Feature Selection - Data Mining with Decision Trees: Theory and Applications - page 210

For the linear model, many of the popular selection criteria are a
penalized sum of squares criterion that can provide a unified framework
for comparisons. This criterion selects the subset model that minimizes:
RSS γ /σ 2 + Fq γ , (13.3)

where F is a preset “dimensionality penalty”. The above penalizes
RSSγ/σ 2by F times qγ , the dimension of the γ th model. AIC and
minimum Cp are equivalent, corresponding to F = 2, and BIC is obtained
by F =log n . Using a smaller penalty, AIC and minimum Cp will select
larger models than BIC (unless n is very small).

13.3.2.3 Principal Component Analysis (PCA)
PCA is linear dimension reduction technique [ Jackson (1991) ] .PCAbased
on the covariance matrix of the variables, is a second-order method. PCA
seeks to reduce the dimension of the data by finding a few orthogonal
linear combinations (the PCs) of the original features with the largest
variance. The first PC, s 1 , is the linear combination with the largest
variance. We have s 1 = x T w 1 ,wherethe p -dimensional coecient vector
w 1 =( w 1 , 1 ,...,w 1 ,p ) T solves:
w 1 =arg max
w =1

x T w Var { } . (13.4)

The second PC is the linear combination with the second largest
variance and orthogonal to the first PC, and so on. There are as many
PCs as the number of original features. For many datasets, the first several
PCs explain most of the variance, so that the rest can be ignored with
minimal loss of information.

13.3.2.4 Factor Analysis (FA)

FA, a linear method based on the second-order data summaries, assumes
that the measured features depend on some unknown factors. Typical
examples include features defined as various test scores of individuals that
might to be related to a common intelligence factor. The goal of FA is to
find out such relations, and thus it can be used to reduce the dimension of
datasets following the factor model.

13.3.2.5 Projection Pursuit (PP)

PP is a linear method which is more computationally intensive than
second-order methods. Given a projection index that defines the merit of a