Information Technology Reference
In-Depth Information
The selection of the main components ( q out of p ) results from an analysis of
the eigenvalues. Before describing it, it is worthwhile reviewing a technique,
similar to PCA, which is extensively used in linear algebra: singular value
decomposition (SVD) [Cichoki 1993]. That technique, which is very useful for
solving linear systems, was mentioned in the previous chapter as a tool for
calculating leverages for nonlinear models.
R n×p matrices, there exist two orthogonal matrices
Theorem. For al l A
R n×p and V
R p×p such that
U
σ 1
0
···
0
0 σ 2
···
0
U T AV = S =
,
.
. . .
. . .
0
0
···
0 σ m
with σ 1
σ 2 ≥···≥
σ m
0 ,wherem =min( p,n ) .
The elements of the diagonal matrix S are the singular values σ j ,ranked
in decreasing order. The singular values σ j are the square roots of the eigen-
values λ j of the positive symmetrical matrix A T A or matrix AA T if m<n .
The columns of the matrix V associated with the change of variables are the
eigenvectors of matrix A T A .
PCA and SVD
Therefore, PCA and SVD are equivalent when operated on centered data.
Unlike diagonalization techniques for square matrices, singular value de-
composition applies to all types of matrices. The index for the 1st singular
value equal to 0 is the rank of the matrix; its condition number is the ratio of
the largest to the smallest singular value σ 1 p .
From the orthogonality of matrices U and V, one has
U T AV = S
A = USV T .
In a modeling application, if A is the matrix of centered observations (defined
in the previous chapter), matrix US = AV describes the same observations in
an “orthogonal” representation: the new inputs obtained after transformation
are not subject to linear correlation. The same technique is used for “cleaning”
signals [Davaud 1991]. In order to reduce the new inputs, matrix U is retained
as a new base of examples: the linear transformation thus becomes S 1 V T x
instead of V T x .
Singular value decomposition of the matrix of centered data X is used to
express the inertia with respect to the singular values σ j
or the eigenvalues
of matrix X T X ,
λ j
p
p
I p =Tr X T X
σ 2
j
I p =
λ j
I p =
.
j =1
j =1
Search WWH ::




Custom Search