Graphics Reference
In-Depth Information
eigenvalues; if none of the eigenvalues are complex, then the matrix
A
can be
decomposed in the form
U
T
A
=
U
Λ
(9.5)
where
U
is a matrix of orthogonal eigenvectors, and
is a diagonal matrix of
the eigenvalues. The eigenvectors corresponding to the smallest eigenvalues are
regarded as the least important in PCA. Setting small eigenvalues to zero in Equa-
tion (9.5) results in an approximation to
A
.
Eigenvalue analysis has some limitations: it only works on a square matrix,
only certain square matrices have only real eigenvalues. Furthermore the eigen-
value decomposition is numerically difficult unless the matrix is symmetric. Orig-
inally, principal component analysis was applied to a specially constructed matrix
that had properties amenable to eigenvalue analysis. PCA is generally applied to
amatrix
AA
T
, which is necessarily symmetric. A more general matrix decompo-
sition is the
singular value decomposition
(SVD), which is a factorization of an
arbitrary matrix
M
into a product of the form
Λ
⎡
⎣
⎤
⎦
⎡
⎣
⎤
⎦
=
v
1
v
.
v
n
σ
1
σ
2
=
u
1
u
m
V
T
M
u
2
···
U
Σ
.
(9.6)
.
.
.
σ
n
(The process of computing the SVD is an important problem in numerical linear
algebra.) The diagonal elements of
are the
singular values
; normally the sin-
gular values are arranged in decreasing order of absolute value. The matrices
U
and
V
are orthonormal square matrices consisting of the
left-singular
and
right-
singular
vectors, which are the eigenvectors of
MM
T
and
M
T
M
, respectively.
Figure 9.9 illustrates the concept graphically. The singular value decomposition
can be regarded as an eigenvalue decomposition for a nonsquare matrix
M
.In
fact, there is a direct connection to the eigenvalues of
MM
T
and
M
T
M
:
Σ
V
Σ
Σ
V
T
M
T
M
T
=
,
(9.7)
U
T
U
T
MM
T
=
ΣΣ
.
(9.8)
In other words, the singular values of
M
are the square roots of the eigenvalues of
M
T
M
and
MM
T
;
V
consists of the eigenvectors of
M
T
M
,and
U
consists of the
eigenvectors of
MM
T
.
The application of the SVD to principal component analysis comes from the
interpretation of the singular values. Each singular value
σ
k
corresponds to a left-
singular vector and a right-singular vector, and in a manner of speaking, provides
the “weight” of those vectors. Setting
σ
k
to zero “turns off” the corresponding