Database Reference
In-Depth Information
Figure 11.10 Dropping the lowest singular value from the decomposition of Fig. 11.7
The resulting matrix is quite close to the matrix M ′ of Fig. 11.8 . Ideally, the entire differ-
ence is the result of making the last singular value be 0. However, in this simple example,
much of the difference is due to rounding error caused by the fact that the decomposition
of M ′ was only correct to two significant digits.
11.3.4
Why Zeroing Low Singular Values Works
The choice of the lowest singular values to drop when we reduce the number of dimensions
can be shown to minimize the root-mean-square error between the original matrix M and its
approximation. Since the number of entries is fixed, and the square root is a monotone op-
eration, we can simplify and compare the Frobenius norms of the matrices involved. Recall
that the Frobenius norm of a matrix M , denoted M , is the square root of the sum of the
squares of the elements of M . Note that if M is the difference between one matrix and its
approximation, then M is proportional to the RMSE (root-mean-square error) between
the matrices.
How Many Singular Values Should We Retain?
A useful rule of thumb is to retain enough singular values to make up 90% of the energy in Σ. That is, the sum of the
squares of the retained singular values should be at least 90% of the sum of the squares of all the singular values. In
Example 11.10 , the total energy is (12.4) 2 + (9.5) 2 + (1.3) 2 = 245.70, while the retained energy is (12.4) 2 + (9.5) 2 =
244.01. Thus, we have retained over 99% of the energy. However, were we to eliminate the second singular value, 9.5,
the retained energy would be only (12.4) 2 /245.70 or about 63%.
To explain why choosing the smallest singular values to set to 0 minimizes the RMSE
or Frobenius norm of the difference between M and its approximation, let us begin with
a little matrix algebra. Suppose M is the product of three matrices M = PQR . Let m ij , p ij ,
q ij , and r ij be the elements in row i and column j of M , P , Q , and R , respectively. Then the
definition of matrix multiplication tells us
Search WWH ::




Custom Search