TOTAL LEAST SQUARES PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 22

rate depends not only on the motor gap but also on the relative spread of the
undesired singular value spectrum. If it is applied to S − 1 [the inverse Lanc-
zos method(ILZ)], it has the same convergence properties as ICI with optimal
bounds. However, round-off errors make all Lanczos methods difficult to use in
practice [75]. For a summary, see Table 1.1.

1.13.5 Rayleigh Quotient Iteration

The Rayleigh quotient iteration (RQI) can be used to accelerate the convergence
rate of the inverse iteration process when the motor gap is small (see Table 1.1),
no adequate shift 0 can be computed, and convergence to only one singular

vector of the desired singular subspace is required. RQI is a variant of II by
applying a variable shift λ 0 ( k ) , which is the RQ r ( q k ) (see Section 2.1) of the
iteration vector q k ,definedby

λ

q k Sq k
q k q k

f (λ) = ( S − λ I ) q k 2 r ( q k ) = minimizing (1.58)

Then the RQI is given by

( S − λ 0 ( k ) I ) q k = q k − 1 (1.59)

With regard to II, RQI need not estimate the fixed shift λ 0 , because it gen-
erates the best shift in each step automatically and therefore converges faster.
Parlett [149] has shown that the RQI convergence is ultimately cubic . However,
RQI requires more computations per iteration step than II. Furthermore, RQI
applies only to square-symmetric matrices S and it is impossible to avoid the
explicit formation of S = C T C , which may affect the numerical accuracy of the
solution. A good start vector q 0 should be available to allow the RQI to converge
to the desired solution. For example, most time-varying problems are character-
ized by abrupt changes at certain time instants that seriously affect the quality
of the start vector and cause RQI to converge to an undesired singular triplet
(see [98]). If convergence to several basis vectors of the desired singular sub-
space is required, RQI extensions must be used (e.g., inverse subspace iteration
with Ritz acceleration) [48].

1.14 RAYLEIGH QUOTIENT MINIMIZATION NONNEURAL
AND NEURAL METHODS

From eqs. (1.22) and (1.21) it is evident that E TLS ( x ) corresponds to the Rayleigh
quotient (see Section 2.1) of [ A ; b ], and therefore the TLS solution corresponds
to its minimization. This minimization is equal to the minor components anal-
ysis(MCA), because it corresponds to the search of the eigenvector associated
with the minimum eigenvalue of [ A ; b ], followed by a scaling of the solution
into the TLS hyperplane. This idea will be explained at greater length in the