GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 203

Table 5.5 Comparison of TLS GAO with TLS EXIN for the TLS benchmark problem
α 0 γ Iterations

TLS EXIN 0.001 0 1718

TLS GAO 0.001 0 2132

TLS EXIN 0.01 0 367

TLS GAO 0.01 0 431

shows the weight phase diagram of this simulation. The weights approach the
solution locus (parameterized in the figure by ) just before the TLS solution,

because the hyperbolic scheduling yields only a single learning step for

ζ

0.
Note the attractive effect of the solution locus, which confirms the analysis in
Section 5.5. Figures 5.30, 5.31 and 5.32 show the weight vector dynamics of the
accelerated versions of GeTLS EXIN for, respectively, the OLS, TLS, and DLS
problems. The initial conditions are null. The DLS is solved by using ζ = 0 . 9999,
thus allowing null initial conditions. Note the very good transients, especially for
the BFGS approach, which also has a very fast convergence: 0.1 ms for OLS,
TLS, and DLS (1 epoch = 5 iterations). The SCG and BFGS learning laws stop
differently; see Section 5.7.4 for the stop criteria chosen.
Table 5.5 shows a comparison between TLS EXIN and TLS GAO for null
initial conditions. TLS GAO has a better transient and a faster convergence.
The differences are smaller for greater α( t ) = α 0 / t γ because it increases the
randomness of the TLS EXIN law, thus giving an effect of the same type of
linearization for the TLS GAO.

ζ =