GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 184

Table 5.1 Second nongeneric TLS benchmark: perturbed system

x 1 x 2

2
√ 5 − 1 × 10 − 8

SVD-based nongeneric TLS − 2 . 618

2
√ 5 10 − 6 10 − 5 SCG TLS EXIN (six epochs)

1 +

2 . 65 × 2 . 7557 ×

−

2
√ 5 − 1 − 7 . 2 × 10 − 8 2 . 4267 × 10 − 6 BFGS TLS EXIN (four epochs)

00
00
010 − 8

0
10 − 8
0

and the observation vector with .

the data matrix with

v n + 1, n + 1 = v 3,3 ≈ 10 − 8

The SVD of the perturbed set of equations yields and

σ n − σ n + 1 = O ( 10 − 8

) . The generic TLS solution yields a very sensitive, unsta-

ble solution given by 1, 10 8 + 10 4 T . Hence, the nongeneric TLS solution must

be computed. Using the SVD, the solution is

2
√ 5

T

1 , 2 × 10 − 8

x = √ 5 (5.132)

− − 3

10 − 8 and then the perturbed solution is stable.
Contrary to the SVD-based techniques, TLS EXIN and its acceleration ver-
sions, for null initial conditions, are stable because they solve the nongeneric
problem without changing the learning law. The results obtained (weights after
convergence) are listed in Table 5.1 together with the number of epochs for the
convergence and the SVD-based result. Figure 5.19 shows the phase diagram
of TLS EXIN for different initial conditions. It has the same interpretation as
that for Figure 5.16. In particular, the domain of convergence is given by the
horizontal half-line with origin the maximum ( − 0 . 618034, 0) and containing the
saddle/solution . The divergence line is the asymptote z 2 and is vertical (parallel
to x 2 ).

Note that x 2 =

5.5 SCHEDULING

The ζ parameterization has been conceived to unify and generalize the three
basic regression problems. The continuity of the solutions ∀ ζ (i.e., the solution
locus cited above) suggests a novel numerical technique for the solution of the
basic problems, which further justifies the intermediary ζ cases. This technique
is defined here as scheduling because it requires a predefined programmation of
the parameter ζ . It uses the solution locus just like an attraction locus for the
trajectory of the solution estimate in the x space and is conceived especially for
solving the DLS problem, which is the most difficult.