GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 181

initial conditions for every TLS gradient flow, just like TLS EXIN, assures the
convergence.

Proposition 122 (Divergence Straight Line) In every plane z n z i , the diver-
gence straight line is between the saddle/minimum and the maximum.

5.4.2 Analysis of Convergence for p < n

In the case where +

1) and all v n + 1, i = 0, i = p + 1, ... , n + 1, there are n − p + 1 critical points
going to infinity in the direction of the eigenvectors associated with the eigen-
values of the corresponding coincident asymptotes. Hence, there is an n − p + 1
divergence plane spanned by these eigenvectors. The complementary subspace
( nongeneric TLS subspace )isthe( p − 1)-dimensional plane through the origin
and orthogonal to the divergence plane. All the reasonings about the case p = n
are also valid here, just recalling that in this case, the lower saddle (minimum) cor-
responds to σ p . In particular, q p = q p + 1 =···= q n = 0. In the space (MCA
vector space), the ( n + 1)-dimensional vectors v p + 1 , ... , v n + 1 are parallel to the
TLS hyperplane.

σ > σ =···= σ 1 , p ≤ n ( σ 1 has multiplicity n − p

p p + 1 n + n +

5.4.3 Simulations
The first benchmark set of equations ([98, Ex. 3.1]) is

√ 6 √ 6

√ 2
4

x 1
x 2

√ 2
4

2
0

−

(5.128)

≈

− √ 3

√ 2
2

√ 2
2

v 3 = [ √ 2 / 2, − √ 2 / 2, 0] T . v n + 1, n + 1 = v 3,3 = 0.

Here, Note that The solution

(using the SVD) is

1
√ 6 ,

T

1
√ 6

x = (5.129)

being x T ;− 1 T

⊥ v 3 . All experiments with TLS EXIN and TLS GAO use a
constant learning rate equal to 0.05. Figure 5.16 shows the phase diagram of
TLS EXIN for different initial conditions: the red (black) initial conditions give
convergent (divergent) trajectories. The black straight line passing through the
red points contains the saddle/solution locus and coincides with the axis z 1 .
The domain of convergence is given by the half-line with origin the maximum
( − 1 . 2247, − 1 . 2247) and containing the saddle/solution . All the nonconverging
trajectories tend to the divergence straight line (dark blue thick line), which is