MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 42

It implies that far from the origin, the RQ cost landscape becomes flatter, and
therefore in this case all the gradient flow methods are too slow. Summarizing:

Proposition 55 (Saddle Cones) Every saddle direction has an infinity of cones
of attraction, each with centers in this direction and containing the directions
of the linear combinations of eigenvectors with bigger eigenvalues than those of
the saddle eigenvalue. It also has an infinity of cones of repulsion, each with
axes orthogonal to the axes of the cones of attraction, with centers in the saddle
direction and containing the directions (dimensions of escape) of the linear com-
binations of the eigenvectors with smaller eigenvalues than the saddle eigenvalue.

Remark 56 (Low Weights) When using a gradient flow, it is better to work
with low weights, to avoid the flatness of the RQ cost landscape.

Consider now moving from a critical direction (at point P ∗ ) toward a linear
combination of this direction and of any other eigenvector:
w ( t ) = ω i 0 ( t ) z i 0 +
j

ε j ( t ) z j (2.66)

with corresponding energy:

λ i 0

2

+ j = i 0 λ j ε

ω i 0 ( t ) + ε i 0 ( t )

2
j

( t )

E = (2.67)

2
2

w ( t )

If the considered critical direction corresponds to the smaller eigenvalue (MC),
it follows that

λ j = λ i 0 + k j with k j >

0. The energy becomes

λ i 0 ( t ) + e ε j ( t ) 2

j = i 0 k j ε

ω ∗ 2

2
j

( t )

i 0

E =

+

( t ) + j ε ( t ) + j ε

(2.68)

ω ∗ 2
i 0

2
j

ω ∗ 2
i 0

2
j

( t ) ( t )

with e ε j ( t ) = 2 ω i 0 ( t ) ε i 0 ( t ) + j ε

2
j

( t ) being a linear combination of the ε j 's.
The first term on the right-hand side of eq. (2.68) is equivalent to (2.54) in the
neighborhood of the minimum direction, reached for e = 0when ε j = 0 ∀ j = i 0 .
The second term is a convex function of the ε j 's, minimized when ε j = 0 ∀ j = i 0 .
Hence, the entire right-hand side is minimized when ε j = 0 ∀ j = i 0 . It implies
that this critical direction is a minimum in any of the directions considered. If the
critical direction does not correspond to the minimum eigenvalue, the condition
k j > 0 is no longer valid and the critical direction is not a minimum for the
energy E .

Proposition 57 (General Facts About Stability) The Rayleigh quotient
( energy E ) of the autocorrelation matrix R has n critical directions in the