MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 39

approximation assumption). Consider now a perturbation from one point P ∗ in
the critical direction i 0 to the critical direction i 1 :

w ( t ) = ω i 0 ( t ) z i 0 + ε i 1 ( t ) z i 1

(2.57)

The corresponding cost error change is given by

2

ε

i 1 (

t )

λ i 1 − λ i 0

E = E − E ∗ = (2.58)

2
2

w ( t )

Thus, the stability of the critical direction is determined by the difference between
the eigenvalues associated with the corresponding eigenvectors. If the critical
direction corresponds to the minor component direction, E is always positive.
However, E can become negative when moving in any direction, corresponding
to an eigenvector with a smaller eigenvalue than the critical direction considered.
This demonstrates the following proposition.

Proposition 53 (Stability 1) The critical direction is a global minimum in the
direction of any eigenvector with a larger eigenvalue and a maximum in the direc-
tion of any eigenvector with a smaller eigenvalue.

Consider moving from a critical direction (at point P ∗ ) toward a linear com-
bination of other eigenvectors. It implies that
w ( t ) = ω i 0 ( t ) z i 0 +
j i 0 ε j ( t ) z j

(2.59)

=

and

( t ) λ j − λ i 0
w ( t )

j = i 0 ε

2
j

E ∗ =

E = E − (2.60)

2
2

Proposition 54 (Stability 2) The critical direction is a global minimum in the
direction of any linear combination of eigenvectors with larger eigenvalues and a
maximum in the direction of any linear combination of eigenvectors with smaller
eigenvalues.

If the critical direction considered corresponds to the minor component direc-
tion, E will always be positive. However, for any other eigenvector direction,
it is possible to find a neighborhood such that E is negative.
Consider the three-dimensional weight vector space ( n = 3); there are three
critical directions: the minimum direction (parallel to z 3 ), the saddle direction