MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 45

Equation (2.76) describes an n -dimensional dynamical system whose equilibrium
points are defined by ω i ( t + 1 ) = ω i ( t ) . In this case this property holds only if
ω j = 0 ∀ j = i 0 (i.e., all the R eigenvectors are equilibrium directions). Consider
a weight perturbation from an equilibrium direction as in eq. (2.57) [i.e. w ( t ) =

ω i 0 ( t ) e i 0 + ε i 1 ( t ) e i 1 ]. In this case, eq. (2.76) yields

# %

i 1 ( t ) λ i 1 − λ i 0

2

ε

$ & ω i 0 ( t )

ω i 0 ( t + ) = + α ( t )

1 1 (2.77)

2

2 2

i 0 ( t )

ω

i 0 ( t ) + ε

# %

) λ i 0 − λ i 1

2

ω

i 0 (

t

$1 + α ( & ε i 1 (

ε i 1 ( t + 1 ) = t ) t ) (2.78)

2

2
i 0

2
i 0

ω ( t ) + ε ( t )

For λ i 0 > λ i 1 , it is possible to find a small enough ε i 1 ( t ) such that the state
of the dynamical system will be farther from the equilibrium direction at time
t + 1thanattime t [i.e., ω i 0 ( t + 1 ) ≈ ω i 0 ( t ) and ε i 1 ( t + 1 ) > ε i 1 ( t ) ]. This result
confirms the fact that because the equilibrium direction considered is a local
maximum in the direction considered, and because the algorithm is a gradient
descent, the equilibrium must be unstable in this direction: Equilibrium directions
are unstable in the direction of the eigenvectors with smaller eigenvalues. This
result will remain valid for any linear combination of such eigenvectors. Around
an equilibrium direction associated with an eigenvector, consider the region of
weight space spanned by the eigenvectors with bigger eigenvalues. Recalling
the basic assumption of the first approximation, in this region the continuous
function E has a unique minimum at the equilibrium direction and is strictly
decreasing since the dynamical system follows the gradient of E . Therefore, E is
a Lyapunov function (this fact will be used in the proof of Theorem 60). Hence,
the Lyapunov theorem justifies the fact that the equilibrium direction considered
is asymptotically stable in the region of the space considered. Furthermore, for
the equilibrium direction associated with the smallest eigenvalue, this region
becomes the entire space (i.e., the global minimum is asymptotically stable in
the n -dimensional weight space).
Remark 59 (Extension) By considering the fact that OJAn and LUO have the
same learning law structure as MCA EXIN [ see eq. ( 2.37 )] , all the theory about
MCA EXIN ODE stability presented here is valid for them. Indeed, the conse-
quences of the learning law are the same, because eqs. ( 2.75 ) to ( 2.78 ) would
differ for a positive ( and then irrelevant ) quantity.

2.5.3 Convergence to the Minor Component

This section deals with the asymptotic behavior of the MCA EXIN ODE, which
can be considered only in the limits of validity of this asymptotic theory (see
[118, Th. 3 and discussion, p. 556]).