VARIANTS OF THE MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 102

minimum. MSA EXIN is faster than MSA LUO in approaching the minor
subspace for initial weights of modulus less than 1.
• MC direction. The weight vectors of the driving neurons for both neural
networks diverge, fluctuating around the first MC direction. Weight moduli
soon become larger than unity. Hence, the fluctuations, which are orthogonal
to the first MC direction, as shown in Section 2.6.2, are larger for LUO than
for EXIN, because the latter has the positive smoothing effect of the squared
weight norm at the denominator of the learning law. Hence, its weight vector
remains closer to the first MC direction. The driven neurons have inputs with
nonnull projections in the lower subspaces.
• Loss of step. If the driving neurons have too large (with respect to MC)
orthogonal weight increments, the driven neurons lose their orthogonal direc-
tion, and their associated minima become saddles again. So the weights tend
to the minor component directions associated with the lower eigenvalues
until they reach the first minor component direction. This loss of the desired
direction is called a loss of step, in analogy with electrical machines. As a
result, this loss propagates from neuron N 1, and

the corresponding weights reach the first MC direction in the same order.
Because of its larger weight increments, MSA LUO always loses a step
before MSA EXIN. Furthermore, MSA EXIN has weight vectors staying
longer near their correct directions. 1
• Sudden divergence. The driving neuron of MSA EXIN diverges at a finite
time, as demonstrated in Section 2.6.2. The other neurons, once they have
lost a step, have the same behavior as that of the driving neuron (i.e., they
diverge at a finite time).

− 1toneuron N − M +

As a consequence of its temporal behavior, MSA LUO is useless for applica-
tions both because their weights do not stay long in their correct directions and
for the sudden divergence problem.

3.3.2.1 Simulations The first example uses, as benchmark, Problem 6.7 in
[121, p. 228]. Here,

2 . 7262 1 . 1921 1 . 0135 − 3 . 7443

" #

1 . 1921 5 . 7048 4 . 3103 − 3 . 0969

R =

1 . 0135 4 . 3103 5 . 1518 − 2 . 6711

− 3 . 7443 − 3 . 0969 − 2 . 6711 9 . 4544

has distinct eigenvalues λ 1 <λ 2 <λ 3 <λ 4 ,where λ 1 = 1 . 0484 with corre-
sponding eigenvector z 1 = [0 . 9052, 0 . 0560, − 0 . 0082, 0 . 4212] T and λ 2 = 1 . 1048
with corresponding eigenvector z 2 = [ − 0 . 0506, 0 . 6908, − 0 . 7212, 0 . 0028] T .
MSA EXIN has common learning rate

α( t ) = const = 0 . 001 . The initial

1 This phenomenon is different from the loss of step, but is not independent.