MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 69

weight space
w w w w w

w 2 w 2

stability

instability

p

p

s

s

p

4 s

p

s

x

4

w 2 w 2

w w w w w

steady state

Figure 2.12 MCA EXIN: stability subspaces of the weight space with respect to the input
vector (two-dimensional space). The angle σ is proportional to the weight modulus. The
negative instability bounds are also valid for OJAn and LUO.

The positive scalar st represents the interval of absence of fluctuations
(dynamic stability) and depends on the following quantities:

γ . Near convergence, α

• is very low, because of the Robbins-Monro con-

ditions [161]; then γ

can be taken very low, and this choice improves the

dynamic stability.
• s ( p )

. This quantity has different expressions according to the learning law
[see eq. (2.97)]; as discussed before (see Section 2.6.2.1 and Remark 64),
the initial weight vector must be chosen low; then, after a certain number
of iterations, the weight vector diverges; near convergence,

st for LUO
decreases, so its dynamic stability worsens; on the contrary, the weight
divergence has a positive effect on the dynamic stability of MCA EXIN
(see Figure 2.12, where the angle σ increases for increasing p ).

• x . Data with a high modulus and close to the weight direction worsen

the dynamic stability.

( t )

Remark 72 (Bias and Variance) From the point of view of the variance/bias
dilemma the MCA EXIN learning law is a high variance/low bias algorithm.
Indeed, choosing low initial conditions implies a high variance in the first tempo-
ral part of the weight time evolution, but not at the expenses of the final accuracy,