VARIANTS OF THE MCA EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 92

If the absolute value function is chosen, then

( sign y ( t )) x ( t ) −

α ( t ) | y ( t ) | w ( t )

w

w ( t + 1 ) = w ( t ) − (3.13)

w

T

( t ) w ( t )

T

( t ) w ( t )

and the MCA EXIN neuron is nonlinear because its activation function is the
hard-limiter [eq. (3.6)]. If the logistic function is chosen, then

x ( t ) tanh β y ( t ) −

α ( t ) y ( t ) w ( t ) tanh β y ( t )

w

w ( t + 1 ) = w ( t ) −

w

T

( t ) w ( t )

T

( t ) w ( t )

(3.14)

and the MCA EXIN neuron is nonlinear because its activation function is the
sigmoid [eq. (3.4)]. In the simulations, only the sigmoid is taken in account and
its steepness β is always chosen equal to 1.

Remark 82 (NMCA EXIN vs. MCA EXIN) The MCA EXIN learning law is a
particular case of the NMCA EXIN learning law for f ( d i ) = d i . As a consequence
of the shape of the loss function, for small orthogonal distances ( inputs not too far
from the fitting hyperplane ) , the behaviors of the nonlinear and linear neuron are
practically the same, but for larger orthogonal distances, the nonlinear neuron
better rejects the noise. The nonlinear unit inherits exactly all the MCA EXIN
properties and, in particular, the absence of the finite time ( sudden ) divergence
phenomenon.

Remark 83 (Robusting the MCA Linear Neurons) The same considerations
that give the NMCA EXIN learning law can be applied to OJA, OJAn, and LUO. It
can be shown that these robust versions inherit the properties of the corresponding
linear units. In particular, they still diverge because this phenomenon is not linked
to the activation function chosen, but to the structure of the learning law, which
implies orthogonal updates with respect to the current weight state. Furthermore,
the presence of outliers anticipates the sudden divergence because the learning
law is directly proportional to the squared weight modulus, which is amplified by
the outlier; it increases the difficulty in choosing a feasible stop criterion.

For details on this subject, see [28].

3.2.1 Simulations for the NMCA EXIN Neuron

Here a comparison is made between the NMCA EXIN neuron and the non-
linear neuron of [143-145] (NOJA + ), which is the nonlinear counterpart of
OJA + . Under the same very strict assumptions, NOJA + still converges. The
first simulation uses, as a benchmark, the example in Sections 2.6.2.4 and 2.10,