GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 187

DLS

DLS basin

initial conditions TLS

TLS basin

increasing

z

attraction locus

OLS

OLS equilevel hypersurfaces

Figure 5.20 DLS scheduling. ( See insert for color representation of the figure .

= 1 ζ + x T x

2
2

where y (ζ ) = ( 1 − ζ ) + ζ x − 1, it is easy to verify the fol-

lowing properties:
1. If the weight vector x is internal to the unit hypersphere, then the GeTLS
EXIN energy is directly proportional to ζ . In particular,

x 2 < 1 ⇒ E DLS > E TLS > E OLS (5.136)

2. If the weight vector x is on the unit hypersphere, then the GeTLS EXIN
energy is invariant with respect to ζ. In particular:

x 2 = 1 ⇒ E DLS = E TLS = E OLS (5.137)

3. If the weight vector x is external to the unit hypersphere, then the GeTLS
EXIN energy is inversely proportional to ζ . In particular,

x 2 > 1 ⇒ E DLS < E TLS < E OLS (5.138)

From these properties it is evident that it is necessary to have low weights after
the first iteration and a scheduling for weight vectors into the unit hypersphere,
because at a certain weight position x , the energy value rises for increasing ζ ,
thus accelerating the method.

Proposition 123 (DLS Scheduling) A scheduling of the parameter ζ , defined
as a continuous function from 0 to 1, combined with not too high a learning rate,
accelerates the DLS EXIN neuron and guarantees its convergence in the absence