GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 160

For γ > γ min , the equilevel family is only composed of hyperellipsoids [( n −
1)-dimensional ellipsoids] with axes parallel to the A eigenvectors, the same
center,

x c = x = V − 1 q (5.89)

and analytical expression

z T

z = 2 (γ − γ ) (5.90)

min

Hence, the squared semiaxes k i s [see eq. (5.52)] are given by

2 γ − γ

min

k i = (5.91)

λ i

From the point of view of a neural algorithm in the form of a gradient flow, it
is better to have isotropy of the hyperellipsoids just to avoid preferred unknown
directions. It implies a clustering of the squared singular values of A . Too distant
λ i s, as in the case of an ill-conditioned matrix ( λ λ n ), give strong hyperel- 1

lipsoid anisotropy.

5.3.7 Barrier of Convergence

5.3.7.1 Two-Dimensional Case For λ 2 / 2 ζ ≤ γ ≤ λ 1 / 2 ζ , the family of
equilevel curves for the cost function is made up of hyperbolas with focal axes
parallel to the v 2 direction for heights less than the saddle height and of hyper-

bolas with focal axes parallel to the v 1 direction for heights more than the saddle

height. This last subfamily is important from the point of view of the convergence
domain of the steepest descent learning law. In fact, the locus of the intersections
of the hyperbolas with the corresponding focal axes gives the crest for the cost
function (i.e., the barrier for the gradient method). Note that this analysis does not
take into account the influence of the maximum on the crest, which is valid for
the inferior crest branch (i.e., the branch toward the maximum). Therefore, the
following analytical considerations are exact only for the superior crest branch
and for the part of the inferior crest branch nearer the saddle point. Figure 5.3
shows the benchmark problem (5.57).
For γ ≤ γ ≤ λ / 2 ζ the hyperbolas have the analytical expression [see saddle 1

eq. (5.52)]

2

z i
k i =

z 1
k 1 +

z 2
k 2 = 1

(5.92)

i = 1

where k i = g (γ , ζ ) /λ i − 2 γζ ( k 2 < 0, k 1 > 0). The intersection with the z 1 axis

is given by √ k 1 . Recalling that

V z

x = + x c (5.93)