GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 152

γ → λ n − 1 / v n − 1 has a value of ζ λ n − 1 / ζ

2 . The infinity in the direction of 2

for the cost function:

E GeTLS EXIN x 0 + s v n − 1 = λ n − 1
2

n

lim
s →±∞

∀ x 0 ∈ (5.54)

ζ

γ → λ n − 1 / because the hyperboloids tend to infinity as 2 ζ .

The density of these equilevel hypersurfaces is proportional to the gap between
λ n / 2 ζ and λ n − 1 / 2 ζ (e.g., for a small gap, the cost landscape is flat). The point
γ = γ ∗ is very important from a geometric point of view because it represents
the passage from a two-sheeted hyperboloid to the opposite one-sheeted hyper-
boloid. Hence, it represents a saddle point. Indeed, every positive k i represents a
minimum in the corresponding direction; so, for levels of the cost function under
the point of level γ ∗ , only the v n direction is a minimum, the others being max-
ima. For levels above γ ∗ ,all A eigenvector directions are minima, just attracting
in the corresponding level range. For ζ = 0 . 5 (TLS), γ ∗ = σ

n .

5.3.2.3 The Case The same considerations of the last

case cited can be made here. g (γ , ζ) is strictly increasing from −∞ for γ =
λ i / 2 ζ to +∞ for γ = λ i − 1 / 2 ζ this implies a zero for g at γ = γ ∗∗ . For each
level of the cost function, the family is composed of hyperboloids (for n ≥
4) with at least 2 positive and negative k i s. Hence, they intersect each of the
hyperplanes z i = const. along some hyperboloid of smaller dimensions (even
one degenerating into a cone). For λ i / 2 ζ ≤ γ ≤ γ ∗∗ , the equilevel hyperboloids
intersect the ( n − i + 1 ) -dimensional plane

λ i / 2 ζ ≤ γ ≤ λ i −1 / 2 ζ

z i − 1 = const i − 1 , z i − 2 = const i − 2 , ... , z 1 = const 1 (5.55)

in a ( n − i ) -dimensional ellipsoid. Hence, the space spanned by v n , v n − 1 , ... , v i
is a minimum for the cost function. For γ ∗∗ ≤ γ ≤ λ i − 1 / 2 ζ ,the k i s change

of sign and now the ( i − 1 ) -dimensional complementary plane of eq. (5.55)

γ = γ ∗∗ represents the level of a saddle. For is a minimum. Thus, ζ = 0 . 5

(TLS), γ ∗∗ = σ

i . The equilevel hyperboloids tend to stretch on the v i direction
as γ → λ i / 2 ζ + andtendtostretchonthe v i − 1 direction as γ → λ i − 1 / 2 ζ n − .
The density of these hypersurfaces is proportional to the gap between λ i / 2 ζ

2

and

λ i − 1 / 2 ζ . The infinity in the direction of v i − 1 has a value of λ i − 1 / 2 ζ for the cost

function:

E GeTLS EXIN x 0 v i = λ

i − 1

n

lim + s ∀ x 0 ∈ (5.56)

2 ζ

s →±∞

because the hyperboloids tend to infinity as γ → λ i − 1 / 2 ζ + .