GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 159

Recast z n as

z n ≈ Gq n

(5.83)

where

b T b
q n

G = (5.84)

It follows that

G > H (5.85)

Indeed, using property (5.68),

b T b + λ n b T b + λ n

q n + λ

b T b
q n

H = 2

q n <

< = G (5.86)

λ n b T b + 2 λ

n

+

n

Hence, z n > | z n | .
It is easy to extend the inequalities to the other z components. Indeed, ∀ i =
1, ... , n ,

$

z i

q i
λ i

q i
λ i − λ n + H − 1

q i
λ i − 2 γ min ζ I n ≈

( z i )

z sol i (γ min , ζ ) = (5.87)

%

λ i − λ n + G − 1 z i

q i

Thus:
• z i > z i because λ i − λ n + G − 1 <λ i [ G − 1 <λ n for the property (5.68)].

• z i > | because G − 1 H − 1 . z i | <

| z i | > z i because λ i − λ n + H − 1 <λ i . Indeed, | z n | > z n ⇒ H − 1

• <λ n .

These properties, a fortiori, justify eq. (5.80).

5.3.6 Existence of the OLS Solution and the Anisotropy
of the OLS Energy

A direct consequence of property (5.68) is the existence of the OLS solution.
Indeed, from eq. (5.50),

n

q i

b T b g (γ

min ,0

) =

λ i +

2 γ − = 0 (5.88)

min

i = 1

iff b T b ≥ and the zero (energy height at the minimum) is nonnegative

i = 1 q i /λ i (i.e., the above-mentioned property).