GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 195

[ A ; b ] with the TLS hyperplane, this neuron solves the problem; after the
convergence, normalization of the weights is needed in order to respect the
constraint (5.143).
2. MCA EXIN . Taking as input each row of U , picked up randomly, this
neuron solves the problem by directly minimizing the Rayleigh quotient of
U T U . Recalling the divergence problem, normalization is needed after the
stopping criterion is satisfied, in order to respect the constraint (5.143).
3. MCA EXIN 0, this neuron solves the problem,

but needs further normalization, as in the other cases.

+ .Taking A = U and b =

The choice of the neuron is problem dependent and is a consequence of its
own properties.

5.7.2 Underdetermined Unidimensional Linear Systems

m

Given the underdetermined unidimensional system Ax = b ,where A ∈ × n ,

m ,and m < n , using the usual notation it follows that

b ∈

rank[ A ; b ] = m ⇒ σ > σ =···= σ = 0 (5.143)

m m + 1 n + 1

(i.e., the last n − m + 1 singular values are null). Recalling the interlacing prop-
erty (1.16), it also follows that the last n − m eigenvalues of A T A are null. The
existence condition is σ m > σ m + 1 , which implies that σ m ≥ σ m > 0. The TLS
problem (1.9) still yields a solution, but it is not unique. Indeed, any linear
combination of singular vectors

v n + 1 solves the TLS problem after
normalization (intersection with the TLS hyperplane). For reasons of stability
and minimal sensitivity, it is desirable to single out the solution with minimal L 2
norm. Recalling eqs. (1.30) and (1.7), the compatibility of the system, given by
σ n + 1

v m + 1 , ...

,

= 0, yields

x = x = ( A T A ) − 1 A T b = A + b (5.144)

as the minimal L 2 norm solution. This norm is
x

= b T AA T − 1 b

2
2

2
2

= x (5.145)

and the corresponding error cost is null [see, e.g., eq. (1.21)]. The following
theorem extends this analysis.

Theorem 130 (Underdetermined GTLS Problems) If the system of linear
equations Ax = b is underdetermined, all the minimal L 2 norm GeTLS solutions
( i.e., ∀ ζ ) are equal.

Proof. Considering that rank[ A ] = m < n ,thelast n − m eigenvalues of A T A
are null. Recalling the theory in Section 5.3.1, the function g (γ , ζ ) has n − m