GENERALIZATION OF LINEAR REGRESSION PROBLEMS - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 154

80

70

60

50

40

30

20

10

0

− 10

− 20

4

2

0 − 1.5

− 1

x − 0.5

0

0.5

2

− 1

1.5

y

Figure 5.1 TLS cost landscape and corresponding equilevel curves for a benchmark
overdetermined system of the two unknowns x and y .

where x , x ,and x are, respectively, the OLS, TLS, and DLS solution. The TLS
cost landscape for this problem is given in Figure 5.1. Its critical points are

min = [0 . 8544, 0 . 2587] T (5.59a)

saddle = [ − 1 . 4442, 0 . 9037] T (5.59b)

max = [ − 0 . 5629 ;− 2 . 0061] T (5.59c)

The equilevel curves are better visualized in Figure 5.2, which also shows the
critical point locations and one section of the cost landscape along the straight line
passing through the saddle and the minimum, which is a valley in the maximum
direction (see Remark 93). Observe the barrier given by the saddle in this section:
a gradient method with initial conditions on the left of the saddle cannot reach
the minimum.

5.3.4 Analysis of the Convergence Keys

Theorem 101 (First Characterization) The convergence keys are the compo-
nents of the OLS solution x =

A + b with respect to the unidimensional eigenspaces
of A T A, weighted by the corresponding eigenvalues.

Proof. From the definition of the vector v j as a right singular vector of A ,

A T A v j = λ j v j