INTRODUCTION TO THE TLS EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 124

and

x T x A T A − ( ∇ x E TLS ) x T

− E TLS I n

2

T

H TLS ( x ) = − x ( ∇ x E TLS ) (4.32)

1 +

which, in the critical points, is

1 + x c x c A T A − σ

c I n = h TLS

2

2

H TLS ( x c ) =

As a consequence, the inverse of the Hessian matrix can be computed. This
implies the feasibility of Newton's method, which uses the Newton direction

d ( t ) =− H − 1

∇ E (w( t )) (4.33)

H being the Hessian matrix of E (w) computed at time t . However, the Hessian
matrix must be inverted at each step and it is computationally prohibitive; more
interesting are the quasi-Newton or variable metric methods [157,187,189], which
are based on eq. (4.33), but instead of calculating H directly and then evaluating
its inverse, they build up an approximation to the inverse over a number of steps
(compare with Remark 45). This approach generates a sequence of matrices
G ( t ) which represent increasingly accurate approximations to H − 1 using only
information on the first derivatives of the error function. The two most commonly
used update formulas for G ( t ) are the Davidson-Fletcher-Powell (DFP) and the
Broyden-Fletcher-Goldfarb-Shanno (BFGS) procedures. Here, only the BFGS
is given because it is generally regarded as superior:

w( t + ) = w( t ) + α( t ) d ( t ) = w( t ) − α( t ) G ( t ) ∇ E (w( t ))

1 (4.34a)

G ( 0 ) = I (4.34b)

pp T
p T

T G ( t )

+ v

T G ( t )v uu T

v − ( G ( t )v) v

G ( t + 1 ) = G ( t ) + (4.34c)

v

T G

( t )v

where

p = w( t + 1 ) − w( t ) (4.34d)

v =∇ E (w( t + 1 )) −∇ E (w( t )) (4.34e)

p
p T

G ( t )v

u =

v −

(4.34f)

v

T G ( t )v

where α( t ) is found by line minimization. Derivations of this procedure can
be found in many standard texts on optimization methods, such as Polak [154]
and Luenberger [119]. An important advantage of the quasi-Newton approach
over the conjugate gradient method is that the line search does not need to be
very accurate since it does not form a critical factor in the algorithm (the line
minimizations for the CG must be performed accurately to ensure that the system
of conjugate directions and orthogonal gradients is set up correctly). Hence, the
simple backtracking line search has been implemented for TLS EXIN.