INTRODUCTION TO THE TLS EXIN NEURON - Neural-based Orthogonal Data Fitting: The EXIN Neural Networks - page 122

If the CG local quadratic approximation is poor, the following prescription for
λ( t ) is given 59:

λ( t + 1 ) = λ( t )
2

( t ) > 0 . 75 if then (4.20)

if ( t )< 0 . 25 then λ( t + 1 ) = 4 λ( t ) (4.21)

where

E (w( t )) − E (w( t ) + α( t ) d ( t ))
E

( t ) = (4.22)

(w( t )) − E Q (w( t ) + α( t ) d ( t ))

where E Q (w( t )) is the local quadratic approximation to the error function in the
neighborhood of the point w( t ) .If ( t )< 0, the weights are not updated. The
two stages of increasing λ( t ) aim to ensure that the denominator is positive and
to validate that the local quadratic approximation is applied in succession after
each weight update. If the conjugacy of the search directions tends to deteriorate,
the algorithm is restarted by resetting the search vector to the negative gradient
direction. The SCG approach is fully automated because it includes no criti-
cal user-dependent parameters. In a great many problems it offers a significant
improvement in speed compared to conventional CG algorithms. In [133] it is
asserted that it is faster than the BFGS method. In the simulations for the SCG
TLS EXIN and the BFGS TLS EXIN, the contrary will be shown.

4.3.2 BFGS TLS EXIN

Theorem 87 (Hessians) The n × n Hessian matrix of the TLS cost function eq.
( 4.7 ) is positive definite ( which implies the nonsingularity ) at the cost function
minimum.

Proof. The Hessian matrix of the Rayleigh quotient r ( u , C ) , given in eq.

(2.10), is

C − ( ∇ r ) u T

− rI

2
u

T

H r = − u ( ∇ r ) (4.23)

2
2

In the TLS approach, the Hessian matrix of the Rayleigh quotient r u ,[ A ; b ] T
[ A ; b ] ) constrained to
u n + 1 =− 1 ( u TLS = x T ;− 1 T

n

x ∈ ) (4.24)

is

[ A ; b ] T [ A ; b ]

− EI n + 1 − ( ∇ E ) x T 1 − x T 1 T

2
1 + x T x

T

H TLS = ;− ;− ( ∇ E )

A T A − EI n − ( ∇ E ) x T

T

A T b −∇ E − kx

2
1 + x T x

− x ( ∇ E )

=

b T A kx T

T

b T b

− + ( ∇ E ) − E + 2 k