Neural Networks Approach - Computational Intelligence in Time Series Forecasting - page 98

Direct practical use of this method, however, is hampered by the need for
Hessian matrix calculation, whose elements are the second derivatives of the
performance index with respect to the parameter vector. To overcome this obstacle,
the first and the second derivatives of the performance index

() N

VW

2

w

T

e (3.23)

¦ e e w w

k i k k k

i 1

are built and expressed as

Vw

T

w ew (3.24)

J

k k k

and

N

2 T 2

Vw J w Jw w e w , (3.25)

¦ e

k k k i k i k

i 1

where J ( w k ) is the Jacobian matrix and

ew T Y w , (3.26)

k k

with the target vector T and the actual output of the neural network Y ( w k ).
The Gauss-Newton modification of the method assumes that the second term in
the right-hand side expression of (3.25) is zero. Therefore, applying the former
assumption (3.22) yields the Gauss-Newton method as

1

T T

ª

J º e , (3.27)

WWJ wwJ ww

¬ ¼

k 1 k k k k k

An additional difficulty appears here with when the Hessian matrix is not
positive definite, i.e. its inverse does not exist. In this case the modification of the
Hessian matrix

2

GV P I (3.28)

w

k

should be considered. Suppose that the eigen-values and the eigen-vectors of

are the sets ^ `

O and ^ `

2

V W

z

respectively. Multiplying both sides of

k

(3.28) by z i we have

2

GV P I P (3.29)

z w z z O z z

i k i i i i i

G z P (3.30)

O z

i i i