Neuro-fuzzy Approach - Computational Intelligence in Time Series Forecasting - page 233

6.4.2.3 Levenberg-Marquardt Training Algorithm
Training experiments with a neuro-fuzzy network using the momentum version of
backpropagation algorithm, as well as its modified error index extension form,
have shown that, with the first 200 training (four inputs- one output) data sets of a
Mackey-Glass chaotic series, backpropagation algorithm usually requires several
hundred epochs to bring the SSE value down to the desired error goal (see Palit
and Popvic, 1999). This calls for an alternative, much faster training algorithm.
Hence, to accelerate the convergence speed of neuro-fuzzy network training, the
Levenberg-Marquardt algorithm (LMA) was proposed.
Although being an approximation to Newton's method, based on a Hessian
matrix, the Levenberg-Marquardt algorithm can still implement the second-order
training speed without direct computation of the Hessian matrix (Hagan and
Menhaj, 1994). This is achieved in the following way.
Suppose that a function V ( w ) is to be minimized with respect to the network's
free-parameter vector w using Newton's method. The update of w to be used here
is

> @

1

' w

2

V w (6.21a)

V w

w

w k 1 w k ' (6.21b)

is the Hessian matrix and

2

where is the gradient of V ( w ) . If the

function V ( w ) is taken to be the sum squared error function, i.e .

V w w

N

2

V w 0 . 5 ¦ w (6.22)

e

r

r 1

then the gradient

2

and the Hessian matrix w V ( w ) are generally defined

using the Jacobian matrix J ( w ) as

T

V w w e w (6.23a)

J

N

2 T 2

V w w J w w w , (6.23b)

J ¦ e e

r r

r 1

where