Information Technology Reference
In-Depth Information
search vector
P
k
to calculate the parameter value
W
1
,
based on a current value
W
as
,
(3.17)
p
WW
D
k
1
k
k
k
where
D
is a scalar value. The search vector
P
k
is to be chosen so that the relation
holds, where
V
VW
is the performance index of the network,
generally a sum square error function.
Now, considering the Taylor series expansion of
V
W
W
k
1
k
V
W
at point
W
k
1
k
T
V
V
P
|
V
.
(3.18)
W
W
D
W
V
P
D
W
kk
k
k
k
1
k
k
k
it is obvious that, in order for the cost function
V
to decrease and for a positive
value of
,
D
the second term of (3.18) must be negative. This will be the case if
the steepest descent condition
(3.19)
WW
D
W
k
1
k
k
k
is met. However, the steepest descent method, as discussed earlier, when used in its
original form, exhibits some drawbacks that need to be eliminated for its practical
use. To overcome this, the approximation of the objective function in the
immediate neighbourhood of a strong minimum by a quadratic function with
positive definite Hessian matrix or by using Newton's method for pursuing the
minimization problem is preferred.
Let us now consider the Taylor series expansion
1
2
T
T
2
T
V
|
V
'
(3.20)
W
W
V
'
W
W
V
'
W
W
W
k
1
k
k
k
k
k
k
2
where
' If the gradient of the
truncated Taylor series expansion (3.20) is taken with respect to
V
W
is the Hessian matrix and
.
W
D
P
k
k
k
k
' and set to
zero (since we are looking for the minimum of the cost function), it follows that
k
1
'
ª
2
V
º
V
.
(3.21)
W
W
W
¬
¼
k
k
k
This reduces the Newton method to
1
ª
2
º
VW
VW
.
(3.22)
WW
¬
¼
k
1
k
k
k
Search WWH ::
Custom Search