Geology Reference
In-Depth Information
o
E
o
z
j
¼
z
j
t
j
for 1
j
n
ð
4
:
20
Þ
In the neighborhood of a minimum the error surface, E(w) is approximately
quadratic, considering the second-order Taylor expansion of E(w) around the
minimum point w
*
:
1
2
ð
w
w
Þ
T
T
ð
w
Þþð
w
w
Þ
ð
w
Þþ
H
ð
w
w
Þ
E
ð
w
Þ
¼
E
r
E
ð
4
:
21
Þ
where H is the Hessian matrix. At the minimum w
*
, the linear second term is
eliminated because
ð
w
Þ
¼
r
E
0 and the quadratic error function can be rewrite as
1
2
ð
w
w
Þ
T
E
ð
w
Þ
¼E
ð
w
Þþ
H
ð
w
w
Þ
ð
4
:
22
Þ
One could consider the above-mentioned quadratic error surface minimization
algorithms such as CG descent and quasi-Newton methods (BFGS). Both CG
descent and quasi-Newton method-based BFGS algorithms compute the Hessian
matrix to perform the minimization.
4.4.1 Conjugate Gradient Algorithm
The CG descent approach uses past gradient measures to improve the minimization
process. The main functional difference of this CG minimization from conventional
steepest descent minimization is that it requires only two steps to minimize the
function, whereas the steepest descent minimization requires many more. A detailed
description CG descent can be found in Bishop [
10
]. The basic concept of CG
minimization can found in Fig.
4.10
.
Figure
4.10
shows a minimization along d
j
which is achieved when point w
j+1
is
reached. At this stage, a new conjugate direction d
j+1
is chosen (the gradient parallel
to the direction d
j
remains zero). In Fig.
4.10
, the dotted curve represents the
contours of the function.
d
j
þ
1
H
d
j
¼
0
ð
4
:
23
Þ
of the error surface at the
next point becomes a minimum in the current search direction d
j
. This would help
to achieve successive conjugate search directions. In the above case, H is the
Hessian matrix evaluated at the point w
j+1
.
When choosing successive search directions d
j+1
, it is possible to express (
4.23
)
for a set of conjugate search directions (up to the dimensionality W of the weight
space):
When the above case satis
es, the gradient g
r
E
ð
w
Þ
