Information Technology Reference
In-Depth Information
0.6
V
R
2
0.5
0.4
0.3
0.2
−0.5
w
1
0
2
0.5
1
1
w
2
0
1.5
2
−1
Fig. 3.18 A segment of the Rényi's information potential for the two-dimensional
arctan perceptron
(
w
0
=0
,μ
=1)
with vertical lines signaling the critical points.
2
2
f
E
(e)
f
E
(e)
1.8
1.8
1.6
1.6
1.4
1.4
1.2
1.2
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
e
e
0
0
−3
−2
−1
0
1
2
3
−3
−2
−1
0
1
2
3
(a)
(b)
T
points (Example
Fig. 3.19
Error PDFs of the arctan perceptron at three
[
w
1
w
2
]
3.9): (a) critical point
[0
√
2]
with
V
R
2
=0
.
1995
; (b) critical point
[
2
/
30]
T
T
T
with
V
R
2
=0
.
3455
(solid line) and at
[4 0]
with
V
R
2
=0
.
8815
(dashed line).
Figure 3.18 also shows an infinite maximum at [0 0]
T
corresponding to
σ
=0. This zero-weights maximum is of no practical consequences. For in-
stance, when considering the application of gradient ascent to
V
R
2
(
w
1
,w
2
),at
a certain point the gradient is so high (error PDFs close to Dirac-
δ
functions
at
π
2
) that it will exceed machine range and cause an overshoot to some
other [
w
1
w
2
]
T
point.
We are then left with the only interesting critical point, with
w
2
=0,a
saddle point as predicted by expressions (3.65) to (3.68). From this point
onwards, progressing towards higher
w
1
values along the
w
2
=0direction,
one obtains increasing
V
R
2
(
w
1
,w
2
) values corresponding to error PDFs ap-
proaching two Dirac-
δ
functions at the origin. Figure 3.19 shows the error
PDF at [4 0]
T
, illustrating what happens for growing
w
1
.
±
