Information Technology Reference
In-Depth Information
+
∞
1
4
(
f
E|−
1
(
e
)+
f
E|
1
(
e
))
2
de .
V
R
2
(
E
)=exp(
−
H
R
2
(
E
)) =
(3.15)
−∞
Applying Theorem F.1 (Appendix F), we have
+
∞
f
E|−
1
(
e
)
de
=
+
∞
−∞
f
E|
1
(
e
)
de
=
g
(0; 0
,
√
2
σ
);
−∞
(3.16)
+
∞
2
d,
√
2
σ
)
.
f
E|−
1
(
e
)
f
E|
1
(
e
)
de
=
g
(0;
−
−∞
Hence:
1+exp
.
d
2
σ
2
1
4
√
πσ
V
R
2
(
w
)=
−
(3.17)
As seen in the preceding section, we may apply to
V
R
2
a training algorithm
of maximum ascent, using (3.11) with
2
d
2
σ
2
1
exp
1
;
∂V
R
2
∂σ
d
2
σ
2
1
4
√
πσ
2
=
−
−
−
(3.18)
2
√
πσ
3
exp
.
d
2
σ
2
∂V
R
2
∂d
d
=
−
−
(3.19)
Near the maximum of
V
R
2
, in order to prevent an overshoot with a consequent
decrease of
V
R
2
, one should decrease
η
until obtaining an increased
V
R
2
.
Let us select
d
=1and
σ
=0
.
9 as initial parameter values; Fig. 3.1a
shows the initial
f
E
(
e
). The following Figs. 3.1b through 3.1d show the result,
at iterations 28, 30 and 31, of applying gradient ascent to the information
potential with
η
=0
.
1.
There is a clear convergence of
f
E
(
e
) towards a single Dirac-
δ
at the origin.
(In fact, towards two infinitely close Dirac-
δ
functions; the distinction from
the single Dirac-
δ
is unimportant.) Iteration 31 outputs
d
=
σ
=0
.
19 with
P
e
≈
0. The following iteration 32 produces the same Dirac-
δ
-type solution
with smaller magnitudes of
d
and
σ
, but demands an overshoot correction
with a decrease of
η
; this behavior repeats itself for following iterations. The
converging behavior is essentially the same for other values of the initial
learning rate.
We now select as initial parameter values the same
d
=1but a smaller
σ
=
0
.
4; the initial
f
E
(
e
) is shown in Fig. 3.2a. Gradient ascent with
η
=0
.
01
1
produces the evolution shown by Figs. 3.2b through 3.2d. The convergence is
now towards two Dirac-
δ
functions, one very close to
1 the other very close
to 1 (
d
=0
.
9997). A small departure from the asymptotic solution means in
−
1
The value of
η
influences the convergence rate. The present
η
=0
.
01
value
was chosen so that a convenient number of illustrative intermediary PDFs were
obtained.
