Information Technology Reference
In-Depth Information
Hence,
α > α
b
(
x
,
w
(
t
))
α
b
>
if
0
ρ (α)
>
1
⇔
(2.154)
α >
α
b
<
0
if
0
Considering that it always holds that
α
≥
0, if
α
b
is negative, then
r
is always
greater than
r
independent of the value of the learning rate
α
. Here this phe-
nomenon is called
negative instability
. The following cases can generate insta-
bility (fluctuations or divergence):
1.
α
b
<
0 (negative instability). This case is equivalent to
pq
−
u
=
A
(
q
,
p
,
u
) <
0
(2.155)
Then there is negative instability when
u
p
=
x
(
t
)
2
2
sin
2
2
2
cos
2
q
=
x
(
t
)
ϑ
x
w
<
ϑ
x
w
(2.156)
That is,
tan
2
ϑ
x
w
<
1
(2.157)
which means that there is negative instability for
−
π/
4
<ϑ
x
w
<π/
4and
for
<ϑ
x
w
<
3
π/
4. This is illustrated in Figure 2.12, which shows
the instability bounds in the weight space with respect to the input vector
x
−
π/
3
4
2, because of the def-
inition of MC (the dotted area in Figure 2.12), the negative instability is
typically valid for the
transient state
and in case of
outliers
. This study
holds for all three neurons.
2. 0
<α
b
≤
γ<
1. In this case,
α > α
b
generates instability. It implies that
too low a value of
γ
certainly requires a lower learning rate
α
to avoid this
problem. As a consequence,
γ
can be considered as an upper bound for the
learning rate. This case is equivalent to
A
(
q
,
p
,
u
) >
0and
B
(
q
,
p
,
u
) >
0,
where
(
t
)
. Considering that in steady state,
ϑ
x
w
→±
π/
2
p
s
(
p
)
B
(
q
,
p
,
u
)
=
(γ
p
)
q
−
γ
u
+
Hence, there are no fluctuations (instability) when
u
p
<
q
<
u
p
+
1
γ
2
0
≤
(2.158)
s
(
p
)
which becomes
1
γ
2
s
(
p
)
1
1
<
tan
2
ϑ
x
w
<
1
+
=
1
+
st
(2.159)
2
x
(
t
)
2
cos
2
ϑ
x
w
Search WWH ::
Custom Search