Information Technology Reference
In-Depth Information
Hence,
α > α b ( x ,
w ( t ))
α b >
if
0
ρ (α) > 1
(2.154)
α >
α b <
0
if
0
Considering that it always holds that α 0, if α b is negative, then r is always
greater than r independent of the value of the learning rate α . Here this phe-
nomenon is called negative instability . The following cases can generate insta-
bility (fluctuations or divergence):
1.
α b < 0 (negative instability). This case is equivalent to
pq u = A ( q , p , u ) < 0
(2.155)
Then there is negative instability when
u
p = x ( t )
2
2 sin 2
2
2 cos 2
q = x ( t )
ϑ x w <
ϑ x w
(2.156)
That is,
tan 2
ϑ x w < 1
(2.157)
which means that there is negative instability for π/ 4 x w <π/ 4and
for
x w < 3 π/ 4. This is illustrated in Figure 2.12, which shows
the instability bounds in the weight space with respect to the input vector
x
π/
3
4
2, because of the def-
inition of MC (the dotted area in Figure 2.12), the negative instability is
typically valid for the transient state and in case of outliers . This study
holds for all three neurons.
2. 0 b γ< 1. In this case, α > α b generates instability. It implies that
too low a value of γ certainly requires a lower learning rate α to avoid this
problem. As a consequence, γ can be considered as an upper bound for the
learning rate. This case is equivalent to A ( q , p , u ) > 0and B ( q , p , u ) > 0,
where
(
t
)
. Considering that in steady state,
ϑ x w →± π/
2 p
s ( p )
B ( q , p , u ) = p ) q
γ u +
Hence, there are no fluctuations (instability) when
u
p < q <
u
p +
1
γ
2
0
(2.158)
s
(
p
)
which becomes
1
γ
2
s ( p )
1
1 < tan 2
ϑ x w < 1 +
= 1 + st
(2.159)
2
x ( t )
2 cos 2
ϑ x w
Search WWH ::




Custom Search