Information Technology Reference
In-Depth Information
Section 2.5.3). This study will also lead to the very important problem of sudden
divergence [181].
The following analysis is original, rethinks and criticizes the existing theory,
but above all, proves the superiority of MCA EXIN in the most general MCA
problems.
2.6.1 Against the Constancy of the Weight Modulus
2.6.1.1 OJAn, LUO, and MCA EXIN
The OJAn, LUO, and MCA EXIN
stochastic discrete learning laws
have a similar analysis, because of their common
structure, here repeated for convenience:
w (
t
+
1
)
=
w(
t
)
+
δw(
t
)
=
w (
t
)
−
α (
t
)
S
neuron
(w (
t
)
,
x
(
t
))
y
(
t
)
x
(
t
)
−
y
2
(
t
p
w(
t
)
=
w (
t
)
+−
α (
t
)
s
(
p
)
(2.96)
where
!
p
for LUO
1
for OJAn
s
(
p
)
=
(2.97)
1
p
"
for EXIN
2
and
p
=
w (
t
)
2
. They have in common the following property:
y
(
t
)
x
(
t
)
−
y
2
(
t
)
T
w
(
t
)
(
t
) w (
t
)
w(
t
)
=
0
(2.98)
T
w
That is,
the weight increment at each iteration is orthogonal to the weight direc-
tion
. This arises from the fact that they are gradient flows of the Rayleigh quotient
and exploit the property of orthogonality (2.4). The squared modulus of the weight
vector at instant
t
+
1isthengivenby
!
2
+
α
(
t
)
4
2
2
2
2
4
2
sin
2
2
ϑ
x
w
w (
t
)
w (
t
)
x
(
t
)
for OJAn
2
+
α
(
t
)
4
2
2
w (
t
+
1
)
=
2
2
6
2
4
2
sin
2
2
ϑ
x
w
w (
t
)
w (
t
)
x
(
t
)
for LUO
"
2
+
α
(
t
)
4
2
2
w (
t
)
−
2
2
4
2
sin
2
2
ϑ
x
w
w (
t
)
x
(
t
)
for MCA EXIN
where
ϑ
x
w
is the angle between the directions of
x
(
t
)
and
w (
t
)
.Fromthese
formulas the dependence of the squared modulus on the square of the learning
rate, which is a consequence of the property (2.98), is apparent. As a consequence,
Search WWH ::
Custom Search