Information Technology Reference
In-Depth Information
Hence,
t
∞
depends on the spreading of the eigenvalue spectrum of
R
;ifthe
eigenvalues of
R
are clustered, the sudden divergence appears late. Furthermore,
t
∞
is proportional to the inverse of
λ
n
(high
λ
n
means noisy data).
Finally, for
s
(
p
)
=
1
/
p
(MCA EXIN),
1
2
−
λ
)
t
p
(
t
)
=
+
n
+
λ
n
tr
(
R
(2.112)
still divergence, but at a slower rate. This confirms Theorem 68, which is based
on the analysis of Section 2.6.1.
For the general case of eq. (2.96), it is easy to verify that the condition for
sudden divergence is given by
∞
dp
ps
2
(
p
)
<
∞
(2.113)
p
0
where
p
0
is the value of
p
at
t
=
0 (note that
p
0
=
0 is excluded because it is
outside the domain of the RQ of
R
). If
s
(
p
)
is a polynomial in
p
of degree
β
,
then
pq
2
(
p
)
∼
p
2
β
+
1
for
p
→∞
(2.114)
β >
0 the integral (2.113) converges and the sudden divergence holds.
9
Hence, for
2.6.2.2 OJA
The sudden divergence of OJA can be analyzed both with the
analysis above (see [181]) and using the corresponding ODE (2.20). In this case
it holds that
2
2
=
2
w
(
t
)
R
w (
t
)
w (
t
)
−
1
d
w (
t
)
T
2
2
(2.115)
dt
Assuming that the MC direction has been approached and recalling Remark 65,
eq. (2.115) can be approximated as
dp
dt
=
2
λ
n
p
(
p
−
1
)
(2.116)
2
where
p
=
w
(
t
)
2
, as usual. Notice the importance of sgn
(
p
−
1
)
with respect to
the increment of the weight modulus, as seen in Section 2.6.1. Define the instant
of time in which the MC direction is approached as
t
0
and the corresponding
9
What is the best possible
β<
0 that can be chosen in devising a novel MCA learning law? Certainly,
as seen before, a large
|
β
|
has a positive inertial effect on the weight modulus. In this case it can
always be classified as RQ gradient flow with its own Riemannian metric. However, for increasing
|
β
|
, the flop cost per iteration quickly becomes prohibitive, above all for high-dimensional weight
vectors. On the contrary,
β
=
2 (MCA EXIN) yields the best compromise between inertia and
computational cost.
Search WWH ::
Custom Search