Digital Signal Processing Reference
In-Depth Information
9.4.4 Approximate Newton Search: The FastICA Algorithm
One of the most popular methods for kurtosis optimization is the FastICA algo-
rithm [
35
,
36
], see also [
40
], which is based on Newton rather than gradient up-
dates. After deriving the algorithm in Sect.
9.4.4.1
, we provide an interpretation as
a gradient-like method in Sect.
9.4.4.2
.
9.4.4.1 Derivation of the Algorithm
Newton methods are based on the second-order Taylor approximation of the contrast
around the current point
w
:
J
κ
w
+
≈
J
κ
(
w
)
+∇
J
κ
(
w
)
T
w
+
−
w
+
2
w
+
−
w
T
H
(
w
)
w
+
−
w
(9.17)
1
where
H
(
w
)
represents the Hessian matrix of the second-order derivatives, with
elements
J
κ
(
w
)/∂w
i
∂w
j
. The Newton update selects the vector
w
+
that cancels out the gradient of the second-order approximation on the left-hand side
of Eq. (
9.17
), yielding
H
(
w
)
]
ij
=
∂
2
[
w
+
=
H
(
w
)
−
1
−
∇
J
κ
(
w
).
w
(9.18)
As compared to gradient update (
9.13
), no parameter needs to be fine-tuned here but,
in exchange, the Hessian matrix needs to be inverted at each iteration, which can be
costly and may introduce numerical instabilities. Hessian inversion is probably the
main drawback of Newton methods.
Matrix inversion can sometimes be avoided, and Newton methods consequently
simplified, by approximating the Hessian matrix, as is the case with FastICA. The
algorithm considers the real-valued mixture scenario after prewhitening, with ob-
servation model (
9.15
) and extraction equation (
9.16
), as in the method described in
the previous section. By constraining the extracting vector to lie on the unit sphere,
1, the extractor output is guaranteed to fulfill the unit-variance normalization
convention, E
q
=
y
2
{
}=
1. Under such assumptions, the absolute kurtosis contrast (
9.9
)
J
κ
(
q
)
=|
J
f
(
q
)
−
|
simplifies into
3
, where
E
y
4
J
f
(
q
)
=
(9.19)
is the fourth-order moment of the extractor output; similar contrasts based on the
fourth-order moment had also been proposed in [
9
,
26
]. The gradient and Hessian
of this simplified contrast are given, respectively, by the expressions:
4sign
J
(
q
)
E
y
3
z
,
∇
J
κ
(
q
)
=
(9.20)
12 sign
J
(
q
)
E
y
2
zz
T
.
H
(
q
)
=
(9.21)
Search WWH ::
Custom Search