Digital Signal Processing Reference
In-Depth Information
drawbacks, as it will be shown later, mostly related with a poor performance (slow
convergence rate) with colored input signals.
4.1.1 Stochastic Gradient Approximation
w
(
n
)
=
w
(
n
−
1
)
+
μ
[
r
x
d
−
R
x
w
(
n
−
1
)
]
,
w
(
−
1
).
(4.1)
The idea of the stochastic gradient approximation is to replace the correlation
matrix and cross correlation vector by suitable estimates. The simplest approximation
rely on the instantaneous values of the input and reference signals, i.e.,
R
x
=
x
T
x
(
n
)
(
n
)
and
r
x
d
=
ˆ
d
(
n
)
x
(
n
).
(4.2)
These estimates arise from dropping the expectation operator in the definitions of
the statistics. Replacing the actual statistics in (
4.1
) by their estimates (
4.2
) leads to
w
(
n
)
=
w
(
n
−
1
)
+
μ
x
(
n
)
e
(
n
),
w
(
−
1
).
(4.3)
This is the recursion of the LMS or Widrow-Hoff algorithm. A common choice in
practice is
w
0
.
As in the SD, the step size
(
−
1
)
=
influences the dynamics of the LMS. It will be
shown later that also as in the SD, large values of
μ
cause instability, whereas small
ones give a low convergence rate. However, important differences should be stated.
The filter
w
μ
in (
4.3
) is a random variable while the one in (
4.1
) is not. The MSE
used by the SD is a deterministic function on the filter
w
. The SD moves through
that surface in the opposite direction of its gradient and eventually converges to its
minimum. In the LMS, that gradient is approximated by
(
n
)
w
T
∇
w
J
(
(
−
))
=
(
)
(
−
)
(
)
−
(
)
.
w
n
1
x
n
n
1
x
n
d
n
(4.4)
On the other hand, we can think of (
4.4
) as the actual gradient of the instantaneous
squared value (ISV) cost function, i.e.,
J
2
w
T
2
(
w
(
n
−
1
))
=|
e
(
n
)
|
=|
d
(
n
)
−
(
n
−
1
)
x
(
n
)
|
,
(4.5)
where the factor 2 from the gradient calculation would be incorporated to the step
size
. This function arises from dropping the expectation in the definition of the
MSE, and therefore it is now a random variable. At each time step, when new data is
available, the shape of this cost function changes and the LMS moves in the opposite
direction to its gradient.
μ
