Digital Signal Processing Reference
In-Depth Information
was supposed additive white Gaussian, and if the estimation algorithm is such that
y
(
y
;
) is almost differentiable (Stein 1981) with respect to
y
, Stein's lemma yields
the divergence formula
λ
m
y
i
(
y
;
∂
λ
)
df
(
λ
)
= E
y
,
(7.36)
∂
y
i
i
=
1
where the expectation
E
y
is taken under the distribution of
y
. However, it is notori-
ously difficult to derive the closed-form analytical expression of
df
from the preced-
ing formula for general nonlinear modeling procedures. To overcome the analytical
difficulty, the bootstrap can be used. Ye (1998) and Shen and Ye (2002) proposed
using a data perturbation technique to numerically compute an (approximately) un-
biased estimate for
df
. From equation (7.36), the estimator of
df
takes the form
ϑ,
)
φ
1
τ
df
(
y
(
y
2
I
)
d
2
I
)
λ
)
=
+
ϑ
(
y
;
τ
ϑ, ϑ
∼N
(0
,τ
,
(7.37)
2
2
I
) is the zero-mean
m
-dimensional Gaussian probability density func-
tion (pdf). Ye (1998) proved that this is an unbiased estimate of
df
, that is,
lim
τ
→
0
φ
v
τ
where
(
;
y
df
(
)
=
E
λ
df
(
λ
). It can be computed by Monte Carlo integration with
τ
near 0.6, as devised by Ye (1998).
Zou et al. (2007) recently studied the degrees of freedom of BPDN (Lasso) in
the framework of SURE. They showed that for any given
, the number of nonzero
coefficients in the model is an unbiased and consistent estimate of
df
. However,
for their results to hold rigorously, the linear problem underlying BPDN must be
overdetermined (
T
λ
T
.InDupe et al. (2009), arguments and
experiments were provided that support that even in the underdetermined case, if
the solution is
k
-sparse, a practical estimator of
df
is still given by the cardinal of
the support of
<
m
) with rank(
F
)
=
α
(
y
;
). With such a simple formula on hand, expression of the GCV
in equation (7.35) is readily available.
λ
7.4.2 Sparsity Penalty with
q
Fidelity Constraint
We now consider the inequality-constrained problem
(
P
q
σ
):
min
α
∈R
(
α
)s
.
t
.
y
−
H
α
q
≤
σ,
(7.38)
T
where
q
≥
1;
σ
≥
0 is typically chosen depending on the noise
q
th moment
E
(
ε
q
);
and
,
H
, and
are defined as in (
P
λ
) (see equation (7.26)). We assume that
0. (
P
q
α
=
σ<
) only asks that the recon-
struction be consistent with the data such that the residual error has a bounded
q
th
moment. Solving (
P
q
y
q
to avoid the unique trivial solution
σ
) is challenging for large-scale problems.
σ
0, equation (7.38) becomes an equality-constrained problem, which spe-
cializes to BP (7.3) when
For
σ
=
) is equivalent to (
P
λ
); see
the discussion preceding equation (7.5). If the noise in equation (7.1) were a uni-
form quantization noise, a good choice would be
p
=
2, (
P
2
σ
is the
1
norm. For
q
=∞
, as advocated by Candes
and Tao (2006) and Jacques et al. (2009), and
is the quantization bin width. An-
other interesting variant is when the noise is impulsive (outliers), where the value
p
σ
=
1 would correspond to a good measurement consistency (Nikolova 2000, 2005).
