Information Technology Reference
In-Depth Information
Appendix D
Entropy Dependence on the Variance
When analyzing the MSE and EE risk functionals in a comparative way, one
often wishes to know how the Shannon entropy changes with the variance
(denoted in this Appendix simply as
H
and
V
, respectively). We shall see
in Sect. D.2 that many unimodal parametric PDFs,
f
(
x
;
α
), have an entropy
H
(
α
) which changes with the variance
V
(
α
) in an increasing way but with
decreasing derivative [219]. In order to show this we first need to introduce
the notion of saturating functions.
D.1 Saturating Functions
+
is an
up-saturating
(
down-saturating
) function — denoted USF (DSF) — if it is
strictly concave (convex) and increasing (decreasing).
Definition D.1. A real continuous function
f
(
x
) with domain
R
Remarks:
1. Let us recall that
f
(
x
) is a strictly concave function if
=
x
2
then
f
(
tx
1
+(1
−t
)
x
2
)
>tf
(
x
1
)+(1
−t
)
f
(
x
2
). Function
f
is strictly convex
if
−f
is strictly concave.
2. One could define saturating functions for the whole real line. They are
here restricted to
∀
t
∈
]0
,
1[ and
x
1
+
for simplicity and because variance is a non-negative
R
quantity.
3.
√
x
and ln(
x
) are obvious USFs. The functions digamma,
ψ
(
x
)=
dΓ
(
x
)
dx
,
and trigamma,
ψ
1
(
x
)=
dψ
(
x
)
dx
, to be used below, are USF and DSF, re-
spectively.
Properties. The following are obvious properties of saturating functions:
f
is USF and vice-versa.
ii If
f
is a saturating function and
a
iIf
f
is DSF
−
is a constant,
f
+
a
is a saturating
function of the same type (i.e., both either USF or DSF).
∈
R
