Information Technology Reference
In-Depth Information
Appendix C
Entropy and Variance of Partitioned
PDFs
Consider a PDF
f
(
x
) defined by a weighted sum of functions with disjoint
supports,
f
(
x
)=
i
a
i
f
i
(
x
)
,
(C.1)
such that
1. Each
f
i
(
x
) is a PDF with support
D
i
;
2.
D
i
∩ D
j
=
∅
, ∀i
=
j
;
3. The support of
f
is
D
=
∪
i
D
i
;
4.
i
a
i
=1.
We call such an
f
(
x
) a
partitioned
PDF. From (C.1), and taking the above
conditions into account, the Shannon (differential) entropy of
f
(
x
) is ex-
pressed as
a
k
f
k
(
x
)
ln
k
a
k
f
k
(
x
)
dx
=
H
S
(
f
)=
−
D
k
a
k
f
k
(
x
)
ln
k
a
k
f
k
(
x
)
dx
=
=
−
D
i
i
(C.2)
k
=
−
a
i
f
i
(
x
)[ln
a
i
+ln
f
i
(
x
)]
dx
=
D
i
i
a
i
D
i
=
−
f
i
(
x
)ln
f
i
(
x
)
dx
−
a
i
ln
a
i
.
D
i
i
We then have
H
S
(
f
)=
i
a
i
H
S
(
f
i
)
−
a
i
ln
a
i
.
(C.3)
i
Thus, the Shannon entropy of
f
is a weighted sum of the entropies of each
component
f
i
plus the entropy of the PMF corresponding to the weighting
