Digital Signal Processing Reference
In-Depth Information
The entropy conveys the idea of uncertainty of a random variable and it is
possible to show that the Gaussian distribution presents the largest entropy
among all distributions with the same mean and variance. On the other hand,
if we restrict this comparison only to distributions with finite support, the
largest entropy will be obtained for a uniform distribution [78].
It is worth mentioning that
H
(
α
)
can also be defined for discrete random
variables. In this case,
H
(
α
)
=−
p
i
log
p
i
(6.17)
i
and presents similar properties to the differential entropy.
In a similar fashion, one can define the conditional entropy as follows.
DEFINITION 6.3
(Conditional Entropy) Let
p
α
|
β
(
α
|
β
)
denotes the
conditional pdf of α given β.The
conditional entropy
is given by
∞
∞
E
log
p
α
|
β
(
)
=−
H
(
α
|
β
)
=−
α
|
β
p
α,β
(
τ,
υ)
log
p
α
|
β
(
τ
|
υ)
d
τ
d
υ
−∞
−∞
(6.18)
which is related to the uncertainty of a random variable given the observa-
tion of another random variable.
From Definitions 6.2 and 6.3, we can finally define the mutual informa-
tion
I
(
α, β
)
as follows.
DEFINITION 6.4
(Mutual Information) The mutual information between
two random variables α and β is defined by
I
(
α, β
)
=
H
(
α
)
−
H
(
α
|
β
)
=
H
(
β
)
−
H
(
β
|
α
)
(6.19)
From (6.16) and (6.18), one can show that the mutual information can also
be expressed as
p
α,β
(
p
α,β
(
α, β
)
I
(
α, β
)
=
α, β
)
log
d
α
d
β
(6.20)
p
α
(
α
)
p
β
(
β
)
It is particularly useful to interpret the mutual information in terms of the
so-called Kulback-Leibler divergence [78], defined in (5.107) and rewritten
here for the sake of convenience,
