Databases Reference
In-Depth Information
This also makes sense. For the compression scheme described here, if we know the source
output, we know 4 bits, the first 3 of which are the reconstruction. Therefore, in this example,
knowledge of the source output at a specific time completely specifies the corresponding
reconstruction.
8.4.2 Average Mutual Information
We make use of one more quantity that relates the uncertainty or entropy of two random
variables. This quantity is called
mutual information
and is defined as
log
P
(
x
k
|
y
j
)
i
(
x
k
;
y
j
)
=
(19)
P
(
x
k
)
We will use the average value of this quantity, appropriately called
average mutual information
,
which is given by
log
P
N
−
1
i
M
−
1
(
x
i
|
y
j
)
I
(
X
;
Y
)
=
P
(
x
i
,
y
j
)
(20)
P
(
x
i
)
=
0
j
=
0
log
P
N
−
1
M
−
1
j
(
x
i
|
y
j
)
=
0
P
(
x
i
|
y
j
)
P
(
y
j
)
(21)
P
(
x
i
)
=
i
=
0
We can write the average mutual information in terms of the entropy and the conditional
entropy by expanding the argument of the logarithm in Equation (
21
):
log
P
N
−
1
M
−
1
(
x
i
|
y
j
)
I
(
X
;
Y
)
=
P
(
x
i
,
y
j
)
(22)
P
(
x
i
)
i
=
0
j
=
0
N
−
1
M
−
1
=
P
(
x
i
,
y
j
)
log
P
(
x
i
|
y
j
)
i
=
0
j
=
0
N
−
1
M
−
1
−
P
(
x
i
,
y
j
)
log
P
(
x
i
)
(23)
i
=
0
j
=
0
=
(
)
−
(
|
)
(24)
H
X
H
X
Y
where the second term in Equation (
23
)is
H
. Thus, the
average mutual information is the entropy of the source minus the uncertainty that remains
about the source output after the reconstructed value has been received. The average mutual
information can also be written as
(
X
)
, and the first term is
−
H
(
X
|
Y
)
I
(
X
;
Y
)
=
H
(
Y
)
−
H
(
Y
|
X
)
=
I
(
Y
;
X
).
(25)
We can show this easily by using Bayes' theorem. According to Bayes' theorem
P
(
y
j
|
x
i
)
P
(
x
i
)
P
(
x
i
|
y
j
)
=
P
(
y
j
)
