Biomedical Engineering Reference
In-Depth Information
where the second form is valid only for strictly stationary processes. The mutual
information function measures the degree to which different parts of the series
are dependent on each other.
The entropy rate h of a stochastic process is
h
w
lim
H X
[
|
X
1
]
,
[49]
0
L
L
ld
=
HX
[
|
X d
1
]
.
[50]
0
(the limit always exists for stationary processes), where h measures the process's
unpredictability, in the sense that it is the uncertainty which remains in the next
measurement even given complete knowledge of its past. In nonlinear dynamics,
h is called the Kolmogorov-Sinai (KS) entropy .
For continuous variables, one can define the entropy via an integral,
H [ X ] -, p ( x ) log p ( x ) dx ,
[51]
with the subtlety that the continuous entropy not only can be negative, but de-
pends on the coordinate system used for x . The relative entropy also has the ob-
vious definition,
px
()
w ยจ
D
(P || Q)
p x
( ) log
dx
,
[52]
qx
()
but is coordinate-independent and non-negative. So, hence, is the mutual infor-
mation.
Optimal Coding . One of the basic results of information theory concerns
codes, or schemes for representing random variables by bit strings. That is, we
want a scheme that associates each value of a random variable X with a bit
string. Clearly, if we want to keep the average length of our code-words small,
we should give shorter codes to the more common values of X . It turns out that
the average code-length is minimized if we use -log Pr( x ) bits to encode x , and it
is always possible to come within one bit of this. Then, on average, we will use
E [-log Pr( x )] = H [ X ] bits.
This presumes we know the true probabilities. If we think the true distribu-
tion is Q when it is really P, we will, on average, use E [-log Q( x )] H [ X ]. This
quantity is called the cross-entropy or inaccuracy , and is equal to H [ X ] +
D (P||Q). Thus, finding the correct probability distribution is equivalent to mini-
mizing the cross-entropy, or the relative entropy (160).
Search WWH ::




Custom Search