Digital Signal Processing Reference
In-Depth Information
any message is independent of the probabilities of other messages. In fact all
examples discussed so far in this chapter are based on memory less source.
(d)
Extension of zero memory source:
After a memory less source has produced
the messages, often successive two or more messages are combined together
before transmission. These combined messages form a new set of symbols
or messages. Consider a source, which can produce M messages namely m
1
,
m
2
,m
3
,
...
etc. It randomly produces a string of messages, for example,
m
3
−
etc. Say, two successive messages are com-
bined to make a symbol. Then the new set of messages will be m
35
,m
11
,
m
46
,
m
5
−
m
1
−
m
1
−
m
4
−
m
6
−
...
...
etc., where m
35
means the combination of m
3
and m
5
.
Entropy of an extension of memory less source: If entropy of the basic memory
less source is H and n nos. of successive messages are combined to form the nth
order extension, then entropy of the nth order extended source is given by,
H
(s
n
)
=
n. H(s)
(8.13)
Example 8.2
A discrete memory less source (DMS) can produce four messages viz.
m
1
,m
2
,m
3
and m
4
with probabilities 1/4,
1
/
2
, 1/8 and 1/8. Determine the entropy of
its 2nd order extension and show that it is twice the entropy of the basic source.
The problem has been illustrated in a tabular form in Table
8.1
.
Table 8.1
Entropy of 2nd order extension of DMS
Basic source
2nd order extension
Message
Probability
New message
Combination of
Joint probability
m
1
1/4
m
11
m
1
&m
1
1/4
×
1/4
=
1/16
m
2
1/2
m
12
m
1
&m
2
1/4
×
1/2
=
1/8
m
3
1/8
m
13
m
1
&m
3
1/4
×
1/8
=
1/32
m
4
1/8
m
14
m
1
&m
4
1/4
×
1/8
=
1/32
m
21
m
2
&m
1
1/2
×
1/4
=
1/8
m
22
m
2
&m
2
1/2
×
1/2
=
1/4
m
23
m
2
&m
3
1/2
×
1/8
=
1/16
m
24
m
2
&m
4
1/2
×
1/8
=
1/16
m
31
m
3
&m
1
1/8
×
1/4
=
1/32
m
32
m
3
&m
2
1/8
×
1/2
=
1/16
m
33
m
3
&m
3
1/8
×
1/8
=
1/64
m
34
m
3
&m
4
1/8
×
1/8
=
1/64
m
41
m
4
&m
1
1/8
×
1/4
=
1/32
m
42
m
4
&m
2
1/8
×
1/2
=
1/16
m
43
m
4
&m
3
1/8
×
1/8
=
1/64
m
44
m
4
&m
4
1/8
×
1/8
=
1/64
H
=
1.75 bits/message
H2
=
3.5 bits/message
Search WWH ::
Custom Search