Information Theory - Digital Communication: Principles and System Modelling - page 168

the entropy value is also increased. With maximum entropy, the entropy is also
maximized. Therefore, it is inferred that, entropy is the measure of uncertainty .

8.3 Rate of Information

In the previous section, entropy, H is defined as the average number of bits of infor-
mation per message. Now, if a message source generates r messages per second, the
rate of information is

R = rH bits / s (8.12)

Let's consider two sources with equal entropy H, generating r 1 and r 2 messages per
second, respectively with r 1 r 2 . Therefore, rate of information is greater in the

second source. It means, in a specific duration, source two will transmit more infor-
mation than source one, placing greater demand on the communication channel.
Hence, source is not only described by its entropy, but also by its rate of information.

<

Example 8.1 A CTCV signal is bandlimited to 4 kHz. The signal is quantized in
8 levels of PCM with probabilities 0.25, 0.2, 0.2, 0.1, 0.1, 0.05, 0.05 and 0.05.
Calculate the entropy and the rate of information.

The Nyquist rate of sampling = 4 × 2 = 8 kHz. Each sample is then quantized

to one of the 8 levels.
The entropy is

=− + + + + +

H (0.25 log 0.25 0.2 log 0.2 0.2 log 0.2 0.1 log 0.1 0.1 log 0.1

0.05 log 0.05 + 0.05 log 0.05 + 0.05 log 0.05) = 2.74 bits / message

As the sampling frequency is 8 kHz, the message rate is 8,000 messages/s. Hence
the rate of information is

R = rH = 8,000 × 2.74 = 21,920 bits / s

8.4 Information Sources

(a) Continuous source : The sources that produce continuous signals for transmis-
sion should be classified as continuous source [ 5 , 3 ].
(b) Discrete source : The sources that produce signals only at discrete times are
called discrete sources. A continuous analog signal after sampling is an example
of a discrete source
(c) Zero memory or memory less source : When any message produced by a source
is statistically independent of other messages produced before or after, the
source is called a zero memory or memory less source. The probability of