Information Technology Reference
In-Depth Information
Fig. 2.1
Communication
system (Simplified from
Shannon Fig.
2.1
1948)
Information
Source
Destination
Message
Message
Receiver
Signal
314159265358979323846...
where the digits will be converted into a signal by a transmitter, passed down the
channel of communication (e.g. a telephone cable), and en route may be changed
by the effects of noise (not shown in diagram). The receiver then converts the noisy
signal back into digits. From the destination's point of view we should note that:
The significant aspect (of this communication system) is that the actual final message is
selected from a defined set
of possible messages. (Shannon 1948,
1964
).
If we take the simplest notion of a message at the receiver by considering that each
digit is a message, then our expectation of a message (i.e. a digit) before it arrives
is given by the choice of 1 in 10. So, if each digit were equally possible then the
probability that we could guess at the destination point what the next digit would be
is 0.1. When a message arrives our uncertainty will reduce to zero because there is
now no need to guess (probability is 1). The larger the choice the greater is the initial
uncertainty.
We can therefore propose a measure of 'uncertainty' that is inversely propor-
tional to probability: it increases as the probability of choosing correctly decreases.
However, this inverse probability measure should also reflect our own perception of
uncertainty.
It has been shown that a person's sensitivity to sensations such as hearing or
touch is 'logarithmic'. This natural detection system allows us to cope with very
loud sounds or firm pressures and yet still remain sensitive to very low sounds or
gentle touches (say).
From these observations of choice and the logarithmic scale of sensation, it would
be reasonable to define the 'unit' of uncertainty as 0.5, because uncertainty is highest
when the probability of either choice is the same. That is when there is zero bias in
the choice.
To reflect both the unit of choice and human sensitivity, we can create a function
by using the logarithm measure to the base 2 of the inverse probability of a message.
The inverse of the probability is used because as the probability of a message rises,
the less information it provides. The advantage of this is since a probability is always
less than 1 the inverse will always be greater than 1. When a log is taken it will
always provide a positive number: (Fig.
2.2
)
1
probability
Uncertainity
=
log
2
