Civil Engineering Reference
In-Depth Information
5.4
INFORMATION THEORY
The amount of information in a stimulus depends on the probability that the stimulus
carries relevant information. In this context one also speaks of information uncertainty or
entropy. Shannon and Weaver (1949) pioneered information theory. They defined
information as uncertainty or entropy. Some information has very little information
content. For example, a statement such as “The sun went up this morning” carries no
information, since the sun goes up every morning. The probability is
p=
1.0, and therefore
there is no information uncertainty. A statement such as “There was an earthquake in
Paris” carries much information, since it is very unlikely to happen—Paris is not on a
fault line! The probability of an event therefore affects the amount of information.
Shannon and Weaver (1949) presented a model for calculation of information in terms
of bits. If 2 stimuli are equally likely to occur with p=0.5, there is an information
uncertainty of 1 bit. If there are 4 possible events, each with a probability of p=0.25, there
are 2 bits. The general formula for calculating the number of bits of information is hence
tied to the probability:
H
s
=
log
2
N
where
H
s
is the amount of information and
N
is the number of equal probability
outcomes. For N events the probability that each may occur is therefore
p=
1/
N
.
H
s
=
log
2
(1/
p
)
To summarize the information for all N events we obtain:
H
s
=
log
2
(1/
p
)
EXAMPLE 1
Start with a deck of 64 cards—16 cards in 4 suits. Ask a person to think of card. Your
task is to identify the card the person is thinking of. You can ask the person for hints.
First, let us note that the amount of information uncertainty in the single card that you
will try to identify is as follows:
H
s
=
log
2
N
=log
2
64=6 bits
We can also do the calculations using probabilities. The probability to find a single
card is p=1/64:
H
s
=
log
2
(1/
p
)=log
2
(1/1/64)=log
2
64=6 bits
Given the hint that the card is red, the uncertainty is reduced from 64 to 32:1 bit.
Given that the card is a heart: 1 bit; lower 8 hearts: 1bit; lower 4 cards: 1 bit, lower 2
cards: 1 bit; ace of hearts: 1 bit; 6 bits altogether. This is in agreement with the
calculations above.
