Information Technology Reference
In-Depth Information
Entropy indicates an expectation; it is a measure of the 'surprise' value of an
event. Here 'surprise' is measured as the log of the inverse probability of that event
(or
−
log
2
p). So:
Surprise
=−
log
2
·
p
−
p log
2
p) is the
average degree of
certainty
(surprise) of being able to predict the next bit of information in a stream,
e.g. the next character in a string of characters. (The logarithmic scale used in keeping
with other human sense sensitivities (Shannon and Weaver 1949). So:
In information theory, entropy (given by
The more unlikely the event the greater the surprise.
We use this in a general measure of an agent's confidence about its current view of
the world and its ability to respond correctly to event n. In the equation below we
will use 'A' and subscript 'a' to denote a particular agent. Because the hypotheses
under consideration in this example will be assumed mutually exclusive, the average
surprise will increase as an agent becomes more confident about a smaller number of
hypotheses. We call this general confidence measure for an agent the
model entropy
.
The term model denotes the set of hypotheses that make up the agent's view of the
world. Model entropy is given by:
=−
a
H
E
n
(H
a
)
Entropy (Agent
a
)
∗
Log
2
(E
n
(H
a
))
and from this we can obtain an inverse of the entropy which gives an expected
value for E
n
(H). This will be denoted by I
n
(A). I
n
(A) will be called an
Indifference
Threshold
for the agent A at event (time)
n
:
log
2
−
1
(Entropy
n
(A
a
))
IndifferenceThreshold(a,
n
)
=−
=
I
n
(A)
The expression E
n
(H) is the expected probability of a hypothesis. Values of E
n
(H)
above I
n
(A) are considered to be significant, i.e., the hypothesis is actively believed
by the agent
1
.
I
n
(A) can be treated as a generic confidence over all the hypotheses. If we consider
the case that all hypotheses had this confidence value as one possible state that could
occur we can say that the agent is
indifferent
to them all, thus the name
Indifference
Threshold
.I
n
(A) indicates a level of general confidence an agent has about its view
of the world at time
n
.
1
Since most of the calculations are done by natural logarithms,it is useful to note that: 2
x
=
e
(x
.
log2)
