Probabilities: A Historical Perspective - Information Fusion in Signal and Image Processing - page 265

mechanics, statistics and information theory. He expressed his research as a prob-
lem involving the specification of probabilities when little information is available.
He examined the two objectivist and subjectivist methods, and chose the latter. By
representing a state of knowledge, this makes it possible to express possible conclu-
sions if not enough information is available to obtain conclusions that are certain. It is
therefore more general, and Jaynes adopted it for statistical mechanics.

The intuitive criteria that Jaynes expects from a measure of uncertainty are the
following:
1) the measure has to be positive and continuous;
2) it has to increase when the uncertainty increases;
3) it has to be additive if the sources are independent.

He thus obtains a unique measure of uncertainty represented by a discrete proba-
bility distribution that corresponds to these intuitive criteria:

n

H p 1 ,...,p n =

− K p i log p i . [A.9]

i =1

In the same way Cox rediscovered probabilistic relations from his postulates,
Jaynes thus rediscovered, based on his criteria, the entropy in statistical mechanics and
simultaneously Shannon's entropy [SHA 59].

The maximum entropy principle can then be considered analogous to Laplace's
principle of insufficient reason. The essential difference is that Laplace's principle is
arbitrary in nature and can lead to paradoxes (the concept of equally probable cases
changes if the variable is changed), whereas the maximum entropy principle makes it
possible to make inferences on the basis of partial information in an unbiased fashion.
It can be chosen for the good reason that entropy is determined in a unique way as the
value that “implicates itself the least” with respect to the missing information and not
for the negative reason that there is no reason to think otherwise. However, it can be
criticized because the results it provides depend on how the problem is stated. This
criticism also applies to the principle of insufficient reason as we have seen in section
6.10.

A.3.6. De Finetti and betting theory

The approach suggested by de Finetti, which predates that of Cox, also relies on
simple and intuitive axioms that lead to the properties of probabilities [FIN 37]. We
have of course the same axioms of increasingness and universal comparison, and most
importantly a consistency axiom which serves as the basis of the demonstration.