Machine learning and natural language processing - The Computing Universe: A Journey through a Revolution - page 281

that the coin is biased toward heads. The assumptions we had before tossing
the coin are called prior beliefs - or just priors - because we form them prior to
gathering any evidence about the situation. After incorporating the results of
observations, we modify our beliefs. These modified beliefs are called posterior
beliefs or posteriors . The technology of Bayesian inference that we explain in this
section determines - in a precise, mathematical way - how we should change
our prior beliefs. Bayesian inference is a decision-making technique that takes
into account both observed data and prior beliefs and allows us to eliminate
the least likely options. The frequentist interpretation of probability is fine
for problems where we can make repeatable experiments and measurements.
But such a frequentist approach cannot make predictions about nonrepeatable
events, such as “What is the probability of an earthquake in Seattle next year?”
or, more commonly, “What is the probability of rain in Seattle tomorrow?” The
Bayesian approach can provide a mathematical basis for such predictions.
An English clergyman named Thomas Bayes ( B.14.1 ) introduced what we
now call the Bayesian approach in the 1700s. His goal was to learn the probabil-
ity of a future event given only the number of times such an event had or had
not occurred in the past. His paper “An Essay towards Solving a Problem in the
Doctrine of Chances” contains the following example:
Picture a newborn witnessing his first sunset. Being new to this world, he
doesn't know if the sun will rise again. Making a guess, he gives the chance
of a sunrise even odds and places in a bag a black marble, representing no
sunrise, and a white marble, representing a sunrise. As each day passes, the
child places in the bag a marble based on the evidence he witnesses - in this
case, a white marble for each sunrise. Over time, the black marble becomes
lost in a sea of white, and the child can say with near certainty that the sun
will rise each day. 2
This example illustrates the basic Bayesian approach. The newborn has an ini-
tial degree of belief in whether or not the sun will rise that is just a fifty-fifty
guess. This belief is the baby's prior . As the child gathers more data, he or she
can update this belief to obtain a more accurate prediction for the probability
of a sunrise, the posterior .
In his paper, Bayes describes a thought experiment in laborious detail, an
experiment that we could now simulate very easily with a computer. He ima-
gines that he turns his back to a square table and asks his assistant to throw a
ball onto the table. The ball has just as much chance of landing at any place on
the table as anywhere else. Bayes cannot see the table and has no idea where
B.14.1. Thomas Bayes (1701-61) was an English clergyman who did pioneering work in probabil-
ity theory. In fact, his major work was published after his death and his papers were edited by
the Welsh scientist, clergyman, and philosopher William Price. Bayes' paper, “An Essay towards
Solving a Problem in the Doctrine of Chances,” containing the famous result now known as Bayes
Theorem, was published in Philosophical Transactions of the Royal Society in 1763. William Price had
a remarkable career, was a personal friend of many of the founding fathers of the United States,
and, with George Washington, received an honorary doctorate from Yale in 1781.