Decision Forests - Data Mining with Decision Trees: Theory and Applications - page 105

One of the basic questions that has been investigated in ensemble
learning is: “Can a collection of weak classifiers create a single strong one?”.
Applying the Condorcet Jury Theorem insinuates that this goal might be
achieved. Namely, construct an ensemble that (a) consists of independent
classifiers, each of which correctly classifies a pattern with a probability of
p> 0 . 5; and (b) has a probability of M>p to jointly classify a pattern to
its correct class.
Sir Francis Galton (1822-1911) was an English philosopher and statisti-
cian that conceived the basic concept of standard deviation and correlation.
While visiting a livestock fair, Galton was intrigued by a simple weight-
guessing contest. The visitors were invited to guess the weight of an ox.
Hundreds of people participated in this contest, but no one succeeded to
guess the exact weight: 1,198 pounds. Nevertheless, surprisingly enough,
Galton found out that the average of all guesses came quite close to
the exact weight: 1,197 pounds. Similarly to the Condorcet jury theorem,
Galton revealed the power of combining many simplistic predictions in order
to obtain an accurate prediction.
James Michael Surowiecki, an American financial journalist, published
in 2004 the topic “The Wisdom of Crowds: Why the Many Are Smarter
Than the Few and How Collective Wisdom Shapes Business, Economies,
Societies and Nations”. Surowiecki argues, that under certain controlled
conditions, the aggregation of information from several sources, results in
decisions that are often superior to those that could have been made by
any single individual — even experts.
Naturally, not all crowds are wise (for example, greedy investors of a
stock market bubble). Surowiecki indicates that in order to become wise,
the crowd should comply with the following criteria:

• Diversity of opinion — Each member should have private information
even if it is just an eccentric interpretation of the known facts.

• Independence — Members' opinions are not determined by the opin-

ions of those around them.

• Decentralization — Members are able to specialize and draw conclu-

sions based on local knowledge.

• Aggregation — Some mechanism exists for turning private judgments

into a collective decision.

In statistics, the idea of building a predictive model that integrates
multiple models has been investigated for a long time. The history of
ensemble methods dates back to as early as 1977 with Tukeys Twicing