Decision Forests - Data Mining with Decision Trees: Theory and Applications - page 106

[ 1 ] : an ensemble of two linear regression models. Tukey suggested to fit
the first linear regression model to the original data and the second
linear model to the residuals. Two years later, Dasarathy and Sheela
(1979) suggested to partition the input space using two or more classifiers.
The main progress in the field was achieved during the Nineties. Hansen
and Salamon (1990) suggested an ensemble of similarly configured neural
networks to improve the predictive performance of a single one. At the same
time Schapire (1990) laid the foundations for the award winning AdaBoost
[ Freund and Schapire (1996) ] algorithm by showing that a strong classifier
in the probably approximately correct (PAC) sense can be generated by
combining “weak” classifiers (that is, simple classifiers whose classification
performance is only slightly better than random classification). Ensemble
methods can also be used for improving the quality and robustness
of unsupervised tasks. Nevertheless, in this topic we focus on classifier
ensembles.
In the past few years, experimental studies conducted by the machine-
learning community show that combining the outputs of multiple classi-
fiers reduces the generalization error [Domingos (1996); Quinlan (1996);
Bauer and Kohavi (1999); Opitz and Maclin (1999)] of the individual
classifiers. Ensemble methods are very effective, mainly due to the phe-
nomenon that various types of classifiers have different “inductive biases”
[ Mitchell (1997) ] . Indeed, ensemble methods can effectively make use of
such diversity to reduce the variance-error [ Tumer and Ghosh (1996) ]
[ Ali and Pazzani (1996) ] without increasing the bias-error. In certain
situations, an ensemble method can also reduce bias-error, as shown by
the theory of large margin classifiers [ Bartlett and Shawe-Taylor (1998) ] .
The ensemble methodology is applicable in many fields such as:
finance [ Leigh et al . (2002) ] ; bioinformatics [ Tan et al . (2003) ] ; medicine
[ Mangiameli et al . (2004) ] , cheminformatics [ Merkwirth et al . (2004) ] ;
manufacturing [ Maimon and Rokach (2004) ] ; geography [ Bruzzone et al .
(2004) ] and pattern recognition [ Pang et al . (2003) ] .
Given the potential usefulness of ensemble methods, it is not surprising
that a vast number of methods is now available to researchers and
practitioners. Several surveys on ensemble are available in the literature,
such as [ Clemen (1989) ] for forecasting methods or [ Dietterich (2000b) ]
for machine learning. Nevertheless, this survey proposes an updated and
profound description of issues related to ensemble of classifiers. This chapter
aims to organize all significant methods developed in this field into a
coherent and unified catalog.