Mixing Independently Trained Classifiers - Design and Analysis of Learning Classifier Systems - page 113

Statistical Analysis. A two-way analysis of variance (ANOVA) is used to
determine if the performance of the different mixing models differ significantly,
with the first factor being the type of mixing model (IRLS, IRLSf, LS, LSf,
InvVar, Conf, MaxConf, XCS) and the second factor being the combination
of regression task and type of classifier (Blocks, Bumps, Doppler, Heavisine,
either with averaging classifiers, or classifiers that model straight lines). The
direction of the difference is determined by Tukey's HSD post-hoc test. As the
optimal likelihood as measured by IRLS varies strongly with different sets of
classifiers, the performance is measured as a fraction of the optimal likelihood
for a particular classifier set rather than the likelihood itself.

6.3.2 Results

The first experiment compares the performance of all mixing models when using
K = 50 classifiers. For all functions and both averaging classifiers and classifiers
that model straight lines, 50 experimental runs were performed per function 1 .
To show the different test functions, and to give the reader an intuitive idea
how mixing is performed, Figures 6.1 to 6.4 show the predictions of the different
methods of a single run when using classifiers that model straight lines. The mean
likelihoods over these 50 runs as a fraction of the mean likelihood of the IRLS

Prediction of Blocks function using different mixing models

0.6

InvVar
Conf
MaxConf
LS
IRLS
XCS
f(x)

0.4

0.2

0

-0.2

-0.4

-0.6

0 0.2 0.4 0.6 0.8 1

x

Fig. 6.1. Resulting predictions of a single run, using different mixing models for the
Blocks function. See the text for an explanation of the experimental setup.

1 In our experience, performing the experiments with fewer runs provided insucient
data to permit significance tests to reliably detect the differences.