Generality Is Predictive of Prediction Accuracy - Data Mining: Theory, Methodology, Techniques, and Applications - page 12

will be differing evidence in support of each. Our hypotheses do not relate to
this situation. We are interested only in how to select between alternative rules
when the only source of evidence about their relative prediction performance is
their relative generality.
If it is not possible to develop measures of magnitude of generalization then
it appears to follow that it will never be possible to extend our hypotheses to
provide more specific predictions about the magnitude of the effects that may
be expected from a given generalization or specialization to a rule.
6Conluon
We have presented two hypotheses relating to expectations regarding the ac-
curacy of two alternative classification rules with identical supporting evidence
other than their relative generality. The first hypothesis is that the accuracy
on unseen data of the more general rule will be more likely to be closer to the
accuracy on unseen data of a default rule for the class than will the accuracy on
unseen data of the more specific rule. The second hypothesis is that the accu-
racy on previously unseen data of the more specific rule will be more likely to
be closer to the accuracy of the rules on the training data than will the accuracy
of the more general rule on unseen data.
We have provided experimental support for those hypotheses, both with re-
spect to classification rules formed by C4.5rules and random classification rules.
However, the results with respect to the second hypothesis were not statistically
significant in the case of random rules. These results are consistent with the
two hypotheses, albeit with the effect of the second being weak when there is
low accuracy for the error estimate for a rule derived from performance on the
training data. They are also consistent with the second hypothesis only applying
to a limited class of rule types. Further research into this issue is warranted.
These results may provide a first step towards the development of useful learn-
ing biases based on rule generality that do not rely upon prior domain knowl-
edge, and may be sensitive to alternative knowledge acquisition objectives, such
as trading-off accuracy for cover. Our experiments demonstrated the frequent
existence of rule variants between which traditional rule quality metrics, such
as an information measures, could not distinguish. This shows that the effect
that we discuss is not an abstract curiosity but rather is an issue of immediate
practical concern.
Acknowledgements
We are grateful to the UCI repository donors and librarians for providing the
data sets used in this research. The breast-cancer, lymphography and primary-
tumor data sets were donated by M. Zwitter and M. Soklic of the University
Medical Centre, Institute of Oncology, Ljubljana, Yugoslavia.