Database Reference
In-Depth Information
As can be seen from Table 3, with respect to the conditions formed by creating
an initial rule from the C4.5rules rule with the greatest cover, all win/draw/loss
comparisons but one significantly (at the 0.05 level) support the hypotheses. The
one exception is marginally significant (
p
.
055).
Where the initial rule is the first rule from a C4.5rules rule list (Table 4),
all win/draw/loss records favor the hypotheses, but some results are not sig-
nificant at the 0.05 level. It is plausible to attribute this outcome to greater
unpredictability in the estimates obtained from the performance of the rules on
the training data when the rules cover fewer training cases, and due to the lower
numbers of differences in rules formed in this condition.
Where the initial rule is a random rule (Table 5), all of the results favor the
hypotheses, except for one comparison between the combined and random most
general rules for which a difference in prediction accuracy was only obtained
on one of the fifty data sets. Where more than one difference in prediction
accuracy was obtained, the results are significant at the 0.05 level with respect
to Hypothesis 1, but not Hypothesis 2.
These results appear to lend substantial support to Hypothesis 1. For all but
one comparison (for which only one domain resulted in a variation in performance
between treatments) the win/draw/loss record favors this hypothesis. Of these
eleven positive results, nine are statistically significant at the 0.05 level. There
appears to be good evidence that of two rules with equal empirical and other
support, the more general can be expected to obtain prediction accuracy on
unseen data that is closer to the frequency with which the class is represented
in the data.
The evidence with respect to Hypothesis 2 is slightly less strong, however. All
conditions result in the predicted effect occurring more often than the reverse.
However, only five of these results are statistically significant at the 0.05 level.
The results are consistent with an effect that is weak where the accuracy of the
rules on the training data differs substantially from the accuracy of the rules on
unseen data. An alternative interpretation is that they are manifestations of an
effect that only applies under specific constraints that are yet to be identified.
=0
4
Discussion
We believe that our findings have important implications for knowledge acquisi-
tion. We have demonstrated that in the absence of other suitable biases to select
between alternative hypotheses, biases based on generality can manipulate ex-
pected classification performance. Where a rule is able to achieve high accuracy
on the training data, our results suggest that very specific versions of the rule
will tend to deliver higher accuracy on unseen cases than will more general al-
ternatives with identical empirical support. However, there is another trade-off
that will also be inherent in selecting between two such alternatives. The more
specific rule will make fewer predictions on unseen cases.
Clearly this trade-off between expected accuracy and cover will be dicult to
manage in many applications and we do not provide general advice as to how
Search WWH ::
Custom Search