Evaluation of Classification Trees - Data Mining with Decision Trees: Theory and Applications - page 43

4.2.6.6 Area Under Curve ( AUC )

Evaluating a probabilistic model without using a specific fixed quota is
not a trivial task. Using continuous measures like hit curves, ROC curves
and lift charts, mentioned previously, is problematic. Such measures can
give a definite answer to the question “Which is the best model?” only if
one model dominates in the curve space, meaning that the curves of all
the other models are beneath it or equal to it over the entire chart space.
If a dominating model does not exist, then there is no answer to that
question, using only the continuous measures mentioned above. Complete
order demands no intersections of the curves. Of course, in practice there is
almost never one dominating model. The best answer that can be obtained
is in regard to which areas one model outperforms the others. As shown in
Figure 4.6, every model gets different values in different areas. If a complete
order of model performance is needed, another measure should be used.
Area under the ROC curve (AUC) is a useful metric for classifier
performance since it is independent of the decision criterion selected
and prior probabilities. The AUC comparison can establish a dominance
relationship between classifiers. If the ROC curves are intersecting, the total
AUC is an average comparison between models [ Lee (2000) ] . The bigger it
is, the better the model is. As opposed to other measures, the area under
the ROC curve (AUC) does not depend on the imbalance of the training
set [Kolcz (2003)]. Thus, the comparison of the AUC of two classifiers is
fairer and more informative than comparing their misclassification rates.

True positive

1

0.8

0.6

0.4

0.2

False positive

0.2 0.4 0.6 0.8 1

Fig. 4.6 Areas of dominancy. A ROC curve is an example of a measure that gives areas
of dominancy and not a complete order of the models. In this example, the equally dashed
line model is the best for f.p < 0.2. The full line model is the best for 0.2 < f.p < 0.4. The
dotted line model is best for 0.4 < f.p < 0.9 and from 0.9 to 1 again the dashed line model
is the best.