ASSESSMENT METRICS FOR IMBALANCED LEARNING - Imbalanced Learning: Foundations, Algorithms, and Applications - page 199

interpreted as a classification performance measure rather than as a pure ranking
measure. Furthermore, they found that under that interpretation, the problem
appears only if the thresholds used to construct the ROC curve are assumed to
be optimal. The model dependence of the AUC, they claim, comes only from
these assumptions. They argue that in fact, Hand [16] did not need to make
such assumptions, and they propose a new derivation that, rather than assuming
optimal thresholds in the construction of the ROC curve, considers all the points
in the dataset as thresholds. Under this assumption, Flach et al. [17] are able to
demonstrate that the AUC measure is in fact coherent and that the H -measure is
unnecessary, at least from a theoretical point of view.
An interesting additional result of Flach [17] is that the H -measure proposed
by Hand [16] is, in fact, a linear transformation of the value that the area under
the cost curve would produce. With this simple and intuitive interpretation, the
H -measure would be interesting to pit against the AUC in practical experiments,
although, given the close relation between the ROC Graph and cost curves, it
may be that the two metrics yield very similar if not identical conclusions.
The next section suggests yet another metric that computes an Area under the
Curve, but this time, the curve in question is a PR curve.
8.4.5.2 AU PREC and I AU PREC The measures discussed in this section
are to PR curves what AUC is to ROC analysis or what the H -measure (or a
close cousin of it) is to cost curves. Rather than a single one, two metrics are
presented as a result of Landgrebe et al. [14]'s observations that in the case of PR
curves, because precision depends on the priors, different curves are obtained for
different priors. Landgrebe et al. [14], thus, propose two metrics to summarize
the information contained in PR curves: one that computes the Area Under the
PR curve for a single prior, the AU PREC , and one that computes the Area
Under the PR curve for a whole range of priors, the I AU PREC . The practical
experiments conducted in the paper demonstrate that the AU PREC and I AU
PREC metrics are more sensitive to changes in class distribution than the AUC.
8.4.5.3 B-ROC The final experimental method that we present in this chapter
claims to be another improvement on ROC analysis. In their work on Bayesian-
ROCs (B-ROCs), Cardenas and Baras [18] propose a graphical method that they
claim is better than ROC analysis for analyzing classifier performance on heavily
imbalanced datasets for three reasons. The first reason is that rather than using
the true and false positive rates, they plot the true positive rate and the precision.
This, they claim, allows the user to control for a low false positive rate, which
is important in applications such as intrusion detection systems, better than ROC
analysis. Such a choice, they claim, is also more intuitive than the variables used
in ROC analysis. A second reason is similar to the one given by Landgrebe
et al. [14] with respect to PR curves: B-ROC allows the plotting of different
curves for different class distributions, something that ROC analysis does not
allow. The third reason concerns the comparisons of two classifiers. With ROC