ASSESSMENT METRICS FOR IMBALANCED LEARNING - Imbalanced Learning: Foundations, Algorithms, and Applications - page 193

to skew (as well as cost and other prior probabilities) of the data is generally not
known. In such cases, it is more useful to use evaluation methods that enable
visualization or summarization performance over the full operating range of the
classifier. In particular, such methods allow for the assessment of a classifier's
performance over all possible imbalance or cost ratios.
In this section, we will discuss ROC analysis, cost curves, precision-recall
(PR) curves, as well as a summary scalar metric known as AUC or AUROC
(Area under the ROC curve). In addition, newer and more experimental methods
and metrics will be presented.

8.4.1 ROC Analysis

In the context of the class imbalance problem, the concept of ROC analysis can
be interpreted as follows. Imagine that instead of training a classifier f only at
a given class imbalance level, that classifier is trained at all possible imbalance
levels. For each of these levels, two measurements are taken as a pair, the true
positive rate (or sensitivity), whose definition was given in Equation 8.1, and the
false positive rate (FPR) (or false alarm rate), whose definition is given by:

FP
FP + TN

FPR = (8.15)

Many situations may yield the same measurement pairs, but that does not matter
as duplicates are ignored. Once all the measurements have been made, the points
represented by all the obtained pairs are plotted in what is called the ROC space ,
a graph that plots the true positive rate as a function of the false positive rate. The
points are then joined in a smooth curve, which represents the ROC curve for
that classifier. Figure 8.1 shows two ROC curves representing the performance
of two classifiers f1 and f2 across all possible operating ranges. 2
The closer a curve representing a classifier f is from the top left corner of
the ROC space (small false positive rate, large true positive rate) the better the
performance of that classifier. For example, f 1 performs better than f 2inthe
graph of Figure 8.1. However, the ideal situation of Figure 8.1 rarely occurs in
practice. More often than not, one is faced with a situation as that of Figure 8.2,
where one classifier dominates the other in some parts of the ROC space, but
not in others.
The reason why ROC analysis is well suited to the study of class imbalance
domains is twofold. First, as in the case of the single-class focus metrics of
the previous section, rather than being combined together into a single multi-
class focus metric, performance on each class is decomposed into two distinct
measures. Second, as discussed at the beginning of the section, the imbalance

2 Note that this description of ROC analysis is more conceptual than practical. The actual construction
of ROC curves uses a single training set and modifies the classifier's threshold to generate all the
points it uses to build the ROC curve. More details can be found in [1].