IMBALANCED DATASETS: FROM SAMPLING TO CLASSIFIERS - Imbalanced Learning: Foundations, Algorithms, and Applications - page 52

make 99% of the dataset, while only 1% is made up of minority class instances.
In such a case, it is trivial to obtain an accuracy of 99% if one merely predicts
the majority class. This accuracy is obtained by simply predicting all instances as
majority class. While the 99% accuracy seems high, it is obviously completely
discounting the performance of the classifier on the class that matters (positive
class). Thus, accuracy can be misleading in imbalanced datasets as there are two
different types of errors (false positives and false negatives), and each of them
carries different costs.
We now present a discussion on a number of alternate metrics used for eval-
uating the performance of classifiers on imbalanced datasets.

3.4.1 Balanced Accuracy

While accuracy (and error rate) is not an effective method of evaluating the
performance of classifiers, one common alternative is balanced accuracy . Bal-
anced accuracy differs from accuracy in that instead of computing accuracy ,
one computes

TP TN

2 ( TN

BalancedAccuracy =

FN )) +

(3.7)

2 ( TP + + FP ))

That is, one computes the average of the percentage of positive class instances
correctly classified and the percentage of negative class instances correctly clas-
sified. By giving equal weight to these relative proportions, we see that the
previous problem of the naıve classifier obtaining very good performance has
been eliminated.
To see this, consider the balanced accuracy of the naıve classifier on a dataset
consisting of 99% majority class instances and 1% minority class instances. We
know that the accuracy of the naıve classifier is 99%. The balanced accuracy,
on the other hand, is: 99 /( 2 ( 99 + 0 )) + 0 /( 2 ( 1 + 0 )) = 0 . 5 + 0 = 0 . 5. A perfor-
mance estimate of 0 . 5 is a much more valid assessment of the naıve classifier.

3.4.2 ROC Curves
The receiver operating characteristic (ROC) curve is a standard technique for
evaluating classifiers on datasets that exhibit class imbalance. ROC curves
achieve this skew insensitivity by summarizing the performance of classifiers
over a range of true positive rates (TPRs) and false positive rates (FPRs)
[32]. By evaluating the models at various error rates, ROC curves are able to
determine what proportion of instances will be correctly classified for a given
FPR.
In Figure 3.2, we see an example of an ROC curve. In Figure 3.2, the X -axis
represents the FPR (FPR = FP /( TN + FP ) ), and the Y -axis represents the TPR
(TPR = TP /( TP + FN ) ). For any given problem, the ideal classifier would have