Evaluation of Classification Trees - Data Mining with Decision Trees: Theory and Applications - page 37

Table 4.1 A confusion matrix.
Predicted negative Predicted positive

Negative Examples A B

Positive Examples C D

for each class; the off-diagonal elements indicate the number of observations
that have been incorrectly classified. One benefit of a confusion matrix is
that it is easy to see if the system is confusing two classes (i.e. commonly
mislabeling one as an other).
For every instance in the test set,wecomparetheactualclassto
the class that was assigned by the trained classifier. A positive (negative)
example that is correctly classified by the classifier is called a TP (true
negative); a positive (negative) example that is incorrectly classified is
called a false negative (false positive). These numbers can be organized
in a confusion matrix shown in Table 4.1.
Based on the values in Table 4.1, one can calculate all the measures
defined above:

• Accuracy is: ( a + d ) / ( a + b + c + d )

• Misclassification rate is: ( b + c ) / ( a + b + c + d )

• Precision is: d/ ( b + d )

• True positive rate (Recall) is: d/ ( c + d )

• False positive rate is: b/ ( a + b )

• True negative rate (Specificity) is: a/ ( a + b )

• False negative rate is: c/ ( c + d )

4.2.6 Classifier Evaluation under Limited Resources

The above-mentioned evaluation measures are insucient when probabilis-
tic classifiers are used for choosing objects to be included in a limited
quota. This is a common situation that arises in real-life applications due
to resource limitations that require cost-benefit considerations. Resource
limitations prevent the organization from choosing all the instances. For
example, in direct marketing applications, instead of mailing everybody on
the list, the marketing efforts must implement a limited quota, i.e. target
the mailing audience with the highest probability of positively responding
to the marketing offer without exceeding the marketing budget.
Another example deals with a security ocer in an air terminal.
Following September 11, the security ocer needs to search all passengers