An Overview of Data Mining Techniques - Data Mining Techniques in CRM: Inside Customer Segmentation - page 27

Table 2.3 Misclassification matrix.

Predicted values

Positive Negative

Positive Correct prediction: true

positive record count

Misclassification: false
negative record count

Negative Misclassification: false

positive record count

Correct prediction: true
negative record count

The gains, response, and lift/index tables and charts are also helpful evaluation
tools that can summarize the predictive efficiency of a model with respect to a
specific target category. To illustrate their basic concepts and usage we will present
the results of a hypothetical churn model that was built on a dichotomous output
field which flagged churners.
The first step in the creation of such charts and tables is to select the target
category of interest, also referred to as the hit category. Records/customers are
then ordered according to their hit propensities and binned into groups of equal
size, named quantiles. In our hypothetical example, the target is the category
of churners and the hit propensity is the churn propensity; in other words, the
estimated likelihood of belonging to the group of churners. Customers have been
split into 10 equal groups of 10% each, named deciles. The 10% of customers with
the highest churn propensities comprise tile 1 and those with the lowest churn
propensities, tile 10. In general, we expect that high estimated hit propensities also
correspond to the actual customers of the target category. Therefore, we hope to
find large concentrations of actual churners among the top model tiles.
The cumulative table, Table 2.4, evaluates our churn model in terms of the
gain, response, and lift measures.
But what exactly do these performance measures represent and how are they
used for model evaluation? A brief explanation is as follows:

• Response %: ''How likely is the target category within the examined quantiles?''
Response % denotes the percentage (probability) of the target category within
the quantiles. In our example, 10.7% of the customers of the top 10% model tile
were actual churners, yielding a response % of the same value. Since the overall
churn rate was 2.9%, we expect that a random list would also have an analogous
churn rate. However, the estimated churn rate for the top model tile was 3.71
times (or 371.4%) higher. This is called the lift. Analysts have achieved results
about four times better than randomness in the examinedmodel tile. As we move
from the top to the bottom tiles, the model estimated confidences decrease.