Advanced Analytical Theory and Methods: Classification - Data Science and Big Data Analytics - page 278

Figure 7.5 Entropy of coin flips, where X=1 represents heads

For the bank marketing scenario previously presented, the output variable is
subscribed . The base entropy is defined as entropy of the output variable, that
is . As seen previously, and

. According to Equation 7.1 , the base entropy
.

The next step is to identify the conditional entropy for each attribute. Given an
attribute , its value , its outcome , and its value , conditional entropy

is

the remaining entropy of given , formally defined as shown in Equation 7.2 .

7.2

Consider the banking marketing scenario, if the attribute contact is chosen,
= { cellular , telephone , unknown }. The conditional entropy of contact
considers all three values.

Table 7.1 lists the probabilities related to the contact attribute. The top row of
the table displays the probabilities of each value of the attribute. The next two rows
contain the probabilities of the class labels conditioned on the contact .