Graphics Reference
In-Depth Information
in all FS techniques. Usually, the evaluation metrics work in two ways: (1) supporting
the performance in terms of efficacy and (2) supporting the performance in terms of
efficiency or yielding more understandable outcomes. For instance, in classification,
as the main problem addressed in this chapter, the primary objective is to maximize
predictive accuracy.
7.2.2.1 Information Measures
Information serves to measure the uncertainty of the receiver when she/he receives
a message. If the receiver understands the message, her/his associated uncertainty is
low, but if the receiver is not able to completely understand the message, all mes-
sages have almost equal probability of being received and the uncertainty increases.
Under the context of predictive learning, the message is the output feature or class
in classification. An information measure
U
is defined so that larger values for
U
represent higher levels of uncertainty.
Given an uncertainty function
U
and the prior class probabilities
P
(
c
i
)
where
i
=
1
,
2
,...,
C
, being
C
the number of classes; the information gain from a feature
, is defined as the difference between the prior uncertainty
i
A
,
IG
(
A
)
U
(
P
(
c
i
))
and the expected posterior uncertainty using
A
, i.e.,
E
i
(
)
=
(
(
c
i
))
−
(
(
c
i
|
))
IG
A
U
P
U
P
A
i
where
E
represents the expectations. By Bayes' theorem, we have
P
(
c
i
)
P
(
x
|
c
i
)
P
(
c
i
|
x
)
=
P
(
x
)
P
P
(
x
)
=
(
c
i
)
P
(
x
|
c
i
)
A feature evaluation model inferred from the concept of information gain states
that feature
A
i
is chosen instead of feature
A
j
if
IG
(
A
i
)>
IG
(
A
j
)
; that is, if
A
i
reduces more uncertainty than
A
j
. Since
i
U
(
P
(
c
i
))
is independent of features,
we can rewrite the rule as
A
i
is preferred to
A
j
if
U
(
U
(
, where
U
=
A
i
)<
A
j
)
E
i
. This idea is used in C4.5 [
43
] for selecting a feature to generate
(
(
c
i
|
))
U
P
A
new branches.
A commonly used uncertainty function is Shannon's entropy,
−
P
(
c
i
)
log
2
P
(
c
i
).
i
