Feature Selection - Data Mining with Decision Trees: Theory and Applications - page 218

13.5.2.2 Using Artificial Contrasts

Using the Bayesian approach, a certain attribute should be filtered out
if: P ( a i

∈ / B | B 1 ,...,B ω ) > 0 . 5or P ( a i ∈ / B | B 1 ,...,B ω ) >P ( a i ∈

B A denote the set of relevant features. By using

the Bayes Theorem, we obtain:

| B 1 ,...,B ω )where B ⊆

B 1 ,...,B ω )= P ( B 1 ,...,B ω |

a i / ∈ B ) P ( a i / ∈ B )

P ( a i ∈ / B | . (13.12)

P ( B 1 ,...,B ω )

However, since calculating the above probability might be dicult, we
use the naive Bayes combination. This is a well-known combining method
due to its simplicity and its relatively outstanding results. According to the
naive Bayes assumption, the results of the feature selectors are independent
given the fact that the attribute a i is not relevant. Thus, using this

assumption, we obtain:

ω

P ( a i / ∈ B ) P ( B j | a i / ∈ B )

P ( B 1 ,...,B ω | a i ∈ / B ) P ( a i ∈ / B ) j =1

= .

(13.13)

P ( B 1 ,...,B ω ) P ( B 1 ,...,B ω )

Using Bayes theorem again:

j =1

ω

j =1

ω

P ( a i ∈B|B j )
P ( a i ∈B )

P ( a i ∈ / B ) P ( B j | a i ∈ / B ) P ( a i ∈ / B ) P ( B j )

=

P ( B 1 ,...,B ω ) P ( B 1 ,...,B ω )

j =1

ω

j =1

ω

P ( B j ) P ( a i ∈ / B | B j )

=

B ) .
(13.14)

P ( B 1 ,...,B ω ) · P ω− 1 ( a i ∈ /

Thus, a certain attribute should be filtered out if:
ω

j =1 j =1

ω

j =1

ω

j =1

ω

P ( B j ) P ( a i ∈ / B | B j ) P ( B j ) P ( a i ∈ B | B j )

>

B ) .
(13.15)

P ( B 1 ,...,B ω ) · P ω− 1 ( a i ∈ / B ) P ( B 1 ,...,B ω ) · P ω− 1 ( a i ∈

or after omitting the common term from both sides:
ω

j =1 j =1

ω

P ( a i ∈ / B | B j ) P ( a i ∈ B | B j )

> . (13.16)

P ω− 1 ( a i ∈ / B ) P ω− 1 ( a i ∈ B )