Information Technology Reference
In-Depth Information
0.75
1.813
0.7
^
^
h = 2
H
S
,H
R
2
h = 0.1
S
S
w
0
w
0
w
0
0.45
1.810
0.2
0
0.6
1.2
0
0.6
1.2
0
0.6
1.2
(a)
c
=0
.
2
,d
=1
.
2
0.5
1.5
^
h = 0.1
^
h = 1
H
S
,H
R
2
S
S
0
w
0
w
0
w
0
−8
−0.3
1.1
0
1
5
0
1
5
0
1
5
(b)
c
=0
.
8
,d
=5
Fig. 3.27 The left and middle graphs show
H
S
as a function of
w
0
for different
values of
h
and
w
1
support
[
c, d
]
. The right-most graphs are the corresponding
theoretical curves of Shannon (solid) and Rényi's (dotted) entropies.
2+
e
(
e
+2)ln
|e|
e
+2
+2
e
⎡
⎤
−
1
−
tanh(
a−w
0
)
q
2
4
⎣
⎦
V
R
2
=
−
+
(
b
−
a
)
2
(
e
+2)
e
−
1
−
tanh(
b
−
w
0
)
2+
e
(
e −
2) ln
e
|e−
2
|
⎡
⎤
−
1
−
tanh(
c−w
0
)
−
2
e
+
p
2
4
⎣
⎦
.
(3.76)
c
)
2
(
e
(
d
−
−
2)
e
−
1
−
tanh(
d−w
0
The expression of the Shannon entropy is even more intricate [212]. For both
entropies it is possible to show that neither of them has a minimum at the
optimal split point (the min
P
e
point); as also shown in the cited work, for
class-conditionals with equal-length supports, they have a maximum.
Figure 3.27 compares the behaviors of theoretical and empirical (2000 in-
stances per class) Shannon entropy curves as functions of the split parameter
w
0
, varying the class-conditional configuration. In all cases we assume equal
priors (
p
=
q
=1
/
2) and fix w.l.o.g. [
a, b
]=[0
,
1].
In the top row of Fig. 3.27 the class-conditionals have equal-length support,
which means that the optimal solution is any point of the overlapped region
[
c,
1]. If the kernel bandwidth
h
is too small,
H
S
exhibits a maximum at the
optimal split point, just as
H
S
.
Above a suciently large
h
(and increasingly larger with increasing over-
lap), the Shannon MEE solution is indeed the optimal split point.
