Information Technology Reference
In-Depth Information
d
dx
V
R
2
(
x
)=
P
−
1
f
X|−
1
(
x
)+
−
P
1
)(
f
X|−
1
(
x
)
f
X|
1
(
x
)) +
P
1
f
X|
1
(
x
)
.
(4.26)
+(1
−
P
−
1
−
−
The optimum
x
satisfies
f
X|−
1
(
x
)
f
X|
1
(
x
)
=
1
−
2
P
1
−
P
−
1
(4.27)
1
−
2
P
−
1
−
P
1
implying at
x
∗
that
P
−
1
=
P
1
.
3. Shannon EE risk
R
SEE
is given by formula (4.2)
R
SEE
(
x
)=
−
P
−
1
ln
P
−
1
−
P
1
ln
P
1
−
−
P
−
1
−
P
1
)ln(1
−
P
−
1
−
P
1
)
.
(4.28)
(1
Therefore, after some simple manipulations:
d
dx
R
SEE
(
x
)=
1
P
−
1
1
2
f
X|
1
(
x
)ln
P
1
2
f
X|−
1
(
x
)ln
P
1
−
.
1
−
P
−
1
−
1
−
P
−
1
−
P
1
(4.29)
The optimum
x
satisfies
P
−
1
ln
f
X|
1
(
x
)
f
X|−
1
(
x
)
=
1
−
P
−
1
−
P
1
(4.30)
P
1
ln
1
−
P
−
1
−
P
1
implying at
x
∗
that
P
−
1
=
P
1
.
This result for SEE can in fact be easily proved for any
p
=1
q
setting,
constituting the following theorem whose proof can be found in [216]:
−
Theorem 4.2.
In the univariate two-class problem with continuous class-
conditional PDFs, the
min
P
e
solution
x
∗
is a critical point of the Shannon
error entropy if the error probabilities of each class at
x
∗
are equal.
This result justifies part of what was observed in Example 4.2. If the classes
are not balanced, in the sense of equal error probability, then
x
∗
is not a
critical point
of SEE. However, Theorem 4.2 says nothing about the nature
(maximum or minimum) of such critical points. Indeed, as Fig. 4.3d has
shown, the obtained solution is not guaranteed to be an entropy minimum.
Before performing a more analytical treatment, Fig. 4.4 gives us an intu-
itive explanation for the fact. If the classes are suciently distant
P
E
(
e
)
exhibits a peaky PMF at
x
∗
(almost all probability is concentrated in
P
(
E
=0)), close to a Dirac-
δ
. Therefore, SEE is lower at
x
∗
(see the top
row of Fig. 4.4). On the other hand, when the classes get closer and the
amount of class overlap increases the PMF peakedness is gradually lost. At
