Information Technology Reference
In-Depth Information
Chapter 4
MEE with Discrete Errors
In this chapter we turn our attention to classifiers with a discrete error vari-
able,
E
=
T
Z
. The need to operate with discrete errors arises when clas-
sifiers only produce a discrete output, as for instance the univariate data
splitters used by decision trees. For regression-like classifiers, producing
Z
as
a thresholding of a continuous output,
Z
=
θ
(
Y
), such a need does not arise.
The present analysis of MEE with discrete errors, besides complementing our
understanding of EE-based classifiers will also serve to lay the foundations of
EE-based decision trees later in the chapter.
As before, we focus our attention on the discrete output of simple clas-
sifiers, say with
Z
−
∈{−
1
,
1
}
of two-class problems with targets
{−
1
,
1
}
.In
this framework, the error variable
E
=
T
−
Z
takes value in
{−
2
,
0
,
2
}
where
E
=
2 and
E
=2correspond to misclassification errors (in class
ω
−
1
and
ω
1
, respectively), while
E
=0corresponds to correct classification. The error
probability mass function
(PMF) is therefore written as
−
⎧
⎨
P
−
1
,
e
=
−
2
P
E
(
e
)=
1
−
P
−
1
−
P
1
,e
=0
(4.1)
⎩
P
1
,
e
=2
where
P
−
1
=
P
(
E
=
−
2) and
P
1
=
P
(
E
=2)are the probabilities of error
for each class.
Given the nature of the error variable, discrete entropy formulas already
introduced in (2.39) and (2.40) must be used. We extensively use the Shannon
entropy definition and write:
H
S
(
E
)=
−
P
−
1
ln
P
−
1
−
P
1
ln
P
1
−
(1
−
P
−
1
−
P
1
)ln(1
−
P
−
1
−
P
1
)
.
(4.2)
In the following we analyze (4.2) as a function of the parameters of the
classifier, starting from the special case of a data splitter with univariate
input data and then evolving to the more general discrete-output perceptron.
