Information Technology Reference
In-Depth Information
3.1 Theoretical and Empirical MEE
3.1.1 Computational Issues
As with any other risk functional, the practical application of the MEE ap-
proach relies on using adequate estimates of the EE risks (2.36), (2.37) and
(2.38). This implies using estimates of the error density, based on the
n
error
values produced by the classifier. Given an i.i.d. sample (
x
1
, ..., x
n
) from
some continuous distribution with density
f
(
x
), its estimate can be obtained
in an ecient way by the Parzen window method (see Appendix E), which
produces the estimate
h
K
x
=
n
n
f
n
(
x
)=
1
n
1
−
x
j
1
n
f
(
x
)
≡
K
h
(
x
−
x
j
)
.
(3.2)
h
j
=1
j
=1
This estimate is also known as
kernel density estimate
(KDE). Properties
and optimal choice of the
bandwidth
h
for a
kernel
function
K
are discussed
in Appendix E, where the justification to use the Gaussian kernel
G
h
(
x
)=
exp(
x
2
/
2
h
2
)
/
(
√
2
πh
) is also provided.
Applying (3.2) with a Gaussian kernel to the error values
e
i
,oneobtains
estimates
−
n
f
(
e
i
)=
1
n
G
h
(
e
i
−
e
j
)
,
(3.3)
j
=1
allowing the estimation of the error entropies as follows:
n
n
1
n
1
n
ln
f
(
e
i
)
,
(3.4)
H
S
(
E
)=
H
S
(
E
)=
E
[
−
ln
f
(
e
)] :
−
ln
f
(
e
i
)
≈−
i
=1
i
=1
n
n
ln
1
n
ln
1
n
H
R
2
(
E
)=
f
(
e
i
)
.
H
R
2
(
E
)=
−
ln
E
[
f
(
e
)] :
−
f
(
e
i
)
≈−
i
=1
i
=1
(3.5)
Formulas (3.4) and (3.5) of
H
S
and
H
R
2
are plug-in (or resubstitution) esti-
mates of
H
S
and
H
R
2
(see Appendix F). The minimization of these (or other)
empirical EEs corresponds to the
empirical MEE
approach used in practical
applications.
From the theoretical point of view, we are interested in analyzing the
behavior of the
theoretical MEE
, which for the Shannon and Rényi's quadratic
entropies corresponds to the minimization of the risks expressed by formulas
(2.36) and (2.38). Unfortunately, theoretical EEs can only be analyzed based
on closed-form expressions in simple classifiers with simple input PDFs, such
as uniform or Gaussian. For more realistic settings, involving more complex
distributions, one has to resort to numerical simulation, which can be carried
