Information Technology Reference
In-Depth Information
Chapter 3
MEE with Continuous Errors
The present chapter analyzes the behavior of classifiers characterized by
continuous distributions of the errors, which are trained to minimize error-
entropy functionals, namely the Shannon and Rényi's quadratic entropies,
presented in the preceding chapter. The analysis focus mainly the classifier
problem (does the MEE solution correspond to the min
P
e
solution?), but
consistency and generalization issues are also addressed.
We only consider simple classifiers, used as building blocks of more sophis-
ticated ones. These classifiers are restricted to solving two-class problems; as
seen in Sect. 2.2.1, the corresponding univariate error PDF is then expressed
as follows for class targets in
{−
1
,
1
}
:
f
E
(
e
)=
pf
Y |
1
(1
−
e
)+
qf
Y |−
1
(
−
1
−
e
)
.
(3.1)
Note that the two-class restriction doesn't amount to a loss of generality
because all multiclass problems are solved in practice by applying a battery
of two-class solvers either in parallel or sequentially.
As we shall soon see, error entropy formulas become rather involved even
for simple classifiers applied to simple datasets. Derivation of MEE solutions
by some kind of direct parameter estimation method is out of the question.
Instead, one has to resort to iterative optimization algorithms. We will use one
such algorithm, and a simple one, to analyze the practical behavior of these
machines: the gradient descent algorithm. It is simple enough, facilitates the
inspection of convergent behavior, and with proper care affords global minima
solutions.
