Introduction - Minimum Error Entropy Classification - page 10

their main features. We then move on to derive risk descriptions in terms
of error random variables as a preliminary step to the introduction of error
entropies as risks. Risk functionals are then put under the Information Theo-
retic Learning perspective, advantages and pitfalls on the application of error
entropies to classification tasks are identified, and specific entropic properties
influencing the MEE approach are discussed and exemplified. Chapter 2 con-
cludes with a discussion that risk functionals do not perform equally in what
respects the attainment of min P e solutions, and provides concrete evidence
to this respect.
Chapter 3 presents theoretical and experimental results regarding the ap-
plication of the MEE approach to simple classifiers having continuous error
distributions, and which are used as building blocks of more sophisticated
machines: linear discriminants, perceptrons, hypersphere neurons, and data
splitters. Classification examples for artificial and realistic datasets (based
on real-world data) are presented. Consistency and generalization issues are
studied and discussed. The min P e issue is also analyzed in detail both from a
theoretical (theoretical MEE) and a practical (empirical MEE) perspective.
Regarding the practical perspective and the attainment of min P e solutions,
the influence of kernel smoothing on error PDF estimates is also scrutinized.
In Chap. 4 we devote our attention to the application of MEE to discrete
error distributions, provided by simple classifiers having a threshold activa-
tion function. We study with care the case of a data splitter for univariate
input discrimination and prove some results regarding the min P e issue. Two
versions of empirical MEE splits are proposed and experimentally tested both
with artificial and real-world data: the kernel-based approach and the resub-
stitution estimate approach. The latter introduces the concept of MEE splits
for decision tree classifiers. This new splitting criteria is carefully compared
to the classic splitting criteria. The chapter concludes with the analysis of
a discrete-output perceptron, investigating how error entropy critical points
relate to min P e solutions and then analyzing the special case of bivariate
Gaussian two-classs problems.
Chapter 5 introduces two new risk functionals. First, the EE-inspired Zero-
Error Density (ZED) risk functional is defined and illustrated both for the
discrete and continuous error distribution cases. Its empirical version for the
case of continuous error distributions (empirical ZED) is presented. Empirical
ZED is then tested in a perceptron learning task and its gradient behavior is
compared to the ones of other classic risks. Connections between ZED and the
correntropy measure are also discussed. Finally, we present as a generalization
of the ZED risk, the Exponential (EXP) risk functional, a parameterized risk
su ciently flexible to emulate a whole range of behaviors, including the ones
of ZED and other classic risks.
Chapter 6 describes various types of classifiers applying the MEE con-
cept. Popular classifiers such as multilayer perceptrons and decision trees are
studied, as well as more sophisticated types of classifiers, such as recurrent,