Inference of Co-occurring Classes: Multi-class and Multi-label Classification - Computational Intelligence Paradigms in Advanced Pattern Classification - page 172

often also dictates the choice of representation manner of the information or
knowledge domain, the type and number of features or attributes that serve as in-
puts to the classification system, and the classification method.
This chapter describes in short the common classification process, reviews
challenges in annotation of data with multiple class-labels, and surveys recent ap-
proaches to multi-class and multi-label classification in various knowledge
domains. The chapter then presents the knowledge and behavioral domain of af-
fective states (emotions, mental states, attitudes and the like), and a classification
algorithm which was used for inferring the levels of co-occurring affective states
from their non-verbal expressions in speech. Unlike other fields, the field of affec-
tive states has no definite “ground truth” for verification of the annotation. In addi-
tion, the choice of taxonomy has an important effect on the design. In this case of
multi-label classification, the annotation of the data includes only one label per
sample, therefore the algorithm performs semi-blind multi-label classification. In
addition, there is an inherent sparsity because different sets of features distinguish
different pairs of classes.
2 Applications
The most commonly researched applications of multi-class and multi-label tech-
niques relates to annotation, search and retrieval of documents in huge repositories
of data of various modalities, such as text [13, 23, 43, 49, 67, 77], image [8, 18,
29, 33, 56, 68, 77, 79], video [7, 73] and music [31, 39, 71]. Annotation means
assigning items to categories, or labeling data samples with class-labels (the
semantic term associated with the class concept). Annotation in the context of
applications refers to problems in which the manually labelled sets have to be ex-
tended. Otherwise, most classification problems can be also referred to as annota-
tion, i.e. automatically assigning labels to samples. Search of items that belong to
the required categories in large datasets, and retrieval of relevant items that belong
to these categories.
However, multi-class and multi-label classification schemes apply also to a
very wide variety of current and foreseeable applications that relate to other know-
ledge domains and applications, for example, for the analysis of human behavioral
cues [17, 37, 60, 73, 78]. These include not only the analysis of the verbal speech
but also the analysis of cues such as non-verbal expressions in speech (prosody
and paralinguistic cues), facial expressions, posture, hand and full body gestures
and actions, as well as analysis of physiological cues from various measurement
equipments, ranging from Electro-cardiograms (ECG) to functional magnetic re-
sonance (FMRI) images. These behavioral cues can be analyzed for a single per-
son or for groups.
Multi-class and multi-label classification of human behavioral cues can be used
for identifying and tracking individuals, as well as for analysis of affective and so-
cial cues of emotional and mental states, moods, attitudes, and the like. This has a
wide scope of other applications, such as human-machine interactions (computers,
robots, smart and mobile environments, etc), content-based context retrieval,
video summarization, for assistive technology, technology aimed at improving the