From Annotation to Multimodal Behavior - Coverbal Synchrony in Human-Machine Interaction - page 204

been designed to control them at various levels. Different factors need
to be considered when elaborating on a representation language: as to
which theoretical grounds the representation language is built; does
it depend of a specific technology; which values for its elements does
it consider; how much flexibility does it accept, etc.
In the remainder of this chapter, we first turn our attention to
the presentation of annotation schemes and then to representation
languages. We end the chapter by presenting some future trends.
2. Annotation Schemes
When studying human multimodal behavior, the starting point is
empirical data analysis which provides the necessary generalizations
and characteristics of the phenomena under study. The analysis is
usually conducted by annotating the data according to an appropriate
annotation scheme, and the annotations allow the data to be compared
with similar annotated data as well as being re-used in further studies.
Focusing on the interaction between human users and automated smart
agents (Embodied Conversational Agents, virtual world characters,
robot agents), this section discusses annotations from the point of
view of natural and intuitive communication, and aims to give an
overview of those aspects related to multimodal corpus annotation,
with the purpose of building models that can be used in a variety of
applications.
An annotation scheme is a collection of labels (tags) to be assigned
to data elements such that they describe the material with respect to
various physical, perceptual and functional dimensions. It concerns the
analysis of a phenomenon with the help of analytical categories, and
is to be distinguished from the representation or mark-up language
which refers to the format in which the annotation categories are
represented (see next section). It may include different analysis levels
on which the data labeling is performed, or contain only one level with
in-depth analysis categories. Metadata, such as the collection date and
setting, participants' demographics, etc., are often included within the
annotation scheme as well, and become important in studies related
to social and personality issues.
The main purpose of the annotation is to capture the phenomena
under study. Annotation categories are thus dependent on the diverse
theoretical assumptions made in the description of the phenomena;
for instance, linguistic annotations of the sentence structure may rely
on phrase-structure, dependency, or some other type of grammar
formalisms, all of which furnish the annotators with a different