From Annotation to Multimodal Behavior - Coverbal Synchrony in Human-Machine Interaction - page 203

CHAPTER 8
From Annotation to
Multimodal Behavior
Kristiina Jokinen and Catherine Pelachaud
1. Introduction
The usage of multimodal corpora is crucial for studying human
behavior as well as modeling embodied conversational agents (Cowie
et al., 2010a). Gathering data and creating corpora is complex.
Interested readers can read work by Cowie and Douglas-Cowie on
the topic (Cowie et al., 2010b) wherein different methodologies for
creating corpora are described.
Once the corpora are gathered, there are the issues of annotating
them. Defining an annotation schema is very much in relation to the
phenomenon to be studied. When defining, it is necessary to decide
which information should be marked and in what form, whether it
should be continuous values or discrete ones, on which theoretical
grounds should the annotation schemes rely, and how to mark
dependencies. A lot of effort is put into deriving annotation schemes
that can be used reliably by researchers working on a wide-variety
of projects. Several schemas have been designed (including some
standards such as ISO-Dit++ (Iso-dit)).
Embodied Conversational Agents are autonomous entities able
to communicate verbally and non-verbally. They are part of an
interaction system. They can be controlled at high levels where their
communicative intentions and emotional states are specified or at low
levels when describing their behavior. Representation languages have