TTS-driven Synthetic Behavior Generation Model for Embodied Conversational Agents - Coverbal Synchrony in Human-Machine Interaction - page 334

most closely to a gesture in meaning”. The affiliates are used in order
to establish explicit relationships between linguistic units (word, word
phrases, word types, word-type-order patterns), and conversational
shapes (gestures). The most natural way to establish these relations
is by observing natural conversational behavior (e.g. gesture analysis
by annotation of multimodal corpus). The annotation scheme used for
building the Gesticon used within the PLATTOS system had already
been proposed in Mlakar and Rojc (2012). The gesture topology and
the formal model used are presented in Figure 2. Each conversational
expression is encoded as a series of consecutive (representative) poses,
encoded with additional linguistic information. For example, the
formal model of the annotation captures morphological correlation
separately for each body-part, and in the form of movement phases,
and movement phrases. Each pose is also described with a semiotic
tag, as by meaning the coverbal expression (speech + gesture) carries.
Although there are some significant differences between speaker-styles
(e.g. intensity, frequency, particular shapes used, etc.), some general
trends can be identified (Ng-Thow-Hing et al., 2010). For instance,
specific gesture classes have a tendency to co-occur with certain words,
phrases, or parts-of-speech (semantically and morphologically).
Movement phrase encapsulates sequential segments of coverbal
behavior (sequence of gestures) into a meaningful sequence of
synchronized speech fragments and gestures, e.g. conversational acts.
Each movement phrase is further segmented into movement phases . A
movement phase describes the five stages of movement propagation
of a single gesture. The most important movement phase is the stroke
phase carrying the majority of meaning. The stroke phase is a
mandatory part of any gesture (similarly to noun, verb or pronoun
being mandatory for grammatically correct sentence formulation in
most languages). The stroke phase may be followed by either the post-
stroke hold movement phase, or the retraction phase. The hold phases
are defined as a short stop of movement prior/after the stroke phase.
They are used to synchronize the beginnings of verbal and coverbal
information, and to extend the meaning conveyed through the stroke
movement phase. The hold movement phases are, in speech, indicated
by short pauses ( sil elements), and by the residue (part not following
stroke and not assigned to stroke) of the 'meaningful word (phrase)'.
The holds are most evident in iconic expressions, especially those
Figure 2. The formal-model of expressively oriented annotation scheme.