TTS-driven Synthetic Behavior Generation Model for Embodied Conversational Agents - Coverbal Synchrony in Human-Machine Interaction - page 331

is then manifested at its full span at the end of the emphasized or
primary-accented syllable. And the prosodic features of speech define
the dynamics of coverbal gesture in the form of movement-phases
(Kendon, 2004). The PLATTOS, therefore, synchronizes the meaning
expressed by gesture to be compatible with the one expressed by the
connected speech fragment (e.g. the shape of the dominant expression
manifested whilst synthesizing verbal indicators).
In the system, features driving the coverbal behavior are deduced
from raw text, and the processing steps for the planning and generating
of non-verbal behavior incorporate semiotic grammar, gesture
dictionary, and lexical affiliation. Semiotic grammar, on the one hand,
incorporates rules by the use of which the most meaningful parts of a
text are selected, and by using which the dynamical alignment (in the
forms of movement-phases (Kendon, 2004)) regarding verbal and non-
verbal behavior can be performed. The gesture dictionary and lexical
affiliation, on the other hand, incorporate the semantic relation between
meaning (meaningful word) and meaning representation (shape
manifestations). Additionally, the PLATTOS system synchronizes
the coverbal expressions in such a way that the meaningful part of
a gesture (the so-called stroke) co-occurs with the most prominent
segment of the accompanying generated speech, as proposed in
McNeill (1992). Furthermore, the PLATTOS system also transforms
the generated coverbal expressions into a form understandable to an
ECA-based behavior realization engine supporting mark-up languages,
such as BML (Vilhjalmsson et al., 2007) and EVA-SCRIPT (Mlakar and
Rojc, 2011).
The TTS-driven coverbal behavior synthesis engine is based on
the following baseline (Allwood, 2001; Esposito and Esposito, 2011;
Kendon, 2004; McNeill, 2005). The first part of the human gesture
production process relies on the lexicon and lexical affiliates, and
can, therefore, be linked directly to words, word phrases (or even
to smaller utterances, such as syllables, etc.). Pierce has defined that
such bodily movements can be linguistically grouped into semiotic
types (Peirce, 1958). The properties of semiotic types may be identified
by key-utterances (mostly words, word phrases, word types, and
word-type-orders) (Deacon, 2003). The second part of the human
gesture production process relies on the paralinguistic and extra-
linguistic information conveyed by speech and gestures (Wallbott,
1998). Meaning is, in addition to linguistic relation, also conveyed
through non-lexical expressions carrying specific communicative
values, such as: turn-taking, feedback, speech-repair, and sequencing.
These expressions are generated based on a complex mechanism of
communicative functions (Kipp, 2001; McNeill, 1992; Peirce, 1958).