Modeling Human Communication Dynamics for Virtual Human - Coverbal Synchrony in Human-Machine Interaction - page 370

other hand, a nod can be used for affirmation, emphasis or to hand
over the dialog turn. The context in which the behavior occurs can
transform the interpretation, as can subtle changes in the dynamics
of the behavior: head nods signaling affirmation versus emphasis
typically have different dynamics. Further, behaviors can be composed
with each other, further transforming their interpretation.
Additionally, the behaviors are often tightly synchronized and
changes in this synchronization can lead to significant changes in what
is conveyed to a listener. For instance, the stroke of a hand gesture,
a nod or an eyebrow raise individually or together are often used
to emphasize the significance of a word or phrase in the speech. To
achieve that emphasis, the behavior must be closely synchronized with
the utterance of the associated words being emphasized. Alteration
of the timing will change what words are being emphasized and
consequently change what is conveyed to a listener.
Achieving such synchronization in a virtual human can be difficult,
especially in the case of behaviors such as hand gestures that involve
relatively large-scale motion and multiple phases. Consider a beat
gesture, a staccato, often downward stroke of the hand that can be
used to provide emphasis. To perform a downward motion, the hand
must be raised in preparation for the stroke. After the stroke, the
hand can be held in a pose to provide further emphasis, followed by
a relaxation to a rest position. This sequence of behaviors occur in
alignment with the speech, so there must be sufficient time to prepare
for the stroke; the stroke and any post-stroke hold must be tightly
coordinated with the parts of the dialog that is being emphasized.
Further the relaxation may need to take into account co-articulation
that there will be subsequent gestures to be performed.
In addition to synchronization of speaker's behaviors, there
is synchronization between speaker and listeners as the speaker's
utterance unfolds. Listeners exhibit a variety of behaviors. These
behaviors include generic responses that signals the listener is attending
or broadly comprehending the speaker as well as specific responses
tightly coupled to a deeper understanding of, and reaction to, the
details of what the speaker is saying at that moment (Bavelas et al.,
2000). The speaker in turn will dynamically adapt what they are saying
in response to this feedback from the listener.
Such dynamic adjustments raise challenges in generating both the
verbal and nonverbal behaviors dynamically and incrementally for a
virtual human. Ideally a virtual human listener should respond to a
human speaker, providing generic feedback signaling attention as well
as more specific feedback that signals comprehension and reaction to