Modeling Human Communication Dynamics for Virtual Human - Coverbal Synchrony in Human-Machine Interaction - page 363

￿ Societal dynamic: We categorize the organizational (often referred
as meso-level) and societal (often referred as macro-level)
dynamics in this general category which emphasize the cultural
change in a large group or society. While this proposal does
not focus on societal dynamics, it is important to point out the
bottom-up and top-down influences. The bottom-up approach
emphasizes the influence of micro-dynamics (behavioral,
individual and interpersonal) on large-scale societal behaviors
(e.g., organizational behavior analysis based on audio micro-
dynamics (Pentland, 2004)). As important is the top-down
influence of society and culture on individual and interpersonal
dynamics.
In this topic chapter, we first discuss techniques to model the
behavior dynamics of virtual human as well as human participants.
We then address the challenge of multimodal dynamic and, more
specifically, the synchrony between speech and gestures for virtual
human. Finally, we present approaches to model the interpersonal
dynamic between speaker and listeners using state-of-the-art machine
learning. We finally conclude by discussing the future challenges
related to societal and cultural dynamics.
2. Behavior Dynamic: Virtual Human Animation
Virtual humans can be used to express portray a wide range of
behaviors, including synchronized speech, gestures and facial
expressions. In order to generate such expressions on a virtual human,
an animation must be synthesized and replayed on a digital character.
Such an animation can be generated from a variety of sources, such
as motion capture, hand-designed by a digital artist, or procedurally
generated from a motion synthesis algorithm. The synthesized motion
can be coordinated with audio and lip motion on the digital character
if the gesture is associated with an utterance. Digital 3D characters
that are human-like in appearance and responses are termed virtual
humans.
The use of digital characters is common in video games, simulations
and live action and animated feature films. The most widely used
method of generating a 3D character in motion involves combining a
3D geometric mesh, its surface colors or images, called textures, with
a set of joints combined in a hierarchy, called a skeleton. The skeleton
is then used to modify the 3D geometry via a deformation, or skinning
process, where each joint of the skeleton is related to one of more of
the faces of the geometry and modifies the geometry as the position