Modeling Human Communication Dynamics for Virtual Human - Coverbal Synchrony in Human-Machine Interaction - page 375

The approach for modeling listener backchannel feedback based on
speaker behaviors is depicted in Figure 5. We first describe the main
ideas behind contextual prediction, then show some of the audiovisual
features commonly used and finally describe recent prediction models.
4.1 Contextual prediction
Natural conversation is fluid and highly interactive. Participants seem
tightly enmeshed in something like a dance, rapidly detecting and
responding, not only to each other's words, but to speech prosody,
gesture, gaze, posture, and facial expression movements. These “extra-
linguistic” signals play a powerful role in determining the nature
of social exchange. When these signals are positive, coordinated
and reciprocated, they can lead to feelings of rapport and promote
beneficial outcomes in such diverse areas as negotiations and conflict
resolution (Drolet and Morris, 2000; Goldberg, 2005), psychotherapeutic
effectiveness (Tsui and Schultz, 2005), improved test performance in
classrooms (Fuchs, 1987), and improved quality of child care (Burns,
1984). Not surprisingly, supporting such fluid interactions has become
an important topic of human-centered research.
In the contextual prediction framework, the prediction model
automatically learns which subset of a speaker's verbal and nonverbal
actions influences the listener's nonverbal behaviors, finds the optimal
way to dynamically integrate the relevant speaker actions and outputs
probabilistic measurements describing the likelihood of a listener
Figure 5. Contextual prediction example: online prediction of the listener's backchannel based
on the speaker's contextual features. In the contextual prediction framework, the prediction
model automatically (1) learns which subset of the speaker's verbal and nonverbal actions
infl uences the listener's nonverbal behaviors, (2) fi nds the optimal way to dynamically
integrate the relevant speaker actions and (3) outputs probabilistic measurements describing
how likely listener nonverbal behaviors are.
(Color image of this fi gure appears in the color plate section at the end of the topic.)