APPLICATION EXAMPLES OF THE FACE PROCESSING FRAMEWORK - 3D Face Processing: Modeling, Analysis and Synthesis - page 104

other hand, the face synthesis technique can be used to create synthetic avatar
to interact with users. Here we describe an integrated HCI environment where
our face processing techniques are used.

2.1 Overview

In the Beckman Institute for Advanced Science and Technology at the Uni-
versity of Illinois at Urbana-Champaign, there is an interdisciplinary project
called: “Multimodal Human Computer Interaction: Towards a Proactive Com-
puter”. The goal of the project is to (1) identify ways to provide the computer
with real-time information about the human user's cognitive, motivational and
emotional state; and (2) explore ways of making the computer take actions
proactively based on the states information. In this way, the computer will
no long only take commands from the user passively. Instead, it will initiate
actions based on user's states and thus make the human-computer interaction
more effectively.

The current application area for the project is science education, in which
children learn about the properties of gears via LEGO TM games. By learning
how to put gears together, the children can learn scientific principles about
ratio, forces and etc. The educational philosophy is such that learning through
exploration is valued. That is, rather than directing the child in carrying out
learning tasks, the goal is to (1) encourage exploration of gears and gear trains
in which principles of gears are learned; and (2) ask questions that encourage
thought and insight. The final goal is a HCI learning environment that engages
children in activities with the tasks, tracks their path and progress in the problem
space, and initiates communication (e.g. asking questions, making comments,
showing examples, etc.). These activities will then encourage exploration,
maintain and develop interest and move the child toward new understandings.

In this learning environment, the information exchanged among different
participants is multimodal. The computer is used to analyze the multimodel
input, including facial expression analysis, prosody analysis, context-based
word spotting, visual tracking of the task states and eye-gaze tracking. Based on
the multimodal input analysis, the computer can estimate the user states. Then
the computer can map the user states to actions to be taken. The mapping can
be designed by domain expert or learned from extensive examples. Finally, the
computer executes the action to guide the children in the next step of exploration.
One type of useful output from the computer is synthetic face animation. The
synthetic face avatar can be used to represent the computer to interact with
children and help engage user in the learning session.