Modeling and Synthesis of Realistic Visual Speech in 3D - 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body - page 280

process is quite time consuming and one would not want to repeat it for every
single face that has to be animated. This section describes how novel faces can
be animated, using visemes which could not be observed beforehand.
Such animation requires a number of steps:

1. Personalizing the visemes
The shape or “physiognomy” of the novel face is taken into account by
determining the face's relative position with respect to the neutral example
faces in a Face Space. This information is used to generate a set of visemes
specific for the novel face.
2. Automatic, audio-based animation
From fluent speech a file is generated that contains visemes and their
timing. This file is automatically transformed into an animation of the face
by producing a sequence of viseme expressions combined with intermedi-
ate co-articulation effects.
3. Possible modifications by the animator
As the animator should remain in control, tools are provided that allow the
animator to modify the result as desired. A “Viseme Space” can be roamed
using its independent components.

Personalizing the Visemes

A good animation requires visemes that are adapted to the shape or “physiog-
nomy” of the face at hand. Hence, one cannot simply copy or “clone” the
deformations that have been extracted from one of the example faces to a novel
face. Although it is not precisely known at this point how the viseme deforma-
tions depend on the physiognomy, visual improvements were observed by
adapting the visemes in a simple way described in this section.
Faces can be efficiently represented as points in a so-called “Face Space”
(Blanz et al., 1999). These points actually represent their deviation from the
average face. This can be done for the neutral faces from which the example
visemes have been extracted using the procedure described in the section,
Learning Viseme Expressions , as well as for a neutral, novel face. The example
faces span a hyper-plane in Face Space. By orthogonally projecting the novel
face onto this plane, a linear combination of the example faces is found that
comes closest to the projected novel face. This procedure is illustrated in Figure
10. Suppose we put the Face Space coordinates of the face that corresponds to
this projection into a single column vector F and, similarly, the coordinates of

nov