Modeling and Synthesis of Realistic Visual Speech in 3D - 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body - page 291

an animator, who still has to be able to modify the results. This remains fully
possible within the proposed framework.
The proposed method yields realistic results, with more detail in the underlying,
3D models than usual. The RBF morphing, described in the section, Learning
Viseme Expressions, has been implemented with special care to make this
possible. The way of combining the visemes to a novel face seems to be both
novel and effective. By giving the animator control over independent, rather than
the more usual, principal components this space can be navigated in a more
intuitive way.
Although we believe that our results can already be of help to an animator,
several improvements can be imagined. The following issues are a few ex-
amples. Currently, a fixed texture map is used for all the visemes, but the 3D
acquisition method allows us to extract a separate texture map for every viseme.
This will help to create the impression of wrinkles on rounded lips, for instance.
Another issue is the selection of the visemes. For the moment, only a rounded and
widened version of the consonants has been included. In reality, an /m/ in ama
lies between that in omo and imi . There is a kind of gradual change from umu ,
over omo , ama , and eme , up to imi . Accordingly, more versions of the visemes
can be considered. Another possible extension is the inclusion of tongue position
in the viseme classification. Some of the consonant classes have to be subdivided
in that case. A distinction has to be made between, e.g., / l / and / n / on the one
hand, and / g / and / k / on the other.
It is also possible to take more example images, until they span Face Space very
well. In that case, the final viseme cloning step in our viseme personalization can
probably be left out.
Last, but not least, speech-oriented animation needs to be combined with other
forms of facial deformations. Emotions are probably the most important ex-
ample. It will be interesting to see what else is needed to combine, e.g., visemes
and emotions and keep the overall impression natural. All these expressions call
on the same facial muscles. It remains to be seen whether linear superpositions
of the different deformations really suffice, as it was the case in our preliminary
experiments.
Acknowledgments
This research has been supported by the ETH Research Council (Visemes
Project), the KULeuven Research Council (GOA VHS+ Project), and the EC