Analysis and Synthesis of Facial Expressions - 3D Modeling and Animation: Synthesis and Analysis Techniques for the Human Body - page 246

optical flow or motion field, additional constraints are required. Instead of using
heuristical smoothness constraints, explicit knowledge about the shape and
motion characteristics of the object is exploited. Any 2-D motion model can be
used as an additional motion constraint in order to reduce the number of
unknowns to the number of motion parameters of the corresponding model. In
that case, it is assumed that the motion model is valid for the complete object. An
over-determined system of equations is obtained that can be solved robustly for
the unknown motion and deformation parameters in a least-squares sense.
In the case of facial expression analysis, the motion and deformation model can
be taken from the shape and the motion characteristics of the head model
description. In this context, a triangular B-spline model (Eisert et al., 1998a) is
used to represent the face of a person. For rendering purposes, the continuous
spline surface is discretized and approximated by a triangle mesh as shown in
Figure 6. The surface can be deformed by moving the spline's control points and
thus affecting the shape of the underlying mesh. A set of facial animation
parameters (FAPs) according to the MPEG-4 standard (MPEG, 1999) charac-
terizes the current facial expression and has to be estimated from the image
sequence. By concatenating all transformations in the head model deformation
and using knowledge from the perspective camera model, a relation between
image displacements and FAPs can be analytically derived

( ) d = f FAP , FAP , , FAP . (2)

0 1 N − 1

Combining this motion constraint with the optical flow constraint (1) leads to a
linear system of equations for the unknown FAPs. Solving this linear system in
a least squares sense, results in a set of facial animation parameters that
determines the current facial expression of the person in the image sequence.

Hierarchical Framework

Since the optical flow constraint equation (1) is derived assuming the image
intensity to be linear, it is only valid for small motion displacements between two
successive frames. To overcome this limitation, a hierarchical framework can be
used (Eisert et al., 1998a). First, a rough estimate of the facial motion and
deformation parameters is determined from sub-sampled and low-pass filtered
images, where the linear intensity assumption is valid over a wider range. The
3-D model is motion compensated and the remaining motion parameter errors are
reduced on frames having higher resolutions.