Image Processing Reference
which is resolved by using shading information and, when necessary, smoothness constraints. This
approach uses more of the textural information than all those that rely solely on interest points,
which ignore most of the image pixels. As discussed in Chapter 2 , when operating in a well-defined
domain such as face reconstruction for which there exists not only a geometric model but also an
appearance model, it becomes possible to use the image texture even more extensively by using an
analysis-by-synthesis approach Blanz and Vetter [ 1999 ], Romdhani and Vetter [ 2003 ] to estimate
both shape and illumination parameters, as shown in Fig. 4.10 .
While the mesh-based parameterization is the most common representation for template-
based reconstruction, it is not the only possible one. As mentioned in Section 2.3 , control points
based parameterizations have been proposed to model non-rigid objects. Recently, in Brunet et al.
[ 2010 ], a free-form deformation model was recently used for monocular reconstruction. This model
has the advantage of making it easy to compute a global smoothness regularizer by exploiting the
second derivatives of the B-spline basis functions that define the deformations. This regularizer used
in conjunction with additional distance constraints was shown to outperform several state-of-the-art
methods in terms of reconstruction accuracy.
18.104.22.168 Local Smoothness
The methods of Section 22.214.171.124 , which rely on regularization models expressed as linear combinations
of deformation modes, are good at recovering the shape of surfaces that deform relatively smoothly.
However, they do not perform as well when deformations are more local or sharper, such as those
depicted by Figs. 4.4 and 4.11 where folds appear on the surface. In theory, handling such local
deformations could be achieved by using a much larger number of global modes. However, in practice,
this would mean introducing far more variables—the weights associated to the modes—which, for
computational reasons, could easily make the previous approaches impractical.
An approach to overcoming this problem by replacing global smoothness constraints with
local ones was introduced in Salzmann et al. [ 2008b ]. It starts from the following observations. First,
locally, all parts of a physically homogeneous surface obey the same deformation rules. Second, these
local deformations are more constrained than those of the global surface and can be learned from
fewer examples. To exploit this, it is the manifold of local, as opposed to global, surface deformations
that is represented. In Salzmann et al. [ 2008b ], these local models were learned using a nonlinear
technique. However, this yields non-convex objective functions, and is therefore only appropriate in
a tracking framework. Thus, these nonlinear models were later replaced by linear ones, where each
local patch is represented as a linear combination of modes Salzmann and Fua [ 2011 ].
In Salzmann and Fua [ 2011 ], this representation was used to regularize the reconstruction of
the global surface by penalizing large local shape deviations from the learned linear manifold. To
this end, as shown in Fig. 4.12 , the mesh representing the surface is subdivided into overlapping
patches. Each patch is taken as an N p × N p square mesh, with N p =
5 in Salzmann and Fua [ 2011 ].
Note that this does not truly limit the approach to rectangular surfaces, since patches can be defined
partially outside the global shape. Given an instance of a surface, each one of its local patches is