FormalizingNon-Rigid Structure fromMotion - Deformable Surface 3D Reconstruction from Monocular Images - page 52

We will first introduce a general formulation of NRSFM under a weak perspective, or affine,
camera model. In this case, the projection of a 3D point q i can be written as

d u i
v i

=

Rq i + t , (5.1)

where R contains the first two rows of the full camera rotation matrix, and t is the 2 1 camera

translation vector. Assuming no distortion, the matrix of internal camera parameters reduces to a
single focal length, which was here absorbed by the scalar d . Since, in NRSFM, the notion of facet
is absent, the same d is used for all the points. We will then discuss NRSFM under full perspective
projection, where the projection of a 3D point q i

×

is expressed as

⎡ ⎤

u i
v i
1

A Rq i + t ,

⎣ ⎦ =

d i (5.2)

with A the 3 × 3 matrix of internal camera parameters.
As in the case of template-based reconstruction, correspondences alone are insufficient for
unambiguous reconstruction. To reduce the ambiguities, most NRSFM methods rely on a linear
subspace model to constrain the deformations of the 3D points. Whereas in the template-based
case, the deformation modes could be infered from the reference mesh using a technique such
as Salzmann et al. [ 2007c ], this is no longer the case for NRSFM. Consequently, the modes are
typically taken as additional unknown variables and recovered at the same time as their coefficients,
along with the rotation and translation for each frame of the sequence. Note that, since the modes
are obtained neither from a large dataset of deformed surfaces as in Salzmann et al. [ 2007c ] nor
by modal analysis, they do not necessarily favor smooth deformations. Depending on the number
of basis shapes involved, they rather encourage the deformations to remain simple. Since most
NRSFM techniques exploit this linear subspace representation, we will describe it as part of the
general approach. However, as will be discussed in Chapter 6 , some very recent methods depart
significantly from this initial formulation.
In theory, NRSFM is more generally applicable than template-based shape recovery, since
it requires neither a calibrated camera, nor a reference template. In practice, however, because of
the higher number of degrees-of-freedom, NRSFM methods are subject to more ambiguities and
are more sensitive to measurement noise. As a consequence, to be as effective as template-based
approaches, they often require stronger constraints and a good initialization. In many cases, the
latter is obtained by applying a rigid structure-from-motion algorithm.
In the remainder of this chapter, we start by reviewing the problem formulation for the weak
perspective case as introduced in Bregler et al. [ 2000 ]. We then discuss NRSFM in the perspective
case, as proposed by Hartley and Vidal [ 2008 ], Xiao and Kanade [ 2005 ].