FormalizingNon-Rigid Structure fromMotion - Deformable Surface 3D Reconstruction from Monocular Images - page 56

temporal smoothness. Similarly, there also is an ambiguity between the magnitude of the basis shapes
and their corresponding coefficients.
Furthermore, the same global rotation-translation ambiguity as in the rigid case re-
mains in non-rigid structure from motion solutions. In the full perspective case, it was shown
in Hartley and Vidal [ 2008 ] that, in the uncalibrated case, the solution is only determined up to
a linear transformation. However, with a calibrated camera, this ambiguity reduces to the same
undetermined global rotation as in the weak perspective case.
Finally, in addition to the ambiguities inherent to the problem that have been formally proved
in the above-mentioned papers, other ambiguities have been observed in practice. In Torresani et al.
[ 2003 ], it was noted that if too many basis shapes were required, the reconstruction problem became
ambiguous. Similarly, in Bartoli et al. [ 2008 ], the problem that treating all modes equally results
in ambiguities due to the potential dependencies of the modes. Solutions to these problem involv-
ing higher order deformation models, or coarse-to-fine modes computation will be discussed in
Chapter 6 .
5.5 THEMISSINGDATA PROBLEM
A weakness of non-rigid structure-from-motion techniques is their sensitivity to missing data and
mismatches. Several solutions to these problems have been proposed.
The first publications to address the missing data problem were Brand [ 2001 ], Torresani et al.
[ 2001 ]. Both proposed to exploit fully tracked points to infer the missing data. In particular, they
follow the idea introduced by Irani [ 1999 ] for optical flow estimation, and establish a basis flow
using the fully tracked points. More specifically, assuming that W has rank r , all columns of W can
be modeled as linear combinations of r basis tracks. If at least r points have been tracked throughout
the whole sequence, this basis can be obtained by singular value decomposition of the corresponding
part of W . To fill in the missing data in matrix W , they follow the formulation of Lucas and Kanade
[ 1981 ], and express the locations of the image points in terms of the basis coefficients. The resulting
system of equations is then solved iteratively to obtain the correct coefficients and thus predict the
location of points that were lost during tracking.
While this is effective when some points have been tracked in all images, it would still fail in the
more realistic case where points are visible for some frames and then disappear. In Olsen and Bartoli
[ 2008 ], this problem was addressed by separating the sequence into blocks of frames whose motion
parameters can then be estimated. Given the motion in the individual blocks, the basis shapes can
be estimated later by exploiting priors.
A different line of works followed the idea introduced in Marques and Costeira [ 2009 ] to deal
with missing data in rigid structure-from-motion. In this paper, it was noted that rigid SFM with
missing data can become ambiguous when the available points are in a degenerate configuration, such
as on a plane. To overcome this issue, the authors introduced the notion of motion manifold , which
constrains the recovered motion to be feasible. The available tracks can then be projected on this
manifold, which makes reconstruction robust to those degenerate scenarios. In Paladini et al. [ 2009 ],