Matchmoving - Computer Vision for Visual Effects - page 207

6

Matchmoving

Matchmoving , also known as camera tracking , is a major aspect of modern visual
effects. It's the key underlying process that allows visual effects artists to convincingly
insert computer-generated elements and characters into a live-action plate, so that
everything appears to “live in” a consistent three-dimensional world. Ineverymodern
action movie (and even many non-action movies), the first step after acquiring live
footage is to track the camera to enable the additionof spatially accurate visual effects.
The basic problem is to determine, using a given video sequence as input, the
three-dimensional location and orientation of the camera at every frame with respect
to landmarks in the scene. Depending on the situation, we may have some prior
information — such as estimates of the focal length from the camera's lens barrel
or labeled landmarks with known 3D coordinates — or the video may come from an
entirely unknown camera and environment.
Matchmoving is fundamentally the same as a computer vision problem called
structure frommotion . In fact, several of the main matchmoving software packages
for visual effects grew directly out of academic research discussed in this chapter.
In turn, structure from motion is closely related to photogrammetry , mathematical
techniques used by surveyors to estimate the shape of buildings and terrain from
multiple images. Many structure frommotion techniques “discovered” by computer
vision researchers in the 1990s share key steps with photogrammetric techniques
developed by cartographers and geodesists 1 in the 1950s or earlier. Finally, structure
from motion is closely related to the problem of simultaneous location and map-
ping or SLAM from robotics, in which a mobile robot must self-localize by taking
measurements of its environment.
In a way, matchmoving ties together the topics discussed so far —matting, com-
positing, feature tracking, and dense correspondence — as well as the remaining
topics in the topic, motion capture and three-dimensional data acquisition. That is,
the matchmoving process begins with a set of feature points tracked though the
video sequence using any of the methods discussed in Chapter 4 . We can then
apply matting techniques from Chapter 2 to separate foreground and background
objects, and compositing techniques fromChapter 3 to insert new elements between
the segmented layers. These elements could include computer-generated charac-
ters animated with the help of motion capture data, as discussed in Chapter 7 .

1 Scientists who study geodesy, or the measurement of different aspects of the earth.

207