Graphics Reference
In-Depth Information
that cabinet that counts. The artist will specifically choose points that are close to
where the effect is going to happen, which may or may not show up in an automatic
corner detector. They might be near a gradual slope or a pattern.
The artist can also draw a pattern or outline around that point and say, “track
this stuff.” That outline doesn't need to be a square or rectangle — for example, a
rectangle might contain background pixels that you don't want to deal with. It could
be an arbitrary shape, which avoids the need to do some sort of automatic outlier
rejection on pixels you don't care about.
To follow the pattern through a sequence, we use a template-basedmethod based
on cross-correlation in three-channel color space. Of course, there are lots of prob-
lems with that. The object may rotate or slide around. The pattern tracker that we use
solves for not just translation and rotation, but also skew, perspective, and lighting
changes using an affine or nonlinear transformation as necessary. I definitely started
with Shi and Tomasi's Good Features to Track, because that's the classic approach,
but these days my approach is more related to a paper by J. P. Lewis that does a good
job of addressing lighting changes [
276
]. Of course, motion blur and occlusions are
always a real pain to deal with.
RJR: SIFT is incredibly popular in the computer vision community; what about in
visual effects?
Roble: Right now, we don't really use SIFT all that much here. It's isn't necessary for
the kind of pattern tracking through frames of a shot that we deal with all the time.
Also, the SIFT descriptor contains a much different kind of information than artists
are used to.
One idea I wanted to investigate using SIFT was as a pre-process before doing
corner matching. The problem is occlusion: the actors on the sets walk in front of the
features all the time. With a corner-matching algorithm, or something that's purely
template-based, once a feature gets occluded, by the time it becomes visible again,
the camera's often moved significantly, and it comes in as a whole new track. SIFT
seems like it could potentially help hook those tracks back together across the gap.
RJR: What's it like to place artificial features on a movie set?
Shankar: Every set is different; it's totally free-form. This is a good example of the
pragmatic side of filmmaking. It would be awesome to be able to place regularly
spaced markers with coded patterns on the set and write software to automatically
recognize their unique IDs, but the reality is you have to get in and out fast without
interfering with the crew or the actors.
When we're on a stage, the camera crew alone can take up half the afternoon and
you've got to throwupwhat markers you can in a fewminutes. You'll see little squares
of gaffer's tape on corners here and there because sometimes that's all you can get.
We often use markers that consist of big orange triangles, which contrast well with
blue screens. An advantage of these is that by looking at their edges they help us a
bit with estimating motion blur frommoving footage—but tracking throughmotion
blur is always a hard problem.
