Graphics Reference
In-Depth Information
things have to be very robust and give very predictable results almost all the time, so
they'd much rather have something that requires a little bit of user interaction but
gets you eighty to ninety percent of the way there all the time. In academia, it's often
all about minimizing user interaction, giving you fantastic results for a subset of the
images you'd experience in reality. You look at many of these matting papers and
you see very similar types of test images. Thankfully academic research is coming
around to the point of view that interactive user input is considered to be good, such
as adding incremental foreground and background scribbles to refine a matte. My
experience was that academic papers sometimes used very complicated models that
tend to work very well for some datasets and extremely poorly on other datasets,
whereas the film industry tends to split the difference and go for something that
works reliably most of the time.
Another big issue with the film industry is that you're always working on video.
A lot of these single-image techniques work really well given user input, but getting
something to work well on video is incredibly hard because you need that temporal
consistency. On the other hand, most algorithms that have temporal consistency
built into them are amazingly slow and will only work on very small, very short image
sequences. It's hard to get the best of bothworlds. As far as I understand, everything's
very manual in the film industry when it comes to things like matting. People will
still roto (rotoscope) out stuff by hand if they don't have a blue screen shot instead of
using some of the more advanced computer vision techniques. That's disappointing
since I'd like to see some of the progress that's made on the academic side come back
to the film industry — but getting that reliability is hard.
RJR: Can you describe how an artist does blue-screen or green-screen matting in
practice?
Lambert: Say I'm assigned a shot that has a green-screen background and I have to
pull a key to put over a particular new background. My keying programwill have sev-
eral built-in algorithms to try: Ultimatte, Primatte, Knockout, channel differencing,
and so on. Each of these corresponds to a rule for processing the RGB colors of each
pixel to produce a matte. For example, you may subtract weighted combinations of
the red and blue from the green channel to make the alpha, changing the weights
of the red and green channels depending on the subject matter. Often as part of the
process you click around on the green screen to find a background color that will
produce a good matte. From experience, you know that if you pick a brighter color,
you'll pick up a certain amount of foreground detail, and if you go with a darker color,
you'll pick up another kind of detail.
I invented one algorithm called the IBK keyer that's part of a software pack-
age called Nuke. It generates a background model on a per-pixel basis, so rather
than just feeding the keyer one background green color you actually feed it a
whole varying-green image. In essence, you're expanding the green-screen col-
ors into the foreground to make a clean green-screen plate that has a nice color
gradient.
When I was first trying to work all of this stuff out, I was obsessed with finding a
perfect way to do keying, but over time you realize that you never pull a key without
