Graphics Reference
In-Depth Information
Chatterjee: At Cinesite, I was able to apply some of the texture synthesis research from
my PhD for inpainting andwire removal problems. At that time, in themid-nineties, I
was using Markov RandomField approaches to resynthesize texture; the artist would
outline the areas of defects, and I would run my texture synthesis program to fill in
the pixels. A common example was painting out marker balls that had been placed
around a set for camera tracking.
We used to use a wire removal tool that was similar to PDE-based inpainting, but
it didn't work very well; you could see that something had been done. They used
to call it the “glass rod” effect because that's what the filled-in wire region kind of
looked like. Later I moved on to Efros et al.'s technique, and then to Criminisi et al.'s
algorithm.
The same kinds of tools apply to general problems of image restoration. For exam-
ple, a frame may contain random dirt, which is very hard to automatically detect. It
was also common to see scratches on frames that came from something inside the
projector periodically or continuously contacting the film. It may show up as a long,
unbroken vertical scratch, or intermittently recur in some parts of some frames. For
historical films — for example I helped on the restoration of
To Kill a Mockingbird
—
you don't want to touch a pixel unless it's absolutely essential. For very thin scratches
you can use something simple like amedian filter, but the artifacts you introduce can
be very subtle. If you freeze a frame and look at the one before and the one after you
won't notice a problem, but if youwatch themoving footage, you'll be very suspicious
about that area—somethingmust have happened there but you can't put your finger
on it — some subtle motion that evokes critical analysis in the viewer.
For the recent movie
Hereafter
, there was a problemwith a “hair in the gate” in one
scene — a hair got stuck in the film scanner and was there throughout the sequence.
We used patch-based inpainting, motion compensated to pull from other frames, to
remove the hair. In another scene outside the San Francisco airport, a dark cloudy
scene with a taxicab, there was actually a fairly large scratch on the film—about forty
pixels at its widest point. The artist created a matte painting for one frame and the
software was able to track that scratch throughout the whole sequence and inpaint it
from the artist's background.
RJR: How is the aspect ratio change between a widescreen cinema release and a DVD
or Blu-Ray handled?
Sloan: You might hope that there's an automatic solution, but there are several rea-
sons why it's problematic. At any step in production before the final shot is done,
you run the risk of things running into each other in a way that you don't expect.
So you can't do content-aware retargeting before all the content is in the scene. The
more important issue is that a well-composed frame is like a visual art form. Yes,
an automatic algorithm could do a nice job of preserving the content and making
sure everyone's heads were in frame when it got cropped down, but every frame
is kind of like a painting. It has a composition and it has spatial relationships that
are probably best preserved by a human, making the decision during actual film-
ing about what part's going to be in the frame and what part's going to be out.
Skilled cinematographers do that all the time; there's this idea of an importance gra-
dient that kind of flows out from a central rectangle that's literally marked in their
