Graphics Reference
In-Depth Information
Before that we used optical flow for a movie called
What Dreams May Come
.We
extracted optical flow vectors and used them to “modulate” several scenes. In one
scene, the main character slides down a hill into a lake, and the optical flow motion
vectors from the clean plate were used to re-render the scene as if he was smearing
the surface of a brightly colored oil painting. A related application of optical flow
was in a kids' movie called
Clockstoppers
. We estimated the motion vectors from the
scene, and used them to create and enhance the motion blur on CGI objects like cars
inserted into the scene.
In another movie called
Practical Magic
we used it in a shot where a man's ghost
comes up out of the ground. The director wanted mud stuck to him, but when they
shot it there was no actual mud. We painted the mud in one keyframe and used the
optical flow vectors to track themud onto him in the rest of the frames. At some point
the ghost vanishes into smoke; we used optical flow there as well. We recorded the
lighting of a match, reversed the video, and estimated the motion vectors as the full-
grown flame shrank to a single spark. We applied those motion vectors to the pixels
on the ghost character to create the dissolving effect. We used a similar approach
to create the “bamfing” effect of the Nightcrawler character in
X2: X-Men United
,by
shooting smoke against a black background, computing the optical flow vectors, and
applying them to the character.
RJR: Which optical flow algorithms did you find useful?
Chatterjee: Initially I was working on an optical flowalgorithmmyself, amodification
of Black and Anandan's classic approach. Later I tried Brox et al.'s algorithm and
the results were very good, but at the time it was much too slow to handle the 2K-
resolution images in digital cinema. That got me interested in looking at Bruhn et al.'s
algorithm which was much faster and also gave high-quality results. I made several
modifications to the algorithm and implemented it on the Nvidia GPU, where it's
near real time. This modified algorithm was recently used to restore the films from
the Apollo 11 moon landing long thought lost. You can see a little bit of diffusion
effects from optical flow around the edges since they're very high contrast images;
the astronaut is in a bright white suit and space is deep black.
Ryan: We were inspired by Black and Anandan's earlier optical flow work, but
more recently we've also implemented and had good success with Bruhn et al.'s
optical flow algorithms. Adding the gradient consistency assumption really made it
successful.
RJR: How does academic research on optical flow differ from the requirements of
feature films?
Ryan: There are many reasons why movie frames violate the usual assumptions of
optical flow. The biggest difference is the size of the frames. Academic optical flow
sequences tend to be a few hundred pixels wide, but we need to track HD plates that
are at least 1080p — it's a whole different world. There's often motion blur, which
affects heavy movement. Directors love anamorphic lens flare. With a lot of effects-
heavy shows, it's very common that action sequences are shot at night, like the big
