Introduction - Computer Vision for Visual Effects - page 2

Like all forms of creative expression, visual effects have both an artistic side and
a technological side. On the artistic side are visual effects artists: extremely tal-
ented (and often underappreciated) professionals who expertly manipulate software
packages to create scenes that support a director's vision. They're attuned to the film-
making aspects of a shot such as its composition, lighting, and mood. In the middle
are the creators of the software packages: artistically minded engineers at companies
like The Foundry, Autodesk, and Adobe who create tools like Nuke , Maya , and After
Effects that the artists use every day. On the technological side are researchers, mostly
in academia, who conceive, prototype, and publish new algorithms, some of which
eventually get incorporated into the software packages. Many of these algorithms are
from the field of computer vision, the main subject of this topic.
Computer vision broadly involves the research and development of algorithms
for automatically understanding images. For example, we may want to design an
algorithm to automatically outline people in a photograph, a job that's easy for a
human but that can be very difficult for a computer. In the past forty years, computer
vision has made great advances. Today, consumer digital cameras can automatically
identifywhether all the people in an image are facing forward and smiling, and smart-
phone camera apps can read bar codes, translate images of street signs and menus,
and identify tourist landmarks. Computer vision also plays a major role in image
analysis problems in medical, surveillance, and defense applications. However, the
application in which the average personmost frequently comes into contact with the
results of computer vision — whether he or she knows it or not — is the generation
of visual effects in film and television production.
To understand the types of computer vision problems that are “under the hood”
of the software packages that visual effects artists commonly use, let's consider a
scene of a human actor fighting a computer-generated creature (for example, Rick
O'Connell vs. Imhotep, Jack Sparrow vs. Davy Jones, or Kate Austen vs. The Smoke
Monster). First, the hero actor is filmed on a partially built set interacting with a
stunt performer who plays the role of the enemy. The built set must be digitally
extended to a larger environment, with props and furniture added and removed after
the fact. The computer-generated enemy's actions may be created with the help of
themotion-captured performance of a second stunt performer in a separate location.
Next, the on-set stunt performer is removed fromthe scene and replacedby the digital
character. This process requires several steps: the background pixels behind the stunt
performer need to be recreated, the camera's motion needs to be estimated so that
the digital character appears in the right place, and parts of the real actor's body need
to appropriately pass in front of and behind the digital character as they fight. Finally,
the fight sequence may be artificially slowed down or sped up for dramatic effect. All
of the elements in the final shot must seamlessly blend so they appear to “live” in the
same frame, without any noticeable visual artifacts. This topic describes many of the
algorithms critical for each of these steps and the principles behind them.

1.1 COMPUTER VISION FOR VISUAL EFFECTS

This topic, Computer Vision for Visual Effects , explores the technological side of visual
effects, and has several goals: