Graphics Reference
In-Depth Information
Dense Correspondence and Its
Applications
5
In the last chapter we focused on detecting and matching distinctive
features
.
Typically, features are sparsely distributed — that is, not every pixel location has
a feature centered at it. However, for several visual effects applications, we require
a
dense correspondence
between pixels in two images, even in relatively flat or fea-
tureless areas. One of the most common applications of dense correspondence in
filmmaking is for slowing down or speeding up a shot after it's been filmed for dra-
matic effect. To create the appropriate intermediate frames, we need to estimate the
trajectory of every pixel in the video sequence over the course of a shot, not just a few
pixels near features.
More mathematically, we want to compute a vector field
(
u
(
x
,
y
)
,
v
(
x
,
y
))
over
the pixels of the first image
I
1
, so that the vector at each pixel
(
x
,
y
)
points to
a corresponding location in the second image
I
2
. That is, the pixels
I
1
(
x
,
y
)
and
I
2
(
x
+
u
(
x
,
y
)
,
y
+
v
(
x
,
y
))
correspond. We usually abbreviate the vector field as
(
u
,
v
)
with the understanding that both elements are functions of
x
and
y
.
Defining what constitutes a correspondence in this context can be tricky. As in
feature matching, our intuition is that a correspondence implies that both pixels
arise from the same point on the surface of some object in the physical world. The
vector
is induced by the motion of the camera and/or the object in the interval
between taking the two pictures. Thus, it is often called a
motion vector
. Unfortu-
nately, changes in pixel intensity can occur in the absence of camera/object motion
and vice versa. For example, if we move a spotlight around a static object between
taking two pictures with a stationary camera, the pixel intensities are likely to be
very different even though the object has not moved. On the other hand, two pic-
tures of an ideal, matte-white rotating sphere under identical lighting conditions
will look the same at different times, even though the points on the surface of the
sphere havemoved. In practice, we try to avoid these worst-case scenarios by assum-
ing sufficiently textured scenes in which the interval between taking the pictures is
small enough to avoid major lighting changes. This means that changes in appar-
ent pixel brightness are for the most part caused by camera and object motion.
The dense correspondence field we seek in this chapter is thus generally assumed
to arise from underlying physical point correspondence, just like in the previous
chapter.
Obtaining dense correspondence between images is one of the oldest and most
well-studied problems in computer vision, with literally thousands of papers on the
topic in several different forms. The easiest case is when the two images are assumed
(
u
,
v
)
148
