Graphics Reference
In-Depth Information
where
a
,
b
, and
c
can be estimatedwith robust methods [
486
]. In this case, we can add
a
segmentation-based
regularization term to the data term of a stereo cost function,
such as
E
segment
(
L
(
i
)
−
(
a
i
x
(
i
)
+
b
i
y
(
i
)
+
c
i
))
(5.56)
i
∈
V
where
a
i
,
b
i
, and
c
i
are the estimated plane parameters for the segment containing
pixel
i
, and
E
segment
(
couldbe based onone of the robust cost functions inTables
5.1
and
5.2
. Of course, now we must perform an extra step of segmenting the image
into roughly constant-intensity pieces, which is commonly solved using the mean-
shift algorithm [
102
]. If necessary, initial estimates of the disparity map within each
segment can be obtained by an algorithm like Lucas-Kanade. High-performing stereo
algorithms that use a segmentationapproach include Sunet al. [
480
], Klaus et al. [
242
],
Wang and Zheng [
537
], and Yang et al. [
563
]. Bleyer et al. [
50
] extended the approach
to incorporate a term based on minimum description length to penalize the number
of segments and to allow higher-order disparity surfaces such as B-splines.
Yang et al. [
563
] also noted that quadratic interpolation could be used to enhance
the quantized disparity estimates from a stereo algorithm, recovering a sub-pixel
disparity image. This step would likely be critical for obtaining good results for the
applications of dense correspondence we discuss in the next three sections.
x
)
5.6
VIDEO MATCHING
We can extend the two-frame dense correspondence problem in several ways. One
possibility is to consider simultaneous correspondences betweenadditional synchro-
nized cameras at different locations in the scene; this problem is called
multi-view
stereo
andwill be discussed indetail inSection
8.3
. Another possibility is to extend the
dense correspondence problem to video sequences, generalizing optical flow in the
case of a single camera and stereo in the case of a rigidly mounted pair of cameras.
These cases can generally be handled by adding a temporal regularization term to the
cost function that encourages the flow values
or disparity labels
L
to be similar
to those of the previous frame. For example, for stereo video this termmight look like
(
u
,
v
)
L
t
L
t
−
1
E
temporal
(
(
i
)
−
(
i
))
(5.57)
i
∈
V
where the superscript
t
indicates the time index in the video. Alternately (or in
addition), the flow field/disparity map from the previous frame can be used as an
initial estimate for the current frame. Sawhney et al. [
421
] described an algorithm for
high-resolution dense correspondence for stereo video in this vein.
A third situation arises when we consider a pair of video cameras that move
through a scene at different times and different velocities. This is related to the visual
effects problem of
motion control
— that is, the synchronization of multiple cam-
era passes over the same scene. Motion control for an effects-quality shot typically
requires a computer-controlled rig that moves through an environment along a pre-
programmed path with extremely high precision. In this way, an environment can be
set up multiple times so that different elements can be independently filmed (e.g.,
