Graphics Reference
In-Depth Information
high- and low-probability correspondence; low-probability regions are caused by
areas that have changed between the two video sequences.
Finally, we can compute a matching score for the candidate frame pair
I
1
(
t
)
,
I
2
(
t
+
δ)
based on the estimated dense correspondence field
(
u
,
v
)
. This score is computed
as a weighted average as follows:
2
−
2
2
x
i
y
i
x
j
y
j
x
i
+
x
j
+
u
(
x
i
,
y
i
)
u
(
x
j
,
y
j
)
λ
−
−
y
i
+
v
(
x
i
,
y
i
)
y
j
+
v
(
x
j
,
y
j
)
(
x
i
,
y
i
)
∈
I
1
(
t
)
(
x
j
,
y
j
)
∈
I
1
(
t
)
2
2
u
(
x
,
y
)
+
(5.59)
v
(
x
,
y
)
(
x
,
y
)
∈
I
1
(
t
)
The first term in Equation (
5.59
) is based on the
parallax
between a pair of
matches — that is, the difference between their distance in the first image and their
distance in the second image. The parallax is invariant to image rotation and transla-
tion. Here, we compute the average parallax over all correspondences introduced by
the optical flow field, as a measure of the introduced image distortion. The second
term in Equation (
5.59
) is the average optical flow vector magnitude, which is small
when the overlap between the two images is large. Sand and Teller used
λ
=
5to
emphasize the importance of parallax.
Nowwe can use the pairwisematching cost between pairs of frames in the first and
second video sequences to build a set of frame-to-frame correspondences. The user
initializes the process by selecting the first pair of corresponding frames
(
I
1
(
t
0
)
,
I
2
(
t
0
+
δ
. Then we iteratively determine the frame-to-frame correspondences with the
following procedure:
))
0
1.
Set
k
=
0.
2.
δ
k
+
1
as aweighted average of the five previous
offsets, where the weight decreases as we move back in time.
3. Compute thematching cost between
I
1
Set the initial guess for the offset
(
t
k
+
1
)
(
t
k
+
δ
k
+
1
)
and the set of frames
I
2
,
.
4. Fit a quadratic function to these costs as illustrated in Figure
5.20
. Determine
the minimizer
I
2
(
t
k
+
δ
k
+
1
+
1
)
,
I
2
(
t
k
+
δ
k
+
1
−
1
)
,
I
2
(
t
k
+
δ
k
+
1
+
5
)
, and
I
2
(
t
k
+
δ
k
+
1
−
5
)
δ
∗
of the quadratic function.
δ
∗
=
δ
k
+
1
— that is, the minimizer stays in the same place — set
k
5.
If
=
k
+
1
δ
k
+
1
=
δ
∗
and go to Step 3.
and go to Step 2. Otherwise, set
Once the two videos are spatially and temporally synchronized, we can applymany
of the algorithms from Chapters
2
and
3
. For example, if one sequence contains live
action and the other contains an empty background, we have a strong prior estimate
of the alignment required for video matting and inpainting. We can also film one
person on the left side of a moving shot, and the same person on the right side of a
similar shot, compositing the two videos along a seam to create a “twinning” effect.
Alternately, we can replace a stand-in from a live-action plate with a computer-
generated character composited over a clean plate.
