Information Technology Reference
In-Depth Information
⎣ ⎦
∑∑
−
W
/
2
⎣ ⎦
W
/
2
1
R
(
x
,
y
)
=
⋅
R
(
x
+
i
,
y
+
j
)
(6)
k
k
n
i
=
⎣ ⎦
W
/
2
j
=
−
⎣ ⎦
W
/
2
where
R
(
x
,
y
)
k
is the interpolated pixel value at the hole position (
x
,
y
) of the
k
th
color segment in the initial ROI depth image
R
using the valid neighboring pixel
value
R
(
x
+
i
,
y
+
j
)
k
in the
k
th
color segment. The term
n
is the valid number of pix-
els within a
W
×
W
window. Since the hole-filling algorithm is performed in each
color segment, the valid depth pixels in its neighboring segments will not affect
the holes in the color segment. Figure 5(c) shows an ROI depth image.
Finally, we obtain the background depth image applying a stereo matching algo-
rithm, such a belief propagation method [18]. Then, we combine the background
depth image with the ROI depth image to generate an ROI enhanced depth image.
Figure 5(d) shows the ROI enhanced depth image. The pair consisting of the left
image and its ROI enhanced depth image becomes a frame of video-plus-depth.
3 Hierarchical Decomposition of Depth Images
For rendering a 3D scene with video-plus-depth data, we employ depth image-
based rendering using meshes [19]. In this chapter, we introduce the hierarchical
decomposition of depth images to represent a dynamic 3D scene represented by
video-plus-depth. In the hierarchical decomposition, we decompose a depth image
into three layers: regular mesh, boundary, and feature point layers. The main bene-
fit of hierarchical decomposition is to maintain geometric regularity by using 3D
shape patterns induced by these three layers so that we can reconstruct a 3D sur-
face rapidly.
First, we extract edge information by applying the Sobel filter to a depth image
vertically and horizontally. The reason using a depth image instead of its color im-
age for edge extraction is that it is not disturbed by lights or surroundings. There-
after, we divide the region of the depth image uniformly into pixel blocks or grid
cells. According to the existence of edge information in a grid cell, we divide the
depth image [4] into regions of edges and regions without edges, as shown in
Fig. 6. The region of edges is the set of grid cells that includes edge information,
referred to as edge-grid cells; similarly, the region without edges is the set of grid
cells excluding edge information, referred to as no-edge-grid cells.
We define the size of a grid cell as 2
m
×2
n
resolution, such as 16×16, 8×8, or
16×8. Once we choose the size of a grid cell, we should maintain it for each depth
image during the hierarchical decomposition. In addition, we should be careful to
select the size of a grid cell, because it is inversely proportional to the amount of
distortion of generated 3D scenes. We usually set the size of a grid cell as 4×4 or
8×8.
A regular mesh layer is obtained by downsampling the depth image. When the
size of a grid cell is
p
×
q
, the regular mesh layer is generated by downsampling its
depth image with the horizontal sampling rate
p
and the vertical sampling rate
q
.
In other words, we gather the four depth pixels at the corner of each grid cell to
