Three-Dimensional Video Contents Exploitation in Depth Camera-Based Hybrid Camera System - High-Quality Visual Experience - page 360

Video Encoder I Left Image Seq.

ROI
Enhanced
Depth Image
Generation

Relative
Camera
Calibration

3D
Image
Warping

MPEG-4
Framework
(MUX)

Video Encoder II Depth Image Seq.

Stereo camera

Computer Graphic
Models

BIFS Encoder

Depth Data
Enhancing

Color Image Seq.

Scene Description
for Composition

Depth camera

Depth Image Seq.

Channel

Hierarchical
Decomposition

Video Decoder I Left Image Seq.

MPEG-4
Synchronizer

MPEG-4
Framework
(DEMUX)

Depth Image Seq.

Video Decoder II

Graphic
Render

Computer
Graphic Models

BIFS
Decoder

3D video contents

Scene Description
for Composition

Fig. 8 3D Video Contents Generation

for depth image sequence. Other multimedia data and the scene description infor-
mation are compressed by their coders, respectively. These compressed bitstreams
are multiplexed into one bitstream in the MPEG-4 system.
At the client side, we extract video-plus-depth, scene description information,
and other multimedia data from the transmitted bitstream by their decoders.
Thereafter, we construct 3D scenes from video-plus-depth using the hierarchical
decomposition method as introduced in Section 3. Other multimedia data are
combined with the dynamic 3D scene by referring the scene description informa-
tion. Finally, we can experience various interactions with the immersive content.
A major difference in MPEG-4, with respect to previous audio-visual stan-
dards, is the object-based audio-visual representation model. In the MPEG-4 mul-
timedia framework, an object-based scene is built using individual objects that
have relationships in space and time. Based on this relationship, the MPEG-4 sys-
tem allows us to combine a variety of media objects into a scene. The MPEG-4
BIFS defines how the objects are spatio-temporally combined for presentation. All
visible objects in the 3D scene are described within the Shape node in MPEG-4
BIFS. The Shape node should have both appearance and geometry information;
the appearance is expressed by the color image sequence through a MovieTexture
node.
However, although the geometry should be expressed by the depth image se-
quence, MPEG-4 BIFS does not support a node related to this geometry. There-
fore, we design a new node representing the depth image sequence, referred to as a
DepthMovie node. A new DepthMovie node that can be stored in the geometry
field is designed as follows.