Information Technology Reference
In-Depth Information
as shown in Fig. 1(b). Frame averaging will to a certain extent smooth motion as
compared to frame repetition, but the blending is an undesired artifact.
Better results are obtained when one computes the 2D optic flow between known
frames and then compensate for the motion in the new frame: Only when know-
ing the flow one can truly create the data in moving regions of the frame. This is
motion compensated frame rate conversion, which we denote temporal super reso-
lution, TSR. Variational frameworks offer methods for computing both dense and
accurate flow fields and high quality motion compensated intensity interpolation to
get optimal TSR results as seen in Fig. 1(c).
1.1
Frame Rate Requirements
We will focus on the frame rate requirements of humans viewers as our TSR algo-
rithm is aimed at application in video processors in broadcast or home entertainment
systems where pleasing human viewers is the final goal. The properties of the hu-
man visual system guides what minimum frame rates should be used to keep the
viewing experience pleasing. The two main requirements are:
•
The
phi-effect
should be obtained to create apparent living pictures with natural
motion portrayal.
•
Flickering should be avoided when displaying image sequences.
The phi-effect is the effect of recording and showing a set of still images so fast
after each other that any motion in the depicted scene appears real and natural as the
HVS will interpolate the simplest (linear) motion between the frames [1]. To create
the phi-effect, the frame rate has to high enough for the HVS to
perceive
all motion
as natural in spite of this interpolation.
Flickering occurs on when an image is not updated often enough, that is the
update frequency (frame refresh rate) is so low that the eye senses flicker.
Determining the exact minimum required frame rate of image sequences is a
difficult, multi-parameter problem (in HVS properties and viewing conditions) but
the consensus is that more than 50 fps is needed to fulfill the requirements above.
However, it ultimately depends on the tracking done in the eye of the viewer. The
rise of 100 Hz TV in Europe and no 120Hz in the US and Asia indicates that 60
Hz might suffice for TVs. Flicker can be avoided by frame repetition, but to get the
phi-effect—perceived smooth and natural motion—motion compensated frame rate
conversion is necessary.
1.2
Blur Acceptance in Human Vision
Blur is a very common artifact in image sequences, but depth of focus blur and
motion blur is accepted by viewers and in de-noising of images blur is often the
side effect, but is preferred over local, high contrast artifacts like noise, block effect
(JPEG and MPEG material) as edges (high contrast) is the key input to the HVS.
Doing TSR in a wrong or incomplete way will most likely create artifacts in the
