Information Technology Reference
In-Depth Information
(a)
(b)
(c)
Fig. 1 a
True frame from a sequence where the square moves diagonally,
b
bad TSR by
frame interpolation, and
c
good motion compensated TSR (variational in this case).
In terms of number of frames per second (fps) created from a given input, TSR is not
just an upscaling, but also a downscaling. This means we will have to create entirely
new frames even if we downscale in time, and thus we use the term TSR also for
downscaling (although the term 'super resolution' implies a higher resolution). Both
spatial super resolution and TSR are ill-posed
sampling
problems.
Upscaling in time is widely needed as most displays (projectors, plasma, LCDs
and CRTs) have higher frame refresh rates than the frame rate used when recording
the displayed material. All cinematographic movies are for instance shot at 24 fps,
while practically all displays today have refresh rates of at least 50 Hz. The higher
display frequencies are necessary to stop the perceived image sequence from flick-
ering. In the human visual system (HVS), any part of a scene projected onto the
retina away from the fovea (center of a focus) is subject to flicker as these regions
of the retina are highly sensitive to flickering.
To increase the frame rate, many flat panel TV-sets and most PC displays do not
use real TSR but just repeat the same frame once or twice, and in cinemas every
frame is shown again two or three times to avoid flicker. At small viewing angles
(the part of the field of view covered by the screen) frame repetition works fine most
of the time, but at larger viewing angles motion will start to appear jerky. The
phi-
effect
—the effect of perceiving a sequences of still images as motion pictures—is
halted [1]. The problem is typically seen around high contrast edges in motion as
edges are the major information perceived and processed in lower level vision. The
archetype example of jerky motion is a horizontal camera pan, e.g. in well lit interior
scenes or exterior shots of houses and cities.
There are three different methods for frame rate conversion: a) frame repetition,
b) frame averaging and c) motion compensated interpolation. Frame repetition is
the simplest and does not create any artifacts from bad temporal interpolation, but
the motion portrayal stays unnatural. When conversion ratio it not integer, e.g. 24
to 60 fps, frames will be repeated a different number of time adding some nonlinear
jumps to the motion, possibly increasing the unnaturalness of the motion. Frame
averaging, where the two nearest known frames are weighed by the inverse of their
distance to the new frame, yields double exposure-like images in case of motion
