Information Technology Reference
In-Depth Information
Variational TSR needs to be benchmarked against other motion compensated TSR
algorithms to show its full potential.
3.4
Discussion: Improving Variational Optic Flow
for Motion Compensated Methods
Scenes with complex motion can cause problems even to advanced optic flow algo-
rithms. It is important to have a robust motion compensated algorithm that switches
off temporal input when flows are unreliable, but the limitations of the human vi-
sual system will also help in optimizing the algorithm: In scenes with complex
motion the HVS will not be able to track all the motions and thus we might get
away with producing suboptimal outputs. Still, optimal results require precise and
reliable flows, but the modeling of e.g. accelerations, whirls and transparent mo-
tions is still complex. Variational methods are getting better and better at this,
while e.g. block matching motion estimation will fail by its basic motion modeling
assumptions.
The problems we see with changes motion magnitude and directions (accelera-
tion) e.g. in
Control Panel
is most likely due to the fact that we use a 3D local
spatiotemporal prior on the flow,
E
3
in (6), reported to give better flow results than a
purely spatial 2D prior on sequences with slow temporal changes (e.g.
Yosemite
)
in [15, 34]. In [24] we showed that the problem might be solved by processing the
sequence in very short bites (2-4 frames) but a more robust solution would be to
consider spatiotemporal sequences as 2D+1D instead of the unnatural 3D, separat-
ing space and time, but still linking them together (naturally) along the optic flow
field. Alternatively, a flow acceleration prior could be added to the variational for-
mulation of our problem.
4Con lu ion
In this chapter we have discussed the requirements put on the design of temporal
super resolution algorithms by the human visual system, and have presented a novel
idea of simultaneous flow and intensity calculation in new frames of an image se-
quence. A novel variational temporal super resolution method has been introduced,
and it has been implemented and tested for the subproblem of frame rate doubling.
Our results showed that the use of the gradient constancy assumption gives no major
improvements on the image sequence output, but as indicated in our discussion, it
might do so as variational optic flow modeling improves.
Even though the new variational TSR algorithms do not always create perfect
new frames, they do provide high quality 50 fps video from 25 fps video without
noticeable artifacts during video playback, thus reestablishing the phi-effect in the
troublesome case of high contrast edges in motion. The framework presented also
has the potential to be used for other frame rate conversions than frame rate dou-
bling, the problems of implementation being mainly of practical character.
