Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 4, pp. 1–11
Abstract: Frame interpolation is the process of synthesizing a new frame in between the existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects since its large scale use in the making of the movie “The Matrix.” This paper presents a review and a new unified view of the classical algorithms used to create in-between frames, representing most of the past 20 years of their evolution. This is used to benchmark the recent deep learning algorithms against two of the best industrial retimers available. A significantly expanded data set of 140,000 frames is used for testing. In the context of highresolution material, we find that techniques relying principally on deep neural networks (DNNs) do not clearly outperform the classical ideas. It is only with the emergence of hybrid approaches in 2019 that we see DNNs adding significantly to the performance in this space. Despite the hype surrounding DNNs, we find that there is still more work to be done.
Publication Date: 2021-05-01
DOI: 10.5594/JMI.2021.3067960
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2021.3067960
Author(s): Anil Kokaram
bio
Anil Kokaram graduated from the Engineering Department, Cambridge University, Cambridge, U.K., in 1993, with a PhD in signal processing. He is now a professor at Trinity College Dublin, Ireland. He was the lead of the Media Algorithms Team at YouTube/Google from 2011-2017. Before that, he founded a media dsp group at Trinity (www.sigmedia.tv,) working in the broad areas of DSP for video processing, Bayesian inference, and motion estimation. He has published more than 200 peer-reviewed papers in these areas. He is a former associate editor of the IEEE Transactions on Video Technology and IEEE Transactions on Image Processing. He founded the company GreenParrotPictures, which produced video enhancement software and was acquired by Google in 2011. In 2007, he was honored with a Science and Engineering Academy Award from the American Academy of Motion Picture Arts and Sciences for his work in video processing for post-production applications.
Davinder Singh
bio
Davinder Singh graduated as a computer engineer from the School of Engineering, Trinity College Dublin, Ireland, in 2019. After a year-long stint as an intern with the Sigmedia group, he is now a software developer working in Dublin.
Simon Robinson
bio
Simon Robinson is the co-founder of Foundry, London, U.K. (www.thefoundry.co.uk). He began his career at IBM Research in New York after receiving a BA (Hons.) in engineering and computing from Cambridge University, U.K. He subsequently worked in technology development at companies including Digital Pictures, Rushes, Framestore, and MPC, before launching Foundry in 1996. He continues to play a key role in the company’s direction and development. He received a Technical Oscar from the Academy of Motion Picture Arts and Sciences in 2007.
Keyword(s): Frame Interpolation, Compression, Optic Flow, Retiming
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

Yu-lun Liu, et al., “Deep Video Frame Interpolation Using Cyclic Frame Generation,” Proc. AAAI Conf. Artif. Intell. (AAAI 2019), pp. 8794–8802, Jan. 2019. EXTERNAL
Bai, W., et al., “Visual-Weighted Motion Compensation Frame Interpolation With Motion Vector Refinement,” Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), pp. 500–503, May 2012. EXTERNAL
Y. Zhang et al., “A Spatio-Temporal Autoregressive Frame Rate Up Conversion Scheme,” IEEE Int. Conf. Image Process., San Antonio, TX, USA, vol. I, pp. 441–444, 2007. EXTERNAL
S. C. Dabner, “Real Time Motion Measurement Hardware: Phase Correlation Unit,” BBC Research and Development Report EXTERNAL
D.M. Martinez, “Model Based Motion Estimation and Its Application to Restoration and Interpolation of Motion Pictures,” Technical Report MIT Dept. Electronic Eng. Comput. Sci. Tr-530, pp. 1–167, Jun. 1987. EXTERNAL
K.-W. Hung and W.-C. Siu, “Fast Video Interpolation/ Upsampling Using Linear Motion Model,” Proc. 18th IEEE Int. Conf. Image Process. (ICIP), p. 1341, 2011. EXTERNAL
K. Lee and J. Jeong, “Bilateral Frame Rate Up-Conversion Algorithm Based on the Comparison of Texture Complexity,” Electron. Lett., 52(5): 354–355, 2016. EXTERNAL
H. Jiang, et al., “Super Slo-Mo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation,” IEEE Conf. Comput. Vis. Pattern Recognit., pp. 9000–9008, Jun. 2018. EXTERNAL
S. Niklaus, and F. Liu, “Context-Aware Synthesis for Video Frame Interpolation,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 1701–1710, 2018. EXTERNAL
Bao, W., et al., “MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement,” IEEE Trans. Pattern Anal. Mach. Intell., to appear, pp. 1–1, Dec. 2019. EXTERNAL
S. Kang, S. Yoo, and Y. H. Kim, “Dual Motion Estimation for Frame Rate Up-Conversion,” IEEE Trans. Circuits Syst. Video Technol., 20(12): 1909–1914, Dec. 2010. EXTERNAL
S. Meyer, et al., “Phase-Based Frame Interpolation for Video,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Boston, MA, pp. 1410–1418, Jun. 2015. EXTERNAL
S. Niklaus, L. Mai, and F. Liu, “Video Frame Interpolation Via Adaptive Convolution,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 2270–2279, Jul. 2017. EXTERNAL
R. Thoma and M. Bierling, “Motion Compensating Interpolation Considering Covered and Uncovered Background,” Signal Process. Image Comm., 1(2): 191–212, Oct. 1989. EXTERNAL
S. H. Yang and C. C. Yang, “Fast Frame Rate Up-Conversion Based on Multiple Frames,” Proc. IEEE Int. Conf. Multimedia Expo, pp. 1–4, Jul. 2011. EXTERNAL
P. Haavisto, J. Juhola, and Y. Neuvo, “Fractional Frame Rate Up-Conversion Using Weighted Median Filters,” IEEE Trans. Consumer Electron. 35(3): 272–278, Aug. 1989. EXTERNAL
R. Tseng, et al., “Adaptive Frame Interpolation Using an End-To-End Deep Net With High Quality Flow Estimation,” 2019 Int. Conf. Technol. Appl. Artif. Intell. (TAAI), pp. 1–4, 2019. EXTERNAL
S. Lee and S. Yang, “Adaptive Motion-Compensated Frame Rate Up-Conversion,” Electron. Lett., 38(10): 451–452, May 2002. EXTERNAL
Z. Liu, et al., “Video Frame Synthesis Using Deep Voxel Flow,” Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 4473– 4481, 2017. EXTERNAL
D. Guo and Z. Lu, “Motion-Compensated Frame Interpolation With Weighted Motion Estimation and Hierarchical Vector Refinement,” Neurocomputing, 181: 76–85, 2016. EXTERNAL
Biswas, M. and Nguyen, T., “A Novel Motion Estimation Algorithm Using Phase Plane Correlation for Frame Rate Conversion,” Proc. 69th Asilomar Conf. Signals Syst. Comput., no. 11, pp. 492–496, Nov. 2002. EXTERNAL
Ahn, H.-E., Jeong, J., and Kim, J. W., “A Fast 4K Video Frame Interpolation Using a Hybrid Task-Based Convolutional Neural Network,” Symmetry, 11(5):1, May 2019. EXTERNAL
G. A. Thomas, “A Comparison of Motion Compensated Interlace to Progressive Conversion Methods,” BBC Research and Development Report BBC RD 1996/9, 1996. EXTERNAL
Wang, Y., Ma, S., and Gao, W., “Frame Rate Up Conversion Via Bayesian Motion Estimation,” Proc. SPIE Visual Comm. Image Process. 2010, vol. 7744, Jul. 2010. EXTERNAL
Couteur, G. M. L., “Field Store Standards Conversion: A Technical Description of the c06/508 Converter,” BBC Research and Development Report BBC RD 1972/22, 1972. EXTERNAL
C. Lu, M. Hirsch, and B. Schölkopf, “Flexible Spatio-Temporal Networks for Video Prediction,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), no. 7, pp. 2137–2145, Jul. 2017. EXTERNAL
S. Niklaus and F. Liu, “Softmax Splatting for Video Frame Interpolation,” ArXiv (CVPR 2020), arXiv:2003.05534, 2020. EXTERNAL
Y. Zhang, et al., “Weighted Convolutional Motion-Compensated Frame Rate Up-Conversion Using Deep Residual Network,” IEEE Trans. Circuits Syst. Video Technol., 12: 1–1, Dec. 2018. EXTERNAL
Dao, M., et al. “Video Frame Interpolation Via Weighted Robust Principal Component Analysis,” Proc. IEEE Int. Conf. Acoustics, Speech Signal Process., pp. 1404–1408, May 2013. EXTERNAL
G. de Haan, Video Processing for Multimedia Systems, Eindhoven University Press, Sept. 2000. EXTERNAL
M. Park, et al., “Robust Video Frame Interpolation With Exceptional Motion Map,” IEEE Trans. Circuits Syst. Video Technol., to appear, 3: 1–1, 2020. EXTERNAL
A. Dosovitskiy, et al., “FlowNet: Learning Optical Flow With Convolutional Networks,” IEEE Int. Conf. Comput. Vis. (ICCV), pp. 2758–2766, Dec. 2015. EXTERNAL
Choi, B., et al., “Motion-Compensated Frame Interpolation Using Bilateral Motion Estimation and Adaptive Overlapped Block Motion Compensation,” IEEE Trans. Circuits Syst. Video Technol., 17(4): 407–416, Apr. 2007. EXTERNAL
X. Hoangvan, X., J. ASCENSO, and F. Pereira, “Improved Matching Criterion for Frame Rate Up-Conversion With Trilateral Filtering,” Electron. Lett., 49(2): 106–107, Jan. 2013. EXTERNAL
A. Kappeler, et al., “Video Super-Resolution With Convolutional Neural Networks,” IEEE Trans. Comput. Imaging, 2(2): 109–122, Jun. 2016. EXTERNAL
A. Ranjan and M. J. Black, “Optical Flow Estimation Using a Spatial Pyramid Network,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 2720–2729, Jul. 2017. EXTERNAL
E. Ilg, et al., “FlowNet 2.0: Evolution of Optical Flow Estimation With Deep Networks,” IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1647–1655, Jul. 2017. EXTERNAL
A. N. Netravali and J. D. Robbins, “Motion-Compensated Television Coding: Part I,” Bell Syst. Techn. J., 58: 631–670, Mar. 1979. EXTERNAL
X. Hoang Van, “Statistical Search Range Adaptation Solution for Effective Frame Rate Up-Conversion,” IET Image Process., 12(1): 113–120, Jan. 2018. EXTERNAL
H. Liu, et al., “Multiple Hypotheses Bayesian Frame Rate Up-Conversion by Adaptive Fusion of Motion-Compensated Interpolations,” IEEE Trans. Circuits Syst. Video Technol., 22(8): 1188–1198, Aug. 2012. EXTERNAL
O. A. Ojo and G. De Haan, “Robust Motion-Compensated Video Upconversion,” IEEE Trans. Consumer Electron., 43(4): 1045–1056, Nov. 1997. EXTERNAL
Bao, W., et al., “Depth-Aware Video Frame Interpolation,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 3698–3707, 2019. EXTERNAL
Kokaram, A., Motion Picture Restoration: Digital Algorithms for Artefact Suppression in Archived Images and Video, Springer Verlag: Berlin, Germany, Apr. 1998. EXTERNAL
C. Wang, et al., “Frame Rate Up-Conversion Using Trilateral Filtering,” IEEE Trans. Circuits Syst. Video Technol., 20(6): 886–893, Jun. 2010. EXTERNAL
R. Castagno, P. Haavisto, and G. Ramponi,“A Method for Motion Adaptive Frame Rate Up-Conversion,” IEEE Trans. Circuits Syst. Video Technol., 6:436–436, Oct. 1996. EXTERNAL
S. Niklaus, L. Mai, and F. Liu, “Video Frame Interpolation Via Adaptive Separable Convolution,” Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 261–270, Oct. 2017. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2021.3067960.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Anil Kokaram, Davinder Singh, and Simon Robinson; Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 4, May 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3067960

Snippet:

Anil Kokaram, Davinder Singh, and Simon Robinson; Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 4, May 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3067960

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Anil Kokaram, Davinder Singh, and Simon Robinson; <cite>Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 4, May 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2021.3067960" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2021.3067960</a></span>

SMPTE's HTML Pub

Preview:

Anil Kokaram, Davinder Singh, and Simon Robinson; Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 4, May 2021); SMPTE, 2021
doi: 10.5594/JMI.2021.3067960
url: https://doi.org/10.5594/JMI.2021.3067960

Snippet:

<li>
Anil Kokaram, Davinder Singh, and Simon Robinson; <cite id="bib-10-5594-jmi-2021-3067960">Moving Image Frame Interpolation: Neural Networks and Classical Toolsets Compared</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 4, May 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2021.3067960</span>
</li>