API Build-data JSON Resources
Theme

Choose how MSRBot.io looks on this device.

Auto (match system) Light mode Dark mode

Preference is stored in this browser only.

SMPTE Meetings and Conferences ( November 2020)
[ACTIVE]

Twenty years of Frame Interpolation for Retiming in the Movies

Metadata

Publisher
SMPTE — White Plains, NY, USA
Doc Type
Conference Paper
Content Type
Original Research
Volume
2020, No. 11, pp. 1–20
Abstract
Frame interpolation is the process of synthesising a new frame in-between existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects since its use at a large scale in the making of the movie “The Matrix”. This paper presents a review and a new unified view of the classical algorithms used to create inbetween frames, representing most of the last 20 years of their evolution. This is used to benchmark the recent Deep Learning algorithms against two of the best industrial retimers available. A significantly expanded dataset of 140,000 frames is used for testing. In the context of high resolution material we find that techniques relying principally on DNNs do not clearly outperform the classical ideas. It is only with the emergence of hybrid approaches since 2019 that we see DNNs adding significantly to the performance in this space. Despite the hype surrounding DNNs, we find that there is still something left to do.
Publication Date
2020-11-10
DOI
10.5594/M001927
Link
https://doi.org/10.5594/M001927
Author(s)
Anil KokaramSigmedia Group, Electronic Engineering, Trinity College Dublin, Ireland
Davinder SinghSigmedia Group, Electronic Engineering, Trinity College Dublin, Ireland
Simon RobinsonChief Scientist, Foundry, London, UK
Keyword(s)
Frame Interpolation, Retiming, Compression, Optic Flow
Copyright
© 2020 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

  • HAAVISTO, P., JUHOLA, J., AND NEUVO, Y. Fractional frame rate up-conversion using weighted median filters. IEEE Transactions on Consumer Electronics 35, 3 (Aug 1989), 272–278. EXTERNAL
  • THOMAS, G. A. A comparison of motion compensated interlace to progressive conversion methods. BBC Research and Development Report BBC RD 1996/9 (1996). EXTERNAL
  • DAO, M., SUO, Y., CHIN, S., AND TRAN, T. Video frame interpolation via weighted robust principal component analysis. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (May 2013), pp. 1404–1408. EXTERNAL
  • DABNER, S. C. Real time motion measurement hardware: Phase correlation unit. BBC Research and Development Report BBC RD 1990/11 (1990). EXTERNAL
  • THOMA, R., AND BIERLING, M. Motion compensating interpolation considering covered and uncovered background. Signal Processing: Image Communication 1, 2 (1989), 191 – 212. 64 kbit/s Coding of Moving Video. EXTERNAL
  • NIKLAUS, S., MAI, L., AND LIU, F. Video frame interpolation via adaptive separable convolution. In 2017 IEEE International Conference on Computer Vision (ICCV) (Oct 2017), pp. 261–270. EXTERNAL
  • KOKARAM, A. Motion Picture Restoration: Digital algorithms for artefact suppression in archived images and video. Springer Verlag ISBN 3-540-76040-7, April 1998. EXTERNAL
  • KAPPELER, A., YOO, S., DAI, Q., AND KATSAGGELOS, A. K. Video super-resolution with convolutional neural networks. IEEE Transactions on Computational Imaging 2, 2 (June 2016), 109–122. EXTERNAL
  • BAO, W., LAI, W., MA, C., ZHANG, X., GAO, Z., AND YANG, M. Depth-aware video frame interpolation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 3698–3707. EXTERNAL
  • HOANGVAN, X., ASCENSO, J., AND PEREIRA, F. Improved matching criterion for frame rate upconversion with trilateral filtering. Electronics Letters 49, 2 (January 2013), 106–107. EXTERNAL
  • AHN, H.-E., JEONG, J., AND KIM, J. W. A fast 4k video frame interpolation using a hybrid taskbased convolutional neural network. Symmetry 11, 5 (2019). EXTERNAL
  • NIKLAUS, S., MAI, L., AND LIU, F. Video frame interpolation via adaptive convolution. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017), pp. 2270–2279. EXTERNAL
  • DOSOVITSKIY, A., FISCHER, P., ILG, E., HÄUSSER, P., HAZIRBAS, C., GOLKOV, V., V. D. SMAGT, P., CREMERS, D., AND BROX, T. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV) (Dec 2015), pp. 2758–2766. EXTERNAL
  • LIU, Z., YEH, R. A., TANG, X., LIU, Y., AND AGARWALA, A. Video frame synthesis using deep voxel flow. In 2017 IEEE International Conference on Computer Vision (ICCV) (2017), pp. 4473– 4481. EXTERNAL
  • WANG, Y., MA, S., AND GAO, W. Frame rate up conversion via bayesian motion estimation. In Proceedings of SPIE Visual Communications and Image Processing 2010 (7 2010), vol. 7744. EXTERNAL
  • ZHANG, Y., CHEN, L., YAN, C., QIN, P., JI, X., AND DAI, Q. Weighted convolutional motioncompensated frame rate up-conversion using deep residual network. IEEE Transactions on Circuits and Systems for Video Technology 12 (December 2018), 1–1. EXTERNAL
  • RANJAN, A., AND BLACK, M. J. Optical flow estimation using a spatial pyramid network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017), pp. 2720–2729. EXTERNAL
  • OJO, O. A., AND DE HAAN, G. Robust motion-compensated video upconversion. IEEE Transactions on Consumer Electronics 43, 4 (Nov 1997), 1045–1056. EXTERNAL
  • NIKLAUS, S., AND LIU, F. Softmax splatting for video frame interpolation. ArXiv (CVPR 2020), arXiv:2003.05534 (2020). EXTERNAL
  • CHOI, B., HAN, J., KIM, C., AND KO, S. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation. IEEE Transactions on Circuits and Systems for Video Technology 17, 4 (April 2007), 407–416. EXTERNAL
  • CASTAGNO, R., HAAVISTO, P., AND RAMPONI, G. A method for motion adaptive frame rate upconversion. IEEE Transactions on circuits and Systems for Video Technology 6 (October 1996), 436–436. EXTERNAL
  • NIKLAUS, S., AND LIU, F. Context-aware synthesis for video frame interpolation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (June 2018), pp. 1701–1710. EXTERNAL
  • GUO, D., AND LU, Z. Motion-compensated frame interpolation with weighted motion estimation and hierarchical vector refinement. Neurocomputing 181 (2016), 76 – 85. EXTERNAL
  • JIANG, H., SUN, D., JAMPANI, V., YANG, M., LEARNED-MILLER, E., AND KAUTZ, J. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE Conference on Computer Vision and Pattern Recognition (June 2018), pp. 9000–9008. EXTERNAL
  • ZHANG, Y., ZHAO, D., JI, X., WANG, R., AND CHEN, X. A spatio-temporal autoregressive frame rate up conversion scheme. In 2007 IEEE International Conference on Image Processing (Sep. 2007), vol. 1, pp. I – 441–I – 444. c© 2020 Society of Motion Picture & Television Engineers R© (SMPTE R© ) c© 20 EXTERNAL
  • BISWAS, M., AND NGUYEN, T. A novel motion estimation algorithm using phase plane correlation for frame rate conversion. In Thirty-Sixth Asilomar Conference on Signals, Systems and Computers (Nov 2002), no. 11, pp. 492–496. EXTERNAL
  • YU-LUN LIU, YI-TUNG LIAO, Y.-Y. L. Y.-Y. C. Deep video frame interpolation using cyclic frame generation. In AAAI Conference on Artificial Intelligence (AAAI 2019) (January 2019), pp. 8794– 8802. EXTERNAL
  • HUNG, K.-W., AND SIU, W.-C. Fast video interpolation/upsampling using linear motion model. In 18th IEEE International Conference on Image Processing (ICIP) (2011), p. 1341. EXTERNAL
  • COUTEUR, G. M. L. Field store standards conversion: A technical description of the c06/508 converter. BBC Research and Development Report BBC RD 1972/22 (1972). EXTERNAL
  • TSENG, R., LIU, Y., CHEN, J., AND LIN, K. W. Adaptive frame interpolation using an end-to-end deep net with high quality flow estimation. In 2019 International Conference on Technologies and Applications of Artificial Intelligence (TAAI) (2019), pp. 1–4. EXTERNAL
  • SHIH-HSUAN YANG, AND CHIA-CHI YANG. Fast frame rate up-conversion based on multiple frames. In 2011 IEEE International Conference on Multimedia and Expo (July 2011), pp. 1–4. c© 2020 Society of Motion Picture & Television Engineers R© (SMPTE R© ) c© 19 EXTERNAL
  • HOANG VAN, X. Statistical search range adaptation solution for effective frame rate up-conversion. IET Image Processing 12, 1 (Jan 2018), 113–120. EXTERNAL
  • BAO, W., LAI, W., ZHANG, X., GAO, Z., AND YANG, M. MEMC-Net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Transactions on Pattern Analysis and Machine Intelligence To appear (December 2019), 1–1. EXTERNAL
  • PARK, M., KIM, H. G., LEE, S., AND RO, Y. M. Robust video frame interpolation with exceptional motion map. IEEE Transactions on Circuits and Systems for Video Technology To appear, 3 (2020), 1–1. EXTERNAL
  • ILG, E., MAYER, N., SAIKIA, T., KEUPER, M., DOSOVITSKIY, A., AND BROX, T. Flownet 2.0: Evolution of optical flow estimation with deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017), pp. 1647–1655. EXTERNAL
  • KANG, S., YOO, S., AND KIM, Y. H. Dual motion estimation for frame rate up-conversion. IEEE Transactions on Circuits and Systems for Video Technology 20, 12 (Dec 2010), 1909–1914. EXTERNAL
  • MEYER, S., WANG, O., ZIMMER, H., GROSSE, M., AND SORKINE-HORNUNG, A. Phase-based frame interpolation for video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (Boston, June 2015), pp. 1410–1418. EXTERNAL
  • WANG, C., ZHANG, L., HE, Y., AND TAN, Y. Frame rate up-conversion using trilateral filtering. IEEE Transactions on Circuits and Systems for Video Technology 20, 6 (June 2010), 886–893. EXTERNAL
  • LEE, K., AND JEONG, J. Bilateral frame rate up-conversion algorithm based on the comparison of texture complexity. Electronics Letters 52, 5 (2016), 354–355. EXTERNAL
  • DE HAAN, G. Video processing for multimedia systems. ISBN 90-9014015-8, Sept 2000. c© 2020 Society of Motion Picture & Television Engineers R© (SMPTE R© ) c© 17 EXTERNAL
  • LU, C., HIRSCH, M., AND SCHÖLKOPF, B. Flexible spatio-temporal networks for video prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017), no. 7, pp. 2137–2145. EXTERNAL
  • LIU, H., XIONG, R., ZHAO, D., MA, S., AND GAO, W. Multiple hypotheses bayesian frame rate up-conversion by adaptive fusion of motion-compensated interpolations. IEEE Transactions on Circuits and Systems for Video Technology 22, 8 (Aug 2012), 1188–1198. EXTERNAL
  • LEE, S., AND YANG, S. Adaptive motion-compensated frame rate up-conversion. Electronics Letters 38, 10 (May 2002), 451–452. c© 2020 Society of Motion Picture & Television Engineers R© (SMPTE R© ) c© 18 EXTERNAL
  • BAI, W., LIU, J., REN, J., AND GUO, Z. Visual-weighted motion compensation frame interpolation with motion vector refinement. In 2012 IEEE International Symposium on Circuits and Systems (ISCAS) (May 2012), pp. 500–503. EXTERNAL
Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001927.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree
Reference this Doc

Plain text (ISO 690 compliant)

Preview:
Anil Kokaram, Davinder Singh, and Simon Robinson; Twenty years of Frame Interpolation for Retiming in the Movies, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020. Available at https://doi.org/10.5594/M001927
Snippet:
Anil Kokaram, Davinder Singh, and Simon Robinson; Twenty years of Frame Interpolation for Retiming in the Movies, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020. Available at https://doi.org/10.5594/M001927

HTML (ISO 690 compliant)

Preview:
Anil Kokaram, Davinder Singh, and Simon Robinson; Twenty years of Frame Interpolation for Retiming in the Movies, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020. Available at https://doi.org/10.5594/M001927
Snippet:
<span class="citation">Anil Kokaram, Davinder Singh, and Simon Robinson; <cite>Twenty years of Frame Interpolation for Retiming in the Movies</cite>, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020. Available at <a href="https://doi.org/10.5594/M001927" target="_blank" rel="noopener">https://doi.org/10.5594/M001927</a></span>

SMPTE Icon SMPTE's HTML Pub

Preview:
Anil Kokaram, Davinder Singh, and Simon Robinson; Twenty years of Frame Interpolation for Retiming in the Movies, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020
doi: 10.5594/M001927
url: https://doi.org/10.5594/M001927
Snippet:
<li>
Anil Kokaram, Davinder Singh, and Simon Robinson; <cite id="bib-10-5594-m001927">Twenty years of Frame Interpolation for Retiming in the Movies</cite>, SMPTE Meetings and Conferences ( November 2020); SMPTE, 2020
<span class="doi">10.5594/M001927</span>
</li>