The future of video compression - Moving beyond hybrid codecs with machine learning

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2023, No. 10, pp. 16–22
Abstract: The consumption of video content on the internet is increasing at a constant pace, along with an increase of video quality. As an answer to the ever-growing demand for high quality video, compression technology improves steadily. About every decade, a new major video compression standard is issued, providing a decrease of bitrate by a factor two. Interestingly, the technology does not change radically between codecs generations. Instead, the same principles are re-used and pushed further. There has been several attempts to depart from this model, but none achieved to be competitive. Recently, the research community has started focusing on deep learning-based strategies. Could it be the new contender to the classical approach? This paper analyzes the benefits and limitations of deep learning-based video compression methods, and investigates practical aspects such as rate control, delay, memory consumption and power consumption. Overlapping patch-based end-to-end video compression strategy is proposed to overcome memory limitations.
Publication Date: 2023-10-16
DOI: 10.5594/M002026
ISBN: Print: 978-1-61482-964-5
Link: https://doi.org/10.5594/M002026
Author(s): Thomas GuionnetAteme
Marwa TarchouliAteme
Thomas BurnichonAteme
Mickael RauletAteme
Keyword(s): Video Compression, Video Codec, MPEG-2, H.264, AVC, HEVC, VVC, Artificial Intelligence, Machine Learning, Deep Learning, End-to-end Video Encoding
Copyright: © 2023 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

ISO/IEC 14496-2:2004 [Active, Amended]
N. Imam and T. A. Cleland, "Rapid online learning and robust recall in a neuromorphic olfactory circuit," nature machine intelligence, p. 181–191, 2020. EXTERNAL
S. Santurkar, D. Budden and N. Shavit, "Generative Compression," 2018 Picture Coding Symposium (PCS), 2018. EXTERNAL
F. Galpin et al., "AHG11: Deep-learning based inter prediction blending", JVET-V0076-v2," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 22nd Meeting,, teleconference, 20–28 Apr. 2021. EXTERNAL
J. Deng, W. Dong, R. Socher, L. J. Li, K. Li and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," IEEE conference on computer vision and pattern recognition (pp. 248-255), 2009. EXTERNAL
L. F. W. Anthony, B. Kanding and R. Selvan, "Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models," ICML Workshop on "Challenges in Deploying and monitoring Machine Learning Systems", 2020. EXTERNAL
Y. Bai, C. Dong and C. Wang, "PS-NeRV: Patch-wise Stylized Neural Representations for Videos," arXiv preprint, 2022. EXTERNAL
Sutskever, A., I. Hinton and E. G. Krizhevsky, "Imagenet classification with deep convolutional neural networks.," Advances in neural information processing systems, 2012. EXTERNAL
P. Chen and J. W. Woods, "Bidirectional MC-EZBC with lifting implementation," Oct. 2004,. EXTERNAL
T. Ladune, P. Philippe, W. Hamidouche, L. Zhang and O. Deforges, "Conditional Coding for Flexible Learned Video Compression," Neural Compression Workshop ICLR 2021, 2021. EXTERNAL
J. C. Lee, D. Rho, J. H. Ko and E. Park, "FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos," arXiv preprint, 2022. EXTERNAL
L. Jianping, L. Dong, L. Houqiang and W. Feng, "M-lvc: : Multiple frames prediction for learned video compression," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. EXTERNAL
"Cisco Annual Internet Report (2018–2023) White Paper, https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html". EXTERNAL
"JVET-Z0012-v1 "JVET AHG report: Enhanced compression beyond VVC capability (AHG12)" Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 26th Meeting, by teleconference, 20–29 April 2022.". EXTERNAL
X. Sheng, J. Li, B. Li, L. Li, L. Dong and L. Yan, "Temporal Context Mining for Learned Video Compression," IEEE Transactions on Multimedia, vol. 25, pp. 7311-7322, 2023 DOI: 10.1109/TMM.2022.3220421.. EXTERNAL
"Video Encoder Market with COVID-19 Impact by Number of Channel (Single, Multichannel), Mounting Type (Standalone, Rack-mounted), Application (Broadcast, Surveillance (Commercial, Residential, Institutional)), and Geography - Global Forecast to 2025.," https://www.marketsandmarkets.com/Market-Reports/video-encoder-market-109133493.html.. EXTERNAL
J. Li, B. Li and Y. Lu, "Deep Contextual Video Compression," Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. EXTERNAL
[Online]. Available: https://hailo.ai/products/hailo-8/. EXTERNAL
V. Bottreau, M. Benetiere, B. Felts and B. Pesquet-Popescu, "A fully scalable 3D subband video codec,," 2001. EXTERNAL
H. Chen, B. He, H. Wang, Y. Ren, L. S. Nam and A. Shrivastava, "NeRV: Neural Representations for Videos," Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. EXTERNAL
Q. Zhao, M. S. Asif and Z. Ma, "DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2031-2040, 2023. EXTERNAL
G. Sullivan, "Overview of International Video Coding Standards (preceding H.264/AVC)," ITU-T VICA Workshop, 22-23 July 2005, ITU Headquarter, Geneva, 2005 . EXTERNAL
F. Mentzer, G. Toderici, D. Minnen, S.-J. Hwang, S. Caelles, M. Lucic and E. Agustsson, "VCT: A Video Compression Transformer," NeurIPS'22, 2022. EXTERNAL
R. Yang, F. Mentzer, L. V. Gool and R. Timofte, "Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model," IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 388 - 401, 2021. EXTERNAL
M. Meyer and C. Rohlfing, "AHG11-related: Investigation on CNN-based Intra Prediction," JVET-U0105-v3, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 21st Meeting, teleconference, 6–15 Jan. 2021. EXTERNAL
Chaoyi Lin et al., "EE1-2.3: CNN-based Super Resolution for Video Coding Using Decoded Information, JVET-Y0069-v21," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 January 2022 EXTERNAL
Z. Li, M. Wang, H. Pi, K. Xu, J. Mei and Y. Liu, "E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context," European Conference on Computer Vision, p. 267–284, 2022. EXTERNAL
Zizheng Liu et al., "AHG11: NN-based Reference Frame Interpolation for VVC Hierarchical Coding Structure, JVET-Y0096-v1," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 25th Meeting, teleconference, 12–19 Jan.2022. EXTERNAL
S. V. K. S. G. ATHISHA, "An Overview Of H.26x Series And Its Applications," International Journal of Engineering Science and Technology, vol. 2(9), pp. 4622-4631, 2010. EXTERNAL
E. Dupont, A. Goliński, M. Alizadeh, Y. W. Teh and A. Doucet, "COIN: COmpression with Implicit Neural representations," Computer Vision and Pattern Recognition, 2021. EXTERNAL
"JVET-Y0150-v2 "EE2-1.1: Tests on unsymmetric partitioning methods" Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 25th Meeting, by teleconference, 12–21 January 2022.". EXTERNAL
"Workshop and Challenge on learned image compression," [Online]. Available: Https://www.compression.cc/, 2020.. EXTERNAL
G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai and Z. Gao, "DVC: An End-To-End Deep Video Compression Framework," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11006-11015, 2019. EXTERNAL
W. Thomas and S. Heiko, "Video Coding: Part II of Fundamentals of Source and Video Coding," Foundations and Trends® in Signal Processing, http://dx.doi.org/10.1561/2000000078, vol. 10 No. 1–3, pp. 1-346, 2016. EXTERNAL
O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle and L. Bourdev, "ELF-VC: Efficient Learned Flexible-Rate Video Coding," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14479-1448, 2021. EXTERNAL
Z. Cheng, H. Sun, M. Takeuchi and J. Katto, "Learned image compression with discretized gaussian mixture likelihoods and attention modules," 2020. EXTERNAL
[Online]. Available: https://developer.nvidia.com/tensorrt. EXTERNAL
Liqiang Wang et al., "Neural Network Based in-Loop Filter with Constrained Memory," in 2022 IEEE International Conference on Multimedia and Expo (ICME), 2022 DOI: 10.1109/ICME52920.2022.9859910. EXTERNAL
"JVET-Z0023 "EE1: Summary of Exploration Experiments on Neural Network-based Video Coding" Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 26th Meeting, by teleconference, 20–29 April 2022.". EXTERNAL
Elena Alshina et al., "JVET AHG report: Neural network-based video coding," JVET-AA0011-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 27th Meeting, teleconference, 13–22 July 2022. EXTERNAL
G. Bjøntegaard, "Calculation of average PSNR differences between RD-curves," in Technical Report VCEG-M33, ITU-T SG16/Q6,, Austin, Texas, USA, 2001. EXTERNAL
"MPEG-1 (Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s)," ISO/IEC JTC 1/SC 29, ISO/IEC 11172-x:1993. EXTERNAL
H. Chen, M. Gwilliam, B. He, S.-N. Lim and A. Shrivastava, "CNeRV: Content-adaptive Neural Representation for Visual Data," BMVC 2022, 2022. EXTERNAL
Y. Sun, Xu, T. ,. X. M. and J. Lu, "Online dictionary learning based intra-frame video coding via sparse representation," 2012. EXTERNAL
"ITU-T Recommendation H.120 (11/88). https://www.itu.int/rec/T-REC-H.120-198811-S/en". EXTERNAL
O. Gafni, A. Polyak, O. Ashual, S. Sheynin, D. Parikh and Y. Taigman, "Make-a-scene: Scene-based text-to-image generation with human priors.," arXiv preprint arXiv:2203.13131., 2022. EXTERNAL
[Online]. Available: https://www.mpeg.org/. EXTERNAL
E. Moyano, F. Quiles, A. Garrido, T. Orozco-Barbosa and J. Duato, "Efficient 3D wavelet transform decomposition for video compression," pp. 118-125 , 2001 DOI: 10.1109/DCV.2001.929950. EXTERNAL
T. Ladune, P. Philippe, F. Henry, G. Clare and T. Leguay, "COOL-CHIC: Coordinatebased Low Complexity Hierarchical Image Codec," arXiv preprint, 2022. EXTERNAL
D. Thomas, "Reducing Machine Learning Inference Cost for PyTorch Models," aws online tech talks, 2020. EXTERNAL
T. Biatek, M. Abdoli, T. Guionnet, A. Nasrallah and M. Raulet, "Future MPEG standards VVC and EVC: 8K broadcast enabler," Available at: https://www.ibc.org/technical-papers/future-mpeg-standards-vvc-and-evc-8k-broadcast-enabler/6754.article, 14 September 2020.. EXTERNAL
"ITU-T Recommendation H.265 (08/21). https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=14660&lang=en". EXTERNAL
A. Habibian, T. van Rozendaal, J. M. Tomczak and T. S. Cohen, "Video Compression With Rate-Distortion Autoencoders," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7033-7042, 2019. EXTERNAL
"ITU-T Recommendation H.264 (08/21) https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=14659&lang=en". EXTERNAL
F. Bossen, X. Li and K. Suehrin, "AHG report: Test model software development (AHG3)," JVET-T0003-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 20th Meeting, teleconference, 7 – 16 Oct. 2020. EXTERNAL
R. D. Kell., "Improvements relating to electric picture transmission systems.". British Patent Patent 341,811, 1929.. EXTERNAL
[Online]. Available: https://www.itu.int/. EXTERNAL
Yue Li et al., "EE1-1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, JVET-Y0143-v2,," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 January 2022. EXTERNAL
Chaoyi Lin et al., "AHG11: Variable rate end-to-end image compression", JVET-U0102,," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 21st Meeting, teleconference, 6–15 Jan. 2021.. EXTERNAL
H. Liu, M. Lu, Z. Ma, F. Wang, Z. Xie, X. Cao and Y. Wang, "Neural Video Coding Using Multiscale Motion Compensation and Spatiotemporal Context Model," IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, pp. 3182 - 3196, 2021. EXTERNAL
M. Mahdavi-Nasab and H. Irannejad, "Block Matching Video Compression Based on Sparse Representation and Dictionary Learning.," 2018. EXTERNAL
P. Lambert, W. P. De Neve, I. Moerman, P. Demeester and R. Van de Walle, "Ratedistortion performance of H.264/AVC compared to state-of-the-art video codecs," Jan 2006. EXTERNAL
"ITU-T Recommendation H.266 (04/22). https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=14948&lang=en". EXTERNAL
"ITU-T Recommendation H.263 (01/05)". EXTERNAL
[Online]. Available: https://en.wikipedia.org/wiki/Machine_learning. EXTERNAL
M. Tarchouli, S. Pelurson, T. Guionnet, W. Hamidouche, M. Outtas and O. Deforges, "Patch-Based Image Coding with End-To-End Learned Codec using Overlapping.," 12th International Conference on Artificial Intelligence, Soft Computing and Applications , pp. 53-63. , 2022 DOI 10.5121/csit.2022.122305.. EXTERNAL
Takeshi Chujoh et al., "EE1.2: Additional experimental results of NN-based super resolution (JVET-U0053)," JVET-V0073-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 22nd Meeting, teleconference, 20–28 Apr. 2021. EXTERNAL
T. Dumas et al., "EE1 test 3.1: intra prediction using neural networks," JVET-Y0082-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 January 2022. EXTERNAL
S. Kim, S. Yu, J. Lee and J. Shin, "Scalable Neural Video Representations with Learnable Positional Features," Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. EXTERNAL
R. Yang, R. Timofte and L. V. Gool, "Perceptual Learned Video Compression with Recurrent Conditional GAN," Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22), 2021. EXTERNAL
H. Chen, M. Gwilliam, S.-N. Lim and A. Shrivastava, "HNeRV: A Hybrid Neural Representation for Videos," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10270-10279, 2023. EXTERNAL
"Video trace library. 2001.," [Online]. Available: http://trace.eas.asu.edu/yuv/index.html,. [Accessed 11 11 2020]. EXTERNAL
[Online]. Available: https://www.videolan.org/developers/x265.html. EXTERNAL
V. J. Reddi, C. Cheng, D. Kanter, P. Mattson and .. Y. Z. Guenther Schmuelling, "MLPerf Inference Benchmark," ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020. EXTERNAL
R. Yang, F. Mentzer, L. V. Gool and R. Timofte, "Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6628-6637, 2020. EXTERNAL
D. S. TAUBMAN, M. W. MARCELIN and M. RABANI, "JPEG2000 Image Compression Fundamentals, Standards and Practice," Springer Science & Business Media., vol. 642, 2012. EXTERNAL
"ITU-T Recommendation H.261 (03/93) https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=1088&lang=en". EXTERNAL
[Online]. Available: https://www.videolan.org/developers/x264.html. EXTERNAL
D. Amodei and D. Hernandez, "AI and compute," 2018. [Online]. Available: https://openai.com/research/ai-and-compute. EXTERNAL
"ITU-T Recommendation H.262 (02/12) https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=11627&lang=en". EXTERNAL
A. M. Turing, "Computing machinery and intelligence," MIND--COMPUTING-MACHINERY-AND-INTELLIGENCE, pp. 433-460, 1950. EXTERNAL
D. He, Z. Yang, W. Peng, R. Ma, H. Qin and Y. Wang, "ELIC: Efficient Learned Image Compression With Unevenly Grouped Space-Channel Contextual Adaptive Coding," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5718-5727, 2022. EXTERNAL
J. Li, B. Li and Y. Lu, "Hybrid spatial-temporal entropy modelling for neural video compression," Proceedings of the 30th ACM International Conference on Multimedia, pp. pp. 1503-1511, 2022. EXTERNAL
A. Reuther, P. Michaleas, M. Jones, V. Gadepally and J. K. Siddharth Samsi, "AI and ML Accelerator Survey and Trends," 2022 IEEE High Performance Extreme Computing Conference (HPEC), 2022. EXTERNAL
[Online]. Available: https://docs.openvino.ai/latest/index.html#. EXTERNAL
G.-H. Wang, J. Li, B. Li and Y. Lu, "EVC: Towards Real-Time Neural Image Compression with Mask Decay," ICLR 2023, 2023. EXTERNAL
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler and S. Hochreiter, "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium," Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. EXTERNAL
Ming Lu et al., "EE1: Tests on Decomposition, Compression, Synthesis (DCS)-based Technology, JVET-V0149," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 22th Meeting, teleconference, 20 – 28 Apr. 2021. EXTERNAL
F. Bossen, J. Boyce, X. Li, V. Seregin and K. Sühring, "VTM Common Test Conditions and Software Reference Configurations for SDR Video," document JVET-T2010 of JVET, Oct 2020. EXTERNAL
J. Mairal, F. Bach and J. Ponce, "Sparse Modeling for Image and Vision Processing .," 2014 DOI: https://doi.org/10.48550/arXiv.1411.3230. EXTERNAL
Changyue Ma et al., "AHG11: Neural Network Based Motion Compensation Enhancement for Video Coding", JVET-Y0090-v1," Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 January 2022. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M002026.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Thomas Guionnet, Marwa Tarchouli, Thomas Burnichon, and Mickael Raulet; The future of video compression - Moving beyond hybrid codecs with machine learning, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002026

Snippet:

Thomas Guionnet, Marwa Tarchouli, Thomas Burnichon, and Mickael Raulet; The future of video compression - Moving beyond hybrid codecs with machine learning, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002026

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Thomas Guionnet, Marwa Tarchouli, Thomas Burnichon, and Mickael Raulet; <cite>The future of video compression - Moving beyond hybrid codecs with machine learning</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at <a href="https://doi.org/10.5594/M002026" target="_blank" rel="noopener">https://doi.org/10.5594/M002026</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Thomas Guionnet, Marwa Tarchouli, Thomas Burnichon, and Mickael Raulet; <cite id="bib-10-5594-m002026">The future of video compression - Moving beyond hybrid codecs with machine learning</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5
<span class="doi">10.5594/M002026</span>
</li>