Real-World Use of AI for Better Video Compression

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 132, No. 4, pp. 6–15
Abstract: Video compression is a complex task, with an almost unlimited number of potential encoding decision permutations. While software-based encoders can, in theory, exhaustively explore all the possible encoding options in a bid to maximize compression efficiency, such an approach means there is the potential to consume as much central processing unit (CPU) power as is available. A direct consequence is a poor tradeoff between the processing required and the reduction in bitrate achieved, especially since technical and commercial limitations often place constraints on the amount of processing for any given application. A long-standing aim for encoders is, therefore, to optimize the use of the available computing resources, either by driving down the bitrates (and therefore delivery costs) or reducing the CPU cost to achieve the required quality. A common approach to reduce CPU costs is to take shortcuts and make decisions upfront based on less information, in an attempt to guess or shortlist decisions without exhaustively evaluating all options. However, the diverse and extensive number of permutations of options available in modern codecs make these decisions very complex, meaning that this approach normally leads to a less bitrate-efficient compression. If means can be found to utilize the CPU more effectively, then the objective of improved encoder performance, measured in either bitrate reduction, infrastructure cost, or a combination of both, can be achieved. This is the application space considered in this article: how to use artificial intelligence (AI) and machine learning (ML) to efficiently perform encoding decisions in realtime based on the characteristics of the incoming content, allowing the use of the CPU power available more effectively (or equivalently, reducing the need for CPU power). Considering the AI processing itself also requires CPU power, such an application requires carefully tuned approaches to guarantee that the combination of the AI processing plus its dynamically adapted encode algorithm achieves a better bitrate versus CPU for the same quality than can be achieved using standard algorithms.
Publication Date: 2023-05-01
DOI: 10.5594/JMI.2023.3263505
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2023.3263505
Author(s): Nelson Francisco
bio
Nelson Francisco is a principal video compression engineer at MediaKind, Southampton, U.K., formerly Ericsson Media Solutions, which he joined in 2013 to work on the development of core compression algorithms used for both hardware and software encoding solutions. His research currently focuses on media processing and delivery applications of artificial intelligence (AI) and machine learning (ML), ranging from encoder optimization to emerging topics such as super-resolution or video semantics analysis. From 2007 to 2013, he collaborated with the Portuguese Telecommunication Institute (IT), Lisbon, Portugal, on several government-funded projects, while being an assistant professor at the Polytechnic Institute of Leiria, Leiria, Portugal. He received an MSc degree in electronics engineering from the University of Tras-os-Montes e Alto Douro, Vila Real, Portugal, in 2008, and a PhD degree in signal processing from the Federal University of Rio de Janeiro, Brazil, in 2012.
Julien Le Tanou
bio
Julien Le Tanou is a video technology expert and the head of the Video Research Team at MediaKind, Rennes, France, formerly Ericsson Media Solutions. In this role, he is responsible for leading research, innovation, and software development in video processing, compression, and associated encoding technologies. Since 2012, he has been with Envivio, France, in charge of research and algorithms design for Envivio’s software video encoding solutions. Before Envivio, he has been with Orange Labs, France and Dolby Laboratories, San Francisco, CA, where he conducted research and development for the next generation of video coding standards. He received an MS degree in signal and image processing from Telecom Paris, France, in 2010, and an MSc degree in computer and electrical engineering from the Institut Superieure d’Electronique de Paris, in 2008.
Tony Jones
bio
Tony Jones has 30 years of experience in advanced cutting-edge digital video technology. Starting in research and development designing post-production digital video effects technology at Questech, Wokingham, U.K., he moved to digital transmission systems for satellites, cable, Internet Protocol Television (IPTV), and streaming, through Tandberg Television, Southampton, U.K., Ericsson, Southampton, and now Mediakind, Southampton, U.K. As a principal technologist, he is currently part of the MediaKind strategy group working on a wide range of technologies including compression, cloud technology, machine learning, and adaptive bitrate (ABR) delivery optimization. He has a BSc (Eng.) degree from the Imperial College of Science and Technology, London, U.K., and has had 12 patents granted relating to advanced digital media.
Keyword(s): Computational Complexity, Context Awareness, Video Codec, Artificial Intelligence
Copyright: © 2023 SMPTE

Bibliographic Reference(s)

J. De Cock, “The (Non)Sense of Artificial Intelligence in Real-Time Video Encoding,” IBC Conf. Proc., 2021. EXTERNAL
L. Merritt and R. Vanam, “X264: A High Performance H.264/ AVC Encoder,” 2006. EXTERNAL
International Telecommunication Union-Telecommunications (ITU-T), “H.265: High Efficiency Video Coding,” 2013. EXTERNAL
International Telecommunication Union-Telecommunications (ITU-T), “H.264: Advanced Video Coding for Generic Audiovisual Services,” 2003. EXTERNAL
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object Detectors Emerge in Deep Scene CNNs,” Int. Conf. on Learn. Representations, 2015. EXTERNAL
International Telecommunication Union-Telecommunications (ITU-T), “H.266: Versatile Video Coding,” 2020. EXTERNAL
I.-K. Kim, J. Min, T. Lee, W.-J. Han, and J. Park, “Block Partitioning Structure in the HEVC Standard,” IEEE Trans. Circuits Syst. Video Technol., 22:1697–1706, Dec. 2012. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2023.3263505.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Nelson Francisco, Julien Le Tanou, and Tony Jones; Real-World Use of AI for Better Video Compression, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 4, May 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3263505

Snippet:

Nelson Francisco, Julien Le Tanou, and Tony Jones; Real-World Use of AI for Better Video Compression, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 4, May 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3263505

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Nelson Francisco, Julien Le Tanou, and Tony Jones; <cite>Real-World Use of AI for Better Video Compression</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 4, May 2023); SMPTE, 2023. Available at <a href="https://doi.org/10.5594/JMI.2023.3263505" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2023.3263505</a></span>

SMPTE's HTML Pub

Preview:

Nelson Francisco, Julien Le Tanou, and Tony Jones; Real-World Use of AI for Better Video Compression, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 4, May 2023); SMPTE, 2023
doi: 10.5594/JMI.2023.3263505
url: https://doi.org/10.5594/JMI.2023.3263505

Snippet:

<li>
Nelson Francisco, Julien Le Tanou, and Tony Jones; <cite id="bib-10-5594-jmi-2023-3263505">Real-World Use of AI for Better Video Compression</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 4, May 2023); SMPTE, 2023
<span class="doi">10.5594/JMI.2023.3263505</span>
</li>