Encoding Optimizations for Video Game Live Streaming

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 132, No. 9, pp. 39–49
Abstract: In recent years, live video-game-centric streaming platforms have experienced dramatic growth, driven by the explosion of the esports market. Besides, live-event rightsholders and broadcasters are successfully “gamifying” their video offerings. Given the rise of video game live streaming, we address, in this article, the questions of how to optimize and adapt the encoding strategies for efficient game content coding. We first characterize game content, focusing on how its signal characteristics differ from “natural” content. We then share insights on selected encoding strategies targeting improvements in both compression efficiency and density/encoder run time for signals with such characteristics. Additionally, we examine the relevance of screen content coding (SCC) tools, as adopted in HEVC SCC and VVC standards, in a video game coding context. Finally, we conclude the article by highlighting the benefits of using game-content-aware technology with better compression efficiency under realtime constraints.
Publication Date: 2023-10-01
DOI: 10.5594/JMI.2023.3303499
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2023.3303499
Author(s): Julien Le Tanou
bio
Julien Le Tanou is a video technology expert and the head of video research at MediaKind, formerly Ericsson Media Solutions. In this role, he is responsible for leading research, innovation, and software development in video processing, compression, and associated encoding technologies. Since 2012, he has been with Envivio France in charge of research and algorithm design for Envivio’s SW video encoding solutions. Before Envivio, he has been with Orange Labs in France and Dolby Laboratories in the USA, where he conducted research and development for the next generation of video coding standards. He received an MS degree in signal and image processing from Telecom Paris, France, in 2010, and an MSc degree in computer and electrical engineering from the Institut Superieure d’Electronique de Paris, France, in 2008.
Nelson Francisco
bio
Nelson Francisco is a principal video compression engineer at MediaKind, formerly Ericsson Media Solutions, joining in 2013, to work on the development of core compression algorithms used for both hardware and software encoding solutions. His research currently focuses on media processing and delivery applications of AI and ML, ranging from encoder optimization to emerging topics such as super-resolution or video semantics analysis. Between 2007 and 2013, he collaborated with the Portuguese Telecommunication Institute (IT) on several government-funded projects, while being an assistant professor at the Polytechnic Institute of Leiria, Portugal. He received an MSc degree in electronics engineering from the University of Tras-os-Montes e Alto Douro, Portugal, in 2008, and a PhD degree in signal processing from the Federal University of Rio de Janeiro, Brazil, in 2012.
Keyword(s): Encoding, Live Game Streaming, Screen Content Coding, Video Game Coding
Copyright: © 2023 SMPTE

Bibliographic Reference(s)

M. Bichon, J. Le Tanou, M. Ropert, W. Hamidouche, and EXTERNAL
SRO Motorsports Group. 2021. [Online]. Available: https://www.sro-motorsports.com/news/53/sro-esports-launches-global-championships-for-sim-drivers-and-teams EXTERNAL
S. G. Blasi, I. Zupancic, and E. Izquierdo, “Adaptive Precision Motion Estimation for HEVC Coding,” Proc. 31th Picture Coding Symp., PCS, Cairns, Australia, pp. 144–148, 2015. EXTERNAL
A. Norkin et al., “HEVC Deblocking Filter,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 22(12):1746–1754, Dec. 2012. EXTERNAL
J. Lainema et al., “Intra Coding of the HEVC Standard,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 22(12):1792–1801, Dec. 2012. EXTERNAL
N. Barman, S. Zadtootaghajy, S. Schmidty, M. G. Martini, and S. Moller, “GamingVideoSET: A Dataset for Gaming Video Streaming Applications,” Proc. of 2018 16th Annu. Workshop on Netw. and Syst. Support for Games (NetGames), Jun. 2018. EXTERNAL
T. Nguyen et al., “Overview of the Screen Content Support in VVC: Applications, Coding Tools, and Performance,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 31(10):3801–3817, Oct. 2021. EXTERNAL
Q. Zhang, Y. Dai, and C.-C. Kuo, “Fast Sub-Pel Motion Vector Prediction Via Block Classification,” 2009 16th IEEE Int. Conf. on Image Process. (ICIP), pp. 1569–1572, Nov. 2009. EXTERNAL
X. Xu and S. Liu, “Overview of Screen Content Coding in Recently Developed Video Coding Standards,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 32(2):839–852, Feb. 2022. while the ones that provide the lowest ratios are prioritized to be disabled if required. To validate this content-aware approach for video game content, we compared it to a reference case where encoding tools are dynamically enabled in a fixed order, independent of the input content, and defined to maximize the compression efficiency on natural content. Both configurations were run in the same computational resources and only standard encoding tools were included in this evaluation (the specific screen content tools mentioned in the previous sections are out of scope for this evaluation). It was observed that the contentaware approach achieved compression gains of 7.4% in H.264 and 6.9% in HEVC (average BD-SSIM), by simply prioritizing encoding tools with a larger impact while encoding video game content. Overall, it was observed that the ME strategy has a very significant impact on compression efficiency for video game content, while the motion refinement itself has a lower impact than for natural content. Similarly, picture partitioning and mode decision strategy have a higher weight on the compression efficiency of video game content, with the compression efficiency being somewhat less dependent on the GOP sequencing and transition management. Conclusion Motivated by the growing traffic of live video game streaming and the related esport market, we investigated, in this work, the possible encoding optimizations and normative toolset to best compress and stream game content. We first identify the main distinguished signal characteristics of game content in comparison to natural content. From this knowledge, we share some practical examples of relevant encoding strategies into an optimized HEVC encoder that provide significant improvements in compression efficiency or encode runtime/CPU cycle saving in the context of video game coding. The suggested optimization techniques are ranging from bitrate trading and AQP within a GOP, frame-adaptative deblocking and smoother in-loop reference sample filtering, the use of 10-bit encoding accuracy to adaptive motion precision estimation, and fast subpel ME. All combined, they show potential for about 10% average BD-rate gain in compression efficiency. Additionally, we review normative SCC-related tools, as adopted in HEVC SCC extension and VVC, discuss their design, and assess their compression performance for game content based on reference SW models. This specific toolset provides substantial bitrate saving for the same MS-SSIM or PSNR score with the main interesting features being the IBC and the AMVR. Finally, leveraging on AI-based content-aware method, we report that compression efficiency can be improved by 7% in H.264 or HEVC-optimized implementations by simply prioritizing encoding tools with a larger impact while encoding videogame content. Overall, we EXTERNAL
N. Francisco and J. Le Tanou, “Optimizing Real-Time Video Encoders With ML,” Proc. of the 1st ACM Mile-High Video Conf. (MHV '22), Mar. 2022. EXTERNAL
A. Pennington, “The State of Live Streaming in 2021,” Streaming Media, 2021. [Online]. Available: https://www.stream-ingmedia.com/Articles/ReadArticle.aspx?ArticleID=146325 EXTERNAL
“HM+SCM Reference Software for HEVC SCC Extension, Tag HM-16.21+SCM-8.8.” [Online]. Available: https://vcgit.hhi.fraunhofer.de/jvet/HM/-/tree/HM-16.21+SCM-8.8 EXTERNAL
G. Bjøntegaard, “Calculation of Average PSNR Differences Between RD-Curves,” Technical Report, VCEG-M33, ITU-T SG16/Q6, 2001. EXTERNAL
Park, S.-H., “A Sub-Pixel Motion Estimation Skipping Method for Fast HEVC Encoding,” ICT Express, 5(2):136–140, Jun. 2019. EXTERNAL
J. Xu, R. Joshi, and R. A. Cohen, “Overview of the Emerging HEVC Screen Content Coding Extension,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 26(1):50–62, Jan. 2016. EXTERNAL
M. Ropert, J. Le Tanou, and M. Blestel, “Mastering Quantization is Key for Video Compression,” Proc. of 2021 Int. Broadcast. Conv. (IBC), Dec. 2021. EXTERNAL
J. Pfaff et al., “Intra Prediction and Mode Coding in VVC,” IEEE Trans. Circuits Syst. Video Technol. (TCSVT), 31(10):3834– 3847, Oct. 2021. EXTERNAL
C. Settimi, “The NBA’s Coronavirus Shutdown Led the Phoenix Suns into a Virtual Arena,” Forbes, 2020. [Online]. Available: https://w w w.forbes.com /sites/christinasettimi /2020/03/20/ the-nbas-coronavirus-shutdown-led-the-phoenix-suns-into-avirtual-arena-when-they-drew-3-million-fans-the-sports-worldtook-note/?sh=5fd5b7ff74b8 EXTERNAL
W. Dai, O. C. Au, W. Zhu, W. Hu, P. Wan, and J. Li, “A Robust Interpolation-Free Approach for Sub-Pixel Accuracy Motion Estimation,” Proc. IEEE Int. Conf. on Image Process., ICIP, Melbourne, Australia, pp. 1767–1771, 2013. EXTERNAL
“VTM Reference Software for VVC, Tag VTM-16.” [Online]. Available: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/tree/VTM-16.0 EXTERNAL
Y. Li, Z. Liu, X. Ji, and D. Wang, “HEVC Fast FME Algorithm Using IME RD-Costs Based Error Surface Fitting Scheme,” Proc. Visual Commun. and Image Process., VCIP, Chengdu, China, 2016. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2023.3303499.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Julien Le Tanou and Nelson Francisco; Encoding Optimizations for Video Game Live Streaming, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 9, October 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3303499

Snippet:

Julien Le Tanou and Nelson Francisco; Encoding Optimizations for Video Game Live Streaming, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 9, October 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3303499

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Julien Le Tanou and Nelson Francisco; <cite>Encoding Optimizations for Video Game Live Streaming</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 9, October 2023); SMPTE, 2023. Available at <a href="https://doi.org/10.5594/JMI.2023.3303499" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2023.3303499</a></span>

SMPTE's HTML Pub

Preview:

Julien Le Tanou and Nelson Francisco; Encoding Optimizations for Video Game Live Streaming, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 9, October 2023); SMPTE, 2023
doi: 10.5594/JMI.2023.3303499
url: https://doi.org/10.5594/JMI.2023.3303499

Snippet:

<li>
Julien Le Tanou and Nelson Francisco; <cite id="bib-10-5594-jmi-2023-3303499">Encoding Optimizations for Video Game Live Streaming</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 9, October 2023); SMPTE, 2023
<span class="doi">10.5594/JMI.2023.3303499</span>
</li>