Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming

Metadata

Publisher: SMPTE
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Motion Imaging Jour.
Volume: 133, No. 5, pp. 60–70
Abstract: Pixelweave Chromascale is a new open-source library that can leverage GPU computing to efficiently convert video frame pixel formats. The library enhances the computational performance of high-quality real-time video streaming by accelerating pre-frame conversions and offloading resource-intensive tasks from the CPU. It currently supports a wide range of pixel formats, including RGBA and chroma-subsampled YUV, using up to 16 bits per color channel. It is accessed through a C++ interface that triggers a GPU compute pipeline. It uses a novel approach that samples $2 \times 2$ pixel windows, ensuring the code is easily extensible and independent of source and destination formats. Evaluations of multiple video samples demonstrate significant improvement in frames-per-second when compared to existing state-of-the-art CPU-based libraries and show improved video quality in real-time streaming applications. Pixelweave technology has been integrated into a production tool to create numerous award-winning films, enabling more performant and color-accurate real-time streaming.
Publication Date: 2024-09-01
DOI: 10.5594/JMI.2024/YPJG9446
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2024/YPJG9446
Author(s): Jose Aguerre
Bruno Sena
Damien Stolarz
Keyword(s): Real-Time Video, GPU Computing, Pixel Formats
Copyright: © 2024 SMPTE

Bibliographic Reference(s)

T. Huang, R. Zhang, C. Zhou, and L. Sun, “QARC: Video quality aware rate control for real-time video streaming based on deep reinforcement learning,” in Proc. of the 26th ACM International Conference on Multimedia, pp. 1208–1216, 2018. EXTERNAL
G. K. Illahi, T. Van Gemert, M. Siekkinen, E. Masala, A. Oulasvirta and A. Ylä-Jääski, “Cloud gaming with foveated video encoding,” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 16 (1): pp. 1–24, 2020. EXTERNAL
J. Lawrence, D. Goldman, and S. Achar, “Project Starline: A High-Fidelity Telepresence System,” ACM Transactions on Graphics 40(6), Article 242, Dec. 2021. [Online]. Available: https://doi.org/10.1145/3478513.3480490. EXTERNAL
F. Heide, M. Steinberger, Y. Tsai, M. Rouf, D. Pająk, D. Reddy, O. Gallo, J. Liu, W. Heidrich, K. Egiazarian, “Flexisp: A flexible camera image processing framework,” ACM Trans. on Graphics 33 (6): pp. 1–13, Nov. 2014. EXTERNAL
J. Žáadník, M Mäkitalo, J. Vanne, and P. Jääskeläinen, “Image and Video Coding Techniques for Ultra-low Latency,” ACM Computing Surveys (CSUR) 54 (11s): pp. 1–35, Sep. 2022. EXTERNAL
S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Transactions on Graphics (ToG) 35 (6): pp. 1–12, Dec. 2016. EXTERNAL
L. Lou, P. Nguyen, J. Lawrence, and C. Barnes, “Image perforation: Automatically accelerating image pipelines by intelligently skipping samples,” ACM Transactions on Graphics (TOG) 35 (5): pp. 1–14, 2016. EXTERNAL
W. Liu, P. Zhang, X. Huang, J. Yang, C. Shen, and I. Reid, “Real-time Image Smoothing via Iterative Least Squares,” ACM Trans. on Graphics (TOG) 39 (3): pp. 1–24, Jun. 2020. EXTERNAL
D.R. Bull and F. Zhang, Intelligent Image and Video Compression: Communicating Pictures, Academic Press, 2021. EXTERNAL
Z. Li, Z. Duanmu, W. Liu, and Z. Wang, “AVC, HEVC, VP9, AVS2 OR AV1? - A comparative study of state-of-the-art video encoders on 4K videos,” in Proc. 76th International Conference on Image Analysis and Recognition (ICIAR 2079), Waterloo, ON, Canada, pp. 162–173, Aug. 2019. EXTERNAL
V. Subhash, “Quick Start Guide to FFmpeg,” Quick Start Guide to FFmpeg: Learn to Use the Open Source Multimedia-Processing Tool like a Pro, Apress, pp. 11–15, 2023. EXTERNAL
N. Blum, S. Lachapelle, and H. Alvestrand, “WebRTC: Real-Time Communication for the Open Web Platform,” Communications of the ACM, 64(8): pp. 50–54, Jul. 2021. [Online]. Available]: https://doi.org/10.1145/3453182. EXTERNAL
F. Mellbin, “Zimg: Scaling, colorspace conversion, and dithering library,” 2023, Accessed 24 Apr. 2023. [Online]. Available: https://github.com/sekrit-twc/zimg. EXTERNAL
Blackmagicdesign, “Decklink SDK Documentation,” 2021. Accessed 24 Apr. 2023. [Online]. Available: https://www.blackmagicdesign.com/developer/. EXTERNAL
J. Shin, J. Ahn, and J. Park, “Stereoscopic low-latency vision system via ethernet network for humanoid teleoperation,” in Proc. IEEE 19th International Conference on Ubiquitous Robots (UR), 313–317, 2022. EXTERNAL
A. Kaur and S. Singh, “A Survey of Streaming Protocols for Video Transmission,” in Proc. International Conference on Data Science, Machine Learning and Artificial Intelligence, pp. 186–191, 2021. EXTERNAL
C. Haine, “Color Grading 101: Getting Started Color Grading for Editors, Cinematographers, Directors, and Aspiring Colorists,” Routledge, 2019. EXTERNAL
T. Lottes, “Using Vulkan* Device Memory - gpuopen.com,” July 2016. Accessed 16 May 2023. [Online]. Available: https://gpuopen.com/learn/vulkan-device-memory/. EXTERNAL
S. White, A. Jenks, J. Kennedy, and M. Jacobs, “Direct3D 12 programming guide,” 2021. Accessed 18 May 2023. [Online]. Available: https://learn.microsoft.com/enus/windows/win32/direct3d12/directx-12-programming-guide. EXTERNAL
A. Adeyemi-Ejeye, M. Alreshoodi, L. Al-Jobouri, and M. Fleury, “Prospects for live higher resolution video streaming to mobile devices: achievable quality across wireless links,” Journal of Real-Time Image Processing 16, pp. 127–141, 2019. EXTERNAL
S. Gül, D. Podborski, T. Buchholz, T. Schierl, and C. Hellge, “Low-latency cloud-based volumetric video streaming using head motion prediction,” in Proc. of the 30th ACM Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 27–33, Jun. 2020. EXTERNAL
G. Baig, J. He, M. Adnan Qureshi, L. Qiu, G. Chen, P. Chen, and Y. Hu, “Jigsaw: Robust live 4k video streaming,” in Proc. 25th Annual International Conference on Mobile Computing and Networking, pp. 1–16, 2019. EXTERNAL
G. Grossi, P. Paglierani, F. Pedersini, and A. Petrini, “Enhanced multicore-manycore interaction in high performance video encoding,” J. Real-Time Image Processing, 17 (4): pp. 887–902, 2020. EXTERNAL
Google chromium, “libyuv: open source YUV scaling and conversion,” 2023. Accessed 24 Apr 2023. [Online]. Available: https://chromium.googlesource.com/libyuv/libyuv/. EXTERNAL
VideoLan, “VLC media player, 2006. Accessed 18 May 2023. [Online]. Available: https://www.videolan.org/vlc/. EXTERNAL
Open Effects Association, “OpenFX Image Processing Standard,” 2022. Accessed 18 May 2023. [Online]. Available: https://openeffects.org/. EXTERNAL
S. Mori, R. Ichikari, F. Shibata, A. Kimura, and H. Tamura, “Enabling on-set stereoscopic MR-based previsualization for 3D filmmaking,” in Proc. SIGGRAPH Asia 2011 Sketches, pp. 1–2, Dec. 2011. EXTERNAL
R. S. Overbeck, D. Erickson, D. Evangelakos, M. Pharr, and P. Debevec, “A system for acquiring, processing, and rendering panoramic light field stills for virtual reality,” ACM Transactions on Graphics (TOG) 37 (6): pp. 1–15, 2018. EXTERNAL
O. Mossad, K. Diab, I. Amer, and M. Hefeeda, “Deepgame: Efficient video encoding for cloud gaming,” in Proc. of the 29th ACM International Conference on Multimedia, pp. 1387–1395, Oct. 2021. [Online]. Available: https://doi.org/10.1145/3474085.3475594. EXTERNAL
NewTek, “NDI 5 SDK Documentation.” Accessed 24-Apr-2023. [Online]. Available: https://ndi.video/. EXTERNAL
Evercast/pixelweave. [Online]. Available: https://github.com/evercast/pixelweave. EXTERNAL
J. Unterguggenberger, B. Kerbl, and M. Wimmer, “Vulkan all the way: Transitioning to a modern low-level graphics API in academia,” Computers &amp Graphics, Vol. 111, pp. 155–165, Apr. 2023, [Online]. Available: https://doi.org/10.1016/j.cag.2023.02.001. EXTERNAL
S. Aaltonen, “Optimizing GPU occupancy and resource usage with large thread groups,” AMD GPUopen, May 2017. Accessed 16 May 2023. [Online]. Available: https://gpuopen.com/learn/optimizinggpu-occupancy-resource-us-age-Iarge-thread-groups/. EXTERNAL
Blender Studio, “Blender Open Movies,” 2013. [Online]. Available: https://studio.blender.org/films/. EXTERNAL
A. Antsiferova, S. Zvezdakov, D. Kondranin, D. Kulikov, D. Vatolin, and M. Erofeev, “HEVC Video Codecs Comparison 2018 (Part II: FullHD Content, Subjective Evaluation),” Nov. 2018. [Online]. Available: https://doi.org/10.13140/RG.2.2.34856.32005. EXTERNAL
SMPTE ST 268:2003 [Superseded]

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2024-YPJG9446.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Jose Aguerre, Bruno Sena, and Damien Stolarz; Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming, MIJ 2024, Volume 133, Number 5 (pp. 60 to 70); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/YPJG9446

Snippet:

Jose Aguerre, Bruno Sena, and Damien Stolarz; Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming, MIJ 2024, Volume 133, Number 5 (pp. 60 to 70); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/YPJG9446

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Jose Aguerre, Bruno Sena, and Damien Stolarz; <cite>Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming</cite>, MIJ 2024, Volume 133, Number 5 (pp. 60 to 70); SMPTE, 2024. Available at <a href="https://doi.org/10.5594/JMI.2024/YPJG9446" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2024/YPJG9446</a></span>

SMPTE's HTML Pub

Preview:

Jose Aguerre, Bruno Sena, and Damien Stolarz; Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming, MIJ 2024, Volume 133, Number 5 (pp. 60 to 70); SMPTE, 2024
doi: 10.5594/JMI.2024/YPJG9446
url: https://doi.org/10.5594/JMI.2024/YPJG9446

Snippet:

<li>
Jose Aguerre, Bruno Sena, and Damien Stolarz; <cite id="bib-10-5594-jmi-2024-ypjg9446">Using GPU-Accelerated Pixel Format Conversions for Efficient Real-Time Video Streaming</cite>, MIJ 2024, Volume 133, Number 5 (pp. 60 to 70); SMPTE, 2024
<span class="doi">10.5594/JMI.2024/YPJG9446</span>
</li>