Incremental Live Video Processing in IP-Based GPU-Leveraging Software Workflows

Metadata

Publisher: SMPTE — Hollywood, CA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 00, pp. 1–14
Abstract: Live media production is in transition from fixed-function appliances towards general-purpose commercial-off-the-shelf (COTS) hardware in order to take advantage of the economy of scale and flexibility that software-driven media workflows provide. This added flexibility of general-purpose hardware shifts much of the responsibility of meeting a media flow's throughput and latency requirements to the software stack. While it is important for the media function's software to have sufficient performance to meet these requirements, it is also critical to have optimal performance in order to maximize the mutti-tenency potential of media workflows across a computer system. That is, by making efficient use of the COTS system's CPUs, GPUs, memory, buses, and network interface cards (NICs) they become more available to other software media functions running on the system. This increases the amount of media processing per unit hardware (or “density”), and therefore the amount of media processing per dollar - whether this cost is in equipping and maintaining a data center, or in cloud service subscription fees. Recently it has been shown that for IP-based software media workflows running on COTS which deal with uncompressed or mezzanine video streams, the majority of the media processing can be offloaded to GPUs and high-bandwidth NICs. GPUs are suitable for image processing functions, as well as video decoding, and encoding. This frees up CPU resources and accelerates the execution of these operations compared to CPUs. Additionally, NICs with GPU Remote Direct Memory Access (RDMA) can transfer incoming video frames directly into GPU memory - thereby saving a lot of bus traffic (typically PCI bus). Live media is inherently paced and so a software program would need to wait some amount of time to begin processing the incoming media. Currently, IP-based video workflows that ingest uncompressed or mezzanine video streams into a GPU-RDMA-capable NIC will wait for an entire video frame to arrive before doing any image processing and/or encoding on a GPU. We demonstrate a novel approach for incrementally processing a SMPTE-211 0–20 stream as a frame is being received. By taking advantage of SMPTE-2110's reliable pacing and HEVC's and A V1's slice specification, we demonstrate an un compressed media ingest pipeline which then encodes it into either HEVC or A V1. Preliminary results show a substantial decrease in the overall latency, as well as being able to run more streams on a single system.
Publication Date: 2024-10-21
DOI: 10.5594/MOO/3005
ISBN: Electronic: 978-1-61482-965-2
Link: https://doi.org/10.5594/MOO/3005
Author(s): Dennis Sandler
Thomas True
Keyword(s): Software Defined Workflows, Encoding, GPU, NIC, Latency, Stream Density
Copyright: © 2024 SMPTE

Bibliographic Reference(s)

Foundations for New Media Nodes [Active]
G. Sylvester-Bradly, P Sethia, R. Hastie and T. True, “A Platform for the Development and Deployment of Software-Defined Media Processing Pipelines,” Proceedings of the NAB Broadcast Engineering Information Technology Conference, April 2024. EXTERNAL
European Broadcast Union, “The Dynamic Media Facility - Media Value Chain Transformation,” https://tech.ebu.ch/docs/white_papers/White%20Paper%20-%20EBU%20Media%20Factory%20Software%20-%20Media%20Functions.pdf EXTERNAL
SMPTE: “Overview of SMPTE ST 2110 Suite of Standards”, https://www.smpte.org/standards/st2110 EXTERNAL
Haivision, “SRT: Everything You Need to Know About the Secure Reliable Transport Protocol,” https://www.haivision.com/products/srt-secure-reliable-transport/ EXTERNAL
Video Services Forum, “RIST - Reliable Internet Stream Transport,” https://vsf.tv/RIST.shtml EXTERNAL
NDI, https://ndi.video/ EXTERNAL
AMWA, “NMOS Overview”, https://www.amwa.tv/nmos-overview EXTERNAL
Intel, “Intel Xeon Process Scalable Family Technical Overview,” https://www.intel.com/contenUwww/us/en/developer/articies/technical/xeon-processor-scalable-family-technical-overview.html. EXTERNAL
AMD, “Infinity Architecture,” https://www.amd.com/en/technologies/infinity-architecture EXTERNAL
Intel, “Intel Xeo Gold 6448Y Processor,” https://www.intel.com/contenUwww/us/en/products/sku/232384/intel-xeon-gold-6448y-processor-60m-cache-2–10-ghz/specifications.html EXTERNAL
AMD, “4th Generation AMD EPYC Processors,” https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series.html EXTERNAL
NVIDIA, “NVIDIA L40S, Unparalleled AI and Graphics Performance for the Data Center,” https://www.nvidia.com/en-us/data-center/140s/ EXTERNAL
NVIDIA, “ConnectX-7 400G Adapters: Accelerated Networking for Modern Data Center Infrastructures,” https://resources.nvidia.com/en-us-accelerated-networking-resource-library/connectx-7-datasheet EXTERNAL
Kubernetes, https://kubernetes.io/ EXTERNAL
NVIDIA, “Streamlining Kubernetes Networking in Scale-out GPU Clusters with the new NVIDIA Network Operator 1.0,” https://developer.nvidia.com/blog/streamlining-kubernetes-networking-in-scale-out-gpu-clusters-with-the-new-nvidia-network-operator-1–0/ EXTERNAL
Red Hat OpenShift, “Configuring the SR-IOV Operator,” https://docs.openshift.com/container-platform/4.13/networking/hardwarenetworks/configuring-sriov-operator.html EXTERNAL
NVIDIA, “NVIDIA GPU Operator: Simplifying GPU Management in Kubernetes,” https://developer.nvidia.com/blog/nvidia-gpu-operator-simplifying-gpu-management-in-kubernetes/ EXTERNAL
NVIDIA, “Cloud Native Technologies: Time-Slicing GPUs in Kubernetes,” https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/gpu-sharing.html EXTERNAL
Docker, “Docker: Accelerated Container Application Development,” https://www.docker.com/ EXTERNAL
NVIDIA, “DeepStream SDK,” https://developer.nvidia.com/deepstream-sdk EXTERNAL
gstreamer, “What is GStreamer?,” https:/gstreamer.freedesktop.org/ EXTERNAL
NVIDIA, “DeepStream Reference Application - deepstream-nmos app,” https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_ref_app_nmos.html EXTERNAL
NVIDIA, “GXF Overview,” https://docs.nvidia.com/metropolis/deepstream/dev-guide/graphtools-docs/docs/text/GXFOverview.html EXTERNAL
NVIDIA, “NVIDIA Rivermax: Optimized Networking SDK for Media and Data Streaming Applications,” https://developer.nvidia.com/networking/Rivermax EXTERNAL
FFmpeg, “About FFmpeg,” https://www.ffmpeg.org/about.html EXTERNAL
NVIDIA, “NVIDIA Video Codec SDK,” https://developer.nvidia.com/video-codec-sdk EXTERNAL
NVIDIA, “PyNvVideoCodec Overview,” https://docs.nvidia.com/video-technologies/pynvvideocodec/overview.html EXTERNAL
NVIDIA, “CUDA Toolkit,” https://developer.nvidia.com/cuda-toolkit EXTERNAL
True, T, Reid, A and Jones, J, “Direct to GPU Transfers”, Proceedings of the 2011 SMPTE Fall Technical Conference. EXTERNAL
International Telecommunication Union, “H.264 SERIES H: Audiovisual and Multimedia Systems Infrastructure of audiovisual services - Coding of moving video, Advanced video coding for generic audiovisual services” H.264: Advanced video coding for generic audiovisual services (itu.int). EXTERNAL
International Telecommunication Union, “H.265 SERIES H: Audiovisual and Multimedia Systems Infrastructure of audiovisual services - Coding of moving video, High efficiency video coding” H.265: High efficiency video coding (itu.int). EXTERNAL
Alliance for Open Media, “AV1 Bitstream & Decoding Process Specification”, AV1 Bitstream & Decoding Process Specification (aomediacodec.github.io). EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-MOO-3005.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Dennis Sandler and Thomas True; Incremental Live Video Processing in IP-Based GPU-Leveraging Software Workflows, MTS 2024, Article 5 (pp. 1 to 14); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3005

Snippet:

Dennis Sandler and Thomas True; Incremental Live Video Processing in IP-Based GPU-Leveraging Software Workflows, MTS 2024, Article 5 (pp. 1 to 14); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3005

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Dennis Sandler and Thomas True; <cite>Incremental Live Video Processing in IP-Based GPU-Leveraging Software Workflows</cite>, MTS 2024, Article 5 (pp. 1 to 14); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at <a href="https://doi.org/10.5594/MOO/3005" target="_blank" rel="noopener">https://doi.org/10.5594/MOO/3005</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Dennis Sandler and Thomas True; <cite id="bib-10-5594-moo-3005">Incremental Live Video Processing in IP-Based GPU-Leveraging Software Workflows</cite>, MTS 2024, Article 5 (pp. 1 to 14); SMPTE, 2024, ISBN: 978-1-61482-965-2
<span class="doi">10.5594/MOO/3005</span>
</li>