Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 131, No. 7, pp. 6–16
Abstract: In this article, we focus on the machine-learning approach developed for automatic audio source recognition and mixing for the U.K. Government Department of Culture Media and Sport (DCMS) funded collaborative project called 5G Edge-XR. Leveraging graphics processing unit (GPU) acceleration, we deployed innovative algorithms in the cloud so that content can be automatically mixed on-the-fly for a personalized, immersive, and interactive experience for audiences. We describe the algorithms involved, the system architecture, how it has been implemented for immersive live boxing, and also how we are using it to enhance a live in-stadium experience.
Publication Date: 2022-08-01
DOI: 10.5594/JMI.2022.3184849
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2022.3184849
Author(s): Rob G. Oldfield
bio
Rob G. Oldfield is an engineer, a researcher, and an entrepreneur, in Manchester, U.K. He is the co-founder of Salsa Sound Ltd., Manchester, which developed a patented AI engine that automates and enhances audio mixing for live sports broadcasts. Salsa’s technology improves mix quality, consistency, and production efficiency and enables the production of content that helps viewers fully connect with the action on the field and the crowd in the stands. He has been working in the audio/broadcast industry for more than 15 years. He received a PhD degree in object-based and immersive audio from the University of Salford, Salford, U.K., in 2013.
Max S. S. Walley
bio
Max S. S. Walley is the lead software developer and researcher with, Salsa Sound Ltd., Manchester, U.K. He received first-class honors in broadcast audio and music technology from the University of the West of England, Bristol, U.K. His research interests include audio analysis, AI in large-scale audio applications, cloud computing, hardware emulation, and XR game development.
Ben G. Shirley
bio
Ben G. Shirley is the co-founder and director of Salsa Sound Ltd., Manchester, U.K., which develops live sports audio broadcast solutions. He is a senior lecturer in audio technology with the University of Salford, Salford, U.K., where he carries out research in object-based audio personalization and broadcast accessibility. He received an MSc degree from Keele University, Keele, U.K., in 2000, and a PhD degree from the University of Salford in 2013.
Doug L. Williams
bio
Doug L. Williams has worked with BT Applied Research, Ipswich, U.K., for more than 30 years. For the past 20 years, he has focused on developing and running collaborative projects that focus on enabling media experiences that use the affordances of IP delivery. He was privileged to coordinate the interdisciplinary teams working on the U.K. Government Department of Culture Media and Sport (DCMS) funded 5G Edge-XR project, the results of which are reported in this article. 5G Edge-XR explored how GPU-based edge compute and 5G technologies can enable immersive experiences that address sports (of immediate relevance to BT) and applications across a range of other industries. Prior to this, he led three major H2020 projects, all looking at new experiences enabled by emerging technology capabilities. These projects explored reconfigurable narrative, multicamera viewpoint video conferencing, the mixing of games with communication, and personalized TV using object-based media.
Keyword(s): 5G, Artificial Intelligence, Audio, Augmented Reality, Automated Production, Broadcast, Extended Reality, Immersive Audio, Machine Learning, Object-based Audio, Personalization
Copyright: © 2022 SMPTE

Bibliographic Reference(s)

R. Oldfield, B. Shirley, and J. Spille, “Object-Based Audio for Interactive Football Broadcast,” Multimed. Tools Appl., 74(8):2717– 2741, 2015. EXTERNAL
B. Shirley et al., “Platform Independent Audio,” in Media Production, Delivery and Interaction for Platform Independent Systems: Format-Agnostic Media, John Wiley & Sons, Inc.: Hoboken, NJ, pp. 130–165, 2013. EXTERNAL
Accessed: July 1, 2022. [Online]. Available: https://www.5gedgexr.com EXTERNAL
M. Mann et al., “Object-Based Audio Applied to Football Broadcasts,” Proc. of the 2013 ACM Int. Workshop on Immersive Media Experiences, pp. 13–16, 2013. EXTERNAL
M. A. Gerzon, “Ambisonics in Multichannel Broadcasting and Video,” J. Audio Eng. Soc., 33(11):859–871, Nov. 1985. EXTERNAL
C. Knapp and G. Carter, “The Generalized Correlation Method for Estimation of Time Delay,” IEEE Trans. Acoust. Speech Signal Process., 24(4): 320–327, Aug. 1976. EXTERNAL
S. Bertet et al., “Investigation on Localisation Accuracy for First & Higher Order Ambisonics Reproduced Sound Sources,” Acta Acustica United Ac., 99(4):642–657, Jul. 2013. EXTERNAL
M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” 2015. Accessed: 1st July 2022. [Online]. Available: https://www.tensorflow.org EXTERNAL
European Broadcasting Union (EBU) R 128, “Loudness Normalisation and Permitted Maximum Level of Audio Signals,” European Broadcasting Union, Geneva, Jun. 2014. EXTERNAL
ITU-R BS.1770-4 [Active]
R-REC-BS.2076-2 — International Telecommunication Union-Radiocommunication (ITU-R) Recommendation BS.2076-2, “Audio Definition Model,” Oct. 2019. EXTERNAL
R-REC-BS.2125-0 — International Telecommunication Union-Radiocommunication (ITU-R) Recommendation BS 2125-0, “A Serial Representation of the Audio Definition Model,” Jan. 2019. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3184849.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Rob G. Oldfield, Max S. S. Walley, Ben G. Shirley, and Doug L. Williams; Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 7, August 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3184849

Snippet:

Rob G. Oldfield, Max S. S. Walley, Ben G. Shirley, and Doug L. Williams; Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 7, August 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3184849

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Rob G. Oldfield, Max S. S. Walley, Ben G. Shirley, and Doug L. Williams; <cite>Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 7, August 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3184849" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3184849</a></span>

SMPTE's HTML Pub

Preview:

Rob G. Oldfield, Max S. S. Walley, Ben G. Shirley, and Doug L. Williams; Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 7, August 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3184849
url: https://doi.org/10.5594/JMI.2022.3184849

Snippet:

<li>
Rob G. Oldfield, Max S. S. Walley, Ben G. Shirley, and Doug L. Williams; <cite id="bib-10-5594-jmi-2022-3184849">Cloud-Based AI for Automatic Audio Production for Personalized Immersive XR Experiences</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 7, August 2022); SMPTE, 2022
<span class="doi">10.5594/JMI.2022.3184849</span>
</li>