Exploring Automated Voice Casting for Content Localization Using Deep Learning

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 3, pp. 12–18
Abstract: Casting voice-actors to dub source language content into a target language—known as voice casting—consists largely of a manual workflow that could benefit immensely from increased levels of automation. Recent advancements in deep learning architectures for sequential data processing are providing the needed impetus to the realization of various AI-enabled audio-processing workflows. Specifically, applications such as speaker verification and speech synthesis have been gaining immense traction due to the advent and maturity of recurrent neural networks. We explore the viability of leveraging advancements in deep learning for text-independent speaker verification (TI-SV) for use in computer-aided voice casting. To this end, we propose and develop an automated voice-casting tool that uses similarity scores generated from neural network embeddings—from a robust autoencoder model trained for the task of TI-SV—to rank voiceover artists across different languages in voice-casting process. To evaluate the dexterity of the proposed approach, we conduct a subjective study emulating a simplified voice-casting process on actual voice-testing kits (dubbing auditions) from our content. We also use casting decisions from casting experts to further evaluate the tool as well as the subjectivity involved in the voice-casting process. We achieve promising results for the automated tool and prove that it could be a viable approach to automating the voice-casting process and warrants further exploration.
Publication Date: 2021-04-01
DOI: 10.5594/JMI.2021.3057695
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2021.3057695
Author(s): Aansh Malik
bio
Aansh Malik is an AI engineer in the Emerging and Creative Technologies Team at WarnerMedia. His work focuses on leveraging advancements in AI to enable more interactive storytelling, facilitate experiences in immersive mediums such as AR/VR, and augmenting current workflows in the media production lifecycle. His projects are a confluence of fields including reinforcement learning in game engines, computer vision, recurrent neural networks for audio processing, among others. He has a BS degree in electrical engineering with a specialization in machine learning from the University of California at San Diego (UCSD), La Jolla, CA.
Ha Nguyen
bio
Ha Nguyen is the director of emerging technology at WarnerMedia’s Emerging and Creative Technologies, Burbank, California. With a background in computer science, she has been with Warner Bros., now WarnerMedia, for 16 years. Starting out as a software engineer for mobile, back-end operations, and web, she ventured into digital and physical media, its application, post-production, and distribution services. She then switched gears to work in the emerging technology arena, where she helped to launch all WarnerMedia titles in ultra-high-definition media format. Nguyen brings mixed reality, artificial intelligence, and future storytelling concepts to the studio, and participates in research and development work for next-generation media content creation.
Keyword(s): Artificial Intelligence, Audio Processing, Automation, Content Localization, Deep Learning, Dubbing, Machine Learning, Neural Network, Voice Casting, Voice Similarity
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

Y. Jia et al. (Jan. 2, 2019). Transfer Learning from Speaker Verification to Multispeaker Text-to-Speech Synthesis. Retrieved Jul. 9, 2020, from https://arxiv.org/abs/1806.04558 EXTERNAL
L. Deng et al. “Binary Coding of Speech Spectrograms Using a Deep Auto-Encoder,” Proc. INTERSPEECH-2010, pp. 1692–1695, 2010. EXTERNAL
A. Gresse, M. Rouvier, R. Dufour, V. Labatut, and J.-F. Bonastre, “Acoustic Pairing of Original and Dubbed Voices in the Context of Video Game Localization,” Proc. Interspeech, Stockholm, Sweden, pp. 2839–2843, 2017, doi: 10.21437/Interspeech.2017-1311. hal-01572151. EXTERNAL
E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez, “Deep Neural Networks for Small Footprint Text-Dependent Speaker Verification,” Proc. IEEE Int. Conf. Acoustics Speech Signal Process. (ICASSP ), Florence, Italy, pp. 4052–4056, 2014, doi: 10.1109/ICASSP.2014.6854363. EXTERNAL
N. Obin, A. Roebel, and G. Bachman, “On Automatic Voice Casting for Expressive Speech: Speaker Recognition vs. Speech Classification,” Proc. IEEE Int. Conf. Acoustics Speech Signal Process, (ICASSP ), Florence, Italy, pp. 950–954, 2014, doi: 10.1109/ EXTERNAL
N. Obin and A. Roebel, “Similarity Search of Acted Voices for Automatic Voice Casting,” IEEE/ACM Trans. Audio Speech Language Process. 24(9):1642–1651, Sep. 2016, doi: 10.1109/ EXTERNAL
W. M. Campbell, D. E. Sturim, and D. A. Reynolds, “Support Vector Machines Using GMM Supervectors for Speaker Verification,” IEEE Signal Process. Lett. 13(5):308–311, May 2006, doi: 10.1109/LSP.2006.870086. EXTERNAL
A. J. Scott and G. A. F. Seber, “Difference of Proportions From the Same Survey,” Am. Stat., 37(4a):319–320, 1983, doi: 10.1080/00031305.1983.10483130. EXTERNAL
L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized End-to-End Loss for Speaker Verification,” Proc. IEEE Int. Conf. Acoustics Speech Signal Process. (ICASSP ), pp. 4879–4883, 2018, doi: 10.1109/icassp.2018.8462665. EXTERNAL
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani, “VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop,” Proc. Int. Conf. Learning Representations (ICLR), 2018. EXTERNAL
D. Snyder, P. Ghahremani, D. Povey, D. Garcia-Romero, Y. Carmiel, and S. Khudanpur, “Deep Neural Network-Based Speaker Embeddings for End-to-End Speaker Verification,” Proc. IEEE Spoken Language Technol. Workshop (SLT), San Diego, CA, pp. 165–170, 2016, doi: 10.1109/SLT.2016.7846260. EXTERNAL
D. A. Reynolds, T. F. Quatieri, and R. B. Dunn, “Speaker Verification Using Adapted Gaussian Mixture Models,” Digit. Signal Process., 10:19–41, Jan. 2000. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2021.3057695.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Aansh Malik and Ha Nguyen; Exploring Automated Voice Casting for Content Localization Using Deep Learning, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3057695

Snippet:

Aansh Malik and Ha Nguyen; Exploring Automated Voice Casting for Content Localization Using Deep Learning, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3057695

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Aansh Malik and Ha Nguyen; <cite>Exploring Automated Voice Casting for Content Localization Using Deep Learning</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2021.3057695" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2021.3057695</a></span>

SMPTE's HTML Pub

Preview:

Aansh Malik and Ha Nguyen; Exploring Automated Voice Casting for Content Localization Using Deep Learning, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
doi: 10.5594/JMI.2021.3057695
url: https://doi.org/10.5594/JMI.2021.3057695

Snippet:

<li>
Aansh Malik and Ha Nguyen; <cite id="bib-10-5594-jmi-2021-3057695">Exploring Automated Voice Casting for Content Localization Using Deep Learning</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2021.3057695</span>
</li>