Exploring Realtime Conversational Virtual Characters
Metadata
- Publisher
- SMPTE — White Plains, NY, USA
- Doc Type
- Journal Article
- Content Type
- Original Research
- Abbreviated Title
- SMPTE Mot. Imag. J
- Volume
- 131, No. 3, pp. 25–34
- Abstract
- Advancements in artificial intelligence (AI) such as speech to text (STT), language understanding models, language generation models, and text to speech (TTS) enable various types of applications, one of which is realtime conversational virtual characters. Building an end-to-end framework with the right AI technology components enables relatable and multidimensional virtual characters, who can naturally converse in creatively controlled domains, while consistently maintaining their state and personality in predetermined narratives. In this work, we designed such a conversational framework with interchangeable and loosely coupled components to support granular creative details in character performance, efficiency in mass creation of virtual characters, and flexibility to embrace future improvements of each component in the fields. We then evaluated the robustness and modularity of the framework by creating Melodie, a virtual character who is fond of music and is a fan and promoter of the Eurovision Song Contest. With Melodie, we went through the full cycle from processing a speaker’s audio signals, to generating a proper response using a natural language generation model, to synthesizing the response in a character’s voice font, to finally synchronizing the synthesized response with corresponding body and facial movements to produce a coherent and believable character performance. Testing and analyzing the implementation of Melodie brought forth areas of improvement and ethical considerations that are, and continue to be, essential to the design of our future applications involving virtual characters.
- Publication Date
- 2022-04-01
- DOI
10.5594/JMI.2022.3153646- ISSN
- Print:
1545-0279| Electronic:2160-2492 - Link
- https://doi.org/10.5594/JMI.2022.3153646
- Author(s)
- Ha Nguyen
bio
Ha Nguyen is the director of emerging technology at WarnerMedia’s Emerging and Creative Technologies, Burbank, CA. With a background in computer science, she has been with Warner Bros, now WarnerMedia, for 17 years. Starting out as a software engineer for mobile, back-end operations, and web, she ventured into digital and physical media, its application, post-production, and distribution services. She then switched gears to work in the emerging technology arena, where she helped to launch all WarnerMedia titles in ultrahigh-definition media format. At present, she brings mixed reality, artificial intelligence, and future storytelling concepts to the studios and participates in research and development work for next-generation media content creation.Aansh Malikbio
Aansh Malik is an AI engineer on the emerging and creative technologies team at WarnerMedia, Burbank, CA. His work focuses on leveraging advancements in AI to enable more interactive storytelling, facilitate experiences in immersive mediums such as AR/VR, as well as augmenting current workflows in the media production lifecycle. His projects are a confluence of fields including reinforcement learning in game engines, computer vision, recurrent neural networks for audio processing, among others. He received a BS degree in electrical engineering with a specialization in machine learning from the University of California at San Diego, La Jolla, CA.Michael Zinkbio
Michael Zink is the vice president of emerging and creative technologies at WarnerMedia, Burbank, CA, where he is responsible for exploring emerging technologies to enhance WarnerMedia’s storytelling capabilities across different platforms. This includes assessing the latest technologies, developing innovative experiences, and integrating new solutions into creative production workflows. He also participates in several industry associations, serves as the president and chairman for the UHD Alliance and Education, and the Vice President for SMPTE. Prior to joining Warner Bros., in 2014, he worked at Technicolor London, U.K., and Los Angeles, CA, for over ten years, most recently as the vice president of technology strategy, where he was responsible for launching the production efforts around various new distribution formats and adoption of Technicolor technology solutions within industry groups. Earlier in his career, he worked for several media production facilities in Germany. - Keyword(s)
- Artificial Intelligence, Conversational AI, Conversational Characters, Virtual Characters
- Copyright
- © 2022 SMPTE
Bibliographic Reference(s)
- Microsoft, “LUIS (Language Understanding) - Cognitive Services,” Nov. 2021. [Online]. Available: https://www.luis.ai/ EXTERNAL
- OpenAI, “GPT-3 Powers the Next Generation of Apps,” Mar. 2021. [Online]. Available: https://openai.com/blog/gpt-3-apps/ EXTERNAL
- Oculus, “Oculus Lipsync for Unity Development,” Nov. 2021. [Online]. Available: https://developer.oculus.com/documentation/unity/audio-ovrlipsync-unity/ EXTERNAL
- Unity, “About AI Planner,” Nov. 2021. [Online]. Available: https://docs.unity3d.com/Packages/com.unity.ai.planner@0.0/manual/index.html EXTERNAL
- T. B. Brown, et al. “Language Models Are Few-Shot Learners,” arXiv.org. Sep. 2021. [Online]. Available: https://arxiv.org/abs/2005.14165 EXTERNAL
- Microsoft, “About the Speech SDK - Speech Service - Azure Cognitive Services,” Sep. 2021. [Online]. Available: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-sdk EXTERNAL
- S. Gehman et al., “Real Toxicity Prompts: Evaluating Neural Toxic Degeneration in Language Models,” arXiv:2009.11462, Sep. 2020. EXTERNAL
- Resemble.AI. Clone Synthetic AI Voices With Neural Text to Speech. Nov. 2021. [Online]. Available: https://www.resemble.ai/ EXTERNAL
Source Data (JSON)
Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3153646.json
Reference Tree
Explore all references and references to this document, as a navigable tree.
Open Reference TreeReference this Doc
Plain text (ISO 690 compliant)
Preview:
Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3153646
Snippet:
Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3153646
HTML (ISO 690 compliant)
Preview:
Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3153646
Snippet:
<span class="citation">Ha Nguyen, Aansh Malik, and Michael Zink; <cite>Exploring Realtime Conversational Virtual Characters</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3153646" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3153646</a></span>
SMPTE's HTML Pub
Preview:
Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3153646
url: https://doi.org/10.5594/JMI.2022.3153646
doi: 10.5594/JMI.2022.3153646
url: https://doi.org/10.5594/JMI.2022.3153646
Snippet:
<li> Ha Nguyen, Aansh Malik, and Michael Zink; <cite id="bib-10-5594-jmi-2022-3153646">Exploring Realtime Conversational Virtual Characters</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 3, April 2022); SMPTE, 2022 <span class="doi">10.5594/JMI.2022.3153646</span> </li>