Exploring Realtime Conversational Virtual Characters

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2021, No. 11, pp. 1–20
Abstract: Advancements in Artificial Intelligence (AI) such as Speech-To-Text, Language Understanding Models, Language Generation Models, and Text-To-Speech enable various types of applications, one of which is real-time conversational Virtual Characters. Building an end-to-end framework with the right AI technology components enables relatable and multi-dimensional Virtual Characters, who can naturally converse in creatively controlled domains, while consistently maintaining their state and personality in pre-determined narratives. In this work, we designed such a conversational framework with interchangeable, and loosely coupled components to support granular creative details in character performance, efficiency in mass creation of Virtual Characters, and flexibility to embrace future improvements of each component in the fields. We then evaluated the robustness and modularity of the framework by creating Melodie, a Virtual Character who is fond of music, and is a fan and promoter of the Eurovision Song Contest. With Melodie, we went through the full cycle from processing a speaker's audio signals, to generating a proper response using a Natural Language Generation model, to synthesizing the response in a character's Voice Font, to finally synchronizing the synthesized response with corresponding body and facial movements to produce a coherent and believable character performance. Testing and analyzing the implementation of Melodie brought forth areas of improvement and ethical considerations that are, and continue to be, essential to the design of our future applications involving Virtual Characters.
Publication Date: 2021-11-09
DOI: 10.5594/M001944
Link: https://doi.org/10.5594/M001944
Author(s): Ha NguyenWarnerMedia
Aansh MalikWarnerMedia
Michael ZinkWarnerMedia
Keyword(s): Virtual Characters, Conversational AI, Conversational Characters, Artificial Intelligence
Copyright: © 2021 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

OpenAI. (2021, March 25). GPT-3 Powers the Next Generation of Apps. https://openai.com/blog/gpt-3-apps/ EXTERNAL
Microsoft. (n.d.). LUIS (Language Understanding) - Cognitive Services. https://www.luis.ai/ EXTERNAL
S. Gehman et al., “RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models”, arXiv:2009.11462, September 2020 EXTERNAL
Resemble.AI. (n.d.). Clone Synthetic AI Voices with Neural Text to Speech. https://www.resemble.ai/ EXTERNAL
Oculus Lipsync for Unity Development: Unity: Oculus Developers. (n.d.). Retrieved from https://developer.oculus.com/documentation/unity/audio-ovrlipsync-unity/ EXTERNAL
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020, July 22). Language models are few-shot learners. arXiv.org. Retrieved September 19, 2021, from https://arxiv.org/abs/2005.14165. EXTERNAL
Unity. (n.d.). About AI Planner. https://docs.unity3d.com/Packages/com.unity.ai.planner@0.0/manual/index.html EXTERNAL
Microsoft. (2021, September 5). About the Speech SDK - Speech Service - Azure Cognitive Services. https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-sdk EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001944.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Meetings and Conferences ( November 2021); SMPTE, 2021. Available at https://doi.org/10.5594/M001944

Snippet:

Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Meetings and Conferences ( November 2021); SMPTE, 2021. Available at https://doi.org/10.5594/M001944

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Ha Nguyen, Aansh Malik, and Michael Zink; <cite>Exploring Realtime Conversational Virtual Characters</cite>, SMPTE Meetings and Conferences ( November 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/M001944" target="_blank" rel="noopener">https://doi.org/10.5594/M001944</a></span>

SMPTE's HTML Pub

Preview:

Ha Nguyen, Aansh Malik, and Michael Zink; Exploring Realtime Conversational Virtual Characters, SMPTE Meetings and Conferences ( November 2021); SMPTE, 2021
doi: 10.5594/M001944
url: https://doi.org/10.5594/M001944

Snippet:

<li>
Ha Nguyen, Aansh Malik, and Michael Zink; <cite id="bib-10-5594-m001944">Exploring Realtime Conversational Virtual Characters</cite>, SMPTE Meetings and Conferences ( November 2021); SMPTE, 2021
<span class="doi">10.5594/M001944</span>
</li>