“AI News Anchor” With Deep Learning-Based Speech Synthesis

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 3, pp. 19–27
Abstract: Deep learning-based text to speech (TTS) is used in various situations, and the sound quality is close to that of humans. We previously developed a news-specific deep learning-based TTS (DL-TTS) system and implemented it with our AI news anchor for live broadcast programs and automatic news-speech distribution services. We also developed our DL-TTS system for the control of speaking style and speech rate, pitch, intonation, and volume to facilitate the creation of various programs. More specifically, this method enables the changing of specific speaking styles, such as news style, which mimics the style of news reporters, and conversation style. The purpose of creating this system was to eliminate the discomfort due to differences in speech and speaking styles. Controlling speaking style is important in news speech because a mismatched speaking style does not appropriately convey news articles. For this study, we conducted an evaluation experiment on the conveying of simple news articles for language learners regarding speaking-style control and found appropriate speaking styles for automatically generated news speech. We conducted another evaluation experiment on whether synthetic speech generated from our system for “easy news” for Japanese language learners can help people understand the news in Japanese. We also discuss practical applications of our system. Our news-specific deep neural network-based TTS system was found to effectively provide news services to broadcast stations. In the future, we will consider various use cases of flexible production by using a cloud system. The coronavirus pandemic has forced broadcasters to adopt new working styles. Thus, we will explore a new production system, such as a cloud-based system, for news-speech automation for this new normal.
Publication Date: 2021-04-01
DOI: 10.5594/JMI.2021.3057703
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2021.3057703
Author(s): Kiyoshi Kurihara
bio
Kiyoshi Kurihara received BE and ME degrees in electronics and communications from Meiji University, Tokyo, Japan, in 2004 and 2006, respectively. After his graduation, he joined NHK (Japan Broadcasting Corporation), Tokyo, where he was a master control and radio studio engineer in a broadcasting station. He joined the Science and Technology Research Laboratories (STRL) in 2016. He joined the Development and Strategy Center in 2020. His current interest includes speech synthesis for broadcast.
Nobumasa Seiyama
bio
Nobumasa Seiyama received BE and ME degrees in electrical engineering from Waseda University, Tokyo, Japan, in 1987 and 1989, respectively. He joined NHK in 1989. Since then, he has worked with the Science and Technology Research Laboratories, engaging in speech signal processing, voice conversion, speech synthesis, and development of a high-quality speech rate conversion system for the elderly. He was with NHK Engineering Services, Inc., from 2007 to 2010.
Tadashi Kumano
bio
Tadashi Kumano is a principal research engineer with the Science and Technology Research Laboratories NHK. He joined NHK STRL after getting his master’s degree in computer science from the Graduate School of Information Science and Engi?neering, Tokyo Institute of Technology, Tokyo, in 1995. He was also a member of the Advanced Telecommunication Research (ATR) Institute International Spoken Language Translation Laboratories from 2000 to 2004. He has been engaged in the research of natural language processing and machine translation, and recently became interested in spoken language generation such as computer-producing voice commentaries for sports programs.
Takashi Fukaya
bio
Takashi Fukaya is a senior manager with the Development and Strategy Center News Department. After graduating from Kyushu Institute of Design, Fukuoka, Japan, he joined NHK in 1992. While with the NHK Science and Technical Research Laboratories, he conducted research on human interaction and released “Invisible Light” and “Virtual Scope.” He has also collaborated with media artist Toshio IWAI to present “Morphovision” at SIGGRAPH 2006 and ARS Electronica 2006. Currently, he has been engaged in NHK News website and application development work. He received a PhD degree in design from the Kyushu Institute of Design.
Kazunari Saito
bio
Kazunari Saito is the technical director of the Development and Strategy Center News Department. After graduating from Keio University, Tokyo, Japan, he joined NHK in 2002. After working on data broadcasting and interactive con?tent development, he focused on the NHK News website and application development. He is currently active as a key member of the cloud infrastructure design team.
Satoshi Suzuki
bio
Satoshi Suzuki is a CG super?visor, an expert in the field of CG for over 20 years. He joined NHK in 1997 and worked at their Osaka Broadcasting Station. Since 2001, he has been with the Broadcast Engineering department, where he has been engaged in developing NHK election systems and CG production systems. More recently, he has been working as a CG supervisor mainly to develop the “big data visualization system.”
Keyword(s): AI News Anchor, Automatic Generated News Speech, Text-to-Speech
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

Automatic Generation of Audio Descriptions for Sports Programs [Active]
A. Imai, N. Tazawa, T. Takagi, T. Tanaka, and T. Ifukube, “A New Touchscreen Application to Retrieve Speech Information Efficiently,” IEEE Trans. Consumer Electronics, 59(1):200–206, Apr. 2013, doi: 10.1109/TCE.2013.6490260. EXTERNAL
Unity. Accessed: Feb. 24, 2021. [Online]. Available: https://unity.com/ EXTERNAL
H. Segi, R. Takou, N. Seiyama, T. Takagi, H. Saito, and S. Ozawa, “Template-Based Methods for Sentence Generation and Speech Synthesis,” Proc. IEEE Int. Conf. Acoustics, Speech Signal Process. (ICASSP), pp. 1757–1760, 2011, doi: 10.1109/ EXTERNAL
J. Shen et al., “Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions,” Proc. IEEE Int. Conf. Acoustics, Speech Signal Processing (ICASSP), pp. 4779–4783, 2018, doi: 10.1109/ICASSP.2018.8461368. EXTERNAL
H. Zen and A. Senior, “Deep Mixture Density Networks for Acoustic Modeling in Statistical Parametric Speech Synthesis,” Proc. IEEE Int. Conf. Acoustics, Speech Signal Processing (ICASSP), pp. 3844–3848, 2014, doi: 10.1109/ICASSP.2014.6854321. EXTERNAL
M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications,” IEICE Trans. Inf. Syst., E99D(7): 1877–1884, Jul. 2016, doi: 10.1587/transinf.2015EDP7457. EXTERNAL
NHK. “On the Second-Generation NHK Speech Synthesis System.” Accessed: Feb. 24, 2021. [Online]. Available: https://www.nhk.or.jp/corporateinfo/english/comment_top/execu-tive/2020/2010.html EXTERNAL
Sogou, “Sogou Introduces World’s First 3D AI News Anchor,” Accessed: Feb. 24, 2021. [Online]. Available: http://ir.sogou.com/2020-05-21-Sogou-Introduces-Worlds-First-3D-AI-News-Anchor EXTERNAL
K. Kurihara, N. Seiyama, A. Imai, and T. Takagi, “The DNN-Based Speech Synthesis Using Speaker Codes and Emotional Codes (in Japanese),” Proc. 2017 IEICE General Conf., p. 150, 2017. EXTERNAL
H. Segi et al., “An Automatic Broadcast System for a Weather Report Radio Program,” IEEE Trans. Broadcast., 59(3):548–555, Sep. 2013, doi: 10.1109/TBC.2013.2272406. EXTERNAL
Z. Wu, O. Watts, and S. King, “Merlin: An Open Source Neural Network Speech Synthesis System,” Proc. 9th ISCA Speech Synthesis Workshop (SSW9), pp. 218–223, 2016. EXTERNAL
NHK, “NHK News Web Easy,” Accessed: Feb. 24, 2021. [Online]. Available: https://www3.nhk.or.jp/nhkworld/en/learnjapanese/audionews/ EXTERNAL
D. F. Glas, T. Minato, C. T. Ishi, T. Kawahara, and H. Ishiguro, “ERICA: The ERATO Intelligent Conversational Android,” Proc. 25th IEEE Int. Symp. Robot and Human Interactive Commun., pp. 22–29, Nov. 2016, doi: 10.1109/ROMAN.2016. 7745086. EXTERNAL
K. Kurihara, N. Seiyama, T. Kumano, and A. Imai, “Study on Switching Method of Speaking Styles in Japanese End-to-End Speech Synthesis (in Japanese),” Proc. Inst. Image Inf. Television Engineers Winter Annual Convention (ITE), 2018. EXTERNAL
“Japanese Language for Broadcasting,” NHK Broadcasting Culture Research Institute. Accessed: Feb. 24, 2021. [Online]. Available: https://www.nhk.or.jp/bunken/english/research/kotoba/index.html EXTERNAL
Google, “Cloud Text-to-Speech.” Accessed: Feb. 24, 2021. [Online]. Available: https://cloud.google.com/text-to-speech EXTERNAL
Netflix, “Recording Dubbed Dialogue in a Home Environment.” Accessed: Jun. 11, 2020. [Online]. Available: https://partnerhelp.netflixstudios.com/hc/en-us/articles/360047052733-Recording-Dubbed-Dialogue-In-A-Home-Environment EXTERNAL
NHK’s New Dictionary of Japanese Pronunciation and Accentuation. Tokyo: NHK Broadcasting Culture Research Institute (in Japanese), 2016. EXTERNAL
K. Kurihara, N. Seiyama, and T. Kumano, “Prosodic Features Control by Symbols as Input of Sequence-to-Sequence Acoustic Modeling for Neural TTS,” IEICE Trans. Inf. Syst., E104-D(2):302-311, Feb. 2021, doi: 10.1587/transinf.2020EDP7104. EXTERNAL
NHK STRL, “Automated Production of Weather Information Radio Programs.” Accessed: Feb. 24, 2021. [Online]. Available: https://www.nhk.or.jp/strl/open2019/tenji/17_e.html EXTERNAL
H. Segi, R. Takou, N. Seiyama, and T. Takagi, “Development of a Prototype Data-Broadcast Receiver with a High-Quality Voice Synthesizer,” IEEE Trans. Consumer Electron., 56(1):169–174, Mar. 2010, doi: 10.1109/TCE.2010.5439141. EXTERNAL
NHK, “News Yomiko (in Japanese).” Accessed: Feb. 24, 2021. [Online]. Available: https://www.nhk.or.jp/voice/yomiko/ EXTERNAL
H. Nakajima and T. Yamada, “Innovation of Techniques Identifying Error (Proofreading Support) Device,” Nikkei Innovation Lab. Accessed: Feb. 24, 2021. [Online]. Available: EXTERNAL
A. v. d. Oord et al., “WaveNet: A Generative Model for Raw Audio,” arXiv preprint arXiv:1609.03499, 2016. EXTERNAL
NHK, “AI Announcer Has Made a Debut on “News Check 11” Program,” Accessed: Jun. 11, 2018. [Online]. Available: https://www.nhk.or.jp/strl/publica/bt/bt74/pdf/nt-editors0074.pdf EXTERNAL
Vizrt. Accessed: Feb. 24, 2021. [Online]. Available: https://www.vizrt.com/products/viz-virtual-studio EXTERNAL
The Mainichi Shimbun, “Public Broadcaster’s AI Newsreader to Make Senryu Poetry Debut.” Accessed: Feb. 24, 2021. [Online]. Available: https://mainichi.jp/english/articles/20190313/p2a/00m/0et/014000c EXTERNAL
Amazon, “Polly.” Accessed: Feb. 24, 2021. [Online]. Available: https://aws.amazon.com/polly/automatic generation of advanced description such as TTS input of TTS is also challenging. For more flexible program production, the automatic generation of new speaking styles and descriptions from past programs and multimodal support are topics for further study. We are working on applying our DNN-TTS system to a variety of applications. For example, we created automated audio descriptions for the visually impaired.3 We plan to use an automatic news reader that uses our system for local radio.29 When it is a challenge to arrange an anchor for emergency news at off-peak hours, it may be possible to solve such problems with our system. For the visually impaired, a service that automatically reads aloud captions of breaking news is needed. Moreover, by developing a controllable TTS system, these in conjunction with more new services using TTS deserve further study. Conclusion We described various applications for our DNN-TTS system with regard to automated services and a TV-studio program for AI news anchor and we experimentally demonstrated that synthesized speech with a slow rate is much better than standard-speed speech synthesis for language learners. We also found that our system contributes not only to production automation, but also universal services. Our system can not only replace existing broadcasting services, but also generate new services. Speech synthesis is a promising technology for broadcasting services but there is room for improvement. TTS has made it possible to automate many broadcasting services, for example, providing the latest news through synthesized speech. The benefits of operating TTS for broadcasters are not only directly related to their services, but also the ease of acquiring training data for deep learning-based TTS. Collecting data for TTS is very expensive, but archives of TV programs can easily be used for training data. We hope that broadcasters will use TTS for their programs to create interesting content and improve business efficiency. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2021.3057703.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Kiyoshi Kurihara, Nobumasa Seiyama, Tadashi Kumano, Takashi Fukaya, Kazunari Saito, and Satoshi Suzuki; “AI News Anchor” With Deep Learning-Based Speech Synthesis, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3057703

Snippet:

Kiyoshi Kurihara, Nobumasa Seiyama, Tadashi Kumano, Takashi Fukaya, Kazunari Saito, and Satoshi Suzuki; “AI News Anchor” With Deep Learning-Based Speech Synthesis, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3057703

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Kiyoshi Kurihara, Nobumasa Seiyama, Tadashi Kumano, Takashi Fukaya, Kazunari Saito, and Satoshi Suzuki; <cite>“AI News Anchor” With Deep Learning-Based Speech Synthesis</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2021.3057703" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2021.3057703</a></span>

SMPTE's HTML Pub

Preview:

Kiyoshi Kurihara, Nobumasa Seiyama, Tadashi Kumano, Takashi Fukaya, Kazunari Saito, and Satoshi Suzuki; “AI News Anchor” With Deep Learning-Based Speech Synthesis, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
doi: 10.5594/JMI.2021.3057703
url: https://doi.org/10.5594/JMI.2021.3057703

Snippet:

<li>
Kiyoshi Kurihara, Nobumasa Seiyama, Tadashi Kumano, Takashi Fukaya, Kazunari Saito, and Satoshi Suzuki; <cite id="bib-10-5594-jmi-2021-3057703">“AI News Anchor” With Deep Learning-Based Speech Synthesis</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2021.3057703</span>
</li>