Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools

Metadata

Publisher: SMPTE
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Motion Imaging Jour.
Volume: 133, No. 2, pp. 48–57
Abstract: This research offers an in-depth review of current Automatic Speech Recognition (ASR) methods and their significant impact on media production, with a focus on the transformer model's self-attention mechanism for understanding sequential relationships. It compares accuracy and performance of top ASR models like Meta's Multilingual Machine Speech, OpenAI's Whisper, and Google's Universal Speech Model along with services from Microsoft Azure, Amazon Web Services, and Google Cloud Platform. The study examines key ASR aspects, including voice activity detection, language identification, and multilanguage support, and evaluates their accuracy metrics. Challenges such as limited data for certain languages and complexities in linguistic nuances are highlighted. Additionally, the paper discusses ASR's role in media production, from creating time-based captions to transforming editing techniques. By analyzing the ASR process from audio preprocessing to post-processing, the research bridges academic and practical perspectives, enabling media producers to utilize advanced ASR technologies effectively.
Publication Date: 2024-04-01
DOI: 10.5594/JMI.2024/IPYX8877
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2024/IPYX8877
Author(s): Randy Fayan
Zahra Montajabi
Rob Gonsalves
Keyword(s): Artificial Intelligence, Machine Learning, Automatic Speech Recognition
Copyright: © 2024 SMPTE

Bibliographic Reference(s)

ANSI/CTA-708-E S-2023 [Active]
ISO/IEC 14496-2:2004 [Active, Amended]
A. Radford, “Robust Speech Recognition via Large-Scale Weak Supervision,” ArXiv, abs/2212.04356, 2022. EXTERNAL
V. Pratap,, “M. Scaling Speech Technology to 1,000+ Languages,” ArXiv, abs/2305.13516, 2023. EXTERNAL
Y. Zhang, “Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages,” ArXiv, abs/2303.01037, 2023. EXTERNAL
Park, Tae Jin, Naoyuki Kanda, Dimitrios Dimitriadis, Kyu J. Han, Shinji Watanabe, and Shrikanth Narayanan, “A Review of Speaker Diarization: Recent Advances with Deep Learning,” Comp. Speech & Lang. 72 (2022): pp. 101317, 2022. EXTERNAL
S. S. Stevens, J. Volkmann, and E. B. Newman. “A Scale for the Measurement of the Psychological Magnitude Pitch,” J. of the Acoust. Soc. of America 8(3): pp. 185–190, (1937. EXTERNAL
E. Sejdić, I. Djurović I., J. Jiang (2009). “Time-frequency Feature Representation Using Energy Concentration: An Overview of Recent Advances,” Digital Signal Processing, 19(1): pp. 153–183. doi:10.1016/j.dsp.2007.12.004. EXTERNAL
Google. WebRTC, 2011. Accessed Aug. 2023. [Online]. Available: https://webrtc.org/. EXTERNAL
S. Team, “Silero VAD: pre-trained enterprise-grade voice activity detector (VAD), number detector and language classifier,” 2021. EXTERNAL
M. Bhatia, “VoIP: An In-Depth Analysis - Voice Activity Detection,” in Cisco, Oct. 2006. EXTERNAL
M. Heck, Automatic Language Identification for Natural Speech Processing Systems, The Department of Informatics Institute of Anthropomatics (IFA) Interactive Systems Laboratories (ISL), 2011. EXTERNAL
R. Wicks and M Post, “A unified approach to sentence segmentation of punctuated text in many languages,” in Proc. 59th Ann. Meeting of the Assoc. for Comput. Ling., pp. 3995–4007, 2021. EXTERNAL
B. Minixhofer, “Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation,” arXiv preprint arXiv:2305.18893, 2023. EXTERNAL
Park, Tae Jin, “Auto-tuning spectral clustering for speaker diarization using normalized maximum eigengap,” IEEE Signal Processing Letters, 27: pp. 381–385, 2019. EXTERNAL
Graham Jones, “Implementing closed captioning for DTV,” in Proc. Broadcast Engineering Conference, National Association of Broadcasters (NAB), p. 8, 2004. EXTERNAL
Society of Cable Telecommunications Engineers (SCTE), SCTE 21 2012 - “Standard for Carriage of VBI Data in Cable Digital Ttransport Streams,” SCTE 21: 13. 2012. EXTERNAL
European Broadcasting Union (EBU), “Specification of the EBU Subtitling Data Exchange Format,” PDF. February 1991. Retrieved 10 Mar. 2013. EXTERNAL
European Broadcasting Union (EBU), Tech 3380, EBU-TT-D Subtitling Distribution Format Version 1.0. EXTERNAL
“Timed Text (TT) Authoring Format 1.0 - Distribution Format Exchange Profile (DFXP),” 2018. EXTERNAL
A. Mehrish, N. Majumder, R. Bhardwaj, R. Mihalcea, and S. Poria, “A Review of Deep Learning Techniques for Speech Processing,” Information Fusion, Apr. 2023. [Online]. Available: https://arxiv.org/abs/2305.00359. EXTERNAL
G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean. “Multilingual Acoustic Models Using Distributed Deep Neural Networks,” in Proc. of ICASSP, 2013. EXTERNAL
Abdel-Hamid, O., Mohamed, A., Jiang, H., Deng, L., Penn, G., & Yu, D, “Convolutional Neural Networks for Speech Recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22: pp. 1533–1545, 2014. EXTERNAL
Alex Graves, Abdel-Rahman Mohamed, Gregory Hinton, “Speech Recognition with Deep Recurrent Neural Networks,” in ICASSP, Proc. IEEE Intern. Conf. on Acoustics, Speech and SIgnal Process. 38. 10.1109/ICASSP.2013.6638947, 2013. EXTERNAL
A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is All you Need, NIPS,” 2017. EXTERNAL
D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” CoRR, abs/1409.0473, 2014. EXTERNAL
Chirp: University Speech Model. [Online]. Available: https://cloud.google.com/speech-to-text/v2/docs/chirp-model. EXTERNAL
Azure: Speech to text. [Online]. Available: https://azure.microsoft.com/en-us/products/ai-services/speech-to-text#features. EXTERNAL
Amazon Transcribe Features. [Online]. Available: https://aws.amazon.com/transcribe/features/?nc=sn&loc=2. EXTERNAL
Google Cloud's new visual interface for Speech-to-Text API | Google Cloud Blog. [Online]. Available: https://cloud.google.com/blog/products/ai-machine-learning/google-clouds-new-visual-interface-for-speech-to-text-api. EXTERNAL
A.C. Morris, V. Maier, and P.D. Green, “From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition,” Interspeech, 2004. EXTERNAL
Hugging Face. [Online]. Available: https://huggingface.co/learn/audio-course/en/chapter5/evaluation#character-error-rate. EXTERNAL
Learn. Azure. Test Accuracy of a Custom Speech Model. [Online]. Available: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-custom-speech-evaluate-data?pivots=speech-studio#resolve-errors-and-improve-wer. EXTERNAL
R. Ardila, “Common voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1972.06670, 2019. EXTERNAL
A. Conneau, M. Ma, S. Khanuja, Y. Zhang, V. Axelrod, S. Dalmia, J. Riesa, C. Rivera, C., and A. Bapna, “FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech,” in IEEE Spoken Language Technology Workshop (SLT), pp. 798–805, 2022. EXTERNAL
List of languages by total number of speakers. [Online]. Available: https://en.wiki-pedia.org/wiki/List_of_languages_by_total_number_of_speakers. EXTERNAL
F. Galpin, “AHG11: Deep-learning based inter prediction blending”, JVET-V0076-v2” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29 22nd Meeting, teleconference, 20–28 Apr. 2021. EXTERNAL
Changyue Ma, “AHG11: Neural Network Based Motion Compensation Enhancement for Video Coding, “JVET-Y0090-v1” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 Jan. 2022. EXTERNAL
Zizheng Liu, “AHG11: NN-based Reference Frame Interpolation for VVC Hierarchical Coding Structure, JVET-Y0096-v1,” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 25th Meeting, teleconference, 12–19 Jan. 2022. EXTERNAL
Liqiang Wang, “Neural Network Based in-Loop Filter with Constrained Memory,” in Proc. 2022 IEEE Internat. Conf. on Multimedia and Expo (ICME), 2022, DOI: 10.1109/ICME52920.2022.9859910. EXTERNAL
Yue Li, “EE1–1.2: Test on Deep In-Loop Filter with Adaptive Parameter Selection and Residual Scaling, JVET-Y0143-v2,” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 Jan. 2022. EXTERNAL
Ming Lu, “EE1: Tests on Decomposition, Compression, Synthesis (DCS)-based Technology, JVET-V0149,” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 22th Meeting, teleconference, 20–28 Apr. 2021. EXTERNAL
Takeshi Chujoh, “EE1.2: Additional experimental results of NN-based super resolution (JVET-U0053),” in JVET-V0073-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 22nd Meeting, teleconference, 20–28 Apr. 2021. EXTERNAL
Chaoyi Lin, “EE1–2.3: CNN-based Super Resolution for Video Coding Using Decoded Information, JVET-Y0069-v21,” in Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 25th Meeting, teleconference, 12–21 January 2022 DOI: 10.1109/VCIP53242.2021.9675417. EXTERNAL
Workshop and Challenge on Learned Image Compression (CLIC). [Online]. Available: https://www.compression.cc/. EXTERNAL
G.-H. Wang, J. Li, B. Li and Y. Lu, “EVC: Towards Real-Time Neural Image Compression with Mask Decay,” in The Eleventh International Conference on Learning Representations (ICLR), May 2023. EXTERNAL
D. He, Z. Yang, W. Peng, R. Ma, H. Qin and Y. Wang, “ELIC: Efficient Learned Image Compression With Unevenly Grouped Space-Channel Contextual Adaptive Coding,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5718–5727, 2022. EXTERNAL
G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai and Z. Gao, “DVC: An End-To-End Deep Video Compression Framework,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11006–11015, 2019. EXTERNAL
L. Jianping, L. Dong, L. Houqiang and W. Feng, “M-Ivc:: Multiple frames prediction for learned video compression,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. EXTERNAL
R. Yang, F. Mentzer, L. V. Gool and R. Timofte, “Learning for Video Compression With Recurrent Auto-Encoder and Recurrent Probability Model,” IEEE Journal of Selected Topics in Signal Processing, 15 (2): pp. 388–401, 2021. EXTERNAL
VideoLAN. [Online]. Available: https://www.videolan.org/developers/x264.html. EXTERNAL
VideoLAN. [Online]. Available: https://www.videolan.org/developers/x265.html. EXTERNAL
R. Yang, F. Mentzer, L. V. Gool and R. Timofte, “Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6628–6637, 2020. EXTERNAL
H. Liu, M. Lu, Z. Ma, F. Wang, Z. Xie, X. Cao and Y. Wang, “Neural Video Coding Using Multiscale Motion Compensation and Spatiotemporal Context Model,” IEEE Transactions on Circuits and Systems for Video Technology, 31: pp. 3182–3196, 2021. EXTERNAL
T. Ladune, P. Philippe, W. Hamidouche, L. Zhang and O. Deforges, “Conditional Coding for Flexible Learned Video Compression,” in Neural Compression Workshop ICLR 2027, 2021. EXTERNAL
J. Li, B. Li and Y. Lu, “Deep Contextual Video Compression,” Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. EXTERNAL
X. Sheng, J. Li, B. Li, L. Li, L. Dong and L. Yan, “Temporal Context Mining for Learned Video Compression,” IEEE Transactions on Multimedia, 25: pp. 7311–7322, 2023, DOI: 10.1109/TMM.2022.3220421. EXTERNAL
A. Habibian, T. van Rozendaal, J. M. Tomczak and T. S. Cohen, “Video Compression With Rate-Distortion Autoencoders,” in Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7033–7042, 2019. EXTERNAL
S. Santurkar, D. Budden and N. Shavit, “Generative Compression,” in 2018 Picture Coding Symposium (PCS), 2018. EXTERNAL
F. Mentzer, G. Toderici, D. Minnen, S.-J. Hwang, S. Caelles, M. Lucic and E. Agustsson, “VCT: A Video Compression Transformer,” in NeurIPS'22, 2022. EXTERNAL
E. Dupont, A. Goliński, M. Alizadeh, Y. W. Teh and A. Doucet, “COIN: COmpression with Implicit Neural representations,” Computer Vision and Pattern Recognition, 2021. EXTERNAL
S. Kim, S. Yu, J. Lee and J. Shin, “Scalable Neural Video Representations with Learnable Positional Features,” in Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. EXTERNAL
T. Ladune, P. Philippe, F. Henry, G. Clare and T. Leguay, “COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec,” arXiv preprint, 2022. EXTERNAL
H. Chen, B. He, H. Wang, Y. Ren, L. S. Nam and A. Shrivastava, “NeRV: Neural Representations for Videos,” in Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. EXTERNAL
Z. Li, M. Wang, H. Pi, K. Xu, J. Mei and Y. Liu, “E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context,” in Proc. European Conference on Computer Vision, pp. 267–284, 2022. EXTERNAL
Y. Bai, C. Dong and C. Wang, “PS-NeRV: Patch-wise Stylized Neural Representations for Videos,” arXiv preprint, 2022. EXTERNAL
H. Chen, M. Gwilliam, S.-N. Lim and A. Shrivastava, “HNeRV: A Hybrid Neural Representation for Videos,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10270–10279, 2023. EXTERNAL
H. Chen, M. Gwilliam, B. He, S.-N. Lim and A. Shrivastava, “CNeRV: Content-adaptive Neural Representation for Visual Data,” in BMVC 2022, 2022. EXTERNAL
Q. Zhao, M. S. Asif and Z. Ma, “DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videos,” in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2031–2040, 2023. EXTERNAL
J. C. Lee, D. Rho, J. H. Ko and E. Park, “FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos,” arXiv preprint, 2022. EXTERNAL
O. Rippel, A. G. Anderson, K. Tatwawadi, S. Nair, C. Lytle and L. Bourdev, “ELF-VC: Efficient Learned Flexible-Rate Video Coding,” in Proc. of the IEEE/CVF Intern. Conf. on Comp. Vision (ICCV), pp. 14479–1448, 2021. EXTERNAL
Chaoyi Lin, “AHG11: Variable rate end-to-end image compression,” in JVET-U0102” Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29, 21st Meeting, teleconference, 6–15 Jan. 2021. EXTERNAL
A. Reuther, P. Michaleas, M. Jones, V. Gadepally and J. K. Siddharth Samsi, “AI and ML Accelerator Survey and Trends,” in 2022 IEEE High Performance Extreme Computing Conference (HPEC), 2022. EXTERNAL
V. J. Reddi, C. Cheng, D. Kanter, P. Mattson and Y. Z. Guenther Schmuelling “MLPerf Inference Benchmark” in ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020. EXTERNAL
OpenVINO. [Online]. Available: https://docs.openvino.ai/latest/index.html#. EXTERNAL
NVIDIA. [Online]. Available: https://developer.nvidia.com/tensorrt. EXTERNAL
N. Imam and T. A. Cleland, “Rapid online learning and robust recall in a neuromorphic olfactory circuit,” Nature Machine Intelligence, pp. 181–191, 2020. EXTERNAL
Hailo-8. [Online]. Available: https://hailo.ai/products/hailo-8/. EXTERNAL
M. Tarchouli, S. Pelurson, T. Guionnet, W. Hamidouche, M. Outtas and O. Deforges, “Patch-Based Image Coding with End-To-End Learned Codec using Overlapping.,” in 12th International Conference on Artificial Intelligence, Soft Computing and Applications, pp. 53–63, 2022, DOI 10.5121/csit.2022.122305. EXTERNAL
F. Bossen, J. Boyce, X. Li, V. Seregin and K. Sühring, “VTM Common Test Conditions and Software Reference Configurations for SDR Video,” document JVET-T2010 of JVET, Oct. 2020. EXTERNAL
SMPTE ST 2052-1:2010 [Superseded]
T-REC-H.120 — “H.120: Codecs for videoconferencing using primary digital group transmission,” ITU-T Recommendation H.120 (11/88) [Online]. Available: https://www.itu.int/rec/T-REC-H.120-198811-S/en. EXTERNAL
T-REC-H.261 — ITU-T H.261 (03/2005), Video Codec for Audiovisual Services at p x 64 kb/s EXTERNAL
T-REC-H.262 — International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 13818-2/ITU-T H.262, “Generic Coding of Moving Pictures and Associated Audio Information: Video,” 1996. EXTERNAL
T-REC-H.263 — Recommendation ITU-T H.263, “Video Coding for Low Bit Rate Communication”, Annex B “Hypothetical Reference Decoder,” Jan 1998. EXTERNAL
ITU-T REC H.264 [Active]
ITU-T REC H.265 [Active]
T-REC-H.266 — “Versatile video coding,” ITU-T Recommendation H.266 (04/22) [Online]. Availabe: https://www.itu.int/ITU-T/recommendations/rec.aspx?rec=14948&lang=en. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2024-IPYX8877__2024-SMPTEMIJAPRIL2024_18-18.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Randy Fayan, Zahra Montajabi, and Rob Gonsalves; Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools, MIJ 2024, Volume 133, Number 2 (pp. 48 to 57); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/IPYX8877

Snippet:

Randy Fayan, Zahra Montajabi, and Rob Gonsalves; Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools, MIJ 2024, Volume 133, Number 2 (pp. 48 to 57); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/IPYX8877

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Randy Fayan, Zahra Montajabi, and Rob Gonsalves; <cite>Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools</cite>, MIJ 2024, Volume 133, Number 2 (pp. 48 to 57); SMPTE, 2024. Available at <a href="https://doi.org/10.5594/JMI.2024/IPYX8877" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2024/IPYX8877</a></span>

SMPTE's HTML Pub

Preview:

Randy Fayan, Zahra Montajabi, and Rob Gonsalves; Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools, MIJ 2024, Volume 133, Number 2 (pp. 48 to 57); SMPTE, 2024
doi: 10.5594/JMI.2024/IPYX8877
url: https://doi.org/10.5594/JMI.2024/IPYX8877

Snippet:

<li>
Randy Fayan, Zahra Montajabi, and Rob Gonsalves; <cite id="bib-10-5594-jmi-2024-ipyx8877__2024-smptemijapril2024_18-18">Automatic Speech Recognition with Machine Learning: Techniques and Evaluation of Current Tools</cite>, MIJ 2024, Volume 133, Number 2 (pp. 48 to 57); SMPTE, 2024
<span class="doi">10.5594/JMI.2024/IPYX8877</span>
</li>