Toward a Subjective Assessment System for Closed Captioning Quality

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 3, pp. 35–44
Abstract: A novel quality assessment system design for closed captioning (CC) is proposed. CC was originally designed to serve hearing-impaired audiences. Traditional quality assessment models focus on empirical methods only, measuring quantitative accuracy by counting the number of word errors in the captions for a segment. However, hearing-impaired audiences have been outspoken about their dissatisfaction with the quality of current CC. One solution to this problem may involve inviting human evaluators, who represent different groups, to assess the quality of CC at the end of each segment, but in reality, this may be difficult to do and very labor-intensive. To solve this challenge, we propose the training and use of an artificial intelligence system to predict subjective CC quality based on human assessment data. In this paper, the design and development process, to be used in developing this system, is described.
Publication Date: 2021-04-01
DOI: 10.5594/JMI.2021.3059344
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2021.3059344
Author(s): Somang Nam
bio
Somang Nam was born in Bielefeld, Germany, in 1989. He received an HBSc degree in computer science with a mathematics and music minor from the University of Toronto, Toronto, ON, Canada, in 2013, and the MSc degree in computer science from Ryerson University, Toronto, in 2016. He is currently a PhD candidate in industrial engineering with the University of Toronto. His main areas of research interest are human factors, human–computer interaction, and accessibility.
Deborah Fels
bio
Deborah Fels received a PhD degree in human factors from industrial engineering with the University of Toronto, Toronto, ON, Canada, in 1994. She is currently employed as a professor with the Ted Rogers School of Information Technology Management, Toronto, and the director of the Inclusive Media and Design Centre at Ryerson University, Toronto. Her research interests involve inclusive design, access to media and technology for people with disabilities and older adults, and inclusive video game design and inclusive business. She has published over 150 articles on inclusive technologies and applications and received three patents. She is also a professional engineer.
Mark Chignell
bio
Mark Chignell received a PhD degree in psychology from the University of Canterbury, Christchurch, New Zealand, and a master’s degree in industrial and systems engineering from Ohio State University, Columbus, OH. He is a professor of mechanical and industrial engineering with the University of Toronto, Toronto, ON, Canada. He carries out research in human factors and user interface design, with particular interest in health care, aging, telecommunications, and driving applications.
Keyword(s): Artificial Intelligence, Neural Network, Captions, Machine Learning, User Centered Design
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

J. R. Bormuth, “Cloze Test Readability : Criterion Reference Scores,” J. Educ. Meas., 5(3):189–196, Sep. 1968. EXTERNAL
S. Zdenek, Reading Sounds, 1st ed., University of Chicago Press: Chicago, IL, 2015. EXTERNAL
Canadian Radio-television and Telecommunications Commission-Government of Canada, “Broadcasting Regulatory Policy CRTC 2012-362 Quality Standards for English-Language Closed Captioning,” 2012. EXTERNAL
National Association of Broadcasters (NAB), “Closed Captioning Quality,” 2221(2014), 2009. which would require further investigation on the prediction model-independent variables and the relationship between variables. Another limitation was the logical mapping between the quality factor representative values and the probability distribution. The fundamental assumption of the SDT framework is normal probability distributions for the signal and errors.38 Therefore, a mapping function that better reflects the real distribution of each quality factor should lead to more accurate user models. A third important limitation was the small participant sample. Adding more participants would increase the opportunity of collecting representative data, and the larger sample size would increase the power of statistical analysis and the developed prediction models. Future research could include a fine-tuning stage where active learning algorithms are employed.56,57 Active learning will allow the system to learn directly from humans by querying the caption quality. The selection algorithm would prioritize caption variations that had greater uncertainty associated with them, providing a more efficient training process. Conclusion In this paper, the design and development of a novel system to assess the quality of CC based on caption viewer subjective quality ratings is described. Based on findings from a user study with hearing-impaired viewers, a set of human decision probability models dealing with captioning variations was developed. With these user models and prediction models, we then generated artificial data, which reflected human assessment of caption quality. We then trained deep neural networks with the generated data, creating an AI system to predict human subjective ratings for different caption variations. The proposed solution fills a gap in existing quality assessment models for CC, which does not include any viewer quality assessment factors. The proposed system should improve as more data are collected, allowing the core prediction model to learn from the new user data over time. EXTERNAL
S. Nam and D. Fels, “Assessing Closed Captioning Quality Using a Multilayer Perceptron,” IEEE Int. Conf. Artificial Intelligence and Knowledge Engineering (AIKE), pp. 9–16, 2018, doi: 10.1109/ EXTERNAL
W. L. Taylor, “‘Cloze Procedure’: A New Tool for Measuring Readability,” J. Bull., 30(4):415–433, 1953, doi: 10.1177/107769905303000401. EXTERNAL
Ofcom, “Subtitling – An Issue of Speed?,” 2005. EXTERNAL
Ofcom, “Ofcom’s Code on Television Access Services 1 Code on Television Access Services,” London, U.K., 2015. EXTERNAL
M. Bontoft and G. Pullin, “What Is an Inclusive Design Process?,” in Inclusive Design, 1st ed., vol. 1, J. Clarkson, S. Keates, R. Coleman, and C. Lebbon, Eds. Springer: London, U.K., pp. 520–531, 2003. EXTERNAL
E. Jones et al., “SciPy: Open Source Scientific Tools for Python,” 2001. [Online]. Available: http://www.scipy.org/ EXTERNAL
C. M. Connine, J. Mullennix, E. Shernoff, and J. Yelen, “Word Familiarity and Frequency in Visual and Auditory Word Recognition,” J. Exp. Psychol. Learn. Mem. Cogn., 16(6):1084– 1096, Nov. 1990, doi: 10.1037/0278-7393.16.6.1084. EXTERNAL
N. A. Macmillan and C. D. Creelman, Detection Theory: A User’s Guide, 2nd ed., Taylor & Francis: New York, NY, 2005. EXTERNAL
U. Strauss, P. Grzybek, and G. Altmann, “Word Length and Word Frequency,” Contrib. Sci. Text Lang., (1990): 277–294, 2007, doi: 10.1007/978-1-4020-4068-9_13. EXTERNAL
R. Rashid, Q. Vy, R. Hunt, and D. I. Fels, “Dancing With Words: Using Animated Text for Captioning,” Int. J. Hum. Comput. Interact., 24(5):505–519, Jun. 2008, doi: 10.1080/10447310802142342. EXTERNAL
J. A. Swets, W. P. Tanner, Jr., and T. G. Birdsall, “The Evidence for a Decision Making Theory of Visual Detection,” Tech. Rep., Dept. of Electrical Engineering, Michigan Univ., Ann Arbor, MI, 1955. EXTERNAL
W. J. B. van Heuven, P. Mandera, E. Keuleers, and M. Brysbaert, “SUBTLEX-UK: A New and Improved Word Frequency Database for British English,” Q. J. Exp. Psychol., 67(6):1176–1190, Jun. 2014, doi: 10.1080/17470218.2013.850521. EXTERNAL
H. Stanislaw and N. Todorov, “Calculation of Signal Detection Theory Measures,” Behav. Res. Methods. Instrum. Comput., 31(1):137–149, Feb. 1999, doi: 10.3758/BF03207704. EXTERNAL
J. Tang, C. Deng, and G.-B. Huang, “Extreme Learning Machine for Multilayer Perceptron,” IEEE Trans. Neural Netwo. Learn. Syst., 27(4):809–821, Apr. 2016, doi: 10.1109/ EXTERNAL
P. Romero-Fresco, “Final Thoughts: Viewing Speed in Subtitling,” Recept. Subtitles Deaf Hard Hear. Eur., Neves 2008, 2016, doi: 10.3726/978-3-0351-0888-0/24. EXTERNAL
J. Ohene-Djan, J. Wright, and K. Combie-Smith, “Emotional Subtitles: A System and Potential Applications for Deaf and Hearing Impaired People,” Conf. Work. Assist. Technol. People With Vis. Hear. Impair., 63(2), pp. 1–6, 2007, doi: 10.7790/tja.v63i2.406. EXTERNAL
R. J. Ryder and M. Hughes “The Effect on Text Comprehension of Word Frequency,” J. Educ. Res., 78(5):286– 291, 1985, doi: 10.1080/00220671.1985.10885617. EXTERNAL
Federal Communications Commission, “Consumer Guide,” Consumer and Governmental Affairs Bureau, Washington, DC, 2017. [Online]. Available: https://www.fcc.gov/consumers/guides/closed-captioning-television EXTERNAL
Canadian Association of the Deaf, “Final Project Report: Understanding User Responses to Live Closed Captioning in Canada,” Toronto, ON, Canada, 2016, doi: 10.2172/1000008. EXTERNAL
M. Hersh, “Deaf People’s Experiences, Attitudes and Requirements of Contextual Subtitles: A Two-Country Survey,” Telecomm. J. Australia, 63(2):23.1–23.14, May 2013. EXTERNAL
A. Szarkowska, I. Krejtz, Z. Klyszejko, and A. Wieczorek, “Verbatim, Standard, or Edited?: Reading Patterns of Different Captioning Styles Among Deaf, Hard of Hearing, and Hearing Viewers,” 156(4):363–378, Fall 2011. EXTERNAL
D. W. Ruck, S. K. Rogers, and M. Kabrisky, “Feature Selection Using a Multilayer Perceptron,” J. Neural Network Comput., 2(2):40–48, 1990. EXTERNAL
Media Access Austrailia, “Caption Quality: International Approaches to Standards and Measurement,” Sydney, NSW, Australia, 2007. [Online]. Available: https://mediaaccess.org.au/sites/default/files/files/MAA_CaptionQuality-Whitepaper.pdf EXTERNAL
P. Romero-Fresco, “More Haste Less Speed: Edited Versus Verbatim Respoken Subtitles,” Vigo Int. J. Appl. Linguist., 6(1):109–133, 2009. EXTERNAL
P. Romero-Fresco and J. M. Pérez, “Accuracy Rate in Live Subtitling: The NER Model,” Audiov. Transl. Glob. Context, pp. 28–50, 2015, doi: 10.1057/9781137552891_3. EXTERNAL
Canadian Radio-television and Telecommunications Commission-Government of Canada, “English-Language Closed Captioning Quality Standard Related to the Accuracy Rate for Live Programming,” 2016. EXTERNAL
M. Brysbaert and B. New, “Moving Beyond Kucera and Francis: A Critical Evaluation of Current Word Frequency Norms and the Introduction of a New and Improved Word Frequency Measure for American English,” Behav. Res. Methods, 41(4):977– 990, Nov. 2009, doi: 10.3758/BRM.41.4.977. EXTERNAL
P. Romero-Fresco, “Accessing Communication: The Quality of Live Subtitles in the U.K.,” Lang. Commun., 49:56-69, Jul. 2016, doi: 10.1016/j.langcom.2016.06.001. EXTERNAL
S. Schuster, S. Hawelka, F. Hutzler, M. Kronbichler, and F. Richlan, “Words in Context: The Effects of Length, Frequency, and Predictability on Brain Responses During Natural Reading,” Cereb. Cortex, 26(10):3889–3904, Oct. 2016, doi: 10.1093/cercor/bhw184. EXTERNAL
A. F. Newell and P. Gregor, “‘User Sensitive Inclusive Design’— In Search of a New Paradigm,” Proc. Conf. Univ. Usability (CUU), pp. 39–44, 2000, doi: 10.1145/355460.355470. EXTERNAL
R. E. Pastore and C. J. Scheirer, “Signal Detection Theory: Considerations for General Application,” Psychol. Bull., 81(12):945–958, 1974, doi: 10.1037/h0037357. EXTERNAL
Canadian Radio-television and Telecommunications Commission-Government of Canada, “Canadian NER Evaluation Guidelines,” pp. 1–9, 2018. [Online]. Available: https://crtc.gc.ca/eng/archive/2019/2019-9.htm EXTERNAL
National Court Reporters Association, “What Is an Error ?,” vol. 180, pp. 12–14, Oct. 2013. EXTERNAL
J. Sandford, “White Paper WHP 306: The Impact of Subtitle Display Rate on Enjoyment Under Normal Television Vewing Conditions,” London, U.K., 2015. EXTERNAL
J. Goetz, A. Tewari, and P. Zimmerman, “Active Learning for Non-parametric Regression Using Purely Random Trees,” Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 2018, pp. 2537–2546, Dec. 2018. [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85064808972&partnerID=40&md5=9020e74fd3d13c121d64dac6e57085c0 EXTERNAL
N. A. Macmillan and C. D. Creelman, “Response Bias: Characteristics of Detection Theory, Threshold Theory, and ‘Nonparametric’ Indexes,” Psychol. Bull., 107(3):401–413, May 1990, doi: 10.1037/0033-2909.107.3.401. EXTERNAL
J. A. Nevin, “ Signal Detection Theory and Operant Behavior: A Review of David M. Green and John A. Swets’ Signal Detection Theory and Psychophysics. 1 ,” J. Exp. Anal. Behav., 12(3):475– 480, May 1969, doi: 10.1901/jeab.1969.12-475. EXTERNAL
C. Jensema, R. McCann, and S. Ramsey, “Closed-Captioned Television Presentation Speed and Vocabulary,” Am. Ann. Deaf, 141(4):284–290, Oct. 1996, doi: 10.1353/aad.2012.0377. EXTERNAL
Federal Communications Commission, “Closed Captioning Quality Order,” 2016. EXTERNAL
Ofcom, “Measuring the Quality of Live Subtitling,” London, U.K., 2013. [Online]. Available: https://www.ofcom.org.uk/research-and-data/tv-radio-and-on-demand/tv-research/live-subtitling EXTERNAL
Canadian Broadcasting Accessibility Fund, “Understanding User Responses to Live Closed Captioning in Canada,” Toronto, ON, Canada, 2019. EXTERNAL
N. A. Macmillan, “Signal Detection Theory,” in Stevens’ Handbook of Experimental Psychology, 3rd ed., H. Pashler and J. Wixted, Eds. Hoboken, NJ: John Wiley&Sons, Inc., pp. 43–85, Aug. 2002. EXTERNAL
Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, 521(7553):436–444, May 2015, doi: 10.1038/nature14539. EXTERNAL
C. Jensema, “Viewer Reaction to Different Television Captioning Speeds,” Am. Ann. Deaf, 143(4):318–324, Oct. 1998, doi: 10.1353/aad.2012.0073. EXTERNAL
B. Settles, “Computer Sciences Active Learning Literature Survey,” Sci. York, Jan. 2009. EXTERNAL
Canadian Association of the Deaf, “Key Findings: Understanding User Responses to Live Closed Captioning in Canada,” Toronto, ON, Canada, 2016. EXTERNAL
J. R. Koza, F. H. Bennett, D. Andre, and M. A. Keane, “Automated Design of Both the Topology and Sizing of Analog Electrical Circuits Using Genetic Programming,” Artif. Intell. Des., pp. 151–170, 1996, doi: 10.1007/978-94-009-0279-4_9. EXTERNAL
V. Louthan, “Some Systematic Grammatical Deletions and Their Effects on Reading Comprehension,” English J., 54(4):295, Apr. 1965, doi: 10.2307/811113. EXTERNAL
A. Raake and S. Egger, “Quality and Quality of Experience,” in Quality of Experience, Springer: Cham, Switzerland, pp. 11–33, Mar. 2014. EXTERNAL
J. Schmidhuber, “Deep Learning in Neural Networks: An Overview,” Neural Netw., 61:85–117, Jan. 2015, doi: 10.1016/j. neunet.2014.09.003. EXTERNAL
L. Postman, J. S. Bruner, and R. D. Walk, “The Perception of Error,” Br. J. Psychol. Gen. Sect., 42(1):1–10, Mar.–May 1951, doi: 10.1111/j.2044-8295.1951.tb00277.x. EXTERNAL
T. Apone, B. Botkin, M. Brooks, and L. Goldberg, “Caption Accuracy Metrics Project Research into Automated Error Ranking of Real-Time Captions in Live Television News Programs,” 2011. [Online]. Available: https://dcmp.org/learn/300-caption-viewer-sur-vey-error-ranking-of-real-time-captions-in-live-television-news EXTERNAL
SMPTE OV 2052-0:2010 [Superseded]

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2021.3059344.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Somang Nam, Deborah Fels, and Mark Chignell; Toward a Subjective Assessment System for Closed Captioning Quality, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3059344

Snippet:

Somang Nam, Deborah Fels, and Mark Chignell; Toward a Subjective Assessment System for Closed Captioning Quality, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3059344

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Somang Nam, Deborah Fels, and Mark Chignell; <cite>Toward a Subjective Assessment System for Closed Captioning Quality</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2021.3059344" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2021.3059344</a></span>

SMPTE's HTML Pub

Preview:

Somang Nam, Deborah Fels, and Mark Chignell; Toward a Subjective Assessment System for Closed Captioning Quality, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
doi: 10.5594/JMI.2021.3059344
url: https://doi.org/10.5594/JMI.2021.3059344

Snippet:

<li>
Somang Nam, Deborah Fels, and Mark Chignell; <cite id="bib-10-5594-jmi-2021-3059344">Toward a Subjective Assessment System for Closed Captioning Quality</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 3, April 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2021.3059344</span>
</li>