Testing 4K HDR-WCG professional video content for subjective quality using a remote testing approach

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2023, No. 10, pp. 1–3
Abstract: Modern content is rapidly shifting from the standard dynamic range (SDR) color volume defined in ITU-R BT.709 towards the high dynamic range and wide color gamut (HDR-WCG) color volume defined in ITU-R BT.2100. For optimizing delivery channels of this content it is crucial to have high performance video quality metrics. However, existing subjective databases used to design and qualify such video quality metrics do not provide adequate test coverage for professional HDR-WCG video content. In this paper we describe a subjective video quality study aimed to be relevant for existing as well as near future HDR-WCG. — Due to the Covid pandemic, the experiment was designed for remote testing, rather than a typical laboratory environment. We carefully analyzed the subjective results and concluded that it is possible to obtain comparable reliability of data in our remote testing environment with a controlled laboratory environment. This opens the door for large-scale video quality testing, enabling continual refinement of video quality metrics, reducing the turn-around time for subjective testing, and eliminating bias in the training data by accessing a much larger and more diverse set of users. — We applied the subjective data set to compare existing image/video quality metrics such as VMAF, HDR-VDP and HDR-VQM. Furthermore, we identified several ways the VMAF metric can be adjusted to improve its accuracy on HDR-WCG content.
Publication Date: 2023-10-16
DOI: 10.5594/M002001
ISBN: Print: 978-1-61482-964-5
Link: https://doi.org/10.5594/M002001
Author(s): Anustup ChoudhuryDolby Laboratories Inc.
Lukas KrasulaNetflix Inc.
Scott DalyDolby Laboratories Inc.
Zhi LiNetflix Inc.
Robin AtkinsDolby Laboratories Inc.
Ludovic MalfaitDolby Laboratories Inc.
Keyword(s): HDR, Visual Quality Metric, Contrast Sensitive Function
Copyright: © 2023 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

M. Narwaria, R. K. Mantiuk, M. Perreira Da Silva, P. Le Callet, "HDR-VDP-2.2: A Calibrated Method for Objective Quality Prediction of High Dynamic Range and Standard Images," Journal of Electronic Imaging, 24(1), 2015. EXTERNAL
N. Moroney, "Unconstrained web-based color naming experiment", Proc. SPIE 5008, pp. 36-46 (2003). EXTERNAL
R. Mantiuk, K. J. Kim, A. G. Rempel, W. Heidrich, "HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions," ACM Transactions on Graphics, 30(4), 40:1--40:14, 2011, doi: 10.1145/2010324.1964935. EXTERNAL
ITU-T Technical Report PSTR-CROWDS, "Subjective evaluation of media quality using a crowdsourcing approach", May 2018. EXTERNAL
A. K. Venkataramanan, C. Wu, A. C. Bovik, I. Katsavounidis and Z. Shahid, "A Hitchhiker’s Guide to Structural Similarity," in IEEE Access, vol. 9, pp. 28872-28896, 2021, doi: 10.1109/ACCESS.2021.3056504 EXTERNAL
J. Gille and J. Larimer, Using the Human Eye to Characterize Displays, Human Vision and Electronic Imaging VI, Proc. SPIE, 4299, pp. 439-454 (2001). EXTERNAL
Z. Li, C. G. Bampis, L. Krasula, L. Janowski, I. Katsavounidis, "A Simple Model for Subject Behavior in Subjective Experiments," arXiv:2004.02067, doi: 10.48550/arXiv.2004.02067. EXTERNAL
Z. Li, C. Bampis, J. Novak, A. Aaron, K. Swanson, A. Moorthy, J. De Cock, “VMAF: The Journey Continues,” Netflix Technology Blog, [online]. https://tinyurl.com/bdf6z3jz EXTERNAL
M. Narwaria, M. Perreira da Silva, P. Le Callet, "HDR-VQM: An Objective Quality Measure for High Dynamic Range Video," Signal Processing: Image Communication, 2015, 35, pp.46-60. EXTERNAL
Z. Li, K. Swanson, C. G. Bampis, L. Krasula, A. Aaron, "Toward a Better Quality Metric for the Video Community," Netflix Technology Blog, [online]. https://tinyurl.com/2p8fce64. EXTERNAL
S. Li, F. Zhang, L. Ma and K. N. Ngan, "Image Quality Assessment by Separately Evaluating Detail Losses and Additive Impairments," in IEEE Transactions on Multimedia, vol. 13, no. 5, pp. 935-949, Oct. 2011, doi: 10.1109/TMM.2011.2152382. EXTERNAL
L. Krasula, A. Choudhury, S. Daly, Z. Li, R. Atkins, L. Malfait, A. Mavlankar, “Subjective video quality for 4K HDR-WCG content using a browser-based approach for" at-home" testing,” Electronic Imaging, vol. 35, pp. 263—1-8 (2023) EXTERNAL
N. Moroney and G. Beretta, "The world wide “gamma”", IS&T Color Imaging Conference 2010. EXTERNAL
T. Hoβfeld, R. Schatz and S. Egger, "SOS: The MOS is not enough!," 2011 Third International Workshop on Quality of Multimedia Experience, Mechelen, 2011, pp. 131-136. EXTERNAL
H.R. Sheikh.and A.C. Bovik, "Image information and visual quality," IEEE Transactions on Image Processing, vol.15, no.2, pp. 430- 444, Feb. 2006. EXTERNAL
H. Schober and R. Hilz, “Contrast Sensitivity of the Human Eye for Square Wave Gratings”, EXTERNAL
ITU-R BT.2100-2 [Active]
T-REC-P.808 — International Telecommunication Union-Telecommunication (ITU-T) Recommendation P.808, “Subjective evaluation of speech quality with a crowdsourcing approach,” Jun. 2021. EXTERNAL
T-REC-P.910 — ITU-T Recommendation P.910, “Subjective Video Quality Assessment Methods for Multimedia Applications,” International Telecommunication Union, Geneva, Switzerland, Apr. 2008. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M002001.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Anustup Choudhury, Lukas Krasula, Scott Daly, Zhi Li, Robin Atkins, and Ludovic Malfait; Testing 4K HDR-WCG professional video content for subjective quality using a remote testing approach, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002001

Snippet:

Anustup Choudhury, Lukas Krasula, Scott Daly, Zhi Li, Robin Atkins, and Ludovic Malfait; Testing 4K HDR-WCG professional video content for subjective quality using a remote testing approach, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002001

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Anustup Choudhury, Lukas Krasula, Scott Daly, Zhi Li, Robin Atkins, and Ludovic Malfait; <cite>Testing 4K HDR-WCG professional video content for subjective quality using a remote testing approach</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at <a href="https://doi.org/10.5594/M002001" target="_blank" rel="noopener">https://doi.org/10.5594/M002001</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Anustup Choudhury, Lukas Krasula, Scott Daly, Zhi Li, Robin Atkins, and Ludovic Malfait; <cite id="bib-10-5594-m002001">Testing 4K HDR-WCG professional video content for subjective quality using a remote testing approach</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5
<span class="doi">10.5594/M002001</span>
</li>