HDR and WCG Image Quality Assessment Using Color Difference Metrics

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 129, No. 9, pp. 38–49
Abstract: High dynamic range (HDR) and wide color gamut (WCG) imagery is now mainstream across content creation. Thus, having a reliable way of evaluating HDR systems is essential. One common way of measuring the quality of an HDR system is measuring the color errors introduced along the imaging pipeline. Unfortunately, development of color difference metrics has been based on test patches, as opposed to natural imagery, with evaluation mostly limited to standard dynamic range (SDR) databases. In this paper, we evaluate several color difference metrics on five publicly available HDR distortion databases consisting of natural images and subjective scores. Distortion types include lower frequency distortions such as those from tone mapping as well as higher-frequency distortions resulting from compression artifacts by the various compression schemes such as Joint Photographic Experts Group (JPEG). We analyze these databases with the well-established CIE L^\star a^\star b^\star metrics ( \Delta E_00, \Delta E_94) as well as two HDR-specific metrics: \Delta E_z (j_za_zb_z) and \Delta E_ITP (IC_TC_P). To quantify the performance, we use four standard performance evaluation procedures. We observe that \Delta E_ITPoutperforms the other color difference metrics on four out of five databases. We also perform statistical analysis, which demonstrates the effectiveness of using \Delta E_ITPto assess the overall image quality of HDR/WCG content. Weaknesses of all metrics for one database suggest that more advanced spatio-chromatic visual models are worth pursuing.
Publication Date: 2020-10-01
DOI: 10.5594/JMI.2020.3010390
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2020.3010390
Author(s): Anustup Choudhury
bio
Anustup Choudhury received PhD and MS degrees in computer science from the University of Southern California (USC), Los Angeles, CA. He currently works at Applied Vision Science Group, Dolby Laboratories, Sunnyvale, CA. His research interests include image/video analysis and processing, pattern recognition, computer vision, and machine learning. His work has been published in various reputed international conferences and journals, and some of his papers have received special recognition. He is also an inventor of a dozen patents.
Jaclyn Pytlarz
bio
Jaclyn Pytlarz is a staff researcher at Dolby Laboratories, Sunnyvale, CA, where she has worked in Dolby’s Applied Vision Science Group, since 2014. Her research interest includes vision science surrounding technologies for high dynamic range and wide color gamut displays. As the display management core algorithm lead, her focus is on developing color mapping and display management algorithms to help maintain consistent imagery on a wide variety of displays. She holds a BS degree in motion picture science from the Rochester Institute of Technology, Rochester, NY, and is currently pursuing an MS degree in computational and mathematical engineering at Stanford University, Stanford, CA.
Scott Daly
bio
Scott Daly is an applied perception scientist at Dolby Laboratories, Sunnyvale, CA, with specialties in spatial, temporal, and chromatic vision. He has significant experience in applications toward display engineering, image processing, and video compression with over 100 technical papers. Current focal areas include high dynamic range, auditory-visual interactions, physiological assessment, and preserving artistic intent. He has a BS degree in bioengineering from North Carolina State University (NCSU), Raleigh, NC, and an MS degree in bioengineering from the University of Utah, Salt Lake City, UT. Past accomplishments led to the Otto Schade award from the Society for Information Display (SID) in 2011, and a team technical Emmy in 1990. He is a member of the IEEE, SID, and SMPTE. He recently completed the 100-patent dash in just under 30 years.
Keyword(s): ΔEITP, Color Difference Metric, HDR, ICTCP, Wide Color Gamut
Copyright: © 2020 SMPTE

Bibliographic Reference(s)

Perceptual Signal Coding for More Efficient Usage of Bit Codes [Active]
Hitting the Mark—A New Color Difference Metric for HDR and WCG Imagery [Active]
An Assessment of Reference Levels in HDR Content [Active]
Black Level Visibility as a Function of Ambient Illumination [Active]
B. Keelan, “Predicting Multivariate Image Quality From Individual Perceptual Attributes,” Proc. IS&T’s 2002 PICS Conf., pp. 82–87, 2002. EXTERNAL
T. Kunkel et. al., “Assessing Color Discernibility in HDR Imaging Using Adaptation Hulls,” IS&T CIC, pp. 336–343, 2018. EXTERNAL
D. H. Alman et al., “Performance Testing of Color Difference Metrics Using a Color Tolerance Dataset,” COLOR Res. Appl., 14:139–151, Jun. 1989. EXTERNAL
A. Choudhury and S. Daly, “Comparing a Spatial Extension of ICTCP Color Representation With S-CIELAB and Other Recent Color Metrics for HDR and WCG Quality Assessment,” Electronic Imaging, Color Imaging XXIV: Displaying, Processing, Hardcopy, and Applications, 2020. EXTERNAL
E. Kirchner and N. Dekker, “Performance Measures of Color Difference Equations: Correlation Coefficient Versus Standardized Residual Sum of Squares,” J. Opt. Soc. Am. A 28:1841–1848, 2011. EXTERNAL
R. Dror, A. Willsky, and E. Adelson, “Statistical Characterization of Real-World Illumination,” J. Vis. 4(9):821–837, 2004. EXTERNAL
M. Narwaria et al., “Tone Mapping-Based High-Dynamic-Range Image Compression: Study of Optimization Criterion and Perceptual Quality,” Opt. Eng. 52:102008, 2013. EXTERNAL
E. Reinhard et al., “Photographic Tone Reproduction for Digital Images,” ACM Trans. Graph., 21(3):267–276, Jul. 2002. EXTERNAL
VQEG, “Final Report From the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment,” 2003. EXTERNAL
T. C. Aydin, R. Mantiuk, and H.-P. Seidel, “Extending Quality Metrics to Full Luminance Range Images,” Proc. SPIE, pp. 109–118, 2008. EXTERNAL
International Telecommunication Union – Radiocommunication (ITU-R) BT.2124, “Objective Metric for the Assessment of the Potential Visibility of Colour Differences in Television.” EXTERNAL
M. Rousselot et. al., “Quality Assessment of HDR/WCG Images Using HDR Uniform Color Spaces,” J. Imag. 5(1):1–18 2019. EXTERNAL
P. A. García et al., “Measurement of the Relationship Between Perceives and Computed Color Differences,” Opt. Soc. Am., 24(7):1823–1829, Jul. 2007. EXTERNAL
International Organization for Standardization/International Commission on Illumination (ISO/CIE) 11664-6, “2014(E) Colorimetry—Part 6: CIEDE2000 Colour-Difference Formula.” EXTERNAL
Z. Mai et al., “Optimizing a Tone Curve for Backward-Compatible High Dynamic Range Image and Video Compression,” IEEE Trans. Image Process. 20(6):1558–1571, Jun. 2011. EXTERNAL
P. Korshunov et al., “Subjective Quality Assessment Database of HDR Images Compressed With JPEG XT,” QoMEX, pp. 1–6, May 2015. EXTERNAL
M. Safdar et al., “Perceptually Uniform Color Space for Image Signals Including High Dynamic Range and Wide Gamut,” Opt. Express, 25(13):15131–15151, Jun. 2017. equivalent to all the color difference metrics that we considered. Likewise, although ΔEITP outperforms other color difference metrics on Database 4 (Table 5), its performance is statistically equivalent to the other color difference metrics. One of the reasons that the performance of ΔEITP is not superior for Database 4 could be because the size of the database is relatively small (96 images). In addition, Database 4 was the one where all the metrics underperformed, possibly as a result of the widely varying color distortions requiring a more advanced model than averaging across the entire image without any frequency-weighting. Conclusion and Future Work In this paper, we evaluate several color difference metrics to assess the quality of HDR content. In particular, we consider the two most commonly used color difference metrics: ΔE00 and ΔE94. Additionally, we evaluate the performance of two color difference metrics designed with HDR/WCG content in mind: ΔEz and ΔEITP. We use five different databases containing a wide variety of distortions for evaluation and show that overall, ΔEITP outperforms the other color difference metrics on four out of five databases. We also perform statistical analyses and observed that ΔEITP EXTERNAL
International Organization for Standardization/International Commission on Illumination (ISO/CIE) 11664-4, “2019(E) Colorimetry—Part 4: CIE 1976 L*a*b* Colour Space.” EXTERNAL
“Tone-Mapping Using Matlab.” Accessed: 5 Sep. 2019. [Online]. Available: https://www.mathworks.com/help/images/ref/localtonemap.html EXTERNAL
M. Fairchild, P. H. Chen, “Brightness, Lightness, and Specifying Color in High-Dynamic-Range Scenes and Images,” Proc. SPIE, pp. 233–246, 7867: 2011. EXTERNAL
R. Wanat et al., “Spatiochromatic Model for Image Quality Prediction of High Dynamic Range and Wide Color Gamut Content,” Proc. SID Symp. Dig. Tech. Papers, 2020. EXTERNAL
E. Zerman, G. Valenzise, and F. Dufaux, “An Extensive Performance Evaluation of Full-Reference HDR Image Quality Metrics,” Qual. User Experience, 2(1):5, Apr. 2017. EXTERNAL
G. Valenzise et al., “Performance Evaluation of Objective Quality Metrics for HDR Image Compression,” SPIE Optical Engineering + Applications, International Society for Optics and Photonics, p. 92170C, 2014. EXTERNAL
M. Rousselot et al., “Impacts of Viewing Conditions on HDRvdp2,” Proc. EUSIPCO, pp. 1442–1446, 2018. EXTERNAL
J. Kuang, G. M. Johnson, and M. D. Fairchild, “iCAM06: A Refined Image Appearance Model for HDR Image Rendering,” J. Vis. Comm. Image Represent., 18(5):406–414, 2007. EXTERNAL
S. Y. Choi et al., “Investigation of Large Display Color Image Appearance I: Important Factors Affecting Perceived Quality” J. Inf. Syst. Telecomm., 52(4): 40904-1–40904-11, 2008. EXTERNAL
International Telecommunication Union – Telecommunication (ITU-T), “Methods, Metrics and Procedures for Statistical Evaluation, Qualification and Comparison of Objective Quality Prediction Models,” 2012. EXTERNAL
R. Mantiuk, K. Myszkowski, and H.-P. Seidel, “A Perceptual Framework for Contrast Processing of High Dynamic Range Images,” ACM Trans. Appl. Percept., 3(3):286–308, Jul. 2006. EXTERNAL
A. Lahoulou et al., “Knowledge-Based Taxonomic Scheme for Full-Reference Objective Image Quality Measurement Models” J. Inf. Syst. Telecomm., 60(6): 60406-1–60406-15, Nov./Dec. 2016. EXTERNAL
M. Vranjes, S. Rimac-Drlje, and K. Grgic, “Review of Objective Video Quality Metrics and Performance Comparison Using Different Databases,” J. Image Comm., 28(1):1–19, Jan. 2013. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2020.3010390.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Anustup Choudhury, Jaclyn Pytlarz, and Scott Daly; HDR and WCG Image Quality Assessment Using Color Difference Metrics, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2020.3010390

Snippet:

Anustup Choudhury, Jaclyn Pytlarz, and Scott Daly; HDR and WCG Image Quality Assessment Using Color Difference Metrics, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2020.3010390

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Anustup Choudhury, Jaclyn Pytlarz, and Scott Daly; <cite>HDR and WCG Image Quality Assessment Using Color Difference Metrics</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at <a href="https://doi.org/10.5594/JMI.2020.3010390" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2020.3010390</a></span>

SMPTE's HTML Pub

Preview:

Anustup Choudhury, Jaclyn Pytlarz, and Scott Daly; HDR and WCG Image Quality Assessment Using Color Difference Metrics, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020
doi: 10.5594/JMI.2020.3010390
url: https://doi.org/10.5594/JMI.2020.3010390

Snippet:

<li>
Anustup Choudhury, Jaclyn Pytlarz, and Scott Daly; <cite id="bib-10-5594-jmi-2020-3010390">HDR and WCG Image Quality Assessment Using Color Difference Metrics</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020
<span class="doi">10.5594/JMI.2020.3010390</span>
</li>