Toward Generalized Psychovisual Preprocessing For Video Encoding

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 131, No. 4, pp. 39–44
Abstract: Deep perceptual preprocessing has recently emerged as a new way to enable further bitrate savings across several generations of video encoders without breaking standards or requiring any changes in client devices. In this article, we lay the foundation for a generalized psychovisual preprocessing framework for video encoding and describe one of its promising instantiations that is practically deployable for video-on-demand, live, gaming, and user-generated content (UGC). Results using state-of-the-art advanced video coding (AVC), high efficiency video coding (HEVC), and versatile video coding (VVC) encoders show that average bitrate [Bjontegaard delta-rate (BD-rate)] gains of 11%–17% are obtained over three state-of-the-art reference-based quality metrics [Netflix video multi-method assessment fusion (VMAF), structural similarity index (SSIM), and Apple advanced video quality tool (AVQT)], as well as the recently proposed nonreference International Telecommunication Union-Telecommunication?(ITU-T) P.1204 metric. The proposed framework on CPU is shown to be twice faster than × 264 medium-preset encoding. On GPU hardware, our approach achieves 714 frames/sec for 1080p video (below 2 ms/frame), thereby enabling its use in very-low-latency live video or game streaming applications.
Publication Date: 2022-05-01
DOI: 10.5594/JMI.2022.3160801
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2022.3160801
Author(s): Aaron Chadha
bio
Aaron Chadha received MA and MEng degrees (Hons.) in information and computer engineering from the University of Cambridge, Cambridge, U.K., in 2014, and a PhD degree in computer vision and machine learning from the University College London, London, U.K., in 2018. He is currently a senior machine learning engineer with iSIZE Technologies, London, U.K. His research interests include machine learning, image/video recognition, compression, and domain adaptation. He has experience in working with multiple modalities, including event-based neuromorphic vision sensing.
Mohammad Ashraful Anam
bio
Mohammad Ashraful Anam received a PhD degree in electronic engineering from the University College London, London, U.K., where he received the Lombardi Prize for the Best PhD thesis in electronic engineering. He is currently a senior software engineer at iSIZE Technologies. His research interests include machine learning, cloud computing and distributed systems, error tolerant computing, and high-performance cloud computing.
Matthias Treder
bio
Matthias Treder received a PhD in human pattern recognition from Radboud University Nijmegen, Nijmegen, The Netherlands, in 2009. He then worked on the application of machine learning to neuroimaging data at the Technical University of Berlin, Berlin, Germany, from 2009 to 2013; the University of Cambridge, Cambridge, U.K., from 2013 to 2015; and the University of Birmingham, Birmingham, U.K., from 2016 to 2017. He joined Cardiff University, Cardiff, U.K., in 2017 and became an assistant professor in computer science in 2018. In 2021, he joined iSIZE Technologies as a senior machine learning engineer.
Ilya Fadeev
bio
Ilya Fadeev studied cyber-security at Tver State University, Tver, Russia, from 2012 to 2017. He is currently a senior software engineer with iSIZE Technologies. He has a strong background in C++ and general-purpose graphics processing unit (GPGPU) programming, including hardware-specific algorithm and code optimizations. He has experience working with machine learning, computer vision, and video codecs.
Yiannis Andreopoulos
Keyword(s): Deep Neural Networks, Perceptual Optimization, Video Delivery
Copyright: © 2022 SMPTE

Bibliographic Reference(s)

S. Deng et al., “VMAF Based Rate-Distortion Optimization for Video Coding,” Proc. IEEE 22nd Intl. Workshop on Multimedia Signal Processing (MMSP), Sep. 2020. EXTERNAL
Q. Huynh-Thu and M. Ghanbari, “Scope of Validity of PSNR in Image/Video Quality Assessment,” Electronics Letters, 44(13):800–801, Feb. 2008. EXTERNAL
Z. Li, “Video @Scale 2020: VMAF,” 2020 At Scale Conference, Nov. 2020. EXTERNAL
A. Chadha et al., “Escaping the Complexity-Bitrate-Quality Barriers of Video Encoders via Deep Perceptual Optimization,” Proc. Applications of Digital Image Processing XLIII, 11510, pp. 38–52, Sep. 2021. EXTERNAL
S. L. Regunathan et al., “Efficient Measurement of Quality at Scale in Facebook Video Ecosystem,” Proc. Applications of Digital Image Processing XLIII, 11510, Aug. 2020. EXTERNAL
A. Raake et al., “Multi-Model Standard for Bitstream-, Pixel-Based and Hybrid Video Quality Assessment of UHD/4K: ITU-T P. 1204,” IEEE Access, 8:193020–193049, 2020. EXTERNAL
A. K. Venkataramanan et al., “A Hitchhiker’s Guide to Structural Similarity,” IEEE Access, 9:28872–28896, 2021, doi: 10.1109/ACCESS.2021.3056504. EXTERNAL
I. Katsavounidis and L. Guo, “Video Codec Comparison Using the Dynamic Optimizer Framework,” Proc. Applications of Digital Image Processing XLI, 10752:107520Q, Sep. 2018, doi: 10.1117/12.2322118. EXTERNAL
VMAF, “Code Repository,” Sep. 2021. [Online]. Available: https://github.com/Netflix/vmaf EXTERNAL
A. Chadha and Y. Andreopoulos, “Deep Perceptual Preprocessing for Video Coding,” Proc. IEEE/CVF Conf. Computer Vision Pattern Recognit, pp. 14852–14861, Jun. 2021. EXTERNAL
“Apple AVQT Presentation,” 2021 Apple Worldwide Developers Conference (WWDC), Sep. 2021, doi: 10.1109/ EXTERNAL
T-REC-P.910 — ITU-T Recommendation P.910, “Subjective Video Quality Assessment Methods for Multimedia Applications,” International Telecommunication Union, Geneva, Switzerland, Apr. 2008. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3160801.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Aaron Chadha, Mohammad Ashraful Anam, Matthias Treder, Ilya Fadeev, and Yiannis Andreopoulos; Toward Generalized Psychovisual Preprocessing For Video Encoding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3160801

Snippet:

Aaron Chadha, Mohammad Ashraful Anam, Matthias Treder, Ilya Fadeev, and Yiannis Andreopoulos; Toward Generalized Psychovisual Preprocessing For Video Encoding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3160801

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Aaron Chadha, Mohammad Ashraful Anam, Matthias Treder, Ilya Fadeev, and Yiannis Andreopoulos; <cite>Toward Generalized Psychovisual Preprocessing For Video Encoding</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3160801" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3160801</a></span>

SMPTE's HTML Pub

Preview:

Aaron Chadha, Mohammad Ashraful Anam, Matthias Treder, Ilya Fadeev, and Yiannis Andreopoulos; Toward Generalized Psychovisual Preprocessing For Video Encoding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3160801
url: https://doi.org/10.5594/JMI.2022.3160801

Snippet:

<li>
Aaron Chadha, Mohammad Ashraful Anam, Matthias Treder, Ilya Fadeev, and Yiannis Andreopoulos; <cite id="bib-10-5594-jmi-2022-3160801">Toward Generalized Psychovisual Preprocessing For Video Encoding</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022
<span class="doi">10.5594/JMI.2022.3160801</span>
</li>