AI-Based Saliency-Aware Video Coding

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 131, No. 4, pp. 21–29
Abstract: The demand for video through over-the-top (OTT) has been constantly increasing in recent years. During the COVID-19 pandemic, demand skyrocketed, hence leading to the need for better video compression. The human visual system (HVS) can quickly select visually important regions in its visual field. These regions are captured at high resolution, while other peripheral regions receive little attention. Saliency maps are a way to imitate the HVS attention mechanism. Recently, deep learning-based saliency models have achieved tremendous improvements. This article leverages state-of-the-art deep learning-based saliency models to improve video coding efficiency. First, a saliency-based rate control scheme is integrated in a high-efficiency video encoder (HEVC). Then, a saliency-guided preprocessing filtering step is introduced. Finally, the two approaches are combined. Objective and subjective evaluations show that it can lower the bitrate from 6% to almost 30% while maintaining the same visual quality.
Publication Date: 2022-05-01
DOI: 10.5594/JMI.2022.3160541
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2022.3160541
Author(s): Sébastien Pelurson
bio
Sébastien Pelurson received a PhD degree in computer science from the University of Grenoble Alpes, Grenoble, France, in 2016. From 2016 to 2019, he worked in the field of augmented reality, and more specifically on the use of deep learning models to improve 3D tracking algorithms on mobile devices. He joined Ateme, Rennes, France, in 2020. His research interests include video coding optimization based on machine learning technology.
Josselin Cozanet
bio
Josselin Cozanet received an engineering diploma from École nationale supérieure d’ingénieurs de Caen & Centre de Recherche (ENSICAEN) National Graduate School of Engineering, Caen, Rennes, France, in 2019. He started working at Ateme, Rennes, in 2019, for an internship in the innovation team on artificial intelligence used to improve encoding latency. He is still working on various topics at Ateme, including artificial intelligence applied to video compression and content delivery network intercommunication with open caching.
Thomas Guionnet
bio
Thomas Guionnet is a fellow research engineer at Ateme, Rennes, France, where he currently leads the Innovation team’s research on artificial intelligence applied to video compression. Beyond his work for Ateme, he has also contributed to the International Standards Organization/Motion Picture Experts Group (ISO/MPEG) - International Telecommunication Union-Telecommunication/Video Coding Experts Group (ITU-T/VCEG) - Versatile Video Coding (VVC), HEVC, and HEVC-3D standardization process; he teaches video compression at the ESIR engineering school in Rennes; and he has authored numerous publications including patents, international conference papers, and journal papers. Prior to joining Ateme, he spent ten years at Envivio conducting research on realtime encoding, video preprocessing, and video quality assessment. He holds a PhD from University of Rennes 1, Rennes.
Mohsen Abdoli
bio
Mohsen Abdoli received a master of engineering degree from Sharif University of Technology, Tehran, Iran, in 2013, and a PhD jointly from Université Paris-Saclay and CentraleSupélec, Paris, France, in 2018. In 2015, he joined Orange Labs, Rennes, France, where he actively contributed to the development of several video compression techniques, targeting the VVC standard by JVET, particularly in the domain of intraprediction and residual coding of natural/screen content. In 2018, he joined Ateme, Rennes, where he contributed to the world-first implementation of realtime VVC transcoder for video-on-demand and live streaming applications. He is currently with IRT b<> com, Rennes, holding the position of standardization engineer. His research interests include areas of signal processing, encoder optimization, machine learning, and quality assessment.
Thibaud Biatek
bio
Thibaud Biatek received a PhD degree in signal and image processing from the Institut National des Sciences Appliquées, Rennes, France, in 2016. From 2013 to 2017, he was a doctoral and post-doctoral fellow with TDF, Cesson-Sévigné, France. From 2017 to 2019, he was a video coding expert with TDF, where he was involved in MPEG and DVB standardization activities. In 2019, he was a senior engineer with Qualcomm, San Diego, CA, working on VVC standardization. Since 2020, he has been director of technology and standards at Ateme, working on partnership projects and multimedia standards, contributing to MPEG, DVB, and 3GPP groups. His research interests include compression, processing, and delivery of audiovisual signals over broadcast and broadband networks.
Keyword(s): Artificial Intelligence, Saliency, Video Coding
Copyright: © 2022 SMPTE

Bibliographic Reference(s)

L. Zhang et al., “SUN: A Bayesian Framework for Saliency Using Natural Statistics,” J. Vision, 8(7):32, Dec. 2008. EXTERNAL
C. Qin et al., “Integration of the Saliency-Based Seed Extraction and Random Walks for Image Segmentation,” Neurocomputing, Elsevier: Amsterdam, The Netherlands, 129:378–391, Apr. 2014. EXTERNAL
J. Pan et al., “Salgan: Visual Saliency Prediction With Generative Adversarial Networks,” arXiv preprint arXiv:1701.01081. EXTERNAL
Z. Sun et al., “Real-Time Video Saliency Prediction via 3D Residual Convolutional Neural Network,” IEEE Access, 7:147743–147754, Oct. 2019. EXTERNAL
K. Xu et al., “Show, Attend and Tell: Neural Image Caption Generation With Visual Attention,” Proc. Intl. Conf. Mach. Learn. (PMLR), pp. 2048–2057, 2015. EXTERNAL
A. Polakovi˘c et al., “An Approach to Video Compression Using Saliency Based Foveation,” Proc. 2018 Intl. Symp. ELMAR, pp. 169–172, Sep. 2018. EXTERNAL
L. Itti, “Automatic Foveation for Video Compression Using a Neurobiological Model of Visual Attention,” IEEE Trans. Image Processing, 13(10):1304–1318, Oct. 2004. EXTERNAL
R. Gupta et al., “A Scheme for Attentional Video Compression,” Intl. Conf. Pattern Recognit. Mach. Intelligence, Springer: Berlin, Heidelberg, pp. 458–465, 2011. EXTERNAL
International Telecommunication Union-Telecommunication (ITU-T), “H.266: Versatile Video Coding,” [Online]. Available: https://www.itu.int/rec/T-REC-H.266 EXTERNAL
International Telcommunication Union-Radiocommunication (ITU-R), “Methodologies for the Subjective Assessment of Quality of Television Images.” EXTERNAL
S. Kruthiventi et al., “Deepfix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations,” IEEE Trans. Image Process., 26(9):4446–4456, Jun. 2017. EXTERNAL
T. Judd et al., “A Benchmark of Computational Models of Saliency to Predict Human Fixations,” MIT Tech Report, 2012. EXTERNAL
H. Fu et al., “Object-Based Multiple Foreground Segmentation in RGBD Video,” IEEE Trans. Image Process., 26(3):1418–1427, Jan. 2017. EXTERNAL
Y. Fang et al., “A Video Saliency Detection Model in Compressed Domain,” IEEE Trans. Circuits Systems Video Technol., 24(1):27–38, Jan. 2014. EXTERNAL
K. Simonyan et al., Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv preprint arXiv:1409.1556. EXTERNAL
T. Judd et al., “Learning to Predict Where Humans Look,” Proc. IEEE 12th Intl. Conf. Comput. Vision, pp. 2106–2113, 2009. EXTERNAL
C. Koch et al., “Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry,” in Matters of Intelligence, Springer: Dordrecht, The Netherlands, pp. 115–141, 1997. EXTERNAL
A. Kroner et al., “Contextual Encoder–Decoder Network for Visual Saliency Prediction,” Neural Networks, 129:261–270. EXTERNAL
D. Gao et al., “Discriminant Saliency for Visual Recognition From Cluttered Scenes,” Advances in Neural Information Processing Systems, 17, 481–488, Jan 2004. EXTERNAL
C. Ma et al., “A Saliency Prior Context Model for Real-Time Object Tracking,” IEEE Trans. Multimedia, 19(11):2415–2424, Apr. 2017. EXTERNAL
Digital Video Broadcasting-Multi Adaptive Bit Rate (DVB -MABR), “DVB-MABR: Adpative Media Streaming over IP Multicast,” [Online]. Available: https://dvb.org/?standard=adaptive-media-streaming-over-ip-multicast EXTERNAL
J. Harel et al., “Graph-Based Visual Saliency,” Advances in Neural Information Processing Systems, 19, 2006. EXTERNAL
E. Vig et al., “Large-Scale Optimization of Hierarchical Features For Saliency Prediction in Natural Images,” Proc. IEEE Conf. Comput. Vision Pattern Recognit., pp. 2798–2805, Jun. 2014. EXTERNAL
H. Wei et al., “Visual Saliency Based Perceptual Video Coding in HEVC,” 2016 IEEE Intl. Symp. Circuits Syst. (ISCAS), pp. 2547–2550, May 2016. EXTERNAL
N. Liu et al., “Learning to Predict Eye Fixations via Multiresolution Convolutional Neural Networks,” IEEE Trans. Neural Netw. Learn. Syst., 29(2):392–404, Nov. 2016. EXTERNAL
N. Bruce et al., “Saliency, Attention, and Visual Search: An Information Theoretic Approach,” J. Vision, 9(3):5, Mar. 2009. EXTERNAL
Cisco, “VNI Report 2021,” [Online]. Available: https://www.cisco.com/c/dam/m/en_us/solutions/service-provider/vni-fore-cast-highlights/pdf/Global_2021_Forecast_Highlights.pdf EXTERNAL
Alliance for Open Media, “AV1 Bitstream & Decoding ProcessSpecification,” Jan. 2019. [Online]. Available: https://aomediacodec.github.io/av1-spec/av1-spec.pdf. EXTERNAL
C. Bak et al., “Spatio-Temporal Saliency Networks for Dynamic Saliency Prediction,” IEEE Trans. Multimedia, 20(7):1688–1698, Jul. 2018. EXTERNAL
P. Wennersten et al., “GOP-Based Temporal Filter Improvements,” Joint Video Experts Team (JVET), 22nd Meeting, by Teleconference, 20–28 Apr. 2021, JVET-V0056. EXTERNAL
V. Lyudvichenko et al. “Improving Video Compression With Deep Visual-Attention Models,” Proc. 2019 Intl Conf. Intelligent Medicine Image Process., pp. 88–94, Apr. 2019. EXTERNAL
M. Tangemann, M. Kümmerer, T. S. Wallis, and M. Bethge, “Measuring the Importance of Temporal Features in Video Saliency,” European Conf. Computer Vision, Springer: Cham, Switzerland, pp. 667–684, Aug. 2020. EXTERNAL
H. Lee et al., “Salient Region-Based Online Object Tracking,” Proc. 2018 IEEE Winter Conf. Appl. Comput. Vision (WACV), pp. 1170–1177, Mar. 2018. EXTERNAL
S. Zhu et al., “Spatiotemporal Visual Saliency Guided Perceptual High Efficiency Video Coding With Neural Network,” Neurocomputing, 275:511–522, Jan. 2018. EXTERNAL
A. Borji and L. Itti, “State-of-the-Art in Visual Attention Modeling,” IEEE Trans. Pattern Analysis Mach. Intelligence, 35(1):185–207, Apr. 2012. EXTERNAL
Y. Li et al., “The Secrets of Salient Object Segmentation,” Proc. 2014 IEEE Conf. Comput. Vision Pattern Recognit., pp. 280– 287, Jun. 2014. EXTERNAL
L. Itti et al., “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis,” IEEE Trans. Pattern Analysis Mach. Intelligence, 20(11): 1254–1259, Nov. 1998. EXTERNAL
Z. Li et al., “Visual Attention Guided Bit Allocation in Video Compression,” Image and Vision Computing, 29(1):1–14, Jan. 2011. EXTERNAL
Netflix, “Toward A Practical Perceptual VideoQuality Metric,” [Online]. Available: https://netflixtechblog.com/toward-a-practical-perceptual-video-quality-metric-653f208b9652 EXTERNAL
X. Huang et al., “Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks,” Proc. IEEE Intl. Conf. Comput. Vision, pp. 262–270, Dec. 2015. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3160541.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Sébastien Pelurson, Josselin Cozanet, Thomas Guionnet, Mohsen Abdoli, and Thibaud Biatek; AI-Based Saliency-Aware Video Coding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3160541

Snippet:

Sébastien Pelurson, Josselin Cozanet, Thomas Guionnet, Mohsen Abdoli, and Thibaud Biatek; AI-Based Saliency-Aware Video Coding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3160541

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Sébastien Pelurson, Josselin Cozanet, Thomas Guionnet, Mohsen Abdoli, and Thibaud Biatek; <cite>AI-Based Saliency-Aware Video Coding</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3160541" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3160541</a></span>

SMPTE's HTML Pub

Preview:

Sébastien Pelurson, Josselin Cozanet, Thomas Guionnet, Mohsen Abdoli, and Thibaud Biatek; AI-Based Saliency-Aware Video Coding, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3160541
url: https://doi.org/10.5594/JMI.2022.3160541

Snippet:

<li>
Sébastien Pelurson, Josselin Cozanet, Thomas Guionnet, Mohsen Abdoli, and Thibaud Biatek; <cite id="bib-10-5594-jmi-2022-3160541">AI-Based Saliency-Aware Video Coding</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 4, May 2022); SMPTE, 2022
<span class="doi">10.5594/JMI.2022.3160541</span>
</li>