Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 132, No. 3, pp. 18–26
Abstract: In the context of convolutional neural network (CNN)-based video compressions, motivated by the lower acuity of the human visual system for color differences when compared with luma, we investigate a video compression framework using autoencoder networks that encode and decode videos by using less chroma information than luma information. For this purpose, instead of converting Y’CbCr 4:2:2/4:2:0 videos to and from RGB 4:4:4 as per the current state of the art, we have kept the video in Y’CbCr 4:2:2/4:2:0 and merged the luma and chroma channels after the luma is downsampled to match the chroma size. We have performed an inverse function for the decoder. The performance of our models against the 4:4:4 baseline is evaluated by using color peak signal-to-noise ratio (CPSNR), multiscale structural similarity (MS-SSIM), and video multimethod assessment fusion (VMAF) metrics. Our experiments reveal that, when compared to video compression involving conversion to and from RGB 4:4:4, the proposed method increases the video quality by about 5% for Y’CbCr 4:2:2 and 6% for Y’CbCr 4:2:0 while reducing the amount of computation by nearly 37% for Y’CbCr 4:2:2 and 40% for Y’CbCr 4:2:0. These results point us to optimization for 4:2:2 and 4:2:0 video of the current state-of-the-art autoencoder.
Publication Date: 2023-04-01
DOI: 10.5594/JMI.2023.3245675
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2023.3245675
Author(s): Vahid Khorasani Ghassab
bio
Vahid Khorasani Ghassab has a BSc degree in pure mathematics from the Ferdowsi University of Mashhad, Mashhad, Iran, an MSc degree in electronic engineering from the Polytechnic University of Turin, Turin, Italy, and a PhD degree in information and systems engineering from Concordia University, Montreal, QC, Canada. He was a researcher with Avid Technology Inc., Montreal, and a post-doctoral fellow with the Concordia Institute for Information Systems Engineering, Concordia University. He is currently an Artificial Intelligence (AI) engineer with IPtoki Inc., Montreal. His current research interests include computer vision and machine learning (ML). He has been a member of IEEE for six years.
Rob Gonsalves
bio
Rob Gonsalves joined Avid Technology, as their 15th employee in 1989. He helped to develop the industry’s preeminent nonlinear editing system Avid Media composer, specializing in programming video effects and color correction. His work on multicamera editing software led to Avid winning the Technology and Engineering Emmy Award. He later spearheaded the development of the architecture of Avid’s media enterprise products. He is currently researching the use of AI for media production. He holds over 50 patents in the media and entertainment industry.
Shailendra Mathur
bio
Shailendra Mathur is responsible for the technology and architecture of Avid’s products in the role of vice president and chief architect. He has been awarded several patents and presented papers at numerous conferences, panels, and journals on subjects such as AI, computer vision, medical imaging, media creation tools, security, immersive entertainment, asset management, 4K/high dynamic range (HDR), high-performance computing, and cloud technologies. With more than 25 years of innovation experience in the media industry, he was privileged to be one of the team members who received the Philo T. Farnsworth Prime Time Engineering Emmy Award on behalf of Avid in 2018.
Nizar Bouguila
bio
Nizar Bouguila has an engineering degree from the University of Tunis, Tunis, Tunisia, in 2000, and MSc and PhD degrees in computer science from Sherbrooke University, Sherbrooke, QC, Canada, in 2002 and 2006, respectively. He is currently a professor with the Concordia Institute for Information Systems Engineering (CIISE), Concordia University, Montreal, QC, Canada. His research interests include image processing, ML, data mining, computer vision, pattern recognition, smart buildings, and energy. He is a licensed professional engineer registered in Ontario and a Senior Member of the IEEE for 12 years.
Keyword(s): Avid, Chroma Subsampling, Concordia, CNN, Machine Learning, Neural Network, Video Compression
Copyright: © 2023 SMPTE

Bibliographic Reference(s)

VC-3 Codec Updates for Handling Better, Faster, and More Pixels [Active]
Video Compression Using Convolutional Neural Networks of Video with Chroma Subsampling [Active]
Y. Chen et al., “An Overview of Coding Tools in AV1: The First Video Codec From the Alliance for Open Media,” APSIPA Trans. Signal and Inform. Process., p. 9, 2020. EXTERNAL
F. Mentzer, G. Toderici, M. Tschannen, and E. Agustsson, “Highfidelity Generative Image Compression,” 2020, arXiv preprint arXiv:2006.09965. EXTERNAL
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Trans. Circuits Syst. Video Technol., 22(12):1649–1668, Dec. 2012. EXTERNAL
R. Song, D. Liu, H. Li, and F. Wu, “Neural Network-Based Arithmetic Coding of Intra Prediction Modes in HEVC,” Proc. 2017 IEEE Visual Commun. Image Process. (VCIP), IEEE, pp. 1–4, 2017. EXTERNAL
F. Mentzer et al., “VCT: A Video Compression Transformer,” 2022, arXiv preprint arXiv:2206.07307. EXTERNAL
“Video Multi-Method Assessment Fusion (VMAF).” Accessed: Feb. 2023. [Online]. Available: https://github.com/Netflix/vmaf EXTERNAL
O. Rippel, S. Nair, C. Lew, S. Branson, A. G. Anderson, and EXTERNAL
S. Wang, K. Gu, S. Ma, and W. Gao, “Joint Chroma Downsampling and Upsampling for Screen Content Image,” IEEE Trans. Circuits Syst. Video Technol., 26(9):1595–1609, 2015. EXTERNAL
K.-L. Chung, C.-C. Huang, and T.-C. Hsu, “Adaptive Chroma Subsampling-Binding and Luma-Guided Chroma Reconstruction Method for Screen Content Images,” IEEE Trans. Image Process., 26(12):6034–6045, Dec. 2017. EXTERNAL
S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” Int. Conf. Mach. Learning, PMLR, pp. 448–456, 2015. EXTERNAL
“Netflix Public Database.” Accessed: Feb. 2023. [Online]. Available: https://github.com/Netflix/vmaf/blob/master/resource/dataset EXTERNAL
T. Lin, P. Zhang, S. Wang, K. Zhou, and X. Chen, “Mixed Chroma Sampling-Rate High Efficiency Video Coding for Full-Chroma Screen Content,” IEEE Trans. Circuits Syst. Video Technol., 23(1):173–185, 2012. EXTERNAL
A. Krizhevsky and G. Hinton, “Learning Multiple Layers of Features from Tiny Images,” Apr. 2009. EXTERNAL
S. H. Lee and N. I. Cho, “Intra Prediction Method Based on the Linear Relationship Between the Channels for YUV 4:2:0 Intra Coding,” Proc. 2009 16th IEEE Int. Conf. Image Process. (ICIP), IEEE, pp. 1037–1040, 2009. EXTERNAL
B. C. Song, Y. G. Lee, and N. H. Kim, “Block Adaptive Inter-Color Compensation Algorithm for RGB 4:4:4 Video Coding,” IEEE Trans. Circuits Syst. Video Technol., 18(10):1447–1451, 2008. EXTERNAL
T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC Video Coding Standard,” IEEE Trans. Circuits Syst. Video Technol., 13(7):560–576, Jul. 2003. EXTERNAL
Y.-H. Tsai, M.-Y. Liu, D. Sun, M.-H. Yang, and J. Kautz, “Learning Binary Residual Representations for Domain-Specific Video Streaming,” Proc. AAAI Conf. Artificial Intell., vol. 32, pp. 7363–7370, 2018. EXTERNAL
Z. Chen, T. He, X. Jin, and F. Wu, “Learning for Video Compression,” IEEE Trans. Circuits Syst. Video Technol., 30(2):566– 576, Jan. 2019. EXTERNAL
T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma, “Deepcoder: A Deep Neural Network Based Video Compression,” Proc. 2017 IEEE Visual Commun. and Image Process. (VCIP), IEEE, pp. 1–4, 2017. EXTERNAL
Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale Structural Similarity for Image Quality Assessment,” The Thrity-Seventh Asilomar Conf. Signals, Syst. Comput., IEEE, vol. 2, pp. 1398–1402, 2003. EXTERNAL
D. A. Kerr, “Chrominance Subsampling in Digital Images,” Pumpkin, 2012(3):1–15, Jan. 2012. EXTERNAL
R. Yang, F. Mentzer, L. V. Gool, and R. Timofte, “Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement,” Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognit., pp. 6628–6637, 2020. EXTERNAL
S. Zhu, C. Cui, R. Xiong, Y. Guo, and B. Zeng, “Efficient Chroma Sub-Sampling and Luma Modification for Color Image Compression,” IEEE Trans. Circuits Syst. Video Technol., 29(5):1559–1563, 2019. EXTERNAL
G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “DVC: An End-to-End Deep Video Compression Framework,” Proc. IEEE/CVF Conf. Comput. Vision Pattern Recognit., pp. 11006– 11015, 2019. EXTERNAL
A. Habibian, T. van Rozendaal, J. M. Tomczak, and T. S. Cohen, “Video Compression With Rate-Distortion Autoencoders,” Proc. IEEE/CVF Int. Conf. Comput. Vision, pp. 7033–7042, 2019. EXTERNAL
Avid DNxHD, XML Pull List, X Mac OS, and XP Windows, “Avid-Based Workflow.” EXTERNAL
D. Mukherjee, S. Li, Y. Chen, A. Anis, S. Parker, and J. Bankoski, “A Switchable Loop-Restoration With Side-Information Framework for the Emerging AV1 Video Codec,” Proc. 2017 IEEE Int. Conf. Image Process. (ICIP), IEEE, pp. 265–269, 2017. EXTERNAL
C. Poynton, Digital Video and HD: Algorithms and Interfaces, Elsevier: Toronto, Canada, 2012. EXTERNAL
Z. Liu, X. Yu, Y. Gao, S. Chen, X. Ji, and D. Wang, “Cu Partition Mode Decision for HEVC Hardwired Intra Encoder Using Convolution Neural Network,” IEEE Trans. Image Process., 25(11):5088–5103, Nov. 2016. EXTERNAL
F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and EXTERNAL
G. Lu, W. Ouyang, D. Xu, X. Zhang, Z. Gao, and M.-T. Sun, “Deep Kalman Filtering Network for Video Compression Artifact Reduction,” Proc. European Conf. Comput. Vision (ECCV), pp. 568–584, 2018. EXTERNAL
C.-Y. Wu, N. Singhal, and P. Krahenbuhl, “Video Compression Through Image Interpolation,” Proc. European Conf. Comput. Vision (ECCV ), pp. 416–431, 2018. EXTERNAL
A. Wieckowski et al., “Towards a Live Software Decoder Implementation for the Upcoming Versatile Video Coding (VVC) Codec,” Proc. 2020 IEEE Int. Conf. Image Process. (ICIP), IEEE, pp. 3124–3128, 2020. EXTERNAL
W. Gish and S. Miller, “Unambiguous Video Pipeline Description Motivated by HDR,” 2016 IEEE Int. Conf. Image Process. (ICIP), IEEE, pp. 909–912, 2016. EXTERNAL
K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” Proc. IEEE Conf. on Comput. Vision and Pattern Recognit., pp. 770–778, 2016. EXTERNAL
ITU-R BT.2020-2 [Active]

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2023.3245675.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Vahid Khorasani Ghassab, Rob Gonsalves, Shailendra Mathur, and Nizar Bouguila; Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 3, April 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3245675

Snippet:

Vahid Khorasani Ghassab, Rob Gonsalves, Shailendra Mathur, and Nizar Bouguila; Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 3, April 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3245675

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Vahid Khorasani Ghassab, Rob Gonsalves, Shailendra Mathur, and Nizar Bouguila; <cite>Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 3, April 2023); SMPTE, 2023. Available at <a href="https://doi.org/10.5594/JMI.2023.3245675" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2023.3245675</a></span>

SMPTE's HTML Pub

Preview:

Vahid Khorasani Ghassab, Rob Gonsalves, Shailendra Mathur, and Nizar Bouguila; Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 3, April 2023); SMPTE, 2023
doi: 10.5594/JMI.2023.3245675
url: https://doi.org/10.5594/JMI.2023.3245675

Snippet:

<li>
Vahid Khorasani Ghassab, Rob Gonsalves, Shailendra Mathur, and Nizar Bouguila; <cite id="bib-10-5594-jmi-2023-3245675">Optimizing Video Compression With CNN-Based Autoencoders With Chroma Subsampling</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 3, April 2023); SMPTE, 2023
<span class="doi">10.5594/JMI.2023.3245675</span>
</li>