Optimizing UHD Production with Super Resolution Techniques

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 131, No. 10, pp. 45–54
Abstract: Super resolution (SR) is a technique that is getting increased attention. It was first deployed in high-end TVs to upconvert content, and now content providers are looking at how it can be used on the production side to guarantee the same high quality on all devices. Different techniques can be used to apply SR to standard definition (SD) and high definition (HD) content for transmitting ultra high definition (UHD) resolution. During a recent trial with France TV at the French Open, SD content with a 16:9 aspect ratio was processed offline to create UHD content, using a hybrid pipeline of deep restoration and SR methods. A pretrained deep neural network was used to produce an intermediate upscaled and deblurred HD version, which was then passed on to a faster, less memory-intensive, and machine-learning (ML)-based SR filter to produce the final UHD 4K resolution output. The result is a significantly better subjective viewing experience in terms of details and sharpness compared with classical filters used in live systems. Adequately addressing analog tape recordings, which have been digitized, is a significantly bigger technical challenge, which will need to be resolved in the future. While a one-model-fits-all approach for video enhancement sounds attractive due to its simplicity, the best visual quality is produced by applying purpose-built models trained for addressing the general types of degradation found in the source video.
Publication Date: 2022-11-01
DOI: 10.5594/JMI.2022.3207816
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2022.3207816
Author(s): Xavier Ducloux
bio
Xavier Ducloux is the senior innovation manager at Harmonic, Cesson-Sévigné, France, where he is in charge of monitoring technological breakthroughs and developing innovation in the video delivery domain, including technology and ecosystem partnerships. With 28 years of experience in the digital TV industry, he has led a vast range of video compression projects with the Thomson Group, Cesson-Sévigné, and Harmonic, representing both companies in MPEG. He frequently speaks at industry-leading conferences across the world, is the author of many international publications, and holds multiple patents in video compression and delivery.
Jon Frydensbjerg
bio
Jon Frydensbjerg is the co-founder and chief technology officer at Pixop, Odense, Denmark, where he oversees research and product development enabling the innovation and growth of Pixop’s machine-learning (ML)-based video enhancement technology. With 20 years of experience in global technology, he has led and contributed to the development of a wide range of computer vision and cloud computing projects. He holds an MSc in informatics engineering, specializing in computer vision and graphics, from Aalborg University, Aalborg, Denmark.
Morten Kolle
bio
Morten Kolle is the co-founder and chief executive officer at Pixop, Odense, Denmark, where he is responsible for sales, marketing, and business development leading the global growth and performance of Pixop. With more than 25 years of experience as an entrepreneur and manager in the Danish IT sector, he has been a leader in building successful software development teams and bringing innovative technologies to the market. He founded the casual game development company Certus in 1996, which he sold successfully after running it for 10 years.
Keyword(s): Artificial Intelligence, Live Distribution, Restoration, Super Resolution, UHD-1
Copyright: © 2022 SMPTE

Bibliographic Reference(s)

T. Fautier, X. Ducloux, and P. Gendron, “Improving Streaming Quality and Bitrate Efficiency with Dynamic Resolution Selection,” MHV'22: Proc. of the 1st Mile-High Video Conf., 2022. Accessed: 2022. [Online]. Available: https://www.mile-high.video/_files/ugd/3b5446_2ccbd408c67245cabf3c9bae36b39145.pdfRight Time for Migrating SR in the Headend Up to now, upscaling to 4K has been deployed in end-user devices, such as UHD 4K TVs, but not on the headend side before delivery. The main reason is that UHD 4K TVs took a significant market share in the last few years, while the vast majority of linear channels throughout the world are still in SD or HD resolutions. The number of UHD 4K channels is still limited, because of the extra bandwidth needed to deliver higher resolution. For the targeted bandwidth capacity, the delivery in HD resolution and upscaling into 4K in the TV has been judged as good enough. Based on the evaluations above, this type of upscaling cannot offer a “true” 4K experience similar to what SR could achieve. Moreover, when the HD content suffers from compression artifacts due to bandwidth constraints, these artifacts can be amplified by the upscaling process, leading to a very disappointing 4K experience. Embedding CPU-intensive deep restoration (DR)/SR algorithms in end-user devices may not be economically feasible or good for the planet’s power consumption, but embedding these algorithms in the headend before delivery only makes sense if 4K can be delivered with a sufficient quality at a reasonable bandwidth (i.e., limited extra bandwidth compared with HD). It will not be long before this happens because there are powerful technologies on the horizon that will enable it. There are, of course, new codecs, appearing on the market, such as AV1,11 as well as codecs just being standardized, such as Versatile Video Coding (VVC)12 and Essential Video Coding (EVC).13 However, it may take a few years before these codecs become mainstream. Alternative technologies can make use of already deployed codecs, such as HEVC, and be applied in a straightforward manner to 4K delivery, with compatibility for the millions of devices already deployed. Low complexity enhancement video coding (LCEVC),14 which has just been standardized by MPEG, is one option with a two-layer encoding scheme. Dynamic resolution encoding (DRE), which is already deployed by Netflix for video on demand (VOD) delivery and can be applied for live delivery thanks to artificial intelligence (AI),15 is another one. These two technologies can limit the extra bandwidth of 4K delivery compared with HD delivery at a very reasonable level and make the upscaling of 4K in the headend a good option. Conclusion This article has shown that SR techniques are now mature enough to be used in the professional video market and deliver a “true” 4K experience with digital video content captured at a lower resolution. The complexity of the algorithm under study is compatible with upscaling live HD content into 4K. For archives with a lower resolution and poor quality, it is recommended to first apply deep restoration and noise reduction processes that will preferably run offline and be trained to address specific degradation. EXTERNAL
C. Thomas, “Deep Learning Based Super Resolution, Without Using a GAN.” Towards Data Science, 2019. Accessed: Oct. 3, 2022. [Online]. https://towardsdatascience.com/deep-learning-based-super-resolution-without-using-a-gan-11c9bb5b6cd5 EXTERNAL
“Augmented Tennis at Roland-Garros 2020, Technical Aspects.” Accessed: Oct. 3, 2022. [Online]. Available: htt ps://w w w.f rancet vlab.f r/en /posts/aug mented-ten nis-atroland-garros-2020-technical-aspects EXTERNAL
Topaz Labs, “Video Enhancement,” 2021. Accessed: Oct. 3, 2022. [Online]. Available: https://www.topazlabs.com/video-enhance-ai EXTERNAL
B. Bross et al., “Overview of the Versatile Video Coding (VVC) Standard and Its Applications,” IEEE Trans. Circuits Syst. Video Technol., 31(10):3736–3764, Oct. 2021. EXTERNAL
K. Nasrollahi and T. B. Moeslund, “Super-Resolution: A Comprehensive Survey,” Mach. Vision Appl., 25(6):1423–1468, Jun. 2014. EXTERNAL
J. Samuelsson et al., “MPEG-5 EVC,” SMPTE Conf., Los Angeles, CA, 2019, pp. 1–11. EXTERNAL
“Pixop Deep Restoration,” 2021. Accessed: Oct. 3, 2022. [Online]. Available: https://www.pixop.com/filters/dvres EXTERNAL
“Pixop Super Resolution,” 2021. Accessed: Oct. 3, 2022. [Online]. Available: https://www.pixop.com/filters/pixsr EXTERNAL
Y. Romano, J. Isidoro, and P. Milanfar, “RAISR: Rapid and Accurate Image Super Resolution,” 2016, arXiv:1606.01299v3 [cs. CV]. EXTERNAL
G. Cohen and R. Giryes, “Generative Adversarial Networks,” 2022, arXiv:2203.00667v1 [cs.CV]. EXTERNAL
D. Bank, N. Koenigstein, and R. Giryes, “Autoencoders,” 2021, arXiv:2003.05991v2 [cs.LG]. EXTERNAL
H. Liu et al., “Video Super Resolution Based on Deep Learning: A Comprehensive Survey,” 2022, arXiv:2007.12928v2 [cs.CV]. EXTERNAL
Y. Chen et al., “An Overview of Core Coding Tools in the AV1 Video Codec,” 2018 Picture Coding Symp. (2018 PCS), San Francisco, CA, 2018. Accessed: Oct. 3, 2022. [Online]. Available: https://jmvalin.ca/papers/AV1_tools.pdf EXTERNAL
S. Battista et al., “Overview of the Low-Complexity Enhancement Video Coding (LCEVC) Standard,” IEEE Trans. Circuits Syst. Video Technol., Jun. 2022. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2022.3207816.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Xavier Ducloux, Jon Frydensbjerg, and Morten Kolle; Optimizing UHD Production with Super Resolution Techniques, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 10, 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3207816

Snippet:

Xavier Ducloux, Jon Frydensbjerg, and Morten Kolle; Optimizing UHD Production with Super Resolution Techniques, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 10, 2022); SMPTE, 2022. Available at https://doi.org/10.5594/JMI.2022.3207816

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Xavier Ducloux, Jon Frydensbjerg, and Morten Kolle; <cite>Optimizing UHD Production with Super Resolution Techniques</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 10, 2022); SMPTE, 2022. Available at <a href="https://doi.org/10.5594/JMI.2022.3207816" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2022.3207816</a></span>

SMPTE's HTML Pub

Preview:

Xavier Ducloux, Jon Frydensbjerg, and Morten Kolle; Optimizing UHD Production with Super Resolution Techniques, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 10, 2022); SMPTE, 2022
doi: 10.5594/JMI.2022.3207816
url: https://doi.org/10.5594/JMI.2022.3207816

Snippet:

<li>
Xavier Ducloux, Jon Frydensbjerg, and Morten Kolle; <cite id="bib-10-5594-jmi-2022-3207816">Optimizing UHD Production with Super Resolution Techniques</cite>, SMPTE Motion Imaging Journal ( Volume: 131, Issue: 10, 2022); SMPTE, 2022
<span class="doi">10.5594/JMI.2022.3207816</span>
</li>