8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 1, pp. 50–58
Abstract: In this paper, we present a set of tools designed to simplify the distribution, playback, and adaptation of high-quality immersive content for large-screen immersive environments. The tools are based on a high-performance high-efficiency video coding (HEVC) codec that allows one to encode, decode, and play ultrahigh-resolution video in 8K and 16K resolutions using standard computing systems. A flexible video-rendering engine allows the content to be adapted to different output display environments. In addition, an immersive audio workflow, based on high-order ambisonics, provides support for 3D and interactive audio. The new tools have been validated in two advanced immersive installations using new immersive content created using various techniques that has been produced in the context of the Immersify European Project. The results show that the new tools enable ultrahigh-resolution immersive video playback combined with 3D audio, and first evaluations indicate that a better immersive experience for final users can be achieved, while at the same time using a simpler workflow.
Publication Date: 2021-01-01
DOI: 10.5594/JMI.2020.3036711
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2020.3036711
Author(s): Mauricio Alvarez-Mesa
bio
Mauricio Alvarez-Mesa is the CEO and co-founder of Spin Digital, Berlin, Germany. He received a bachelor’s degree in electronics engineering from the University of Antioquia, Medellin, Colombia, and a PhD degree in computing engineering from the Polytechnic University of Catalonia, Barcelona, Spain. He has worked as a researcher for several international institutions such as IBM Labs, Haifa, Israel, Fraunhofer HHI, Berlin, and Technical University of Berlin, Germany. He was the technical coordinator of the EU-funded Immersify project. His research interests include video coding, parallel computing, and human visual perception.
Sergio Sanz-Rodríguez
bio
Sergio Sanz-Rodríguez is a senior engineer and co-founder of Spin Digital, Berlin, Germany. He received a PhD degree in multimedia and communications in 2011 from the Carlos III University of Madrid, Spain. From 2005 to 2012, he was a teaching assistant at the Carlos III University of Madrid. In January 2013, he joined the Image Communication Group of Technical University of Berlin (TUB), and collaborated with the Multimedia Communications Group of Fraunhofer HHI in Berlin. From July 2014 to June 2016, he worked as a researcher at the AES group of TUB. His research interests include video coding, subjective quality assessment, and ambisonics.
Chi Ching Chi
bio
Chi Ching Chi is the chief technology officer of Spin Digital, Berlin, Germany. He received a BSc degree in electrical engineering in 2008 and an MSc degree in computer engineering in 2010 from the Delft University of Technology, The Netherlands. Until 2016, he was working toward a PhD degree in the Embedded Systems Architecture Group at TU Berlin, Germany. The work he performed during this period became the foundation of Spin Digital, which he co-founded in 2015. Since then, he has been advancing the company’s core technologies in video coding and expanding it in related topics, such as image processing, omnidirectional mapping, video rendering, and HDR conversion, resulting in highly advanced implementations with high quality and performance.
Maciej Glowiak
bio
Maciej Glowiak graduated from the Poznań University of Technology, Poznań, Poland, in 2003. Since 2003, he has been working for Poznań Supercomputing and Networking Center (PSNC) in the Network Department. He has been involved in many Polish and European research projects. He was a coordinator of the Immersify project focusing on cutting-edge technologies for recording, encoding, streaming, and playback of the high-resolution immersive media. Currently, he heads the New Media Department of PSNC, conducting research projects, next-generation networks, multimedia applications, interactive technologies, and digital electronics design.
Roland Haring
bio
Roland Haring studied media technology and design at the Hagenberg University of Applied Sciences, Hagenberg, Austria. Since 2003, he has been a member of the Ars Electronica Futurelab and one of the driving forces behind the lab’s R&D efforts. His activities include research and development in several large R&D projects with academic, artistic, and commercial partners and collaborators. Currently, he is the technical director of Ars Electronica Futurelab and co-responsible for its general management, content conception, and technical development. With his many years of experience in the (software) technical management of large-scale, research-intensive projects, he is an expert in the design, architecture, and development of interactive applications.
Keyword(s): 8K, 16K, HEVC, High-Order Ambisonics (HOA), Immersive Audio, Immersive Media, Video Coding, Video Rendering
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

How Independent are HDR, WCG, and HFR in Human Visual Perception and the Creative Process? [Active]
M. Alvarez-Mesa et al., “Advanced Video Compression and Rendering for Highly Immersive 8K+ Applications,” presented at IBC 2019 Conf., Amsterdam, The Netherlands, pp. 1–9, Sep. 2019. [Online]. Available: https://www.ibc.org/advanced-video-compression-and-rendering-for-highly-immersive-8k-applica-tions/5060.article EXTERNAL
C. Nachbar et al., “AmbiX – A Suggested Ambisonics Format,” Ambisonics Symp., Lexington, KY, Jun. 2011. EXTERNAL
D. Flynn et al., “Overview of the Range Extensions for the HEVC Standard: Tools, Profiles, and Performance,” IEEE Trans. Circuits Syst. Video Technol., 26(1):4–19, Jan. 2016. EXTERNAL
J. Xu, R. Joshi, and R. A. Cohen, “Overview of the Emerging HEVC Screen Content Coding Extension,” IEEE Trans. Circuits Syst. Video Technol., 26(1):50–62, Sep. 2015. EXTERNAL
M. Sugawara and K. Masaoka, “UHDTV Image Format for Better Visual Experience,” Proc. IEEE, 101(1):8–17, Jan. 2013. EXTERNAL
D. Rudrich, “IEM Plug-in Suite.” Accessed: Oct. 1, 2020. [Online]. Available: https://plugins.iem.at/ EXTERNAL
M. Alvarez-Mesa and C. C. Chi, “HEVC Software Media Player for Ultra-High-Quality Video: 8K and Beyond,” Proc. 25th Int. Display Workshops, Nagoya, Japan, pp. 1437–1440, 2018. EXTERNAL
K. Kurowski et al., “PSNC Advanced Multimedia and Visualization Infrastructures, Services and Applications,” Proc. 23rd Int. ACM Conf. 3D Web Technol.,” pp. 34:1–34:2, Poznan´ , Poland, June 2018. EXTERNAL
M. Wien et al., “Standardization Status of Immersive Video Coding,” IEEE J. Emerg. Sel. Top. Circuits Syst., 9(1):5–17, Mar. 2019. EXTERNAL
S. Schwarz et al., “Emerging MPEG Standards for Point Cloud Compression,” IEEE J. Emerg. Sel. Top. Circuits Syst., 9(1):133–148, Mar. 2019. EXTERNAL
S. Sanz-Rodriguez and A. Nikrang, “Report on Quality Assessment and Content Preparation Guidelines,” Technical Report Immersify Project, 2020. [Online]. Available: https://immersify.eu/wp-content/uploads/2020/10/D5.3-Report-on-QA-and-Content-Preparation-Guidelines-v2.1.pdf EXTERNAL
M. Glowiak, “Report on Immersive Content Production,” Technical Report Immersify Project, 2019. [Online]. Available: https://immersify.eu/wp-content/uploads/2019/03/Deliverable_D5.1-Report_on_Immersive_Content_Production_Part_1.pdf EXTERNAL
ITU-R BS.2051-3 [Active]

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2020.3036711.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Mauricio Alvarez-Mesa, Sergio Sanz-Rodríguez, Chi Ching Chi, Maciej Glowiak, and Roland Haring; 8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 1, 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2020.3036711

Snippet:

Mauricio Alvarez-Mesa, Sergio Sanz-Rodríguez, Chi Ching Chi, Maciej Glowiak, and Roland Haring; 8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 1, 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2020.3036711

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Mauricio Alvarez-Mesa, Sergio Sanz-Rodríguez, Chi Ching Chi, Maciej Glowiak, and Roland Haring; <cite>8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 1, 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2020.3036711" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2020.3036711</a></span>

SMPTE's HTML Pub

Preview:

Mauricio Alvarez-Mesa, Sergio Sanz-Rodríguez, Chi Ching Chi, Maciej Glowiak, and Roland Haring; 8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 1, 2021); SMPTE, 2021
doi: 10.5594/JMI.2020.3036711
url: https://doi.org/10.5594/JMI.2020.3036711

Snippet:

<li>
Mauricio Alvarez-Mesa, Sergio Sanz-Rodríguez, Chi Ching Chi, Maciej Glowiak, and Roland Haring; <cite id="bib-10-5594-jmi-2020-3036711">8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 1, 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2020.3036711</span>
</li>