Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 128, No. 10, pp. 33–39
Abstract: After decennia of developing leading-edge 2D video compression technologies, the Moving Picture Expert Group (MPEG) is currently working on the new era of coding for immersive applications, referred to as MPEG-I, where “I” refers to the “Immersive” aspects. It ranges from 360° video with head-mounted displays to free navigation in 3D space with head-mounted and 3D light field displays. Two families of coding approaches, covering typical industrial workflows, are currently considered for standardization—MultiView + Depth (MVD) Video Coding and Point Cloud Coding—both supporting high-quality rendering at bitrates of up to a couple of hundreds of megabits per second. This paper provides a technical/historical overview of the acquisition, coding, and rendering technologies considered in the MPEG-I standardization activities.
Publication Date: 2019-11-01
DOI: 10.5594/JMI.2019.2941362
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2019.2941362
Author(s): Gauthier Lafruit
bio
Gauthier Lafruit received an ME degree in electromechanics from the Free University of Brussels, Belgium, in 1989, where he also obtained his doctoral dissertation in 1995 in the field of wavelet imaging, for which he was awarded the Barco Prize. In 1996, he joined imec, Leuven, Belgium, the microelectronics research institute, specializing in compression and image analysis for space applications and broadcasting. This gradually led him to follow standardization committees of European Space Agency (Consulative Committee for Space Data Systems) as well as audiovisual and multimedia standards in general (JPEG and MPEG). After a year at the University of Hasselt, Belgium, taking advantage of a specialization scholarship in research, he joined the Université Libre de Bruxelles, Brussels, the French wing of the Free University of Brussels, in 2014, where he is currently an associate professor in 3D imaging in all its forms, including stereoscopy, multicamera acquisitions, 3D video games, virtual reality, and digital holography.
Daniele Bonatto
bio
Daniele Bonatto received a computational intelligence software engineering degree in applied sciences from the Université Libre de Bruxelles, Brussels, Belgium, in 2016. He is currently enrolled in a PhD program in realtime 3D computing, jointly between the Université Libre de Bruxelles and the Vrije Universiteit Brussel, respectively the French and Dutch wing of the Free University of Brussels, Brussels, Belgium. He works on realtime free-viewpoint rendering of natural scenery with sparse multicamera acquisition setups. Jointly with the Moving Picture Experts Group (MPEG), Bonatto developed the reference view synthesis software in 2018 and two high-density natural scene data sets for static and dynamic content explorations.
Christian Tulvan
bio
Christian Tulvan received an engineering degree in electronics and a master's degree in embedded systems from the University of Transylvania, Braşov, Romania. He is currently a research engineer at the Institut Mines-Télécom, Paris, France. Tulvan has been the chairman of the Reconfigurable Media Coding Ad hoc Group (AhG) and Moving Picture Expert Group (MPEG) Assets AhG, International Organization for Standardization (ISO) MPEG. He contributes to various ISO standards with technologies in the field of 3D graphics. He is part of a research team focusing on 3D graphics, cloud architectures, and interactive media. Tulvan received several ISO certifications of appreciation.
Marius Preda
bio
Marius Preda received an engineering degree in electronics from the Politehnica University of Bucharest, Bucharest, Romania, in 1998, and a PhD degree in mathematics and informatics from Paris V University - René Descartes, Paris, France, in 2001. Preda has been actively involved in Moving Picture Expert Group (MPEG)-4 since 1998, focusing in particular on coding for video and face and body animation (FBA). He was the main contributor of the new animation tools dedicated to generic synthetic objects, promoted by MPEG-4 as part of Animation Framework eXtension (AFX) specifications and currently leads the Point Cloud Coding activities in the MPEG-I standardization. Preda's research interests include generic virtual character animation, rendering, low-bitrate compression and transmission of animated data, multimedia composition, and multimedia standardization.
Lu Yu
bio
Lu Yu is a professor at the College of Information Science and Electronic Engineering at Zhejiang University, Hangzhou, China. She received a BEng degree (Hons.) in radio engineering and a PhD degree in communication and electronic systems from Zhejiang University in 1991 and 1996, respectively. Her research interests include visual perception, visual perceptual quality assessment, visual signal representation and coding, multimedia processing, and related architecture design. Yu has published more than 160 peer-reviewed papers and was granted more than 80 patents. She acted as video subgroup cochair and chair of the Audio and Video Coding Standardization Working Group of China (AVS) for 16 years and has been ISO/IEC JTC 1/SC 29/WG11 (MPEG) video subgroup chair since January 2018.
Keyword(s): 3/6 Degrees Of Freedom (3/6-DoF), Depth Image-based Rendering (DIBR), MPEG-I, Point Cloud Compression
Copyright: © 2019 SMPTE

Bibliographic Reference(s)

ISO.23090 — Moving Picture Experts Group, MPEG-I, “Coded Representation of Immersive Media.” ISO/IEC 23090. [Online]. Available: https://mpeg.chiariglione.org/standards/mpeg-i, 2018. EXTERNAL
MPEG 3DG, “Call for Proposals for Point Cloud Compression V2.” ISO/IEC JTC1/SC29/WG11. MPEG2017/N16763, Hobart, Australia, Apr. 2017. EXTERNAL
B. Salahieh et al., “Test Model 2 for Immersive Video,” ISO/IEC JTC1/SC29/WG11. MPEG2019/N18577, Gothenburg, Sweden, Jul. 2019. EXTERNAL
T. Senoh et al., “Updated eDERS to Higher Precision,” ISO/ IEC JTC1/SC29/WG11. MPEG2018/m42525, San Diego, CA, Apr. 2018. EXTERNAL
G. J. Sullivan et al., “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Trans. Circ. Syst. Video Techn., 22(12):1649–1668, Dec. 2012. EXTERNAL
S. Schwarz et al., “Emerging MPEG Standards for Point Cloud Compression,” IEEE J. Emerg. Select. Topics Circ. Syst., 9(1):133–148, Mar. 2019. EXTERNAL
M. Preda, “Report on PCC CfP Answers,” ISO/IEC JTC1/ SC29/WG11. MPEG2017/N17251, Macao, China, Oct. 2017. EXTERNAL
R. Koenen, “MPEG Standardization Roadmap,” ISO/ IEC JTC1/SC29/WG11. MPEG2018/N17506, San Diego, CA, Apr. 2018. EXTERNAL
K. C. Karthikeyan, “How the Matrix Bullet Time Works?” Geekswipe, Aug. 12, 2017. [Online]. Available: https://geekswipe.net/art/films/how-matrix-bullet-time-works/ EXTERNAL
J. Boyce, R. Doré, and V. K. Malamal Vadakital, “Working Draft 2 of Immersive Video,” ISO/IEC JTC1/SC29/WG11. MPEG2019/N18576, Gothenburg, Sweden, Jul. 2019. EXTERNAL
M. Budagavi et al., “PCC Core Experiments for Category 2,” ISO/IEC JTC1/SC29/WG11. MPEG2018/N17346, Gwangju, Korea, Jan. 2018. EXTERNAL
S. Rogge et al., “Refactoring DERS 8.0 to RDE,” ISO/IEC JTC1/SC29/WG11. MPEG2019/M48075, Gothenburg, Sweden, Jul. 2019. EXTERNAL
J. Jung, “Workshop on Coding Technologies for Immersive Audio/Visual Experiences: How to Achieve 6DoF Compression,” ISO/IEC JTC1/SC29/WG11. MPEG2019/N18559, Gothenburg, Sweden, Jul. 10, 2019. EXTERNAL
M. Panahpour Tehrani et al., “Overview of MPEG-I Visual Test Materials,” ISO/IEC JTC1/SC29/WG11. MPEG2018/ N17606, San Diego, CA, Apr. 2018. EXTERNAL
G. Aguiar, “Blender Beginner Tutorial - UV Mapping (Part 4),” YouTube, Sep. 17, 2017. [Online]. https://www.you-tube.com/watch?v=Vj0k4-I33lQ EXTERNAL
W. Sun et al. “An Overview of Free Viewpoint Depth-Image-Based Rendering (DIBR),” Proc. Second APSIPA Annu. Summit Conf., Singapore, pp. 1023–1030, Dec. 2010. EXTERNAL
R. Doré et al. “3DoF+ Intermediate View Synthetizer Proposal,” ISO/IEC JTC1/SC29/WG11. MPEG2018/m42486, San Diego, CA, 2018, April 2018. EXTERNAL
M. Dou et al., “Fusion4D: Real-Time Performance Capture of Challenging Scene,” ACM Trans. Graphics, 35(4), Jul. 2016, pp. 114:1–114:13. EXTERNAL
R. Doré, G. Briand, and T. Tapie, “Technicolor 3DoFPlus Test Materials,” ISO/IEC JTC1/SC29/WG11. MPEG2018/ m42349, San Diego, CA, Apr. 2018. EXTERNAL
S. Fachada et al., “Depth Image-Based View Synthesis with Multiple Reference Views for Virtual Reality.” 3DTV-CON, Jun. 2018. EXTERNAL
M. Botsch, M. Spernat, and L. Kobbelt, “Phong Splatting,” Proc. First Eurographics Conf. Point-Based Graphics, Switzerland, pp. 25–32, 2004. EXTERNAL
B. Blizard, “The Art of Photogrammetry: Introduction to Software and Hardware,” Tested, Feb. 11, 2014. [Online]. Available: http://www.tested.com/art/makers/460057-tested-dark-art-photogrammetry/ EXTERNAL
R. Schnabel and R. Klein, “Octree-Based Point-Cloud Compression,” Proc. Symp. Point-Based Graphics, Eurographics, pp. 111–121, Jul. 2006. EXTERNAL
A. Hinds et al., “Toward the Realization of Six Degreesof-Freedom with Compressed Light Fields,” Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2017. EXTERNAL
T. Senoh et al., “Enhanced VSRS to Four Reference Views,” ISO/IEC JTC1/SC29/WG11. MPEG2018/m42526, San Diego, CA, Apr. 2018. EXTERNAL
J. Fleureau et al., “Description of Technicolor Intel Response to MPEG-I 3DoF+ Call for Proposal,” ISO/IEC JTC1/SC29/WG11. MPEG2019/m47445, Geneva, Switzerland, Mar. 2019. EXTERNAL
D. Mieloch, G. Lafruit, and B. Kroon, “Summary on MPEG-I Visual Activities,” ISO/IEC JTC1/SC29/WG11. MPEG2019/ N18560, Gothenburg, Sweden, Jul. 2019. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2019.2941362.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Gauthier Lafruit, Daniele Bonatto, Christian Tulvan, Marius Preda, and Lu Yu; Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications, SMPTE Motion Imaging Journal ( Volume: 128, Issue: 10, 2019); SMPTE, 2019. Available at https://doi.org/10.5594/JMI.2019.2941362

Snippet:

Gauthier Lafruit, Daniele Bonatto, Christian Tulvan, Marius Preda, and Lu Yu; Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications, SMPTE Motion Imaging Journal ( Volume: 128, Issue: 10, 2019); SMPTE, 2019. Available at https://doi.org/10.5594/JMI.2019.2941362

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Gauthier Lafruit, Daniele Bonatto, Christian Tulvan, Marius Preda, and Lu Yu; <cite>Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications</cite>, SMPTE Motion Imaging Journal ( Volume: 128, Issue: 10, 2019); SMPTE, 2019. Available at <a href="https://doi.org/10.5594/JMI.2019.2941362" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2019.2941362</a></span>

SMPTE's HTML Pub

Preview:

Gauthier Lafruit, Daniele Bonatto, Christian Tulvan, Marius Preda, and Lu Yu; Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications, SMPTE Motion Imaging Journal ( Volume: 128, Issue: 10, 2019); SMPTE, 2019
doi: 10.5594/JMI.2019.2941362
url: https://doi.org/10.5594/JMI.2019.2941362

Snippet:

<li>
Gauthier Lafruit, Daniele Bonatto, Christian Tulvan, Marius Preda, and Lu Yu; <cite id="bib-10-5594-jmi-2019-2941362">Understanding MPEG-I Coding Standardization in Immersive VR/AR Applications</cite>, SMPTE Motion Imaging Journal ( Volume: 128, Issue: 10, 2019); SMPTE, 2019
<span class="doi">10.5594/JMI.2019.2941362</span>
</li>