Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2023, No. 10, pp. 12–15
Abstract: This study proposes a volumetric capturing studio, called NHK Meta Studio, that can acquire various material attributes and light-field parameters to enable photo-realistic relighting. Volumetric videos captured from Meta Studio comprise a multidimensional extended point cloud for each frame, and hence, have large amounts of data. Although it is possible to export all parameters into existing formats, the large amounts of data make this approach less optimal. Therefore, we propose a hierarchical volumetric video format that comprises base and enhancement layers. The base layer is designed for compatibility with the existing format and rendering for a simple relighting method. The uncompressed base-layer format is represented in the Stanford triangle format (.ply) and the compressed base layer is used for an extension of the MPEG volumetric video format called visual volumetric video-based coding (V3C). As a conventional file format is adopted, integration with existing 3D applications becomes easy. The enhancement layer includes the other components and the advanced rendering shader program, which enables us to fully utilize our advanced photo-realistic rendering method; however, full real-time rendering is still under development. Prior to full real-time rendering, we developed an efficient rendering system for the base layer. This system was verified for integration with game engines. Even real-time rendering using full information is not accomplished, real-time rendering will make it possible to make wide use of Meta Studio's volumetric data.
Publication Date: 2023-10-16
DOI: 10.5594/M002011
ISBN: Print: 978-1-61482-964-5
Link: https://doi.org/10.5594/M002011
Author(s): Yuma WakaharaNHK Science & Technology Research Laboratories, 1-10-11 Kinuta, Setagaya, Tokyo 157- 8510, Japan
Toshie MisuNHK Science & Technology Research Laboratories, 1-10-11 Kinuta, Setagaya, Tokyo 157- 8510, Japan
Kensuke HisatomiNHK Science & Technology Research Laboratories, 1-10-11 Kinuta, Setagaya, Tokyo 157- 8510, Japan
Keyword(s): Volumetric Video, Rendering, Shading, Virtual Video Production, Immersive Audio, Point Cloud, Free Viewpoint, Extended Reality
Copyright: © 2023 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

ISO.23090-5.2021 — International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 23090-5:2021, “Information Technology—Coded Representation of Immersive EXTERNAL
H. Morioka, T. Misu, T. Suginoshita, H. Mitsumine, “NHK Meta Studio: A Compact Volumetric TV Studio for 3-D Reconstruction,” IEEE Transactions on Broadcasting 69 (1): 2-9, Mar. 2023. EXTERNAL
MPEG-I 3DG, “OpenV3C – Multi-platform Open-source Implementation of the V-PCC,” EXTERNAL
“Volumetric Video Studio—Kawasaki.”2022. [Online]. Available: https://global.canon/en/vvs/ EXTERNAL
“The Metaverse Needs Authenticity.”2022. [Online]. Available: https://metastage.com/ EXTERNAL
“Powering the Future of Human Connection.”2022. [Online]. Available: https://8i.com/ EXTERNAL
“Intel Studios.”2020. [Online]. Available: https://newsroom.intel.com/press-kits/intel-studios/ EXTERNAL
I. Mustafa, et al. “Humanrf: High-fidelity Neural Radiance Fields for Humans in Motion.” arXiv preprint arXiv:2305.06356 (2023). EXTERNAL
M. Kowalski, J. Naruniec, M. Daniluk, “Livescan3d: A Fast and Inexpensive 3d Data Acquisition System for Multiple Kinect v2 Sensors,” 2015 International Conference on 3D Vision, 2015. EXTERNAL
K. Guo, “The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting,” ACM Trans. Graph., 38 (6): 1-19, Dec. 2019. [Online]. Available: https://dl.acm.org/doi/10.1145/3355089. 3356571 EXTERNAL
“The Future of Immersive Entertainment.”2022. [Online]. Available: https://www.tetavi.com EXTERNAL
C. Guede, J. Ricard, S. Lasserre, J. Llach, “Technicolor Point Cloud Renderer,” ISO/IEC EXTERNAL
“Volumetric Capture Technology That Goes Beyond Omnidirectional Visualization.”2020. [Online]. Available: https://www.sony.com/en/SonyInfotechnology/stories/Volumetric_Capture/ EXTERNAL
A. Yamada, T. Suginoshita, H. Morioka, T. Misu, H. Mitsumine, J. Arai, “3D Subject Area Estimation Method for Automatic Subject Tracking Photography,” Proceedings of the 2022 Institute of Image Information and Television Engineers (ITE) Annual Convention, 21B-2 (2022) (in Japanese). EXTERNAL
“4DViews.”https://www.4dviews.com/, last accessed 7/19/2023. EXTERNAL
S. Markus. “"Potree: Rendering Large Point Clouds in Web Browsers." Technische Universität Wien, Wiedeń (2016). EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M002011.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002011

Snippet:

Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at https://doi.org/10.5594/M002011

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; <cite>Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5. Available at <a href="https://doi.org/10.5594/M002011" target="_blank" rel="noopener">https://doi.org/10.5594/M002011</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; <cite id="bib-10-5594-m002011">Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction</cite>, SMPTE Meetings and Conferences ( October 2023); SMPTE, 2023, ISBN: 978-1-61482-964-5
<span class="doi">10.5594/M002011</span>
</li>