Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction

Metadata

Publisher: SMPTE
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Motion Imaging Jour.
Volume: 133, No. 3, pp. 17–25
Abstract: We are developing a photo-realistic volumetric capture system called Japan Broadcasting Corp. (NHK) Meta Studio. Meta Studio can acquire various material and light-field parameters to enable photo-realistic relighting. Captured data from Meta Studio comprise a multi-dimensional extended point cloud for each frame and is considerable in size. Hence, we adopt a hierarchical format that consists of base and enchancement layers. The base layer is designed for compatibility with the existing format and rendering for a simple relighting method. The enhancement layer includes other components and the advanced rendering shader program, which enables us to fully utilize our advanced photo-realistic rendering method; however, full real-time rendering is still under development. In preparation for full real-time rendering, we have developed an efficient rendering system for the base layer. Although full processing is not complete yet, real-time rendering will allow wide use of Meta Studio's volumetric data.
Publication Date: 2024-05-01
DOI: 10.5594/JMI.2024/ZRBF6236
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2024/ZRBF6236
Author(s): Yuma Wakahara
Toshie Misu
Kensuke Hisatomi
Keyword(s): Volumetric Video, Rendering, Shading, Virtual Video Production, Immersive Audio
Copyright: © 2024 SMPTE

Bibliographic Reference(s)

ISO.23090-5.2021 — International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 23090-5:2021, “Information Technology—Coded Representation of Immersive EXTERNAL
H. Morioka, T. Misu, T. Suginoshita, H. Mitsumine, “NHK Meta Studio: A Compact Volumetric TV Studio for 3-D Reconstruction,” IEEE Trans. on Broadcast. 69 (1): 2–9, Mar. 2023. EXTERNAL
M. Kowalski, J. Naruniec, M. Daniluk, “Livescan3d: A Fast and Inexpensive 3d Data Acquisition System for Multiple Kinect v2 Sensors,” Proc. 2015 International Conference on 3D Vision, Oct. 2015. EXTERNAL
Intel Studios Showcases Volumetric Production at 77th Venice International Film Festival. [Online]. Available: https://www.intel.com/content/www/us/en/news-room/news/studios-volumetric-production-venice-film-festival.html. EXTERNAL
“The Metaverse Needs Authenticity.” 2022. [Online]. Available: https://metastage.com/. EXTERNAL
“Volumetric Video Studio-Kawasaki.” 2022. [Online]. Available: https://global.canon/en/vvs/. EXTERNAL
“Volumetric Capture Technology That Goes Beyond Omnidirectional Visualization.” 2020. [Online]. Available: https://www.sony.com/en/Sonylnfo/technology/stories/entries/Volumetric_Capture/. EXTERNAL
“The Future of Immersive Entertainment.” 2022. [Online]. Available: https://yoom.com/. EXTERNAL
I. Mustafa, “Humanrf: High-fidelity Neural Radiance Fields for Humans in Motion.” arXiv preprint arXiv:2305.06356 (2023). EXTERNAL
“Powering the Future of Human Connection.” 2022. [Online]. Available: https://8i.com/. EXTERNAL
“4DViews.” [Online]. Available: https://www4dviews.com/. Accessed 7/19/2023. EXTERNAL
S. Markus “Potree: Rendering Large Point Clouds in Web Browsers.” Technische Universität Wien, Wiedeń (2016). EXTERNAL
K. Guo, “The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting,” ACM Trans. Graph., 38 (6): 1–19, Dec. 2019. [Online]. Available: https://dl.acm.org/doi/10.1145/3355089.3356571. EXTERNAL
A. Yamada, T. Suginoshita, H. Morioka, T. Misu, H. Mitsumine, J. Arai, “3D Subject Area Estimation Method for Automatic Subject Tracking Photography,” Proceedings of the 2022 Institute of Image Information and Television Engineers (ITE) Annual Convention, 21B-2 (2022) (in Japanese). EXTERNAL
C. Guede, J. Ricard, S. Lasserre, J. Llach, “Technicolor Point Cloud Renderer,” ISO/ IEC JTCl/SC29/WG11 MPEG, N16902, 2017. EXTERNAL
MPEG-I 3DG, “OpenV3C - Multi-platform Open-source Implementation of the V-PCC,” ISO/IEC JTC 1/SC 29/WG 11 N19375, MPEG, 2020. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2024-ZRBF6236.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction, MIJ 2024, Volume 133, Number 3 (pp. 17 to 25); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/ZRBF6236

Snippet:

Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction, MIJ 2024, Volume 133, Number 3 (pp. 17 to 25); SMPTE, 2024. Available at https://doi.org/10.5594/JMI.2024/ZRBF6236

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; <cite>Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction</cite>, MIJ 2024, Volume 133, Number 3 (pp. 17 to 25); SMPTE, 2024. Available at <a href="https://doi.org/10.5594/JMI.2024/ZRBF6236" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2024/ZRBF6236</a></span>

SMPTE's HTML Pub

Preview:

Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction, MIJ 2024, Volume 133, Number 3 (pp. 17 to 25); SMPTE, 2024
doi: 10.5594/JMI.2024/ZRBF6236
url: https://doi.org/10.5594/JMI.2024/ZRBF6236

Snippet:

<li>
Yuma Wakahara, Toshie Misu, and Kensuke Hisatomi; <cite id="bib-10-5594-jmi-2024-zrbf6236">Advanced Volumetric Video Format for Enhancing Photo-Realistic Lighting Reproduction</cite>, MIJ 2024, Volume 133, Number 3 (pp. 17 to 25); SMPTE, 2024
<span class="doi">10.5594/JMI.2024/ZRBF6236</span>
</li>