Lessons Learned During One year of Commercial Volumetric Video Production

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 129, No. 9, pp. 31–37
Abstract: In June 2018, Fraunhofer Heinrich Hertz Institute (HHI) together with Studio Babelsberg, ARRI, Universum Film AG (UFA), and Interlake founded the joint venture Volucap GmbH and opened a commercial volumetric video studio on the film campus of Potsdam-Babelsberg. After a testing phase, commercial productions went live in November 2018. The core technology for volumetric video production is 3D human body reconstruction (3DHBR), developed by Fraunhofer HHI. This technology captures real persons with our novel volumetric capture system and creates naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented environment. Thanks to the large number of test productions and new requirements from customers, several lessons have been learned during the first year of commercial activity. The processing workflow for capture and production of volumetric video has been continuously evolved such as process-dependent color grading and a new encoding scheme for volumetric video. Furthermore, specific challenges have been tackled, for instance, fast movements and relighting of actors.
Publication Date: 2020-10-01
DOI: 10.5594/JMI.2020.3010399
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2020.3010399
Author(s): Oliver Schreer
bio
Oliver Schreer is the head of the Immersive Media & Communication Research Group, Vision & Imaging Technologies Department, Fraunhofer Heinrich Hertz Institute (HHI), Berlin, Germany, and an associate professor at the Technical University of Berlin, Berlin. Since December 2019, he has been coordinating the Horizon 2020 Coordination and Support Action XR4ALL. His main research fields are 3D video processing and immersive and interactive media services and applications exploiting augmented and virtual reality technologies. He has published more than 100 articles and was the main editor of the books 3D Videocommunication (2005) and Media Production, Delivery and Interaction for Platform Independent Systems: Format-Agnostic Media (2014) both published by Wiley. In May 2019, he and his colleagues, I. Feldmann and P. Kauff, were awarded the Joseph-von-Fraunhofer Prize “Realistic people in virtual worlds—A movie as a true experience.”
Ingo Feldmann
bio
Ingo Feldmann is the head of the Immersive Media & Communication Group, Vision & Imaging Technologies Department, Fraunhofer HHI. He received a Dipl.-Ing. degree in electrical engineering from the Technical University of Berlin, in 2000. Since September 2000, he has been with Fraunhofer HHI, where he has been engaged in various research activities in the field of 2D image processing, 3D scene reconstruction and modeling, digital cinema, multiview projector–camera systems, realtime 3D video conferencing, and immersive TV applications. His current research focuses on the field of future virtual reality, augmented reality and mixed reality applications, and volumetric video acquisition. In May 2019, he and his colleagues, O. Schreer and P. Kauff, were awarded the Joseph-von-Fraunhofer Prize “Realistic people in virtual worlds—A movie as a true experience.”
Peter Kauff
bio
Peter Kauff is the former head of the Vision & Imaging Technologies Department, Fraunhofer HHI. He also acted as a co-head of the Capture & Display Systems Group in his department and was a co-founder of the “Tomorrow’s immersive Media Experience Laboratory (TiME Lab)” at Fraunhofer HHI. He received a Dipl.-Ing. degree in electrical engineering and telecommunication from the Technical University of Aachen, Aachen, Germany, in 1984. He has been with Fraunhofer HHI since then. He was involved in numerous German and European projects related to advanced 3D video, 3D cinema, 3D television, tele-presence, and immersive media. In May 2019, he and his colleagues, O. Schreer and I. Feldmann, were awarded the Joseph-von-Fraunhofer Prize “Realistic people in virtual worlds—A movie as a true experience.”
Peter Eisert
bio
Peter Eisert is a professor for visual computing at the Humboldt University of Berlin, Berlin, Germany, and heading the Vision & Imaging Technologies Department at Fraunhofer HHI. He received a PhD degree with the highest honors from the University of Erlangen, Erlangen, Germany, in 2000. Then, he worked as a postdoctoral fellow at Stanford University, Stanford, CA, on 3D image analysis, facial animation, and computer graphics. In 2002, he joined Fraunhofer HHI, where he is coordinating and initiating numerous national and international research projects. He has published more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as in the Editorial Board of the Journal of Visual Communication and Image Representation. His research interests include 3D image/video analysis and synthesis, face and body processing, image-based rendering, deep learning, computer vision, computer graphics in application areas like multimedia, security, and medicine.
Danny Tatzelt
bio
Danny Tatzelt works as a freelance cinematographer, editor, and director based in Berlin. He graduated with a degree in cinematography from the University for Film and Television, Konrad Wolf, Potsdam-Babelsberg, in 2014 with the documentary Viajar, followed by a masterscholars program completed with a 360° film Berlin in 2016. During his academic years, he held the Studienstiftung des deutschen Volkes scholarship. In addition to classical film works, he has also specialized in virtual reality (VR)- and augmented reality (AR)-film productions. In close collaboration with Fraunhofer HHI, he designed the Volumetric Capture Studio that led to the founding of Volucap GmbH, Potsdam-Babelsberg, Germany, the first commercial volumetric video studio on the European continent.
Cornelius Hellge
bio
Cornelius Hellge heads the Multimedia Communications Group, Fraunhofer HHI, since 2015. He received a Dipl.-Ing. degree in media technology from the Ilmenau University of Technology, Ilmenau, Germany, in 2006 and a Dr.-Ing. degree (summa cum laude) with distinction from the Berlin University of Technology, Berlin, Germany, in 2013. In 2014, he was a visiting researcher with the Network Coding and Reliable Communications Group, Massachusetts Institute of Technology, Cambridge, MA. He is responsible for various scientific as well as industry-funded projects. Together with his team, he regularly contributes to Third-Generation Partnership Project (3GPP) (RAN1, RAN2, and SA4), Motion Picture Experts Group (MPEG), Joint Collaborative Team on Video Coding (JCT-VC), and JVET. He has authored more than 60 journal and conference papers, predominantly in the area of video communication and next-generation networks and he holds more than 30 internationally issued patents and patent applications in these fields. His current research interest is in new formats for volumetric video for mixed reality services over 5G networks.
Karsten Müller
bio
Karsten Müller received a Dr. Ing. degree in electrical engineering and a Dipl. Ing. degree from the Technical University of Berlin, Germany, in 2006 and 1997, respectively. He has been with the Fraunhofer Institute for Telecommunications, HHI, since 1997, where he is currently the head of group. His research interests include coding of 3D and multidimensional video, as well as compression and representation of neural networks. He has been involved in international standardization activities, successfully contributing to the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group for work items on visual media content description, multiview, multitexture, 3D video coding, and neural network representation. Here, he cochaired an ad hoc group on 3D video coding from 2003 to 2012. He has served as the chair and an editor for IEEE conferences and as an associate editor for the IEEE Transactions on Image Processing
Sven Bliedung
bio
Sven Bliedung is a CG industry veteran and the director and chief executive officer of Volucap, a state-of-the-art volumetric capture studio, located in Postsdam-Babelsberg, Germany. He founded Slice, an immersive media production company after he founded a high-profile visual effects company called VFXbox. He is also the founder of the Virtual Reality Association Berlin Brandenburg (VRBB e.V.) and today he is the chief executive officer of Volucap, the state-of-the-art volumetric capture studio.
Thomas Ebner
bio
Thomas Ebner joined the Fraunhofer HHI in 2008. He is a scientist and a software developer in the Immersive Media & Communication Group of the Vision & Imaging Technologies Department. As a project manager for volumetric video for virtual reality/augmented reality (VR/AR), he took a leading role in integrating dynamic 3D reconstructions of real persons into immersive experiences and expanding the group’s X Reality activities. Since 2018, he has also been the chief technology officer of Volucap, a volumetric capture studio located in Potsdam-Babelsberg, Germany. He holds a diploma degree in media and computer science from the Technical University of Dresden, Dresden, Germany. From 2005 to 2006, he was with the Division of Computer Engineering, Digital Dongseo University Busan, Busan, South Korea.
Keyword(s): Augmented Reality, Production, Virtual Reality, Volumetric Video
Copyright: © 2020 SMPTE

Bibliographic Reference(s)

Corto, Accessed: 20 Jun. 2019. [Online]. Available: https://github.com/cnr-isti-vclab/corto EXTERNAL
W. Morgenstern, A. Hilsmann, and P. Eisert, “Progressive Non-Rigid Registration of Temporal Mesh Sequences,” Proc. Eur. Conf. Visual Media Production, London, U.K., pp. 1–10, Dec. 2019. EXTERNAL
“Unreal Engine,” Accessed: 6 Aug. 2020. [Online]. Available: https://www.unrealengine.com EXTERNAL
W. Waizenegger et al., “Real-Time Patch Sweeping for High-Quality Depth Estimation in 3D Videoconferencing Applications,” Presented at SPIE Conf. Real-Time Image Video Process., San Francisco, CA, 2011, doi: 10.1117/12.872868. EXTERNAL
Feldmann, were awarded the Joseph-von-Fraunhofer Prize “Realistic people in virtual worlds—A movie as a true experience.” Peter Eisert is a professor for visual computing at the Humboldt University of Berlin, Berlin, Germany, and heading the Vision & Imaging Technologies Department at Fraunhofer HHI. He received a PhD degree with the highest honors from the University of Erlangen, Erlangen, Germany, in 2000. Then, he worked as a postdoctoral fellow at Stanford University, Stanford, CA, on 3D image analysis, facial animation, and computer graphics. In 2002, he joined Fraunhofer HHI, where he is coordinating and initiating numerous national and international research projects. He has published more than 200 conference and journal papers and is an Associate Editor of the International Journal of Image and Video Processing as well as in the Editorial Board of the Journal of Visual Communication and Image Representation. His research interests include 3D image/ video analysis and synthesis, face and body processing, image-based rendering, deep learning, computer vision, computer graphics in application areas like multimedia, security, and medicine. EXTERNAL
Google, “Introducing Draco: Compression for 3D Graphics,” Accessed: 20 Jun. 2019. [Online]. Available: https://opensource.googleblog.com/2017/01/introducing-draco-compression-for-3d.html EXTERNAL
“8i,” Accessed: 6 Aug. 2020. [Online]. Available: https://8i.com/ EXTERNAL
R. M. Diaz Fernandez et al., “Region Dependent Mesh Refinement for Volumetric Video Workflows,” IEEE Int. Conf. 3D Immersion (IC3D), Brussels, Belgium, pp.1–8, Dec. 2019. EXTERNAL
“Open 3D Graphics Compression,” Accessed: 20 Jun. 2019. [Online]. Available: https://github.com/KhronosGroup/glTF/wiki/Open-3D-Graphics-Compression EXTERNAL
“Unity,” Accessed: 6 Aug. 2020. [Online]. Available: https://unity.com EXTERNAL
J. Rossignac, “Edgebreaker: Connectivity Compression for Triangle Meshes,” IEEE Trans. Vis. Comput. Graph., 5(1):47–61, Jan./Mar. 1999. EXTERNAL
T. Ebner et al., “Multi-View Reconstruction of Dynamic Real-World Objects and Their Integration in Augmented and Virtual Reality Applications,” J. Soc. Inf. Display, 25(3):151–157, Mar. 2017, doi: 10.1002/jsid.538. EXTERNAL
W. Waizenegger et al., “Real-Time 3D Body Reconstruction for Immersive TV,” Proc. 23rd Int. Conf. Image Process. (ICIP 2016), Phoenix, AZ, pp. 360–364, 25–28 Sep. 2016. EXTERNAL
S. Schwarz et al., “Emerging MPEG Standards for Point Cloud Compression,” IEEE J. Emerg. Sel. Top. Circuits Syst., 9(1):133–148, Mar. 2019. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2020.3010399.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Oliver Schreer, Ingo Feldmann, Peter Kauff, Peter Eisert, Danny Tatzelt, Cornelius Hellge, Karsten Müller, Sven Bliedung, and Thomas Ebner; Lessons Learned During One year of Commercial Volumetric Video Production, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2020.3010399

Snippet:

Oliver Schreer, Ingo Feldmann, Peter Kauff, Peter Eisert, Danny Tatzelt, Cornelius Hellge, Karsten Müller, Sven Bliedung, and Thomas Ebner; Lessons Learned During One year of Commercial Volumetric Video Production, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2020.3010399

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Oliver Schreer, Ingo Feldmann, Peter Kauff, Peter Eisert, Danny Tatzelt, Cornelius Hellge, Karsten Müller, Sven Bliedung, and Thomas Ebner; <cite>Lessons Learned During One year of Commercial Volumetric Video Production</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020. Available at <a href="https://doi.org/10.5594/JMI.2020.3010399" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2020.3010399</a></span>

SMPTE's HTML Pub

Preview:

Oliver Schreer, Ingo Feldmann, Peter Kauff, Peter Eisert, Danny Tatzelt, Cornelius Hellge, Karsten Müller, Sven Bliedung, and Thomas Ebner; Lessons Learned During One year of Commercial Volumetric Video Production, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020
doi: 10.5594/JMI.2020.3010399
url: https://doi.org/10.5594/JMI.2020.3010399

Snippet:

<li>
Oliver Schreer, Ingo Feldmann, Peter Kauff, Peter Eisert, Danny Tatzelt, Cornelius Hellge, Karsten Müller, Sven Bliedung, and Thomas Ebner; <cite id="bib-10-5594-jmi-2020-3010399">Lessons Learned During One year of Commercial Volumetric Video Production</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 9, October 2020); SMPTE, 2020
<span class="doi">10.5594/JMI.2020.3010399</span>
</li>