Bringing Stage Events with Real Actors to Your Home in VR: A Low Complexity Solution for Immersive Media Streaming

Metadata

Publisher: SMPTE — Hollywood, CA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 00, pp. 1–21
Abstract: In this paper we demonstrate a novel method how to integrate stereo content of a live act into a virtual reality (VR) environment. This opens a new level of immersion, when watching the recording of a stage event using a head mounted display. In contrast to existing solutions, we offer proper depth perception, allow for a low complexity recording and distribution and still avoid any motion sickness caused by objects being close to the observer. To this end, we separate the digitization of the event location from the recording of the actor's performance. As a main contribution of the paper, we show how to achieve high visual quality when compositing the stereo recording showing the actors with the reconstructed 3D-model of the event location. To do so, we have created a tool that allows to simulate the 3D perception of the stereo video depending on the selected stereo parameters such as inter-camera distance and focal length. By these means, we can ensure a smooth transition between the stereo video and the 3D model of the event location, leading to superior visual quality. Numerous lessons learned conclude our paper showing how we overcame practical problems encountered when recording a real theater play.
Publication Date: 2024-10-21
DOI: 10.5594/MOO/3010
ISBN: Electronic: 978-1-61482-965-2
Link: https://doi.org/10.5594/MOO/3010
Author(s): Joachim Keinert
Tobias Jaschke
Hendrik Hawer
Andree Josef
Frank Dittrich
Siegfried Fossel
Keyword(s): Stereo, Parallax, Actor, Live Act, Stage, Event Location, 3D Reconstruction, Blender, Unity, Virtual Reality, Head Mounted Display, Broadcast, 360° Video, Motion Capture, Photogrammetry, LiDAR, Focal Length, Time & Sync
Copyright: © 2024 SMPTE

Bibliographic Reference(s)

Parallax. In: Wikipedia [Internet]. 2024 [cited 2024 Jul 22]. Available from: https://en.wikipedia.org/w/index.php?title=Parallax&oldid=1221302278 EXTERNAL
VRChat [Internet]. [cited 2024 Aug 1]. Available from: https://hello.vrchat.com/ EXTERNAL
Metallica Rocks Fortnite with a New Music Experience & More! [Internet]. [cited 2024 Aug 1]. Available from: https://www.fortnite.com/news/metallica-lights-up-fortnite-with-a-new-music-experience-festival-season-and-more EXTERNAL
Winkler A, Won J, Ye Y. QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars. In: SIGGRAPH Asia 2022 Conference Papers [Internet]. Daegu Republic of Korea: ACM; 2022 [cited 2023 Jun 2]. p. 1–8. Available from: https://dl.acm.org/doi/10.1145/3550469.3555411 EXTERNAL
Xia S, Gao L, Lai YK, Yuan MZ, Chai J. A Survey on Human Performance Capture and Animation. J Comput Sci Technol. 2017 May; 32(3):536–54. EXTERNAL
Weissig C, Schreer O, Eisert P, Kauff P. The Ultimate Immersive Experience: Panoramic 3D Video Acquisition. In: Schoeffmann K, Merialdo B, Hauptmann AG, Ngo CW, Andreopoulos Y, Breiteneder C, editors. Advances in Multimedia Modeling [Internet]. Berlin, Heidelberg: Springer Berlin Heidelberg; 2012 [cited 2023 Jun 1]. p. 671–81. (Lecture Notes in Computer Science; vol. 7131). Available from: http://link.springer.com/10.1007/978-3-642-27355-1_72 EXTERNAL
Matzen K, Cohen MF, Evans B, Kopf J, Szeliski R. Low-cost 360 stereo photography and video capture. ACM Trans Graph. 2017 Jul20;36(4):1–12. EXTERNAL
Hannuksela MM, Wang YK, Hourunranta A. An Overview of the OMAF Standard for 360° Video. In: 2019 Data Compression Conference (DCC). 2019. p. 418–27. EXTERNAL
Sreedhar KK, Curcio IDD, Hourunranta A, Lepisto M. Immersive media experience with MPEG OMAF multi-viewpoints and overlays. In: Proceedings of the 11th ACM Multimedia Systems Conference [Internet]. Istanbul Turkey: ACM; 2020 [cited 2023 Jul 9]. p. 333–6. Available from: https://dl.acm.org/doi/10.1145/3339825.3393576 EXTERNAL
A rapid prototyping tool to produce 360° video-based immersive experiences enhanced with virtual/multimedia elements. Procedia Computer Science. 2018 Jan 1; 138:441–53. EXTERNAL
Wang M, Lyu XQ, Li Y J, Zhang FL. VR content creation and exploration with deep learning: A survey. Comp Visual Media. 2020 Mar; 6(1):3–28. EXTERNAL
Valenzise G, Alain M, Zerman E, Ozcinar C, editors. Immersive Video Technologies [Internet]. Academic Press; 2022. 630 p. Available from: https://www.elsevier.com/books/immersive-video-technologies/valenzise/978-0-323-91755-1 EXTERNAL
Perazzi F, Sorkine-Hornung A, Zimmer H, Kaufmann P, Wang O, Watson S, Panoramic Video from Unstructured Camera Arrays. Computer Graphics Forum. 2015 May; 34(2):57–68. EXTERNAL
Complete List of VR Camera Systems 2019 [Internet]. 2019 [cited 2020 Jul 15]. Available from: https://delight-vr.com/blog/complete-list-of-vr-cameras-2019/ EXTERNAL
Schroers C, Bazin JC, Sorkine-Hornung A. An Omnistereoscopic Video Pipeline for Capture and Display of Real-World VR. ACM Trans Graph. 2018 Jun 30; 37(3):1–13. EXTERNAL
User I. Introducing Facebook Surround 360: An open, high-quality 3D–360 video capture system [Internet]. Engineering at Meta. 2016 [cited 2023 Jul 8]. Available from: https://engineering.fb.com/2016/04/12/video-engoneering/introducing-facebook-surround-360-an-open-high-quality-3d-360-video-capture-system/ EXTERNAL
Anderson R, Gallup D, Barron JT, Kontkanen J, Snavely N, Hernandez C,, Jump: virtual reality video. ACM Trans Graph. 2016 Nov 11; 35(6):1–13. EXTERNAL
Attal B, Ling S, Gokaslan A, Richardt C, Tompkin J. MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images. arXiv: 200806534 [cs] [Internet]. 2020 Aug 14 [cited 2021 May 3]; Available from: http://arxiv.org/abs/2008.06534 EXTERNAL
Dziembowski A, Mieloch D, Stankiewicz O, Domanski M, Lee G, Seo J. Virtual View Synthesis for 3DoF+ Video. In: 2019 Picture Coding Symposium (PCS). 2019. p. 1–5. EXTERNAL
Dashwood T. A Beginner's Guide to Shooting Stereoscopic 3D. EXTERNAL
Wimmer P. Aufnahme und Wiedergabe stereoskopischer Videos. EXTERNAL
Xu D, Coria L, Nasiopoulos P. Guidelines for capturing high quality stereoscopic content based on a systematic subjective evaluation. In: 2010 17th IEEE International Conference on Electronics, Circuits and Systems [Internet]. Athens, Greece: IEEE; 2010 [cited 2023 Jul 8]. p. 162–5. Available from: http://ieeexplore.ieee.org/document/5724479/ EXTERNAL
Gunnewiek RK, Vandewalle P. HOW TO DISPLAY 3D CONTENT REALISTICALLY. EXTERNAL
Lang M, Hornung A, Wang O, Poulakos S, Smolic A, Gross M. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans Graph. 2010 Jul 26;29(4):1–10. EXTERNAL
Hornung A, Smolic A, Gross M. Novel Stereoscopic Content Production Tools. EXTERNAL
Zilly F, Kluger J, Kauff P. Production Rules for Stereo Acquisition. Proc IEEE. 2011 Apr; 99(4):590–606. EXTERNAL
Zhang J, Liu X, Ye X, Zhao F, Zhang Y, Wu M,, Editable Free-viewpoint Video Using a Layered Neural Representation. arXiv: 210414786 [cs] [Internet]. 2021 Apr 30 [cited 2021 May 3]; Available from: http://arxiv.org/abs/2104.14786 EXTERNAL
Collet A, Chuang M, Sweeney P, Gillett D, Evseev D, Calabrese D,, High-quality streamable free-viewpoint video. ACM Trans Graph. 2015 Jul 27; 34(4):1–13. EXTERNAL
Martin-Brualla R, Pandey R, Yang S, Pidlypenskyi P, Taylor J, Valentin J, LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering. arXiv: 181105029 [cs] [Internet]. 2018 Nov 12 [cited 2020 Aug 6]; Available from: http://arxiv.org/abs/1811.05029 EXTERNAL
Choi J, Jeong JB, Lee S, Ryu ES. Overview of the Volumetric Video Capturing System for Immersive Media. In: 2022 13th International Conference on Information and Communication Technology Convergence (ICTC) [Internet]. Jeju Island, Korea, Republic of: IEEE; 2022 [cited 2023 Jul 8]. p. 574–7. Available from: https://ieeexpiore.ieee.org/document/9953020/ EXTERNAL
Joo H, Liu H, Tan L, Gui L, Nabbe B, Matthews I,, Panoptic Studio: A Massively Multiview System for Social Motion Capture. In: 2015 IEEE International Conference on Computer Vision (ICCV) [Internet]. Santiago, Chile: IEEE; 2015 [cited 2023 Jul 8]. p. 3334–42. Available from: http://ieeexplore.ieee.org/document/7410738/ EXTERNAL
de Aguiar E, Stoll C, Theobalt C, Ahmed N, Seidel HP, Thrun S. Performance Capture from Sparse Multi-view Video. EXTERNAL
Alldieck T, Magnor M, Xu W, Theobalt C, Pons-Moll G. Video Based Reconstruction of 3D People Models. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition [Internet]. Salt Lake City, UT: IEEE; 2018 [cited 2023 Jul 8]. p. 8387–97. Available from: https://ieeexplore.ieee.org/document/8578973/ EXTERNAL
Martinez-Cano FJ. Volumetric filmmaking, new mediums and formats for digital audiovisual storytelling. A|C [Internet]. 2021 Feb 26 [cited 2023 Jul 8]; Available from: https://publication.avanca.org/index.php/avancacinema/article/view/168 EXTERNAL
Eisert P, Schreer O, Feldmann I, Hellge C, Hilsmann A. Volumetric video- acquisition, interaction, streaming and rendering. In: Valenzise G, Alain M, Zerman E, Ozcinar C, editors. Immersive Video Technologies [Internet]. Academic Press; 2023 [cited 2023 May 25]. p. 289–326. Available from: https://www.sciencedirect.com/science/article/piilB9780323917551000171 EXTERNAL
Wu G, Masia B, Jarabo A, Zhang Y, Wang L, Dai Q,, Light Field Image Processing: An Overview. IEEE J Sel Top Signal Process. 2017 Oct; 11(7):926–54. EXTERNAL
Herfet T. 5D Light Field Video Capture. EXTERNAL
Sabater N, Boisson G, Vandame B, Kerbiriou P, Babon F, Hog M,, Dataset and Pipeline for Multi-view Light-Field Video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017. p. 1743–53. EXTERNAL
Flynn J, Broxton M, Debevec P, DuVall M, Graham Fyffe, Overbeck R,, DeepView: View Synthesis With Learned Gradient Descent. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) [Internet]. Long Beach, CA, USA: IEEE; 2019 [cited 2020 May 12]. p. 2362–71. Available from: https://ieeexpiore.ieee.org/document/8953705/ EXTERNAL
Pan Y, Liu Y, Zhang L. LiTrix: A Lightweight Live Light Field Video Scheme for Metaverse Stereoscopic Applications. IEEE Internet Things M. 2023 Jun;6(2):137–42. EXTERNAL
Ziegler M, Engelhardt A, Muller S, Keinert J, Zilly F, Foessel S,, Multi-camera system for depth based visual effects and compositing. In New York, NY, USA: Association for Computing Machinery; 2015 [cited 2021 Mar 23]. p. 1–10. (CVMP '15). Available from: https://doi.org/10.114512824840.2824845 EXTERNAL
Yang JC, Everett M, Buehler C, McMillan L. A Real-Time Distributed Light Field Camera.: 10. EXTERNAL
Herfet T, Chelli K, Le Pendu M. Chapter 6- Acquisition of light field images & videos: Capturing light rays. In: Valenzise G, Alain M, Zerman E, Ozcinar C, editors. Immersive Video Technologies [Internet]. Academic Press; 2023 [cited 2023 Jul 15]. p. 163–71. Available from: https://www.sciencedirect.com/science/article/piilB9780323917551000122 EXTERNAL
Herfet T, Chelli K, Le Pendu M. Chapter 7- Light field representation: The dimensions in light fields. In: Valenzise G, Alain M, Zerman E, Ozcinar C, editors. Immersive Video Technologies [Internet]. Academic Press; 2023 [cited 2023 Jul 15]. p. 173–99. Available from: https://www.sciencedirect.com/science/article/piilB9780323917551000134 EXTERNAL
Feng BY, Jabbireddy S, Varshney A. VIINTER: View Interpolation with Implicit Neural Representations of Images. In: SIGGRAPH Asia 2022 Conference Papers [Internet]. Daegu Republic of Korea: ACM; 2022 [cited 2022 Dec 7]. p. 1–9. Available from: https://dl.acm.org/doi/10.1145/3550469.3555417 EXTERNAL
Shum H, Kang SB. Review of image-based rendering techniques. In: Visual communications and image processing 2000. 2000. p. 2–13. EXTERNAL
Levoy M, Hanrahan P. Light field rendering. In: Proceedings of the 23rd annual conference on Computer graphics and interactive techniques- SIGGRAPH '96 [Internet]. Not Known: ACM Press; 1996 [cited 2020 Jun 30]. p. 31–42. Available from: http://portal.acm.org/citation.cfm?doid=237170.237199 EXTERNAL
Buehler C, Bosse M, McMillan L, Gortler S, Cohen M. Unstructured lumigraph rendering. In: Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH) [Internet]. Los Angeles, CA, USA: ACM Press; 2001 [cited 2020 Jun 6]. p. 425–32. Available from: http://portal.acm.org/citation.cfm?doid=383259.383309 EXTERNAL
Gyeong Ja Jang, Keun Ho Kim, In Kyu Park. Depth image-based rendering of 3D object on mobile device. In: Proceedings of 2004 International Symposium on Intelligent Signal Processing and Communication Systems, 2004 ISPACS 2004. 2004. p. 630–3. EXTERNAL
Bogaert LV, Bonatto D, Fachada S, Lafruit G. Novel view synthesis in embedded virtual reality devices. The Engineering Reality of Virtual Reality. 2022; 6. EXTERNAL
Penner E, Zhang L. Soft 3D reconstruction for view synthesis. ACM Trans Graph. 2017 Nov 20;36(6): 1–11. EXTERNAL
Lange T, Petrovic G, Herfet T. Real-time virtual view rendering for video communication systems using OpenCL. In: 2014 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2014. p. 1–5. EXTERNAL
Vandame B, Sabater N, Boisson G, Doyen D, Allie V, Babon F,, Pipeline for Real-Time Video View Synthesis. In London, UK; 2020. p. 1–6. EXTERNAL
Ziegler M. Advanced image processing for immersive media applications using sparse light-fields [PhD Thesis]. [Erlangen]: Friedrich-Alexander-Universitat Erlanqen-Nurnberq (FAU); 2019. EXTERNAL
Broxton M, Flynn J, Overbeck R, Erickson D, Hedman P, DuVall M,, Immersive Light Field Video with a Layered Mesh Representation. ACM Transactions on Graphics. 2020 Jul; 39(4): 15. EXTERNAL
Mildenhall B, Srinivasan PP, Ortiz-Cayon R, Kalantari NK, Ramamoorthi R, Ng R,, Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. arXiv: 190500889 [cs] [Internet]. 2019 May 2 [cited 2020 May 12]; Available from: http://arxiv.org/abs/1905.00889 EXTERNAL
Gao K, Gao Y, He H, Lu D, Xu L, Li J. NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review [Internet]. arXiv; 2022 [cited 2022 Oct 10]. Available from: http://arxiv.org/abs/2210.00379 EXTERNAL
Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In: Vedaldi A, Bischof H, Brox T, Frahm JM, editors. Cham: Springer; 2020. p. 405–21. EXTERNAL
Xie Y, Takikawa T, Saito S, Litany O, Yan S, Khan N,, Neural Fields in Visual Computing and Beyond. 2022 Apr 5 [cited 2022 Jun 22]; Available from: http://arxiv.org/abs/2111.11426 EXTERNAL
Virtualization Chemnitz Opera House Part 1: Photogrammetry-SocialSTAGE-VR [Internet]. [cited 2024 Jul 8]. Available from: https://socialstagevr.de/virtualization-chemnitz-opera-house-part-1-photogrammetryl EXTERNAL
Virtualization Chemnitz Opera House Part 2: Post-Modeling and Optimization-SocialSTAGE-VR [Internet]. [cited 2024 Jul 8]. Available from: https://socialstagevr.de/virtualization-chemnitz-opera-house-part-2-post-modeling-and-optimizationl EXTERNAL
Ziegler M, Keinert J, Holzer N, Wolf T. IMMERSIVE VIRTUAL REALITY FOR LIVE-ACTION VIDEO USING CAMERA ARRAYS. EXTERNAL
RX0 ll premium tiny tough camera |DSC-RXOM2G | Sony United Kingdom [Internet]. [cited 2023 Jul 29]. Available from: https://www.sony.co.uk/electronics/cyber-shot-compact-cameras/dsc-rx0m2?locale=enGB EXTERNAL
3D rig. In: Wikipedia [Internet]. 2023 [cited 2024 Jul 8]. Available from: https://en.wikipedia.org/w/index.php?title=3D_rig&oldid=1157862944 EXTERNAL
Reda F, Kontkanen J, Tabellion E, Sun D, Pantofaru C, Curless B. FILM: Frame Interpolation for Large Motion. In: Avidan S, Brostow G, Cisse M, Farinella GM, Hassner T, editors. Computer Vision-ECCV 2022 [Internet]. Cham: Springer Nature Switzerland; 2022 [cited 2024 Jul 8]. p. 250–66. (Lecture Notes in Computer Science; vol. 13667). Available from: https://link.springer.com/10.1007/978-3-031-20071-7_15 EXTERNAL
Blackmagic URSA Cine Immersive [Internet]. [cited 2024 Jul 8]. Available from: https://www.blackmagicdesign.com/media/release/20240611-02 EXTERNAL
Schonberger JL, Frahm JM. Structure-from-Motion Revisited. In Las Vegas, NV, USA; 2016 [cited 2021 Sep 3]. p. 4104–13. Available from: http://ieeexplore.ieee.org/document/77808141 EXTERNAL
Ma J, Jiang X, Fan A, Jiang J, Yan J. Image Matching from Handcrafted to Deep Features: A Survey. Int J Comput Vis. 2021 Jan; 129 (1):23–79. EXTERNAL
Scharstein D, Szeliski R, Zabih R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In Kauai, HI, USA; 2001 [cited 2021 Nov 27]. p. 131–40. Available from: http://ieeexplore.ieee.org/document/988771/ EXTERNAL
Hamzah RA, Ibrahim H. Literature Survey on Stereo Vision Disparity Map Algorithms. Journal of Sensors. 2016;1–23. EXTERNAL
Laga H, Jospin LV, Boussaid F, Bennamoun M. A Survey on Deep Learning Techniques for Stereo-based Depth Estimation. IEEE Trans Pattern Anal Mach Intell. 2022 Apr 1; 44(4):1738–64. EXTERNAL
72. Swan Lake. In: Wikipedia [Internet]. 2024 [cited 2024 Jul 22]. Available from: https://en.wikipedia.org/w/index.php?title=Swan_Lake&oIdid=1235185024 EXTERNAL
Cinderella. In: Wikipedia [Internet]. 2024 [cited 2024 Jul 22]. Available from: https://en.wikipedia.org/w/index.php?title=Cinderella&oldid=1235190617 EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-MOO-3010.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Joachim Keinert, Tobias Jaschke, Hendrik Hawer, Andree Josef, Frank Dittrich, and Siegfried Fossel; Bringing Stage Events with Real Actors to Your Home in VR: A Low Complexity Solution for Immersive Media Streaming, MTS 2024, Article 10 (pp. 1 to 21); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3010

Snippet:

Joachim Keinert, Tobias Jaschke, Hendrik Hawer, Andree Josef, Frank Dittrich, and Siegfried Fossel; Bringing Stage Events with Real Actors to Your Home in VR: A Low Complexity Solution for Immersive Media Streaming, MTS 2024, Article 10 (pp. 1 to 21); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at https://doi.org/10.5594/MOO/3010

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Joachim Keinert, Tobias Jaschke, Hendrik Hawer, Andree Josef, Frank Dittrich, and Siegfried Fossel; <cite>Bringing Stage Events with Real Actors to Your Home in VR: A Low Complexity Solution for Immersive Media Streaming</cite>, MTS 2024, Article 10 (pp. 1 to 21); SMPTE, 2024, ISBN: 978-1-61482-965-2. Available at <a href="https://doi.org/10.5594/MOO/3010" target="_blank" rel="noopener">https://doi.org/10.5594/MOO/3010</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Joachim Keinert, Tobias Jaschke, Hendrik Hawer, Andree Josef, Frank Dittrich, and Siegfried Fossel; <cite id="bib-10-5594-moo-3010">Bringing Stage Events with Real Actors to Your Home in VR: A Low Complexity Solution for Immersive Media Streaming</cite>, MTS 2024, Article 10 (pp. 1 to 21); SMPTE, 2024, ISBN: 978-1-61482-965-2
<span class="doi">10.5594/MOO/3010</span>
</li>