Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 132, No. 5, pp. 30–37
Abstract: Recent improvements in extended reality (XR) technology created an increase in XR products and solutions in the industry while raising new requirements for new or improved architectural concepts. This need can be particularly complex as XR applications often relate both to 3D geometric rendering and multimedia paradigms. This article outlines the main concepts relevant to XR, both from a game engineering and a multimedia streaming system perspective. XR requires new metadata and media/game orchestration to allow complex interaction between users, objects, and (volumetric) multimedia content, which also results in new requirements on synchronization (i.e., for global object state and positioning). Furthermore, the article presents the functional blocks needed in new XR system architectures and how they will glue both (game and media) spaces together. The discussion of functional components and architecture relates to the ongoing activities in relevant standardization bodies like Khronos, Moving Picture Experts Group (MPEG) and The 3rd Generation Partnership Project (3GPP). To make XR successful in the long term, the industry needs to agree on interoperable solutions and how to merge both game and media paradigms to allow complex multiuser XR applications.
Publication Date: 2023-06-01
DOI: 10.5594/JMI.2023.3269752
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2023.3269752
Author(s): Simon N. B. Gunkel
bio
Simon N. B. Gunkel is a senior research and development scientist at TNO, The Netherlands Organization for Applied Scientific Research, The Hague, The Netherlands.
Emmanouil Potetsianakis
bio
Emmanouil Potetsianakis is a researcher with Xiaomi Technology Netherlands, Dordrecht, The Netherlands, working on media systems, technologies, and standards.
Tessa E. Klunder
bio
Tessa E. Klunder is a research and development scientist at TNO, The Netherlands Organization for Applied Scientific Research, The Hague, The Netherlands.
Alexander Toet
bio
Alexander Toet is a retired senior research scientist from TNO Human Factors, Soesterberg, The Netherlands, where he investigated multisensory (visual, auditory, tactile, and olfactory) perception and the crossmodal interactions between these different senses.
Sylvie S. Dijkstra-Soudarissanane
bio
Sylvie S. Dijkstra-Soudarissanane is a senior scientist at TNO, The Netherlands Organization for Applied Scientific Research, The Hague, The Netherlands such as 5G/6G, SDN/NFV, edge/cloud computing, as well as 3D volumetric media processing elements like point clouds and light field.
Keyword(s): Augmented Reality, Extended Reality, Game Engine, Immersive Media, Multimedia Streaming, Virtual Reality
Copyright: © 2023 SMPTE

Bibliographic Reference(s)

ISO.23090-2 — International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) 23090-2, “Information Technology—Coded Representation of Immersive Media (MPEG-I)—Part 2: Omnidirectional Media Format (OMAF).” EXTERNAL
30. J. M. Boyce et al., “MPEG Immersive Video Coding Standard,” Proc. IEEE, 109(9):1521–1536, 2021. EXTERNAL
C. Willemse, “Social Touch Technologies. How They feel and How They Make You Feel,” PhD thesis, University of Twente, 2018. EXTERNAL
P. Poller, M. Chikobava, J. Hodges, M. Kritzler, F. Michahelles, and T. Becker, Back-End Semantics for Multimodal Dialog on XR Devices,” 26th Int. Conf. on Intell. User Interfaces, Association for Computing Machinery: New York, NY, 2021. EXTERNAL
S. Gül, D. Podborski, T. Buchholz, T. Schierl, and C. Hellge, “Low-Latency Cloud-Based Volumetric Video Streaming Using Head Motion Prediction,” 30th ACM Workshop on Netw. and Operating Syst. Support for Digital Audio and Video (NOSSDAV ’20), Istanbul, Turkey, 2020. EXTERNAL
S. Kim, M. Billinghurst, and K. Kim, “Multimodal Interfaces and Communication Cues for Remote Collaboration,” J. Multimodal User Interfaces, 14:313–319, Oct. 2020. EXTERNAL
A Global Resource Guide to XR Collaboration, XR IGNITE, 2021. EXTERNAL
WG2 MPEG Requirements, “Annex B Report of Characterizing Rendering Methods for Immersive Applications,” International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) JTC 1/SC 29/WG 2 N 0035, 2021. EXTERNAL
European Telecommunications Standards Institute Technical Specification (ETSI TS) 103 286-2 V1.1.1, “Digital Video Broadcasting (DVB); Companion Screens and Streams; Part 2: EXTERNAL
O. Holland et al., “The IEEE 1918.1 “Tactile Internet” Standards Working Group and Its Standards,” Proc. IEEE, 107(2):256–279, 2019. EXTERNAL
J. B. F. van Erp, “Guidelines for the Use of Vibro-Tactile Displays in Human Computer Interaction,” 2002. EXTERNAL
M. Montagud, P. Cesar, F. Boronat, and J. Jansen, MediaSync: Handbook on Multimedia Synchronization, Springer International Publishing, 2018. EXTERNAL
O. S. Good, “How Lucasfilm Used Unreal Engine to Make the Mandalorian—You Need the World’s Most Badass Curved Monitor, for Starters.” Accessed: May 21, 2021. [Online]. Available: https://www.polygon.com/tv/2020/2/20/21146152/the-mandalorian-making-of-video-unreal-engine-projection-screen EXTERNAL
A. Vetro, T. Wiegand, and G. J. Sullivan, “Overview of the Stereo and Multiview Video Coding Extensions of the H. 264/ MPEG-4 AVC Standard,” Proc. IEEE, 99(4):626–642, 2011. EXTERNAL
M. Broxton et al., “Immersive Light Field Video With a Layered Mesh Representation,” ACM Trans. Graphics (TOG), 39(4):86–81, 2020. EXTERNAL
J. M. Silva, M. Orozco, J. Cha, A. E. Saddik, and E. M. Petriu, “Human Perception of Haptic-to-Video and Haptic-to-Audio Skew in Multimedia Applications,” ACM Trans. Multimedia Comput. Commun. Appl. (TOMM), 9(2):1–16, May 2013. EXTERNAL
K. Ahuja et al., “Interactive Eye Sensing Using Off-the-Shelf, Smartphone-Based VR Headsets,” Proc. 2018 ACM on Interact., Mobile, Wearable and Ubiquitous Technol., Jul. 2018. EXTERNAL
H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang, “Pix2vox: Context-Aware 3D Reconstruction From Single and Multi-View Images,” IEEE/CVF Int. Conf. on Comput. Vision, IEEE, 2019. EXTERNAL
A. Patney et al., “Perceptually-Based Foveated Virtual Reality,” Proc. 2016 ACM SIGGRAPH Emerg. Technol., pp. 1–2, Jul. 2016. EXTERNAL
A. Steed, D. Roberts, R. Schroeder, and I. Heldal, “Interaction Between Users of Immersion Projection Technology Systems,” 11th Int. Conf. on Human Comput. Interact. (HCI), 2005. EXTERNAL
Y. Zhao et al., “Identity Preserving Face Completion for Large Ocular Region Occlusion,” BMVC2018, 2018. EXTERNAL
O. Bimber and R. Raskar, Spatial Augmented Reality: Merging Real and Virtual Worlds, AK Peters/CRC Press: New York, 2019. EXTERNAL
P. Hoschka, “An Introduction to the Synchronized Multimedia Integration Language,” IEEE MultiMedia, 5(4):84–88, Oct.-Dec. 1998. EXTERNAL
N. Numan, F. T. Haar, and P. Cesar, “Generative RGB-D Face Completion for Head-Mounted Display Removal,” Proc 2021 IEEE Conf. on Virtual Reality and 3D User Interfaces Abstr. and Workshops (VRW), pp. 109–116, 2021. EXTERNAL
Y. A. G. V. Boas, “Overview of Virtual Reality Technologies,” Interact. Multimedia Conf., vol. 2013, 2013. EXTERNAL
D. Graziosi, O. Nakagami, S. Kuma, A. Zaghetto, T. Suzuki, and A. Tabatabai, “An Overview of Ongoing Point Cloud Compression Standardization Activities: Video-Based (V-PCC) and Geometry-Based (G-PCC),” APSIPA Trans. Signal Inf. Process., 9:E13, 2020. EXTERNAL
S. Rokhsaritalemi, A. Sadeghi-Niaraki, and S. M. Choi, “A Review on Mixed Reality: Current Trends, Challenges and Prospects,” Appl. Sci., 10:636, Jan. 2020. EXTERNAL
Y. Zhao et al., “Mask-Off: Synthesizing Face Images in the Presence of Head-Mounted Displays,” Proc. 2019 IEEE Conf. on Virtual Reality and 3D User Interfaces (VR), pp. 267–276, 2019. EXTERNAL
S. Shekhar, S. K. Feiner, and W. G. Aref, “Spatial Computing,” Commun. ACM, 59:72–81, 2015. EXTERNAL
F. Biocca and C. Harms, “Defining and Measuring Social Presence: Contribution to the Networked Minds Theory and Measure,” Proc. PRESENCE, Jan. 2002. EXTERNAL
A. Doumanoglou et al., “Benchmarking Open-Source Static 3D Mesh Codecs for Immersive Media Interactive Live Streaming,” IEEE J. Emerg. Sel. Topics Circuits Syst., 9(1):190–203, 2019. EXTERNAL
3rd Generation Partnership Project (3GPP), “Technical Specification Group Services and System,” Technical Report TR 26.928 V1.2.1, Extended Reality (XR) in 5G, Dec. 2020. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2023.3269752.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Simon N. B. Gunkel, Emmanouil Potetsianakis, Tessa E. Klunder, Alexander Toet, and Sylvie S. Dijkstra-Soudarissanane; Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 5, June 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3269752

Snippet:

Simon N. B. Gunkel, Emmanouil Potetsianakis, Tessa E. Klunder, Alexander Toet, and Sylvie S. Dijkstra-Soudarissanane; Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 5, June 2023); SMPTE, 2023. Available at https://doi.org/10.5594/JMI.2023.3269752

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Simon N. B. Gunkel, Emmanouil Potetsianakis, Tessa E. Klunder, Alexander Toet, and Sylvie S. Dijkstra-Soudarissanane; <cite>Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 5, June 2023); SMPTE, 2023. Available at <a href="https://doi.org/10.5594/JMI.2023.3269752" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2023.3269752</a></span>

SMPTE's HTML Pub

Preview:

Simon N. B. Gunkel, Emmanouil Potetsianakis, Tessa E. Klunder, Alexander Toet, and Sylvie S. Dijkstra-Soudarissanane; Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 5, June 2023); SMPTE, 2023
doi: 10.5594/JMI.2023.3269752
url: https://doi.org/10.5594/JMI.2023.3269752

Snippet:

<li>
Simon N. B. Gunkel, Emmanouil Potetsianakis, Tessa E. Klunder, Alexander Toet, and Sylvie S. Dijkstra-Soudarissanane; <cite id="bib-10-5594-jmi-2023-3269752">Immersive Experiences and XR: A Game Engine or Multimedia Streaming Problem?</cite>, SMPTE Motion Imaging Journal ( Volume: 132, Issue: 5, June 2023); SMPTE, 2023
<span class="doi">10.5594/JMI.2023.3269752</span>
</li>