Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production

Metadata

Publisher: SMPTE
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Motion Imaging Jour.
Volume: 134, No. 6, pp. 50–58
Abstract: The ability to capture a volumetric representation of performers in a production environment would enable a multitude of novel on-set and post-production workflows. Current methods are impractical due to their reliance on large numbers of cameras and consistent lighting conditions. We propose a framework for creating 2.5D assets from a single monocular video, allowing digitized performers to be viewed from a range of angles with the appearance of depth. An application processes the video offline, using depth and segmentation AI models to create a packaged 2.5D asset. This is then loaded into Disguise Designer for pre-visualisation of the performers within the virtual stage. Analysis of state-of-the-art depth inference models, using videos captured to represent the challenges of production environments, shows that it is possible to obtain coherent video depth maps in these conditions. However, metric models do not always accurately identify absolute depth values, and it is necessary to use models tailored specifically for video to ensure temporal consistency in the results. This work aims to serve as a foundation for more comprehensive 3D volumetric capture of performers in real-world production environments.
Publication Date: 2025-10-01
DOI: 10.5594/JMI.2025/FOOS6445
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2025/FOOS6445
Author(s): Chris Nash
Nathan Butt
Andrea Loriedo
Robin Spooner
Taegyun Ha
Philip Coulam-Jones
James Bentley
Aljosa Smolic
Keyword(s): Volumetric Capture, Depth Estimation, Computer Vision, Virtual Production
Copyright: © 2025 SMPTE

Bibliographic Reference(s)

Universitat Pompeu Fabra, “Emerald consortium homepage,” 2024. [Online]. Available: https://www.upf.edu/web/emerald. EXTERNAL
A. Smolic, “FaVoRe - Fast Volumetric Reconstruction for Media and Video Communication,” 2024. [Online]. Available: https://www.hslu.ch/de-ch/hochschule-luzern/forschung/projekte/detail/?pid=6705. EXTERNAL
Universitat Pompeu Fabra, “MAX-R: Mixed Augmented and Extended Reality Media Pipeline,” 2023. [Online]. Available: https://www.upf.edu/web/max-r. EXTERNAL
Disguise, “Disguise homepage,” 2025. [Online]. Available: https://www.disguise.one/. EXTERNAL
D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-NeRF: Structured view-dependent appearance for neural radiance fields,” arXiv:2112.03907, 2021. EXTERNAL
J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-NeRF 360: Unbounded anti-aliased neural radiance fields,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVRR), pp. 5470–5479, 2022. EXTERNAL
F. Wang, M.-J. Rakotosaona, M. Niemeyer, R. Szeliski, M. Pollefeys, and F. Tombari, “UniSDF: Unifying neural representations for high-fidelity 3D reconstruction of complex scenes with reflections,” arXiv:2312.13285, 2023. EXTERNAL
S. Kheradmand, D. Rebain, G. Sharma, W. Sun, J. Tseng, H. lsack, A. Kar, A. Tagliasacchi, and K. M. Yi, “3D Gaussian Splatting as Markov Chain Monte Carlo,” arXiv:2404.09591, 2024. EXTERNAL
D. Charatan, S. Li, A. Tagliasacchi, and V. Sitzmann, “pixelSplat: 3D Gaussian Splats from image pairs for scalable generalizable 3D reconstruction,” Proc. IEEE/CVF Cenf. Comput. Vis. Pattern Recognit. (CVPR), pp. 19457–19467, Jun. 2024. EXTERNAL
J. C. Lee, D. Rho, X. Sun, J. H. Ko, and E. Park, “Compact 3D Gaussian representation for radiance field,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 21719–21728, 2024. EXTERNAL
J. Sun, H. Jiao, G. Li, Z. Zhang, L. Zhao, and W. Xing, “3DGStream: On-the-fly training of 3D Gaussians for efficient streaming of photorealistic free-viewpoint videos,” Proc. IEEE/CVF Conf Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024. EXTERNAL
M. Kocabas, J.-H. R. Chang, J. Gabriel, O. Tuzel, and A. Ranjan, “HUGS: Human Gaussian splatting,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024. EXTERNAL
S. Croci, A. Charisoudis, P. Haslbauer, L. K. Yung, and A. Smolic, “Reconstruction Principles,” Fast Volumetric Reconstruction: Deliverable D3.1, 2024. EXTERNAL
S. Zheng, B. Zhou, R. Shao, B. Liu, S. Zhang, L. Nie, and Y. Liu, “Generalizable pixel-wise 3D Gaussian splatting for real-time human novel view synthesis,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024. EXTERNAL
R. Birkl, D. Wofk, and M. Müller, “MiDaS v3.1-A model zoo for robust monocular relative depth estimation,” arXiv:2307.14460, 2023. EXTERNAL
L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth Anything: Unleashing the power of large-scale unlabeled data,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024. EXTERNAL
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth Anything V2,” arXiv:2406.09414, 2024. EXTERNAL
A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. R. Richter, and V. Koltun, “Depth Pro: Sharp Monocular Metric Depth Less Than a Second,” arXiv:2410.02073, Oct. 2024. arXiv. EXTERNAL
B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler, “Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation,” Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. ResearchGate. EXTERNAL
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” arXiv:2112.10752, 2021. EXTERNAL
W. Hu, X. Gao, X. Li, S. Zhao, X. Cun, Y. Zhang, L. Quan, and Y. Shan, “DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videos,” arXiv:2409.02095, Sep. 2024. EXTERNAL
B. Ke, D. Narnhofer, S. Huang, L. Ke, T. Peters, K. Fragkiadaki, A. Obukhov, and K. Schindler, “Video Depth without Video Models,” arXiv:2411.79189, 2024. EXTERNAL
S. Chen, H. Guo, S. Zhu, F. Zhang, Z. Huang, J. Feng, and B. Kang, “Video Depth Anything: Consistent Depth Estimation for Super-Long Videos,” arXiv:2501.12375, Jan. 2025. EXTERNAL
J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv:1804.02767, 2018. EXTERNAL
W. Abdulla, “Mask R-CNN for Object Detection and Instance Segmentation on Keras and TensorFlow,” 2017. [Online]. Available: https://github.com/matterport/Mask_RCNN. EXTERNAL
L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation,” arXiv:1802.02611, 2018. EXTERNAL
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick, “Segment Anything,” arXiv:2304.02643, 2023. EXTERNAL
N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Radle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. Alwala, N. Vasudev, N. Carion, and C.Y. Wu, “SAM 2: Segment Anything in Images and Videos,” arXiv:2408.00714, 2024. EXTERNAL
J. Yang, M. Gao, Z. Li, S. Gao, F. Wang, and F. Zheng, “Track Anything: Segment Anything Meets Videos,” arXiv:2304.11968, 2023. EXTERNAL
Disguise, “2.5D Format Specification,” 2024. [Online]. Available: https://developer.disguise.one/formats/2p5d/. EXTERNAL
J. P. Telotte, “Ub Iwerks' (Multi)Plain Cinema,” Animation, 1 (1): pp. 9–24, Jul. 2006. EXTERNAL
Cuebric, “Cuebric Homepage,” 2025. [Online]. Available: https://cuebric.com/. EXTERNAL
S. Orts-Escolano, C. Rhemann, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson, S. Khamis, and M. Dou, “Holoportation: Virtual 3D Teleportation in Real-time,” presented at the 29th Annu. Symp. User Interface Softw. Technol., Oct. 16–19.2016. EXTERNAL
S. Weder, J. L. Schonberger, M. Pollefeys, and M. R. Oswald, “NeuralFusion: Online Depth Fusion in Latent Space,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 3162–3172, 2021. EXTERNAL
D. Azinovic, R. Martin-Brualla, D. B. Goldman, M. Niessner, and J. Thies, “Neural RGB-D Surface Reconstruction,” Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 6290–6302, Jun. 2022. EXTERNAL
Z. Zeng, V. Deschaintre, I. Georgiev, Y.- Geoffroy, Y. Hu, F. Luan, L.-Q. Yan, and M. Hasan, “RGB-X: Image Decomposition and Synthesis Using Material- and Lighting-Aware Diffusion Models,” ACM SIGGRAPH Conf. Papers, 2024. EXTERNAL
R. Liang, Z. Gojcic, H. Ling, J. Munkberg, J. Hasselgren, Z.-H. Lin, J. Gao, A. Keller, N. Vijaykumar, S. Fidler, and Z. Wang, “DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models,” arXiv:2501.18590, 2025. EXTERNAL
G. A. Thomas, F. M. Rivera, L. Kelso, B. Weir, P. Rich, and O. Moolan-Feroze, “Live Music in Immersive Virtual Spaces,” Int. Broadcasting Conv. (IBC) Tech. Papers, 2024. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2025-FOOS6445.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Chris Nash, Nathan Butt, Andrea Loriedo, Robin Spooner, Taegyun Ha, Philip Coulam-Jones, James Bentley, and Aljosa Smolic; Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production, MIJ 2025, Volume 134, Number 6 (pp. 50 to 58); SMPTE, 2025. Available at https://doi.org/10.5594/JMI.2025/FOOS6445

Snippet:

Chris Nash, Nathan Butt, Andrea Loriedo, Robin Spooner, Taegyun Ha, Philip Coulam-Jones, James Bentley, and Aljosa Smolic; Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production, MIJ 2025, Volume 134, Number 6 (pp. 50 to 58); SMPTE, 2025. Available at https://doi.org/10.5594/JMI.2025/FOOS6445

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Chris Nash, Nathan Butt, Andrea Loriedo, Robin Spooner, Taegyun Ha, Philip Coulam-Jones, James Bentley, and Aljosa Smolic; <cite>Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production</cite>, MIJ 2025, Volume 134, Number 6 (pp. 50 to 58); SMPTE, 2025. Available at <a href="https://doi.org/10.5594/JMI.2025/FOOS6445" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2025/FOOS6445</a></span>

SMPTE's HTML Pub

Preview:

Chris Nash, Nathan Butt, Andrea Loriedo, Robin Spooner, Taegyun Ha, Philip Coulam-Jones, James Bentley, and Aljosa Smolic; Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production, MIJ 2025, Volume 134, Number 6 (pp. 50 to 58); SMPTE, 2025
doi: 10.5594/JMI.2025/FOOS6445
url: https://doi.org/10.5594/JMI.2025/FOOS6445

Snippet:

<li>
Chris Nash, Nathan Butt, Andrea Loriedo, Robin Spooner, Taegyun Ha, Philip Coulam-Jones, James Bentley, and Aljosa Smolic; <cite id="bib-10-5594-jmi-2025-foos6445">Integration of Depth and Volumetric Capture Workflows for 2.5D Virtual Production</cite>, MIJ 2025, Volume 134, Number 6 (pp. 50 to 58); SMPTE, 2025
<span class="doi">10.5594/JMI.2025/FOOS6445</span>
</li>