GPU-Based Real-Time System for Cinematic Virtual Reality Production

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2016, No. 10, pp. 1–13
Abstract: Relatively inexpensive Head Mounted Display (HMD) devices made possible by advances in digital imaging technology are moving the immersive virtual and augmented reality mediums into the entertainment mainstream. This departure from the traditional art of linear story telling requires a richer production workflow and toolset capable of seamlessly integrating numerous potential narratives. An effective Virtual Reality (VR) workflow requires low-latency real-time synchronized capture of multiple live high-resolution video streams, real-time stitching and viewing of the captured assets, and real-time composition with Computer Generated (CG) elements optimally rendered for the HMD. This paper presents a real-time system for virtual reality production that encompasses optimal high-bandwidth data transfers, Graphics Processor Unit (GPU) -based video stitching, and techniques for optimally rendering to the individual display characteristics of specific VR HMDs for on-set preview.
Publication Date: 2016-10-25
DOI: 10.5594/M001699
ISBN: Print: 978-1-61482-957-7
Link: https://doi.org/10.5594/M001699
Author(s): Thomas TrueNVIDIA Corporation
Dennis SandlerNVIDIA Corporation
Pablo OdoricoNVIDIA Corporation
Keyword(s): Virtual Reality, Video Stitching, GPU, Post Production
Copyright: © 2016 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

Direct to GPU Video Transfers [Active]
M. Grundmann, V. Kwatra, and I. Essa, “Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths,” IEEE Conference on Computer Vision and Pattern Recognition, 2011. EXTERNAL
L. Shuaicheng, L. Yuan, P. Tan and J. Sun, “Bundled Camera Paths for Video Stabilization,” ACM Transactions on Graphics, Vol. 32, No. 4, July 2013. EXTERNAL
https://en.wikipedia.org/wiki/Circle-Vision_360 EXTERNAL
A. Karpenko, D. Jacobs, J Baek and M. Levoy, “Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes,” Stanford Tech Report, March 2011. EXTERNAL
http://360designs.io/ EXTERNAL
S. Battiato, G. Messina and A Castorina, “Exposure Correction for Imaging Devices: An Overview,” Single-Sensor Imaging: Methods and Applications for Digital Camera, CRC Press: Boca Raton, FL, 2008. EXTERNAL
https://developer3.oculus.com/downloads/platform/1.8.0/Oculus_Platform_SDK/ EXTERNAL
Peter Eisert and Philippe Bekaert, “Omni-Directional Video,” Digital Representations of the Real World: How to Capture, Model and Render Visual Reality, CRC Press: Boca Raton, FL, 2015. EXTERNAL
A. Efros and W. Freeman, “Image Quilting for Texture Synthesis and Transfer,” Proceedings of ACM SIGGRAPH 2001. EXTERNAL
S. Bell, A. Troccoli and K. Pulli, “A Non-Linear Filter for Gyroscope-Based Video Stabilization,” Computer Vision – ECCV 2014. EXTERNAL
http://www.kolor.com/autopano EXTERNAL
James Davis, “Mosaics of Scenes with Moving Objects,” Proceedings of CVPR 1998, Santa Barbara, CA, June 1998. EXTERNAL
C. Corment, C. Leiserson, and R. Rivest, Introduction to Algorithms, MIT Press, 1990. EXTERNAL
https://shop.gopro.com/virtualreality/omni---all-inclusive/MHDHX-006.html EXTERNAL
F Liu, M. Gleicher, J. Wang, J. Hailin and A. Agrawala, “Subspace Video Stabilization,” ACM Transactions on Graphics, Vol. 30, No. 1, 2011. EXTERNAL
W. Jiang and J. Gu, “Video Stitching with Spatial-Temporal Content-Preserving Warping,” IEEE 2015. EXTERNAL
S. Baker, E. Bennett, S. B. Kang and R Szeliski, “Removing Rolling Shutter Wobble,” IEEE Conference on Computer Vision and Pattern Recognition, 2010. EXTERNAL
E. Adelson, C. Anderson, J. Bergen, P. Burt and J. Ogden, “Pyramid Method in Image Processing”, RCA Engineer, Vol. 29, No 6, pp 33-41, November 1984. EXTERNAL
http://www.video-stitch.com/ EXTERNAL
M. Grundmann, V. Kwatra, D. Castro and I. Essa, “Calibration-Free Rolling Shutter Removal,” IEEE International Conference on Computational Photography, 2012. EXTERNAL
P. Burt and E. Adelson, “A Multiresolution Spline With Application to Image Mosaics,” ACM Transactions on Graphics, Vol. 2, No. 4, pp 217-236, October 1983, EXTERNAL
http://www.htcvive.com/us/ EXTERNAL
https://www3.oculus.com/en-us/rift/ EXTERNAL
http://360designs.io/product/mini-eye-4-professional-360-camera/ EXTERNAL
http://www.pufferfishdisplays.co.uk/ EXTERNAL
http://www.nvidia.com/content/gtc/documents/1060_gtc09.pdf EXTERNAL
Y. Boykov and M. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images,” Proceedings of ICCV, Vancouver Canada, July 2001. EXTERNAL
M. L. Gleicher and F. Liu, “Re-Cinematography: Improving the Camera Dynamics of Casual Video,” ACM Transactions on Multimedia, Vol. 5, No. 1, October 2008. EXTERNAL
https://en.wikipedia.org/wiki/IMAX EXTERNAL
https://en.wikipedia.org/wiki/PCI_Express EXTERNAL
F. Liu and Jin Hailin, “Content-Preserving Warps for 3D Video Stabilization” ACM Transactions on Graphics, Vo. 28 , No. 3, pp. 1-9, July, 2009. EXTERNAL
https://facebook360.fb.com/ EXTERNAL
Y. Li, J. Sun, C. Tang and H. Shum, “Lazy Snapping,” ACM, 2004. EXTERNAL
https://gopro.com/odyssey EXTERNAL
Y. Matsushita, E. Ofek, W. Ge, T. X. Tang and H-Y Shum, “Full-Frame Video Stabilization with Motion Inpainting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 28, No 7. July 2008. EXTERNAL
http://www.video-stitch.com/live-vr/ EXTERNAL
SMPTE ST 2081-10:2015 [Superseded]
SMPTE ST 2082-10:2015 [Superseded]

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001699.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Thomas True, Dennis Sandler, and Pablo Odorico; GPU-Based Real-Time System for Cinematic Virtual Reality Production, SMPTE Meetings and Conferences ( October 2016); SMPTE, 2016, ISBN: 978-1-61482-957-7. Available at https://doi.org/10.5594/M001699

Snippet:

Thomas True, Dennis Sandler, and Pablo Odorico; GPU-Based Real-Time System for Cinematic Virtual Reality Production, SMPTE Meetings and Conferences ( October 2016); SMPTE, 2016, ISBN: 978-1-61482-957-7. Available at https://doi.org/10.5594/M001699

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Thomas True, Dennis Sandler, and Pablo Odorico; <cite>GPU-Based Real-Time System for Cinematic Virtual Reality Production</cite>, SMPTE Meetings and Conferences ( October 2016); SMPTE, 2016, ISBN: 978-1-61482-957-7. Available at <a href="https://doi.org/10.5594/M001699" target="_blank" rel="noopener">https://doi.org/10.5594/M001699</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Thomas True, Dennis Sandler, and Pablo Odorico; <cite id="bib-10-5594-m001699">GPU-Based Real-Time System for Cinematic Virtual Reality Production</cite>, SMPTE Meetings and Conferences ( October 2016); SMPTE, 2016, ISBN: 978-1-61482-957-7
<span class="doi">10.5594/M001699</span>
</li>