GPU-Based Realtime System for Cinematic Virtual Reality Production

Bibliographic Reference(s)

• Direct to GPU Video Transfers [Active]
• S. Battiato, G. Messina, and A. Castorina, “Exposure Correction for Imaging Devices: An Overview,” in Single-Sensor Imaging: Methods and Applications for Digital Camera, CRC Press: Boca Raton, FL, 2008. EXTERNAL
• S. Bell, A. Troccoli, and K. Pulli, “A Non-Linear Filter for Gyroscope-Based Video Stabilization,” Computer Vision—ECCV 2014. EXTERNAL
• C. Corment, C. Leiserson, and R. Rivest, Introduction to Algorithms, MIT Press: Cambridge, MA, 1990. EXTERNAL
• E. Adelson et al., “Pyramid Method in Image Processing,” RCA Engineer, 29(6):33–41, Nov. 1984. EXTERNAL
• GoPro, Odyssey, https://gopro.com/odyssey EXTERNAL
• M. Grundmann, V. Kwatra, and I. Essa, “Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths,” in Proc. IEEE Conf. Comput. Vision Pattern Recognition, pp. 225–232, 2011. same central processing unit (CPU) within the system in order to avoid a data transfer over the slower Intel QuickPath Interconnect (QPI) between CPUs in current Non-Uniform Memory Architecture (NUMA) system architectures. For this reason, we utilized a single CPU system. Display We support a variety of target displays in our stitching pipeline. These include a regular 4K computer display, an HMD, and a Pufferfish display. Computer Display In the case of a standard 16:9 4K computer display, the panorama resulting from the video stitching operation is texture mapped onto the inside of a sphere. The viewer is placed into the center of the sphere with a 180° field of view projected onto the flat computer display. This method was used to validate our implementation and to permit different people to view our realtime results. HMD The stitched panorama was also rendered for an Oculus VR headset using the OpenVR38 toolkit. This rendered view allows the wearer of the HMD to see the whole 360 view from the position of the camera by looking around. Because the content is video and the camera is stationary, the viewer cannot zoom, but only pan and tilt to see the complete 360° view. Spherical Display In an effort to provide a spherical projection that could be viewed by more than one person, useful for on-set review and monitoring of captured content, we support and also use a Pufferfish spherical display39 by Pufferfish, Ltd. of Edinburgh for our realtime streaming demonstration at SIGGRAPH 2016 as shown in Fig. 3. The Pufferfish display provides a rear projection inside a translucent sphere. The input to the PufferSphere is a standard HD resolution image, and an internal computer performs the required warp for the projection on to the inside of EXTERNAL
• A. Efros and W. Freeman, “Image Quilting for Texture Synthesis and Transfer,” in Proc. ACM Siggraph, pp. 341–346, 2001. EXTERNAL
• Vive, http://www.htcvive.com/us/ EXTERNAL
• M. Grundmann et al., “Calibration-Free Rolling Shutter Removal,” in Proc. IEEE Int. Conf. Computational Photography, pp. 1–8, 2012. EXTERNAL
• A. Karpenko et al., “Digital Video Stabilization and Rolling Shutter Correction Using Gyroscopes,” Stanford Tech Report, Mar. 2011. EXTERNAL
• Orah, Vahana VR, http://www.video-stitch.com/live-vr/ EXTERNAL
• 360 Designs, Mini Eye4 360 Professional Camera, http://360designs.io/product/mini-eye-4-professional-360-camera/ EXTERNAL
• Circle Vision 360, https://en.wikipedia.org/wiki/Circle-Vision_360 EXTERNAL
• GPU Technology Conference, http://www.nvidia.com/con-tent/gtc/documents/1060_gtc09.pdf EXTERNAL
• Kolor, http://www.kolor.com/autopano EXTERNAL
• P. Burt and E. Adelson, “A Multiresolution Spline With Application to Image Mosaics,” ACM Trans. Graphics, 2(4):217–36, Oct. 1983. EXTERNAL
• L. Shuaicheng et al., “Bundled Camera Paths for Video Stabilization,” ACM Trans. Graphics, 32(4), July 2013. EXTERNAL
• PCI Express, https://en.wikipedia.org/wiki/PCI_Express EXTERNAL
• Y. Boykov and M. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images,” in Proc. ICCV, p. 105, July 2001. EXTERNAL
• S. Baker et al., “Removing Rolling Shutter Wobble,” in Proc. IEEE Conf. Comput. Vision Pattern Recognition, pp. 2392–2399, 2010. EXTERNAL
• Oculus, https://www3.oculus.com/en-us/rift/ EXTERNAL
• Puffer Fish, http://www.pufferfishdisplays.co.uk/ EXTERNAL
• Oculus Platform SDK, https://developer3.oculus.com/downloads/platform/1.8.0/Oculus_Platform_SDK/ EXTERNAL
• W. Jiang and J. Gu, “Video Stitching with Spatial-Temporal Content-Preserving Warping,” in Proc. IEEE Comp. Soc. pp. 42–48, 2015. EXTERNAL
• F. Liu and J. Hailin, “Content-Preserving Warps for 3D Video Stabilization,” ACM Trans. Graphics, 28(3):1–9, July 2009. EXTERNAL
• J. Davis, “Mosaics of Scenes with Moving Objects,” in Proc. CVPR 1998, Santa Barbara, CA, June 1998. EXTERNAL
• M. L. Gleicher and F. Liu, “Re-Cinematography: Improving the Camera Dynamics of Casual Video,” ACM Trans. Multimedia, 5(1), Oct. 2008. EXTERNAL
• Y. Matsushita et al., “Full-Frame Video Stabilization with Motion Inpainting,” IEEE Trans. Pattern Analysis and Machine Intelligence, 28(7), July 2008. EXTERNAL
• Facebook 360, https://facebook360.fb.com/ EXTERNAL
• Y. Li et al., “Lazy Snapping,” Proc. ACM Siggraph, pp. 303–308, 2004. EXTERNAL
• P. Eisert and P. Bekaert, “Omni-Directional Video,” in Digital Representations of the Real World: How to Capture, Model and Render Visual Reality, CRC Press: Boca Raton, FL, 2015. EXTERNAL
• F Liu et al., “Subspace Video Stabilization,” ACM Trans. Graphics, 30(1), 2011. EXTERNAL
• GoPro, https://shop.gopro.com/virtualreality/omni—all-inclusive/MHDHX-006.html EXTERNAL
• Video Stitch, http://www.video-stitch.com/ EXTERNAL
• Imax, https://en.wikipedia.org/wiki/IMAX EXTERNAL
• 360 Designs, http://360designs.io/ EXTERNAL
• SMPTE ST 2081-10:2015 [Superseded]
• SMPTE ST 2082-10:2015 [Superseded]