Integrating Machine-Learning-Based Operators in Visual Effects Toolsets

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Journal Article
Content Type: Original Research
Abbreviated Title: SMPTE Mot. Imag. J
Volume: 130, No. 5, pp. 26–34
Abstract: Post-production workflows rely heavily on image processing and computer vision algorithms for the implementation of their visual effects (VFX) tools. In these fields, machine learning has been shown to be disruptive. By integrating the domain statistics from a given training data set, machine-learning-based operators may be more efficient at solving existing problems and may enable new problems to be solved, expanding the VFX toolset. In this study, we share our experience developing and integrating several machine-learning-based operators into software for the post-production industry. We will present a sky segmentation operator, a depth map estimation operator, and an operator to compute face geometric maps (UVs,* depth, and normals) from sequences of images. More specifically, these operators consist of trained deep convolutional neural networks (DCNNs) taking as input an RGB color image and outputting the associated maps, that is, a matte, depth, normals, and/or UV maps. Such maps permit the application of many different effects to the input image during color grading, including beautification or even relighting of faces. These studies will serve as a case study to review and discuss the multiple challenges posed by the implementation, integration, and deployment of machine-learning-based operators into a VFX toolset.
Publication Date: 2021-06-01
DOI: 10.5594/JMI.2021.3072684
ISSN: Print: 1545-0279 | Electronic: 2160-2492
Link: https://doi.org/10.5594/JMI.2021.3072684
Author(s): Nicolas Moënne-Loccoz
bio
Nicolas Moënne-Loccoz joined the Creative Finishing team at Autodesk, Montréal, Canada, in 2011, where he is working on the machine learning and computer vision (optical flow estimation, planar and camera tracking, sky and human segmentation, and face geometry maps predictions) and computer graphics (rendering workflows and post-processing effects) challenges for the flame 3D compositing tools. Prior to this, he has spent five years working on machine learning and computer vision applied to face tracking, reconstruction, and recognition at SAFRAN Morpho, Paris, France, and four years working on his PhD in content-based indexing and retrieval of video at the University of Geneva, Geneva, Switzerland.
Keyword(s): Face Relighting, Machine Learning, Monocular Depth Estimation, Sky Segmentation, Visual Effects
Copyright: © 2021 SMPTE

Bibliographic Reference(s)

L. Zhengqi and S. Noah, “MegaDepth: Learning Single-View Depth Prediction from Internet Photos,” IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2018. EXTERNAL
Tzeng, E., et al., “Adversarial Discriminative Domain Adaptation,” IEEE Conf. Comput. Vis. Pattern Recognit., 2017. EXTERNAL
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016. EXTERNAL
E. Zakharov et al., “Few-Shot Adversarial Learning of Realistic Neural Talking Head Models,” IEEE Int. Conf. Comput. Vis. (ICCV), pp. 9458–9467, Oct. 2019. EXTERNAL
L. Lettry, K. Vanhoey, and L. Van Gool, “Unsupervised Deep Single-Image Intrinsic Decomposition Using Illumination-Varying Image Sequences,” IEEE Winter Conf. Appl. Comput. Vis., pp. 409–419, Sep. 2018. EXTERNAL
O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” Proc. Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, 2015. EXTERNAL
M. Tan and Q. Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Int. Conf. Mach. Learn., May 2019. EXTERNAL
H. Jiang et al., “Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation,” IEEE Conf. Computer Vision Pattern Recognit., pp. 9000–9008, Jul. 2018. EXTERNAL
T.-Y. Lin et al., “Microsoft COCO: Common Objects in Context,” Eur. Conf. Comput. Vis., pp. 740–755, 2014. EXTERNAL
J. S. Yoon et al., “Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking,” IEEE Conf. Comput. Vis. Pattern Recognit., 2019. EXTERNAL
J. Zhang et al., “All-Weather Deep Outdoor Lighting Estimation,” IEEE Conf. Comput. Vis. Pattern Recognit., Aug. 2019. EXTERNAL
P. Paysan et al., “A 3D Face Model for Pose and Illumination Invariant Face Recognition,” IEEE Int. Conf. Adv. Video Signal Based Surveillance, pp. 296–301, 2009. EXTERNAL
T. Park et al., “Semantic Image Synthesis With Spatially-Adaptive Normalization,” IEEE Conf. Comput. Vis. Pattern Recognit., Nov. pp. 2332–2341, 2019. EXTERNAL
A. Ranjan et al., “Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation,” IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2019. EXTERNAL
P. Isola et al., “Image-to-Image Translation With Conditional Adversarial Nets,” IEEE Conf. Computer Vision and Pattern Recognition, pp. 5967–5976, 2017. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2021.3072684.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Nicolas Moënne-Loccoz; Integrating Machine-Learning-Based Operators in Visual Effects Toolsets, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 5, June 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3072684

Snippet:

Nicolas Moënne-Loccoz; Integrating Machine-Learning-Based Operators in Visual Effects Toolsets, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 5, June 2021); SMPTE, 2021. Available at https://doi.org/10.5594/JMI.2021.3072684

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Nicolas Moënne-Loccoz; <cite>Integrating Machine-Learning-Based Operators in Visual Effects Toolsets</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 5, June 2021); SMPTE, 2021. Available at <a href="https://doi.org/10.5594/JMI.2021.3072684" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2021.3072684</a></span>

SMPTE's HTML Pub

Preview:

Nicolas Moënne-Loccoz; Integrating Machine-Learning-Based Operators in Visual Effects Toolsets, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 5, June 2021); SMPTE, 2021
doi: 10.5594/JMI.2021.3072684
url: https://doi.org/10.5594/JMI.2021.3072684

Snippet:

<li>
Nicolas Moënne-Loccoz; <cite id="bib-10-5594-jmi-2021-3072684">Integrating Machine-Learning-Based Operators in Visual Effects Toolsets</cite>, SMPTE Motion Imaging Journal ( Volume: 130, Issue: 5, June 2021); SMPTE, 2021
<span class="doi">10.5594/JMI.2021.3072684</span>
</li>