Integrating Machine Learning Based Operators in Visual Effects Toolsets

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2019, No. 10, pp. 1–17
Abstract: Post-production workflows rely heavily on image processing and computer vision algorithms for the implementation of their visual effect (VFX) tools. In these fields, machine learning has been shown to be disruptive. By integrating the domain statistics from a given training dataset, machine learning based operators may be more efficient at solving existing problems and may enable new problems to be solved, expanding the VFX toolset. In this paper we are sharing our experience on developing and integrating several machine-learning based operators into software for the Post-production industry. We will present a sky segmentation operator, a depth map estimation operator and an operator to compute face geometric maps (UVs, depth and normals) from sequences of images. More specifically, these operators consist in trained deep convolutional neural networks (DCNN) taking as input an RGB color image and outputting the associated maps, i.e. a matte, depth, normals, and/or UVs maps. Such maps permit to apply many different effects to the input image during color grading, including beautification or even relighting of faces. These works will serve as a case study to review and discuss the multiple challenges posed by the implementation, integration and deployment of machine learning based operators into a VFX toolsets.
Publication Date: 2019-10-21
DOI: 10.5594/M001866
Link: https://doi.org/10.5594/M001866
Author(s): Nicolas Moenne-LoccozAutodesk, Montréal
Keyword(s): Visual Effects, Machine Learning, Sky Segmentation, Monocular Depth Estimation, Face Relighting
Copyright: © 2019 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

Paysan, P., Knothe, R., Amberg, B., Romdhani, S., & Vetter, T. (2009). A 3D Face Model for Pose and Illumination Invariant Face Recognition. IEEE International Conference on Advanced Video and Signal Based Surveillance. EXTERNAL
Yoon, J. S., Shiratori, T., Yu, S.-I., & Park, H. S. (2019). Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Lettry, L., Vanhoey, K., & Van Gool, L. (2018). Unsupervised Deep Single-Image Intrinsic Decomposition using Illumination-Varying Image Sequences. IEEE Winter Conference on Applications of Computer Vision. EXTERNAL
Lin, T.-Y., Maire, M., Belongie, S. J., Bourdev, L. D., Girshick, R. B., Hays, J., . . . Ramanan, D. (2014). Microsoft COCO: Common Objects in Context. EXTERNAL
Zhengqi, L., & Noah, S. (2018). MegaDepth: Learning Single-View Depth Prediction From Internet Photos. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned-Miller, E., & Kautz, J. (2018). Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019). Semantic Image Synthesis With Spatially-Adaptive Normalization. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019). Few-Shot Adversarial Learning of Realistic Neural Talking Head Models. EXTERNAL
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention. EXTERNAL
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-Image Translation with Conditional Adversarial Nets. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Ian, G., Yoshua, B., & Aaron, C. (2016). Deep Learning. EXTERNAL
Tan, M., & Le, Q. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. International Conference on Machine Learning. EXTERNAL
Zhang, J., Sunkavalli, K., Hold-Geoffroy, Y., Hadap, S., Eisenman, J., & Lalonde, J.-F. (2019). All-Weather Deep Outdoor Lighting Estimation. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Ranjan, A., Jampani, V., Balles, L., Kim, K., Sun, D., Wulff, J., & Black, M. J. (2019). Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial Discriminative Domain Adaptation. IEEE Conference on Computer Vision and Pattern Recognition. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001866.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Nicolas Moenne-Loccoz; Integrating Machine Learning Based Operators in Visual Effects Toolsets, SMPTE Meetings and Conferences ( October 2019); SMPTE, 2019. Available at https://doi.org/10.5594/M001866

Snippet:

Nicolas Moenne-Loccoz; Integrating Machine Learning Based Operators in Visual Effects Toolsets, SMPTE Meetings and Conferences ( October 2019); SMPTE, 2019. Available at https://doi.org/10.5594/M001866

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Nicolas Moenne-Loccoz; <cite>Integrating Machine Learning Based Operators in Visual Effects Toolsets</cite>, SMPTE Meetings and Conferences ( October 2019); SMPTE, 2019. Available at <a href="https://doi.org/10.5594/M001866" target="_blank" rel="noopener">https://doi.org/10.5594/M001866</a></span>

SMPTE's HTML Pub

Preview:

Nicolas Moenne-Loccoz; Integrating Machine Learning Based Operators in Visual Effects Toolsets, SMPTE Meetings and Conferences ( October 2019); SMPTE, 2019
doi: 10.5594/M001866
url: https://doi.org/10.5594/M001866

Snippet:

<li>
Nicolas Moenne-Loccoz; <cite id="bib-10-5594-m001866">Integrating Machine Learning Based Operators in Visual Effects Toolsets</cite>, SMPTE Meetings and Conferences ( October 2019); SMPTE, 2019
<span class="doi">10.5594/M001866</span>
</li>