SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020)
[ACTIVE]
Rotoscope Automation with Deep Learning
Metadata
- Publisher
- SMPTE — White Plains, NY, USA
- Doc Type
- Journal Article
- Content Type
- Original Research
- Abbreviated Title
- SMPTE Mot. Imag. J
- Volume
- 129, No. 2, pp. 16–26
- Abstract
- We present a deep learning-based algorithm that can automatically rotoscope people in a given scene, without any user input. Current approaches to image matting require a significant amount of human input, irrespective of whether it is manually rotoscoped or through a chroma key. This study shows that this algorithm can perform as well as and even surpass the rotoscoping capabilities of the Adobe After Effects’ RotoBrush tool, in a variety of scenes comprising different lighting conditions, movements, and subjects. This makes it suitable for integration within a visual effects (VFX) pipeline.
- Publication Date
- 2020-03-01
- DOI
10.5594/JMI.2019.2959967- ISSN
- Print:
1545-0279| Electronic:2160-2492 - Link
- https://doi.org/10.5594/JMI.2019.2959967
- Author(s)
- Oscar Estrada Torrejon
bio
Oscar Estrada Torrejonis a recent graduate from the Motion Picture Science Program at the Rochester Institute of Technology (RIT), Rochester, NY. His interests include content development for digital platforms, post-production technology, and visual effects. His experience includes creating content marketing videos for his university, as well as managing broadcast equipment and post-production workflows on board oceanographic research vessels. Torrejon graduated Summa Cum Laude in 2019 and received the Outstanding Undergraduate Scholar Award and the Outstanding International Student Service Award for his service to his alma matter.Nicholas Perettibio
Nicholas Perettirecently graduated from the Rochester Institute of Technology (RIT), Rochester, NY, with a major in software engineering and a minor in imaging science. While at RIT, he worked for one year at Apple Inc., Cupertino, CA, beginning with the Photography Group and then spent six months with the Computer Vision and Machine Learning Team. He is currently the head of machine learning at Scythe Robotics based in Boulder, CO, which is creating a self-driving lawn mower for golf courses and municipalities. Peretti enjoys basketball and snowboarding, when not interacting with a computer.Ricardo Figueroabio
Ricardo Figueroais currently an associate professor, the program director for the Motion Picture Science Program, and an interim co-director of the School of Film and Animation, Rochester Institute of Technology (RIT), Rochester, NY. Figueroa joined RIT after working at Eastman Kodak, Rochester, for ten years, where he held positions as film and digital lab manager, digital/hybrid technologies regional director, digital imaging research engineer, and Kodak operating system manager. Figueroa holds a BSEE and an MSEE from the University of Puerto Rico, Mayagüez, Puerto Rico, and is currently pursuing a PhD in computing and information sciences at RIT. His research interests are in the areas of machine learning and deep learning for image processing applications. Outside of work, Figueroa is an avid triathlete, soccer player, and golfer. He is also active in various professional and civic organizations. - Keyword(s)
- Compositing, Deep Learning, Image Matting, Instance Segmentation
- Copyright
- © 2019 SMPTE
Bibliographic Reference(s)
- M. Seymour, “The Art of Roto: 2011,” fxguide, 2019. Accessed: 2 June 2019. [Online]. Available: https://www.fxguide.com/featured/the-art-of-roto-2011/ EXTERNAL
- A. Karpathy, “CS231n Convolutional Neural Networks for Visual Recognition,” Cs231n.github.io, 2019. Accessed: 2 June 2019. [Online]. Available: http://cs231n.github.io EXTERNAL
- J. Hui, “Understanding Feature Pyramid Networks for Object Detection (FPN),” Medium, 2019. Accessed: 2 June 2019. [Online]. Available: https://medium.com/@jonathan_hui/understanding-feature-pyramid-networks-for-object-detection-fpn-45b227b9106c EXTERNAL
- X. Bai, J. Wang, D. Simons, and G. Sapiro, “Video SnapCut,” ACM SIGGRAPH 2009 Papers, 2009. Accessed: 2 June 2019. doi: 10.1145/1576246.1531376 EXTERNAL
- C. Singh, “Rotobrush,” Cmsc426.github.io, 2019. Accessed: 2 June 2019. [Online]. Available: https://cmsc426.github.io/rotobrush/ EXTERNAL
- T. Lin et al., “Microsoft COCO: Common Objects in Context,” Computer Vision—ECCV 2014, pp. 740–755, 2014. Accessed: 2 June 2019. doi: 10.1007/978-3-319-10602-1_48 EXTERNAL
- A. Rosebrock, “Intersection over Union (IoU) for Object Detection—PyImageSearch,” PyImageSearch, 2019. Accessed: 2 June 2019. [Online]. Available: https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/ EXTERNAL
- Q. Zhu, P. Heng, L. Shao, and X. Li, “What’s the Role of Image Matting in Image Segmentation?,” Proc. 2013 IEEE Int. Conf. Robot. Biomimetics (ROBIO), 2013. Accessed: 2 June 2019. doi: 10.1109/robio.2013.6739711 EXTERNAL
- Y. Aksoy, T. Aydin, M. Pollefeys, and A. Smolic´, “Interactive High-Quality Green-Screen Keying via Color Unmixing,” ACM Trans. Graphics, 36(4):1, 2016. Accessed: 2 June 2019. doi: 10.1145/3072959.2907940 EXTERNAL
- N. Xu, B. Price, S. Cohen, and T. Huang, “Deep Image Matting,” Proc. 2017 IEEE Conf. Comput.Vision Pattern Recognition (CVPR), 2017. Accessed: 2 June 2019. doi: 10.1109/cvpr.2017.41 EXTERNAL
- J. Olafenwa, “Review of Deep Residual Learning for Image Recognition,” Medium, 2019. Accessed: 2 June 2019. [Online]. Available: https://medium.com/deepreview/review-of-deep-resid-ual-learning-for-image-recognition-a92955acf3aa EXTERNAL
- M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes (VOC) Challenge,” Int. J. Comput. Vis., 88(2):303–338, 2009. Accessed: 2 June 2019. doi: 10.1007/s11263-009-0275-4 EXTERNAL
- F. Chollet, “Xception: Deep Learning with Depthwise Separable Convolutions,” Proc. 2017 IEEE Conf. Comput. Vision Pattern Recognition (CVPR), 2017. Accessed: 2 June 2019. doi: 10.1109/cvpr.2017.195 EXTERNAL
- J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” Proc. 2009 IEEE Conf. Comput. Vision Pattern Recognition, 2009. Accessed: 2 June 2019. doi: 10.1109/cvpr.2009.5206848 EXTERNAL
- C. Szegedy et al., “Going Deeper with Convolutions,” Proc. 2015 IEEE Conf. Comput. Vision Pattern Recognition (CVPR), 2015. Accessed: 2 June 2019. doi: 10.1109/cvpr.2015.7298594 EXTERNAL
- D. Lowe, “Object Recognition from Local Scale-Invariant Features,” Proc. 7th IEEE Int. Conf. Comput. Vision, 1999. Accessed: 2 June 2019. doi: 10.1109/iccv.1999.790410 EXTERNAL
- S. Tsang, “Review: Xception—With Depthwise Separable Convolution, Better than Inception-v3 (Image Classification),” Towards Data Science, 2019. Accessed: 2 June 2019. [Online]. Available: https://towardsdatascience.com/review-xception-with-depthwise-separable-convolution-better-than-inception-v3-image-dc967dd42568 EXTERNAL
- A. Smith and J. Blinn, “Blue Screen Matting,” Proc. 23rd Annu. Conf. Comput. Graphics Interactive Techn.—SIGGRAPH’96, 1996. Accessed: 2 June 2019. doi: 10.1145/237170.237263 EXTERNAL
Source Data (JSON)
Full registry record with provenance metadata. Open directly: /api/doc/10.5594-JMI.2019.2959967.json
Reference Tree
Explore all references and references to this document, as a navigable tree.
Open Reference TreeReference this Doc
Plain text (ISO 690 compliant)
Preview:
Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; Rotoscope Automation with Deep Learning, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2019.2959967
Snippet:
Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; Rotoscope Automation with Deep Learning, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2019.2959967
HTML (ISO 690 compliant)
Preview:
Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; Rotoscope Automation with Deep Learning, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020. Available at https://doi.org/10.5594/JMI.2019.2959967
Snippet:
<span class="citation">Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; <cite>Rotoscope Automation with Deep Learning</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020. Available at <a href="https://doi.org/10.5594/JMI.2019.2959967" target="_blank" rel="noopener">https://doi.org/10.5594/JMI.2019.2959967</a></span>
SMPTE's HTML Pub
Preview:
Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; Rotoscope Automation with Deep Learning, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020
doi: 10.5594/JMI.2019.2959967
url: https://doi.org/10.5594/JMI.2019.2959967
doi: 10.5594/JMI.2019.2959967
url: https://doi.org/10.5594/JMI.2019.2959967
Snippet:
<li> Oscar Estrada Torrejon, Nicholas Peretti, and Ricardo Figueroa; <cite id="bib-10-5594-jmi-2019-2959967">Rotoscope Automation with Deep Learning</cite>, SMPTE Motion Imaging Journal ( Volume: 129, Issue: 2, March 2020); SMPTE, 2020 <span class="doi">10.5594/JMI.2019.2959967</span> </li>