In-camera, Photorealistic Style Transfer for On-set Automatic Grading

Metadata

Publisher: SMPTE — White Plains, NY, USA
Doc Type: Conference Paper
Content Type: Original Research
Volume: 2018, No. 10, pp. 1–12
Abstract: In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time.
Publication Date: 2018-10-22
DOI: 10.5594/M001835
ISBN: Print: 978-1-61482-960-7
Link: https://doi.org/10.5594/M001835
Author(s): Itziar ZabaletaUniversitat Pompeu Fabra
Marcelo BertalmioUniversitat Pompeu Fabra
Keyword(s): Style Transfer, Color Transfer, Color Grading, Video
Copyright: © 2018 Society of Motion Picture and Television Engineers, Inc.

Bibliographic Reference(s)

X. Xiao, L. Ma, “Color transfer in correlated color space”, Proceedings of the 2006 ACM international conference on Virtual reality continuum and its applications (VRCIA), 2006. EXTERNAL
L. A. Gatys, A. S. Ecker, M. Bethge, "Image style transfer using convolutional neural networks", IEEE Computer Vision and Pattern Recognition, 2016. EXTERNAL
E. Reinhard, M. Ashikhmin, B. Gooch and P. Shirley, "Color transfer between images", IEEE Computer Graphics and Applications, 2001. EXTERNAL
M. Brown, S. Susstrunk, "Multi-spectral SIFT for Scene Category Recognition", International Conference on Computer Vision and Pattern Recognition, 2011. EXTERNAL
P. Cyriac, D. Kane and M. Bertalmío, "Optimized tone curve for in-camera image processing", IS&T Electronic Imaging Conference, 2016. EXTERNAL
Y. HaCohen, E. Shechtman, D. B. Goldman, D. Lischinski, "Non-rigid dense correspondence with applications for image enhancement", ACM Transactions on Graphics (Proceedings of SIGGRAPH 2011), 2011. EXTERNAL
F. Luan, S. Paris, E. Shechtman, K. Bala, "Deep photo style transfer", IEEE Computer Vision and Pattern Recognition, 2017. EXTERNAL
M. Mudrová, A. Proch́azka, "Principal Component Analysis in image processing", Proceedings of MATLAB Technical Computing Conference, 2005. [Pitié] F. Pitié, A. Kokaram, R. Dahyot, "Automated colour grading using colour distribution transfer", Computer Vision and Image Understanding, 2007. EXTERNAL
H. Kotera, "A Scene-referred color transfer for pleasant imaging on display", IEEE International Conference on Image Processing, 2005. EXTERNAL
F. Ebner, "Derivation and modelling of HUE uniformity and development of the IPT color space", Ph.D. degree dissertation, 1998. EXTERNAL
J. Vázquez-Corral, M. Bertalmío, "Color stabilization along time and across shots of the same scene, for one or several cameras of unknown specifications", IEEE Transactions on Image Processing, 2014. EXTERNAL
O. Frigo, N. Sabater, J. Delon, P. Hellier, "Video Style Transfer by Consistent Adaptive Patch Sampling", The Visual Computer, 2018. EXTERNAL
N. Bonneel, K. Sunkavalli, S. Paris, H. Pfister, "Example-based video color grading", ACM Transactions on Graphics (Proceedings of SIGGRAPH 2013), 2013. EXTERNAL
C. Poynton, "Gamma", in Digital Video and HD: Algorithms and Interfaces, Morgan Kaufmann, 2003. EXTERNAL
I. Zabaleta, M. Bertalmio, “Photorealistic style transfer for cinema shoots”, to appear Proceedings in Colour and Visual Computing Symposium, 2018. EXTERNAL

Source Data (JSON)

Full registry record with provenance metadata. Open directly: /api/doc/10.5594-M001835.json

Reference Tree

Explore all references and references to this document, as a navigable tree.

Open Reference Tree

Reference this Doc

Plain text (ISO 690 compliant)

Preview:

Itziar Zabaleta and Marcelo Bertalmio; In-camera, Photorealistic Style Transfer for On-set Automatic Grading, SMPTE Meetings and Conferences ( October 2018); SMPTE, 2018, ISBN: 978-1-61482-960-7. Available at https://doi.org/10.5594/M001835

Snippet:

Itziar Zabaleta and Marcelo Bertalmio; In-camera, Photorealistic Style Transfer for On-set Automatic Grading, SMPTE Meetings and Conferences ( October 2018); SMPTE, 2018, ISBN: 978-1-61482-960-7. Available at https://doi.org/10.5594/M001835

HTML (ISO 690 compliant)

Preview:

Snippet:

<span class="citation">Itziar Zabaleta and Marcelo Bertalmio; <cite>In-camera, Photorealistic Style Transfer for On-set Automatic Grading</cite>, SMPTE Meetings and Conferences ( October 2018); SMPTE, 2018, ISBN: 978-1-61482-960-7. Available at <a href="https://doi.org/10.5594/M001835" target="_blank" rel="noopener">https://doi.org/10.5594/M001835</a></span>

SMPTE's HTML Pub

Preview:

Snippet:

<li>
Itziar Zabaleta and Marcelo Bertalmio; <cite id="bib-10-5594-m001835">In-camera, Photorealistic Style Transfer for On-set Automatic Grading</cite>, SMPTE Meetings and Conferences ( October 2018); SMPTE, 2018, ISBN: 978-1-61482-960-7
<span class="doi">10.5594/M001835</span>
</li>