research-article

Public Access

Portrait Lighting Transfer Using a Mass Transport Approach

Authors:

Kalyan Sunkavalli,

Dimitris SamarasAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 37, Issue 1

Article No.: 2, Pages 1 - 15

https://doi.org/10.1145/3095816

Published: 10 October 2017 Publication History

Abstract

Lighting is a critical element of portrait photography. However, good lighting design typically requires complex equipment and significant time and expertise. Our work simplifies this task using a relighting technique that transfers the desired illumination of one portrait onto another. The novelty in our approach to this challenging problem is our formulation of relighting as a mass transport problem. We start from standard color histogram matching that only captures the overall tone of the illumination, and we show how to use the mass-transport formulation to make it dependent on facial geometry. We fit a three-dimensional (3D) morphable face model to the portrait, and for each pixel, we combine the color value with the corresponding 3D position and normal. We then solve a mass-transport problem in this augmented space to generate a color remapping that achieves localized, geometry-aware relighting. Our technique is robust to variations in facial appearance and small errors in face reconstruction. As we demonstrate, this allows our technique to handle a variety of portraits and illumination conditions, including scenarios that are challenging for previous methods.

Supplementary Material

MP4 File (tog-41.mp4)

Download
457.86 MB

References

[1]

Yael Adini, Yael Moses, and Shimon Ullman. 1997. Face recognition: The problem of compensating for changes in illumination direction. IEEE Trans. PAMI 19, 7 (1997), 721.

Digital Library

[2]

Oleg Alexander, Mike Rogers, William Lambeth, Matt Chiang, and Paul Debevec. 2009. The digital emily project: Photoreal facial modeling and animation. In Proceedings of the ACM SIGGRAPH 2009 Courses. 12.

Digital Library

[3]

Soonmin Bae, Sylvain Paris, and Frédo Durand. 2006. Two-scale tone management for photographic look. ACM Trans. Graph. 25, 3 (2006), 637.

Digital Library

[4]

Dmitri Bitouk, Neeraj Kumar, Samreen Dhillon, Peter Belhumeur, and Shree K. Nayar. 2008. Face swapping: Automatically replacing faces in photographs. ACM Trans. Graph. 27, 3 (2008), 39.

Digital Library

[5]

Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of SIGGRAPH. 187.

Digital Library

[6]

Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. 2015a. Sliced and radon wasserstein barycenters of measures. J. Math. Imag. Vision 51, 1 (2015), 22.

Digital Library

[7]

Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. 2015b. Blind video temporal consistency. ACM Trans. Graph. 34, 6 (2015).

Digital Library

[8]

Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. 2011. Displacement interpolation using lagrangian mass transport. ACM Trans. Graph. 30, 6 (2011), 158.

Digital Library

[9]

Xiaowu Chen, Mengmeng Chen, Xin Jin, and Qinping Zhao. 2011. Face illumination transfer through edge-preserving filters. In Proceedings of CVPR. IEEE, 281--287.

Digital Library

[10]

Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Proceedings of NIPS. 2292--2300.

[11]

Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of SIGGRAPH. 145.

Digital Library

[12]

Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol. 2014. Regularized discrete optimal transport. SIAM J. Imag. Sci. 7, 3 (2014), 1853.

[13]

Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. PAMI 23, 6 (2001), 643.

Digital Library

[14]

John Haddon and David Forsyth. 1998. Shading primitives: Finding folds and shallow grooves. In Proceedings of ICCV. IEEE, 236--241.

[15]

Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided image filtering. IEEE Trans. PAMI 35, 6 (2013), 1397.

Digital Library

[16]

David W. Jacobs, Peter N. Belhumeur, and Ronen Basri. 1998. Comparing images under variable illumination. In Proceedings of CVPR. IEEE, 610--617.

[17]

Pierre-Yves Laffont, Zhile Ren, Xiaofeng Tao, Chao Qian, and James Hays. 2014. Transient attributes for high-level understanding and editing of outdoor scenes. ACM Trans. Graph. 33, 4 (2014), 1.

Digital Library

[18]

Zicheng Liu, Ying Shan, and Zhengyou Zhang. 2001. Expressive expression mapping with ratio images. In Proceedings of SIGGRAPH. 271--276.

Digital Library

[19]

Pieter Peers, Naoki Tamura, Wojciech Matusik, and Paul Debevec. 2007. Post-production facial performance relighting using reflectance transfer. ACM Trans. Graph. 26, 3 (2007), 52.

Digital Library

[20]

Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. ACM Trans. Graph. 22, 3 (2003), 313.

Digital Library

[21]

Francois Pitie, C. Kokaram Anil, and Rozenn Dahyot. 2005. N-dimensional probability density function transfer and its application to color transfer. In Proceedings of ICCV. IEEE, 1434--1439.

Digital Library

[22]

Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. 2012. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision. Springer, 435.

Digital Library

[23]

E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. 2001. Color transfer between images. Comput. Graph. Appl. IEEE 21, 5 (2001), 34--41.

Digital Library

[24]

Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. 2000. The earth mover’s distance as a metric for image retrieval. Int. J. Comput. Vis. 40, 2 (2000), 99.

Digital Library

[25]

Amnon Shashua and Tammy Riklin-Raviv. 2001. The quotient image: Class-based re-rendering and recognition with varying illuminations. IEEE Trans. PAMI 23, 2 (2001), 129.

Digital Library

[26]

Xiaoyong Shen, Aaron Hertzmann, Jiaya Jia, Sylvain Paris, Brian Price, Eli Shechtman, and Ian Sachs. 2016. Automatic portrait segmentation for image stylization. Comput. Graph. Forum 35, 2 (2016), 93.

[27]

YiChang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014. Style transfer for headshot portraits. ACM Trans. Graph. 33, 4 (2014), 148.

Digital Library

[28]

Yichang Shih, Sylvain Paris, Frédo Durand, and William T. Freeman. 2013. Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32, 6 (2013), 200.

Digital Library

[29]

Justin Solomon, Fernando de Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. 2015. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. ACM Trans. Graph. 34, 4 (2015), 66.

Digital Library

[30]

Kalyan Sunkavalli, Micah K. Johnson, Wojciech Matusik, and Hanspeter Pfister. 2010. Multi-scale image harmonization. ACM Trans. Graph. 29, 4 (2010), 125.

Digital Library

[31]

Cédric Villani. 2003. Topics in Optimal Transportation (Graduate Studies in Mathematics, Vol. 58). American Mathematical Society.

[32]

Cédric Villani. 2008. Optimal Transport: Old and New (1st ed.).

[33]

Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, and Dimitris Samaras. 2009. Face relighting from a single image under arbitrary unknown lighting conditions. IEEE Trans. PAMI 31, 11 (2009), 1968.

Digital Library

[34]

Zhen Wen, Zicheng Liu, and Thomas S. Huang. 2003. Face relighting with radiance environment maps. In Proceedings of CVPR, Vol. 2. IEEE, II--158.

[35]

Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25, 3 (2006), 1013.

Digital Library

[36]

Fei Yang, Jue Wang, Eli Shechtman, Lubomir Bourdev, and Dimitri Metaxas. 2011. Expression flow for 3D-aware face component transfer. ACM Trans. Graph. 30, 4 (2011), 60.

Digital Library

Cited By

Choi JChristman MSengupta R(2024)Personalized Video Relighting With an At-Home Light StageComputer Vision – ECCV 202410.1007/978-3-031-73661-2_22(394-410)Online publication date: 10-Nov-2024
https://doi.org/10.1007/978-3-031-73661-2_22
Bonneel NDigne J(2023)A survey of Optimal Transport for Computer Graphics and Computer VisionComputer Graphics Forum10.1111/cgf.1477842:2(439-460)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14778
Avi-Aharon MArbelle ARaviv T(2023)Differentiable Histogram Loss Functions for Intensity-based Image-to-Image TranslationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3278287(1-12)Online publication date: 2023
https://doi.org/10.1109/TPAMI.2023.3278287
Show More Cited By

Index Terms

Portrait Lighting Transfer Using a Mass Transport Approach
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Portrait Lighting Transfer Using a Mass Transport Approach

Lighting is a critical element of portrait photography. However, good lighting design typically requires complex equipment and significant time and expertise. Our work simplifies this task using a relighting technique that transfers the desired ...
Face relighting using discriminative 2D spherical spaces for face recognition

As part of the face recognition task in a robust security system, we propose a novel approach for the illumination recovery of faces with cast shadows and specularities. Given a single 2D face image, we relight the face object by extracting the nine ...
Color face image decomposition under complex lighting conditions

In this paper, we proposed a method to recover the reflectance and shading images of face images, such as portraits, captured under complex lighting conditions. Under such lighting conditions, traditional intrinsic image decomposition can hardly be ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 37, Issue 1

February 2018

167 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3151031

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2017

Received: 01 October 2017

Accepted: 01 April 2017

Revised: 01 March 2017

Published in TOG Volume 37, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
1,054
Total Downloads

Downloads (Last 12 months)111
Downloads (Last 6 weeks)14

Reflects downloads up to 17 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Choi JChristman MSengupta R(2024)Personalized Video Relighting With an At-Home Light StageComputer Vision – ECCV 202410.1007/978-3-031-73661-2_22(394-410)Online publication date: 10-Nov-2024
https://doi.org/10.1007/978-3-031-73661-2_22
Bonneel NDigne J(2023)A survey of Optimal Transport for Computer Graphics and Computer VisionComputer Graphics Forum10.1111/cgf.1477842:2(439-460)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14778
Avi-Aharon MArbelle ARaviv T(2023)Differentiable Histogram Loss Functions for Intensity-based Image-to-Image TranslationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.3278287(1-12)Online publication date: 2023
https://doi.org/10.1109/TPAMI.2023.3278287
Sheng YZhang JPhilip JHold-Geoffroy YSun XZhang HLing LBenes B(2023)PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01597(16643-16653)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.01597
Mattavelli GGorrino ICesana EDe Angelis JRicciardelli P(2022)Illumination and gaze effects on face evaluation: The Bi-AGI databaseFrontiers in Psychology10.3389/fpsyg.2022.94814213Online publication date: 13-Oct-2022
https://doi.org/10.3389/fpsyg.2022.948142
Li GMeka AMueller FBuehler MHilliges OBeeler T(2022)EyeNeRFACM Transactions on Graphics10.1145/3528223.353013041:4(1-16)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530130
Ohkawara MFujishiro I(2022)Illumination-aware group portrait compositorThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02508-z38:12(4009-4018)Online publication date: 20-May-2022
https://dl.acm.org/doi/10.1007/s00371-022-02508-z
Petikam LAnjyo KRhee T(2021)Shading Rig: Dynamic Art-directable Stylised Shading for 3D CharactersACM Transactions on Graphics10.1145/346169640:5(1-14)Online publication date: 24-Sep-2021
https://dl.acm.org/doi/10.1145/3461696
Zhang YTsang ILuo YHu CLu XYu X(2021)Recursive Copy and Paste GAN: Face Hallucination from Shaded ThumbnailsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.3061312(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2021.3061312
Xu WXie XLai J(2021)RelightGAN: Instance-level Generative Adversarial Network for Face Illumination TransferIEEE Transactions on Image Processing10.1109/TIP.2021.306193330(3450-3460)Online publication date: 2021
https://doi.org/10.1109/TIP.2021.3061933
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents