research-article

Expression flow for 3D-aware face component transfer

Authors:

Lubomir Bourdev,

Dimitri MetaxasAuthors Info & Claims

SIGGRAPH '11: ACM SIGGRAPH 2011 papers

Article No.: 60, Pages 1 - 10

https://doi.org/10.1145/1964921.1964955

Published: 25 July 2011 Publication History

Abstract

We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending using existing compositing tools results in semantically unnatural composites, since expression is a global effect and the local component in one expression is often incompatible with the shape and other components of the face in another expression. To solve this problem we present Expression Flow, a 2D flow field which can warp the target face globally in a natural way, so that the warped face is compatible with the new facial component to be copied over. To do this, starting with the two input face photos, we jointly construct a pair of 3D face shapes with the same identity but different expressions. The expression flow is computed by projecting the difference between the two 3D shapes back to 2D. It describes how to warp the target face photo to match the expression of the reference photo. User studies suggest that our system is able to generate face composites with much higher fidelity than existing methods.

Supplementary Material

Supplemental material. (a60-yang.zip)

Download
9.68 MB

MP4 File (tp045_11.mp4)

Download
17.46 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. In Proceedings of ACM SIGGRAPH, vol. 23, 294--302.

Digital Library

[2]

Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., and Nayar, S. K. 2008. Face swapping: automatically replacing faces in photographs. In Proceedings of ACM SIGGRAPH, 39:1--39:8.

Digital Library

[3]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of ACM SIGGRAPH, 187--194.

Digital Library

[4]

Blanz, V., Basso, C., Poggio, T., and Vetter. 2003. Rean-imating faces in images and video. Computer Graphics Forum 22, 3.

[5]

Blanz, V., Scherbaum, K., Vetter, T., and Seidel, H.-P. 2004. Exchanging faces in images. Computer Graphics Forum 23, 3, 669--676.

[6]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Analysis and Machine Intelligence 23.

Digital Library

[7]

Cootes, T., Taylor, C., Cooper, D., and Graham, J. 1995. Active shape models: Their training and application. Computer Vision and Image Understanding 61, 1, 38--59.

Digital Library

[8]

Cootes, T. F., Edwards, G. J., and Taylor, C. J. 2001. Active appearance models. IEEE Trans. Pattern Analysis and Machine Intelligence 23, 681--685.

Digital Library

[9]

Decarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38, 99--127.

Digital Library

[10]

Dovgard, R., and Basri, R. 2004. Statistical symmetric shape from shading for 3d structure recovery of faces. In Proceedings of ECCV, 99--113.

[11]

Faigin, G. 1991. The Artist's Complete Guide to Facial Expression. Watson-Guptill Publications Inc., New York.

[12]

Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. In Proceedings of ACM SIGGRAPH, vol. 28, 67:1--67:9.

Digital Library

[13]

Fattal, R. 2009. Edge-avoiding wavelets and their applications. In Proceedings of ACM SIGGRAPH, vol. 28.

Digital Library

[14]

Felzenszwalb, P. F., and Huttenlocher, D. P. 2005. Pictorial structures for object recognition. International Journal of Computer Vision 61, 55--79.

Digital Library

[15]

Joshi, N., Matusik, W., Adelson, E. H., and Kriegman, D. J. 2010. Personal photo enhancement using example images. ACM Trans. Graphics 29, 12:1--12:15.

Digital Library

[16]

Leyvand, T., Cohen-Or, D., Dror, G., and Lischinski, D. 2008. Data-driven enhancement of facial attractiveness. In Proceedings of ACM SIGGRAPH, 38:1--38:9.

Digital Library

[17]

Liang, L., Xiao, R., Wen, F., and Sun, J. 2008. Face alignment via component-based discriminative search. In Proceedings of ECCV.

Digital Library

[18]

Liu, Z., Shan, Y., and Zhang, Z. 2001. Expressive expression mapping with ratio images. In Proceedings of ACM SIGGRAPH, 271--276.

Digital Library

[19]

Liu, C., Shum, H.-Y., and Freeman, W. T. 2007. Face hallucination: Theory and practice. International Journal of Computer Vision 75, 115--134.

Digital Library

[20]

Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 1981 DARPA Image Understanding Workshop, 121--130.

Digital Library

[21]

Milborrow, S., and Nicolls, F. 2008. Locating facial features with an extended active shape model. In Proceedings of ECCV, 504--513.

Digital Library

[22]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. In Proceedings of ACM SIGGRAPH, 313--318.

Digital Library

[23]

Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of ACM SIGGRAPH, 75--84.

Digital Library

[24]

Romdhani, S., and Vetter, T. 2003. Efficient, robust and accurate fitting of a 3d morphable model. In Proceedings of ICCV, 59--66.

Digital Library

[25]

Saragih, J., Lucey, S., and Cohn, J. 2009. Face alignment through subspace constrained mean-shifts. In Proceedings of the 12th ICCV, 1034--1041.

[26]

Shlizerman, I. K., and Basri, R. 2011. 3d face reconstruction from a single image using a single reference face shape. IEEE Trans. Pattern Analysis and Machine Intelligence 33, 394--405.

Digital Library

[27]

Shlizerman, I. K., Sankar, A., Shechtman, E., and Seitz, S. M. 2010. Being john malkovich. In Proceedings of ECCV, 341--353.

Digital Library

[28]

Singular Inversions Inc. 2009. Facegen modeller manual. In www.facegen.com.

[29]

Sunkavalli, K., Johnson, M. K., Matusik, W., and Pfister, H. 2010. Multi-scale image harmonization. In Proceedings of ACM SIGGRAPH, vol. 29.

Digital Library

[30]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. In Proceedings of ACM SIGGRAPH, vol. 24, 426--433.

Digital Library

[31]

Wang, Y., Huang, X., Lee, C.-S., Zhang, S., Li, Z., Samaras, D., Metaxas, D., Elgammal, A., and Huang, P. 2004. High resolution acquisition, learning and transfer of dynamic 3-d facial expressions. In Proceedings of EuroGraphics, 677--686.

[32]

Williams, L. 1990. Performance-driven facial animation. In Proceedings of ACM SIGGRAPH, vol. 24, 235--242.

Digital Library

[33]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In Proceedings of ACM SIGGRAPH, 548--558.

Digital Library

Cited By

Wu YShu CPai D(2024)RightSizing: Disentangling Generative Models of Human Body Shapes with Metric ConstraintsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3671658(1-10)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3671658
Fan YJiang XLan SYou S(2023)Facial expression transfer based on generative adversarial networks: a reviewThird International Conference on Computer Vision and Pattern Analysis (ICCPA 2023)10.1117/12.2684264(73)Online publication date: 1-Aug-2023
https://doi.org/10.1117/12.2684264
Kumar RLuo JPang ADavis J(2023)Disjoint Pose and Shape for 3D Face Reconstruction2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW60793.2023.00336(3107-3117)Online publication date: 2-Oct-2023
https://doi.org/10.1109/ICCVW60793.2023.00336
Show More Cited By

Index Terms

Expression flow for 3D-aware face component transfer
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Expression flow for 3D-aware face component transfer

We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending ...
Generating 3D views of facial expressions from frontal face video based on topographic analysis
MULTIMEDIA '04: Proceedings of the 12th annual ACM international conference on Multimedia

In this paper, we report our newly developed 3D face modeling system with arbitrary expressions in a high level of detail using the topographic analysis and mesh instantiation process. Given a sequence of images of facial expressions at frontal views, ...
A bi-modal face recognition framework integrating facial expression with facial appearance

Among many biometric characteristics, the facial biometric is considered to be the least intrusive technology that can be deployed in the real-world visual surveillance environment. However, in facial biometric, little research attention has been paid ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '11: ACM SIGGRAPH 2011 papers

August 2011

869 pages

ISBN:9781450309431

DOI:10.1145/1964921

Editor:
Hugues Hoppe

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGGRAPH '11

Sponsor:

SIGGRAPH

SIGGRAPH '11: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 7 - 11, 2011

British Columbia, Vancouver, Canada

Acceptance Rates

SIGGRAPH '11 Paper Acceptance Rate 82 of 432 submissions, 19%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,657
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu YShu CPai D(2024)RightSizing: Disentangling Generative Models of Human Body Shapes with Metric ConstraintsProceedings of the 50th Graphics Interface Conference10.1145/3670947.3671658(1-10)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3671658
Fan YJiang XLan SYou S(2023)Facial expression transfer based on generative adversarial networks: a reviewThird International Conference on Computer Vision and Pattern Analysis (ICCPA 2023)10.1117/12.2684264(73)Online publication date: 1-Aug-2023
https://doi.org/10.1117/12.2684264
Kumar RLuo JPang ADavis J(2023)Disjoint Pose and Shape for 3D Face Reconstruction2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)10.1109/ICCVW60793.2023.00336(3107-3117)Online publication date: 2-Oct-2023
https://doi.org/10.1109/ICCVW60793.2023.00336
Tan DSoeseno JHua K(2022)Controllable and Identity-Aware Facial Attribute TransformationIEEE Transactions on Cybernetics10.1109/TCYB.2021.307117252:6(4825-4836)Online publication date: Jun-2022
https://doi.org/10.1109/TCYB.2021.3071172
Tu XZou YZhao JAi WDong JYao YWang ZGuo GLi ZLiu WFeng J(2022)Image-to-Video Generation via 3D Facial DynamicsIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2021.308325732:4(1805-1819)Online publication date: Apr-2022
https://doi.org/10.1109/TCSVT.2021.3083257
Luo JKhan FMori Ide Silva ARuezga ELiu MPang ADavis J(2022)How much does input data type impact final face model accuracy?2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.01841(18963-18972)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.01841
Fu CHu YWu XWang GZhang QHe R(2021)High-Fidelity Face Manipulation With Extreme Poses and ExpressionsIEEE Transactions on Information Forensics and Security10.1109/TIFS.2021.305006516(2218-2231)Online publication date: 2021
https://doi.org/10.1109/TIFS.2021.3050065
Luo JKhan FMori ISilva ARuezga EDavis J(2021)Face Models: How Good Does My Data Need To Be?2021 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP42928.2021.9506668(3188-3192)Online publication date: 19-Sep-2021
https://doi.org/10.1109/ICIP42928.2021.9506668
Wang RAmy Ahn YMessinger DNwogu I(2021)Towards the Synthesis of Parent-Infant Facial Interactions2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)10.1109/FG52635.2021.9667019(1-8)Online publication date: 15-Dec-2021
https://doi.org/10.1109/FG52635.2021.9667019
Wang XLi WHuang D(2021)Expression-Latent-Space-guided GAN for Facial Expression Animation based on Discrete Labels2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)10.1109/FG52635.2021.9666959(1-8)Online publication date: 15-Dec-2021
https://doi.org/10.1109/FG52635.2021.9666959
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents