research-article

Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition

Authors:

Hsiang-Tao WuAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 30, Issue 4

Article No.: 74, Pages 1 - 10

https://doi.org/10.1145/2010324.1964969

Published: 25 July 2011 Publication History

Abstract

This paper introduces a new approach for acquiring high-fidelity 3D facial performances with realistic dynamic wrinkles and fine-scale facial details. Our approach leverages state-of-the-art motion capture technology and advanced 3D scanning technology for facial performance acquisition. We start the process by recording 3D facial performances of an actor using a marker-based motion capture system and perform facial analysis on the captured data, thereby determining a minimal set of face scans required for accurate facial reconstruction. We introduce a two-step registration process to efficiently build dense consistent surface correspondences across all the face scans. We reconstruct high-fidelity 3D facial performances by combining motion capture data with the minimal set of face scans in the blendshape interpolation framework. We have evaluated the performance of our system on both real and synthetic data. Our results show that the system can capture facial performances that match both the spatial resolution of static face scans and the acquisition speed of motion capture systems.

Supplemental Material

MP4 File

Download
25.96 MB

ZIP File - Supplemental material.

Download
87.22 MB

References

[1]

Alexa, M. 2003. Differential coordinates for local mesh morphing and deformation. The Visual Computer. 19(2):105--114.

[2]

Alexander, O., Rogers, M., Lambeth, W., Chiang, M., and Debevec, P. 2009. The digital emily project: photo-real facial modeling and animation. In ACM SIGGRAPH 2009 Courses, 12:1--12:15.

Digital Library

[3]

Beeler, T., Bickel, B., Beardsley, P., Sumner, B., and Gross, M. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4, 40:1--40:9.

Digital Library

[4]

Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Trans. Graph. 26, 3, 33:1--33:10.

Digital Library

[5]

Blanz, V., Basso, C., Poggio, T., and Vetter, T. 2003. Reanimating faces in images and video. In Computer Graphics Forum. 22(3):641--650.

[6]

Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2003. Universal capture: image-based facial animation for "the matrix reloaded". In ACM SIGGRAPH 2003 Sketches & Applications, 1--1.

Digital Library

[7]

Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1--41:10.

Digital Library

[8]

Cazals, F., and Pouget, M. 2003. Estimating differential quantities using polynomial fitting of osculating jets. In Proceedings of the 2003 Eurographics/ACM SIGGRAPH symposium on Geometry processing, 177--187.

Digital Library

[9]

Chai, J., and Hodgins, J. 2007. Constraint-based motion optimization using a statistical dynamic model. ACM Transactions on Graphics 26, 3, 8:1--8:9.

Digital Library

[10]

Chai, J., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3D facial animation. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 193--206.

Digital Library

[11]

DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. International Journal of Computer Vision. 38(2):99--127.

Digital Library

[12]

Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads using input from video. In Proceedings of Computer Animation Conference. 68--79.

Digital Library

[13]

Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making Faces. In Proceedings of ACM SIGGRAPH 1998, 55--66.

Digital Library

[14]

James, D. L., and Twigg, C. D. 2005. Skinning mesh animations. ACM Transactions on Graphics 24, 3, 399--407.

Digital Library

[15]

Lau, M., Chai, J., Xu, Y.-Q., and Shum, H. 2009. Face poser: Interactive modeling of 3d facial expressions using facial priors. ACM Transactions on Graphics 29, 1, 3:1--3:17.

Digital Library

[16]

Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5, 175:1--175:10.

Digital Library

[17]

Li, H., Weise, T., and Pauly, M. 2010. Example-based facial rigging. ACM Transactions on Graphics 29, 4, 32:1--32:6.

Digital Library

[18]

Ma, W.-C., Jones, A., Chiang, J.-Y., Hawkins, T., Frederiksen, S., Peers, P., Vukovic, M., Ouhyoung, M., and Debevec, P. 2008. Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27, 5, 121:1--121:10.

Digital Library

[19]

Papenberg, N., Bruhn, A., Brox, T., Didas, S., and Weickert, J. 2006. Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vision 67, 2, 141--158.

Digital Library

[20]

Pighin, F., and Lewis, J. P. 2006. Facial motion retargeting. In ACM SIGGRAPH 2006 Courses, SIGGRAPH '06.

Digital Library

[21]

Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3D model-based tracking. In International Conference on Computer Vision. 143--150.

[22]

Sorkine, O., Cohen-Or, D., Lipman, Y., Alexa, M., Rössl, C., and Seidel, H.-P. 2004. Laplacian surface editing. In SGP '04: Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing, 175--184.

Digital Library

[23]

Sumner, R. W., Schmid, J., and Pauly, M. Embedded deformation for shape manipulation. ACM Trans. Graph. 26, 3, 80:1--80:7.

Digital Library

[24]

Vicon Systems, 2011. http://www.vicon.com.

[25]

Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3, 426--433.

Digital Library

[26]

Williams, L. 1990. Performance driven facial animation. In Proceedings of ACM SIGGRAPH 1990. 24(4):235--242.

Digital Library

[27]

XYZ RGB Systems, 2011. http://www.xyzrgb.com/.

[28]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3, 548--558.

Digital Library

Cited By

Shu ZSun XPang CXin S(2024)3D Shape Segmentation via Attentive Nonuniform DownsamplingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343209534:12_Part_1(12184-12196)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3432095
Zhang SXu GWu HGu RQi LPang Y(2024)Medial hex-meshing: high-quality all-hexahedral mesh generation based on medial meshEngineering with Computers10.1007/s00366-023-01925-540:4(2537-2557)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1007/s00366-023-01925-5
Liu HBommes D(2023)Locally Meshable Frame FieldsACM Transactions on Graphics10.1145/359245742:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592457
Show More Cited By

Index Terms

Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Animation
    2. Image manipulation

Recommendations

Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition
SIGGRAPH '11: ACM SIGGRAPH 2011 papers

This paper introduces a new approach for acquiring high-fidelity 3D facial performances with realistic dynamic wrinkles and fine-scale facial details. Our approach leverages state-of-the-art motion capture technology and advanced 3D scanning technology ...
Multi-scale capture of facial geometry and motion

We present a novel multi-scale representation and acquisition method for the animation of high-resolution facial geometry and wrinkles. We first acquire a static scan of the face including reflectance data at the highest possible quality. We then ...
Controllable high-fidelity facial performance transfer

Recent technological advances in facial capture have made it possible to acquire high-fidelity 3D facial performance data with stunningly high spatial-temporal resolution. Current methods for facial expression transfer, however, are often limited to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 30, Issue 4

July 2011

829 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2010324

Issue’s Table of Contents

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2011

Published in TOG Volume 30, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

92
Total Citations
View Citations
2,529
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shu ZSun XPang CXin S(2024)3D Shape Segmentation via Attentive Nonuniform DownsamplingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343209534:12_Part_1(12184-12196)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3432095
Zhang SXu GWu HGu RQi LPang Y(2024)Medial hex-meshing: high-quality all-hexahedral mesh generation based on medial meshEngineering with Computers10.1007/s00366-023-01925-540:4(2537-2557)Online publication date: 10-Jan-2024
https://dl.acm.org/doi/10.1007/s00366-023-01925-5
Liu HBommes D(2023)Locally Meshable Frame FieldsACM Transactions on Graphics10.1145/359245742:4(1-20)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592457
Reberol MVerhetsel KHenrotte FBommes DRemacle J(2023)Robust Topological Construction of All-hexahedral Boundary Layer MeshesACM Transactions on Mathematical Software10.1145/357719649:1(1-32)Online publication date: 21-Mar-2023
https://dl.acm.org/doi/10.1145/3577196
Prabhune BSuresh K(2023)On why mesh untangling may not be requiredEngineering with Computers10.1007/s00366-023-01913-940:3(1357-1374)Online publication date: 28-Oct-2023
https://dl.acm.org/doi/10.1007/s00366-023-01913-9
Putrayudanto PHwang YLai JSong PTsai YHsu C(2023)An enhanced approach for inner and outer faces recognition of complex thin-shell partsEngineering with Computers10.1007/s00366-023-01802-140:1(493-525)Online publication date: 18-Mar-2023
https://dl.acm.org/doi/10.1007/s00366-023-01802-1
Pietroni NCampen MSheffer ACherchi GBommes DGao XScateni RLedoux FRemacle JLivesu M(2022)Hex-Mesh Generation and Processing: A SurveyACM Transactions on Graphics10.1145/355492042:2(1-44)Online publication date: 18-Oct-2022
https://dl.acm.org/doi/10.1145/3554920
Pietroni NCampen MSheffer ACherchi GBommes DGao XScateni RLedoux FRemacle JLivesu MJung S(2022)A Course on Hex-Mesh Generation and ProcessingSIGGRAPH Asia 2022 Courses10.1145/3550495.3558207(1-78)Online publication date: 6-Dec-2022
https://dl.acm.org/doi/10.1145/3550495.3558207
Zhang TFang GHuang YDutta NLefebvre SKilic ZWang C(2022)S3-SlicerACM Transactions on Graphics10.1145/3550454.355551641:6(1-15)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555516
Yang JMo KLai YGuibas LGao L(2022)DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape GenerationACM Transactions on Graphics10.1145/352621242:1(1-17)Online publication date: 12-Aug-2022
https://dl.acm.org/doi/10.1145/3526212
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents