research-article

Open access

MoSh: motion and shape capture from sparse markers

Authors:

Naureen Mahmood,

Michael J. BlackAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 6

Article No.: 220, Pages 1 - 13

https://doi.org/10.1145/2661229.2661273

Published: 19 November 2014 Publication History

Abstract

Marker-based motion capture (mocap) is widely criticized as producing lifeless animations. We argue that important information about body surface motion is present in standard marker sets but is lost in extracting a skeleton. We demonstrate a new approach called MoSh (Motion and Shape capture), that automatically extracts this detail from mocap data. MoSh estimates body shape and pose together using sparse marker data by exploiting a parametric model of the human body. In contrast to previous work, MoSh solves for the marker locations relative to the body and estimates accurate body shape directly from the markers without the use of 3D scans; this effectively turns a mocap system into an approximate body scanner. MoSh is able to capture soft tissue motions directly from markers by allowing body shape to vary over time. We evaluate the effect of different marker sets on pose and shape accuracy and propose a new sparse marker set for capturing soft-tissue motion. We illustrate MoSh by recovering body shape, pose, and soft-tissue motion from archival mocap data and using this to produce animations with subtlety and realism. We also show soft-tissue motion retargeting to new characters and show how to magnify the 3D deformations of soft tissue to create animations with appealing exaggerations.

Supplementary Material

ZIP File (a220.zip)

Supplemental material.

Download
44.98 MB

References

[1]

Allen, B., Curless, B., and Popović, Z. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. (Proc. SIGGRAPH) 22, 3, 587--594.

Digital Library

[2]

Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. 2005. SCAPE: Shape Completion and Animation of PEople. ACM Trans. Graph. (Proc. SIGGRAPH 24, 3, 408--416.

Digital Library

[3]

Bogo, F., Romero, J., Loper, M., and Black, M. J. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[4]

de Aguiar, E., Theobalt, C., Stoll, C., and Seidel, H.-P. 2007. Marker-less deformable mesh tracking for human shape and motion capture. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 1--8.

[5]

de Aguiar, E., Zayer, R., Theobalt, C., Seidel, H. P., and Magnor, M. 2007. A simple framework for natural animation of digitized models. In Computer Graphics and Image Processing, 2007. SIBGRAPI 2007. XX Brazilian Symposium on, 3--10.

Digital Library

[6]

de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (Aug.), 98:1--98:10.

Digital Library

[7]

Griewank, A., and Walther, A. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, second ed. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.

Digital Library

[8]

Hirshberg, D. A., Loper, M., Rachlin, E., and Black, M. J. 2012. Coregistration: Simultaneous alignment and modeling of articulated 3d shape. In Computer Vision ECCV 2012, A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid, Eds., vol. 7577 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, 242--255.

Digital Library

[9]

Hong, Q. Y., Park, S. I., and Hodgins, J. K. 2010. A data-driven segmentation for the shoulder complex. Computer Graphics Forum 29, 2, 537--544.

[10]

Jain, A., Thormählen, T., Seidel, H.-P., and Theobalt, C. 2010. MovieReshape: Tracking and reshaping of humans in videos. ACM Transactiosn on Graphics (Proc. SIGGRAPH) 29, 6 (Dec.), 148:1--148:10.

Digital Library

[11]

Kwon, J.-Y., and Lee, I.-K. 2007. Rubber-like exaggeration for character animation. In Proceedings of the 15th Pacific Conference on Computer Graphics and Applications, IEEE Computer Society, Washington, DC, USA, PG '07, 18--26.

Digital Library

[12]

Leardini, A., Chiari, L., Croce, U. D., and Cappozzo, A. 2005. Human movement analysis using stereophotogrammetry: Part 3. soft tissue artifact assessment and compensation. Gait & Posture 21, 2, 212--225.

Digital Library

[13]

Liu, Y., Gall, J., Stoll, C., Dai, Q., Seidel, H.-P., and Theobalt, C. 2013. Markerless motion capture of multiple characters using multiview image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 11, 2720--2735.

Digital Library

[14]

Livne, M., Sigal, L., Troje, N., and Fleet, D. 2012. Human attributes from 3D pose tracking. Computer Vision and Image Understanding 116, 5, 648--660.

Digital Library

[15]

Loper, M., 2014. Chumpy autodifferentiation library. http://chumpy.org/.

[16]

Neumann, T., Varanasi, K., Hasler, N., Wacker, M., Magnor, M., and Theobalt, C. 2013. Capture and statistical modeling of arm-muscle deformations. Computer Graphics Forum 32, 2 (May), 285--294.

[17]

Neumann, T., Varanasi, K., Wenger, S., Wacker, M., Magnor, M., and Theobalt, C. 2013. Sparse localized deformation components. ACM Trans. Graph. 32, 6 (Nov.), 179:1--179:10.

Digital Library

[18]

Nocedal, J., and Wright, S. J. 2006. Numerical Optimization, 2nd ed. Springer, New York.

[19]

Park, S. I., and Hodgins, J. K. 2006. Capturing and animating skin deformation in human motion. ACM Trans. Graph. (Proc. SIGGRAPH) 25, 3 (July), 881--889.

Digital Library

[20]

Park, S. I., and Hodgins, J. K. 2008. Data-driven modeling of skin and muscle deformation. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (Aug.), 96:1--96:6.

Digital Library

[21]

Robinette, K., Blackwell, S., Daanen, H., Boehmer, M., Fleming, S., Brill, T., Hoeferlin, D., and Burnsides, D. 2002. Civilian American and European Surface Anthropometry Resource (CAESAR) final report. Tech. Rep. AFRL-HE-WP-TR-2002-0169, US Air Force Research Laboratory.

Digital Library

[22]

Stark, J., and Hilton, A. 2007. Surface capture for performance-based animation. IEEE Computer Graphics and Applications 27, 3, 21--31.

Digital Library

[23]

Tsoli, A., Mahmood, N., and Black, M. J. 2014. Breathing life into shape: Capturing, modeling and animating 3D human breathing. ACM Trans. Graph., (Proc. SIGGRAPH) 33, 4 (July), 52:1--52:11.

Digital Library

[24]

Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W. T. 2013. Phase-based video motion processing. ACM Trans. Graph., (Proc. SIGGRAPH) 32, 4 (July), 80:1--80:10.

Digital Library

[25]

Wang, H., Xu, N., Raskar, R., and Ahuja, N. 2007. Videoshop: A new framework for spatio-temporal video editing in gradient domain. Graph. Models 69, 1, 57--70.

Digital Library

[26]

Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., and Freeman, W. T. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 4 (July), 65:1--65:8.

Digital Library

Cited By

Udayan J DTV JRaman RKim HNedungadi P(2025)Deep learning in monocular 3D human pose estimation: Systematic review of contemporary techniques and applicationsMultimedia Tools and Applications10.1007/s11042-024-20495-2Online publication date: 6-Feb-2025
https://doi.org/10.1007/s11042-024-20495-2
Chen FChoi J(2024)ATGT3D: Animatable Texture Generation and Tracking for 3D AvatarsElectronics10.3390/electronics1322456213:22(4562)Online publication date: 20-Nov-2024
https://doi.org/10.3390/electronics13224562
Ray LZhou BSuh SLukowicz P(2024)A comprehensive evaluation of marker-based, markerless methods for loose garment scenarios in varying camera configurationsFrontiers in Computer Science10.3389/fcomp.2024.13799256Online publication date: 5-Apr-2024
https://doi.org/10.3389/fcomp.2024.1379925
Show More Cited By

Index Terms

MoSh: motion and shape capture from sparse markers
1. Computing methodologies
  1. Computer graphics
    1. Animation

Recommendations

Dyna: a model of dynamic human shape in motion

To look human, digital full-body avatars need to have soft-tissue deformations like those of real people. We learn a model of soft-tissue deformations from examples using a high-resolution 4D capture system and a method that accurately registers a ...
Capturing and animating skin deformation in human motion
SIGGRAPH '06: ACM SIGGRAPH 2006 Papers

During dynamic activities, the surface of the human body moves in many subtle but visually significant ways: bending, bulging, jiggling, and stretching. We present a technique for capturing and animating those motions using a commercial motion capture ...
Capturing and animating skin deformation in human motion

During dynamic activities, the surface of the human body moves in many subtle but visually significant ways: bending, bulging, jiggling, and stretching. We present a technique for capturing and animating those motions using a commercial motion capture ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 6

November 2014

704 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2661229

Issue’s Table of Contents

Copyright © 2014 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 November 2014

Published in TOG Volume 33, Issue 6

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

276
Total Citations
View Citations
3,819
Total Downloads

Downloads (Last 12 months)512
Downloads (Last 6 weeks)75

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Udayan J DTV JRaman RKim HNedungadi P(2025)Deep learning in monocular 3D human pose estimation: Systematic review of contemporary techniques and applicationsMultimedia Tools and Applications10.1007/s11042-024-20495-2Online publication date: 6-Feb-2025
https://doi.org/10.1007/s11042-024-20495-2
Chen FChoi J(2024)ATGT3D: Animatable Texture Generation and Tracking for 3D AvatarsElectronics10.3390/electronics1322456213:22(4562)Online publication date: 20-Nov-2024
https://doi.org/10.3390/electronics13224562
Ray LZhou BSuh SLukowicz P(2024)A comprehensive evaluation of marker-based, markerless methods for loose garment scenarios in varying camera configurationsFrontiers in Computer Science10.3389/fcomp.2024.13799256Online publication date: 5-Apr-2024
https://doi.org/10.3389/fcomp.2024.1379925
Ross GZhao XTroje NFischer SGraham R(2024)Assessing inter- and intra-rater reliability of movement scores and the effects of body-shape using a custom visualisation tool: an exploratory studyBMC Sports Science, Medicine and Rehabilitation10.1186/s13102-024-00988-116:1Online publication date: 30-Sep-2024
https://doi.org/10.1186/s13102-024-00988-1
Kim KSeo SHan DKang H(2024)DAMO: A Deep Solver for Arbitrary Marker Configuration in Optical Motion CaptureACM Transactions on Graphics10.1145/369586544:1(1-14)Online publication date: 14-Sep-2024
https://dl.acm.org/doi/10.1145/3695865
Hewitt CSaleh FAliakbarian SPetikam LRezaeifar SFlorentin LHosenie ZCashman TValentin JCosker DBaltrusaitis T(2024)Look Ma, no markers: holistic performance capture without the hassleACM Transactions on Graphics10.1145/368777243:6(1-12)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687772
Wu EKhirodkar RKoike HKitani K(2024)SolePoser: Full Body Pose Estimation using a Single Pair of Insole SensorProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676418(1-9)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676418
Xie ZHe HZou GWu JLiu GZhao JWang YLin HLin WGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Visibility-guided Human Body Reconstruction from Uncalibrated Multi-view CamerasProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658110(589-598)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658110
Milef NKeyser JKong S(2024)Towards Unstructured Unlabeled Optical Mocap: A Video Helps!ACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657522(1-11)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657522
Panev SKim ESi Namburu SNikolova Dde Melo CDe La Torre FHodgins J(2024)Exploring the Impact of Rendering Method and Motion Quality on Model Performance when Using Multi-view Synthetic Data for Action Recognition2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00453(4580-4590)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00453
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents