research-article

Real-time finger tracking using active motion capture: a neural network approach robust to occlusions

Authors:

Thibault Porssut,

Bruno Herbelin,

Ronan BoulicAuthors Info & Claims

MIG '18: Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games

Article No.: 6, Pages 1 - 10

https://doi.org/10.1145/3274247.3274501

Published: 08 November 2018 Publication History

Abstract

Hands deserve particular attention in virtual reality (VR) applications because they represent our primary means for interacting with the environment. Although marker-based motion capture with inverse kinematics works adequately for full body tracking, it is less reliable for small body parts such as hands and fingers which are often occluded when captured optically, thus leading VR professionals to rely on additional systems (e.g. inertial trackers). We present a machine learning pipeline to track hands and fingers using solely a motion capture system based on cameras and active markers. Our finger animation is performed by a predictive model based on neural networks trained on a movements dataset acquired from several subjects with a complementary capture system. We employ a two-stage pipeline, which first resolves occlusions, and then recovers all joint transformations. We show that our method compares favorably to inverse kinematics by inferring automatically the constraints from the data, provides a natural reconstruction of postures, and handles occlusions better than three proposed baselines.

Supplementary Material

MP4 File (a6-pavllo.mp4)

Download
43.06 MB

References

[1]

Simon Alexanderson, Carol O'Sullivan, and Jonas Beskow. 2017. Real-time labeling of non-rigid motion capture marker sets. Computers & Graphics 69, Supplement C (2017), 59 -- 67.

Digital Library

[2]

Ahmed RJ Almusawi, L Canan Dülger, and Sadettin Kapucu. 2016. A New Artificial Neural Network Approach in Solving Inverse Kinematics of Robotic Arm (Denso VP6242). Computational intelligence and neuroscience (2016).

Digital Library

[3]

Sheldon Andrews, Ivan Huerta, Taku Komura, Leonid Sigal, and Kenny Mitchell. 2016. Real-time physics-based motion capture with sparse sensors. In Proceedings of the 13th European Conference on Visual Media Production (CVMP). ACM, 5.

Digital Library

[4]

Andreas Aristidou, Jonathan Cameron, and Joan Lasenby. 2008. Real-Time Estimation of Missing Markers in Human Motion Capture. In 2008 2nd International Conference on Bioinformatics and Biomedical Engineering. IEEE, 1343--1346.

[5]

Andreas Aristidou, Joan Lasenby, Yiorgos Chrysanthou, and Ariel Shamir. 2017. Inverse Kinematics Techniques in Computer Graphics: A Survey. Computer Graphics Forum 37, 6 (2017), 35--58.

[6]

H. Bourlard and Y. Kamp. 1988. Auto-association by Multilayer Perceptrons and Singular Value Decomposition. Biol. Cybern. 59, 4-5 (Sept. 1988), 291--294.

Digital Library

[7]

Sidney Bovet, Henrique Galvan Debarba, Bruno Herbelin, Eray Molla, and Ronan Boulic. 2018. The critical role of self-contact for embodiment in virtual reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1428--1436.

Digital Library

[8]

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. 315--323.

[9]

F Sebastian Grassia. 1998. Practical parameterization of rotations using the exponential map. Journal of graphics tools 3, 3 (1998), 29--48.

Digital Library

[10]

Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D Twigg, and Kenrick Kin. 2018. Online optical marker-based hand tracking with deep labels. In SIGGRAPH.

Digital Library

[11]

L. Herda, P. Fua, R. Plänkers, R. Boulic, and D. Thalmann. 2000. Skeleton-Based Motion Capture for Robust Reconstruction of Human Motion. In Proceedings of the Computer Animation (CA '00). IEEE Computer Society, Washington, DC, USA, 77-.

Digital Library

[12]

Arthur E. Hoerl and Robert W. Kennard. 1970. Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics 12, 1 (1970), 55--67.

Digital Library

[13]

Daniel Holden. 2018. Robust Solving of Optical Motion Capture Data by Denoising. In SIGGRAPH.

Digital Library

[14]

Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning Motion Manifolds with Convolutional Autoencoders. In SIGGRAPH Asia 2015 Technical Briefs (SA '15). ACM, New York, NY, USA, Article 18, 4 pages.

Digital Library

[15]

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2, 5 (jan 1989), 359--366. arXiv:arXiv:1011.1669v3

[16]

K Jiang, H Chen, and S Yuan. 2005. Classification for Incomplete Data Using Classifier Ensembles. Proc International Conference on Neural Networks and Brain, 2005. (ICNN&B 05) 1 (2005), 559--563.

[17]

W. Kabsch. 1976. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A 32, 5 (Sep 1976), 922--923.

[18]

Byoung-Ho Kim. 2014. An Adaptive Neural Network Learning-based Solution for the Inverse Kinematics of Humanoid Fingers. International Journal of Advanced Robotic Systems 11, 1 (2014), 3.

[19]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). arXiv:1412.6980

[20]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS'12). Curran Associates Inc., USA, 1097--1105.

Digital Library

[21]

Thomas B. Moeslund, Adrian Hilton, and Volker Krüger. 2006. A survey of advances in vision-based human motion capture and analysis. Computer Vision and Image Understanding 104, 2 (2006), 90 -- 126.

Digital Library

[22]

Eray Molla, Henrique Galvan Debarba, and Ronan Boulic. 2017. Egocentric Mapping of Body Surface Constraints. IEEE Transactions on Visualization and Computer Graphics (2017), 1--1.

[23]

Christos Mousas and Christos-Nikolaos Anagnostopoulos. 2017. Real-time performance-driven finger motion synthesis. Computers & Graphics 65, Supplement C (2017), 1 -- 11.

[24]

Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In Proceedings of International Conference on Computer Vision (ICCV), Vol. 10.

[25]

Vinod Nair and Geoffrey E. Hinton. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML'10). Omnipress, USA, 807--814.

Digital Library

[26]

Tommaso Piazza, Johan Lundström, Andreas Kunz, and Morten Fjeld. 2009. Predicting Missing Markers in Real-Time Optical Motion Capture. LNCS 5903 (2009), 125--136.

Digital Library

[27]

Karen Rafferty, K Nickleson, Scott Devine, and Chris Herdman. 2017. Improving the ergonomics of hand tracking inputs to VR HMDs. In International Conferences in Central Europe on Human Computer Interaction, Pilsen, Czech Republic.

[28]

Matthias Schröder, Jonathan Maycock, and Mario Botsch. 2015. Reduced Marker Layouts for Optical Motion Capture of Hands. In Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games (MIG '15). ACM, New York, NY, USA, 7--16.

Digital Library

[29]

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. 2011. Real-time human pose recognition in parts from single depth images. In CVPR 2011. 1297--1304.

Digital Library

[30]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929--1958. arXiv:1102.4807

Digital Library

[31]

Graham W. Taylor, Geoffrey E Hinton, and Sam T. Roweis. 2007. Modeling Human Motion Using Binary Latent Variables. In Advances in Neural Information Processing Systems 19, B. Schölkopf, J. C. Platt, and T. Hoffman (Eds.). MIT Press, 1345--1352. http://papers.nips.cc/paper/3078-modeling-human-motion-using-binary-latent-variables.pdf

Digital Library

[32]

Robert Tibshirani. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) (1996), 267--288.

[33]

Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-meshes for Real-time Hand Modeling and Tracking. ACM Trans. Graph. 35, 6, Article 222 (Nov. 2016), 11 pages.

Digital Library

[34]

Tim Waegeman and Benjamin Schrauwen. 2011. Towards Learning Inverse Kinematics with a Neural Network Based Tracking Controller. In Neural Information Processing, Bao-Liang Lu, Liqing Zhang, and James Kwok (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 441--448.

[35]

D.J. Wiley and J.K. Hahn. 1997. Interpolation synthesis of articulated figure motion. IEEE Computer Graphics and Applications 17, 6 (1997), 39--45.

Digital Library

[36]

Xingyi Zhou, Xiao Sun, Wei Zhang, Shuang Liang, and Yichen Wei. 2016a. Deep kinematic pose regression. In ECCV Workshop on Geometry Meets Deep Learning. Springer, 186--201.

[37]

Xingyi Zhou, Qingfu Wan, Wei Zhang, Xiangyang Xue, and Yichen Wei. 2016b. Model-based Deep Hand Pose Estimation. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI'16). AAAI Press, 2421--2427.

Digital Library

Cited By

Sun JHuang LWang HZheng CQiu JIslam MXie EZhou BXing LChandrasekaran ABlack M(2024)Localization and recognition of human action in 3D using transformersCommunications Engineering10.1038/s44172-024-00272-73:1Online publication date: 3-Sep-2024
https://doi.org/10.1038/s44172-024-00272-7
Sierra Picon GRocha Castano CSarmiento W(2021)Non-Traditional Rig for Virtual Environment Systems. Case study: Arm rigProceedings of the 23rd Symposium on Virtual and Augmented Reality10.1145/3488162.3488230(162-166)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3488162.3488230
Li JXiao DLi KLi J(2021)Graph Matching for Marker Labeling and Missing Marker Reconstruction With Bone Constraint by LSTM in Optical Motion CaptureIEEE Access10.1109/ACCESS.2021.30603859(34868-34881)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3060385
Show More Cited By

Index Terms

Real-time finger tracking using active motion capture: a neural network approach robust to occlusions
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Robust online motion capture labeling of finger markers
MIG '16: Proceedings of the 9th International Conference on Motion in Games

Passive optical motion capture is one of the predominant technologies for capturing high fidelity human skeletal motion, and is a workhorse in a large number of areas such as bio-mechanics, film and video games. While most state-of-the-art systems can ...
Real-Time Multi-person Motion Capture from Multi-view Video and IMUs
Abstract
A real-time motion capture system is presented which uses input from multiple standard video cameras and inertial measurement units (IMUs). The system is able to track multiple people simultaneously and requires no optical markers, specialized ...
Real-time labeling of non-rigid motion capture marker sets

A method for real-time labeling of passive motion capture markers is proposed.The method is optimized for capturing non-rigid structures such as fingers and faces.By creating multiple assignment hypotheses, the method is robust to occlusions and ghost ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MIG '18: Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games

November 2018

185 pages

ISBN:9781450360159

DOI:10.1145/3274247

Conference Chairs:
Panayiotis Charalambous
The Cyprus Institute
,
Yiorgos Chrysanthou
The University of Cyprus & RISE
,
Program Chairs:
Ben Jones
University of Utah
,
Jehee Lee
Seoul National University

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Student Paper

Author Tags

Qualifiers

Research-article

Funding Sources

Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung

Conference

MIG '18

Sponsor:

SIGGRAPH

MIG '18: Motion, Interaction and Games

November 8 - 10, 2018

Limassol, Cyprus

Acceptance Rates

Overall Acceptance Rate -9 of -9 submissions, 100%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
524
Total Downloads

Downloads (Last 12 months)37
Downloads (Last 6 weeks)5

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sun JHuang LWang HZheng CQiu JIslam MXie EZhou BXing LChandrasekaran ABlack M(2024)Localization and recognition of human action in 3D using transformersCommunications Engineering10.1038/s44172-024-00272-73:1Online publication date: 3-Sep-2024
https://doi.org/10.1038/s44172-024-00272-7
Sierra Picon GRocha Castano CSarmiento W(2021)Non-Traditional Rig for Virtual Environment Systems. Case study: Arm rigProceedings of the 23rd Symposium on Virtual and Augmented Reality10.1145/3488162.3488230(162-166)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3488162.3488230
Li JXiao DLi KLi J(2021)Graph Matching for Marker Labeling and Missing Marker Reconstruction With Bone Constraint by LSTM in Optical Motion CaptureIEEE Access10.1109/ACCESS.2021.30603859(34868-34881)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3060385
Chatzitofis AZarpalas DDaras PKollias S(2021)DeMoCap: Low-Cost Marker-Based Motion CaptureInternational Journal of Computer Vision10.1007/s11263-021-01526-z129:12(3338-3366)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1007/s11263-021-01526-z
Rosskamp JWeller RKluss TMaldonado C. JZachmann G(2020)Improved CNN-Based Marker Labeling for Optical Hand TrackingVirtual Reality and Augmented Reality10.1007/978-3-030-62655-6_10(165-177)Online publication date: 25-Nov-2020
https://dl.acm.org/doi/10.1007/978-3-030-62655-6_10

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents