research-article

Towards Deep Learning based Hand Keypoints Detection for Rapid Sequential Movements from RGB Images

Authors:

Srujana Gattupalli,

Ashwin Ramesh Babu,

James Robert Brady,

Fillia Makedon,

Vassilis AthitsosAuthors Info & Claims

PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference

Pages 31 - 37

https://doi.org/10.1145/3197768.3201538

Published: 26 June 2018 Publication History

Abstract

Hand keypoints detection and pose estimation has numerous applications in computer vision, but it is still an unsolved problem in many aspects. An application of hand keypoints detection is in performing cognitive assessments of a subject by observing the performance of that subject in physical tasks involving rapid finger motion. As a part of this work, we introduce a novel hand keypoints benchmark dataset that consists of hand gestures recorded specifically for cognitive behavior monitoring. We explore the state of the art methods in hand keypoint detection and we provide quantitative evaluations for the performance of these methods on our dataset. In future, these results and our dataset can serve as a useful benchmark for hand keypoint recognition for rapid finger movements.

References

[1]

Mohamad Baydoun, Alejandro Betancourt, Pietro Morerio, Lucio Marcenaro, Matthias Rauterberg, and Carlo S. Regazzoni. 2017. Hand pose recognition in First Person Vision through graph spectral analysis. In ICASSP. IEEE, 1872--1876.

[2]

Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek, and Tae-Kyun Kim. 2017. First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations. CoRR abs/1704.02463 (2017).

[3]

Srujana Gattupalli, Dylan Ebert, Michalis Papakostas, Fillia Makedon, and Vassilis Athitsos. 2017. CogniLearn: A Deep Learning-based Interface for Cognitive Behavior Assessment. In Proceedings of the 22Nd International Conference on Intelligent User Interfaces (IUI '17). 577--587.

Digital Library

[4]

Liuhao Ge, Hui Liang, Junsong Yuan, and Daniel Thalmann. 2016. Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs. In CVPR. IEEE Computer Society, 3593--3601.

[5]

De-An Huang, Minghuang Ma, Wei-Chiu Ma, and Kris M. Kitani. 2015. How do we use our hands? Discovering a diverse set of common grasps. In CVPR. IEEE Computer Society, 666--675.

[6]

Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. 2016. DeeperCut: A deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer Vision. Springer, 34--50.

[7]

Cem Keskin, Furkan KiraÃğ, Yunus Emre Kara, and Lale Akarun. 2011. Real time hand pose estimation using depth sensors. In ICCV Workshops. IEEE, 1228--1234.

[8]

Markus Oberweger and Vincent Lepetit. 2017. DeepPrior++: Improving Fast and Accurate 3D Hand Pose Estimation. In ICCV Workshops. IEEE Computer Society, 585--594.

[9]

Grégory Rogez, Maryam Khademi, James Steven Supancic III, J. M. M. Montiel, and Deva Ramanan. 2014. 3D Hand Pose Detection in Egocentric RGB-D Images. In ECCV Workshops (1) (Lecture Notes in Computer Science), Vol. 8925. Springer, 356--371.

[10]

B. Stenger S. Jain S. Yuan, Q. Ye and T.-K. Kim. 2017. BigHand2.2m benchmark: Hand pose dataset and state of the art analysis. In CVPR.

[11]

Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand Keypoint Detection in Single Images using Multiview Bootstrapping. In CVPR.

[12]

Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. 2016. Real-Time Joint Tracking of a Hand Manipulating an Object from RGB-D Input. In ECCV (2) (Lecture Notes in Computer Science), Vol. 9906. Springer, 294--310.

[13]

Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. 2013. Interactive Markerless Articulated Hand Motion Tracking Using RGB and Depth Data. In ICCV. IEEE Computer Society, 2456--2463.

Digital Library

[14]

Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. 2015. Cascaded hand pose regression. In CVPR. IEEE Computer Society, 824--832.

[15]

W. J. Tsai, J. C. Chen, and K. W. Lin. 2016. Depth-Based Hand Pose Segmentation with Hough Random Forest. In 2016 3rd International Conference on Green Technology and Sustainable Development (GTSD). 166--167.

[16]

Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation. International Journal of Computer Vision 118, 2 (2016), 172--193.

Digital Library

[17]

Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional Pose Machines. In CVPR. 4724--4732.

[18]

Yimin Zhou, Guolai Jiang, and Yaorong Lin. 2016. A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recognition 49 (2016), 102--114.

Digital Library

[19]

Christian Zimmermann et al. 2017. Learning to Estimate 3D Hand Pose from Single RGB Images. In IEEE International Conference on Computer Vision (ICCV). https://lmb.informatik.uni-freiburg.de/projects/hand3d/ https://arxiv.org/abs/1705.01389.

Cited By

Mishra PSarawadekar K(2024)Multiple-Hand 2D Pose Estimation From a Monocular RGB ImageIEEE Access10.1109/ACCESS.2024.337642612(40722-40735)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3376426
Jaiswal ARamesh Babu AZaki Zadeh MWylie GMakedon F(2023)Detecting Cognitive Fatigue in Subjects with Traumatic Brain Injury from FMRI Scans Using Self-Supervised LearningProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594868(83-90)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594868
Karim ENambiappan HAcharya SMakedon F(2023)Remote Operated Human Robot Interactive System using Hand Gestures for Persons with DisabilitiesProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594832(137-139)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594832
Show More Cited By

Index Terms

Recommendations

Hand pose recognition from monocular images by geometrical and texture analysis

One challenging research problem of hand pose recognition is the accurate detection of finger abduction and flexion with a single camera. The detection of flexion movements from a 2D image is difficult, because it involves estimation of finger movements ...
Pointing gesture recognition based on 3D-tracking of face, hands and head orientation
ICMI '03: Proceedings of the 5th international conference on Multimodal interfaces

In this paper, we present a system capable of visually detecting pointing gestures and estimating the 3D pointing direction in real-time. In order to acquire input features for gesture recognition, we track the positions of a person's face and hands on ...
HAND-REHA: dynamic hand gesture recognition for game-based wrist rehabilitation
PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments

Hand-gesture recognition systems have recently gained more popularity. Moreover, there is a growing interest in building games for other purposes apart from entertainment, such as education and rehabilitation. This paper focuses on developing a novel ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '18: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference

June 2018

591 pages

ISBN:9781450363907

DOI:10.1145/3197768

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

NSF: National Science Foundation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PETRA '18

PETRA '18: The 11th PErvasive Technologies Related to Assistive Environments Conference

June 26 - 29, 2018

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
182
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mishra PSarawadekar K(2024)Multiple-Hand 2D Pose Estimation From a Monocular RGB ImageIEEE Access10.1109/ACCESS.2024.337642612(40722-40735)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3376426
Jaiswal ARamesh Babu AZaki Zadeh MWylie GMakedon F(2023)Detecting Cognitive Fatigue in Subjects with Traumatic Brain Injury from FMRI Scans Using Self-Supervised LearningProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594868(83-90)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594868
Karim ENambiappan HAcharya SMakedon F(2023)Remote Operated Human Robot Interactive System using Hand Gestures for Persons with DisabilitiesProceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3594806.3594832(137-139)Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1145/3594806.3594832
Santos BPais PRibeiro FLima JGonçalves GPinto V(2023)Deep Convolutional Neural Networks applied to Hand Keypoints Estimation2023 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)10.1109/ICARSC58346.2023.10129621(93-98)Online publication date: 26-Apr-2023
https://doi.org/10.1109/ICARSC58346.2023.10129621
Kapoor RJaiswal AMakedon FMakedon F(2022)Light-Weight Seated Posture Guidance System with Machine Learning and Computer VisionProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3535341(595-600)Online publication date: 29-Jun-2022
https://dl.acm.org/doi/10.1145/3529190.3535341
Pavel HKarim EZaki Zadeh MJaiswal AKapoor RMakedon FMakedon F(2022)Automated System to Measure Static Balancing in Children to Assess Executive FunctionProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3534750(569-575)Online publication date: 29-Jun-2022
https://dl.acm.org/doi/10.1145/3529190.3534750
Bayd HGuyot PBardy BSlangen P(2022)Automatic Scoring of Synchronization from Fingers Motion Capture and Music BeatsImage Analysis and Processing. ICIAP 2022 Workshops10.1007/978-3-031-13321-3_21(235-245)Online publication date: 7-Aug-2022
https://doi.org/10.1007/978-3-031-13321-3_21
Bell MWeinstein APittman BGorman RAbujelala M(2021)The Activate Test of Embodied Cognition (ATEC): Reliability, concurrent validity and discriminant validity in a community sample of children using cognitively demanding physical tasks related to executive functioningChild Neuropsychology10.1080/09297049.2021.192368627:7(973-983)Online publication date: 13-May-2021
https://doi.org/10.1080/09297049.2021.1923686
Ramesh Babu AZadeh MJaiswal ALueckenhoff AKyrarini MMakedon FTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)A Multi-modal System to Assess Cognition in Children from their Physical MovementsProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3418829(6-14)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3382507.3418829
Babu AZakizadeh MBrady JCalderon DMakedon F(2019)An Intelligent Action Recognition System to assess Cognitive Behavior for Executive Function Disorder2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)10.1109/COASE.2019.8843199(164-169)Online publication date: Aug-2019
https://doi.org/10.1109/COASE.2019.8843199

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents