research-article

A system for pose analysis and selection in virtual reality environments

Authors:

Anban W. Pillay,

Deshendran MoodleyAuthors Info & Claims

SAICSIT '20: Conference of the South African Institute of Computer Scientists and Information Technologists 2020

Pages 210 - 216

https://doi.org/10.1145/3410886.3410909

Published: 14 September 2020 Publication History

Abstract

Depth cameras provide a natural and intuitive user interaction mechanism in virtual reality environments by using hand gestures as the primary user input. However, building robust VR systems that use depth cameras are challenging. Gesture recognition accuracy is affected by occlusion, variation in hand orientation and misclassification of similar hand gestures. This research explores the limits of the Leap Motion depth camera for static hand pose recognition in virtual reality applications. We propose a system for analysing static hand poses and for systematically identifying a pose set that can achieve a near-perfect recognition accuracy. The system consists of a hand pose taxonomy, a pose notation, a machine learning classifier and an algorithm to identify a reliable pose set that can achieve near perfect accuracy levels. We used this system to construct a benchmark hand pose data set containing 2550 static hand pose instances, and show how the algorithm can be used to systematically derive a set of poses that can produce an accuracy of 99% using a Support Vector Machine classifier.

References

[1]

M. Alimanova, S. Borambayeva, D. Kozhamzharova, N. Kurmangaiyeva, D. Ospanova, G. Tyulepberdinova, G. Gaziz, and A. Kassenkhan. 2017. Gamification of Hand Rehabilitation Process Using Virtual Reality Tools: Using Leap Motion for Hand Rehabilitation. In 2017 First IEEE International Conference on Robotic Computing (IRC). 336–339. https://doi.org/10.1109/IRC.2017.76

[2]

Nathan Beattie, Ben Horan, and Sophie McKenzie. 2015. Taking the LEAP with the Oculus HMD and CAD - Plucking at thin Air?Procedia Technology 20 (Jan. 2015), 149–154. https://doi.org/10.1016/j.protcy.2015.07.025

[3]

J. Blaha and M. Gupta. 2014. Diplopia: A virtual reality game designed to help amblyopics. In Virtual Reality (VR), 2014 iEEE. 163–164. https://doi.org/10.1109/VR.2014.6802102

[4]

Eunjung Choi, Heejin Kim, and Min K. Chung. 2014. A taxonomy and notation method for three-dimensional hand gestures. International Journal of Industrial Ergonomics 44, 1 (Jan. 2014), 171–188. https://doi.org/10.1016/j.ergon.2013.10.011

[5]

Ching-Hua Chuan, E. Regina, and C. Guardino. 2014. American Sign Language Recognition Using Leap Motion Sensor. In 2014 13th International Conference on Machine Learning and Applications (ICMLA). 541–544. https://doi.org/10.1109/ICMLA.2014.110

[6]

Andrew Clark and Deshendran Moodley. 2016. A System for a Hand Gesture-Manipulated Virtual Reality Environment. In Proceedings of the Annual Conference of the South African Institute of Computer Scientists and Information Technologists(SAICSIT ’16). ACM, New York, NY, USA, 10:1–10:10. https://doi.org/10.1145/2987491.2987511

Digital Library

[7]

Bruno Fanini. 2014. A 3D Interface to Explore and Manipulate multi-scale Virtual Scenes using the Leap Motion Controller. In ACHI 2014 : The Seventh International Conference on Advances in Computer-Human Interactions. http://citeseerx.ist.psu.edu/viewdoc/citations;jsessionid=DBFED2BDCFA6A3B6FE84489C8DA2528E?doi=10.1.1.673.7112

[8]

Jože Guna, Grega Jakus, Matevž Pogačnik, Sašo Tomažič, and Jaka Sodnik. 2014. An Analysis of the Precision and Reliability of the Leap Motion Sensor and Its Suitability for Static and Dynamic Tracking. Sensors 14, 2 (Feb. 2014), 3702–3720. https://doi.org/10.3390/s140203702

[9]

Isabelle Guyon, Vassilis Athitsos, Pat Jangyodsuk, and Hugo Jair Escalante. 2014. The ChaLearn Gesture Dataset (CGD 2011). Mach. Vision Appl. 25, 8 (Nov. 2014), 1929–1951. https://doi.org/10.1007/s00138-014-0596-3

Digital Library

[10]

R. W. Hamming. 1950. Error detecting and error correcting codes. The Bell System Technical Journal 29, 2 (April 1950), 147–160. https://doi.org/10.1002/j.1538-7305.1950.tb00463.x

[11]

D. E. Holmes, D. K. Charles, P. J. Morrow, S. McClean, and S. M. McDonough. 2016. Using Fitt’s Law to Model Arm Motion Tracked in 3D by a Leap Motion Controller for Virtual Reality Upper Arm Stroke Rehabilitation. In 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS). 335–336. https://doi.org/10.1109/CBMS.2016.41

[12]

Maria Karam and M. C. Schraefel. 2005. A Taxonomy of Gestures in Human Computer Interactions. Master’s thesis. North Dakota State University.

[13]

Stoyan Kerefeyn and Stoyan Maleshkov. 2015. Manipulation of virtual objects through a LeapMotion optical sensor. ResearchGate 12, 5 (Oct. 2015), 52–57. https://www.researchgate.net/publication/283643278_Manipulation_of_virtual_objects_through_a_LeapMotion_optical_sensor

[14]

S. Khattak, B. Cowan, I. Chepurna, and A. Hogue. 2014. A real-time reconstructed 3D environment augmented with virtual objects rendered with correct occlusion. In 2014 IEEE Games Media Entertainment (GEM). 1–8. https://doi.org/10.1109/GEM.2014.7048102

[15]

C. Khundam. 2015. First person movement control with palm normal and hand gesture interaction in virtual reality. In 2015 12th International Joint Conference on Computer Science and Software Engineering (JCSSE). 325–330. https://doi.org/10.1109/JCSSE.2015.7219818

[16]

W. Lu, Z. Tong, and J. Chu. 2016. Dynamic Hand Gesture Recognition With Leap Motion Controller. IEEE Signal Processing Letters 23, 9 (Sept. 2016), 1188–1192. https://doi.org/10.1109/LSP.2016.2590470

[17]

Rajesh B. Mapari and Govind Kharat. 2016. American Static Signs Recognition Using Leap Motion Sensor. In Proceedings of the Second International Conference on Information and Communication Technology for Competitive Strategies(ICTCS ’16). ACM, New York, NY, USA, 67:1–67:5. https://doi.org/10.1145/2905055.2905125

[18]

G. Marin, F. Dominio, and P. Zanuttigh. 2014. Hand gesture recognition with leap motion and kinect devices. In 2014 IEEE International Conference on Image Processing (ICIP). 1565–1569. https://doi.org/10.1109/ICIP.2014.7025313

[19]

M. Mohandes, S. Aliyu, and M. Deriche. 2015. Prototype Arabic Sign language recognition using multi-sensor data fusion of two leap motion controllers. In 2015 12th International Multi-Conference on Systems, Signals Devices (SSD). 1–6. https://doi.org/10.1109/SSD.2015.7348113

[20]

Javier Molina, JosÃ© A. Pajuelo, Marcos Escudero-ViÃ±olo, JesÃºs BescÃ³s, and JosÃ© M. MartÃnez. 2014. A natural and synthetic corpus for benchmarking of hand gesture recognition systems. Machine Vision and Applications 25, 4 (May 2014), 943–954. https://doi.org/10.1007/s00138-013-0576-z

Digital Library

[21]

Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-defined Motion Gestures for Mobile Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’11). ACM, New York, NY, USA, 197–206. https://doi.org/10.1145/1978942.1978971

Digital Library

[22]

K. Sabir, C. Stolte, B. Tabor, and S.I. O’Donoghue. 2013. The Molecular Control Toolkit: Controlling 3D molecular graphics via gesture and voice. In 2013 IEEE Symposium on Biological Data Visualization (BioVis). 49–56. https://doi.org/10.1109/BioVis.2013.6664346

[23]

D. Shukla, Ã– Erkent, and J. Piater. 2016. A multi-view hand gesture RGB-D dataset for human-robot interaction scenarios. In 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). 1084–1091. https://doi.org/10.1109/ROMAN.2016.7745243

[24]

Gurminder Singh, Steven K. Feiner, and Daniel Thalmann. 1994. Virtual Reality Software & Technology: Proceedings of the VRST ’94 Conference, 23-26 August 1994, Singapore. World Scientific. Google-Books-ID: ywTGrWPf518C.

[25]

M. Sourial, A. Elnaggar, and D. Reichardt. 2016. Development of a virtual coach scenario for hand therapy using LEAP motion. In 2016 Future Technologies Conference (FTC). 1071–1078. https://doi.org/10.1109/FTC.2016.7821736

[26]

Jeremy Sutton. 2013. Air Painting with Corel Painter Freestyle and the Leap Motion Controller: A Revolutionary New Way to Paint!. In ACM SIGGRAPH 2013 Studio Talks(SIGGRAPH ’13). ACM, New York, NY, USA, 21:1–21:1. https://doi.org/10.1145/2503673.2503694

Digital Library

[27]

Michail Theofanidis, Saif Iftekar Sayed, Alexandros Lioulemes, and Fillia Makedon. 2017. VARM: Using Virtual Reality to Program Robotic Manipulators. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments(PETRA ’17). ACM, New York, NY, USA, 215–221. https://doi.org/10.1145/3056540.3056541

Digital Library

[28]

Fereydoon Vafaei. 2013. Taxonomy of Gestures in Human Computer Interaction. Master’s thesis. North Dakota State University. http://library.ndsu.edu/tools/dspace/load/?file=/repository/bitstream/handle/10365/23110/Vafaei_Taxonomy%20of%20Gestures%20in%20Human%20Computer%20Interaction.pdf?sequence=1

[29]

Radu-Daniel Vatavu and Ionut-Alexandru Zaiti. 2014. Leap Gestures for TV: Insights from an Elicitation Study. In Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video(TVX ’14). ACM, New York, NY, USA, 131–138. https://doi.org/10.1145/2602299.2602316

Digital Library

[30]

Q. Wang, Y. Wang, F. Liu, and W. Zeng. 2017. Hand gesture recognition of Arabic numbers using leap motion via deterministic learning. In 2017 36th Chinese Control Conference (CCC). 10823–10828. https://doi.org/10.23919/ChiCC.2017.8029083

[31]

Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’09). ACM, New York, NY, USA, 1083–1092. https://doi.org/10.1145/1518701.1518866

Digital Library

Cited By

Ergül İÇalış TYücetürk EGür MBulut SBaykal G(2024)Co-Rhythm: Analyzing Children's Performative Gesture-based Interactions in a Music Composition ToolProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659375(686-690)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3628516.3659375
Baykal GLeylekoğlu AArslan SOzer D(2023)Studying Children’s Object Interaction in Virtual Reality: A Manipulative Gesture Taxonomy for VR Hand TrackingExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585865(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585865

Recommendations

Designing in virtual reality (DesIRe): a gesture-based interface
DIMEA '07: Proceedings of the 2nd international conference on Digital interactive media in entertainment and arts

In this paper, our purpose is to focus on detection of dynamic gestures and find the thresholds to detect gestures for a gesture-based system, DesIRe (DESigning In virtual REality). DesIRe treats virtual reality as a platform of virtual clay and allows ...
A Real-Time Interactive System Based on Hand Gesture Recognition in Virtual Fitting
LNCS on Transactions on Edutainment XIII - Volume 10092

Hand gesture interaction has already become a research focus with the rapid development of virtual reality technologies. In this paper, we present a real-time interactive system based on hand gesture recognition in virtual fitting. Firstly, real-time ...
A Quantitative Analysis of Redirected Walking in Virtual Reality Using Saccadic Eye Movements
Advances in Visual Computing
Abstract
In this work we investigated redirected walking in virtual reality using saccadic eye movements to determine when to rotate the participant in their virtual environment. We tested participants by placing them into a virtual environment where they ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SAICSIT '20: Conference of the South African Institute of Computer Scientists and Information Technologists 2020

September 2020

258 pages

ISBN:9781450388474

DOI:10.1145/3410886

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SAICSIT '20

SAICSIT '20: Conference of the South African Institute of Computer Scientists and Information Technologists 2020

September 14 - 16, 2020

Cape Town, South Africa

Acceptance Rates

Overall Acceptance Rate 187 of 439 submissions, 43%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
83
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ergül İÇalış TYücetürk EGür MBulut SBaykal G(2024)Co-Rhythm: Analyzing Children's Performative Gesture-based Interactions in a Music Composition ToolProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659375(686-690)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3628516.3659375
Baykal GLeylekoğlu AArslan SOzer D(2023)Studying Children’s Object Interaction in Virtual Reality: A Manipulative Gesture Taxonomy for VR Hand TrackingExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585865(1-7)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585865

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents