research-article

Multimodal Hand and Foot Gesture Interaction for Handheld Devices

Authors:

Shengzhong Feng,

Shafiq Ur RéhmanAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 11, Issue 1s

Article No.: 10, Pages 1 - 19

https://doi.org/10.1145/2645860

Published: 01 October 2014 Publication History

Abstract

We present a hand-and-foot-based multimodal interaction approach for handheld devices. Our method combines input modalities (i.e., hand and foot) and provides a coordinated output to both modalities along with audio and video. Human foot gesture is detected and tracked using contour-based template detection (CTD) and Tracking-Learning-Detection (TLD) algorithm. 3D foot pose is estimated from passive homography matrix of the camera. 3D stereoscopic and vibrotactile are used to enhance the immersive feeling. We developed a multimodal football game based on the multimodal approach as a proof-of-concept. We confirm our systems user satisfaction through a user study.

References

[1]

Kazi Masudul Alam, Abu Saleh Md Mahfujur Rahman, and Abdulmotaleb El Saddik. 2013. Mobile haptic e-Book system to support 3D immersive reading in ubiquitous environments. ACM Trans. Multimedia Comput. Commun. Appl. 9, 4 (2013), Article 27, 20 pages.

Digital Library

[2]

Jason Alexander, Teng Han, William Judd, Pourang Irani, and Sriram Subramanian. 2012. Putting your best foot forward: Investigating real-world mappings for foot-based gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'12). ACM, New York, NY, 1229--1238.

Digital Library

[3]

Oliver Amft and Paul Lukowicz. 2009. From backpacks to smartphones: Past, present, and future of wearable computers. IEEE Perv. Comput. 8, 3 (2009), 8--13.

Digital Library

[4]

Christoph Amma, Dirk Gehrig, and Tanja Schultz. 2010. Airwriting recognition using wearable motion sensors. In Proceedings of the 1st Augmented Human International Conference (AH'10). ACM, New York, NY, Article 10, 8 pages.

Digital Library

[5]

Thomas Augsten, Konstantin Kaefer, René Meusel, Caroline Fetzer, Dorian Kanitz, Thomas Stoff, Torsten Becker, Christian Holz, and Patrick Baudisch. 2010. Multitoe: High-precision interaction with back-projected floors based on high-resolution multi-touch input. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST'10).

Digital Library

[6]

Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. ShoeSense: A new perspective on gestural interaction and wearable applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI'12). ACM, New York, NY, 1239--1248.

Digital Library

[7]

Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: Multi-“touch” interaction around small devices. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST'08). ACM, New York, NY, 201--204.

Digital Library

[8]

Michael Cohen. 2008. Integration of laptop sudden motion sensor as accelerometric control for virtual environments. In Proceedings of the 7th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (VRCAI'08). ACM, New York, NY, Article 38, 2 pages.

Digital Library

[9]

Andrew Crossan, Stephen Brewster, and Alexander Ng. 2010. Foot tapping for mobile interaction. In Proceedings of the 24th BCS Interaction Specialist Group Conference (BCS'10). British Computer Society, 418--422. http://dl.acm.org/citation.cfm&quest;id=2146303.2146366

Digital Library

[10]

Pedro F. Felzenszwalb and Ramin Zabih. 2011. Dynamic programming and graph algorithms in computer vision. IEEE Trans. Pattern Anal. Mach. Intell. 33, 4 (2011), 721--740.

Digital Library

[11]

Kristen Grauman, Margrit Betke, James Gips, and Gary R. Bradski. 2001. Communication via eye blinks - detection and duration analysis in real time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[12]

Wei Guan, Lu Wang, Jonathan Mooser, Suya You, and Ulrich Neumann. 2009. Robust pose estimation in untextured environments for augmented reality applications. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR'09). IEEE Computer Society, 191--192.

Digital Library

[13]

Wei Guan, Suya You, and Ulrich Newmann. 2012. Efficient matchings and mobile augmented reality. ACM Trans. Multimedia Comput. Commun. Appl. 8, 3s (2012), Article 47, 15 pages.

Digital Library

[14]

Nate Hagbi, Oriel Bergig, Jihad El-Sana, and Mark Billinghurst. 2011. Shape recognition and pose estimation for mobile augmented reality. IEEE Trans. Visual. Comput. Graphics 17, 10 (2011), 1369--1379.

Digital Library

[15]

Alaa Halawani and Haibo Li. 2013. FingerInk: Turn your glass into a digital board. In Proceedings of the Australian Computer-Human Interaction Conference (OzCHI'13).

Digital Library

[16]

Alaa Halawani, Shafiq ur Réhman, Haibo Li, and Adi Anani. 2012. Active vision for controlling an electric wheelchair. Intell. Serv. Robot. 5, 2 (2012), 89--98.

Digital Library

[17]

Teng Han, Jason Alexander, Abhijit Karnik, Pourang Irani, and Sriram Subramanian. 2011. Kick: Investigating the use of kick gestures for mobile interactions. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI'11). ACM, New York, NY, 29--32.

Digital Library

[18]

R. M. Haralick, Hyonam Joo, D. Lee, S. Zhuang, V. G. Vaidya, and M. B. Kim. 1989. Pose estimation from corresponding point data. IEEE Trans. Syst. Man Cyberne. 19, 6 (1989), 1426--1446.

[19]

Michael Hardegger, Gerhard Tröster, and Daniel Roggen. 2013. Improved actionSLAM for long-term indoor tracking with wearable motion sensors. In Proceedings of the International Symposium on Wearable Computers (ISWC'13). ACM, New York, NY, 1--8.

Digital Library

[20]

Chris Harrison and Scott E. Hudson. 2009. Abracadabra: Wireless, high-precision, and unpowered finger input for very small mobile devices. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST'09). ACM, New York, NY, 121--124.

Digital Library

[21]

R. I. Hartley and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (2nd Ed.). Cambridge University Press, ISBN: 0521540518.

Digital Library

[22]

Alejandro Jaimes and Nicu Sebe. 2007. Multimodal human computer interaction: A survey. Comput. Vision Image Understand. 108, 1--2 (October 2007), 116--134.

Digital Library

[23]

Qiang Ji, Mauro S. Costa, Robert M. Haralick, and Linda G. Shapiro. 2000. A robust linear least-squares estimation of camera exterior orientation using multiple geometric features. ISPRS J. Photogrammetry Remote Sens. 55, 2 (2000), 75--93.

[24]

Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas. 2010. Tracking learning detection. IEEE Trans. Pattern Anal. Mach. Intell. 34, 7, (2012), 1409--1402.

Digital Library

[25]

Georg Klein and David Murray. 2007. Parallel tracking and mapping for small AR workspaces. In Proceedings of the 6th IEEE and ACM Symposium on Mixed and Augmented Reality (ISMAR'07). 1--10.

Digital Library

[26]

Ravi Kondapalli and Ben-Zhen Sung. 2011. Daft datum—An interface for producing music through foot-based interaction. In Proceedings of the 7th International Workshop on Networking Issues in Multimedia Entertainment (NIME'11).

[27]

Zhihan Lu, Muhammad Sikandar Lal Khan, and Shafiq ur Réhman. 2013a. Hand and foot gesture interaction for handheld devices. In Proceedings of the 21st ACM International Conference on Multimedia (MM'13). ACM, New York, NY, 621--624.

Digital Library

[28]

Zhihan Lu, Shafiq ur Rehman, M. S. L. Khan, and Haibo Li. 2013b. Anaglyph 3D stereoscopic visualization of 2D video based on fundamental matrix. In Proceedings of the International Conference on Virtual Reality and Visualization (ICVRV). 305--308.

Digital Library

[29]

Zhihan Lu and Shafiq ur Réhman. 2013. Touch-less interaction smartphone on Go&excl;. In Proceedings of the SIGGRAPH Asia Posters (SA'13). ACM, New York, NY, Article 28, 1 page.

Digital Library

[30]

Bruce D. Lucas and Takeo Kanade. 1981. An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence - Volume 2 (IJCAI'81). Morgan Kaufmann Publishers Inc., San Francisco, CA, 674--679. http://dl.acm.org/citation.cfm&quest;id=1623264.1623280

Digital Library

[31]

Zhihan Lv. 2013. Wearable smartphone: Wearable hybrid framework for hand and foot gesture interaction on smartphone. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW'13). IEEE.

Digital Library

[32]

Zhihan Lv, Shengzhong Feng, Muhammad Sikandar Lal Khan, Shafiq ur Réhman, and Haibo Li. 2014. Foot motion sensing: Augmented game interface based on foot interaction for smartphone. In Proceedings of the CHI'14 Extended Abstracts on Human Factors in Computing Systems (CHI EA'14). ACM, New York, NY, 293--296.

Digital Library

[33]

Zhihan Lv, Alaa Halawani, Muhammad Sikandar Lal Khan, Shafiq ur Réhman, and Haibo Li. 2013. Finger in air: Touch-less interaction on smartphone. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia (MUM'13). ACM, New York, NY, Article 16, 4 pages.

Digital Library

[34]

Zhihan Lv and Shafiq ur Réhman. 2013. Multi-gesture based football game in smart phones. In Proceedings of the SIGGRAPH Asia Symposium on Mobile Graphics and Interactive Applications (SA'13). ACM, New York, NY, Article 20, 1 pages.

Digital Library

[35]

Pranav Mistry and Pattie Maes. 2009. SixthSense: A wearable gestural interface. In Proceedings of the ACM SIGGRAPH ASIA Sketches (SIGGRAPH ASIA'09). ACM, New York, NY, Article 11, 1 pages.

Digital Library

[36]

Pranav Mistry, Pattie Maes, and Liyan Chang. 2009. WUW - wear ur world: A wearable gestural interface. In Proceedings of the CHI'09 Extended Abstracts on Human Factors in Computing Systems (CHI EA'09). ACM, New York, NY, 4111--4116.

Digital Library

[37]

Volker Paelke, Christian Reimann, and Dirk Stichling. 2004. Foot-based mobile Interaction with games. In Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE'04). ACM, New York, NY, 321--324.

Digital Library

[38]

Srikumar Ramalingam, Suresh K. Lodha, and Peter Sturm. 2006. A generic structure-from-motion framework. Comput. Vision Image Understand. 103, 3 (September 2006), 218--280.

Digital Library

[39]

S. Réhman, A. Khan, and H. Li. 2012. Interactive feet for mobile immersive interaction. In Proceedings of the ACM International Workshop MobiVis Workshop at MobileHCI.

[40]

Nuttapol Sangsuriyachot and Masanori Sugimoto. 2012. Novel interaction techniques based on a combination of hand and foot gestures in tabletop environments. In Proceedings of the 10th Asia Pacific Conference on Computer Human Interaction.

Digital Library

[41]

Jeremy Scott, David Dearman, Koji Yatani, and Khai N. Truong. 2010. Sensing foot gestures from the pocket. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST'10). ACM, New York, NY, 199--208.

Digital Library

[42]

Ju-Hwan Seo, Jeong-Yean Yang, and Dong-Soo Kwon. 2012. Laser scanner based foot motion detection for intuitive robot user interface system. In Proceedings of the IEEE International Symposium on Robots and Human Interactive Communications (RO-MAN).

[43]

Tyler Simpson, Michel Gauthier, and Arthur Prochazka. 2010. Evaluation of tooth-click triggering and speech recognition in assistive technology for computer access. Neurorehabil. Neural Repair 24, 2 (February 2010), 188--194.

[44]

Khoa Nguyen Tran and Zhiyong Huang. 2007. Design and implementation of a built-in camera based user interface for mobile games. In Proceedings of the 5th International Conference on Computer Graphics and Interactive Techniques in Australia and Southeast Asia (GRAPHITE'07). ACM, New York, NY, 25--31.

Digital Library

[45]

Alex Ufkes and Mark Fiala. 2013. A markerless augmented reality system for mobile devices. In Proceedings of the International Conference on Computer and Robot Vision (CRV). 226--233.

Digital Library

[46]

S. ur Rehman, Li Liu, and Haibo Li. 2007. Lipless tracking and emotion estimation. In Proceedings of the 3rd International IEEE Conference on Signal-Image Technologies and Internet-Based System (SITIS'07). 768--774.

Digital Library

[47]

S. ur Rehman, Jiong Sun, Li Liu, and Haibo Li. 2008. Turn your mobile into the ball: Rendering live football game using vibration. IEEE Trans. Multimedia 10, 6 (2008), 1022--1033.

Digital Library

[48]

D. Valkov, F. Steinicke, G. Bruder, and K. Hinrichs. 2010. Traveling in 3D virtual environments with foot gestures and a multi-touch enabled WIM. In Proceedings of the International Conference on Virtual Reality.

[49]

Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg. 2008. Pose tracking from natural features on mobile phones. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR'08). IEEE Computer Society, 125--134.

Digital Library

[50]

Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and D. Schmalstieg. 2010. Real-time detection and tracking for augmented reality on mobile phones. IEEE Trans. Visual. Comput. Graphics 16, 3 (2010), 355--368.

Digital Library

[51]

Jingtao Wang, Shumin Zhai, and John Canny. 2006. Camera phone based motion sensing: Interaction techniques, applications and performance study. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST'06). ACM, New York, NY, 101--110.

Digital Library

[52]

Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics Bullet. 1, 6 (1945), 80--83.

[53]

Shahrouz Yousefi, Farid Abedan Kondori, and Haibo Li. 2011. 3D gestural interaction for stereoscopic visualization on mobile devices. In Proceedings of the 14th International Conference on Computer Analysis of Images and Patterns - Volume Part II (CAIP'11). Lecture Notes in Computer Science, vol. 6855. Springer-Verlag, Berlin, Heidelberg, 555--562. http://dl.acm.org/citation.cfm&quest;id=2044575.2044654

Digital Library

[54]

Shahrouz Yousefi, Farid Abedan Kondori, and Haibo Li. 2013. Experiencing real 3D gestural interaction with mobile devices. Pattern Recognition Letters 34, 8 (2013), 912--921. Computer Analysis of Images and Patterns.

Digital Library

Cited By

Hedlund MBogdan CMeixner GMatviienko A(2024)Rowing Beyond: Investigating Steering Methods for Rowing-based Locomotion in Virtual EnvironmentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642192(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642192
Arias-Flores HValencia-Aragón KSanchez-Gordón S(2024)Exploring the Effectiveness of Assistive Technology: A Preliminary Case Study Using Makey Makey, Tobii Eye Tracker, and Leap MotionExtended Reality10.1007/978-3-031-71704-8_3(32-42)Online publication date: 18-Sep-2024
https://doi.org/10.1007/978-3-031-71704-8_3
Shen JZhou XWu WWang LChen Z(2023)Worldwide Overview and Country Differences in Metaverse Research: A Bibliometric AnalysisSustainability10.3390/su1504354115:4(3541)Online publication date: 15-Feb-2023
https://doi.org/10.3390/su15043541
Show More Cited By

Index Terms

Multimodal Hand and Foot Gesture Interaction for Handheld Devices

Recommendations

Hand and foot gesture interaction for handheld devices
MM '13: Proceedings of the 21st ACM international conference on Multimedia

In this paper we present hand and foot based immersive multimodal interaction approach for handheld devices. A smart phone based immersive football game is designed as a proof of concept. Our proposed method combines input modalities (i.e. hand & foot) ...
Multimodal interaction systems: information and time features

Multimodal interaction systems combine visual information (involving images, text, sketches and so on) with voice, gestures and other modalities to provide flexible and powerful dialogue approaches, enabling users to choose one or more of the multiple ...
Extending chatterbot system into multimodal interaction framework with embodied contextual understanding
HRI '12: Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction

This work aims to realize multimodal interaction with embodied contextual understanding based on the simple chatterbot system. A system framework is proposed to integrate the dialogue system into a 3D simulation platform, SIGVerse to attain multimodal ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 11, Issue 1s

Special Issue on Multiple Sensorial (MulSeMedia) Multimodal Media : Advances and Applications

September 2014

260 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2675060

Editors:
Ralf Steinmetz
Technische Universität Darmstadt, Germany
,
Gheorghita Ghinea,
Christian Timmerer,
Weisi Lin,
Stephen Gulliver,
Zheng-Jun Zha
Institute of Intelligent Machines, Chinese Academy of Sciences
,
Lei Zhang
Bing Multimedia Search, Microsoft Corporation
,
Max Mühlhäuser
Department of Computer Science, Technische Universität Darmstadt
,
Alan F. Smeaton
Insight Centre for Data Analytics, Dublin City University

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2014

Accepted: 01 May 2014

Revised: 01 March 2014

Received: 01 November 2013

Published in TOMM Volume 11, Issue 1s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

115
Total Citations
View Citations
1,267
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)8

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hedlund MBogdan CMeixner GMatviienko A(2024)Rowing Beyond: Investigating Steering Methods for Rowing-based Locomotion in Virtual EnvironmentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642192(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642192
Arias-Flores HValencia-Aragón KSanchez-Gordón S(2024)Exploring the Effectiveness of Assistive Technology: A Preliminary Case Study Using Makey Makey, Tobii Eye Tracker, and Leap MotionExtended Reality10.1007/978-3-031-71704-8_3(32-42)Online publication date: 18-Sep-2024
https://doi.org/10.1007/978-3-031-71704-8_3
Shen JZhou XWu WWang LChen Z(2023)Worldwide Overview and Country Differences in Metaverse Research: A Bibliometric AnalysisSustainability10.3390/su1504354115:4(3541)Online publication date: 15-Feb-2023
https://doi.org/10.3390/su15043541
Freitas ASantos DLima RSantos CMeiguins B(2023)Pactolo Bar: An Approach to Mitigate the Midas Touch Problem in Non-Conventional InteractionSensors10.3390/s2304211023:4(2110)Online publication date: 13-Feb-2023
https://doi.org/10.3390/s23042110
Ahmed Khan FAbubakar Ibrahim A(2023)Non-Fungible Token based Smart Manufacturing to scale Industry 4.0 by using Augmented Reality, Deep Learning and Industrial Internet of ThingsInternational Journal on Perceptive and Cognitive Computing10.31436/ijpcc.v9i2.4079:2(62-72)Online publication date: 28-Jul-2023
https://doi.org/10.31436/ijpcc.v9i2.407
Xiao YLiu THan YLiu YWang Y(2023)Realtime Recognition of Dynamic Hand Gestures in Practical ApplicationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/356182220:2(1-17)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3561822
Müller FSchmitt DMatviienko ASchön DGünther SKosch TSchmitz M(2023)TicTacToes: Assessing Toe Movements as an Input ModalityProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580954(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580954
Yeo HWu EKim DLee JKim HOh STakagi LWoo WKoike HQuigley A(2023)OmniSense: Exploring Novel Input Sensing and Interaction Techniques on Mobile Device with an Omni-Directional CameraProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580747(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580747
Yue LZongxing LHui DChao JZiqiang LZhoujie L(2023)How to Achieve Human–Machine Interaction by Foot Gesture Recognition: A ReviewIEEE Sensors Journal10.1109/JSEN.2023.328521423:15(16515-16528)Online publication date: 1-Aug-2023
https://doi.org/10.1109/JSEN.2023.3285214
Xu SKaul CGe XMurray-Smith R(2023)Continuous Interaction with A Smart Speaker via Low-Dimensional Embeddings of Dynamic Hand PoseICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP49357.2023.10096097(1-5)Online publication date: 4-Jun-2023
https://doi.org/10.1109/ICASSP49357.2023.10096097
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents