research-article

FingerIO: Using Active Sonar for Fine-Grained Finger Tracking

Authors:

Rajalakshmi Nandakumar,

Shyamnath GollakotaAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 1515 - 1525

https://doi.org/10.1145/2858036.2858580

Published: 07 May 2016 Publication History

Abstract

We present fingerIO, a novel fine-grained finger tracking solution for around-device interaction. FingerIO does not require instrumenting the finger with sensors and works even in the presence of occlusions between the finger and the device. We achieve this by transforming the device into an active sonar system that transmits inaudible sound signals and tracks the echoes of the finger at its microphones. To achieve sub-centimeter level tracking accuracies, we present an innovative approach that use a modulation technique commonly used in wireless communication called Orthogonal Frequency Division Multiplexing (OFDM). Our evaluation shows that fingerIO can achieve 2-D finger tracking with an average accuracy of 8 mm using the in-built microphones and speaker of a Samsung Galaxy S4. It also tracks subtle finger motion around the device, even when the phone is in the pocket. Finally, we prototype a smart watch form-factor fingerIO device and show that it can extend the interaction space to a 0.5×0.25 m2 region on either side of the device and work even when it is fully occluded from the finger.

Supplementary Material

MP4 File (p1515-nandakumar.mp4)

Download
202.59 MB

References

[1]

Adafruit. https://www.adafruit.com/products/1063.

[2]

Apple Watch - Guided Tour: Phone Calls. https://www.youtube.com/watch?v=Zj5KisMVv8.

[3]

Chirp Microsystems. http://www.chirpmicro.com/technology.html.

[4]

A MimioTeach Interaction Whiteboard. http://www.mimio.com/en-NA/Products/MimioTeach-Interactive-Whiteboard.aspx.

[5]

Adib, F., Kabelac, Z., Katabi, D., and Miller, R. C. 3D Tracking via Body Radio Reflections. NSDI 2014, 317--329.

Digital Library

[6]

Aumi, M. T. I., Gupta, S., Goel, M., Larson, E., and Patel, S. DopLink: Using the Doppler Effect for Multi-device Interaction. UbiComp 2013, 583--586.

Digital Library

[7]

Boleskei, H. Principles of MIMO-OFDM wireless systems. 2004.

[8]

Braun, A., Krepp, S., and Kuijper, A. Acoustic Tracking of Hand Activities on Surfaces. WOAR 2015, 1--5.

Digital Library

[9]

Butler, A., Izadi, S., and Hodges, S. SideSight: Multi-"Touch" Interaction Around Small Devices. UIST 2008, 201--204.

Digital Library

[10]

Chan, L., Liang, R.-H., Tsai, M.-C., Cheng, K.-Y., Su, C.-H., Chen, M. Y., Cheng, W.-H., and Chen, B.-Y. FingerPad: Private and Subtle Interaction Using Fingertips. UIST 2013, 255--260.

Digital Library

[11]

Chen, K.-Y., Ashbrook, D., Goel, M., Lee, S.-H., and Patel, S. AirLink: Sharing Files Between Multiple Devices Using In-air Gestures. UbiComp 2014, 565--569.

Digital Library

[12]

Chen, K.-Y., Lyons, K., White, S., and Patel, S. uTrack: 3D Input Using Two Magnetic Sensors. UIST 2013, 237--244.

Digital Library

[13]

Goel, M., Lee, B., Islam Aumi, M. T., Patel, S., Borriello, G., Hibino, S., and Begole, B. Surface Link: Using Inertial and Acoustic Sensing to Enable Multi-device Interaction on a Surface. CHI 2014, 1387--1396.

Digital Library

[14]

Google. Project Soli. https://www.youtube.com/watch?v=_Zj5KisMVv8.

[15]

Gupta, S., Morris, D., Patel, S., and Tan, D. SoundWave: Using the Doppler Effect to Sense Gestures. CHI 2012, 1911--1914.

Digital Library

[16]

Heiskala, J., and Terry, J. OFDM Wireless LANs: A Theoretical and Practical Guide. Sams publishing, 2001.

Digital Library

[17]

Huang, W., Xiong, Y., Li, X.-Y., Lin, H., Mao, X., Yang, P., and Liu, Y. Shake and walk: Acoustic direction finding and fine-grained indoor localization using smartphones. INFOCOM 2014, 370--278.

[18]

Kellogg, B., Talla, V., and Gollakota, S. Bringing Gesture Recognition to All Devices. NSDI 2014, 303--316.

Digital Library

[19]

Khyam, M., Alam, M., Lambert, A., Benson, C., and Pickering, M. High precision multiple ultrasonic transducer positioning using a robust optimization approach. ISSPIT 2013, 192--197.

[20]

Khyam, M., Alam, M., and Pickering, M. OFDM based low-complexity time of arrival estimation in active sonar. OCEANS 2014, 1--5.

[21]

Kienzle, W., and Hinckley, K. LightRing: Always-available 2D Input on Any Surface. UIST 2014, 157--160.

Digital Library

[22]

Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. UIST 2012, 167--176.

Digital Library

[23]

Kratz, S., and Rohs, M. HoverFlow: Expanding the Design Space of Around-device Interaction. MobileHCI 2009, 1--8.

Digital Library

[24]

Liu, J., Wang, Y., Kar, G., Chen, Y., Yang, J., and Gruteser, M. Snooping Keystrokes with Mm-level Audio Ranging on a Single Phone. MobiCom 2015, 142--154.

Digital Library

[25]

MacNeish. The Intersections of Two Conic Sections with a Common Focus. The American Mathematical Monthly 28, 6/7, 260--262.

[26]

Nandakumar, R., Chinatalapudi, K., Padmanaban, V., and Venkatesan, R. Dhwani: Secure Peer-to-Peer Acoustic NFC. Sigcomm 2013 2013.

Digital Library

[27]

Nandakumar, R., Gollakota, S., and Watson, N. Contactless Sleep Apnea Detection on Smartphones. Mobisys 2015, 45--57.

Digital Library

[28]

Ogata, M., Sugiura, Y., Osawa, H., and Imai, M. iRing: Intelligent Ring Using Infrared Reflection. UIST 2012, 131--136.

Digital Library

[29]

Priyantha, N. B., Chakraborty, A., and Balakrishnan, H. The Cricket Location-support System. Mobicom 2000, 32--43.

Digital Library

[30]

Proakis, J., and Salehi, M. Digital Communications. McGraw-hill, 2007.

[31]

Przybyla, R., Tang, H.-Y., Guedes, A., Shelton, S., Horsley, D., and Boser, B. 3D Ultrasonic Rangefinder on a Chip. IEEE Journal of Solid-State Circuits 2015, 320--334.

[32]

Pu, Q., Gupta, S., Gollakota, S., and Patel, S. Whole-home Gesture Recognition Using Wireless Signals. Mobicom 2013, 27--38.

Digital Library

[33]

Reju, V., Khong, A., and Sulaiman, A. Localization of Taps on Solid Surfaces for Human-Computer Touch Interfaces. IEEE Trans. on Multimedia 2013, 1365--1376.

Digital Library

[34]

Saponas, T. S., Harrison, C., and Benko, H. PocketTouch: Through-fabric Capacitive Touch Input. UIST 2011, 303--308.

Digital Library

[35]

Song, J., Soros, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. In-air Gestures Around Unmodified Mobile Devices. UIST 2014, 319--329.

Digital Library

[36]

Sun, L., Sen, S., Koutsonikolas, D., and Kim, K.-H. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. Mobicom 2015, 77--89.

Digital Library

[37]

Sun, Z., Purohit, A., Bose, R., and Zhang, P. Spartacus: Spatially-aware Interaction for Mobile Devices Through Energy-efficient Audio Sensing. MobiSys 2013, 263--276.

Digital Library

[38]

Wang, J., Zhao, K., Zhang, X., and Peng, C. Ubiquitous Keyboard for Small Mobile Devices: Harnessing Multipath Fading for Fine-grained Keystroke Localization. MobiSys 2014, 14--27.

Digital Library

[39]

Xiao, R., Lew, G., Marsanico, J., Hariharan, D., Hudson, S., and Harrison, C. Toffee: Enabling Ad Hoc, Around-device Interaction with Acoustic Time-of-arrival Correlation. MobileHCI 2014, 67--76.

Digital Library

[40]

Yang, X.-D., Grossman, T., Wigdor, D., and Fitzmaurice, G. Magic Finger: Always-available Input Through Finger Instrumentation. UIST 2012, 147--156.

Digital Library

[41]

Yang, X.-D., Hasan, K., Bruce, N., and Irani, P. Surround-see: Enabling Peripheral Vision on Smartphones During Active Use. UIST 2013, 291--300.

Digital Library

[42]

Yun, S., Chen, Y.-C., and Qiu, L. Turning a Mobile Device into a Mouse in the Air. Mobisys 2015, 15--29.

Digital Library

[43]

Zhao, C., Chen, K.-Y., Aumi, M. T. I., Patel, S., and Reynolds, M. S. SideSwipe: Detecting In-air Gestures Around Mobile Devices Using Actual GSM Signal. UIST 2014, 527--534.

Digital Library

Cited By

Joo JKoh JLee H(2024)Hand Gesture Recognition Using Ultrasonic Array with Machine LearningSensors10.3390/s2420676324:20(6763)Online publication date: 21-Oct-2024
https://doi.org/10.3390/s24206763
Wang ZWang YLiao MSun YWang SSun XShi XKang YTian MBao TLu R(2024)FireSonic: Design and Implementation of an Ultrasound Sensing-Based Fire Type Identification SystemSensors10.3390/s2413436024:13(4360)Online publication date: 5-Jul-2024
https://doi.org/10.3390/s24134360
Mahmud SParikh VLiang QLi KZhang RAjit AGunda VAgarwal DGuimbretiere FZhang C(2024)ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the BodyProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997528:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699752
Show More Cited By

Index Terms

FingerIO: Using Active Sonar for Fine-Grained Finger Tracking
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

The natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...
MagiMusic: using embedded compass (magnetic) sensor for touch-less gesture based interaction with digital music instruments in mobile devices
TEI '11: Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction

Playing musical instruments such as chordophones, percussions and keyboard types accompany with harmonic interaction of player's hand with the instruments. In this work, we present a novel approach that enables the user to imitate the music playing ...
MagiTact: interaction with mobile devices based on compass (magnetic) sensor
IUI '10: Proceedings of the 15th international conference on Intelligent user interfaces

In this work, we present a new technique for efficient use of 3D space around a mobile device for interaction with the device. Around Device Interaction (ADI) enables extending interaction space of small mobile and tangible devices beyond their physical ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

Google Faculty Award
NSF

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

330
Total Citations
View Citations
3,182
Total Downloads

Downloads (Last 12 months)353
Downloads (Last 6 weeks)29

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Joo JKoh JLee H(2024)Hand Gesture Recognition Using Ultrasonic Array with Machine LearningSensors10.3390/s2420676324:20(6763)Online publication date: 21-Oct-2024
https://doi.org/10.3390/s24206763
Wang ZWang YLiao MSun YWang SSun XShi XKang YTian MBao TLu R(2024)FireSonic: Design and Implementation of an Ultrasound Sensing-Based Fire Type Identification SystemSensors10.3390/s2413436024:13(4360)Online publication date: 5-Jul-2024
https://doi.org/10.3390/s24134360
Mahmud SParikh VLiang QLi KZhang RAjit AGunda VAgarwal DGuimbretiere FZhang C(2024)ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the BodyProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997528:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699752
Zhou JLi YWang YDing DLu YChen YXue G(2024)Visar: Projecting Virtual Sound Spots for Acoustic Augmented Reality Using Air NonlinearityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785468:3(1-30)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678546
Sun TZhao YXie WLi JMa YZhang J(2024)EyeGesener: Eye Gesture Listener for Smart Glasses Interaction Using Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785418:3(1-28)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678541
Yuan KIbrahim MSong YDeng GNerone RVijayan SGadre AKumar S(2024)ToMoBrush: Exploring Dental Health Sensing Using a Sonic ToothbrushProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785058:3(1-27)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678505
Wang SWang XJiang WMiao CCao QWang HSun KXue HSu L(2024)Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic SignalsACM Transactions on Sensor Networks10.1145/367712220:5(1-32)Online publication date: 26-Aug-2024
https://dl.acm.org/doi/10.1145/3677122
Kim JJung HOakley I(2024)VibraHand: In-Hand Superpower Enabling Spying, Precognition, and TelekinesisAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686728(1-3)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686728
Matulic FKashima TBeker DSuzuo DFujiwara HVogel D(2024)Above-Screen Fingertip Tracking and Hand Representation for Precise Touch Input with a Phone in Virtual RealityProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670961(1-15)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670961
Garg NGhosh ARoy NShu YLiu JTan RHe YChen J(2024)LiTEfoot: Ultra-low-power Localization using Ambient Cellular SignalsProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699356(535-548)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699356
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents