research-article

Open access

WatchSense: On- and Above-Skin Input Sensing through a Wearable Depth Sensor

Authors:

Srinath Sridhar,

Anders Markussen,

Antti Oulasvirta,

Christian Theobalt,

Sebastian BoringAuthors Info & Claims

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

Pages 3891 - 3902

https://doi.org/10.1145/3025453.3026005

Published: 02 May 2017 Publication History

Abstract

This paper contributes a novel sensing approach to support on- and above-skin finger input for interaction on the move. WatchSense uses a depth sensor embedded in a wearable device to expand the input space to neighboring areas of skin and the space above it. Our approach addresses challenging camera-based tracking conditions, such as oblique viewing angles and occlusions. It can accurately detect fingertips, their locations, and whether they are touching the skin or hovering above it. It extends previous work that supported either mid-air or multitouch input by simultaneously supporting both. We demonstrate feasibility with a compact, wearable prototype attached to a user's forearm (simulating an integrated depth sensor). Our prototype---which runs in real-time on consumer mobile devices---enables a 3D input space on the back of the hand. We evaluated the accuracy and robustness of the approach in a user study. We also show how WatchSense increases the expressiveness of input by interweaving mid-air and multitouch for several interactive applications.

Supplementary Material

suppl.mov (pn4498.mp4)

Supplemental video

Download
13.73 MB

suppl.mov (pn4498p.mp4)

Supplemental video

Download
3.68 MB

References

[1]

Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1239--1248.

Digital Library

[2]

Yannick Bernaerts, Matthias Druwé, Sebastiaan Steensels, Jo Vermeulen, and Johannes Schöning. 2014. The Office Smartwatch: Development and Design of a Smartwatch App to Digitally Augment Interactions in an Office Environment. In Proceedings of the 2014 Companion Publication on Designing Interactive Systems (DIS Companion '14). ACM, New York, NY, USA, 41--44.

Digital Library

[3]

Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: multi-touch interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology. ACM, 201--204.

Digital Library

[4]

Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1e Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2527--2530.

Digital Library

[5]

Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 255--260.

Digital Library

[6]

Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 237--244.

Digital Library

[7]

Xiang 'Anthony' Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. Air+Touch: Interweaving Touch & In-air Gestures. In Proc. of UIST '14. ACM, New York, NY, USA, 519--525.

[8]

Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24, 5 (2002), 603--619.

Digital Library

[9]

Antonio Criminisi and Jamie Shotton. 2013. Decision forests for computer vision and medical image analysis. Springer Science & Business Media.

[10]

David A Forsyth and Jean Ponce. 2002. Computer vision: a modern approach. Prentice Hall Professional Technical Reference.

Digital Library

[11]

Sean Gustafson, Daniel Bierwirth, and Patrick Baudisch. 2010. Imaginary Interfaces: Spatial Interaction with Empty Hands and Without Visual Feedback. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 3--12.

Digital Library

[12]

Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary Phone: Learning Imaginary Interfaces by Transferring Spatial Memory from a Familiar Device. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 283--292.

Digital Library

[13]

Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 441--450.

Digital Library

[14]

Chris Harrison and Scott E. Hudson. 2009. Abracadabra: Wireless, High-precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, New York, NY, USA, 121--124.

Digital Library

[15]

Khalad Hasan, David Ahlström, and Pourang Irani. 2013. Ad-binning: Leveraging Around Device Space for Storing, Browsing and Retrieving Mobile Device Content. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 899--908.

Digital Library

[16]

Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1247--1256.

Digital Library

[17]

Brett Jones, Rajinder Sodhi, David Forsyth, Brian Bailey, and Giuliano Maciocci. 2012. Around Device Interaction for Multiscale Navigation. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '12). ACM, New York, NY, USA, 83--92.

Digital Library

[18]

Cem Keskin, Furkan Kraç, Yunus Emre Kara, and Lale Akarun. 2013. Real time hand pose estimation using depth sensors. In Consumer Depth Cameras for Computer Vision. Springer, 119--137.

[19]

David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 167--176.

Digital Library

[20]

Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A Wireless Contact-free Gesture Based Wrist Interface. In Proceedings of the 2007 11th IEEE International Symposium on Wearable Computers (ISWC '07). IEEE Computer Society, Washington, DC, USA, 1--8.

Digital Library

[21]

Sven Kratz and Michael Rohs. 2009. HoverFlow: Expanding the Design Space of Around-device Interaction. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, New York, NY, USA, Article 4, 8 pages.

Digital Library

[22]

Seungyon Claire Lee, Bohao Li, and Thad Starner. 2011. AirTouch: Synchronizing In-air Hand Gesture and On-body Tactile Feedback to Augment Mobile Gesture Interaction. In Proceedings of the 2011 15th Annual International Symposium on Wearable Computers (ISWC '11). IEEE Computer Society, Washington, DC, USA, 3--10.

Digital Library

[23]

Jhe-Wei Lin, Chiuan Wang, Yi Yao Huang, Kuan-Ting Chou, Hsuan-Yu Chen, Wei-Luan Tseng, and Mike Y. Chen. 2015. BackHand: Sensing Hand Gestures via Back of the Hand. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 557--564.

Digital Library

[24]

Simon Mayer and Gábor Sörös. 2014. User Interface Beaming Seamless Interaction with Smart Things using Personal Wearable Computers. In Proceedings of the 11th International Conference on Wearable and Implantable Body Sensor Networks (BSN 2014). Zurich, Switzerland, 46--49.

Digital Library

[25]

Kei Nakatsuma, Hiroyuki Shinoda, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2011. Touch Interface on Back of the Hand. In ACM SIGGRAPH 2011 Emerging Technologies (SIGGRAPH '11). ACM, New York, NY, USA, Article 19, 1 pages.

Digital Library

[26]

Jerome Pasquero, Scott J. Stobbe, and Noel Stonehouse. 2011. A Haptic Wristwatch for Eyes-free Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 3257--3266.

Digital Library

[27]

Simon T. Perrault, Eric Luecolinet, James Eagan, and Yves Guiard. 2013. Watchit: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1451--1460.

Digital Library

[28]

Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim Christoph Rhemann Ido Leichter, Alon Vinnikov Yichen Wei, Daniel Freedman Pushmeet Kohli Eyal Krupka, Andrew Fitzgibbon, and Shahram Izadi. 2015. Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI, Vol. 8.

Digital Library

[29]

Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. 2013. Real-time human pose recognition in parts from single depth images. Commun. ACM 56, 1 (2013), 116--124.

Digital Library

[30]

Jie Song, Fabrizio Pece, Gábor Sörös, Marion Koelle, and Otmar Hilliges. 2015. Joint Estimation of 3D Hand Position and Gestures from Monocular Video for Mobile Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3657--3660.

Digital Library

[31]

Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. 2015. Fast and Robust Hand Tracking Using Detection-Guided Optimization. In Proceedings of Computer Vision and Pattern Recognition (CVPR). 9. http://handtracker.mpi-inf.mpg.de/projects/ FastHandTracker/

[32]

Paul Strohmeier. DIY IR Sensors for Augmenting Objects and Human Skin. In Proc. of Augmented Human International Conference '15. ACM, New York, NY, USA, 181--182.

Digital Library

[33]

Cheng-Yao Wang, Wei-Chen Chu, Po-Tsung Chiu, Min-Chieh Hsiu, Yih-Harn Chiang, and Mike Y. Chen. 2015a. PalmType: Using Palms As Keyboards for Smart Glasses. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 153--160.

Digital Library

[34]

Cheng-Yao Wang, Min-Chieh Hsiu, Po-Tsung Chiu, Chiao-Hui Chang, Liwei Chan, Bing-Yu Chen, and Mike Y. Chen. 2015b. PalmGesture: Using Palms As Gesture Interfaces for Eyes-free Input. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 217--226.

Digital Library

[35]

David Way and Joseph Paradiso. 2014. A Usability User Study Concerning Free-Hand Microgesture and Wrist-Worn Sensors. In Proceedings of the 2014 11th International Conference on Wearable and Implantable Body Sensor Networks (BSN '14). IEEE Computer Society, Washington, DC, USA, 138--142.

Digital Library

[36]

Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2991--3000.

Digital Library

[37]

Martin Weigel, Vikram Mehta, and Jürgen Steimle. 2014. More Than Touch: Understanding How People Use Skin As an Input Surface for Mobile Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 179--188.

Digital Library

[38]

Andrew D Wilson. 2006. Robust computer vision-based detection of pinching for one and two-handed gesture input. In Proceedings of the 19th annual ACM symposium on User interface software and technology. ACM, 255--258.

Digital Library

[39]

Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 193--196.

Digital Library

[40]

Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, and Aaron Quigley. 2016. WatchMI: Pressure Touch, Twist and Pan Gesture Input on Unmodified Smartwatches. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '16). ACM, New York, NY, USA, 394--399.

Digital Library

[41]

Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body As an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1491--1503.

Digital Library

Cited By

Hindelang MWecker HBiedermann TZink A(2024)Continuously monitoring the human machine? – A cross-sectional study to assess the acceptance of wearables in GermanyHealth Informatics Journal10.1177/1460458224126060730:2Online publication date: 20-Jun-2024
https://doi.org/10.1177/14604582241260607
Darbar RHu XYan XWei YLiang HXu WSarcar S(2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682084
Mollyn VHarrison C(2024)EgoTouch: On-Body Touch Input Using AR/VR Headset CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676455(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676455
Show More Cited By

Index Terms

WatchSense: On- and Above-Skin Input Sensing through a Wearable Depth Sensor
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

SkinTrack is a wearable system that enables continuous touch tracking on the skin. It consists of a ring, which emits a continuous high frequency AC signal, and a sensing wristband with multiple electrodes. Due to the phase delay inherent in a high-...
Space Boards: Combining Tangible Interfaces with the Surrounding Space via RGB-D Cameras

This paper introduces a novel take on a well-known user interface that combines the advantages of a number of new technologies. In particular, it presents a new tangible interface with an interactive surrounding space. It demonstrates the technology in ...
Vision-based projected tabletop interface for finger interactions
HCI'07: Proceedings of the 2007 IEEE international conference on Human-computer interaction

We designed and implemented a vision-based projected table-top interface for finger interaction. The system offers a simple and quick setup and economic design. The projection onto the tabletop provides more comfortable and direct viewing for users, and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

May 2017

7138 pages

ISBN:9781450346559

DOI:10.1145/3025453

General Chairs:
Gloria Mark
University of California Irvine
,
Susan Fussell
Cornell University
,
Program Chairs:
Cliff Lampe
University of Michigan
,
m.c. schraefel
University of Southampton
,
Juan Pablo Hourcade
University of Iowa
,
Caroline Appert
Université Paris-Sud
,
Daniel Wigdor
University of Toronto

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Research Council

Conference

CHI '17

Sponsor:

SIGCHI

CHI '17: CHI Conference on Human Factors in Computing Systems

May 6 - 11, 2017

Colorado, Denver, USA

Acceptance Rates

CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
1,939
Total Downloads

Downloads (Last 12 months)256
Downloads (Last 6 weeks)45

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hindelang MWecker HBiedermann TZink A(2024)Continuously monitoring the human machine? – A cross-sectional study to assess the acceptance of wearables in GermanyHealth Informatics Journal10.1177/1460458224126060730:2Online publication date: 20-Jun-2024
https://doi.org/10.1177/14604582241260607
Darbar RHu XYan XWei YLiang HXu WSarcar S(2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682084
Mollyn VHarrison C(2024)EgoTouch: On-Body Touch Input Using AR/VR Headset CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676455(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676455
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Kyu AMao HZhu JGoel MAhuja K(2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642663
Nath PDelamare WHasan K(2024)PalmSpaceInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103219184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103219
Waghmare ABoldu RWhitmire EKienzle W(2023)OptiRing: Low-Resolution Optical Sensing for Subtle Thumb-to-Index Micro-InteractionsProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614538(1-13)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614538
Chen WWang ZQuan PPeng ZLin SSrivastava MMatusik WStankovic J(2023)Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese AdaptationProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606794(1-14)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606794
Rateau HRekik YLank E(2023)Ether-Mark: An Off-Screen Marking Menu For Mobile DevicesProceedings of the 25th International Conference on Multimodal Interaction10.1145/3577190.3614150(224-233)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3577190.3614150
Tran PGadepalli PLee JNittala A(2023)Augmenting On-Body Touch Input with Tactile Feedback Through Fingernail HapticsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581473(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581473
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents