UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable Sensors
Article No.: 230, Pages 1 - 25
Abstract
Wearable devices allow quick and convenient interactions for controlling mobile computers. However, these interactions are often device-dependent, and users cannot control devices in a way they are familiar with if they do not wear the same wearable device. This paper proposes a new method, UnifiedSense, to enable device-dependent gestures even when the device that detects such gestures is missing by utilizing sensors on other wearable devices. UnifiedSense achieves this without explicit gesture training for different devices, by training its recognition model while users naturally perform gestures. The recognizer uses the gestures detected on the primary device (i.e., a device that reliably detects gestures) as labels for training samples and collects sensor data from all other available devices on the user. We conducted a technical evaluation with data collected from 15 participants with four types of wearable devices. It showed that UnifiedSense could correctly recognize 5 gestures (5 gestures × 5 configurations) with an accuracy of 90.9% (SD = 1.9%) without the primary device present.
Supplementary Material
Supplemental video
- Download
- 189.32 MB
References
[1]
Sunggeun Ahn, Seongkook Heo, and Geehyuk Lee. 2017. Typing on a smartwatch for smart glasses. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces. 201--209.
[2]
Ahmad Akl, Chen Feng, and Shahrokh Valaee. 2011. A Novel Accelerometer-Based Gesture Recognition System. IEEE Transactions on Signal Processing 59, 12 (Dec. 2011), 6197--6205. https://doi.org/10.1109/TSP.2011.2165707 Conference Name: IEEE Transactions on Signal Processing.
[3]
Brian Amento, Will Hill, and Loren Terveen. 2002. The sound of one hand: A wrist-mounted bio-acoustic fingertip gesture interface. In CHI'02 Extended Abstracts on Human Factors in Computing Systems. 724--725.
[4]
Apple Inc. 2021. Apple -- iOS 15 -- Continuity. https://www.macrumors.com/guide/universal-control/. Online; accessed Aug 2022.
[5]
Daniel L Ashbrook. 2010. Enabling mobile microinteractions. Georgia Institute of Technology.
[6]
Md Aashikur Rahman Azim, Adil Rahman, and Seongkook Heo. 2022. Over-The-Shoulder Training Between Redundant Wearable Sensors for Unified Gesture Interactions. In Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1--3.
[7]
Frederik Brudy, Christian Holz, Roman Rädle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-device taxonomy: Survey, opportunities and challenges of interactions spanning across multiple devices. In Proceedings of the 2019 chi conference on human factors in computing systems. 1--28.
[8]
Baptiste Caramiaux, Marco Donnarumma, and Atau Tanaka. 2015. Understanding gesture expressivity through muscle sensing. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 6 (2015), 1--26.
[9]
Matthew Chalmers and Areti Galani. 2004. Seamful interweaving: heterogeneity in the theory and design of interactive systems. In Proceedings of the 5th conference on Designing interactive systems: processes, practices, methods, and techniques. 243--252.
[10]
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785--794.
[11]
Xiang "Anthony' Chen, Tovi Grossman, Daniel J Wigdor, and George Fitzmaurice. 2014. Duet: exploring joint interactions on a smart phone and a smart watch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 159--168.
[12]
Pei-Yu Chi and Yang Li. 2015. Weave: Scripting cross-device wearable interaction. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 3923--3932.
[13]
Jungmin Chung, Changhoon Oh, SoHyun Park, and Bongwon Suh. 2018. PairRing: a ring-shaped rotatable smartwatch controller. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1--6.
[14]
Andrew Crossan, Stephen Brewster, and Alexander Ng. 2010. Foot tapping for mobile interaction. Proceedings of HCI 2010 24 (2010), 418--422.
[15]
Andrew Crossan, Mark McGill, Stephen Brewster, and Roderick Murray-Smith. 2009. Head tilting for interaction in mobile contexts. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1--10.
[16]
David Dearman and Jeffery S Pierce. 2008. It's on my other computer! computing with multiple devices. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems. 767--776.
[17]
Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, Honolulu Hawaii USA, 161--166. https://doi.org/10.1145/2642918.2647396
[18]
Angus Dempster, François Petitjean, and Geoffrey I Webb. 2020. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 5 (2020), 1454--1495.
[19]
Travis Deyle, Szabolcs Palinko, Erika Shehan Poole, and Thad Starner. 2007. Hambone: A bio-acoustic gesture interface. In 2007 11th IEEE International Symposium on Wearable Computers. IEEE, 3--10.
[20]
Paul Dourish. 2004. Where the action is: the foundations of embodied interaction. MIT press.
[21]
Facebook's AI Research lab. 2022. PyTorch Manual: The developer's perspective. https://pytorch.org/. Online; accessed Aug 2022.
[22]
Steven Feiner and Ari Shamash. 1991. Hybrid user interfaces: Breeding virtually bigger interfaces for physically smaller computers. In Proceedings of the 4th annual ACM symposium on User interface software and technology. 9--17.
[23]
Fitbit Inc. 2022. Fitbit Versa. https://www.fitbit.com/global/us/products/smartwatches/versa3. Online; accessed Aug 2022.
[24]
Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems 32 (2019).
[25]
Rui Fukui, Masahiko Watanabe, Tomoaki Gyota, Masamichi Shimosaka, and Tomomasa Sato. 2011. Hand shape classification with a wrist contour sensor: development of a prototype device. In Proceedings of the 13th international conference on Ubiquitous computing. 311--314.
[26]
Marcus Georgi, Christoph Amma, and Tanja Schultz. 2015. Recognizing Hand and Finger Gestures with IMU based Motion and EMG based Muscle Activity Sensing. In Biosignals. Citeseer, 99--108.
[27]
Bogdan-Florin Gheran, Radu-Daniel Vatavu, and Jean Vanderdonckt. 2018. Ring x2: Designing gestures for smart rings using temporal calculus. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems. 117--122.
[28]
Nicholas Gillian and Joseph A Paradiso. 2014. The gesture recognition toolkit. The Journal of Machine Learning Research 15, 1 (2014), 3483--3487.
[29]
Jun Gong, Xing-Dong Yang, and Pourang Irani. 2016. WristWhirl: One-handed Continuous Smartwatch Input using Wrist Gestures. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo Japan, 861--872. https://doi.org/10.1145/2984511.2984563
[30]
Jun Gong, Yang Zhang, Xia Zhou, and Xing-Dong Yang. 2017. Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 553--563. https://doi.org/10.1145/3126594.3126615
[31]
Google Inc. 2014. Android Wear: The developer's perspective. https://youtu.be/G7tXr-w35UA. Online; accessed Aug 2022.
[32]
Google Inc. 2022. Google Glass. https://www.google.com/glass/start/. Online; accessed Aug 2022.
[33]
Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New Orleans LA USA, 1059--1070. https://doi.org/10.1145/3332165.3347947
[34]
Nils Y Hammerla and Thomas Plötz. 2015. Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 1041--1051.
[35]
Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems. 453--462.
[36]
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016).
[37]
Seongkook Heo, Michelle Annett, Benjamin J Lafreniere, Tovi Grossman, and George W Fitzmaurice. 2017. No Need to Stop What You're Doing: Exploring No-Handed Smartwatch Interaction. In Graphics Interface. 107--114.
[38]
Ken Hinckley. 2003. Synchronous gestures for multiple persons and computers. In Proceedings of the 16th annual ACM symposium on User interface software and technology. 149--158.
[39]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.
[40]
Jonggi Hong, Seongkook Heo, Poika Isokoski, and Geehyuk Lee. 2015. SplitBoard: A simple split soft keyboard for wristwatch-sized touch screens. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1233--1236.
[41]
Fang Hu, Peng He, Songlin Xu, Yin Li, and Cheng Zhang. 2020. FingerTrak: Continuous 3D hand pose tracking by deep learning hand silhouettes captured by miniature thermal cameras on wrist. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1--24.
[42]
Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. Digitspace: Designing thumb-to-fingers touch interfaces for one-handed and eyes-free interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1526--1537.
[43]
Yasha Iravantchi, Mayank Goel, and Chris Harrison. 2019. BeamBand: Hand gesture sensing with ultrasonic beamforming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--10.
[44]
Yasha Iravantchi, Yang Zhang, Evi Bernitsas, Mayank Goel, and Chris Harrison. 2019. Interferi: Gesture Sensing using On-Body Acoustic Interferometry. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland Uk, 1--13. https://doi.org/10.1145/3290605.3300506
[45]
Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul Lukowicz, and Andreas Bulling. 2014. In the blink of an eye: combining head motion and eye blink frequency for activity recognition with google glass. In Proceedings of the 5th augmented human international conference. 1--4.
[46]
Brian Kenji Iwana and Seiichi Uchida. 2021. An empirical survey of data augmentation for time series classification with neural networks. Plos one 16, 7 (2021), e0254841.
[47]
Lei Jing, Zixue Cheng, Yinghui Zhou, Junbo Wang, and Tongjun Huang. 2013. Magic Ring: a self-contained gesture input device on finger. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Multimedia - MUM'13. ACM Press, Luleå, Sweden, 1--4. https://doi.org/10.1145/2541831.2541875
[48]
Wolf Kienzle and Ken Hinckley. 2014. LightRing: always-available 2D input on any surface. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 157--160.
[49]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 167--176.
[50]
Minwoo Kim, Jaechan Cho, Seongjoo Lee, and Yunho Jung. 2019. IMU sensor-based hand gesture recognition for human-machine interfaces. Sensors 19, 18 (2019), 3827.
[51]
Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Ijcai, Vol. 14. Montreal, Canada, 1137--1145.
[52]
A.H.F. Lam, W.J. Li, Yunhui Liu, and Ning Xi. 2002. MIDS: micro input devices system using MEMS sensors. In IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 2. 1184--1189 vol.2. https://doi.org/10.1109/IRDS.2002.1043893
[53]
Gierad Laput and Chris Harrison. 2019. Sensing fine-grained hand activity with smartwatches. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--13.
[54]
Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo Japan, 321--333. https://doi.org/10.1145/2984511.2984582
[55]
Juyoung Lee, Shaurye Aggarwal, Jason Wu, Thad Starner, and Woontack Woo. 2019. SelfSync: exploring selfsynchronous body-based hotword gestures for initiating interaction. In Proceedings of the 23rd International Symposium on Wearable Computers. 123--128.
[56]
Ho-Man Colman Leung, Chi-Wing Fu, and Pheng-Ann Heng. 2018. TwistIn: Tangible authentication of smart devices via motion co-analysis with a smartwatch. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 1--24.
[57]
Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics 35, 4 (July 2016), 1--19. https://doi.org/10.1145/2897824.2925953
[58]
Guanhong Liu, Yizheng Gu, Yiwen Yin, Chun Yu, Yuntao Wang, Haipeng Mi, and Yuanchun Shi. 2020. Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (June 2020), 1--23. https://doi.org/10.1145/3397308
[59]
Jiayang Liu, Zhen Wang, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2008. uWave: Accelerometer-based personalized gesture recognition. TR0630-08, Rice University and Motorola Labs, Houston, Texas (2008).
[60]
Christian Loclair, Sean Gustafson, and Patrick Baudisch. 2010. PinchWatch: a wearable device for one-handed microinteractions. In Proc. MobileHCI, Vol. 10. Citeseer.
[61]
Simon Mayer and Gábor Sörös. 2014. User Interface Beaming--Seamless Interaction with Smart Things Using Personal Wearable Computers. In 2014 11th International Conference on Wearable and Implantable Body Sensor Networks Workshops. IEEE, 46--49.
[62]
Jess McIntosh, Asier Marzo, and Mike Fraser. 2017. SensIR: Detecting Hand Gestures with a Wearable Bracelet using Infrared Transmission and Reflection. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, Québec City QC Canada, 593--597. https://doi.org/10.1145/3126594.3126604
[63]
Jess McIntosh, Asier Marzo, and Mike Fraser. 2017. Sensir: Detecting hand gestures with a wearable bracelet using infrared transmission and reflection. In Proceedings of the 30th annual ACM symposium on user interface software and technology. 593--597.
[64]
Jess McIntosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. Echoflex: Hand gesture recognition using ultrasound imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1923--1934.
[65]
Stephen J Mckenna and Kenny Morrison. 2004. A comparison of skin history and trajectory-based representation schemes for the recognition of user-specified gestures. Pattern Recognition 37, 5 (2004), 999--1009.
[66]
Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TaplD: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 519--528.
[67]
Chulhong Min, Seungwoo Kang, Chungkuk Yoo, Jeehoon Cha, Sangwon Choi, Younghan Oh, and Junehwa Song. 2015. Exploring current practices for battery use and management of smartwatches. In Proceedings of the 2015 ACM international symposium on wearable computers. 11--18.
[68]
Vivian Genaro Motti and Kelly Caine. 2015. Micro interactions and multi dimensional graphical user interfaces in the design of wrist worn wearables. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 59. SAGE Publications Sage CA: Los Angeles, CA, 1712--1716.
[69]
Miguel A Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of pre-designed and user-defined gesture sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing systems. 1099--1108.
[70]
Shahriar Nirjon, Jeremy Gummeson, Dan Gelb, and Kyu-Han Kim. 2015. TypingRing: A Wearable Ring Platform for Text Input. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, Florence Italy, 227--239. https://doi.org/10.1145/2742647.2742665
[71]
Antti Oulasvirta. 2008. FEATURE When users" do" the Ubicomp. Interactions 15, 2 (2008), 6--9.
[72]
Tom Ouyang and Yang Li. 2012. Bootstrapping personal gesture shortcuts with the wisdom of the crowd and handwriting recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2895--2904.
[73]
Shwetak N Patel, Jeffrey S Pierce, and Gregory D Abowd. 2004. A gesture-based authentication scheme for untrusted public terminals. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 157--160.
[74]
Simon T Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. Watchit: simple gestures and eyes-free interaction for wristwatches and bracelets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1451--1460
[75]
J. Rekimoto. 2001. GestureWrist and GesturePad: unobtrusive wearable interaction devices. 21--27. https://doi.org/10.1109/ISWC.2001.962092 ISSN: 1530-0811.
[76]
Gabriel Reyes, Jason Wu, Nikita Juneja, Maxim Goldshtein, W Keith Edwards, Gregory D Abowd, and Thad Starner. 2018. Synchrowatch: One-handed synchronous smartwatch gestures using correlation and magnetic sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--26.
[77]
Mehran Roshandel, Aarti Munjal, Peyman Moghadam, Shahin Tajik, and Hamed Ketabdar. 2014. Multi-sensor Finger Ring for Authentication Based on 3D Signatures. In Human-Computer Interaction. Advanced Interaction Modalities and Techniques, David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Alfred Kobsa, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Demetri Terzopoulos, Doug Tygar, Gerhard Weikum, and Masaaki Kurosu (Eds.). Vol. 8511. Springer International Publishing, Cham, 131--138. https://doi.org/10.1007/978--3--319-07230--2_13 Series Title: Lecture Notes in Computer Science.
[78]
Antti Salovaara, Antti Oulasvirta, and Giulio Jacucci. 2017. Evaluation of prototypes and the problem of possible futures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2064--2077.
[79]
Stephanie Santosa and Daniel Wigdor. 2013. A field study of multi-device workflows in distributed workspaces. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 63--72.
[80]
T Scott Saponas, Desney S Tan, Dan Morris, and Ravin Balakrishnan. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 515--524.
[81]
T Scott Saponas, Desney S Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A Landay. 2009. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 167--176.
[82]
T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology - UIST '09. ACM Press, Victoria, BC, Canada, 167. https://doi.org/10.1145/1622176.1622208
[83]
Jeremy Scott, David Dearman, Koji Yatani, and Khai N Truong. 2010. Sensing foot gestures from the pocket. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 199--208.
[84]
Marcos Serrano, Barrett M Ens, and Pourang P Irani. 2014. Exploring the use of hand-to-face input for interacting with head-worn displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3181--3190.
[85]
Mohamed Soliman, Franziska Mueller, Lena Hegemann, Joan Sol Roo, Christian Theobalt, and Jürgen Steimle. 2018. Fingerinput: Capturing expressive single-hand thumb-to-finger microgestures. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces. 177--187.
[86]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929--1958.
[87]
Thad Starner, Joshua Weaver, and Alex Pentland. 1997. A wearable computer based american sign language recognizer. In Digest of Papers. First International Symposium on Wearable Computers. IEEE, 130--137.
[88]
Paul Strohmeier, Roel Vertegaal, and Audrey Girouard. 2012. With a flick of the wrist: stretch sensors as lightweight input for mobile devices. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction. 307--308.
[89]
Hoang Truong, Shuo Zhang, Ufuk Muncuk, Phuc Nguyen, Nam Bui, Anh Nguyen, Qin Lv, Kaushik Chowdhury, Thang Dinh, and Tam Vu. 2018. CapBand: Battery-free Successive Capacitance Sensing Wristband for Hand Gesture Recognition. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems. ACM, Shenzhen China, 54--67. https://doi.org/10.1145/3274783.3274854
[90]
Hsin-Ruey Tsai, Min-Chieh Hsiu, Jui-Chun Hsiao, Lee-Ting Huang, Mike Chen, and Yi-Ping Hung. 2016. TouchRing: subtle and always-available input using a multi-touch ring. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct. ACM, Florence Italy, 891--898. https://doi.org/10.1145/2957265.2961860
[91]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[92]
Bryan Wang and Tovi Grossman. 2020. BlyncSync: Enabling Multimodal Smartwatch Gestures with Synchronous Touch and Blink. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.
[93]
Zihan Wang, Jiarong Li, Yuchao Jin, Jiyu Wang, Fang Yang, Gang Li, Xiaoyue Ni, and Wenbo Ding. 2021. Sensing beyond itself: Multi-functional use of ubiquitous signals towards wearable applications. Digital Signal Processing 116 (2021), 103091.
[94]
Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition using an Off-theShelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, San JoseCalifornia USA, 3847--3851. https://doi.org/10.1145/2858036.2858466
[95]
William Antonelli and Jennifer Still. 2021. How to use Spotify on your Apple Watch to play music or control playback. https://www.businessinsider.in/tech/how-to/how-to-use-spotify-on-your-apple-watch-to-play-music-orcontrol-playback/articleshow/82538864.cms. Online; accessed May 2023.
[96]
Erwin Wu, Ye Yuan, Hui-Shyong Yeo, Aaron Quigley, Hideki Koike, and Kris M Kitani. 2020. Back-hand-pose: 3d hand pose estimation for a wrist-worn camera via dorsum deformation network. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 1147--1160.
[97]
Jason Wu, Cooper Colglazier, Adhithya Ravishankar, Yuyan Duan, Yuanbo Wang, Thomas Ploetz, and Thad Starner. 2018. Seesaw: rapid one-handed synchronous gesture interface for smartwatches. In Proceedings of the 2018 ACM International Symposium on Wearable Computers. 17--20.
[98]
Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, Santa Fe New Mexico USA, 9--14. https://doi.org/10.1145/2699343.2699350
[99]
Xuhai Xu, Alexandru Dancu, Pattie Maes, and Suranga Nanayakkara. 2018. Hand range interface: Information always at hand with a body-centric mid-air input surface. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1--12.
[100]
Xuhai Xu, Jun Gong, Carolina Brum, Lilian Liang, Bongsoo Suh, Shivam Kumar Gupta, Yash Agarwal, Laurence Lindsey, Runchang Kang, Behrooz Shahsavari, et al. 2022. Enabling hand gesture customization on wrist-worn devices. In CHI Conference on Human Factors in Computing Systems. 1--19.
[101]
Xuhai Xu, Haitian Shi, Xin Yi, Wenjia Liu, Yukang Yan, Yuanchun Shi, Alex Mariakakis, Jennifer Mankoff, and Anind K Dey. 2020. Earbuddy: Enabling on-face interaction via wireless earbuds. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.
[102]
Xing-Dong Yang, Tovi Grossman, Daniel Wigdor, and George Fitzmaurice. 2012. Magic finger: always-available input through finger instrumentation. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 147--156.
[103]
Hui-Shyong Yeo, Erwin Wu, Juyoung Lee, Aaron Quigley, and Hideki Koike. 2019. Opisthenar: Hand poses and finger tapping recognition by observing back of hand using embedded wrist camera. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 963--971.
[104]
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2114--2124.
[105]
Ben Zhang, Yu-Hsiang Chen, Claire Tuna, Achal Dave, Yang Li, Edward Lee, and Björn Hartmann. 2014. HOBS: head orientation-based selection in physical spaces. In Proceedings of the 2nd ACM symposium on Spatial user interaction. 17--25.
[106]
Cheng Zhang, Xiaoxuan Wang, Anandghan Waghmare, Sumeet Jain, Thomas Ploetz, Omer T Inan, Thad E Starner, and Gregory D Abowd. 2017. FingOrbits: interaction with wearables using synchronized thumb movements. In Proceedings of the 2017 ACM International Symposium on Wearable Computers. 62--65.
[107]
Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, Charlotte NC USA, 167--173. https://doi.org/10.1145/2807442.2807480
[108]
Oura ¯ Health Ltd. 2022. Oura Ring. https://ouraring.com/. Online; accessed Aug 2022.
Index Terms
- UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable Sensors
Recommendations
Over-The-Shoulder Training Between Redundant Wearable Sensors for Unified Gesture Interactions
UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and TechnologyWearable computers are now prevalent, and it is not uncommon to see people wearing multiple wearable devices. These wearable devices are often equipped with sensors to detect the user’s interactions and context. As more devices are worn on the user’s ...
Device-free gesture tracking using acoustic signals
MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and NetworkingDevice-free gesture tracking is an enabling HCI mechanism for small wearable devices because fingers are too big to control the GUI elements on such small screens, and it is also an important HCI mechanism for medium-to-large size mobile devices because ...
Device-free gesture tracking using acoustic signals: demo
MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and NetworkingIn this demo, we present LLAP, a hand tracking system that uses ultrasound to localize the hand of the user to enable device-free gesture inputs. LLAP utilizes speakers and microphones on Commercial-Off-The-Shelf (COTS) mobile devices to play and record ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2023
1017 pages
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 September 2023
Published in PACMHCI Volume 7, Issue MHCI
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 400Total Downloads
- Downloads (Last 12 months)304
- Downloads (Last 6 weeks)39
Reflects downloads up to 21 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in