research-article

Solids on Soli: Millimetre-Wave Radar Sensing through Materials

Authors:

Klen Čopič Pucihar,

Nuwan T. Attygalle,

Christian Sandor,

Luis A. LeivaAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 6, Issue EICS

Article No.: 156, Pages 1 - 19

https://doi.org/10.1145/3532212

Published: 17 June 2022 Publication History

Abstract

Gesture recognition with miniaturised radar sensors has received increasing attention as a novel interaction medium. The practical use of radar technology, however, often requires sensing through materials. Yet, it is still not well understood how the internal structure of materials impacts recognition performance. To tackle this challenge, we collected a large dataset of 14,090 radar recordings for 6 paradigmatic gesture classes sensed through a variety of everyday materials, performed by humans (6 materials) and a robot system (75 materials). Next, we developed a hybrid CNN+LSTM deep learning model and derived a robust indirect method to measure signal distortions, which we used to compile a comprehensive catalogue of materials for radar-based interaction. Among other findings, our experiments show that it is possible to estimate how different materials would affect gesture recognition performance of arbitrary classifiers by selecting just 3 reference materials. Our catalogue, software, models, data collection platform, and labeled datasets are publicly available.

Supplementary Material

ZIP File (v6eics156aux.zip)

Auxiliary Material includes the following files:

Download
59.81 MB

MP4 File (v6eics156.mp4)

Supplemental video

Download
15.86 MB

References

[1]

Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the Human Figure through a Wall. ACM TOG 34, 6 (2015).

[2]

Fadel Adib, Zach Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D tracking via body radio reflections. In Proc. NSDI. 317--329.

[3]

Agilent 1997. Understanding the Fundamental Principles of Vector Network Analysis. Agilent AN 1287--1.

[4]

Shaikh Shawon Arefin Shimon, Courtney Lutton, Zichun Xu, Sarah Morrison-Smith, Christina Boucher, and Jaime Ruiz. 2016. Exploring non-touchscreen gestures for smartwatches. In Proc. CHI. 3822--3833.

Digital Library

[5]

Francisco Bernardo, Nicholas Arner, and Paul Batchelor. 2017. O soli mio: exploring millimeter wave radar for musical interaction. In Proc. NIME. 283--286.

[6]

Amit Bleiweiss, Dagan Eshar, Gershom Kutliroff, Alon Lerner, Yinon Oshrat, and Yaron Yanai. 2010. Enhanced interactive gaming by blending full-body tracking and gesture animation. In ACM SIGGRAPH ASIA 2010 Sketches. 1--2.

Digital Library

[7]

Victor C Chen, Fayin Li, S-S Ho, and Harry Wechsler. 2006. Micro-Doppler effect in radar: phenomenon, model, and simulation study. IEEE TGRS 42, 1 (2006), 2--21.

[8]

Kevin Chetty, Graeme E Smith, and Karl Woodbridge. 2011. Through-the-wall sensing of personnel using passive bistatic wifi radar at standoff distances. IEEE Trans. Geosci. Remote Sens. 50, 4 (2011), 1218--1226.

[9]

Barrett Ens, Aaron Quigley, Hui-Shyong Yeo, Pourang Irani, Thammathip Piumsomboon, and Mark Billinghurst. 2017. Exploring mixed-scale gesture interaction. In ACM SIGGRAPH Asia 2017 Posters. 1--2.

[10]

Luigi Gallo, Alessio Pierluigi Placitelli, and Mario Ciampi. 2011. Controller-free exploration of medical image data: Experiencing the Kinect. In Proc. CBMS. 1--6.

Digital Library

[11]

Brook Galna, Gillian Barry, Dan Jackson, Dadirayi Mhiripiri, Patrick Olivier, and Lynn Rochester. 2014. Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease. Gait & posture 39, 4 (2014), 1062--1068.

[12]

Nils Y. Hammerla, Shane Halloran, and Thomas Plotz. 2016. Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables. In Proc. IJCAI. 1533--1540.

[13]

Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2010. 3D Convolutional Neural Networks for Human Action Recognition. IEEE PAMI 35, 495--502.

[14]

Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Proc. NSDI. 303--316.

[15]

Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The gesture watch: A wireless contact-free gesture based wrist interface. In Proc. ISWC. 15--22.

Digital Library

[16]

Tarmo Koppel, Andrei Shishkin, Heldur Haldre, N. Toropov, and Piia Tint. 2017. Reflection and Transmission Properties of Common Construction Materials at 2.4 GHz Frequency. Energy Procedia 113 (2017), 158--165.

[17]

Sven Kratz and Michael Rohs. 2009. HoverFlow: expanding the design space of around-device interaction. In Proc. MobileHCI. 1--8.

[18]

Christine Kühnel, Tilo Westermann, Fabian Hemmert, Sven Kratz, Alexander Müller, and Sebastian Möller. 2011. I'm home: Defining and evaluating a gesture set for smart-home control. Int. J. Hum. Comput. Stud. 69, 11 (2011), 693--704.

[19]

David R Large, Kyle Harrington, Gary Burnett, and Orestis Georgiou. 2019. Feel the noise: Mid-air ultrasound haptics as a novel human-vehicle interaction paradigm. Appl. Ergon. 81 (2019), 102909.

[20]

Luis A. Leiva, Matjaz Kljun, Christian Sandor, and Klen Copic Pucihar. 2020. The Wearable Radar: Sensing Gestures Through Fabrics. In Proc. MobileHCI. 1--4.

Digital Library

[21]

Yi Li. 2012. Hand gesture recognition using Kinect. In Proc. CSAE. 196--199.

[22]

Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM TOG 35, 4 (2016), 1--19.

Digital Library

[23]

Gan Lu, Lik-Kwan Shark, Geoff Hall, and Ulrike Zeshan. 2012. Immersive manipulation of virtual objects through glove-based hand gesture interaction. Virtual Reality 16, 3 (2012), 243--252.

[24]

Daniel Maturana and Sebastian Scherer. 2015. VoxNet: A 3D Convolutional Neural Network for real-time object recognition. 922--928.

[25]

Jess McIntosh, Mike Fraser, Paul Worgan, and Asier Marzo. 2017. DeskWave: Desktop Interactions Using Low-Cost Microwave Doppler Arrays. In Proc. CHI EA. 1885--1892.

Digital Library

[26]

Andre Mewes, Bennet Hensen, Frank Wacker, and Christian Hansen. 2017. Touchless interaction with software in interventional radiology and surgery: a systematic literature review. IJCARS 12, 2 (2017), 291--305.

[27]

Pranav Mistry and Pattie Maes. 2009. SixthSense: a wearable gestural interface. In Proc. ACM SIGGRAPH ASIA: Art Gallery & Emerg. Tech.: Adapt. 85--85.

[28]

Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Kari Pulli. 2015. Short-range FMCW monopulse radar for hand-gesture sensing. In Proc. RadarCon. 1491--1496.

[29]

Ismail Nasr, Reinhard Jungmaier, Ashutosh Baheti, Dennis Noppeney, Jagjit S Bal, Maciej Wojnowski, Emre Karagozler, Hakim Raja, Jaime Lien, Ivan Poupyrev, et al . 2016. A highly integrated 60 GHz 6-channel transceiver with antenna in package for smart sensing and short-range communications. IEEE J. Solid-State Circuits 51, 9 (2016), 2066--2076.

[30]

J.Y.H. Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. 2015. Beyond short snippets: Deep networks for video classification. In Proc. CVPR.

[31]

Kenton O'Hara, Gerardo Gonzalez, Abigail Sellen, Graeme Penney, Andreas Varnavas, Helena Mentis, Antonio Criminisi, Robert Corish, Mark Rouncefield, Neville Dastur, et al . 2014. Touchless interaction in surgery. Commun. ACM 57, 1 (2014), 70--77.

Digital Library

[32]

Francisco J. Ordónez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16, 1 (2016).

[33]

Michael Otero. 2005. Application of a continuous wave radar for human gait recognition. In Proc. SPIE, Vol. 5809. 538--548.

[34]

C.H. Oxley, J. Williams, R. Hopper, H. Flora, D. Eibeck, and C. Alabaster. 2007. Measurement of the reflection and transmission properties of conducting fabrics at milli-metric wave frequencies. IET Sci., Meas. & Technol. 1, 3 (2007), 166--169.

[35]

Sameera Palipana, Dariush Salami, Luis A. Leiva, and Stephan Sigg. 2021. Pantomime: Mid-Air Gesture Recognition with Sparse Millimeter-Wave Radar Point Clouds. ACM IMWUT 5, 1 (2021), 27:1--27:27.

[36]

Joseph Paradiso, Craig Abler, Kai-yuh Hsiao, and Matthew Reynolds. 1997. The magic carpet: physical sensing for immersive environments. In Proc. CHI EA. 277--278.

Digital Library

[37]

Joseph A Paradiso. 1999. The brain opera technology: New instruments and gestural sensors for musical interaction and performance. JNMR 28, 2 (1999), 130--149.

[38]

François Petitjean, Alain Ketterlin, and Pierre Gançarski. 2011. A global averaging method for dynamic time warping, with applications to clustering. Pattern Recognit. 44, 3 (2011), 678--693.

Digital Library

[39]

Carl A Pickering, Keith J Burnham, and Michael J Richardson. 2007. A research study of hand gesture recognition technologies and applications for human vehicle interaction. In Proc. IET Autom. Elect. 1--15.

[40]

M PourMousavi, M Wojnowski, R Agethen, R Weigel, and A Hagelauer. 2013. Antenna array in eWLB for 61 GHz FMCW radar. In Proc. APMC. 310--312.

[41]

Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proc. MobiCom. 27--38.

Digital Library

[42]

Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proc. CVPR. 652--660.

[43]

Tauhidur Rahman, Alexander T Adams, Ruth Vinisha Ravichandran, Mi Zhang, Shwetak N Patel, Julie A Kientz, and Tanzeem Choudhury. 2015. Dopplesleep: A contactless unobtrusive sleep sensing system using short-range doppler radar. In Proc. Ubicomp. 39--50.

Digital Library

[44]

Siddharth S Rautaray and Anupam Agrawal. 2011. Interaction with virtual game through hand gesture recognition. In Proc. IMPACT. 244--247.

[45]

Richard Hartless Richard Rudd, Ken Craig, Martin Ganley. 2014. Building Materials and Propagation. Technical Report September. 40 pages.

[46]

Andreas Riener, Alois Ferscha, Florian Bachmair, Patrick Hagmüller, Alexander Lemme, Dominik Muttenthaler, David Pühringer, Harald Rogner, Adrian Tappe, and Florian Weger. 2013. Standardization of the in-car gesture interaction space. In Proc. AutomotiveUI. 14--21.

Digital Library

[47]

Marco Roccetti, Gustavo Marfia, and Angelo Semeraro. 2012. Playing into the wild: A gesture-based interface for gaming in public spaces. J. Vis. Commun. Image Represent. 23, 3 (2012), 426--440.

Digital Library

[48]

Christian Sandor and Hiraku Nakamura. 2018. SoliScratch: A Radar Interface for Scratch DJs. In Proc. ISMAR-Adjunct. 427--427.

[49]

R. W. Schafer. 2011. What Is a Savitzky-Golay Filter? [Lecture Notes]. IEEE Signal Process. Mag. 28, 4 (2011), 111--117.

[50]

Gözel Shakeri, John H Williamson, and Stephen Brewster. 2018. May the force be with you: Ultrasound haptic feedback for mid-air gesture interaction in cars. In Proc. AutomotiveUI. 1--10.

Digital Library

[51]

Merrill Ivan Skolnik et al. 1980. Introduction to radar systems. Vol. 3. McGraw-hill NY.

[52]

Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air gestures around unmodified mobile devices. In Proc. UIST. 319--329.

Digital Library

[53]

Peng Song, Wooi Boon Goh, William Hutama, Chi-Wing Fu, and Xiaopei Liu. 2012. A handle bar metaphor for virtual object manipulation with mid-air interaction. In Proc. CHI. 1297--1306.

Digital Library

[54]

Thad Starner, Jake Auxier, Daniel Ashbrook, and Maribeth Gandy. 2000. The gesture pendant: A self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring. In Proc. ISWC. 87--94.

[55]

Ivan E Sutherland. 1964. Sketchpad a man-machine graphical communication system. Simulation 2, 5 (1964), R--3.

[56]

Ivan E Sutherland. 1965. The ultimate display. In Proc. IFIP Congress. 506--508.

[57]

John W. Tukey. 1977. Exploratory data analysis. Addison-Wesley Publishing Co.

[58]

Wouter Van Vlaenderen, Jens Brulmans, Jo Vermeulen, and Johannes Schöning. 2015. Watchme: A novel input method combining a smartwatch and bimanual interaction. In Proc. CHI EA. 2091--2095.

Digital Library

[59]

Klen Copic Pucihar, Christian Sandor, Matja? Kljun, Wolfgang Huerst, Alexander Plopski, Takafumi Taketomi, Hirokazu Kato, and Luis A. Leiva. 2019. The Missing Interface: Micro-Gestures on Augmented Objects. In Proc. CHI EA. 1--6.

[60]

Qian Wan, Yiran Li, Changzhi Li, and Ranadip Pal. 2014. Gesture recognition for smart home applications using portable radar sensors. In Proc. EMBS. 6414--6417.

[61]

Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proc. UIST. 851--860.

Digital Library

[62]

Yazhou Wang and Aly E Fathy. 2011. Micro-Doppler signatures for intelligent human gait recognition using a UWB impulse radar. In Proc. APSURSI. 2103--2106.

[63]

Teng Wei and Xinyu Zhang. 2015. mTrack: High-precision passive tracking using millimeter wave radios. In Proc. MobiCom. 117--129.

Digital Library

[64]

Huiyue Wu and Jianmin Wang. 2012. User-defined body gestures for TV-based applications. In Proc. ICDH. 415--420.

Digital Library

[65]

LI Yang, Jin Huang, TIAN Feng, WANG Hong-An, and DAI Guo-Zhong. 2019. Gesture interaction in virtual reality. VRIH 1, 1 (2019), 84--112.

[66]

Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. RadarCat: Radar Categorization for Input & Interaction. In Proc. UIST. 833--841.

Digital Library

[67]

Renyuan Zhang and Siyang Cao. 2018. Real-time human motion behavior detection via CNN using mmWave radar. IEEE Sens. Lett. 3, 2 (2018), 1--4.

[68]

Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S Reynolds. 2014. SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal. In Proc. UIST. 527--534.

Digital Library

[69]

M. Zhao, T. Li, M. A. Alsheikh, Y. Tian, H. Zhao, A. Torralba, and D. Katabi. 2018. Through-Wall Human Pose Estimation Using Radio Signals. In Proc. CVPR. 7356--7365.

[70]

Yan Zhuang, Chen Song, Aosen Wang, Feng Lin, Yiran Li, Changzhan Gu, Changzhi Li, and Wenyao Xu. 2015. SleepSense: Non-invasive sleep event recognition using an electromagnetic probe. In Proc. BSN. 1--6.

Cited By

Čopič Pucihar KMeneghello FSalami DSluÿters AVatavu R(2024)Beyond Radar Waves: The First Workshop on Radar-Based Human-Computer InteractionCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662836(97-102)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662836
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Sluÿters ALambot SVanderdonckt JVillarreal-Narvaez S(2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3366667
Show More Cited By

Recommendations

Soli: ubiquitous gesture sensing with millimeter wave radar

This paper presents Soli, a new, robust, high-resolution, low-power, miniature gesture sensing technology for human-computer interaction based on millimeter-wave radar. We describe a new approach to developing a radar-based sensor optimized for human-...
Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

This paper proposes a novel machine learning architecture, specifically designed for radio-frequency based gesture recognition. We focus on high-frequency (60]GHz), short-range radar based sensing, in particular Google's Soli sensor. The signal has ...
The Wearable Radar: Sensing Gestures Through Fabrics
MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

Recently, millimeter-wave radar-on-chip sensors such as Google Soli have become readily available in the mobile ecosystem. We envision such radar technology to be integrated into wearables to enable gesture-based interaction possibilities for users ‘on ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 6, Issue EICS

EICS

June 2022

736 pages

EISSN:2573-0142

DOI:10.1145/3544787

Editor:
Jeff Nichols
Apple Inc., United States

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2022

Published in PACMHCI Volume 6, Issue EICS

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article

Funding Sources

European Commission, InnoRenew CoE project
Horizon 2020 FET, ERA-NET Cofund
ARRS

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
247
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)10

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Čopič Pucihar KMeneghello FSalami DSluÿters AVatavu R(2024)Beyond Radar Waves: The First Workshop on Radar-Based Human-Computer InteractionCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662836(97-102)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662836
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Sluÿters ALambot SVanderdonckt JVillarreal-Narvaez S(2024)Analysis of User-Defined Radar-Based Hand Gestures Sensed Through Multiple MaterialsIEEE Access10.1109/ACCESS.2024.336666712(27895-27917)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3366667
Sluÿters ALambot SVanderdonckt JVatavu R(2023)RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training ExamplesACM Transactions on Interactive Intelligent Systems10.1145/358964513:3(1-45)Online publication date: 11-Sep-2023
https://dl.acm.org/doi/10.1145/3589645
Şiean APamparău CSluÿters AVatavu RVanderdonckt J(2023)Flexible gesture input with radars: systematic literature review and taxonomy of radar sensing integration in ambient intelligence environmentsJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-023-04606-914:6(7967-7981)Online publication date: 10-Apr-2023
https://doi.org/10.1007/s12652-023-04606-9

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents