research-article

UltraSnoop: Placement-agnostic Keystroke Snooping via Smartphone-based Ultrasonic Sonar

Authors:

Lei XieAuthors Info & Claims

ACM Transactions on Internet of Things, Volume 4, Issue 4

Article No.: 22, Pages 1 - 24

https://doi.org/10.1145/3614440

Published: 22 November 2023 Publication History

Abstract

Keystroke snooping is an effective way to steal sensitive information from the victims. Recent research on acoustic emanation-based techniques has greatly improved the accessibility by non-professional adversaries. However, these approaches either require multiple smartphones or require specific placement of the smartphone relative to the keyboards, which tremendously restricts the application scenarios. In this article, we propose UltraSnoop, a training-free, transferable, and placement-agnostic scheme that manages to infer user’s input using a single smartphone placed within the range covered by a microphone and speaker. The innovation of Ultrasnoop is that we propose an ultrasonic anchor-keystroke positioning method and a Mel Frequency Cepstrum Coefficients clustering algorithm, synthesis of which could infer the relative position between the smartphone and the keyboard. Along with the keystroke time difference of arrival, our method could infer the keystrokes and even gradually improve the accuracy as the snooping proceeds. Our real-world experiments show that UltraSnoop could achieve more than 85% top-3 snooping accuracy when the smartphone is placed within the range of 30–60 cm from the keyboard.

References

[1]

Dmitri Asonov and Rakesh Agrawal. 2004. Keyboard acoustic emanations. In Proceedings of the IEEE Symposium on Security and Privacy (SP’04). IEEE, 3.

[2]

Yigael Berger, Avishai Wool, and Arie Yeredor. 2006. Dictionary attacks using keyboard acoustic emanations. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’06). ACM, 245–254.

Digital Library

[3]

Huijie Chen, Fan Li, and Yu Wang. 2017. EchoTrack: Acoustic device-free hand tracking on smart phones. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM’17). IEEE, 1–9.

[4]

Yimin Chen, Tao Li, Rui Zhang, Yanchao Zhang, and Terri Hedgpeth. 2018. EyeTell: Video-assisted touchscreen keystroke inference from eye movements. In Proceedings of the IEEE Symposium on Security and Privacy (SP’18). IEEE, 144–160.

[5]

Yuchi Chen, Gong Wei, Jiangchuan Liu, and Cui Yong. 2017. Fine-grained ultrasound range finding for mobile devices: Sensing way beyond the 24 kHz limit of built-in microphones. In Computer Communications Workshops.

[6]

Song Fang, Ian Markwood, Yao Liu, Shangqing Zhao, Zhuo Lu, and Haojin Zhu. 2018. No training hurdles: Fast training-agnostic attacks to infer your typing. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’18). ACM, 1747–1760.

Digital Library

[7]

Yuyi Fang, Zhiwei Zhao, Zi Wang, Geyong Min, Yue Cao, Haojun Huang, and Hao Yin. 2018. Eavesdrop with PoKeMon: Position free keystroke monitoring using acoustic data. Fut. Gener. Comput. Syst. 87 (2018), 704–711. https://www.sciencedirect.com/science/article/abs/pii/S0167739X17300493

Digital Library

[8]

Tyler Giallanza, Travis Siems, Elena Smith, Erik Gabrielsen, Ian Johnson, Mitchell A. Thornton, and Eric C. Larson. 2019. Keyboard snooping from mobile phone arrays with mixed convolutional and recurrent neural networks. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3, 2 (2019), 1–22.

Digital Library

[9]

Wenchao Huang, Xiang-Yang Li, Yan Xiong, Panlong Yang, Yiqing Hu, Xufei Mao, Fuyou Miao, Baohua Zhao, and Jumin Zhao. 2016. WalkieLokie: Sensing relative positions of surrounding presenters by acoustic signals. In Proceedings of the the ACM Conference on Ubiquitous Computing (Ubicomp’16). ACM, 439–450.

Digital Library

[10]

Jesper Højvang Jensen, Daniel P. W. Ellis, Mads Græsbøll Christensen, and Søren Holdt Jensen. 2007. Evaluation of distance measures between gaussian mixture models of MFCCs. In Proceedings of the International Society for Music Information Retrieval (ISMIR’07). 107–108.

[11]

Tomi Kinnunen, Rahim Saeidi, Johan Sandberg, and Maria Hansson-Sandsten. 2010. What else is new than the Hamming window? Robust MFCCs for speaker recognition via multitapering. In Proceedings of the 11th Annual Conference of the International Speech Communication Association.

[12]

Hyewon Lee, Tae Hyun Kim, Jun Won Choi, and Sunghyun Choi. 2015. Chirp signal-based aerial acoustic communication for smart devices. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM’15). IEEE, 2407–2415.

[13]

Mengyuan Li, Yan Meng, Junyi Liu, Haojin Zhu, Xiaohui Liang, Yao Liu, and Na Ruan. 2016. When CSI meets public WiFi: Inferring your mobile phone password via WiFi signals. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’16). ACM, 1068–1079.

Digital Library

[14]

Kang Ling, Haipeng Dai, Yuntang Liu, and Alex X. Liu. 2018. UltraGesture: Fine-grained gesture sensing and recognition. In Proceedings of the IEEE Communications Society Conference on Sensor and Ad Hoc Communications and Networks (SECON’18). IEEE.

Digital Library

[15]

Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. 2015. Snooping keystrokes with mm-level audio ranging on a single phone. In Proceedings of the ACM International Conference on Mobile Computing and Networking (Mobicom’15). ACM, 142–154.

Digital Library

[16]

Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: High-precision acoustic motion tracking. In Proceedings of the ACM International Conference on Mobile Computing and Networking (Mobicom’16). ACM, 69–81.

Digital Library

[17]

K. Sri Rama Murty and Bayya Yegnanarayana. 2006. Combining evidence from residual phase and MFCC features for speaker recognition. IEEE Sign. Process. Lett. 13, 1 (2006), 52–55.

[18]

Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. Fingerio: Using active sonar for fine-grained finger tracking. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’16). ACM, 1515–1525.

Digital Library

[19]

Yasunari Obuchi. 2005. Mixture weight optimization for dual-microphone MFCC combination. In Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding. IEEE, 325–330.

[20]

Zhenquan Qin, Jiaxin Du, Guangjie Han, Gaopeng Yong, Linlin Guo, and Lei Wang. 2019. LOL: Localization-free online keystroke tracking using acoustic signals. Soft Comput. 23, 21 (2019), 11063–11075.

Digital Library

[21]

Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. VSkin: Sensing touch gestures on surfaces of mobile devices using acoustic signals. In Proceedings of the ACM International Conference on Mobile Computing and Networking (Mobicom’18). ACM, 591–605.

Digital Library

[22]

Jiayao Tan, Cam-Tu Nguyen, and Xiaoliang Wang. 2017. SilentTalk: Lip reading through ultrasonic sensing on mobile phones. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM’17). 1–9.

[23]

Jian Wang, Rukhsana Ruby, Lu Wang, and Kaishun Wu. 2016. Accurate combined keystrokes detection using acoustic signals. In Proceedings of the IEEE Mobile Ad-Hoc and Sensor Networks (MSN’16). IEEE, 9–14.

[24]

Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. 2014. Ubiquitous keyboard for small mobile devices: Harnessing multipath fading for fine-grained keystroke localization. In Proceedings of the ACM ACM International Conference on Mobile Systems, Applications, and Services (Mobisys’14). 14–27.

Digital Library

[25]

Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the ACM International Conference on Mobile Computing and Networking (Mobicom’16). ACM, 82–94.

[26]

Jiadi Yu, Li Lu, Yingying Chen, Yanmin Zhu, and Linghe Kong. 2019. An indirect eavesdropping attack of keystrokes on touch screen through acoustic sensing. IEEE Trans. Mobile Comput. 20, 2 (2019), 337–351.

Digital Library

[27]

Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Sumeet Jain, Yiming Pu, Sinan Hersek, Kent Lyons, Kenneth A. Cunefare, Omer T. Inan, and Gregory D. Abowd. 2017. Soundtrak: Continuous 3d tracking of a finger using active acoustics. Proceedings of the ACM Conference on Ubiquitous Computing (Ubicomp’17), 30.

[28]

Qianqian Zhou, Yanni Yang, Feng Hong, Yuan Feng, and Zhongwen Guo. 2016. User identification and authentication using keystroke dynamics with acoustic signal. In Proceedings of the 12th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN’16). IEEE, 445–449.

[29]

Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. 2014. Context-free attacks using keyboard acoustic emanations. In Proceedings of the ACM Conference on Computer and Communications Security (CCS’14). ACM, 453–464.

Digital Library

[30]

Li Zhuang, Feng Zhou, and J. Doug Tygar. 2009. Keyboard acoustic emanations revisited. ACM Trans. Inf. Syst. Secur. 13, 1 (2009), 3.

Digital Library

Cited By

Cai GWang JShu YLiu JTan RHe YChen J(2024)Locating Your Smart Devices with a Single SpeakerProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699320(28-40)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699320
Wang PHu JLiu CLuo JLuo BLiao XXu JKirda ELie D(2024)RefleXnoop: Passwords Snooping on NLoS Laptops Leveraging Screen-Induced Sound ReflectionProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670341(3361-3375)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3670341
Huang JBai JZhang XLiu ZFeng YLiu JSun XDong MLi M(2024)KeystrokeSniffer: An Off-the-Shelf Smartphone Can Eavesdrop on Your Privacy From AnywhereIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.342430119(6840-6855)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3424301

Index Terms

UltraSnoop: Placement-agnostic Keystroke Snooping via Smartphone-based Ultrasonic Sonar
1. Human-centered computing
  1. Ubiquitous and mobile computing

Recommendations

Research of Palmprint Authentication Arithmetic Based on Smartphone
ICNCC '18: Proceedings of the 2018 VII International Conference on Network, Communication and Computing

Palmprint authentication technology can solve the information security problem of mobile devices effectively. Based on smartphone, a new palmprint acquisition scheme and an improved authentication algorithm are proposed in this article. Key points and ...
Snooping Keystrokes with mm-level Audio Ranging on a Single Phone
MobiCom '15: Proceedings of the 21st Annual International Conference on Mobile Computing and Networking

This paper explores the limits of audio ranging on mobile devices in the context of a keystroke snooping scenario. Acoustic keystroke snooping is challenging because it requires distinguishing and labeling sounds generated by tens of keys in very close ...
Accurate Indoor Positioning System Using Near-Ultrasonic Sound from a Smartphone
NGMAST '14: Proceedings of the 2014 Eighth International Conference on Next Generation Mobile Apps, Services and Technologies

This paper describes new concepts and techniques for an indoor positioning system that uses near-ultrasonic sound from a smartphone. The indoor positioning system can be used in many practical applications, for example, in detecting the location of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet of Things

ACM Transactions on Internet of Things Volume 4, Issue 4

Special Issue on Wireless Sensing for IoT: Part 1

November 2023

194 pages

EISSN:2577-6207

DOI:10.1145/3633336

Editor:
Gian Pietro Picco
University of Trento, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 22 November 2023

Online AM: 10 August 2023

Accepted: 24 July 2023

Revised: 29 March 2023

Received: 28 October 2022

Published in TIOT Volume 4, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Natural Science Foundation of Jiangsu Province
A3 Foresight Program of NSFC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
474
Total Downloads

Downloads (Last 12 months)246
Downloads (Last 6 weeks)22

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cai GWang JShu YLiu JTan RHe YChen J(2024)Locating Your Smart Devices with a Single SpeakerProceedings of the 22nd ACM Conference on Embedded Networked Sensor Systems10.1145/3666025.3699320(28-40)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3666025.3699320
Wang PHu JLiu CLuo JLuo BLiao XXu JKirda ELie D(2024)RefleXnoop: Passwords Snooping on NLoS Laptops Leveraging Screen-Induced Sound ReflectionProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670341(3361-3375)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3670341
Huang JBai JZhang XLiu ZFeng YLiu JSun XDong MLi M(2024)KeystrokeSniffer: An Off-the-Shelf Smartphone Can Eavesdrop on Your Privacy From AnywhereIEEE Transactions on Information Forensics and Security10.1109/TIFS.2024.342430119(6840-6855)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIFS.2024.3424301

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents