Nothing Special   »   [go: up one dir, main page]

skip to main content
research-article
Open access

Wavesdropper: Through-wall Word Detection of Human Speech via Commercial mmWave Devices

Published: 07 July 2022 Publication History

Abstract

Most existing eavesdropping attacks leverage propagating sound waves for speech retrieval. However, soundproof materials are widely deployed in speech-sensitive scenes (e.g., a meeting room). In this paper, we reveal that human speech protected by an isolated room can be compromised by portable and commercial off-the-shelf mmWave devices. To achieve this goal, we develop Wavesdropper, a word detection system that utilizes a mmWave probe to sense the targeted speaker's throat vibration and recover speech contents in the obstructed condition. We proposed a CEEMD-based method to suppress dynamic clutters (e.g., human movements) in the room and a wavelet-based processing method to extract the delicate vocal vibration information from the hybrid signals. To recover speech contents from mmWave signals related to the vocal vibration, we designed a neural network to infer the speech contents. Moreover, we explored word detection on a conversation with multiple (two) probes and reveal that the adversary can detect words on multiple people simultaneously with only one mmWave device. We performed extensive experiments to evaluate the system performance with over 60,000 pronunciations. The experimental results indicate that Wavesdropper can achieve 91.3% accuracy for 57-word recognition on 23 volunteers.

References

[1]
[n. d.]. Apple Siri Eavesdropping Puts Millions Of Users At Risk. [online]. https://www.forbes.com/sites/kateoflahertyuk/2019/07/28/apple-siri-eavesdropping-puts-millions-of-users-at-risk/#1dd38829a530 Accessed on March 26, 2021.
[2]
[n. d.]. The Group Dynamics of Smartphone Eavesdropping. [online]. https://www.securitymagazine.com/articles/90577-the-group-dynamics-of-smartphone-eavesdropping, Accessed on March 26, 2021.
[3]
[n. d.]. IWR1642Boost. [online]. https://www.ti.com/tool/IWR1642BOOST, Accessed on March 26, 2021.
[4]
[n. d.]. Why Enterprises Still Have to Worry about Eavesdropping. [online]. https://blog.trendmicro.com/why-enterprises-still-have-to-worry-about-eavesdropping/, Accessed on March 26, 2021.
[5]
Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--13.
[6]
Mamta Agiwal, Abhishek Roy, and Navrati Saxena. 2016. Next generation 5G wireless networks: A comprehensive survey. IEEE Communications Surveys & Tutorials 18, 3 (2016), 1617--1655.
[7]
Adeel Ahmad, June Chul Roh, Dan Wang, and Aish Dubey. 2018. Vital signs monitoring of multiple people using a FMCW millimeter-wave sensor. In 2018 IEEE Radar Conference (RadarConf18). IEEE, 1450--1455.
[8]
S Abhishek Anand and Nitesh Saxena. 2018. Speechless: Analyzing the threat to speech privacy from smartphone motion sensors. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 1000--1017.
[9]
Zhongjie Ba, Tianhang Zheng, Xinyu Zhang, Zhan Qin, Baochun Li, Xue Liu, and Kui Ren. 2020. Learning-based practical smartphone eavesdropping with built-in accelerometer. In Proceedings of the Network and Distributed Systems Security (NDSS) Symposium. 23--26.
[10]
Arijit Banerjee, Dustin Maas, Maurizio Bocca, Neal Patwari, and Sneha Kasera. 2014. Violating privacy through walls by passive monitoring of radio windows. In Proceedings of the 2014 ACM conference on Security and privacy in wireless & mobile networks. 69--80.
[11]
Mattia Brambilla, Monica Nicoli, Sergio Savaresi, and Umberto Spagnolini. 2019. Inertial sensor aided mmWave beam tracking to support cooperative autonomous driving. In 2019 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 1--6.
[12]
A Bunkowski, B Bödeker, S Bader, M Westhoff, P Litterst, and J I Baumbach. 2009. MCC/IMS signals in human breath related to sarcoidosis---results of a feasibility study using an automated peak finding procedure. Journal of Breath Research 3, 4 (sep 2009), 046001.
[13]
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. 2017. Lip reading sentences in the wild. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 3444--3453.
[14]
Pierre Comon and Christian Jutten. 2010. Handbook of Blind Source Separation: Independent component analysis and applications. Academic press.
[15]
Khadidja Ghribi, Mohamed Djendi, and Daoued Berkani. 2016. A wavelet-based forward BSS algorithm for acoustic noise reduction and speech enhancement. Applied Acoustics 105 (2016), 55--66.
[16]
Jian Gong, Xinyu Zhang, Kaixin Lin, Ju Ren, Yaoxue Zhang, and Wenxun Qiu. 2021. RF Vital Sign Sensing under Free Body Movement. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--22.
[17]
Tianbo Gu, Zheng Fang, Zhicheng Yang, Pengfei Hu, and Prasant Mohapatra. 2019. Mmsense: Multi-person detection and identification via mmwave sensing. In Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems. 45--50.
[18]
Unsoo Ha, Sohrab Madani, and Fadel Adib. 2021. WiStress: Contactless Stress Monitoring Using Wireless Signals. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 3, Article 103 (sep 2021), 37 pages. https://doi.org/10.1145/3478121
[19]
Jun Han, Albert Jin Chung, and Patrick Tague. 2017. Pitchln: eavesdropping via intelligible speech reconstruction using non-acoustic sensor fusion. In Proceedings of the 16th ACM/IEEE International Conference on Information Processing in Sensor Networks. 181--192.
[20]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[21]
Yongzhong He, Xuejun Yang, Binghui Hu, and Wei Wang. 2019. Dynamic privacy leakage analysis of Android third-party libraries. Journal of Information Security and Applications 46 (2019), 259--270.
[22]
Jerry Hildenbrand. [n. d.]. Here's how the Pixel 4's Soli radar works and why Motion Sense has so much potential. [online]. https://www.androidcentral.com/how-does-googles-soli-chip-work, Accessed on March 26, 2021.
[23]
Denis Ivanko, Alexey Karpov, Dmitry Ryumin, Irina Kipyatkova, Anton Saveliev, Victor Budkov, Dmitriy Ivanko, and Miloš Železny. 2017. Using a high-speed video camera for robust audio-visual speech recognition in acoustically noisy conditions. In International Conference on Speech and Computer. Springer, 757--766.
[24]
Chengkun Jiang, Junchen Guo, Yuan He, Meng Jin, Shuai Li, and Yunhao Liu. 2020. MmVib: Micrometer-Level Vibration Measurement with Mmwave Radar. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3372224.3419202
[25]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[26]
Andrew Kwong, Wenyuan Xu, and Kevin Fu. 2019. Hard Drive of Hearing: Disks that Eavesdrop with a Synthesized Microphone. In 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 905--919.
[27]
Huining Li, Chenhan Xu, Aditya Singh Rathore, Zhengxiong Li, Hanbin Zhang, Chen Song, Kun Wang, Lu Su, Feng Lin, Kui Ren, et al. 2020. VocalPrint: exploring a resilient and secure voice authentication via mmWave biometric interrogation. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems.
[28]
Yingsong Li, Zelong Shao, Xiangkun Zhang, and Jingshan Jiang. 2018. Vibrations monitoring for highway bridge using mm-Wave radar. In 2018 IEEE Asia-Pacific Conference on Antennas and Propagation (APCAP). IEEE, 501--502.
[29]
Zhengxiong Li, Fenglong Ma, Aditya Singh Rathore, Zhuolin Yang, Baicheng Chen, Lu Su, and Wenyao Xu. 2020. Wavespy: Remote and through-wall screen attack via mmwave sensing. In 2020 IEEE Symposium on Security and Privacy (SP). 49--64.
[30]
Yan Michalevsky, Dan Boneh, and Gabi Nakibly. 2014. Gyrophone: Recognizing speech from gyroscope signals. In 23rd {USENIX} Security Symposium ({USENIX} Security 14). 1053--1067.
[31]
Jochen Moll, Kaspar Bechtel, Bernd Hils, and Viktor Krozer. 2014. Mechanical vibration sensing for structural health monitoring using a millimeter-wave doppler radar sensor.
[32]
Masanori Morise, Hideki Kawahara, and Haruhiro Katayose. 2009. fast and reliable f0 estimation method based on the period extraction of vocal fold vibration of singing voice and speech. journal of the audio engineering society (february 2009).
[33]
Timothy I Murphy. [n. d.]. Analyst's Desktop Binder. [online]. https://www.hsdl.org/?view&did=710020 Accessed on March 26, 2021.
[34]
Rajalakshmi Nandakumar, Alex Takakuwa, Tadayoshi Kohno, and Shyamnath Gollakota. 2017. Covertband: Activity information leakage using music. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--24.
[35]
Ben Nassi, Yaron Pirutin, Adi Shamir, Yuval Elovici, and Boris Zadov. 2020. Lamphone: Real-Time Passive Sound Recovery from Light Bulb Vibrations. BlackHat USA (2020).
[36]
K O'flaherty. 2019. Amazon Staff Are Listening To Alexa Conversations---Here's What To Do'. Forbes (2019).
[37]
Hyojin Park, Christoph Kayser, Gregor Thut, and Joachim Gross. 2016. Lip movements entrain the observers' low-frequency brain oscillations to facilitate speech intelligibility. Elife 5 (2016), e14521.
[38]
Sandeep Rao. 2017. Introduction to mmWave sensing: FMCW radars. Texas Instruments (TI) mmWave Training Series (2017).
[39]
S. Rao and A. V. Mani. 2020. Interference Characterization in FMCW radars. In 2020 IEEE Radar Conference (RadarConf20). 1--6. https://doi.org/10.1109/RadarConf2043947.2020.9266283
[40]
Linda Senigagliesi, Gianluca Ciattaglia, and Ennio Gambi. 2020. Contactless Walking Recognition based on mmWave RADAR. In 2020 IEEE Symposium on Computers and Communications (ISCC). IEEE, 1--4.
[41]
Satish Sinha, Partha S Routh, Phil D Anno, and John P Castagna. 2005. Spectral decomposition of seismic data with continuous-wavelet transform. Geophysics 70, 6 (2005), P19-P25.
[42]
M. Song, J. Lim, and D. Shin. 2014. The velocity and range detection using the 2D-FFT scheme for automotive radars. In 2014 4th IEEE International Conference on Network Infrastructure and Digital Content. 507--510.
[43]
M. E. Torres, M. A. Colominas, G. Schlotthauer, and P. Flandrin. 2011. A complete ensemble empirical mode decomposition with adaptive noise. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4144--4147. https://doi.org/10.1109/ICASSP.2011.5947265
[44]
Vutha Va, Takayuki Shimizu, Gaurav Bansal, and Robert W Heath Jr. 2016. Millimeter wave vehicular communications: A survey. Foundations and Trends® in Networking 10, 1 (2016), 1--118.
[45]
Pradeep Verma, Vivek Sheel Shakya, Divya Sharma, Usha Chauhan, and Abhilash Gaur. 2020. MM-Wave Radar Application for Autonomous Vehicles. In 2020 2nd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN). IEEE, 556--559.
[46]
Fei Wang, Jianwei Feng, Yinliang Zhao, Xiaobin Zhang, Shiyuan Zhang, and Jinsong Han. 2019. Joint Activity Recognition and Indoor Localization With WiFi Fingerprints. IEEE Access 7 (2019), 80058--80068.
[47]
Guanhua Wang, Yongpan Zou, Zimu Zhou, Kaishun Wu, and Lionel M Ni. 2016. We can hear you with Wi-Fi! IEEE Transactions on Mobile Computing 15, 11 (2016), 2907--2920.
[48]
Teng Wei, Shu Wang, Anfu Zhou, and Xinyu Zhang. 2015. Acoustic eavesdropping through wireless vibrometry. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. 130--141.
[49]
Henk Wymeersch, Gonzalo Seco-Granados, Giuseppe Destino, Davide Dardari, and Fredrik Tufvesson. 2017. 5G mmWave positioning for vehicular networks. IEEE Wireless Communications 24, 6 (2017), 80--86.
[50]
Chenhan Xu, Zhengxiong Li, Hanbin Zhang, Aditya Singh Rathore, Huining Li, Chen Song, Kun Wang, and Wenyao Xu. 2019. Waveear: Exploring a mmwave-based noise-resistant speech sensing for voice-user interface. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. 14--26.
[51]
Zhicheng Yang, Parth H Pathak, Yunze Zeng, Xixi Liran, and Prasant Mohapatra. 2016. Monitoring vital signs using millimeter wave. In Proceedings of the 17th ACM international symposium on mobile ad hoc networking and computing. 211--220.
[52]
Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence 29, 6 (2007), 1091--1095.
[53]
Hansong Zeng and Yi Zhao. 2011. Sensing movement: Microsensors for body motion measurement. Sensors 11, 1 (2011), 638--660.
[54]
Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems. 8778--8788.
[55]
Mingmin Zhao, Tianhong Li, Mohammad Abu Alsheikh, Yonglong Tian, Hang Zhao, Antonio Torralba, and Dina Katabi. 2018. Through-wall human pose estimation using radio signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7356--7365.

Cited By

View all
  • (2025)Ambient Light Reflection-Based Eavesdropping Enhanced With cGANIEEE Transactions on Mobile Computing10.1109/TMC.2024.346039224:1(72-85)Online publication date: Jan-2025
  • (2024)HomeOSD: Appliance Operating-Status Detection Using mmWave RadarSensors10.3390/s2409291124:9(2911)Online publication date: 2-May-2024
  • (2024)Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997398:4(1-26)Online publication date: 21-Nov-2024
  • Show More Cited By

Index Terms

  1. Wavesdropper: Through-wall Word Detection of Human Speech via Commercial mmWave Devices

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 2
    July 2022
    1551 pages
    EISSN:2474-9567
    DOI:10.1145/3547347
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2022
    Published in IMWUT Volume 6, Issue 2

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. mmWave sensing
    2. through walls
    3. word detection

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • Leading Innovative and Entrepreneur Team Introduction Program of Zhejiang
    • National Natural Science Foundation of China
    • National Key R&D Program of China

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)420
    • Downloads (Last 6 weeks)31
    Reflects downloads up to 16 Dec 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Ambient Light Reflection-Based Eavesdropping Enhanced With cGANIEEE Transactions on Mobile Computing10.1109/TMC.2024.346039224:1(72-85)Online publication date: Jan-2025
    • (2024)HomeOSD: Appliance Operating-Status Detection Using mmWave RadarSensors10.3390/s2409291124:9(2911)Online publication date: 2-May-2024
    • (2024)Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997398:4(1-26)Online publication date: 21-Nov-2024
    • (2024)Beamforming for Sensing: Hybrid Beamforming based on Transmitter-Receiver Collaboration for Millimeter-Wave SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596198:2(1-27)Online publication date: 15-May-2024
    • (2024)WiFi-CSI Difference ParadigmProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596088:2(1-29)Online publication date: 15-May-2024
    • (2024)BreathPro: Monitoring Breathing Mode during Running with EarablesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596078:2(1-25)Online publication date: 15-May-2024
    • (2024)ICY Interfaces: Exploration of Ice’s Ephemeral Features for Digital Game User ExperienceProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660725(2107-2124)Online publication date: 1-Jul-2024
    • (2024)RFSpy: Eavesdropping on Online Conversations with Out-of-Vocabulary Words by Sensing Metal Coil Vibration of Headsets Leveraging RFIDProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661887(169-182)Online publication date: 3-Jun-2024
    • (2024)Enabling 6D Pose Tracking on Your Acoustic DevicesProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661875(15-28)Online publication date: 3-Jun-2024
    • (2024)F2Key: Dynamically Converting Your Face into a Private Key Based on COTS Headphones for Reliable Voice InteractionProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661860(127-140)Online publication date: 3-Jun-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media