research-article

PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition

Authors:

Yasha Iravantchi,

Chris Harrison,

Alanson SampleAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 198, Pages 1 - 13

https://doi.org/10.1145/3411764.3445169

Published: 07 May 2021 Publication History

Abstract

Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants “eavesdropped’’ on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.

Supplementary Material

VTT File (3411764.3445169_videofigurecaptions.vtt)

Download
2.95 KB

MP4 File (3411764.3445169_videofigure.mp4)

Supplemental video

Download
88.49 MB

References

[1]

Amazon.com. 2019. Pioneer SP-BS22-LR. Website. Retrieved September 20, 2019 from https://www.amazon.com/Pioneer-SP-BS22-LR-Designed-Bookshelf-Loudspeakers/dp/B008NCD2LG.

[2]

D. Arp, E. Quiring, C. Wressnegger, and K. Rieck. 2017. Privacy Threats through Ultrasonic Side Channels on Mobile Devices. In 2017 IEEE European Symposium on Security and Privacy (EuroS P). 35–47.

[3]

Avisoft. 2019. Knowles FG Response Curve. Website. Retrieved September 20, 2019 from https://www.avisoft.com/usg/KnowlesFGO.htm.

[4]

Yusuf Aytar, Carl Vondrick, and Antonio Torralba. 2016. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems. 892–900.

[5]

BlueDesigns. 2019. Blue Yeti Microphone. Website. Retrieved September 20, 2019 from https://www.bluedesigns.com/products/yeti/#.

[6]

Emre Cakır, Giambattista Parascandolo, Toni Heittola, Heikki Huttunen, and Tuomas Virtanen. 2017. Convolutional recurrent neural networks for polyphonic sound event detection. IEEE/ACM Transactions on Audio, Speech, and Language Processing 25, 6(2017), 1291–1303.

Digital Library

[7]

Ray-I Chang, Chien-Chang Huang, Liang-Bin Lai, and Chia-Yun Lee. 2018. Query-Based Machine Learning Model for Data Analysis of Infrasonic Signals in Wireless Sensor Networks. In Proceedings of the 2Nd International Conference on Digital Signal Processing (Tokyo, Japan) (ICDSP 2018). ACM, New York, NY, USA, 114–118. https://doi.org/10.1145/3193025.3193031

Digital Library

[8]

Dodotronic. 2019. Ultramic384K. Website. Retrieved September 20, 2019 from https://www.dodotronic.com/ultramic384k/.

[9]

Benjamin Elizalde, Rohan Badlani, Ankit Shah, Anurag Kumar, and Bhiksha Raj. 2018. Nels-never-ending learner of sounds. (2018). arxiv:1801.05544

[10]

Antti J Eronen, Vesa T Peltonen, Juha T Tuomi, Anssi P Klapuri, Seppo Fagerlund, Timo Sorsa, Gaëtan Lorho, and Jyri Huopaniemi. 2005. Audio-based context recognition. IEEE Transactions on Audio, Speech, and Language Processing 14, 1(2005), 321–329.

Digital Library

[11]

Pasquale Foggia, Nicolai Petkov, Alessia Saggese, Nicola Strisciuglio, and Mario Vento. 2015. Reliable detection of audio events in highly noisy environments. Pattern Recognition Letters 65 (2015), 22–28.

Digital Library

[12]

Frederic Font, Gerard Roma, and Xavier Serra. 2013. Freesound Technical Demo. In Proceedings of the 21st ACM International Conference on Multimedia (Barcelona, Spain) (MM ’13). ACM, New York, NY, USA, 411–412. https://doi.org/10.1145/2502081.2502245

Digital Library

[13]

Jon E. Froehlich, Eric Larson, Tim Campbell, Conor Haggerty, James Fogarty, and Shwetak N. Patel. 2009. HydroSense: Infrastructure-mediated Single-point Sensing of Whole-home Water Activity. In Proceedings of the 11th International Conference on Ubiquitous Computing (Orlando, Florida, USA) (UbiComp ’09). ACM, New York, NY, USA, 235–244. https://doi.org/10.1145/1620545.1620581

Digital Library

[14]

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 776–780.

Digital Library

[15]

Pascal Getreuer, Chet Gnegy, Richard F Lyon, and Rif A Saurous. 2017. Ultrasonic communication using consumer hardware. IEEE Transactions on Multimedia 20, 6 (2017), 1277–1290.

[16]

Mayank Goel, Elliot Saba, Maia Stiber, Eric Whitmire, Josh Fromm, Eric C. Larson, Gaetano Borriello, and Shwetak N. Patel. 2016. SpiroCall: Measuring Lung Function over a Phone Call. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). ACM, New York, NY, USA, 5675–5685. https://doi.org/10.1145/2858036.2858401

Digital Library

[17]

Sidhant Gupta, Matthew S. Reynolds, and Shwetak N. Patel. 2010. ElectriSense: Single-point Sensing Using EMI for Electrical Event Detection and Classification in the Home. In Proceedings of the 12th ACM International Conference on Ubiquitous Computing (Copenhagen, Denmark) (UbiComp ’10). ACM, New York, NY, USA, 139–148. https://doi.org/10.1145/1864349.1864375

Digital Library

[18]

Infiltec. 2019. Infiltec INFRA20 Infrasound Sensor. Website. Retrieved September 20, 2019 from http://www.infiltec.com/Infrasound@home/.

[19]

Yasha Iravantchi, Mayank Goel, and Chris Harrison. 2019. BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 15, 10 pages. https://doi.org/10.1145/3290605.3300245

Digital Library

[20]

Yasha Iravantchi, Yang Zhang, Evi Bernitsas, Mayank Goel, and Chris Harrison. 2019. Interferi: Gesture Sensing Using On-Body Acoustic Interferometry. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, Article 276, 13 pages. https://doi.org/10.1145/3290605.3300506

Digital Library

[21]

KeySight. 2019. Agilent 33521A Function Generator. Website. Retrieved September 20, 2019 from https://www.keysight.com/en/pd-1871159-pn-33521A/function-arbitrary-waveform- generator-30-mhz?&cc=US&lc=eng.

[22]

Mone Kijima, Yuta Miyagawa, Hayato Oshita, Norihisa Segawa, Masato Yazawa, and Masa-yuki Yamamoto. 2018. Multiple door opening/closing detection system using infrasound sensor. In Proceedings of the 17th ACM/IEEE International Conference on Information Processing in Sensor Networks. IEEE Press, 126–127.

Digital Library

[23]

Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). ACM, New York, NY, USA, 213–224. https://doi.org/10.1145/3242587.3242609

Digital Library

[24]

Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Synthetic Sensors: Towards General-Purpose Sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 3986–3999. https://doi.org/10.1145/3025453.3025773

Digital Library

[25]

Eric C. Larson, TienJui Lee, Sean Liu, Margaret Rosenfeld, and Shwetak N. Patel. 2011. Accurate and Privacy Preserving Cough Sensing Using a Low-Cost Microphone. In Proceedings of the 13th International Conference on Ubiquitous Computing (Beijing, China) (UbiComp ’11). Association for Computing Machinery, New York, NY, USA, 375–384. https://doi.org/10.1145/2030112.2030163

Digital Library

[26]

S. Lee, E. Nemati, and J. Kuang. 2018. Configurable Pulmonary-Tuned Privacy Preservation Algorithm for Mobile Devices. In 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). 1107–1112.

[27]

Lydia Manikonda, Aditya Deotale, and Subbarao Kambhampati. 2018. What’s up with Privacy? User Preferences and Privacy Concerns in Intelligent Personal Assistants. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (New Orleans, LA, USA) (AIES ’18). Association for Computing Machinery, New York, NY, USA, 229–235. https://doi.org/10.1145/3278721.3278773

Digital Library

[28]

Jess McIntosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition Using Ultrasound Imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). ACM, New York, NY, USA, 1923–1934. https://doi.org/10.1145/3025453.3025807

Digital Library

[29]

Bjoern H Menze, B Michael Kelm, Ralf Masuch, Uwe Himmelreich, Peter Bachert, Wolfgang Petrich, and Fred A Hamprecht. 2009. A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data. BMC bioinformatics 10, 1 (2009), 213.

[30]

The Cornell Lab of Ornithology. 2019. Elephant Listening Project: Deep into Infrasound. Website. Retrieved September 20, 2019 from https://elephantlisteningproject.org/all-about-infrasound/.

[31]

Future of Privacy Forum. 2020. ”Always On: Privacy Implications of Microphone-Enabled Devices”. Website. Retrieved Sept 17, 2020 from https://www.ftc.gov/system/files/documents/public_comments/2016/08/00003-128652.pdf.

[32]

PUIAudio. 2019. PUI Audio AB1290B. Website. Retrieved September 20, 2019 from http://www.puiaudio.com/product-detail.aspx?categoryId=5&partnumber=AB1290B.

[33]

Dale Purves. 2001. The Audible Spectrum. Sunderland Associates, Sunderland, MA, USA.

[34]

Rolf Quam, Ignacio Martínez, Manuel Rosa, Alejandro Bonmatí, Carlos Lorenzo, Darryl J. de Ruiter, Jacopo Moggi-Cecchi, Mercedes Conde Valverde, Pilar Jarabo, Colin G. Menter, J. Francis Thackeray, and Juan Luis Arsuaga. 2015. Early hominin auditory capacities. Science Advances 1, 8 (2015). https://doi.org/10.1126/sciadv.1500355 arXiv:https://advances.sciencemag.org/content/1/8/e1500355.full.pdf

[35]

RecordingHacks. 2019. Blue Yeti Microphone Response Curve. Website. Retrieved September 20, 2019 from https://doi.org/10.1145/3081333.3081366.

Digital Library

[36]

Nirupam Roy, Haitham Hassanieh, and Romit Roy Choudhury. 2017. BackDoor: Making Microphones Hear Inaudible Sounds. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services(Niagara Falls, New York, USA) (MobiSys ’17). Association for Computing Machinery, New York, NY, USA, 2–14. https://doi.org/10.1145/3081333.3081366

Digital Library

[37]

Yvan Saeys, Thomas Abeel, and Yves Van de Peer. 2008. Robust feature selection using ensemble feature selection techniques. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 313–325.

[38]

Florian Sainjeon, Sébastien Gaboury, and Bruno Bouchard. 2016. Real-Time Indoor Localization in Smart Homes Using Ultrasound Technology. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (Corfu, Island, Greece) (PETRA ’16). ACM, New York, NY, USA, Article 6, 4 pages. https://doi.org/10.1145/2910674.2910718

Digital Library

[39]

Valkyrie Savage, Andrew Head, Björn Hartmann, Dan B. Goldman, Gautham Mysore, and Wilmot Li. 2015. Lamello: Passive Acoustic Sensing for Tangible Input Components. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 1277–1280. https://doi.org/10.1145/2702123.2702207

Digital Library

[40]

ShotSpotter. 2019. ShotSpotter Gunshot Detection, Location and Forensic Analysis. Website. Retrieved September 20, 2019 from https://www.shotspotter.com.

[41]

Takeshi Sugawara, Benjamin Cyr, Sara Rampazzi, Daniel Genkin, and Kevin Fu. 2020. Light Commands: Laser-Based Audio Injection Attacks on Voice-Controllable Systems. In 29th USENIX Security Symposium (USENIX Security 20).

[42]

Ingo R Titze and Daniel W Martin. 1998. Principles of voice production.

[43]

Wikipedia. 2020. ”Wikipedia: Hearing range”. Website. Retrieved Sept 17, 2020 from https://en.wikipedia.org/wiki/Hearing_range.

[44]

Wikipedia. 2020. ”Wikipedia: Mel scale”. Website. Retrieved Sept 17, 2020 from https://en.wikipedia.org/wiki/Mel_scale.

[45]

Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, Omer Inan, and Gregory D. Abowd. 2018. FingerPing: Recognizing Fine-grained Hand Poses Using Active Acoustic On-body Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, Article 437, 10 pages. https://doi.org/10.1145/3173574.3174011

Digital Library

[46]

Hanbin Zhang, Chen Song, Aosen Wang, Chenhan Xu, Dongmei Li, and Wenyao Xu. 2019. PDVocal: Towards Privacy-Preserving Parkinson’s Disease Detection Using Non-Speech Body Sounds. In The 25th Annual International Conference on Mobile Computing and Networking (Los Cabos, Mexico) (MobiCom ’19). Association for Computing Machinery, New York, NY, USA, Article 16, 16 pages. https://doi.org/10.1145/3300061.3300125

Digital Library

Cited By

Alvarez MArjona LIglesias Martínez MBahillo A(2024)Automatic classification of the physical surface in sound uroflowmetry using machine learning methodsEURASIP Journal on Audio, Speech, and Music Processing10.1186/s13636-024-00332-y2024:1Online publication date: 16-Feb-2024
https://dl.acm.org/doi/10.1186/s13636-024-00332-y
Xu ABiehl JLee A(2024)Getting it Just Right: Towards Balanced Utility, Privacy, and Equity in Shared Space SensingACM Transactions on Internet of Things10.1145/36484795:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3648479
Boovaraghavan SZhou HGoel MAgarwal Y(2024)KirigamiProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435028:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643502
Show More Cited By

Index Terms

PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition
1. Hardware

Index terms have been assigned to the content through auto-classification.

Recommendations

Ubicoustics: Plug-and-Play Acoustic Activity Recognition
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology

Despite sound being a rich source of information, computing devices with microphones do not leverage audio to glean useful insights about their physical and social context. For example, a smart speaker sitting on a kitchen countertop cannot figure out ...
A review of transduction techniques used in acoustics echo cancellation
AMTA'07: Proceedings of the 8th WSEAS international conference on Acoustics & music: theory & applications

In this review we shall describe the most typical and conventional transducers used in the acoustic echo cancellation systems and the effect of the transducer characteristics on achieving echo cancellation. We shall also discuss the typical design ...
BackDoor: Making Microphones Hear Inaudible Sounds
MobiSys '17: Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services

Consider sounds, say at 40kHz, that are completely outside the human's audible range (20kHz), as well as a microphone's recordable range (24kHz). We show that these high frequency sounds can be designed to become recordable by unmodified microphones, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
1,217
Total Downloads

Downloads (Last 12 months)229
Downloads (Last 6 weeks)33

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alvarez MArjona LIglesias Martínez MBahillo A(2024)Automatic classification of the physical surface in sound uroflowmetry using machine learning methodsEURASIP Journal on Audio, Speech, and Music Processing10.1186/s13636-024-00332-y2024:1Online publication date: 16-Feb-2024
https://dl.acm.org/doi/10.1186/s13636-024-00332-y
Xu ABiehl JLee A(2024)Getting it Just Right: Towards Balanced Utility, Privacy, and Equity in Shared Space SensingACM Transactions on Internet of Things10.1145/36484795:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3648479
Boovaraghavan SZhou HGoel MAgarwal Y(2024)KirigamiProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435028:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643502
Jain PXu ADownes TKim IKhan TBiehl JLee A(2024)Understanding Perceived Utility and Comfort of In-Home General-Purpose Sensing through Progressive ExposureProceedings of the ACM on Human-Computer Interaction10.1145/36374328:CSCW1(1-32)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637432
Kim TKim HRoberts AJacobs MKay M(2024)Opportunities in Mental Health Support for Informal Dementia Caregivers Suffering from Verbal AgitationProceedings of the ACM on Human-Computer Interaction10.1145/36373818:CSCW1(1-26)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637381
Iravantchi YSample A(2024)T4Train: Rapid Prototyping of ML-Driven Interactive ApplicationsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636269(1-4)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636269
Kim MLee JKoh YLee CLee UKim A(2024)Interrupting for Microlearning: Understanding Perceptions and Interruptibility of Proactive Conversational Microlearning ServicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642778(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642778
Diraco GRescio GCaroppo AManni ALeone A(2023)Human Action Recognition in Smart Living Services and Applications: Context Awareness, Data Availability, Personalization, and PrivacySensors10.3390/s2313604023:13(6040)Online publication date: 29-Jun-2023
https://doi.org/10.3390/s23136040
Huang JChhabria HJain D(2023)“Not There Yet”: Feasibility and Challenges of Mobile Sound Recognition to Support Deaf and Hard-of-Hearing PeopleProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608431(1-14)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3608431
Antar AKratz ABanovic N(2023)Behavior Modeling Approach for Forecasting Physical Functioning of People with Multiple SclerosisProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808877:1(1-29)Online publication date: 28-Mar-2023
https://dl.acm.org/doi/10.1145/3580887
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents