research-article

Public Access

Lasagna: towards deep hierarchical understanding and searching over mobile sensing data

Authors:

Yunhao LiuAuthors Info & Claims

MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking

Pages 334 - 347

https://doi.org/10.1145/2973750.2973752

Published: 03 October 2016 Publication History

All formats PDF

Abstract

The proliferation of mobile devices has enabled extensive mobile-data supported applications, e.g., exercise and activity recognition and quantification. Typically, these applications need predefined features and are only applicable to predefined activities. In this work, we address the issue of deep understanding of arbitrary activities and semantic searching of any activity over massive mobile sensing data. The challenges stem from the rich dynamics and the wide-spectrum of activities that a human being could perform. We propose a hierarchical activity representation, extract common bases of motion data in an unsupervised manner by leveraging the power of deep neural networks, and propose a universal multi-resolution representation for all activities without prior knowledge. Based on this representation, we design an innovative system Lasagna to manage and search motion data semantically. We implement a prototype system and our comprehensive evaluations show that our system can achieve highly accurate activity classification (with precision 98.9%) and search (with recall almost 100% and precision about 90%) over a diverse set of activities.

References

[1]

Anjum, A., and Ilyas, M. U. Activity recognition using smartphone sensors. In 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC), pp. 914--919.

[2]

Bao, L., and Intille, S. S. Activity recognition from user-annotated acceleration data. In International Conference on Pervasive Computing (2004), Springer, pp. 1--17.

[3]

Chen, K.-Y., Ashbrook, D., Goel, M., Lee, S.-H., and Patel, S. Airlink: sharing files between multiple devices using in-air gestures. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 565--569.

Digital Library

[4]

Cheng, H.-T. Learning and Recognizing The Hierarchical and Sequential Structure of Human Activities. PhD thesis, CARNEGIE MELLON UNIVERSITY, 2013.

Digital Library

[5]

George, E. I., and McCulloch, R. E. Variable selection via gibbs sampling. Journal of the American Statistical Association 88, 423 (1993), 881--889.

[6]

Guo, Y., Yang, L., Ding, X., Han, J., and Liu, Y. Opensesame: Unlocking smart phone through handshaking biometrics. In Proceedings of IEEE INFOCOM (2013), pp. 365--369.

[7]

Harbach, M., von Zezschwitz, E., Fichtner, A., De Luca, A., and Smith, M. It's a hard lock life: A field study of smartphone (un) locking behavior and risk perception. In SOUPS 2014, pp. 213--230.

[8]

Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation 14, 8 (2002), 1771--1800.

Digital Library

[9]

Hong, F., Wei, M., You, S., Feng, Y., and Guo, Z. Waving authentication: Your smartphone authenticate you on motion gesture. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (2015), pp. 263--266.

Digital Library

[10]

Huynh, T., Fritz, M., and Schiele, B. Discovery of activity patterns using topic models. In Proceedings of the 10th international conference on Ubiquitous computing (2008), ACM, pp. 10--19.

Digital Library

[11]

Jiang, W., and Yin, Z. Human activity recognition using wearable sensors by deep convolutional neural networks. In Proceedings of the 23rd Annual ACM Conference on Multimedia Conference (2015), pp. 1307--1310.

Digital Library

[12]

Kiros, R., Salakhutdinov, R., and Zemel, R. Multimodal neural language models. In Proceedings of the 31st International Conference on Machine Learning (2014), pp. 595--603.

[13]

Kwapisz, J. R., Weiss, G. M., and Moore, S. A. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12, 2 (2011), 74--82.

Digital Library

[14]

Kwon, Y., Kang, K., and Bae, C. Unsupervised learning for human activity recognition using smartphone sensors. Expert Systems with Applications 41, 14 (2014), 6067--6074.

[15]

Lane, N. D., and Georgiev, P. Can deep learning revolutionize mobile sensing? In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (2015), ACM, pp. 117--122.

Digital Library

[16]

Lane, N. D., Georgiev, P., and Qendro, L. Deepear: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 283--294.

Digital Library

[17]

Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning (2009), ACM, pp. 609--616.

Digital Library

[18]

Lee, H., Pham, P., Largman, Y., and Ng, A. Y. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in neural information processing systems (2009), pp. 1096--1104.

Digital Library

[19]

Li, Z., Li, M., Wang, J., and Cao, Z. Ubiquitous data collection for mobile users in wireless sensor networks. In INFOCOM, 2011 Proceedings IEEE (2011), IEEE, pp. 2246--2254.

[20]

Liu, J., Zhong, L., Wickramasuriya, J., and Vasudevan, V. User evaluation of lightweight user authentication with a single tri-axis accelerometer. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (2009), ACM, p. 15.

Digital Library

[21]

Liu, X., Zhou, Z., Diao, W., Li, Z., and Zhang, K. When good becomes evil: Keystroke inference with smartwatch. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015), pp. 1273--1285.

Digital Library

[22]

Lütkepohl, H. New introduction to multiple time series analysis. Springer Science & Business Media, 2005.

Digital Library

[23]

Matsubara, Y., Sakurai, Y., and Faloutsos, C. Autoplait: Automatic mining of co-evolving time sequences. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pp. 193--204.

Digital Library

[24]

Michalevsky, Y., Boneh, D., and Nakibly, G. Gyrophone: Recognizing speech from gyroscope signals. In 23rd USENIX Security Symposium (2014), pp. 1053--1067.

Digital Library

[25]

Owusu, E., Han, J., Das, S., Perrig, A., and Zhang, J. Accessory: password inference using accelerometers on smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications (2012), ACM.

Digital Library

[26]

Parate, A., Chiu, M.-C., Chadowitz, C., Ganesan, D., and Kalogerakis, E. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (2014), pp. 149--161.

Digital Library

[27]

Rabiner, L. R., and Juang, B.-H. An introduction to hidden markov models. ASSP Magazine, IEEE 3, 1 (1986), 4--16.

[28]

Ramanathan, V., Tang, K., Mori, G., and Fei-Fei, L. Learning temporal embeddings for complex video analysis. In The IEEE International Conference on Computer Vision (2015).

Digital Library

[29]

Ren, Y., Chen, Y., Chuah, M. C., and Yang, J. User verification leveraging gait recognition for smartphone enabled mobile healthcare systems. IEEE Transactions on Mobile Computing 14, 9 (2015), 1961--1974.

Digital Library

[30]

Roy, N., Wang, H., and Roy Choudhury, R. I am a smartphone and i can tell my user's walking direction. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (2014), ACM, pp. 329--342.

Digital Library

[31]

Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 815--823.

[32]

Shahzad, M., Liu, A. X., and Samuel, A. Secure unlocking of mobile touch screen devices by simple gestures: you can see it but you can not do it. In Proceedings of the 19th annual international conference on Mobile computing & networking (2013), ACM, pp. 39--50.

Digital Library

[33]

Shoaib, M., Bosch, S., Incel, O. D., Scholten, H., and Havinga, P. J. Fusion of smartphone motion sensors for physical activity recognition. Sensors 14, 6 (2014), 10146--10176.

[34]

Sun, Y., Chen, Y., Wang, X., and Tang, X. Deep learning face representation by joint identification-verification. In Advances in Neural Information Processing Systems 27. 2014, pp. 1988--1996.

Digital Library

[35]

Sun, Y., Liang, D., Wang, X., and Tang, X. Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873 (2015).

[36]

Sun, Y., Wang, X., and Tang, X. Deeply learned face representations are sparse, selective, and robust. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 2892--2900.

[37]

Sun, Z., Purohit, A., Bose, R., and Zhang, P. Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services (2013), ACM, pp. 263--276.

Digital Library

[38]

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. arXiv preprint arXiv:1409.4842 (2014).

[39]

Tung, Y.-C., and Shin, K. G. Echotag: accurate infrastructure-free indoor location tagging with smartphones. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, pp. 525--536.

Digital Library

[40]

Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), pp. 3156--3164.

[41]

Wang, H., Lai, T. T.-T., and Roy Choudhury, R. Mole: Motion leaks through smartwatch sensors. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, pp. 155--166.

Digital Library

[42]

Wang, P., Wang, H., and Wang, W. Finding semantics in time series. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data (2011), pp. 385--396.

Digital Library

[43]

Wang, R., Harari, G., Hao, P., Zhou, X., and Campbell, A. T. Smartgpa: how smartphones can assess and predict academic performance of college students. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 295--306.

Digital Library

[44]

Wang, X., and Gupta, A. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision (2015), pp. 2794--2802.

Digital Library

[45]

Wang, X., Smith, K., and Hyndman, R. Characteristic-based clustering for time series data. Data mining and knowledge Discovery 13, 3 (2006), 335--364.

Digital Library

[46]

Wu, F.-J., Chu, F.-I., and Tseng, Y.-C. Cyber-physical handshake. In ACM SIGCOMM Computer Communication Review (2011), vol. 41, pp. 472--473.

Digital Library

[47]

Xu, Z., Bai, K., and Zhu, S. Taplogger: Inferring user inputs on smartphone touchscreens using on-board motion sensors. In Proceedings of the fifth ACM conference on Security and Privacy in Wireless and Mobile Networks (2012), pp. 113--124.

Digital Library

[48]

Zhang, L., Liu, K., Jiang, Y., Li, X.-Y., Liu, Y., and Yang, P. Montage: Combine frames with movement continuity for realtime multi-user tracking. In Proceedings of IEEE INFOCOM (2014), pp. 799--807.

[49]

Zhang, L., Pathak, P. H., Wu, M., Zhao, Y., and Mohapatra, P. Accelword: Energy efficient hotword detection through accelerometer. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (2015), ACM, pp. 301--315.

Digital Library

[50]

Zheng, N., Bai, K., Huang, H., and Wang, H. You are how you touch: User verification on smartphones via tapping behaviors. In IEEE 22nd International Conference on Network Protocols (ICNP) (2014), pp. 221--232.

Digital Library

Cited By

Siam SAhn HLiu LAlam SShen HCao ZShroff NKrishnamachari BSrivastava MZhang M(2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1145/3690639
Salehzadeh Niksirat KVelykoivanenko LZufferey NCherubini MHuguenin KHumbert M(2024)Wearable Activity Trackers: A Survey on Utility, Privacy, and SecurityACM Computing Surveys10.1145/364509156:7(1-40)Online publication date: 8-Feb-2024
https://dl.acm.org/doi/10.1145/3645091
Yuan MZhang LHe FTong XSong MXu ZLi X(2024)InFi: End-to-End Learning to Filter Input for Resource-Efficiency in Mobile-Centric InferenceIEEE Transactions on Mobile Computing10.1109/TMC.2023.3275981(1-16)Online publication date: 2024
https://doi.org/10.1109/TMC.2023.3275981
Show More Cited By

Index Terms

Lasagna: towards deep hierarchical understanding and searching over mobile sensing data
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools
    2. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Mobile computing
2. Information systems
  1. World Wide Web
    1. Web searching and information discovery
      1. Web search engines

Recommendations

Capturing Daily Student Life by Recognizing Complex Activities Using Smartphones
MobiQuitous 2017: Proceedings of the 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

In-depth understanding of student life is essential to proactively support students in their academic educations. However, there is no work that identifies and recognizes a sufficient set of activities to capture a daily student life since complex ...
A Survey on Deep Learning for Human Activity Recognition
Human activity recognition is a key to a lot of applications such as healthcare and smart home. In this study, we provide a comprehensive survey on recent advances and challenges in human activity recognition (HAR) with deep learning. Although there are ...
From Multimedia Logs to Personal Chronicles
MM '17: Proceedings of the 25th ACM international conference on Multimedia

Multimodal data streams are essential for analyzing personal life, environmental conditions, and social situations. Since these data streams have different granularities and semantics, the semantic gap becomes even more formidable. To make sense of all ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

MobiCom '16: Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking

October 2016

532 pages

ISBN:9781450342261

DOI:10.1145/2973750

General Chairs:
Yingying Chen
Stevens Institute of Technology
,
Marco Gruteser
Rutgers University
,
Program Chairs:
Y. Charlie Hu
Purdue University, USA
,
Karthik Sundaresan
NEC Labs Princeton, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

MobiCom'16

MobiCom'16: The 22nd Annual International Conference on Mobile Computing and Networking

October 3 - 7, 2016

New York, New York City

Acceptance Rates

MobiCom '16 Paper Acceptance Rate 31 of 226 submissions, 14%;

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
2,467
Total Downloads

Downloads (Last 12 months)172
Downloads (Last 6 weeks)35

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Siam SAhn HLiu LAlam SShen HCao ZShroff NKrishnamachari BSrivastava MZhang M(2024)Artificial Intelligence of Things: A SurveyACM Transactions on Sensor Networks10.1145/369063921:1(1-75)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1145/3690639
Salehzadeh Niksirat KVelykoivanenko LZufferey NCherubini MHuguenin KHumbert M(2024)Wearable Activity Trackers: A Survey on Utility, Privacy, and SecurityACM Computing Surveys10.1145/364509156:7(1-40)Online publication date: 8-Feb-2024
https://dl.acm.org/doi/10.1145/3645091
Yuan MZhang LHe FTong XSong MXu ZLi X(2024)InFi: End-to-End Learning to Filter Input for Resource-Efficiency in Mobile-Centric InferenceIEEE Transactions on Mobile Computing10.1109/TMC.2023.3275981(1-16)Online publication date: 2024
https://doi.org/10.1109/TMC.2023.3275981
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: Oct-2025
https://doi.org/10.1109/COMST.2024.3357591
Wang GGuo HWang YChen BZhou CYan Q(2024)Protecting Activity Sensing Data Privacy Using Hierarchical Information Dissociation2024 IEEE Conference on Communications and Network Security (CNS)10.1109/CNS62487.2024.10735551(1-9)Online publication date: 30-Sep-2024
https://doi.org/10.1109/CNS62487.2024.10735551
Wang HZhou HCheng S(2024)Dynamical system prediction from sparse observations using deep neural networks with Voronoi tessellation and physics constraintComputer Methods in Applied Mechanics and Engineering10.1016/j.cma.2024.117339432(117339)Online publication date: Dec-2024
https://doi.org/10.1016/j.cma.2024.117339
Guo XWang YCheng JChen YGuo XWang YCheng JChen Y(2024)Personalized Fitness Assistance Using Commodity WiFiMobile Technologies for Smart Healthcare System Design10.1007/978-3-031-57345-3_3(49-82)Online publication date: 3-Jul-2024
https://doi.org/10.1007/978-3-031-57345-3_3
Zhu BWei QLi LYang ZLiu WYou ZZhou JLi PSong JLiu SLi D(2023)RF SignWireless Communications & Mobile Computing10.1155/2023/13411932023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/1341193
Zhang LZheng DYuan MHan FWu ZLiu MLi X(2023)MultiSense: Cross-labelling and Learning Human Activities Using Multimodal Sensing DataACM Transactions on Sensor Networks10.1145/357826719:3(1-26)Online publication date: 17-Apr-2023
https://dl.acm.org/doi/10.1145/3578267
Li ZZhang LYuan MSong MSong Q(2023)Efficient Deep Ensemble Inference via Query Difficulty-dependent Task Scheduling2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00082(1005-1018)Online publication date: Apr-2023
https://doi.org/10.1109/ICDE55515.2023.00082
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

EPUB

View this article in ePub.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten