research-article

Leveraging Active Learning and Conditional Mutual Information to Minimize Data Annotation in Human Activity Recognition

Authors:

Rebecca Adaimi,

Edison ThomazAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 3

Article No.: 70, Pages 1 - 23

https://doi.org/10.1145/3351228

Published: 09 September 2019 Publication History

Abstract

A difficulty in human activity recognition (HAR) with wearable sensors is the acquisition of large amounts of annotated data for training models using supervised learning approaches. While collecting raw sensor data has been made easier with advances in mobile sensing and computing, the process of data annotation remains a time-consuming and onerous process. This paper explores active learning as a way to minimize the labor-intensive task of labeling data. We train models with active learning in both offline and online settings with data from 4 publicly available activity recognition datasets and show that it performs comparably to or better than supervised methods while using around 10% of the training data. Moreover, we introduce a method based on conditional mutual information for determining when to stop the active learning process while maximizing recognition performance. This is an important issue that arises in practice when applying active learning to unlabeled datasets.

Supplementary Material

adaimi (adaimi.zip)

Supplemental movie, appendix, image and software files for, Leveraging Active Learning and Conditional Mutual Information to Minimize Data Annotation in Human Activity Recognition

Download
99.56 KB

References

[1]

Zahraa Said Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2015. Adaptive Mobile Activity Recognition System with Evolving Data Streams. Neurocomput. 150, PA (Feb. 2015), 304--317.

Digital Library

[2]

Hande Alemdar, Tim L. M. van Kasteren, and Cem Ersoy. 2011. Using Active Learning to Allow Activity Recognition on a Large Scale. In Proceedings of the Second International Conference on Ambient Intelligence (AmI'11). Springer-Verlag, Berlin, Heidelberg, 105--114.

Digital Library

[3]

Dana Angluin. 1988. Queries and Concept Learning. Mach. Learn. 2, 4 (April 1988), 319--342.

[4]

Les Atlas, David Cohn, Richard Ladner, M. A. El-Sharkawi, and R. J. Marks, II. 1990. Advances in Neural Information Processing Systems 2. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, Chapter Training Connectionist Networks with Queries and Selective Sampling, 566--573. http://dl.acm.org/citation.cfm?id=109230.109294

Digital Library

[5]

M. Bachlin, D. Roggen, G. Troster, M. Plotnik, N. Inbar, I. Meidan, T. Herman, M. Brozgol, E. Shaviv, N. Giladi, and J. M. Hausdorff. 2009. Potentials of Enhanced Context Awareness in Wearable Assistants for Parkinson's Disease Patients with the Freezing of Gait Syndrome. In 2009 International Symposium on Wearable Computers. 123--130.

Digital Library

[6]

Salikh Bagaveyev and Diane J. Cook. 2014. Designing and Evaluating Active Learning Methods for Activity Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). ACM, New York, NY, USA, 469--478.

Digital Library

[7]

Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: using wearable sensors to detect eating episodes in unconstrained environments. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 1, 3 (2017), 37.

Digital Library

[8]

Klaus Brinker. 2003. Incorporating Diversity in Active Learning with Support Vector Machines. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning (ICML'03). AAAI Press, 59--66. http://dl.acm.org/citation.cfm?id=3041838.3041846

Digital Library

[9]

Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A Tutorial on Human Activity Recognition Using Body-worn Inertial Sensors. ACM Comput. Surv. 46, 3, Article 33 (Jan. 2014), 33 pages.

Digital Library

[10]

Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José Del R. Millán, and Daniel Roggen. 2013. The Opportunity Challenge: A Benchmark Database for On-body Sensor-based Activity Recognition. Pattern Recogn. Lett. 34, 15 (Nov. 2013), 2033--2042.

Digital Library

[11]

Keum San Chun, Sarnab Bhattacharya, and Edison Thomaz. 2018. Detecting eating episodes by tracking jawbone movements with a non-contact wearable sensor. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 4.

Digital Library

[12]

Keum San Chun, Ashley B. Sanders, Rebecca Adaimi, Necole Streeper, David E. Conroy, and Edison Thomaz. 2019. Towards a Generalizable Method for Detecting Fluid Intake with Wrist-mounted Sensors and Adaptive Segmentation. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI '19). ACM, New York, NY, USA, 80--85.

Digital Library

[13]

Federico Cruciani, Ian Cleland, Chris Nugent, Paul McCullagh, Kåre Synnes, and Josef Hallberg. 2018. Automatic Annotation for Human Activity Recognition in Free Living Using a Smartphone. Sensors 18, 7 (2018).

[14]

Toon De Pessemier and Luc Martens. 2018. Heart rate monitoring, activity recognition, and recommendation for e-coaching. Multimedia Tools and Applications (26 Jan 2018).

Digital Library

[15]

A. Diete, T. Sztyler, and H. Stuckenschmidt. 2017. A smart data annotation tool for multi-sensor activity recognition. In 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). 111--116.

[16]

Y. Gu, Z. Jin, and S. C. Chiu. 2015. Active learning combining uncertainty and diversity for multi-class image classification. IET Computer Vision 9, 3 (2015), 400--407.

[17]

Tianxu He, Zhang Kui, Jie Xin, Pengpeng Zhao, Jian Wu, Xuefeng Xian, Chunhua Li, and Zhiming Cui. 2014. An Active Learning Approach with Uncertainty, Representativeness, and Diversity. The Scientific World Journal 2014 (08 2014), 827586.

[18]

A. Holub, P. Perona, and M. C. Burl. 2008. Entropy-based active learning for object recognition. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. 1--8.

[19]

S. C. Hsu, C. H. Chuang, C. L. Huang, P. R. Teng, and M. J. Lin. 2018. A video-based abnormal human behavior detection for psychiatric patient monitoring. In 2018 International Workshop on Advanced Image Technology (IWAIT). 1--4.

[20]

Jie Jiang, Riccardo Pozza, Kristrún Gunnarsdóttir, G. Nigel Gilbert, and Klaus Moessner. 2017. Using Sensors to Study Home Activities. J. Sensor and Actuator Networks 6 (2017), 32.

[21]

David D. Lewis and William A. Gale. 1994. A Sequential Algorithm for Training Text Classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '94). Springer-Verlag New York, Inc., New York, NY, USA, 3--12. http://dl.acm.org/citation.cfm?id=188490.188495

Digital Library

[22]

R. Liu, T. Chen, and L. Huang. 2010. Research on human activity recognition based on active learning. In 2010 International Conference on Machine Learning and Cybernetics, Vol. 1. 285--290.

[23]

B. Longstaff, S. Reddy, and D. Estrin. 2010. Improving activity classification for health applications on mobile devices using active and semi-supervised learning. In 2010 4th International Conference on Pervasive Computing Technologies for Healthcare. 1--7.

[24]

Donald McMillan, Barry Brown, Airi Lampinen, Moira McGregor, Eve Hoggan, and Stefania Pizza. 2017. Situating Wearables: Smartwatch Use in Context. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3582--3594.

Digital Library

[25]

Christopher Merck, Christina Maher, Mark Mirtchouk, Min Zheng, Yuxiao Huang, and Samantha Kleinberg. 2016. Multimodality Sensing for Eating Recognition. In Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth '16). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 130--137. http://dl.acm.org/citation.cfm?id=3021319.3021339

[26]

Daniela Micucci, Marco Mobilio, and Paolo Napoletano. 2016. UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones. CoRR abs/1611.07688 (2016). arXiv:1611.07688 http://arxiv.org/abs/1611.07688

[27]

T. Miu, P. Missier, and T. Plötz. 2015. Bootstrapping Personalised Human Activity Recognition Models Using Online Active Learning. In 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing. 1138--1147.

[28]

Georg Ogris, Paul Lukowicz, Thomas Stiefmeier, and Gerhard Tröster. 2012. Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking. Pattern Analysis and Applications 15, 1 (01 Feb 2012), 87--111.

Digital Library

[29]

J. Qi, P. Yang, M. Hanneghan, S. Tang, and B. Zhou. 2018. A Hybrid Hierarchical Framework for Gym Physical Activity Recognition and Measurement Using Wearable Sensors. IEEE Internet of Things Journal (2018), 1--1.

[30]

Julien Rebetez, Héctor F. Satizábal, and Andres Perez-Uribe. 2013. Reducing User Intervention in Incremental Activityrecognition for Assistive Technologies. In Proceedings of the 2013 International Symposium on Wearable Computers (ISWC '13). ACM, New York, NY, USA, 29--32.

Digital Library

[31]

Attila Reiss and Didier Stricker. 2012. Creating and Benchmarking a New Dataset for Physical Activity Monitoring. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '12). ACM, New York, NY, USA, Article 40, 8 pages.

Digital Library

[32]

J. Ryder, B. Longstaff, S. Reddy, and D. Estrin. 2009. Ambulation: A Tool for Monitoring Mobility Patterns over Time Using Mobile Phones. In 2009 International Conference on Computational Science and Engineering, Vol. 4. 927--931.

Digital Library

[33]

A. Sathyanarayana, F. Ofli, L. Fernandez-Luque, J. Srivastava, A. Elmagarmid, T. Arora, and S. Taheri. 2016. Robust Automated Human Activity Recognition and Its Application to Sleep Research. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW). 495--502.

[34]

Tobias Scheffer, Christian Decomain, and Stefan Wrobel. 2001. Active Hidden Markov Models for Information Extraction. In Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis (IDA '01). Springer-Verlag, London, UK, UK, 309--318. http://dl.acm.org/citation.cfm?id=647967.741626

[35]

D. Sculley. 2007. Online Active Learning Methods for Fast Label-Efficient Spam Filtering.

[36]

R. Serra, D. Knittel, P. Di Croce, and R. Peres. 2016. Activity Recognition With Smart Polymer Floor Sensor: Application to Human Footstep Recognition. IEEE Sensors Journal 16, 14 (July 2016), 5757--5775.

[37]

Farhad Shahmohammadi, Anahita Hosseini, Christine E. King, and Majid Sarrafzadeh. 2017. Smartwatch Based Activity Recognition Using Active Learning. In Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE '17). IEEE Press, Piscataway, NJ, USA, 321--329.

Digital Library

[38]

M. Stikic, T. Huynh, K. Van Laerhoven, and B. Schiele. 2008. ADL recognition based on the combination of RFID and accelerometer sensing. In 2008 Second International Conference on Pervasive Computing Technologies for Healthcare. 258--263.

[39]

Maja Stikic, Diane Larlus, Sandra Ebert, and Bernt Schiele. 2011. Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors. IEEE Trans. Pattern Anal. Mach. Intell. 33, 12 (Dec. 2011), 2521--2537.

Digital Library

[40]

Maja Stikic and Bernt Schiele. 2009. Activity Recognition from Sparsely Labeled Data Using Multi-Instance Learning. In Location and Context Awareness, Tanzeem Choudhury, Aaron Quigley, Thomas Strang, and Koji Suginuma (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 156--173.

Digital Library

[41]

M. Stikic, K. Van Laerhoven, and B. Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In 2008 12th IEEE International Symposium on Wearable Computers. 81--88.

Digital Library

[42]

A. Subasi, M. Radhwan, R. Kurdi, and K. Khateeb. 2018. IoT based mobile healthcare system for human activity recognition. In 2018 15th Learning and Technology Conference (L T). 29--34.

[43]

Emmanuel Munguia Tapia, Stephen S. Intille, and Kent Larson. 2004. Activity Recognition in the Home Using Simple and Ubiquitous Sensors. In Pervasive Computing, Alois Ferscha and Friedemann Mattern (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 158--175.

[44]

Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-mounted Inertial Sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 1029--1040.

Digital Library

[45]

Emma L. Tonkin, Alison Burrows, Przemyslaw R. Woznowski, Pawel Laskowski, Kristina Y. Yordanova, Niall Twomey, and Ian J. Craddock. 2018. Talk, Text, Tag? Understanding Self-Annotation of Smart Home Data from a User's Perspective. Sensors 18, 7 (2018).

[46]

Y. Vaizman, K. Ellis, and G. Lanckriet. 2017. Recognizing Detailed Human Context in the Wild from Smartphones and Smartwatches. IEEE Pervasive Computing 16, 4 (October 2017), 62--74.

[47]

Yonatan Vaizman, Katherine Ellis, Gert Lanckriet, and Nadir Weibel. 2018. ExtraSensory App: Data Collection In-the-Wild with Rich User Interface to Self-Report Behavior. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 554, 12 pages.

Digital Library

[48]

Niels van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017. The Experience Sampling Method on Mobile Devices. ACM Comput. Surv. 50, 6, Article 93 (Dec. 2017), 40 pages.

Digital Library

[49]

I. Žliobaitė, A. Bifet, B. Pfahringer, and G. Holmes. 2014. Active Learning With Drifting Streaming Data. IEEE Transactions on Neural Networks and Learning Systems 25, 1 (Jan 2014), 27--39.

[50]

Y. Wang, S. Cang, and H. Yu. 2018. A Data Fusion based Hybrid Sensory System for Older People's Daily Activity and Daily Routine Recognition. IEEE Sensors Journal (2018), 1--1.

[51]

Zuobing Xu, Ram Akella, and Yi Zhang. 2007. Incorporating Diversity and Density in Active Learning for Relevance Feedback. In Proceedings of the 29th European Conference on IR Research (ECIR'07). Springer-Verlag, Berlin, Heidelberg, 246--257. http://dl.acm.org/citation.cfm?id=1763653.1763684

Digital Library

[52]

Aras Yurtman and Billur Barshan. 2016. Human Activity Recognition Using Tag-Based Radio Frequency Localization. Applied Artificial Intelligence 30, 2 (2016), 153--179.

Digital Library

[53]

Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma. 2010. Confidence-based Stopping Criteria for Active Learning for Data Annotation. ACM Trans. Speech Lang. Process. 6, 3, Article 3 (April 2010), 24 pages.

Digital Library

Cited By

Hoelzemann AVan Laerhoven K(2024)A matter of annotation: an empirical study on in situ and self-recall activity annotations from wearable sensorsFrontiers in Computer Science10.3389/fcomp.2024.13797886Online publication date: 18-Jul-2024
https://doi.org/10.3389/fcomp.2024.1379788
Takeda NLegaspi RNishimura YIkeda KMinamikawa APlötz TChernova S(2024)Sensor event sequence prediction for proactive smart homeJournal of Ambient Intelligence and Smart Environments10.3233/AIS-23042916:3(275-308)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.3233/AIS-230429
Le HLakshminarayanan RLi JMishra VIntille S(2024)Collecting Self-reported Physical Activity and Posture Data Using Audio-based Ecological Momentary AssessmentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785848:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678584
Show More Cited By

Index Terms

Leveraging Active Learning and Conditional Mutual Information to Minimize Data Annotation in Human Activity Recognition
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Active learning settings

Recommendations

Confidence-based stopping criteria for active learning for data annotation

The labor-intensive task of labeling data is a serious bottleneck for many supervised learning approaches for natural language processing applications. Active learning aims to reduce the human labeling cost for supervised learning methods. Determining ...
Multiclass autoencoder-based active learning for sensor-based human activity recognition
Abstract
Leveraging the enormous amounts of real-world data collected through Internet of Things (IoT) technologies, human activity recognition (HAR) has become a crucial component of numerous human-centric applications, with the aim of enhancing the ...
Graphical abstract

Display Omitted
Highlights
- We present a deep learning-based active learning for an efficiently labeled dataset.
- The proposed method extends the autoencoder with SVDD in a multiclass scheme.
- We evaluate the proposed active learning method in the scenario of ...
Annotating smart environment sensor data for activity learning
Smart Environments: Technology to Support Healthcare

The pervasive sensing technologies found in smart homes offer unprecedented opportunities for providing health monitoring and assistance to individuals experiencing difficulties living independently at home. In order to monitor the functional health of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 3

September 2019

1415 pages

EISSN:2474-9567

DOI:10.1145/3361560

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Published in IMWUT Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
1,060
Total Downloads

Downloads (Last 12 months)71
Downloads (Last 6 weeks)9

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hoelzemann AVan Laerhoven K(2024)A matter of annotation: an empirical study on in situ and self-recall activity annotations from wearable sensorsFrontiers in Computer Science10.3389/fcomp.2024.13797886Online publication date: 18-Jul-2024
https://doi.org/10.3389/fcomp.2024.1379788
Takeda NLegaspi RNishimura YIkeda KMinamikawa APlötz TChernova S(2024)Sensor event sequence prediction for proactive smart homeJournal of Ambient Intelligence and Smart Environments10.3233/AIS-23042916:3(275-308)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.3233/AIS-230429
Le HLakshminarayanan RLi JMishra VIntille S(2024)Collecting Self-reported Physical Activity and Posture Data Using Audio-based Ecological Momentary AssessmentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785848:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678584
Lafontaine VBouchard KMaître JGaboury S(2024)Generating Synthetic Augmentation Data from a Practical UWB Radar Dataset Using VQ-VAEProceedings of the 2024 International Conference on Information Technology for Social Good10.1145/3677525.3678663(212-215)Online publication date: 4-Sep-2024
https://dl.acm.org/doi/10.1145/3677525.3678663
Bock MVan Laerhoven KMoeller MKostakos VKay JHoang T(2024)Weak-Annotation of HAR Datasets using Vision Foundation ModelsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676613(55-62)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676613
Zhang YZou YXie YChen L(2024)Identifying dynamic interaction patterns in mandatory and discretionary lane changes using graph structureComputer-Aided Civil and Infrastructure Engineering10.1111/mice.1309939:5(638-655)Online publication date: 25-Feb-2024
https://dl.acm.org/doi/10.1111/mice.13099
Carvalho MSoares DMacedo D(2024)A Human-in-the-Loop Based ML Framework to Estimate User’s QoE on Cloud Gaming Using Active Learning2024 Joint European Conference on Networks and Communications & 6G Summit (EuCNC/6G Summit)10.1109/EuCNC/6GSummit60053.2024.10597003(895-900)Online publication date: 3-Jun-2024
https://doi.org/10.1109/EuCNC/6GSummit60053.2024.10597003
Hiremath SPlötz T(2024)Maintenance Required: Updating and Extending Bootstrapped Human Activity Recognition Systems for Smart Homes2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10651685(1-13)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10651685
Park HLee GHan JChoi J(2024)Multiclass autoencoder-based active learning for sensor-based human activity recognitionFuture Generation Computer Systems10.1016/j.future.2023.09.029151(71-84)Online publication date: Feb-2024
https://doi.org/10.1016/j.future.2023.09.029
Arrotta LCivitarese GBettini C(2024)SelfAct: Personalized Activity Recognition Based on Self-Supervised and Active LearningMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63989-0_19(375-391)Online publication date: 19-Jul-2024
https://doi.org/10.1007/978-3-031-63989-0_19
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents