Article

Deep, convolutional, and recurrent models for human activity recognition using wearables

Authors:

Nils Y. Hammerla,

Shane Halloran,

Thomas PlötzAuthors Info & Claims

IJCAI'16: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence

Pages 1533 - 1540

Published: 09 July 2016 Publication History

Abstract

Human activity recognition (HAR) in ubiquitous computing is beginning to adopt deep learning to substitute for well-established analysis techniques that rely on hand-crafted feature extraction and classification methods. However, from these isolated applications of custom deep architectures it is difficult to gain an overview of their suitability for problems ranging from the recognition of manipulative gestures to the segmentation and identification of physical activities like running or ascending stairs. In this paper we rigorously explore deep, convolutional, and recurrent approaches across three representative datasets that contain movement data captured with wearable sensors. We describe how to train recurrent approaches in this setting, introduce a novel regularisation approach, and illustrate how they outperform the state-of-the-art on a large benchmark dataset. We investigate the suitability of each model for HAR, across thousands of recognition experiments with randomly sampled model configurations, explore the impact of hyperparameters using the fANOVA framework, and provide guidelines for the practitioner who wants to apply deep learning in their problem setting.

References

[1]

Mohammad Abu Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, and Hwee-Pink Tan. Deep activity recognition models with triaxial accelerometers. arXiv:1511.04664 , 2015.

[2]

Marc Bachlin, Daniel Roggen, Gerhard Troster, Meir Plotnik, Noit Inbar, Inbal Meidan, Talia Herman, Marina Brozgol, Eliya Shaviv, Nir Giladi, et al. Potentials of enhanced context awareness in wearable assistants for parkinson's disease patients with the freezing of gait syndrome. In ISWC , 2009.

Digital Library

[3]

Andreas Bulling, Ulf Blanke, and Bernt Schiele. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) , 46(3):33, 2014.

Digital Library

[4]

Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. The opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters , 2013.

Digital Library

[5]

Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop , 2011.

[6]

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research , 12:2121-2159, 2011.

Digital Library

[7]

Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. arXiv preprint arXiv:1503.04069 , 2015.

[8]

Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv:1502.04623 , 2015.

[9]

Nils Y Hammerla and Thomas Plötz. Let's (not) stick together: pairwise similarity biases cross-validation in activity recognition. In Ubicomp , 2015.

Digital Library

[10]

Nils Y Hammerla, James M Fisher, Peter Andras, Lynn Rochester, Richard Walker, and Thomas Plötz. Pd disease state assessment in naturalistic environments using deep learning. In AAAI , 2015.

Digital Library

[11]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems , pages 1684-1692, 2015.

[12]

Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation , 18(7):1527-1554, 2006.

Digital Library

[13]

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Digital Library

[14]

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.

[15]

Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. An efficient approach for assessing hyperparameter importance. In ICML , pages 754-762, 2014.

[16]

Aftab Khan, Sebastian Mellor, Eugen Berlin, Robin Thompson, Roisin McNaney, Patrick Olivier, and Thomas Plötz. Beyond activity recognition: skill assessment from accelerometer data. In Ubicomp , pages 1155-1166. ACM, 2015.

Digital Library

[17]

Nicholas D Lane, Petko Georgiev, and Lorena Qendro. Deepear: robust smartphone audio sensing in unconstrained acoustic environments using deep learning. In Ubicomp , pages 283-294. ACM, 2015.

Digital Library

[18]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 2015.

[19]

Natalia Neverova, Christian Wolf, Griffin Lacey, Lex Fridman, Deepak Chandra, Brandon Barbello, and Graham Taylor. Learning human identity from motion patterns. arXiv:1511.03908 , 2015.

[20]

Francisco Javier Ordóñez and Daniel Roggen. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors , 16(1):115, 2016.

[21]

Thomas Plötz, Nils Y Hammerla, and Patrick Olivier. Feature learning for activity recognition in ubiquitous computing. In IJCAI , 2011.

Digital Library

[22]

Thomas Plötz, Nils Y Hammerla, Agata Rozga, Andrea Reavis, Nathan Call, and Gregory D Abowd. Automatic assessment of problem behavior in individuals with developmental disabilities. In Ubicomp , 2012.

Digital Library

[23]

Nastaran Mohammadian Rad, Andrea Bizzego, Seyed Mostafa Kia, Giuseppe Jurman, Paola Venuti, and Cesare Furlanello. Convolutional neural network for stereotypical motor movement detection in autism. arXiv:1511.01865 , 2015.

[24]

Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In ISWC , 2012.

Digital Library

[25]

Charissa Ann Ronao and Sung-Bae Cho. Deep convolutional neural networks for human activity recognition with smartphone sensors. In Neural Information Processing , pages 46-53. Springer, 2015.

[26]

Charissa Ann Ronaoo and Sung-Bae Cho. Evaluation of deep convolutional neural network architectures for human activity recognition with smartphone sensors. In Proc. of the KIISE Korea Computer Congress , pages 858-860, 2015.

[27]

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In NIPS . 2012.

Digital Library

[28]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR , 2014.

Digital Library

[29]

Jian Bo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In IJCAI , 2015.

[30]

Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, Jiang Zhu, Pang Wu, and Juyong Zhang. Convolutional neural networks for human activity recognition using mobile sensors. In MobiCASE , pages 197-205. IEEE, 2014.

[31]

Licheng Zhang, Xihong Wu, and Dingsheng Luo. Human activity recognition with hmm-dnn model. In ICCI , pages 192-197. IEEE, 2015.

Cited By

Haresamudram HTang CSuh SLukowicz PPlötz TKostakos VKay JHoang T(2024)Solving the Sensor-Based Activity Recognition Problem (SOAR): Self-Supervised, Multi-Modal Recognition of Activities from Wearable SensorsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677562(1004-1007)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677562
Zhou YZhao HHuang YRöddiger TKurnaz MRiedel TBeigl M(2024)AutoAugHAR: Automated Data Augmentation for Sensor-based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595898:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659589
Ahmad NLeung H(2024)HyperHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435118:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643511
Show More Cited By

Deep, convolutional, and recurrent models for human activity recognition using wearables
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks
MM '15: Proceedings of the 23rd ACM international conference on Multimedia

Human physical activity recognition based on wearable sensors has applications relevant to our daily life such as healthcare. How to achieve high recognition accuracy with low computational cost is an important issue in the ubiquitous computing. Rather ...
Ensembles of Deep LSTM Learners for Activity Recognition using Wearables

Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with ...
Human activity recognition from multiple sensors data using deep CNNs
Abstract
Smart devices with sensors now enable continuous measurement of activities of daily living. Accordingly, various human activity recognition (HAR) experiments have been carried out, aiming to convert the measures taken from smart devices into ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

IJCAI'16: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence

July 2016

4277 pages

ISBN:9781577357704

Editor:
Gerhard Brewka
Leipzig University, Germany

Sponsors

Sony: Sony Corporation
Arizona State University: Arizona State University
Microsoft: Microsoft
Facebook: Facebook
AI Journal: AI Journal

Publisher

AAAI Press

Publication History

Published: 09 July 2016

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

91
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Haresamudram HTang CSuh SLukowicz PPlötz TKostakos VKay JHoang T(2024)Solving the Sensor-Based Activity Recognition Problem (SOAR): Self-Supervised, Multi-Modal Recognition of Activities from Wearable SensorsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677562(1004-1007)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677562
Zhou YZhao HHuang YRöddiger TKurnaz MRiedel TBeigl M(2024)AutoAugHAR: Automated Data Augmentation for Sensor-based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595898:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659589
Ahmad NLeung H(2024)HyperHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435118:1(1-29)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643511
Yan DYang PShang FHan FYan YLi XPapavassiliou SSchmid S(2024)freeGait: Liberalizing Wireless-based Gait Recognition to Mitigate Non-gait Human BehaviorsProceedings of the Twenty-fifth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing10.1145/3641512.3686362(241-250)Online publication date: 14-Oct-2024
https://dl.acm.org/doi/10.1145/3641512.3686362
Konak OWischmann Avan De Water RArnrich B(2023)A Real-time Human Pose Estimation Approach for Optimal Sensor Placement in Sensor-based Human Activity RecognitionProceedings of the 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence10.1145/3615834.3615848(1-6)Online publication date: 21-Sep-2023
https://dl.acm.org/doi/10.1145/3615834.3615848
Dönnebrink RMoya Rueda FGrzeszick RStach M(2023)Miss-placement Prediction of Multiple On-body Devices for Human Activity RecognitionProceedings of the 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence10.1145/3615834.3615838(1-8)Online publication date: 21-Sep-2023
https://dl.acm.org/doi/10.1145/3615834.3615838
Kim JNam YLee JSuh YHwang I(2023)ProxiFitProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109207:3(1-32)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610920
Haresamudram HTang CSuh SLukowicz PPloetz T(2023)Solving the Sensor-based Activity Recognition Problem (SOAR): Self-supervised, Multi-modal Recognition of Activities from Wearable SensorsAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3605102(759-761)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3605102
Xin CKim SPark KWang MYoon B(2023)A Comparison of Machine Learning Models with Data Augmentation Techniques for Skeleton-based Human Action RecognitionProceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics10.1145/3584371.3612999(1-6)Online publication date: 3-Sep-2023
https://dl.acm.org/doi/10.1145/3584371.3612999
Zhang XChowdhury RZhang JHong DGupta RShang JFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Unleashing the Power of Shared Label Structures for Human Activity RecognitionProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615101(3340-3350)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615101
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents