Human Activity Recognition Using Cell Phone-Based Accelerometer and Convolutional Neural Network
<p>CNN model development [<a href="#B45-applsci-11-12099" class="html-bibr">45</a>].</p> "> Figure 2
<p>Plots representing accelerometer readings for the six human activities (a–f). (<b>a</b>) Accelerometer readings for Walking. (<b>b</b>) Accelerometer readings for Upstairs. (<b>c</b>) Accelerometer readings for Downstairs. (<b>d</b>) Accelerometer readings for Jogging. (<b>e</b>) Accelerometer readings for Standing. (<b>f</b>) Accelerometer readings for Sitting.</p> "> Figure 2 Cont.
<p>Plots representing accelerometer readings for the six human activities (a–f). (<b>a</b>) Accelerometer readings for Walking. (<b>b</b>) Accelerometer readings for Upstairs. (<b>c</b>) Accelerometer readings for Downstairs. (<b>d</b>) Accelerometer readings for Jogging. (<b>e</b>) Accelerometer readings for Standing. (<b>f</b>) Accelerometer readings for Sitting.</p> "> Figure 3
<p>Training examples by activity type.</p> "> Figure 4
<p>Training examples by user.</p> "> Figure 5
<p>Balanced number of training examples for each activity type.</p> "> Figure 6
<p>Different layers of the 2D Convolutional Neural Network.</p> "> Figure 7
<p>Graph depicting model accuracy.</p> "> Figure 8
<p>Graph detecting model loss.</p> "> Figure 9
<p>Confusion matrix showing the prediction scores of each activity.</p> ">
Abstract
:1. Introduction
- Demonstrating the methodology behind the transformation of accelerometer data into appropriate classes, so that it can be fed into the convolutional neural network to perform HAR.
- Illustrating that it is plausible to perform HAR with prevalent sensors such as accelerometers, and devices such as commonly used smartphones, and still achieve accurate results. It does not have to include a sophisticated machinery and devices or highly complex algorithms for most of the time to obtain appreciable and useful outcomes. Thus, this paper gives a fundamental and preliminary approach towards HAR.
- To validate the proposed 2D CNN model through extensive experiments.
2. Related Work
- The dataset used in the proposed methods which showed better results have a smaller number of participants. For producing generic HAR outcomes, a bigger size of data collected from several participants is advantageous.
- Except for one or two proposed methods, there is a meagre analysis and in-depth study carried out on already existing methods, which may still bring comparable results such as the newly developed methods.
- Less emphasis might be given to the evaluation of the models so developed for performing HAR.
3. Research Methodology
3.1. Problem Description
3.2. CNN for Human Activity Recognition
3.3. Software Tools Used
4. Experiment Study
4.1. Data Collection
4.2. The Activities
4.3. Data Balancing
4.4. Data Standardization and Frame Preparation
4.5. Splitting Data for Training and Testing
4.6. The 2D CNN Model
4.6.1. Model Evaluation
4.6.2. The Confusion Matrix
5. Results and Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Garcia-Gonzalez, D.; Rivero, D.; Fernandez-Blanco, E.; Luaces, M.R. A Public Domain Dataset for Real-Life Human Activity Recognition Using Smartphone Sensors. Sensors 2020, 20, 2200. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhu, N.; Diethe, T.; Camplani, M.; Tao, L.; Burrows, A.; Twomey, N.; Kaleshi, D.; Mirmehdi, M.; Flach, P.; Craddock, I.J. Bridging e-Health and the Internet of Things: The SPHERE Project. IEEE Intell. Syst. 2015, 30, 39–46. [Google Scholar] [CrossRef] [Green Version]
- Oliver, N.; Horvitz, E.; Garg, A. Layered representations for human activity recognition. In Proceedings of the Fourth IEEE International Conference on Multimodal Interfaces, Pittsburgh, PA, USA, 16 October 2002; pp. 3–8. [Google Scholar] [CrossRef]
- Aran, O.; Sanchez-Cortes, D.; Do, M.-T.; Gatica-Perez, D. Anomaly Detection in Elderly Daily Behavior in Ambient Sensing Environments. In Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2016; pp. 51–67. [Google Scholar]
- Zerkouk, M.; Chikhaoui, B. Long Short Term Memory Based Model for Abnormal Behavior Prediction in Elderly Persons. In How AI Impacts Urban Living and Public Health; Springer: Cham, Switzerland, 2019; pp. 36–45. [Google Scholar]
- Shoaib, M.; Bosch, S.; Incel, O.D.; Scholten, J.; Havinga, P.J.M. Complex Human Activity Recognition Using Smartphone and Wrist-Worn Motion Sensors. Sensors 2016, 16, 426. [Google Scholar] [CrossRef] [PubMed]
- Attal, F.; Mohammed, S.; Dedabrishvili, M.; Chamroukhi, F.; Oukhellou, L.; Amirat, Y. Physical Human Activity Recognition Using Wearable Sensors. Sensors 2015, 15, 31314–31338. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Parate, A.; Chiu, M.C.; Chadowitz, C.; Ganesan, D.; Kalogerakis, E. RisQ: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, New York, NY, USA, 2 June 2014; pp. 149–161. [Google Scholar]
- Ramos-Garcia, R.I.; Hoover, A.W. A Study of Temporal Action Sequencing During Consumption of a Meal. In Proceedings of the International Conference on Bioinformatics, Computational Biology and Biomedical Informatics, Washington, DC, USA, 22–25 September 2021. [Google Scholar]
- Dong, Y.; Scisco, J.; Wilson, M.; Muth, E.; Hoover, A. Detecting Periods of Eating During Free-Living by Tracking Wrist Motion. IEEE J. Biomed. Health Inform. 2013, 18, 1253–1260. [Google Scholar] [CrossRef] [PubMed]
- Guiry, J.J.; Van De Ven, P.; Nelson, J. Multi-Sensor Fusion for Enhanced Contextual Awareness of Everyday Activities with Ubiquitous Devices. Sensors 2014, 14, 5687–5701. [Google Scholar] [CrossRef] [PubMed]
- Kwapisz, J.R.; Weiss, G.; Moore, S.A. Activity recognition using cell phone accelerometers. ACM SIGKDD Explor. Newsl. 2011, 12, 74–82. [Google Scholar] [CrossRef]
- Bao, L.; Intille, S.S. Activity Recognition from User-Annotated Acceleration Data. In Artificial Intelligence and Soft Computing—ICAISC 2008; Springer: New York, NY, USA, 2004; Volume 3001, pp. 1–17. [Google Scholar]
- Krishnan, N.C.; Colbry, D.; Juillard, C.; Panchanathan, S. Real Time Human Activity Recognition Using Tri-Axial Accelerometers. In Proceedings of the Sensors Signals and Information Processing Workshop, Sedona, AZ, USA, 11–14 May 2008. [Google Scholar]
- Choudhury, T.; Borriello, G.; Consolvo, S.; Haehnel, D.; Harrison, B.; Hemingway, B.; Hightower, J.; Klasnja, P.; Koscher, K.; Lamarca, A.; et al. The Mobile Sensing Platform: An Embedded Activity Recognition System. IEEE Pervasive Comput. 2008, 7, 32–41. [Google Scholar] [CrossRef]
- Voicu, R.-A.; Dobre, C.; Bajenaru, L.; Ciobanu, R.-I. Human Physical Activity Recognition Using Smartphone Sensors. Sensors 2019, 19, 458. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Figueiredo, J.; Gordalina, G.; Correia, P.; Pires, G.; Oliveira, L.; Martinho, R.; Rijo, R.; Assuncao, P.; Seco, A.; Fonseca-Pinto, R. Recognition of human activity based on sparse data collected from smartphone sensors. In Proceedings of the IEEE 6th Portuguese Meeting on Bioengineering, Instituto Superior de Engenharia de Lisboa (ISEL), Lisbon, Portugal, 22 February 2019; pp. 1–4. [Google Scholar]
- Anguita, D.; Ghio, A.; Oneto, L.; Parra, X.; Reyes-Ortiz, J.L. A Public Domain Dataset for Human Activity Recognition Using Smartphones. InEsann 2013, 3, 3. [Google Scholar]
- Choudhury, N.A.; Moulik, S.; Roy, D.S. Physique-Based Human Activity Recognition Using Ensemble Learning and Smartphone Sensors. IEEE Sens. J. 2021, 21, 16852–16860. [Google Scholar] [CrossRef]
- Chen, Z.; Zhu, Q.; Soh, Y.C.; Zhang, L. Robust Human Activity Recognition Using Smartphone Sensors via CT-PCA and Online SVM. IEEE Trans. Ind. Inform. 2017, 13, 3070–3080. [Google Scholar] [CrossRef]
- Ronao, C.A.; Cho, S.-B. Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst. Appl. 2016, 59, 235–244. [Google Scholar] [CrossRef]
- Hernandez, F.; Suarez, L.F.; Villamizar, J.; Altuve, M. Human Activity Recognition on Smartphones Using a Bidirectional LSTM Network. In Proceedings of the 2019 XXII Symposium on Image, Signal Processing and Artificial Vision (STSIVA), Bucaramanga, Colombia, 24–26 April 2019; pp. 1–5. [Google Scholar]
- Badshah, M. Sensor—Based Human Activity Recognition Using Smartphones. Master’s Thesis, San Jose State University, San Jose, CA, USA, 2019. [Google Scholar] [CrossRef]
- Banos, O.; Galvez, J.-M.; Damas, M.; Pomares, H.; Rojas, I. Window Size Impact in Human Activity Recognition. Sensors 2014, 14, 6474–6499. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Hemalatha, C.; Vaidehi, V. Frequent Bit Pattern Mining Over Tri-axial Accelerometer Data Streams for Recognizing Human Activities and Detecting Fall. Procedia Comput. Sci. 2013, 19, 56–63. [Google Scholar] [CrossRef] [Green Version]
- Gallagher, S. Smartphone Sensor Data Mining for Gait Abnormality Detection; Fordham University: New York, NY, USA, 2013. [Google Scholar]
- Lockhart, J.W. The Benefits of Personalized Data Mining Approaches to Human Activity Recognition with Smartphone Sensor Data. Ph.D. Thesis, Fordham University, New York, NY, USA, 2016. [Google Scholar]
- The Burgos Tapestry: Medieval Theatre and Visual Experience|Attitudes towards Immigration Reform in the United States: The Importance of Neighborhoods|The Spontaneous Formation of Selenium Nanoparticles on Gallic Acid Assemblies and Their Antioxidant Properties|A Power Beyond the Reach of Any Magic’: Mythology in Harry Potter|A Canyon Apart: Immigration Politics and Hispanic Mobilization in Arizona. Available online: www.fordham.edu/fcrh/furj (accessed on 29 August 2021).
- Weiss, G.M.; Lockhart, J.W. Identifying user traits by mining smart phone accelerometer data. In Proceedings of the Fifth International Workshop on Knowledge Discovery from Sensor Data—SensorKDD, San Diego, CA, USA, 21 August 2011. [Google Scholar]
- Lockhart, J.W.; Weiss, G.M.; Xue, J.C.; Gallagher, S.T.; Grosner, A.B.; Pulickal, T.T. Design considerations for the WISDM smart phone-based sensor mining architecture. In Proceedings of the Fifth International Workshop on Knowledge Discovery from Sensor Data—SensorKDD, San Diego, CA, USA, 21 August 2011. [Google Scholar]
- Weiss, G.M.; Lockhart, J.W. The Impact of Personalization on Smartphone-Based Activity Recognition. Available online: www.aaai.org (accessed on 29 August 2021).
- Lockhart, J.W.; Pulickal, T.; Weiss, G.M. Applications of mobile activity recognition. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing—UbiComp, Pittsburgh, PA, USA, 5–8 September 2012; pp. 1054–1058. [Google Scholar]
- Weiss, G.M.; Lockhart, J.W. A comparison of alternative client/server architectures for ubiquitous mobile sensor-based applications. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing—UbiComp, Pittsburgh, PA, USA, 5–8 September 2012; pp. 721–724. [Google Scholar]
- Weiss, G.M.; Nathan, A.; Kropp, J.; Lockhart, J.W. WagTag. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication 2013, Zurich, Switzerland, 8–12 September 2013; pp. 405–414. [Google Scholar]
- Lockhart, J.W.; Weiss, G.M. The Benefits of Personalized Smartphone-Based Activity Recognition Models. In Proceedings of the 2014 SIAM International Conference on Data Mining, Philadelphia, PA, USA, 24–26 April 2014. [Google Scholar]
- Lockhart, J.W.; Weiss, G.M. Limitations with activity recognition methodology & data sets. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, Seattle, WA, USA, 13–17 September 2014. [Google Scholar]
- Vrigkas, M.; Enikou, C.; Kakadiaris, I.A. A Review of Human Activity Recognition Methods. Front. Robot. AI 2015, 2, 28. [Google Scholar] [CrossRef] [Green Version]
- Valueva, M.; Nagornov, N.; Lyakhov, P.; Valuev, G.; Chervyakov, N. Application of the residue number system to reduce hardware costs of the convolutional neural network implementation. Math. Comput. Simul. 2020, 177, 232–243. [Google Scholar] [CrossRef]
- Van Den Oord, A.; Dieleman, S.; Schrauwen, B. Deep content-based music recommendation. In Neural Information Processing Systems Conference; Neural Information Processing Systems Foundation: Vancouver, Canada, 2013. [Google Scholar]
- Collobert, R.; Weston, J. A unified architecture for natural language processing: Deep Neural Networks with Multitask Learning. In Proceedings of the 25th International Conference on Machine Learning, New York, NY, USA, 5–9 July 2008; pp. 160–167. [Google Scholar] [CrossRef]
- Avilov, O.; Rimbert, S.; Popov, A.; Bougrain, L. Deep Learning Techniques to Improve Intraoperative Awareness Detection from Electroencephalographic Signals. In Proceedings of the 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Montreal, QC, Canada, 20–24 July 2020; pp. 142–145. [Google Scholar]
- Tsantekidis, A.; Passalis, N.; Tefas, A.; Kanniainen, J.; Gabbouj, M.; Iosifidis, A. Forecasting Stock Prices from the Limit Order Book Using Convolutional Neural Networks. In Proceedings of the 2017 IEEE 19th Conference on Business Informatics (CBI), Thessaloniki, Greece, 24–27 July 2017; Volume 1, pp. 7–12. [Google Scholar]
- Albawi, S.; Mohammed, T.A.; Al-Zawi, S. Understanding of a convolutional neural network. In Proceedings of the 2017 International Conference on Engineering and Technology (ICET), Antalya, Turkey, 21–23 August 2017; pp. 1–6. [Google Scholar]
- Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res. 2014, 15, 1929–1958. [Google Scholar] [CrossRef]
- Ackermann, N. Deep Neural Network Example. In Licensed under Creative Commons CC BY-ND 4.0; Retrieved 17 August 2021; Available online: https://towardsdatascience.com/human-activity-recognition-har-tutorial-with-keras-and-core-ml-part-1-8c05e365dfa0 (accessed on 31 August 2021).
- Chollet, F. Keras. 2015. Available online: https://keras.io/ (accessed on 31 August 2021).
- TensorFlow. Available online: https://www.tensorflow.org/ (accessed on 4 June 2021).
- MatPlotLib. Available online: https://matplotlib.org/ (accessed on 31 August 2021).
- Scikit-Learn. Scikit-Learn: Machine Learning in Python. Scikit-Learn. 2020. Available online: https://scikit-learn.org/stable/ (accessed on 15 July 2021).
- Pandas. Available online: https://pandas.pydata.org/ (accessed on 31 August 2021).
- NumPy. Available online: https://numpy.org/ (accessed on 31 August 2021).
- Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
- Roelofs, R.; Fridovich-Keil, S.; Miller, J.; Shankar, V.; Hardt, M.; Recht, B.; Schmidt, L. A Meta-Analysis of Overfitting in Machine Learning. Available online: https://www.kaggle.com/kaggle/meta-kaggle (accessed on 31 August 2021).
UCI HAR | WISDM (Used in This Work) | UDC HAR | HARSense | |
---|---|---|---|---|
Human activities studied | Walking, Walking upstairs, Walking downstairs, Sitting, Standing, Laying | Walking, Jogging, Upstairs, Downstairs, Sitting, Standing | Inactive, Active, Walking, Driving | Walking, Standing, Upstairs, Downstairs, Running, Sitting |
Smartphone orientation | Fixed | Fixed | Free | Fixed |
Smartphone positioning | Waist | Front pants leg pocket | As per the individual’s choice | Front pocket and waist |
Sensor frequency | Fixed | Fixed | Not fixed | Fixed |
Dissimilar Individuals | Yes | Yes | Yes | No |
Type(s) of sensors used | Accelerometer; gyroscope | Accelerometer; gyroscope | Accelerometer; gyroscope; magnetometer; GPS | Accelerometer; gyroscope |
Number of subjects involved in the study | 30 | 36 | 19 | 12 |
Human Activity | Label |
---|---|
Downstairs | 0 |
Jogging | 1 |
Sitting | 2 |
Standing | 3 |
Upstairs | 4 |
Walking | 5 |
User ID | Activity | Time | x | y | z | |
---|---|---|---|---|---|---|
0 | 33 | Jogging | 49105962326000 | −0.6946377 | 12.680544 | 0.50395286 |
1 | 33 | Jogging | 49106062271000 | 5.012288 | 11.264028 | 0.95342433 |
2 | 33 | Jogging | 49106112167000 | 4.903325 | 10.882658 | −0.08172209 |
3 | 33 | Jogging | 49106222305000 | −0.61291564 | 18.496431 | 3.0237172 |
4 | 33 | Jogging | 49106332290000 | −1.1849703 | 12.108489 | 7.205164 |
x | y | z | Label | |
---|---|---|---|---|
0 | 0.000503 | 0.000503 | 0.000503 | 5 |
1 | 0.073590 | 0.073590 | 0.073590 | 5 |
2 | −0.361275 | −0.361275 | −0.361275 | 5 |
3 | 1.060258 | 1.060258 | 1.060258 | 5 |
4 | −0.237028 | −0.237028 | −0.237028 | 5 |
… | … | … | … | … |
21325 | −0.470217 | −0.470217 | −0.470217 | 3 |
21326 | −0.542658 | −0.542658 | −0.542658 | 3 |
21327 | −0.628514 | −0.628514 | −0.628514 | 3 |
21328 | −0.781444 | −0.781444 | −0.781444 | 3 |
21329 | −0.800225 | −0.800225 | −0.800225 | 3 |
Activity | Prediction Accuracy (%) |
---|---|
Downstairs | 89% |
Jogging | 94% |
Sitting | 100% |
Standing | 100% |
Upstairs | 61% |
Walking | 94% |
Activity | LSTM Accuracy (%) | 2D CNN Accuracy (%) |
---|---|---|
Downstairs | 86% | 89% |
Jogging | 96% | 94% |
Sitting | 92% | 100% |
Standing | 94% | 100% |
Upstairs | 87% | 61% |
Walking | 99% | 94% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Prasad, A.; Tyagi, A.K.; Althobaiti, M.M.; Almulihi, A.; Mansour, R.F.; Mahmoud, A.M. Human Activity Recognition Using Cell Phone-Based Accelerometer and Convolutional Neural Network. Appl. Sci. 2021, 11, 12099. https://doi.org/10.3390/app112412099
Prasad A, Tyagi AK, Althobaiti MM, Almulihi A, Mansour RF, Mahmoud AM. Human Activity Recognition Using Cell Phone-Based Accelerometer and Convolutional Neural Network. Applied Sciences. 2021; 11(24):12099. https://doi.org/10.3390/app112412099
Chicago/Turabian StylePrasad, Ashwani, Amit Kumar Tyagi, Maha M. Althobaiti, Ahmed Almulihi, Romany F. Mansour, and Ayman M. Mahmoud. 2021. "Human Activity Recognition Using Cell Phone-Based Accelerometer and Convolutional Neural Network" Applied Sciences 11, no. 24: 12099. https://doi.org/10.3390/app112412099