Abstract
Human behavior recognition is one of the significant components of Ambient Assisted Living (AAL) systems and personal assistive robots allowing to improve the quality of their lives in terms of safety, autonomy, and well-being. A critical aspect of preventing dangerous situations for users, especially elderlies, is to recognize abnormal human behavior. In spite of the extensive exploration of abnormality recognition in various fields, there remain some challenges in developing effective approaches for recognizing abnormal human behaviors in AAL systems due to the limitations of data-driven and knowledge-driven approaches. In this paper, a context-aware framework combining data-driven and knowledge-driven approaches is proposed to better characterize human behaviors and recognize abnormal behaviors using commonsense reasoning while considering human behavior context. The proposed framework comprises five main modules, which leverage Long Short-Term Memory (LSTM) models and Probabilistic Answer Set Programming (PASP)-based commonsense reasoning to recognize human activities and represent abnormal human behaviors, as well as reason about those behaviors. The proposed framework is evaluated using two datasets, namely Orange4Home and UCI HAR. The obtained results indicate the capability of the proposed framework to characterize human behaviors and recognize abnormal human behaviors with high performance.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The datasets used during the current study are available in [https://amiqual4home.inria.fr/orange4home/] and [shorturl.at/loNTV].
References
Ahmed MA, Zaidan BB, Zaidan AA et al (2021) Real-time sign language framework based on wearable device: analysis of msl, dataglove, and gesture recognition. Soft Comput 25:11101–11122. https://doi.org/10.1007/s00500-021-05855-6
Anguita D, Ghio A, Oneto L, Parra X, Reyes-Ortiz JL (2013) A public domain dataset for human activity recognition using smartphones. Comput Intell 20:6
Aran O, Sanchez-Cortes D, Do M-T, Gatica-Perez D (2016) Anomaly detection in elderly daily behavior in ambient sensing environments. In: International workshop on human behavior understanding. Springer, pp 51–67
Arifoglu D, Bouchachia A (2017) Activity recognition and abnormal behaviour detection with recurrent neural networks. Proced Comput Sci 110:86–93 (14th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2017))
Ariza-Colpas PP, Vicario E, Oviedo-Carrascal AI, Butt Aziz S, Piñeres-Melo MA, Quintero-Linero A, Patara F (2022) Human activity recognition data analysis: history, evolutions, and new trends. Sensors 22(9):3401
Artikis A, Makris E, Paliouras G (2019) A probabilistic interval-based event calculus for activity recognition. Ann Math Artif Intell 20:1–24
Artikis A, Paliouras G (2009) Behaviour recognition using the event calculus. In: IFIP international conference on artificial intelligence applications and innovations. Springer, pp 469–478
Artikis A, Sergot M, Paliouras G (2010) A logic programming approach to activity recognition. In: Proceedings of the 2nd ACM international workshop on events in multimedia, ser. EiMM ’10, pp 3–8
Banovic N, Buzali T, Chevalier F, Mankoff J, Dey AK (2016) Modeling and understanding human routine behavior. In: Proceedings of the CHI conference on human factors in computing systems, ser. CHI ’16. ACM: New York, pp 248–260
Batchuluun G, Kim JH, Hong HG, Kang JK, Park KR (2017) Fuzzy system based human behavior recognition by combining behavior prediction and recognition. Expert Syst Appl 81:108–133
Baxter RH, Robertson NM, Lane DM (2015) Human behaviour recognition in data-scarce domains. Pattern Recogn 48(8):2377–2393
Bloch I (2005) Fusion d’informations numériques: panorama méthodologique. J Natl Rech Robot 2005:79–88
Chen L, Nugent CD (2019) Human activity recognition and behaviour analysis. Springer, Berlin
Chen L, Nugent CD, Mulvenna M, Finlay D, Hong X, Poland M (2008) A logical framework for behaviour reasoning and assistance in a smart home. Int J Assist Robot Mechatron 9(4):20–34
Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 24(6):961–974
Chen L, Nugent CD, Wang H (2012) A knowledge-driven approach to activity recognition in smart homes. IEEE Trans Knowl Data Eng 24(6):961–974
Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C (Appl Rev) 42(6):790–808
Chen L, Nugent C, Okeyo G (2014) An ontology-based hybrid approach to activity modeling for smart homes. IEEE Trans Human Mach Syst 44(1):92–105
Cumin J, Lefebvre G, Ramparany F, Crowley JL (2017) A dataset of routine daily activities in an instrumented home. In: Ochoa SF, Singh P, Bravo J (eds) Ubiquitous computing and ambient intelligence. Springer International Publishing, Cham, pp 413–425
Cumin J, Ramparany F, Crowley JL et al. (2018) Inferring availability for communication in smart homes using context. In: IEEE international conference on pervasive computing and communications workshops (PerCom Workshops). IEEE, pp 1–6
Dash SCB, Mishra SR, Srujan Raju K, Panda G (2021) Human action recognition using a hybrid deep learning heuristic. Soft Comput 25(18):13079–13092
Dey AK (2001) Understanding and using context. Pers Ubiquit Comput 5(1):4–7
Diamantini C, Freddi A, Longhi S, Potena D, Storti E (2016) A goal-oriented, ontology-based methodology to support the design of AAL environments. Expert Syst Appl 64:117–131
Eppe M (2013) Postdictive reasoning in epistemic action theory. Ph.D. dissertation, Staats-und Universitätsbibliothek Bremen
Fahad LG, Khan A, Rajarajan M (2015) Activity recognition in smart homes with self verification of assignments. Neurocomputing 149:1286–1298
Forkan ARM, Khalil I, Tari Z, Foufou S, Bouras A (2015) A context-aware approach for long-term behavioural change detection and abnormality prediction in ambient assisted living. Pattern Recogn 48(3):628–641
Garcia-Ceja E, Galván-Tejada CE, Brena R (2018) Multi-view stacking for activity recognition with sound and accelerometer data. Inf Fusion 40:45–56
Gayathri KS, Elias S, Shivashankar S (2015) Composite activity recognition in smart homes using Markov logic network. In: 2015 IEEE 12th international conference on ubiquitous intelligence and computing and 2015 IEEE 12th international conference on autonomic and trusted computing and 2015 IEEE 15th international conference on scalable computing and communications and its associated workshops (UIC-ATC-ScalCom), pp 46–53
Gayathri K, Elias S, Ravindran B (2015) Hierarchical activity recognition for dementia care using Markov logic network. Pers Ubiquit Comput 19(2):271–285
Gayathri K, Easwarakumar K, Elias S (2017) Probabilistic ontology based activity recognition in smart homes using Markov logic network. Knowl-Based Syst 121:173–184
Gebser M, Kaminski R, Kaufmann B, Schaub T (2014) Clingo = ASP + control: preliminary report. arXiv:1405.3694 [cs]
Gelfond M, Lifschitz V (1988) The stable model semantics for logic programming. MIT Press, New York, pp 1070–1080
Gers FA, Schmidhuber J, Cummins F (1999) Learning to forget: continual prediction with LSTM. In: 1999 ninth international conference on artificial neural networks ICANN 99. (Conf. Publ. No. 470), vol 2, pp 850–855
Gu J, Wang L, Wang H, Wang S (2019) A novel approach to intrusion detection using SVM ensemble with feature augmentation. Comput Secur 86:53–62
Guarino N, Oberle D, Staab S (2009) What is an ontology? Handbook on ontologies. Springer, Berlin, pp 1–17
Gu T, Wang XH, Pung HK, Zhang DQ (2020) An ontology-based context model in intelligent environments. arXiv:2003.05055 (arXiv preprint)
Hochreiter S, Schmidhuber J (1997) LSTM can solve hard long time lag problems. Adv Neural Inf Process Syst 20:473–479
Hossain HS, Khan MAAH, Roy N (2017) Active learning enabled activity recognition. Pervasive Mob Comput 38:312–330
Ismail WN, Hassan MM, Alsalamah HA (2019) Context-enriched regular human behavioral pattern detection from body sensors data. IEEE Access 7:33834–33850
Jakkula VR, Crandall AS, Cook DJ (2009) Enhancing anomaly detection using temporal pattern discovery. Advanced intelligent environments. Springer, US, pp 175–194
Jia H, Chen S (2020) Integrated data and knowledge driven methodology for human activity recognition. Inf Sci 536:409–430
Khan IU, Afzal S, Lee JW (2022) Human activity recognition via hybrid deep learning based model. Sensors 22(1):323
Khowaja SA, Prabono AG, Setiawan F, Yahya BN, Lee S-L (2018) Contextual activity based healthcare internet of things, services, and people (HIOTSP): an architectural framework for healthcare monitoring using wearable sensors. Comput Netw 145:190–206
Knox S, Coyle L, Dobson S (2010) Using ontologies in case-based activity recognition. In: Proceedings of the twenty-third international Florida artificial intelligence research society conference (FLAIRS)
Kordestani H, Mojarad R, Chibani A, Osmani A, Amirat Y, Barkaoui K, Zahran W (2019) Hapicare: a healthcare monitoring system with self-adaptive coaching using probabilistic reasoning. In: 2019 IEEE/ACS 16th international conference on computer systems and applications (AICCSA). IEEE, pp 1–8
Lago P, Jiménez-Guarín C, Roncancio C (2015) Contextualized behavior patterns for ambient assisted living. In: Salah AA, Kröse BJ, Cook DJ (eds) Human behavior understanding, vol 9277. Springer International Publishing, Cham, pp 132–145
Lee SU, Hofmann A, Williams B (2019) A robust online human activity recognition methodology for human-robot collaboration. In: The AAAI workshop on plan, activity, and intent recognition, p 12
Lentzas A, Vrakas D (2019) Non-intrusive human activity recognition and abnormal behavior detection on elderly people: a review. Artif Intell Rev 20:1–47
Li Q, Huangfu W, Farha F, Zhu T, Yang S, Chen L, Ning H (2020) Multi-resident type recognition based on ambient sensors activity. Futur Gener Comput Syst 112:108–115
Lifschitz V (2002) Answer set programming and plan generation. Artif Intell 138(1–2):39–54
Lifschitz V (2019) Answer set programming. Springer, Berlin
Liu L, Wang S, Su G, Huang Z-G, Liu M (2017) Towards complex activity recognition using a Bayesian network-based probabilistic generative framework. Pattern Recogn 68:295–309
Liu HC, Chuah JH, Khairuddin ASM, Zhao XM, Wang XD (2023) Campus abnormal behavior recognition with temporal segment transformers. IEEE Access 11:38471–38484
Lühr S, West G, Venkatesh S (2007) Recognition of emergent human behaviour in a smart home: a data mining approach. Pervasive Mob Comput 3(2):95–116
Mabrouk AB, Zagrouba E (2018) Abnormal behavior recognition for intelligent video surveillance systems: a review. Expert Syst Appl 91:480–491
Maintaining a healthy lifestyle. https://www.foundationforpn.org/living-well/lifestyle/. Accessed 2020-02-00
Makantasis K, Doulamis A, Doulamis N, Psychas K (2016) Deep learning based human behavior recognition in industrial workflows. In: IEEE international conference on image processing (ICIP), pp 1609–1613
McGuinness D, Van Harmelen F et al. (2004) Owl web ontology language overview. W3C recommendation
Meditskos G, Dasiopoulou S, Kompatsiaris I (2016) Metaq: a knowledge-driven framework for context-aware activity recognition combining sparql and owl 2 activity patterns. Pervasive Mob Comput 25:104–124
Mojarad R, Attal F, Chibani A, Amirat Y (2020) Automatic classification error detection and correction for robust human activity recognition. IEEE Robot Autom Lett 5(2):2208–2215
Mojarad R, Attal F, Chibani A, Amirat Y (2021) A context-aware approach to detect abnormal human behaviors. In: Dong Y, Mladenić D, Saunders C (eds) Machine learning and knowledge discovery in databases: applied data science track. Springer International Publishing, Cham, pp 89–104
Mojarad R, Attal F, Chibani A, Amirat Y (2020) A context-aware hybrid framework for human behavior analysis. In: 2020 IEEE 32nd international conference on tools with artificial intelligence (ICTAI). IEEE, pp 460–465
Mojarad R, Attal F, Chibani A, Amirat Y (2020) A hybrid context-aware framework to detect abnormal human daily living behavior. In: Conference on neural networks. IEEE, pp 1–8
Mojarad R, Attal F, Chibani A, Fiorini SR, Amirat Y (2018) Hybrid approach for human activity recognition by ubiquitous robots. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5660–5665
Mueller ET (2015) Chapter 15—commonsense reasoning using answer set programming. In: Mueller ET (ed) Commonsense reasoning (Second Edition), Boston, pp 249–269
Muise C, Wollenstein-Betech S, Booth S, Shah J, Khazaeni Y (2020) Modeling blackbox agent behaviour via knowledge compilation. In: The AAAI workshop on plan, activity, and intent recognition
Nebel B, Bürckert H-J (1995) Reasoning about temporal relations: a maximal tractable subclass of Allen’s interval algebra. J ACM 42(1):43–66
Okeyo G, Chen L, Wang H (2014) Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes. Futur Gener Comput Syst 39:29–43
Owl 2 web ontology language document overview (second edition). https://www.w3.org/TR/2012/REC-owl2-overview-20121211/. Accessed 2020-02-00
Patkos T, Plexousakis D, Chibani A, Amirat Y (2016) An event calculus production rule system for reasoning in dynamic and uncertain domains. Theory Pract Logic Program 20:325–352
Phan N, Dou D, Wang H, Kil D, Piniewski B (2017) Ontology-based deep learning for human behavior prediction with explanations in health social networks. Inf Sci 384:298–313
Qu Y, Tang Y, Yang X, Wen Y, Zhang W (2023) Context-aware mutual learning for semi-supervised human activity recognition using wearable sensors. Expert Syst Appl 219:119679
Rafferty J, Nugent CD, Liu J, Chen L (2017) From activity recognition to intention recognition for assisted living within smart homes. Trans Human Mach Syst 20:368–379
Rastogi S, Singh J (2022) Human fall detection and activity monitoring: a comparative analysis of vision-based methods for classification and detection techniques. Soft Comput 26:3679–3701. https://doi.org/10.1007/s00500-021-06717-x
Reiss A, Hendeby G, Stricker D (2013) A competitive approach for human activity recognition on smartphones. In: European symposium on artificial neural networks, computational intelligence and machine learning, 24–26 April. Belgium, Bruges, pp 455–460
Riboni D, Bettini C (2011) Owl 2 modeling and reasoning with complex human activities. Pervas Mob Comput 7(3):379–395 (Knowledge-Driven Activity Recognition in Intelligent Environments)
Riboni D, Bettini C (2011) Cosar: hybrid reasoning for context-aware activity recognition. Pers Ubiquit Comput 15(3):271–289
Riboni D, Bettini C, Civitarese G, Janjua ZH, Helaoui R (2016) Smartfaber: recognizing fine-grained abnormal behaviors for early detection of mild cognitive impairment. Artif Intell Med 67:57–74
Riboni D, Bettini C, Civitarese G, Janjua ZH, Helaoui R (2015) Fine-grained recognition of abnormal behaviors for early detection of mild cognitive impairment. In: IEEE international conference on pervasive computing and communications (PerCom), pp 149–154
Romera-Paredes B, Aung MS, Bianchi-Berthouze N (2013) A one-vs-one classifier ensemble with majority voting for activity recognition. In: ESANN
Ronao CA, Cho S-B (2016) Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244
Sánchez VG, Lysaker OM, Skeie N-O (2020) Human behaviour modelling for welfare technology using hidden Markov models. Pattern Recogn Lett 137:71–79
Sfar H (2019) Real time intelligent decision making from heterogeneous and imperfect data. Ph.D. dissertation, Paris Saclay
Shen Y-D, Eiter T (2016) Evaluating epistemic negation in answer set programming. Artif Intell 237:115–135
Shin JH, Lee B, Park KS (2011) Detection of abnormal living patterns for elderly living alone using support vector data description. Trans Inf Tech Biomed 15(3):438–448
Soto-Mendoza V, García-Macías JA, Chávez E, Gomez-Montalvo JR, Quintana E (2017) Detecting abnormal behaviours of institutionalized older adults through a hybrid-inference approach. Pervasive Mob Comput 40:708–723
Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Thirty-first AAAI conference on artificial intelligence
Springer T, Turhan A-Y (2009) Employing description logics in ambient intelligence for modeling and reasoning about complex situations. J Ambient Intell Smart Environ 1(3):235–259
Stavropoulos TG, Meditskos G, Andreadis S, Avgerinakis K, Adam K, Kompatsiaris I (2016) Semantic event fusion of computer vision and ambient sensor data for activity recognition to support dementia care. J Ambient Intell Human Comput 20:1–16
Sun J, Shao J, He C (2019) Abnormal event detection for video surveillance using deep one-class learning. Multimed Tools Appl 78(3):3633–3647
Tay NC, Connie T, Ong TS, Teoh ABJ, Teh PS (2023) A review of abnormal behavior detection in activities of daily living. IEEE Access 20:20
Triboan D, Chen L, Chen F (2019) Fuzzy-based fine-grained human activity recognition within smart environments. In: 2019 IEEE SmartWorld, ubiquitous intelligence and computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE 2019, pp 94–101
Tsai M-F, Li M-H (2022) Intelligent attendance monitoring system with spatio-temporal human action recognition. Soft Comput. https://doi.org/10.1007/s00500-022-07582
Van Haaren J, Van den Broeck G, Meert W, Davis J (2016) Lifted generative learning of Markov logic networks. Mach Learn 103(1):27–55
Varshney N, Bakariya B, Kushwaha AKS et al (2023) Rule-based multi-view human activity recognition system in real time using skeleton data from rgb-d sensor. Soft Comput 27(1):405–421
Vert J-P, Tsuda K, Schölkopf B (2004) A primer on kernel methods. Kernel Methods Comput Biol 47:35–70
Villalonga C, Pomares H, Rojas I, Banos O (2017) Mimu-wear: ontology-based sensor selection for real-world wearable activity recognition. Neurocomputing 250:76–100
Wang Y, Cang S, Yu H (2018) A data fusion-based hybrid sensory system for older people’s daily activity and daily routine recognition. IEEE Sens J 18(16):6874–6888
Wen J, Wang Z (2017) Learning general model for activity recognition with limited labelled data. Expert Syst Appl 74:19–28
Wongpatikaseree K, Ikeda M, Buranarach M, Supnithi T, Lim AO, Tan Y (2012) Activity recognition using context-aware infrastructure ontology in smart home domain. In: 2012 seventh international conference on knowledge, information and creativity support systems. IEEE, pp 50–57
Xiang T, Gong S (2008) Video behavior profiling for anomaly detection. IEEE Trans Pattern Anal Mach Intell 30(5):893–908
Ye J, Coyle L, Dobson S, Nixon P (2007) Ontology-based models in pervasive computing systems. Knowl Eng Rev 22(4):315–347
Zambrana C, Palou XR, Vargiu E (2016) Sleeping recognition to assist elderly people at home. Artif Intell Res 20:64–69
Zhang Y, Ding K, Hui J, Lv J, Zhou X, Zheng P (2022) Human-object integrated assembly intention recognition for context-aware human–robot collaborative assembly. Adv Eng Inform 54:101792
Zhao Y, Zhang H, Gao Z, Gao W, Wang M, Chen S (2023) A novel action saliency and context-aware network for weakly-supervised temporal action localization. IEEE Trans Multimed 20:1–14
Funding
The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All the authors declare that no conflict of interest exists.
Human participants
All the procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Mojarad, R., Chibani, A., Attal, F. et al. A hybrid and context-aware framework for normal and abnormal human behavior recognition. Soft Comput 28, 4821–4845 (2024). https://doi.org/10.1007/s00500-023-09188-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-023-09188-4