Abstract
Human Activity Recognition (HAR) using Inertial Measurement Unit (IMU) sensor data has practical applications in healthcare and assisted living environments. However, its use in real-world scenarios has been limited due to the lack of comprehensive IMU-based HAR datasets covering various activities. Zero-shot HAR (ZS-HAR) can overcome these data limitations. However, most existing ZS-HAR methods based on IMU data rely on attributes or word embeddings of class labels as auxiliary data to relate the seen and unseen classes. This approach requires expert knowledge and lacks motion-specific information. In contrast, videos depicting various human activities provide valuable information for ZS-HAR based on inertial sensor data, and they are readily available. Our proposed model, TEZARNet: TEmporal Zero-shot Activity Recognition Network, uses videos as auxiliary data and employs a Bidirectional Long-Short Term IMU encoder to exploit temporal information, distinguishing it from current work. The proposed model outperforms the state-of-the-art accuracy by 4.7%, 7.8%, 3.7%, and 9.3% for benchmark datasets PAMAP2, DaLiAc, UTD-MHAD, and MHEALTH, respectively. The code is available at https://github.com/nipdep/TEZARNet
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ahmed, N., Rafiq, J.I., Islam, M.R.: Enhanced human activity recognition based on smartphone sensor data using hybrid feature selection model. Sensors 20(1), 317 (2020)
Banos, O., et al.: mhealthdroid: a novel framework for agile development of mobile health applications. In: Proceedings of the Ambient Assisted Living and Daily Activities: 6th International Work-Conference, pp. 91–98 (2014)
Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
Chen, C., Jafari, R., Kehtarnavaz, N.: Utd-mhad: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: Proceedings of the IEEE International Conference on Image Processing (ICIP), pp. 168–172 (2015)
Cheng, H.T., Griss, M., Davis, P., Li, J., You, D.: Towards zero-shot learning for human activity recognition using semantic attribute sequence model. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 355–358 (2013)
Cheng, H.T., Sun, F.T., Griss, M., Davis, P., Li, J., You, D.: Nuactiv: recognizing unseen new activities using semantic attribute-based learning. In: Proceeding of the 11th annual international conference on Mobile Systems, Applications, and Services, pp. 361–374 (2013)
Church, K.W.: Word2vec. Nat. Lang. Eng. 23(1), 155–162 (2017)
Khojasteh, S.B., Villar, J.R., Chira, C., González, V.M., De la Cal, E.: Improving fall detection using an on-wrist wearable accelerometer. Sensors 18(5), 1350 (2018)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Yann LeCun, Y.B. (ed.) Proceedings of the 3rd International Conference on Learning Representations, ICLR (2015)
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: Proceedings of the International Conference on Computer Vision (ICCV) (2011)
Leutheuser, H., Schuldhaus, D., Eskofier, B.M.: Hierarchical, multi-sensor based classification of daily life activities: comparison with state-of-the-art algorithms using a benchmark dataset. PLoS ONE 8(10), e75196 (2013)
Li, Y., Wang, L.: Human activity recognition based on residual network and bilstm. Sensors 22(2), 635 (2022)
Matsuki, M., Lago, P., Inoue, S.: Characterizing word embeddings for zero-shot sensor-based human activity recognition. Sensors 19(22), 5043 (2019)
Meedeniya, D.: Deep Learning: A Beginners’ Guide. CRC Press LLC (2023). https://www.routledge.com/9781032473246
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Proceedings of the Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)
Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: Proceedings of the 16th International Symposium on Wearable Computers, pp. 108–109 (2012)
Tong, C., Ge, J., Lane, N.D.: Zero-shot learning for imu-based activity recognition using video embeddings. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 5(4), pp. 1–23 (2021)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, Q., Chen, K.: Alternative semantic representations for zero-shot human action recognition. In: Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 87–102 (2017)
Wu, T., Chen, Y., Gu, Y., Wang, J., Zhang, S., Zhechen, Z.: Multi-layer cross loss model for zero-shot human activity recognition. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 210–221 (2020)
Zebin, T., Sperrin, M., Peek, N., Casson, A.J.: Human activity recognition from inertial sensor time-series using batch normalized deep lstm recurrent networks. In: Proceedings of the 40th annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1–4 (2018)
Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., Eickhoff, C.: A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114–2124 (2021)
Zhang, H., Xiao, Z., Wang, J., Li, F., Szczerbicki, E.: A novel iot-perceptive human activity recognition (har) approach using multihead convolutional attention. IEEE Internet Things J. 7(2), 1072–1080 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Deelaka, P.N., De Silva, D.Y., Wickramanayake, S., Meedeniya, D., Rasnayaka, S. (2024). TEZARNet: TEmporal Zero-Shot Activity Recognition Network. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1969. Springer, Singapore. https://doi.org/10.1007/978-981-99-8184-7_34
Download citation
DOI: https://doi.org/10.1007/978-981-99-8184-7_34
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8183-0
Online ISBN: 978-981-99-8184-7
eBook Packages: Computer ScienceComputer Science (R0)