Abstract
In recent years, graph convolutional neural network (GCNN) has achieved the most advanced results in skeleton action recognition tasks. However, existing models mainly focus on extracting local information from joint-level and part-level, but ignore the global information of frame-level and the relevance between multiple levels, which lead to the loss of hierarchical information. Moreover, these models consider the non-physical connection relationship between nodes but neglect the dependence between body parts. The lose of topology information directly results in poor model performance. In this paper, we propose a structure-aware multi-scale hierarchical graph convolutional network (SAMS-HGCN) model, which includes two modules: a structure-aware hierarchical graph pooling block (SA-HGP Block) and a multi-scale fusion module (MSF module). Specifically, SA-HGP Block establishes a hierarchical network to capture the topological information of multiple levels by using the hierarchical graph pooling (HGP) operation and model the dependence among parts via the structure-aware learning (SA Learning) operation. MSF module fuses information of different scales in each level to obtain multi-scale global structural information. Experiments show that our method achieves comparable performances to state-of-the-art methods on NTU-RGB+D and Kinetics-Skeleton datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhang, Y., Cao, C., Cheng, J., Lu, H.: EgoGesture: a new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans. Multimed. 20, 1038–1050 (2018)
Ziaeefard, M., Bergevin, R.: Semantic human activity recognition: a literature review. Pattern Recognit. 48, 2329–2345 (2015)
Aggarwal, J.K., Xia, L.: Human activity recognition from 3D data: a review. Pattern Recognit. Lett. 48(1), 70–80 (2014)
Han, F., Reily, B., Hoff, W., Zhan, H.: Space-time representation of people based on 3D skeletal data: a review. Comput. Vis. Image Underst. 158(C), 85–105 (2017)
Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed. 19(2), 4–10 (2012)
Zhu, W., Lan, C., Xing, J., Zeng, W., Xie, X.: Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. In: AAAI 2016 (2016)
Ke, Q., Bennamoun, M., An, S., Sohel, F., Boussaid, F.: A new representation of skeleton sequences for 3D action recognition. In: CVPR 2017 (2017)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: AAAI 2018 (2018)
Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W.: Disentangling and unifying graph convolutions for skeleton-based action recognition. IEEE (2020)
Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., Tian, Q.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: CVPR 2019 (2019)
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with directed graph neural networks. In: CVPR 2020 (2020)
Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition (2018)
Luo, W., Li, Y., Urtasun, R., Zemel, R.: Understanding the effective receptive field in deep convolutional neural networks (2017)
Si, C., Jing, Y., Wang, W., Wang, L., Tan, T.: Skeleton-based action recognition with spatial reasoning and temporal stack learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, Part I. LNCS, vol. 11205, pp. 106–121. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_7
Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. Human Action Recognition with Depth Cameras (2014)
Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: International Joint Conference on Artificial Intelligence (2013)
Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: CVPR 2014 (2014)
Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: CVPR, pp. 1110–1118 (2015)
Liu, J., Shahroudy, A., Xu, D., Wang, G.: Spatio-temporal LSTM with trust gates for 3D human action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part III. LNCS, vol. 9907, pp. 816–833. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_50
Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An end-to-end spatio-temporal attention model for human action recognition from skeleton data (2016)
Kim, T.S., Reiter, A.: Interpretable 3D human action analysis with temporal convolutional networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2017)
Liu, M., Hong, L., Chen, C.: Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognit. 68, 346–362 (2017)
Gao, X., Hu, W., Tang, J., Liu, J., Guo, Z.: Optimized skeleton-based action recognition via sparsified graph regression. In: The 27th ACM International Conference (2019)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks (2016)
Martins, A., Astudillo, R.F.: From softmax to sparsemax: a sparse model of attention and multi-label classification. JMLR.org (2016)
Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: a large scale dataset for 3D human activity analysis, pp. 1010–1019. IEEE Computer Society (2016)
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Zisserman, A.: The kinetics human action video dataset (2017)
Zhe, C., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: CVPR 2017 (2017)
Li, C., Zhong, Q., Xie, D., Pu, S.: Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation. In: Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-2018 (2018)
Yang, W., Zhang, J., Cai, J., Xu, Z.: Shallow graph convolutional network for skeleton-based action recognition. Sensors 21(2), 452 (2021)
Acknowledgements
This work was supported by the Artificial Intelligence Program of Shanghai under Grant 2019-RGZN-01077.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
He, C., Liu, S., Zhao, Y., Qin, X., Zeng, J., Zhang, X. (2021). Structure-Aware Multi-scale Hierarchical Graph Convolutional Network for Skeleton Action Recognition. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12893. Springer, Cham. https://doi.org/10.1007/978-3-030-86365-4_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-86365-4_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86364-7
Online ISBN: 978-3-030-86365-4
eBook Packages: Computer ScienceComputer Science (R0)