Nothing Special   »   [go: up one dir, main page]
Skip to main content
Springer Nature Link
Log in

mmGaitSet: multimodal based gait recognition for countering carrying and clothing changes

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This paper studies robust gait features against pedestrian carrying and clothing condition changes. Inspired by the fact that humans pay more attention to pose details based on part movements when completing a gait recognition task, we introduce pose information into the convolutional network without complex computation of human modeling. We construct a multimodal set-based deep convolutional network (mmGaitSet). The mmGaitSet consists of two independent feature extractors which extract the body features from silhouettes and the part features from pose heatmaps, respectively. Joint training of two feature extractors make them complement each other. We combine intra-modal fusion and inter-modal fusion into the network. The intra-modal fusion integrates the low-level structural features and high-level semantic features, to improve the discrimination of single modality features. The inter-modal fusion fully aggregates the complementary information between different modalities to enhance the pedestrian gait presentation. The state-of-the-art results are achieved on the challenging CASIA-B dataset outperforming recent competing methods, with up to 92.5% and 80.3% average rank-1 accuracies under bag-carrying and coat-wearing walking conditions, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

We evaluate our method on the public CASIA-B dataset. The dataset is available at http://www.cbsr.ia.ac.cn/english/G-ait%20Databases.asp.

Code Availability

We will open source code on GitHub.

References

  1. Fan C, Peng Y, Cao C, Liu X, Hou S, Chi J, Huang Y, Li Q, He Z (2020) GaitPart: temporal part-based model for gait recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 14225–14233

  2. Liao R, Yu S, An W, Huang Y (2020) A model-based gait recognition method with body pose and human prior knowledge. Pattern Recogn 98:107069

    Article  Google Scholar 

  3. Iwama H, Muramatsu D, Makihara Y, Yagi Y (2013) Gait verification system for criminal investigation. Inf Med Technol 8(4):1187–1199

    Google Scholar 

  4. Lynnerup N, Larsen PK (2014) Gait as evidence. IET Biom 3(2):47–54

    Article  Google Scholar 

  5. He Y, Zhang J, Shan H, Wang L (2019) Multi-task GANs for view-specific feature learning in gait recognition. IEEE Trans Inf Forens Secur 14(1):102–113

    Article  Google Scholar 

  6. Wu Z, Huang Y, Wang L, Wang X, Tan T (2016) A comprehensive study on cross-view gait based human identification with deep cnns. IEEE Trans Pattern Anal Mach Intell 39(2):209–226

    Article  Google Scholar 

  7. Wolf T, Babaee M, Rigoll G (2016) Multi-view gait recognition using 3D convolutional neural networks. In: 2016 IEEE international conference on image processing (ICIP). IEEE, pp 4165–4169

  8. Chao H, He Y, Zhang J, Feng J (2019) Gaitset: Regarding gait as a set for cross-view gait recognition. In: Proceedings of the AAAI conference on artificial intelligence, pp 8126–8133

  9. Zhao G, Liu G, Li H, Pietikainen M (2006) 3D gait recognition using multiple cameras. In: 7th international conference on automatic face and gesture recognition (FGR06). IEEE, pp 529–534

  10. Yu S, Tan D, Tan T (2006) A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition. In: 18th international conference on pattern recognition (ICPR’06). IEEE, pp 441–444

  11. Hermans A, Beyer L, Leibe B (2017) In defense of the triplet loss for person re-identification. arXiv:170307737

  12. Han J, Bhanu B (2006) Individual recognition using gait energy image. IEEE Trans Pattern Anal Mach Intell 28(2):316–322

    Article  Google Scholar 

  13. Bashir K, Xiang T, Gong S (2009) Gait recognition using gait entropy image

  14. Bashir K, Xiang T, Gong S (2010) Gait recognition without subject cooperation. Pattern Recogn Lett 31(13):2052–2060

    Article  Google Scholar 

  15. Makihara Y, Suzuki A, Muramatsu D, Li X, Yagi Y (2017) Joint intensity and spatial metric learning for robust gait recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5705–5715

  16. Guan Y, Li C-T, Roli F (2014) On reducing the effect of covariate factors in gait recognition: a classifier ensemble method. IEEE Trans Pattern Anal Mach Intell 37(7):1521–1528

    Article  Google Scholar 

  17. Wu Z, Huang Y, Wang L (2015) Learning representative deep features for image set analysis. IEEE Trans Multimed 17(11):1960–1968

    Article  Google Scholar 

  18. Zhang Y, Huang Y, Yu S, Wang L (2020) Cross-View Gait Recognition by Discriminative Feature Learning. IEEE Trans Image Process 29:1001–1015

    Article  MathSciNet  Google Scholar 

  19. Hou S, Cao C, Liu X, Huang Y (2020) Gait lateral network: learning discriminative and compact representations for gait recognition. In: European conference on computer vision. Springer, pp 382–398

  20. Ariyanto G, Nixon MS (2011) Model-based 3D gait biometrics. In: 2011 international joint conference on biometrics (IJCB). IEEE, pp 1–7

  21. Bodor R, Drenner A, Fehr D, Masoud O, Papanikolopoulos N (2009) View-independent human motion classification using image-based reconstruction. Image Vis Comput 27(8):1194–1206

    Article  Google Scholar 

  22. Urtasun R, Fua P (2004) 3D tracking for gait characterization and recognition. In: Sixth IEEE international conference on automatic face and gesture recognition, 2004. Proceedings. IEEE, pp 17–22

  23. Urtasun R, Fua P (2004) 3D tracking for gait characterization and recognition. In: Sixth IEEE international conference on automatic face and gesture recognition, 2004. Proceedings. IEEE, pp 17–22

  24. Yam C, Nixon MS, Carter JN (2004) Automated person recognition by walking and running via model-based approaches. Pattern Recognit 37(5):1057–1072

    Article  Google Scholar 

  25. Feng Y, Li Y, Luo J (2016) Learning effective gait features using LSTM. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE, pp 325–330

  26. Liao R, Cao C, Garcia EB, Yu S, Huang Y (2017) Pose-based temporal-spatial network (PTSN) for gait recognition with carrying and clothing variations. In: Chinese conference on biometric recognition. Springer, pp 474–483

  27. Li X, Makihara Y, Xu C, Yagi Y, Yu S, Ren M (2020) End-to-end model-based gait recognition. In: Proceedings of the Asian conference on computer vision

  28. Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY (2011) Multimodal deep learning. In: ICML

  29. Kumar P, Mukherjee S, Saini R, Kaushik P, Roy PP, Dogra DP (2018) Multimodal gait recognition with inertial sensor data and video using evolutionary algorithm. IEEE Trans Fuzzy Syst 27(5):956–965

    Article  Google Scholar 

  30. Castro FM, Marín-Jiménez MJ, Guil N (2016) Multimodal features fusion for gait, gender and shoes recognition. Mach Vis Appl 27(8):1213–1228

    Article  Google Scholar 

  31. Zaheer M, Kottur S, Ravanbakhsh S, Poczos B, Salakhutdinov R, Smola A (2017) Deep sets. arXiv:1703.06114

  32. Takemura N, Makihara Y, Muramatsu D, Echigo T, Yagi Y (2018) Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition. IPSJ Trans Comput Vis Appl 10(1):4

    Article  Google Scholar 

  33. Wei S-E, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional pose machines. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4724–4732

  34. Fu Y, Wei Y, Zhou Y, Shi H, Huang G, Wang X, Yao Z, Huang T (2019) Horizontal pyramid matching for person re-identification. In: Proceedings of the AAAI conference on artificial intelligence, pp 8295–8302

Download references

Acknowledgements

This research work is partly supported by the Zhejiang Provincial Public Welfare Technology Research Project (No.LGF21F020008), the National Natural Science Foundation of China (No.61906101), the Open Project Program of the State Key Lab of CAD&CG (No.A2119), Zhejiang University, and the Ningbo Municipal Natural Science Foundation of China (No.2018A610057).

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering and Computer Science, Ningbo University, Ningbo, Zhejiang, People’s Republic of China

    Liming Zhao, Lijun Guo, Rong Zhang, Xijiong Xie & Xulun Ye

Authors
  1. Liming Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lijun Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xijiong Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xulun Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lijun Guo.

Ethics declarations

Conflict of Interests

We have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, L., Guo, L., Zhang, R. et al. mmGaitSet: multimodal based gait recognition for countering carrying and clothing changes. Appl Intell 52, 2023–2036 (2022). https://doi.org/10.1007/s10489-021-02484-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02484-2

Keywords

Search

Navigation