research-article

AttenGait: : Gait recognition with attention and rich modalities

Authors:

Francisco M. Castro,

Rubén Delgado-Escaño,

Ruber Hernández-García,

Manuel J. Marín-Jiménez,

Nicolás GuilAuthors Info & Claims

Volume 148, Issue C

https://doi.org/10.1016/j.patcog.2023.110171

Published: 17 April 2024 Publication History

Abstract

Current gait recognition systems employ different types of manual attention mechanisms, like horizontal cropping of the input data to guide the training process and extract useful gait signatures for people identification. Typically, these techniques are applied using silhouettes as input, which limits the learning capabilities of the models. Thus, due to the limited information provided by silhouettes, state-of-the-art gait recognition approaches must use very simple and manually designed mechanisms, in contrast to approaches proposed for other topics such as action recognition. To tackle this problem, we propose AttenGait, a novel model for gait recognition equipped with trainable attention mechanisms that automatically discover interesting areas of the input data. AttenGait can be used with any kind of informative modalities, such as optical flow, obtaining state-of-the-art results thanks to the richer information contained in those modalities. We evaluate AttenGait on two public datasets for gait recognition: CASIA-B and GREW; improving the previous state-of-the-art results on them, obtaining 95.8% and 70.7% average accuracy, respectively. Code will be available at https://github.com/fmcp/attengait.

Highlights

•

Novel cross-view gait recognition model equipped with trainable attention mechanisms.

•

Three novel attention blocks to discover important data regions for gait recognition.

•

Thorough experimental study to measure the impact of the different modalities.

•

State-of-the-art results for the public CASIA-B and GREW datasets.

References

[1]

Sepas-Moghaddam A., Etemad A., Deep gait recognition: A survey, IEEE Trans. Pattern Anal. Mach. Intell. (2022).

[2]

Castro F.M., Marín-Jiménez M.J., Guil N., de la Blanca N.P., Automatic learning of gait signatures for people identification, in: IWANN, Vol. 10306, 2017, pp. 257–270.

[3]

Delgado-Escaño R., Castro F.M., Cózar J.R., Marín-Jiménez M.J., Guil N., An end-to-end multi-task and fusion CNN for inertial-based gait recognition, IEEE Access 7 (2019) 1897–1908.

[4]

Delgado-Escaño R., Castro F.M., Guil N., Marín-Jiménez M.J., GaitCopy: Disentangling appearance for gait recognition by signature copy, IEEE Access 9 (2021) 164339–164347.

[5]

Castro F.M., Marín-Jiménez M.J., Guil N., de la Blanca N.P., Multimodal feature fusion for CNN-based gait recognition: an empirical comparison, Neural Comput. Appl. (2020) 1–21.

[6]

H. Chao, Y. He, J. Zhang, J. Feng, Gaitset: Regarding gait as a set for cross-view gait recognition, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2019.

[7]

C. Fan, Y. Peng, C. Cao, X. Liu, S. Hou, J. Chi, Y. Huang, Q. Li, Z. He, GaitPart: Temporal Part-Based Model for Gait Recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14225–14233.

[8]

S. Hou, C. Cao, X. Liu, Y. Huang, Gait lateral network: Learning discriminative and compact representations for gait recognition, in: European Conference on Computer Vision, 2020, pp. 382–398.

[9]

B. Lin, S. Zhang, X. Yu, Gait recognition via effective global-local feature representation and local temporal aggregation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14648–14656.

[10]

S. d’Ascoli, H. Touvron, M.L. Leavitt, A.S. Morcos, G. Biroli, L. Sagun, Convit: Improving vision transformers with soft convolutional inductive biases, in: International Conference on Machine Learning, 2021, pp. 2286–2296.

[11]

A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, C. Schmid, Vivit: A video vision transformer, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 6836–6846.

[12]

S. Yu, D. Tan, T. Tan, A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition, in: Proc. ICPR, Vol. 4, 2006, pp. 441–444.

[13]

Takemura N., Makihara Y., Muramatsu D., Echigo T., Yagi Y., Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition, IPSJ Trans. Comput. Vis. Appl. 10 (1) (2018) 4.

[14]

Z. Zhu, X. Guo, T. Yang, J. Huang, J. Deng, G. Huang, D. Du, J. Lu, J. Zhou, Gait recognition in the wild: A benchmark, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14789–14799.

[15]

Phuong T.T., et al., Privacy-preserving deep learning via weight transmission, IEEE Trans. Inf. Forensics Secur. 14 (11) (2019) 3003–3015.

[16]

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021.

[17]

C. Fan, J. Liang, C. Shen, S. Hou, Y. Huang, S. Yu, OpenGait: Revisiting Gait Recognition Towards Better Practicality, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 9707–9716.

[18]

Wu Z., Huang Y., Wang L., Wang X., Tan T., A comprehensive study on cross-view gait based human identification with deep CNNs, IEEE Trans. Pattern Anal. Mach. Intell. 39 (2) (2017) 209–226.

Digital Library

[19]

Delgado-Escaño R., Castro F.M., Cózar J.R., Marín-Jiménez M.J., Guil N., Casilari E., A cross-dataset deep learning-based classifier for people fall detection and identification, Comput. Methods Programs Biomed. 184 (2020).

[20]

Z. Meng, S. Fu, J. Yan, H. Liang, A. Zhou, S. Zhu, H. Ma, J. Liu, N. Yang, Gait Recognition for Co-existing Multiple People Using Millimeter Wave Sensing, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2019.

[21]

An W., Yu S., Makihara Y., Wu X., Xu C., Yu Y., Liao R., Yagi Y., Performance evaluation of model-based gait on multi-view very large population database with pose sequences, IEEE Trans. Biom. Behav. Identity Sci. 2 (4) (2020) 421–430.

[22]

Liao R., Yu S., An W., Huang Y., A model-based gait recognition method with body pose and human prior knowledge, Pattern Recognit. 98 (2020).

[23]

X. Li, Y. Makihara, C. Xu, Y. Yagi, S. Yu, M. Ren, End-to-end model-based gait recognition, in: Proceedings of the Asian Conference on Computer Vision, 2020.

[24]

T. Teepe, J. Gilg, F. Herzog, S. Hörmann, G. Rigoll, Towards a deeper understanding of skeleton-based gait recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1569–1577.

[25]

Wang L., Chen J., Liu Y., Frame-level refinement networks for skeleton-based gait recognition, Comput. Vis. Image Underst. 222 (2022).

[26]

J. Zheng, X. Liu, W. Liu, L. He, C. Yan, T. Mei, Gait Recognition in the Wild With Dense 3D Representations and a Benchmark, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

[27]

Xu C., Makihara Y., Li X., Yagi Y., Occlusion-aware human mesh model-based gait recognition, IEEE Trans. Inf. Forensics Secur. (2023).

[28]

F.M. Castro, M.J. Marín-Jiménez, N. Guil, S. López-Tapia, N.P. de la Blanca, Evaluation of CNN architectures for gait recognition based on optical flow maps, in: International Conference of the Biometrics Special Interest Group, 2017, pp. 251–258.

[29]

Kumar P., Mukherjee S., Saini R., Kaushik P., Roy P.P., Dogra D.P., Multimodal gait recognition with inertial sensor data and video using evolutionary algorithm, IEEE Trans. Fuzzy Syst. 27 (5) (2018) 956–965.

[30]

Marín-Jiménez M.J., Castro F.M., Delgado-Escaño R., Kalogeiton V., Guil N., UGaitNet: Multimodal gait recognition with missing input modalities, IEEE Trans. Inf. Forensics Secur. 16 (2021) 5452–5462.

[31]

J. Liang, C. Fan, S. Hou, C. Shen, Y. Huang, S. Yu, GaitEdge: Beyond plain end-to-end gait recognition for better practicality, in: European Conference on Computer Vision, 2022.

[32]

Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, Adv. Neural Inf. Process. Syst. (2017).

[33]

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou, Training data-efficient image transformers & distillation through attention, in: International Conference on Machine Learning, 2021, pp. 10347–10357.

[34]

L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F.E. Tay, J. Feng, S. Yan, Tokens-to-token vit: Training vision transformers from scratch on imagenet, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[35]

K. Li, Y. Wang, P. Gao, G. Song, Y. Liu, H. Li, Y. Qiao, Uniformer: Unified Transformer for Efficient Spatiotemporal Representation Learning, in: International Conference on Learning Representations, 2022.

[36]

Fan H., Xiong B., Mangalam K., Li Y., Yan Z., Malik J., Feichtenhofer C., Multiscale vision transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[37]

H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, L. Zhang, Cvt: Introducing convolutions to vision transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

[38]

Zhang Y., Huang Y., Yu S., Wang L., Cross-view gait recognition by discriminative feature learning, IEEE Trans. Image Process. (2019).

[39]

M. Wang, B. Lin, X. Guo, L. Li, Z. Zhu, J. Sun, S. Zhang, Y. Liu, X. Yu, GaitStrip: Gait Recognition via Effective Strip-based Feature Representations and Multi-Level Framework, in: Proceedings of the Asian Conference on Computer Vision, 2022.

[40]

Li G., Guo L., Zhang R., Qian J., Gao S., TransGait: Multimodal-based gait recognition with set transformer, Appl. Intell. (2022) 1–13.

[41]

Mogan J.N., Lee C.P., Lim K.M., Muthu K.S., Gait-ViT: Gait recognition with vision transformer, Sensors 22 (19) (2022) 7362.

[42]

J. Li, Y. Zhang, H. Shan, J. Zhang, Gaitcotr: Improved Spatial-Temporal Representation for Gait Recognition with a Hybrid Convolution-Transformer Framework, in: IEEE International Conference on Acoustics, Speech, and Signal Processing, 2023.

[43]

Y. Cui, Y. Kang, Multi-Modal Gait Recognition via Effective Spatial-Temporal Feature Fusion, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[44]

T. Teepe, A. Khan, J. Gilg, F. Herzog, S. Hörmann, G. Rigoll, Gaitgraph: Graph convolutional network for skeleton-based gait recognition, in: IEEE International Conference on Image Processing, 2021.

[45]

Fan C., Hou S., Huang Y., Yu S., Exploring deep models for practical gait recognition, 2023, arXiv preprint arXiv:2303.03301.

[46]

Liang J., Fan C., Hou S., Shen C., Huang Y., Yu S., Gaitedge: Beyond plain end-to-end gait recognition for better practicality, in: European Conference on Computer Vision, Springer, 2022, pp. 375–390.

[47]

G. Farnebäck, Two-Frame Motion Estimation Based on Polynomial Expansion, in: Image Analysis: 13th Scandinavian Conference, Vol. 2749, 2003, pp. 363–370.

[48]

X. Huang, D. Zhu, H. Wang, X. Wang, B. Yang, B. He, W. Liu, B. Feng, Context-sensitive temporal feature learning for gait recognition, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12909–12918.

[49]

H. Dou, P. Zhang, W. Su, Y. Yu, Y. Lin, X. Li, GaitGCI: Generative Counterfactual Intervention for Gait Recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[50]

K. Ma, Y. Fu, D. Zheng, C. Cao, X. Hu, Y. Huang, Dynamic Aggregated Network for Gait Recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[51]

R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2017, pp. 618–626.

[52]

McInnes L., Healy J., Melville J., Umap: Uniform manifold approximation and projection for dimension reduction, arXiv (2018).

Cited By

Xi HRen KLu PLi YHu C(2024)SSGait: enhancing gait recognition via semi-supervised self-supervised learningApplied Intelligence10.1007/s10489-024-05385-254:7(5639-5657)Online publication date: 24-Apr-2024
https://dl.acm.org/doi/10.1007/s10489-024-05385-2

Recommendations

On the Generalization Power of Face and Gait in Gender Recognition

Human face/gait-based gender recognition has been intensively studied in the previous literatures, yet most of them are based on the same database. Although nearly perfect gender recognition rates can be achieved in the same face/gait dataset, they ...
Electrical stimulation of hip adductors and abductors improves gait parameters of children with spastic diplegic cerebral palsy: -pilot study-
iCREATe '10: Proceedings of the 4th International Convention on Rehabilitation Engineering & Assistive Technology

Electrical stimulation (ES) of the hip adductor and abductor muscles simultaneously during functional walking training has never been reported as a management option to improve gait in spastic diplegic children. Objective: To investigate the effects of ...
Analyzing Walking with Ankle Foot Orthoses Using Shank-mounted Wearable Movement Sensors
ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility

An Ankle Foot Orthosis (AFO) is a device that supports the ankle joint to help improve gait. When fitting AFOs, clinicians tune the AFO, and the success of the tuning is often determined by observation, e.g. of a patient walking up and down a corridor. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 148, Issue C

Apr 2024

747 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 17 April 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xi HRen KLu PLi YHu C(2024)SSGait: enhancing gait recognition via semi-supervised self-supervised learningApplied Intelligence10.1007/s10489-024-05385-254:7(5639-5657)Online publication date: 24-Apr-2024
https://dl.acm.org/doi/10.1007/s10489-024-05385-2

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents