research-article

Incremental Learning from Low-labelled Stream Data in Open-Set Video Face Recognition

Authors:

Eric Lopez-Lopez,

Carlos V. RegueiroAuthors Info & Claims

Volume 131, Issue C

https://doi.org/10.1016/j.patcog.2022.108885

Published: 01 November 2022 Publication History

Highlights

•

A online approach to unsupervised instance-incremental learning with stream data.

•

Adaptation from pseudo-labels, which are the own predictions of the system.

•

A strategy to deal with catastrophic forgetting and the effect of wrong pseudo-labels.

•

Designed to operate in the open-set, extendable to the class-incremental problem.

•

Method for person re-identification based on face without a reservoir of face images.

Abstract

Deep Learning approaches have brought solutions, with impressive performance, to general classification problems where wealthy of annotated data are provided for training. In contrast, less progress has been made in continual learning of a set of non-stationary classes, mainly when applied to unsupervised problems with streaming data.

Here, we propose a novel incremental learning approach which combines a deep features encoder with an Open-Set Dynamic Ensembles of SVM, to tackle the problem of identifying individuals of interest (IoI) from streaming face data. From a simple weak classifier trained on a few video-frames, our method can use unsupervised operational data to enhance recognition. Our approach adapts to new patterns avoiding catastrophic forgetting and partially heals itself from miss-adaptation. Besides, to better comply with real world conditions, the system was designed to operate in an open-set setting. Results show a benefit of up to 15% F1-score increase respect to non-adaptive state-of-the-art methods.

References

[1]

R. Kemker, M. McClure, A. Abitino, T.L. Hayes, C. Kanan, Measuring catastrophic forgetting in neural networks, AAAI’18/IAAI’18/EAAI’18, AAAI Press, 2018,.

Digital Library

[2]

M. Wang, W. Deng, Deep visual domain adaptation: A survey, Neurocomputing 312 (2018) 135–153,.

Digital Library

[3]

L. Ren, X. Yuan, J. Lu, M. Yang, J. Zhou, Deep reinforcement learning with iterative shift for visual tracking, in: V. Ferrari, M. Hebert, C. Sminchisescu, Y. Weiss (Eds.), Computer Vision – ECCV 2018, Springer International Publishing, Cham, 2018, pp. 697–713,.

Digital Library

[4]

J. He, R. Mao, Z. Shao, F. Zhu, Incremental learning in online scenario, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 13923–13932.

[5]

D. Sahoo, Q. Pham, J. Lu, S.C.H. Hoi, Online deep learning: Learning deep neural networks on the fly, International Joint Conference on Artificial Intelligence (IJCAI), 2018, pp. 2660–2666,.

[6]

X. Tao, X. Hong, X. Chang, S. Dong, X. Wei, Y. Gong, Few-shot class-incremental learning, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 12180–12189,.

[7]

J.-M. Pérez-Rúa, X. Zhu, T.M. Hospedales, T. Xiang, Incremental few-shot object detection, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 13843–13852,.

[8]

M. McCloskey, N.J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, Psychology of Learning and Motivation, volume 24, Academic Press, 1989, pp. 109–165,.

[9]

Z. Huang, S. Shan, R. Wang, H. Zhang, S. Lao, A. Kuerban, X. Chen, A benchmark and comparative study of video-based face recognition on COX face database, IEEE Transactions on Image Processing 24 (12) (2015) 5967–5981,.

Digital Library

[10]

Y. Guo, L. Zhang, Y. Hu, X. He, J. Gao, MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition, European Conference on Computer Vision (ECCV), 2016, pp. 87–102.

[11]

M. Günther, L.E. Shafey, S. Marcel, Face Recognition in Challenging Environments: An Experimental and Reproducible Research Survey, Springer International Publishing, 2016, pp. 247–280.

[12]

E. López-López, X.M. Pardo, C.V. Regueiro, R. Iglesias, F.E. Casado, Dataset bias exposed in face verification, IET Biometrics 8 (4) (2019) 249–258,.

[13]

G. Guo, N. Zhang, A survey on deep learning based face recognition, Computer Vision and Image Understanding 189 (2019) 102805,.

Digital Library

[14]

S. Disabato, M. Roveri, Learning convolutional neural networks in presence of concept drift, International Joint Conference on Neural Networks (IJCNN), 2019, pp. 1–8,.

[15]

D. Maltoni, V. Lomonaco, Continuous learning in single-incremental-task scenarios, Neural Networks 116 (2019) 56–73,.

Digital Library

[16]

W.J. Scheirer, A. de Rezende Rocha, A. Sapkota, T.E. Boult, Toward open set recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (7) (2013) 1757–1772,.

Digital Library

[17]

M. Günther, S. Cruz, E.M. Rudd, T.E. Boult, Toward open-set face recognition, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2017, pp. 573–582,.

[18]

D. Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, Annual Meeting on Association for Computational Linguistics (ACL), 1995, pp. 189–196,.

Digital Library

[19]

C. Geng, S. Huang, S. Chen, Recent advances in open set recognition: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020),.

[20]

Z. Ge, S. Demyanov, Z. Chen, R. Garnavi, Generative openmax for multi-class open set classification, British Machine Vision Conference Proceedings (BMVC), 2017,.

[21]

P. Perera, V.I. Morariu, R. Jain, V. Manjunatha, C. Wigington, V. Ordonez, V.M. Patel, Generative-discriminative feature representations for open-set recognition, Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11811–11820.

[22]

S. Coles, Classical Extreme Value Theory and Models, Springer London, London, 2001, pp. 45–73.

[23]

E.M. Rudd, L.P. Jain, W.J. Scheirer, T.E. Boult, The extreme value machine, IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (3) (2018) 762–768,.

[24]

G. Salomon, A. Britto, R.H. Vareto, W.R. Schwartz, D. Menotti, Open-set face recognition for small galleries using siamese networks, International Conference on Systems, Signals and Image Processing (IWSSIP), 2020, pp. 161–166,.

[25]

F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for face recognition and clustering, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 815–823,.

[26]

G.I. Parisi, R. Kemker, J.L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural Networks 113 (2019) 54–71,.

Digital Library

[27]

T.L. Hayes, C. Kanan, Lifelong machine learning with deep streaming linear discriminant analysis, IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020, pp. 887–896,.

[28]

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A.A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, R. Hadsell, Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences 114 (13) (2017) 3521–3526,.

[29]

J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, C. Jay Kuo, Class-incremental learning via deep model consolidation, IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 1120–1129,.

[30]

G.M. van de Ven, H.T. Siegelmann, A.S. Tolias, Brain-inspired replay for continual learning with artificial neural networks, Nature Communications 11 (1) (2020) 4069,.

[31]

S. Ud Din, J. Shao, J. Kumar, W. Ali, J. Liu, Y. Ye, Online reliable semi-supervised learning on evolving data streams, Information Sciences 525 (2020) 153–171,.

[32]

Y. Li, Y. Wang, Q. Liu, C. Bi, X. Jiang, S. Sun, Incremental semi-supervised learning on streaming data, Pattern Recognition 88 (2019) 383–396,.

Digital Library

[33]

M. De-la Torre, E. Granger, P.V.W. Radtke, R. Sabourin, D.O. Gorodnichy, Partially-supervised learning from facial trajectories for face recognition in video surveillance, Information Fusion 24 (2015) 31–53,.

Digital Library

[34]

P.H. Pisani, A. Mhenni, R. Giot, E. Cherrier, N. Poh, F. de Carvalho André Carlos Ponce, C. Rosenberger, N.E.B. Amara, Adaptive biometric systems: Review and perspectives, ACM Comput. Surv. 52 (5) (2019) 102:1–102:38,.

Digital Library

[35]

G. Orrú, G.L. Marcialis, F. Roli, A novel classification-selection approach for the self updating of template-based face recognition systems, Pattern Recognition 100 (2020) 107–121,.

Digital Library

[36]

A. Franco, D. Maio, D. Maltoni, Incremental template updating for face recognition in home environments, Pattern Recognition 43 (8) (2010) 2891–2903,.

Digital Library

[37]

F. Pernici, A.D. Bimbo, Unsupervised incremental learning of deep descriptors from video streams, IEEE International Conference on Multimedia Expo Workshops (ICMEW), 2017, pp. 477–482,.

[38]

R. Coop, A. Mishtal, I. Arel, Ensemble learning in fixed expansion layer networks for mitigating catastrophic forgetting, IEEE Transactions on Neural Networks and Learning Systems 24 (10) (2013) 1623–1634,.

[39]

B. Krawczyk, L.L. Minku, J. Gama, J. Stefanowski, M. Woźniak, Ensemble learning for data stream analysis: A survey, Information Fusion 37 (2017) 132–156,.

Digital Library

[40]

H.M. Gomes, J.P. Barddal, F. Enembreck, A. Bifet, A survey on ensemble learning for data stream classification, ACM Comput. Surv. 50 (2017),.

Digital Library

[41]

N. Liang, G. Huang, P. Saratchandran, N. Sundararajan, A fast and accurate online sequential learning algorithm for feedforward networks, IEEE Transactions on Neural Networks 17 (6) (2006) 1411–1423,.

Digital Library

[42]

N. Dvornik, J. Mairal, C. Schmid, Diversity with cooperation: Ensemble methods for few-shot classification, 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 3722–3730,.

[43]

X. Zhang, F. Yan, Y. Zhuang, H. Hu, C. Bu, Using an ensemble of incrementally fine-tuned CNNs for cross-domain object category recognition, IEEE Access 7 (2019) 33822–33833,.

[44]

Y. Guo, X. Wang, P. Xiao, X. Xu, An ensemble learning framework for convolutional neural network based on multiple classifiers, Soft Computing 24 (2020),.

Digital Library

[45]

J. Wang, Z. Mo, H. Zhang, Q. Miao, Ensemble diagnosis method based on transfer learning and incremental learning towards mechanical big data, Measurement 155 (2020) 107517,.

[46]

J. Deng, J. Guo, N. Xue, S. Zafeiriou, Arcface: Additive angular margin loss for deep face recognition, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4685–4694,.

[47]

H. Liu, X. Zhu, Z. Lei, S.Z. Li, Adaptiveface: Adaptive margin and sampling for face recognition, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11939–11948,.

[48]

W. Scheirer, A. Rocha, R. Micheals, T. Boult, Robust fusion: Extreme value theory for recognition score normalization, European Conference on Computer Vision (ECCV), 2010, pp. 481–495,.

[49]

N. Li, Y. Yu, Z.-H. Zhou, Diversity regularized ensemble pruning, Machine Learning and Knowledge Discovery in Databases, 2012, pp. 330–345,.

[50]

Z. Cheng, X. Zhu, S. Gong, Surveillance face recognition challenge, arXiv preprint arXiv:1804.09691 (2018).

[51]

R. Goh, L. Liu, X. Liu, T. Chen, The CMU Face In Action (FIA) Database, in: W. Zhao, S. Gong, X. Tang (Eds.), Analysis and Modelling of Faces and Gestures, Springer, 2005, pp. 255–263,.

Digital Library

[52]

L. Wolf, T. Hassner, I. Maoz, Face recognition in unconstrained videos with matched background similarity, CVPR 2011, 2011, pp. 529–534,.

Digital Library

[53]

K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Processing Letters 23 (10) (2016) 1499–1503,.

Cited By

Sun JDong Q(2024)Conditional feature generation for transductive open-set recognition via dual-space consistent samplingPattern Recognition10.1016/j.patcog.2023.110046146:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110046
Parga CPardo XRegueiro C(2024)Mapping the Unknown: A New Approach to Open-World Video RecognitionPattern Recognition10.1007/978-3-031-78189-6_10(144-159)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78189-6_10
Parga CVilariño GPardo XRegueiro C(2023)-LOR: Supervised Stream Learning for Object RecognitionPattern Recognition and Image Analysis10.1007/978-3-031-36616-1_24(300-311)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-36616-1_24

Index Terms

Incremental Learning from Low-labelled Stream Data in Open-Set Video Face Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Open-set face recognition across look-alike faces in real-world scenarios

The open-set problem is among the problems that have significantly changed the performance of face recognition algorithms in real-world scenarios. Open-set operates under the supposition that not all the probes have a pair in the gallery. Most face ...
Enhancing Open-Set Face Recognition by Closing It with Cluster-Inferred Gallery Augmentation
Pattern Recognition
Abstract
In open-set face recognition—as opposed to closed-set face recognition—it is possible that the identity of a given query is not present in the gallery set. In that case, the identity of the query can only be correctly classified as “unknown” when ...
Partially-supervised learning from facial trajectories for face recognition in video surveillance

Face recognition (FR) is employed in several video surveillance applications to determine if facial regions captured over a network of cameras correspond to a target individuals. To enroll target individuals, it is often costly or unfeasible to capture ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Pattern Recognition

Pattern Recognition Volume 131, Issue C

Nov 2022

837 pages

ISSN:0031-3203

Issue’s Table of Contents

Copyright © 2022.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 November 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun JDong Q(2024)Conditional feature generation for transductive open-set recognition via dual-space consistent samplingPattern Recognition10.1016/j.patcog.2023.110046146:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110046
Parga CPardo XRegueiro C(2024)Mapping the Unknown: A New Approach to Open-World Video RecognitionPattern Recognition10.1007/978-3-031-78189-6_10(144-159)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1007/978-3-031-78189-6_10
Parga CVilariño GPardo XRegueiro C(2023)-LOR: Supervised Stream Learning for Object RecognitionPattern Recognition and Image Analysis10.1007/978-3-031-36616-1_24(300-311)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1007/978-3-031-36616-1_24

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents