Abstract
Face recognition is an efficient technique and one of the most preferred biometric modalities for the identification and verification of individuals as compared to voice, fingerprint, iris, retina eye scan, gait, ear and hand geometry. This has over the years necessitated researchers in both the academia and industry to come up with several face recognition techniques making it one of the most studied research area in computer vision. A major reason why it remains a fast-growing research lies in its application in unconstrained environments, where most existing techniques do not perform optimally. Such conditions include pose, illumination, ageing, occlusion, expression, plastic surgery and low resolution. In this paper, a critical review on the different issues of face recognition systems are presented, and different approaches to solving these issues are analyzed by presenting existing techniques that have been proposed in the literature. Furthermore, the major and challenging face datasets that consist of the different facial constraints which depict real-life scenarios are also discussed stating the shortcomings associated with them. Also, recognition performance on the different datasets by researchers are also reported. The paper is concluded, and directions for future works are highlighted.
Similar content being viewed by others
References
Abate AF, Nappi M, Riccio D, Sabatino G (2007) 2D and 3D face recognition: a survey. Pattern Recogn Lett 28:1885–1906
Ali ASO, Sagayan V, Malik A, Aziz A (2016) Proposed face recognition system after plastic surgery. IET Comput Vis 10:344–350
Alkkiomaki O, Kyrki V, Liu Y, Handroos H, and Kalviainen H (2009) Multi-modal force/vision sensor fusion in 6-DOF pose tracking," in Advanced Robotics. ICAR 2009. International conference on 2009,, pp. 1–8.
Angadi SA, Kagawade VC (2017) A robust face recognition approach through symbolic modeling of polar FFT features. Pattern Recogn 71:235–248
Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Trans Neural Netw 13:1450–1464
Belahcene M, Chouchane A, and Ouamane H (2014) 3D face recognition in presence of expressions by fusion regions of interest," in 2014 22nd Signal Processing and Communications Applications Conference (SIU), pp. 2269–2274.
Bhat FA, Wani MA (2016) Elastic bunch graph matching based face recognition under varying lighting, pose, and expression conditions. IAES International Journal of Artificial Intelligence (IJ-AI) 3:177–182
Bolme DS (2003) Elastic bunch graph matching. Colorado State University
Bowyer KW, Chang K, Flynn P (2006) A survey of approaches and challenges in 3D and multi-modal 3D+ 2D face recognition. Comput Vis Image Underst 101:1–15
Breiman L (2001) Random forests. Mach Learn 45:5–32
Brunelli R, Poggio T (1993) Face recognition: features versus templates. IEEE Trans Pattern Anal Mach Intell 15:1042–1052
Cao X, Shen W, Yu L, Wang Y, Yang J, Zhang Z (2012) Illumination invariant extraction for face recognition using neighboring wavelet coefficients. Pattern Recogn 45:1299–1305
Chen L, Liang M, Song W, and Xiao K (2018) A multi-scale parallel convolutional neural network based intelligent human identification using face information. Journal of Information Processing Systems, vol. 14.
Cheng EJ, Chou KP, Jin S, Tanveer M, Lin CT, Young KY, Lin WC, Prasad M (2019) Deep sparse representation classifier for facial recognition and detection system. Pattern Recogn Lett 125:71–77
Chihaoui M, Elkefi A, Bellil W, Ben Amar C (2016) A survey of 2D face recognition techniques. Computers 5:21
Chu Y, Ahmad T, Bebis G, Zhao L (2017) Low-resolution face recognition with single sample per person. Signal Process 141:144–157
Chude-Olisah CC, Sulong G, Chude-Okonkwo UA, Hashim SZ (2014) Face recognition via edge-based Gabor feature representation for plastic surgery-altered images. EURASIP Journal on Advances in Signal Processing 2014:102
Delac K, Grgic M, Grgic S (2005) Independent comparative study of PCA, ICA, and LDA on the FERET data set. Int J Imaging Syst Technol 15:252–260
Deng W, Jiani H, Jun G (2017) Face recognition via collaborative representation: its discriminant nature and superposed representation. IEEE Transaction on pattern analysis and machine intelligence 40:2513–2521
Ding C, Tao D (2017) Pose-invariant face recognition with homography-based normalization. Pattern Recogn 66:144–152
Ding C, Hu Z, Karmoshi S, Zhu M (2017) A novel two-stage learning pipeline for deep neural networks. Neural processing letters
Drira H, Amor BB, Srivastava A, Daoudi M, Slama R (2013) 3D face recognition under expressions, occlusions, and pose variations. IEEE Trans Pattern Anal Mach Intell 35:2270–2283
Feng Z-H, Kittler J, Awais M, Huber P, and Wu X-J (2017) Face detection, bounding box aggregation and pose estimation for robust facial landmark localisation in the Wild, arXiv preprint arXiv:1705.02402.
Fu Y, Wu X, Wen Y, Xiang Y (2017) Efficient locality-constrained occlusion coding for face recognition. Neurocomputing 260:104–111
Gao G, Yang J, Jing X-Y, Shen F, Yang W, Yue D (2017) Learning robust and discriminative low-rank representations for face recognition with occlusion. Pattern Recogn 66:129–143
Gao C-z, Cheng Q, He P, Susilo W, Li J (2018) Privacy-preserving naive Bayes classifiers secure against the substitution-then-comparison attack. Inf Sci 444:72–88
Ghiass RS, Arandjelović O, Bendada A, Maldague X (2014) Infrared face recognition: a comprehensive review of methodologies and datasets. Pattern Recogn 47:2807–2824
Goyal SJ, Upadhyay AK, Jadon R, and Goyal R (2018) Real-life facial expression recognition systems: a review," in Smart Computing and Informatics, ed: Springer, pp. 311–331.
Guo Y, Zhang L, Hu Y, He X, and Gao J (2016) Ms-celeb-1m: A dataset and benchmark for large-scale face recognition, in European Conference on Computer Vision, pp. 87–102.
Hanmandlu M, Gupta D, and Vasikarla S (2013) Face recognition using Elastic bunch graph matching. in Applied Imagery Pattern Recognition Workshop (AIPR): Sensing for Control and Augmentation, 2013 IEEE, pp. 1–7.
Heo J, Marios S (2008) Face recognition across pose using view based active appearance models on CMU multi-PIE dataset. In Proceeding of International Conference on Computer Vision Systems, May, pp 527–535
Hijazi S, Kumar R, and Rowen C (2015) Using convolutional neural networks for image recognition, ed.
Ho C, Morgado P, Persekian A, Vasconcelos N (2019) "PIEs: pose invariant Embeddings," IEEE/CVF conference on computer vision and pattern recognition (CVPR). Long Beach, CA, USA, pp 12369–12378. https://doi.org/10.1109/CVPR.2019.01266
Hsu G-SJ, Ambikapathi A, Chung S-L, Shie H-C (2018) Robust cross-pose face recognition using landmark oriented depth warping. J Vis Commun Image Represent 53:273–280
Hu H (2008) ICA-based neighborhood preserving analysis for face recognition. Comput Vis Image Underst 112:286–295
Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 41:765–781
Jia S, Lansdall-Welfare T, and Cristianini N (2016) Gender classification by deep learning on millions of weakly labelled images, in Data Mining Workshops (ICDMW), 2016 IEEE 16th International Conference on, pp. 462–467.
Jiang L, Li C, Wang S, Zhang L (2016) Deep feature weighting for naive Bayes and its application to text classification. Eng Appl Artif Intell 52:26–39
Jin X, Tan X (2017) Face alignment in-the-wild: a survey. Comput Vis Image Underst 162:1–22
Jin T, Liu Z, Yu Z, Min X, Li L (2017) Locality preserving collaborative representation for face recognition. Neural Process Lett 45:967–979
Kakadiaris IA, Toderici G, Evangelopoulos G, Passalis G, Chu D, Zhao X, Shah SK, Theoharis T (2017) 3D-2D face recognition with pose and illumination normalization. Comput Vis Image Underst 154:137–151
Karamizadeh S, Abdullah SM, Zamani M, Shayan J, and Nooralishahi P (2017) Face recognition via taxonomy of illumination normalization," in Multimedia Forensics and Security, ed: Springer, pp. 139–160.
Kim P (2017) Convolutional Neural Network, in MATLAB Deep Learning, ed: Springer, pp. 121–147.
Kotropoulos C, Pitas I, Fischer S, and Duc B (1997) Face authentication using morphological dynamic link architecture," in Audio-and Video-based Biometric Person Authentication, pp. 169–176.
Lades M, Vorbruggen JC, Buhmann J, Lange J, von der Malsburg C, Wurtz RP, Konen W (1993) Distortion invariant object recognition in the dynamic link architecture. IEEE Trans Comput 42:300–311
Lahasan BM, Venkat I, Al-Betar MA, Lutfi SL, De Wilde P (2016) Recognizing faces prone to occlusions and common variations using optimal face subgraphs. Appl Math Comput 283:316–332
Le QV (2013) Building high-level features using large scale unsupervised learning, in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 8595–8598.
Lei G, Li X-h, Zhou J-l, and Gong X-g (2009) Geometric feature based facial expression recognition using multiclass support vector machines," in Granular Computing, 2009, GRC'09. IEEE International Conference on, pp. 318–321.
Li L-y, Li D-r (2010) Research on particle swarm optimization in remote sensing image enhancement [J]. Journal of Geomatics Science and Technology 2:012
Li M, Yuan B (2005) 2D-LDA: a statistical linear discriminant analysis for image matrix. Pattern Recogn Lett 26:527–532
Li Z, Gong D, Li X, Tao D (2016) Aging face recognition: a hierarchical learning model based on local patterns selection. IEEE Trans Image Process 25:2146–2154
Li Y, Wang Y, Liu J, Hao W (2018) Expression-insensitive 3D face recognition by the fusion of multiple subject-specific curves. Neurocomputing 275:1295–1307
Liao S, Lei Z, Yi D, and Li SZ (2014) A benchmark study of large-scale unconstrained face recognition," in Biometrics (IJCB), 2014 IEEE International Joint Conference on, pp. 1–8.
Liu H-D, Yang M, Gao Y, Cui C (2014) Local histogram specification for face recognition under varying lighting conditions. Image Vis Comput 32:335–347
Long Y, Zhu F, Shao L, and Han J (2017) Face recognition with a small occluded training set using spatial and statistical pooling. Inf Sci.
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628
Luan X, Fang B, Liu L, Yang W, Qian J (2014) Extracting sparse error of robust PCA for face recognition in the presence of varying illumination and occlusion. Pattern Recogn 47:495–508
Ma X, Song H, Qian X (2015) Robust framework of single-frame face Superresolution across head pose, facial expression, and illumination variations. IEEE Transactions on Human-Machine Systems 45:238–250
Manjani I, Sumerkan H, Flynn PJ, and Bowyer KW (2016) Template aging in 3D and 2D face recognition," in Biometrics Theory, Applications and Systems (BTAS), 2016 IEEE 8th International Conference on, pp. 1–6.
Martinez AM (1998) The AR face dataset, CVC technical report, vol. 24.
Martins JA, Lam R, Rodrigues J, du Buf J (2018) Expression-invariant face recognition using a biological disparity energy model. Neurocomputing 297:82–93
Masi L, Rawls S, Medioni G, and Natarajan P (2016) Pose-aware face recognition in the wild. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 4838–4846.
Mi J-X, Liu T (2016) Multi-step linear representation-based classification for face recognition. IET Comput Vis 10:836–841
Nappi M, Ricciardi S, Tistarelli M (2016) Deceiving faces: when plastic surgery challenges face recognition. Image Vis Comput 54:71–82
Oloyede MO, Hancke GP (2016) Unimodal and multimodal biometric sensing systems: a review. IEEE Access 4:7532–7555
Oloyede MO, Hancke GP, and Kapileswar N (2017) Evaluating the effect of occlusion in face recognition systems, In Proceedings of IEEE Africon Conference, pp. 1547–1551.
Oloyede MO, Hancke GP, and Myburgh HC (2018) Improving face recognition systems using a new image enhancement technique, hybrid features and the convolutional neural network. IEEE Access, pp. 1–11.
Oloyede MO, Hancke GP, Myburgh HC, and Onumanyi AJ (2019) A new evaluation function for face image in unconstrained environments using metaheuristic algorithms. Eurasip Journal on Image and Video Processing, pp. 1–18.
Ouyang S, Hospedales T, Song Y-Z, Li X, Loy CC, Wang X (2016) A survey on heterogeneous face recognition: sketch, infra-red, 3d and low-resolution. Image Vis Comput 56:28–48
Patacchiola M, Cangelosi A (2017) Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recogn 71:132–143
Pereira JF, Barreto RM, Cavalcanti GD, and Tsang R (2011) A robust feature extraction algorithm based on class-modular image principal component analysis for face verification, in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pp. 1469–1472.
Petpairote C, Madarasmi S, Chamnongthai K (2017) A pose and expression face recognition method using transformation based on single face neutral reference. In Proceedings of IEEE Internationl Conference on Global Wireless Summit, October:123–126
Qi Z, Tian Y, Shi Y (2013) Robust twin support vector machine for pattern classification. Pattern Recogn 46:305–316
Qian Y, Deng W, and Hu J (2019) Unsupervised face normalization with extreme pose and expressionin the wild , In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9851–9858.
Rakshit RD, Kisku DR (2020) Face identification via strategic combination of local features. In Proceedings of Computational Intelligencein Pattern Recognition:207–217
Rasti P, Uiboupin T, Escalera S, and Anbarjafari G (2016) Convolutional neural network super resolution for face recognition in surveillance monitoring, in International Conference on Articulated Motion and Deformable Objects, pp. 175–184.
Rehman A, Saba T (2014) Neural networks for document image preprocessing: state of the art. Artif Intell Rev 42:253–273
Revina IM, Emmanuel WS (2018) Face expression recognition using LDN and dominant gradient local ternary pattern descriptors. Journal of King Saud University-Computer and Information Sciences
Sabharwal T, Rashimi G (2019) Human identification after plastic surgery using region based score level fusion of local facial features. Journal of information security and application 48:102373
Sable AH, Talbar SN, Dhirbasi HA (2017) Recognition of plastic surgery faces and the surgery types: An approach with entropy based scale invariant features. Journal of King Saud University-Computer and Information Sciences
Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37:1113–1133
Savran A, Sankur B (2017) Non-rigid registration based model-free 3D facial expression recognition. Comput Vis Image Underst 162:146–165
Suri S, Sankaran A, Vasta M, Singh R (2018) On matching faces with alterations due to plastic surgery and disguise. In Proceedings of IEEE Conference on Biometrics Theory, Applications and Systems, pp 1–7
Tan S, Xi S, Wenato C, Lei Q, Ling S (2017) Robust face recognition with kernalized locality-sensitive group sparsity representation. IEEE Transaction on image processing 26:4661–4668
Tefas A, Kotropoulos C, and Pitas I (1998) Variants of dynamic link architecture based on mathematical morphology for frontal face authentication, in Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231), pp. 814–819.
Tong Z, Aihara K, and Tanaka G (2016) A hybrid pooling method for convolutional neural networks, in International Conference on Neural Information Processing, pp. 454–461.
Tsai H-H, Chang Y-C (2017) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput:1–17
Turk MA and Pentland AP (1991) Face recognition using eigenfaces, in Computer Vision and Pattern Recognition. Proceedings CVPR'91., IEEE Computer Society Conference on, 1991, pp. 586–591.
Wang K, Chen Z, Wu QJ, Liu C (2017) Illumination and pose variable face recognition via adaptively weighted ULBP_MHOG and WSRC. Signal Process Image Commun 58:175–186
Wang J-W, Le NT, Lee J-S, Wang C-C (2017) Illumination compensation for face recognition using adaptive singular value decomposition in the wavelet domain. Inf Sci
Xanthopoulos P, Pardalos PM, and Trafalis TB (2013) Linear discriminant analysis, in Robust data mining, ed: Springer, pp. 27–33.
Xu C, Liu Q, Ye M (2017) Age invariant face recognition and retrieval by coupled auto-encoder networks. Neurocomputing 222:62–71
Yang J, Luo L, Qian J, Tai Y, Zhang F, Xu Y (2017) Nuclear norm based matrix regression with applications to face recognition with occlusion and illumination changes. IEEE Trans Pattern Anal Mach Intell 39:156–171
Yang J, Ren P, Zhang D, Chen D, Wen F, Li H, and Hua G (2017) Neural aggregation network for video face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4362–4371.
Yu Y-F, Dai D-Q, Ren C-X, Huang K-K (2017) Discriminative multi-layer illumination-robust feature extraction for face recognition. Pattern Recogn 67:201–212
Zafeiriou S, Zhang C, Zhang Z (2015) A survey on face detection in the wild: past, present and future. Comput Vis Image Underst 138:1–24
Zeng S, Jianping G, Deng L (2017) An antinoise sparse representation method for robust face recognition via joint l1and l2 regularization. Expert System with Application 82:1–9
Zhang P, Ben X, Jiang W, Yan R, Zhang Y (2015) Coupled marginal discriminant mappings for low-resolution face recognition. Optik-International Journal for Light and Electron Optics 126:4352–4357
Zhang Y, Lu Y, Wu H, Wen C, and Ge C (2016) Face occlusion detection using cascaded convolutional neural network, in Chinese Conference on Biometric Recognition, pp. 720–727.
Zhang D-x, An P, Zhang H-x (2018) Application of robust face recognition in video surveillance systems. Optoelectron Lett 14:152–155
Zhang MM, Shang K, Wu H (2019) Learning deep discriminative face features by customized weighted constraint. Nuerocomputing 332:71–79
Zhao S (2018) Pixel-level occlusion detection based on sparse representation for face recognition. Optik 168:920–930
Zhao K, Jingyl X, and Cheng MM (2019) Regukarface: Deep face recognition via exclusive regularization”, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1136–1144.
Zhou H, Lam K-M (2018) Age-invariant face recognition based on identity inference from appearance age. Pattern Recogn 76:191–202
Zhou Z, Wagner A, Mobahi H, Wright J, and Ma Y (2009) Face recognition with contiguous occlusion using markov random fields, in Computer Vision, 2009 IEEE 12th International Conference on, pp. 1050–1057.
Zhou L-F, Du Y-W, Li W-S, Mi J-X, Luan X (2018) Pose-robust face recognition with Huffman-LBP enhanced by divide-and-rule strategy. Pattern Recogn
Zhou Q, Zhang C, Yu W, Fan Y, Zhu H, Xiaofu W (2018) Face recognition via fast dense correspondence. Multimed Tools Appl 77:10501–10519
Zhuang L, Chan T-H, Yang AY, Sastry SS, Ma Y (2015) Sparse illumination learning and transfer for single-sample face recognition with image corruption and misalignment. Int J Comput Vis 114:272–287
Acknowledgments
This work was supported by the Council for Scientific and Industrial Research (CSIR), South Africa.
[ICT: Meraka].
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Oloyede, M.O., Hancke, G.P. & Myburgh, H.C. A review on face recognition systems: recent approaches and challenges. Multimed Tools Appl 79, 27891–27922 (2020). https://doi.org/10.1007/s11042-020-09261-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09261-2