research-article

Combining Facial Parts For Learning Gender, Ethnicity, and Emotional State Based on RGB-D Information

Authors:

Safaa Azzakhnini,

Lahoucine Ballihi,

Driss AboutajdineAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 14, Issue 1s

Article No.: 19, Pages 1 - 14

https://doi.org/10.1145/3152125

Published: 06 March 2018 Publication History

Abstract

With the success of emerging RGB-D cameras such as the Kinect sensor, combining the shape (depth) and texture information to improve the quality of recognition became a trend among computer vision researchers. In this work, we address the problem of face classification in the context of RGB images and depth data. Inspired by the psychological results for human face perception, this article focuses on (i) finding out which facial parts are most effective at making the difference for some social aspects of face perception (gender, ethnicity, and emotional state), (ii) determining the optimal decision by combining the decision rendered by the individual parts, and (iii) extracting the promising features from RGB-D faces to exploit all the potential that this data provide. Experimental results on EurecomKinect Face and CurtinFaces databases show that the proposed approach improves the recognition quality in many use cases.

References

[1]

Andrea F. Abate, Michele Nappi, Daniel Riccio, and Gabriele Sabatino. 2007. 2D and 3D face recognition: A survey. Pattern Recogn. Lett. 28, 14 (2007), 1885--1906.

Digital Library

[2]

Timo Ahonen, Abdenour Hadid, and Matti Pietikainen. 2006. Face description with local binary patterns: Application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28, 12 (2006), 2037--2041.

Digital Library

[3]

Rahul Ajmera, Aditya Nigam, and Phalguni Gupta. 2014. 3D face recognition using kinect. In Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing. ACM, 76.

Digital Library

[4]

Lahoucine Ballihi, Boulbaba Ben Amor, Mohamed Daoudi, Anuj Srivastava, and Driss Aboutajdine. 2012. Boosting 3D-geometric features for efficient face recognition and gender classification. IEEE Trans. Info. Forens. Secur. 7, 6 (2012), 1766--1779.

Digital Library

[5]

Lahoucine Ballihi, Adel Lablack, Boulbaba Ben Amor, Ioan Marius Bilasco, and Mohamed Daoudi. 2015. Positive/negative emotion detection from RGB-D upper body images. In Face and Facial Expression Recognition from Real World Videos. Springer, 109--120.

[6]

Stefano Berretti, Alberto Del Bimbo, and Pietro Pala. 2012. Distinguishing facial features for ethnicity-based 3D face recognition. ACM Trans. Intell. Syst. Technol. (TIST) 3, 3 (2012), 45.

Digital Library

[7]

Mauro Birattari, Zhi Yuan, Prasanna Balaprakash, and Thomas Stützle. 2010. F-race and iterated F-race: An overview. Exper. Methods Anal. Optim. Algor. (2010), 311--336.

[8]

Elhocine Boutellaa, Abdenour Hadid, Messaoud Bengherabi, and Samy Ait-Aoudia. 2015. On the use of kinect depth data for identity, gender and ethnicity classification from facial images. Pattern Recogn. Lett. (2015).

Digital Library

[9]

Kevin W. Bowyer, Kyong Chang, and Patrick Flynn. 2006. A survey of approaches and challenges in 3D and multi-modal 3D+ 2D face recognition. Comput. Vision Image Understand. 101, 1 (2006), 1--15.

Digital Library

[10]

Elizabeth Brown and David I. Perrett. 1993. What gives a face its gender? Perception 22, 7 (1993), 829--840.

[11]

Vicki Bruce, A. Mike Burton, Elias Hanna, Pat Healey, Oli Mason, Anne Coombes, Rick Fright, and Alf Linney. 1993. Sex discrimination: How do we tell the difference between male and female faces? Perception 22, 2 (1993), 131--152.

[12]

A. Mike Burton, Vicki Bruce, and Neal Dench. 1993. What’s the difference between men and women? Evidence from facial measurement. Perception 22, 2 (1993), 153--176.

[13]

João Baptista Cardia Neto and Aparecido Nilceu Marana. 2015. 3DLBP and HAOG fusion for face recognition utilizing kinect as a 3D scanner. In Proceedings of the 30th Annual ACM Symposium on Applied Computing. ACM, 66--73.

Digital Library

[14]

Alessandro Cellerino, Davide Borghetti, and Ferdinando Sartucci. 2004. Sex differences in face gender recognition in humans. Brain Res. Bull. 63, 6 (2004), 443--449.

[15]

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn. 20, 3 (1995), 273--297.

Digital Library

[16]

Xu Dai, Shouyi Yin, Peng Ouyang, Leibo Liu, and Shaojun Wei. 2015. A multi-modal 2D+ 3D face recognition method with a novel local feature descriptor. In Proceedings of the 2015 IEEE Winter Conference on Applications of Computer Vision. IEEE, 657--662.

Digital Library

[17]

Yoav Freund, Robert E. Schapire et al. 1996. Experiments with a new boosting algorithm. In Proceedings of the ICML, Vol. 96. 148--156.

Digital Library

[18]

Jerome H. Friedman. 2001. Greedy function approximation: A gradient boosting machine. Ann. Stat. (2001), 1189--1232.

[19]

P. Gallinari, Yann Lecun, S. Thiria, and F. Fogelman Soulie. 1987. Mémoires associatives distribuées: Une comparaison (distributed associative memories: A comparison). In Proceedings of COGNITIVA 87. Cesta-Afcet.

[20]

Hamed Kiani Galoogahi and Terence Sim. 2012. Inter-modality face sketch recognition. In Proceedings of the 2012 IEEE International Conference on Multimedia and Expo. IEEE, 224--229.

Digital Library

[21]

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. Retrieved from http://www.deeplearningbook.org.

Digital Library

[22]

Gaurav Goswami, Samarth Bharadwaj, Mayank Vatsa, and Rajdeep Singh. 2013. On RGB-D face recognition using kinect. In Proceedings of the 2013 IEEE 6th International Conference on Biometrics: Theory, Applications and Systems (BTAS). IEEE, 1--6.

[23]

Munawar Hayat, Mohammed Bennamoun, and Amar A. El-Sallam. 2016. An RGB--D based image set classification for robust face recognition from kinect data. Neurocomputing 171 (2016), 889--900.

Digital Library

[24]

R. I. Hg, Petr Jasek, Clement Rofidal, Kamal Nasrollahi, Thomas B. Moeslund, and Gabrielle Tranchet. 2012. An RGB-D database using microsoft’s kinect for windows for face detection. In Proceedings of the 2012 8th International Conference on Signal Image Technology and Internet Based Systems (SITIS). IEEE, 42--46.

Digital Library

[25]

Gee-Sern Jison Hsu, Yu-Lun Liu, Hsiao-Chia Peng, and Po-Xun Wu. 2014. RGB-D-based face reconstruction and recognition. IEEE Trans. Info. Forensics Secur. 9, 12 (2014), 2110--2118.

Digital Library

[26]

Yonggang Huang, Yunhong Wang, and Tieniu Tan. 2006. Combining statistics of geometrical and correlative features for 3D face recognition. In Proceedings of BMVC. Edinburgh, 879--888.

[27]

Tri Huynh, Rui Min, and Jean-Luc Dugelay. 2013. An efficient LBP-based descriptor for facial depth images applied to gender recognition using RGB-D face data. In Proceedings of the Computer Vision Workshops (ACCV’12). Springer, 133--145.

Digital Library

[28]

Qiu Jin, Jieyu Zhao, and Yuanyuan Zhang. 2012. Facial feature extraction with a depth AAM algorithm. In Proceedings of the 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD’12). IEEE, 1792--1796.

[29]

Poornima Krishnan and S. Naveen. 2015. RGB-D face recognition system verification using kinect and FRAV3D databases. Procedia Comput. Sci. 46 (2015), 1653--1660.

[30]

Billy Y. L. Li, Ajmal Mian, Wanquan Liu, and Aneesh Krishna. 2013. Using kinect for face recognition under varying poses, expressions, illumination and disguise. In Proceedings of the 2013 IEEE Workshop on Applications of Computer Vision (WACV’13). IEEE, 186--192.

Digital Library

[31]

R. Sala Llonch, Effrosini Kokiopoulou, I. Tošić, and Pascal Frossard. 2010. 3D face recognition with sparse spherical representations. Pattern Recogn. 43, 3 (2010), 824--834.

Digital Library

[32]

Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. 2016. The irace package: Iterated racing for automatic algorithm configuration. Operat. Res. Perspect. 3 (2016), 43--58.

[33]

Alessandra Lumini and Loris Nanni. 2017. Overview of the combination of biometric matchers. Info. Fusion 33 (2017), 71--85.

Digital Library

[34]

Alessandra Lumini, Loris Nanni, and Sheryl Brahnam. 2016. Ensemble of texture descriptors and classifiers for face recognition. Appl. Comput. Info. (2016).

[35]

Stephen Milborrow. 2013. Stasm 4 user manual. Retrieved from http://www.milbo.org.stasm-files/stasm4.pdf.

[36]

S. Milborrow and F. Nicolls. 2014. Active shape models with SIFT descriptors and MARS. VISAPP. Retrieved from http://www.milbo.users.sonic.net/stasm.

[37]

Rui Min, Jongmoo Choi, Gérard Medioni, and Jean-Luc Dugelay. 2012. Real-time 3D face identification from a depth camera. In Proceedings of the 2012 21st International Conference on Pattern Recognition (ICPR’12). IEEE, 1739--1742.

[38]

Rui Min, Neslihan Kose, and Jean-Luc Dugelay. 2014. KinectFaceDB: A kinect database for face recognition. IEEE Trans. Syst., Man, Cybernet.: Syst. 44, 11 (2014), 1534--1548.

[39]

Stepán Mrácek, Martin Drahanskỳ, Radim Dvorák, Ivo Provaznik, and Jan Vána. 2014. 3D face recognition on low-cost depth sensors. In Proceedings of BIOSIG. 195--202.

[40]

Andrew Ng. 2011. Sparse autoencoder. CS294A Lecture Notes 72, 2011 (2011), 1--19.

[41]

Mauricio Pamplona Segundo, Santonu Sarkar, Dmitry Goldgof, Leandro Silva, and Olga Bellon. 2013. Continuous 3D face authentication using RGB-D cameras. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’13). IEEE, 64--69.

Digital Library

[42]

Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1701--1708.

Digital Library

[43]

Chenghua Xu, Yunhong Wang, Tieniu Tan, and Long Quan. 2004. A new attempt to face recognition using 3D eigenfaces. In Proceedings of the Asian Conference on Computer Vision, Vol. 2. Citeseer, 884--889.

[44]

L. E. Yann. 1987. Modèles Connexionnistes de lapprentissage. Ph.D. Dissertation. These de Doctorat, Universite Paris 6.

Cited By

Zhang PDou HZhang WZhao YQin ZHu DFang YLi X(2023)A Large-Scale Synthetic Gait Dataset Towards in-the-Wild Simulation and Comparison StudyACM Transactions on Multimedia Computing, Communications, and Applications10.1145/351719919:1(1-23)Online publication date: 5-Jan-2023
https://dl.acm.org/doi/10.1145/3517199
Ali SGanapathi IAlsarhan TWerghi N(2023)Enhanced authentication system with robust features for the secure user template2023 IEEE 12th International Conference on Communication Systems and Network Technologies (CSNT)10.1109/CSNT57126.2023.10134714(742-747)Online publication date: 8-Apr-2023
https://doi.org/10.1109/CSNT57126.2023.10134714
Sharma UTomar PAli SSaxena NBhadoria R(2021)Optimized Authentication System with High Security and PrivacyElectronics10.3390/electronics1004045810:4(458)Online publication date: 13-Feb-2021
https://doi.org/10.3390/electronics10040458
Show More Cited By

Index Terms

Combining Facial Parts For Learning Gender, Ethnicity, and Emotional State Based on RGB-D Information
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Biometrics

Recommendations

Automatic face analysis system based on face recognition and facial physiognomy
ICHIT'06: Proceedings of the 1st international conference on Advances in hybrid information technology

An automatic face analysis system is proposed which uses face recognition and facial physiognomy. It first detects human's face, extracts its features, and classifies the shape of facial features. It will analyze the person's facial physiognomy and then ...
Facial marks: soft biometric for face recognition
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

We propose to utilize micro features, namely facial marks (e.g., freckles, moles, and scars) to improve face recognition and retrieval performance. Facial marks can be used in three ways: i) to supplement the features in an existing face matcher, ii) to ...
Face recognition using gender information
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

In this paper, we propose a novel method using gender information for achieving better performances of face recognition systems. Gender is one of the important factors for recognizing appearance of human faces and there are many studies on gender ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 14, Issue 1s

Special Section on Representation, Analysis and Recognition of 3D Humans and Special Section on Multimedia Computing and Applications of Socio-Affective Behaviors in the Wild

March 2018

234 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3190503

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 March 2018

Accepted: 01 October 2017

Revised: 01 October 2017

Received: 01 February 2017

Published in TOMM Volume 14, Issue 1s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
236
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang PDou HZhang WZhao YQin ZHu DFang YLi X(2023)A Large-Scale Synthetic Gait Dataset Towards in-the-Wild Simulation and Comparison StudyACM Transactions on Multimedia Computing, Communications, and Applications10.1145/351719919:1(1-23)Online publication date: 5-Jan-2023
https://dl.acm.org/doi/10.1145/3517199
Ali SGanapathi IAlsarhan TWerghi N(2023)Enhanced authentication system with robust features for the secure user template2023 IEEE 12th International Conference on Communication Systems and Network Technologies (CSNT)10.1109/CSNT57126.2023.10134714(742-747)Online publication date: 8-Apr-2023
https://doi.org/10.1109/CSNT57126.2023.10134714
Sharma UTomar PAli SSaxena NBhadoria R(2021)Optimized Authentication System with High Security and PrivacyElectronics10.3390/electronics1004045810:4(458)Online publication date: 13-Feb-2021
https://doi.org/10.3390/electronics10040458
Umair MNazir M(2021)Classification of Demographic Attributes from Facial Image by using CNN2021 International Conference on Artificial Intelligence (ICAI)10.1109/ICAI52203.2021.9445248(68-73)Online publication date: 5-Apr-2021
https://doi.org/10.1109/ICAI52203.2021.9445248
Boumedine ABentaieb SOuamri AMallek A(2021)3D Face Identification Using HOG Features and Collaborative RepresentationAdvances in Computing Systems and Applications10.1007/978-3-030-69418-0_1(3-13)Online publication date: 21-Feb-2021
https://doi.org/10.1007/978-3-030-69418-0_1
Gabani KMehta MNoronha S(2020)Racial Categorization Methods: A SurveyAdvances in Science, Technology and Engineering Systems Journal10.25046/aj0503505:3(388-401)Online publication date: 2020
https://doi.org/10.25046/aj050350
H. de B. Zavan FR. P. Bellon OSilva LMedioni G(2019)Benchmarking parts based face processing in-the-wild for gender recognition and head pose estimationPattern Recognition Letters10.1016/j.patrec.2018.09.023123:C(104-110)Online publication date: 15-May-2019
https://dl.acm.org/doi/10.1016/j.patrec.2018.09.023

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents