research-article

Facial emotion recognition with expression energy

Authors:

Albert C. Cruz,

Bir Bhanu,

Ninad ThakoorAuthors Info & Claims

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Pages 457 - 464

https://doi.org/10.1145/2388676.2388777

Published: 22 October 2012 Publication History

Get Access

Abstract

Facial emotion recognition, the inference of an emotion from apparent facial expressions, in unconstrained settings is a typical case where algorithms perform poorly. A property of the AVEC2012 data set is that individuals in testing data are not encountered in training data. In these situations, conventional approaches suffer because models developed from training data cannot properly discriminate unforeseen testing samples. Additional information beyond the feature vectors is required for successful detection of emotions. We propose two similarity metrics that address the problems of a conventional approach: neutral similarity, measuring the intensity of an expression; and temporal similarity, measuring changes in an expression over time. These similarities are taken to be the energy of facial expressions, measured with a SIFT-based warping process. Our method improves correlation by 35.5% over the baseline approach on the frame-level sub-challenge.

References

[1]

T. Ahonen, A. Hadid, and M. Pietikainen. Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12):2037--2041, 2006.

Digital Library

Google Scholar

[2]

A. Cruz, B. Bhanu, and S. Yang. A psychologically-inspired match-score fusion model for video-based facial expression recognition. In Affective Computing and Intelligent Interaction, Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2011.

Digital Library

Google Scholar

[3]

J. R. Fontaine, K. R. Scherer, E. B. Roesch, and P. C. Ellsworth. The world of emotions is not two-dimensional. Psychological Science, 18(12):1050--1057, 2007.

Crossref

Google Scholar

[4]

H. Gunes and M. Piccardi. Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems Science and Cybernetics Part B, 31(1):64--84, 2009.

Digital Library

Google Scholar

[5]

G. B. Huang, H. Zhou, X. Ding, and R. Zhang. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems Science and Cybernetics Part B, 42(2):513--529, 2012.

Digital Library

Google Scholar

[6]

R. E. Kaliouby and P. Robinson. The emotional hearing aid: an assistive tool for children with asperger syndrome. Universal Access in the Information Society, 4(2):121--134, 2005.

Digital Library

Google Scholar

[7]

C. Liu, J. Yuen, and A. Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5):978--994, 2011.

Digital Library

Google Scholar

[8]

L. Ma and K. Khorasani. Facial expression recognition using constructive feedforward neural networks. IEEE Transactions on Systems Science and Cybernetics Part B, 34(3):1588--1595, 2004.

Digital Library

Google Scholar

[9]

A. Maronidis, D. Bolis, A. Tefas, and I. Pitas. Improving subspace learning for facial expression recognition using person dependent and geometrically enriched training sets. Neural Networks, 24(8):814--823, 2011.

Digital Library

Google Scholar

[10]

T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971--987, 2002.

Digital Library

Google Scholar

[11]

B. Schuller, M. Valstar, F. Eyben, R. Cowie, and M. Pantic. Avec 2012 - the continuous audio/visual emotion challenge. In Proc. Second International Audio/Visual Emotion Challenge and Workshop (AVEC 2012), Grand Challenge and Satellite of ACM ICMI 2012, 2012.

Digital Library

Google Scholar

[12]

L. Wolf, T. Hassner, and Y. Taigman. Effective unconstrained face recognition by combining multiple descriptors and learned background statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10):1978--1990, 2011.

Digital Library

Google Scholar

Cited By

View all

Li JYang CYang FHuang JWei WZhang SZuo XZhang S(2022)Face Detection and Tracking Based on Neural Network2022 3rd International Conference on Information Science, Parallel and Distributed Systems (ISPDS)10.1109/ISPDS56360.2022.9874114(257-260)Online publication date: 22-Jul-2022
https://doi.org/10.1109/ISPDS56360.2022.9874114
Canedo DNeves A(2019)Facial Expression Recognition Using Computer Vision: A Systematic ReviewApplied Sciences10.3390/app92146789:21(4678)Online publication date: 2-Nov-2019
https://doi.org/10.3390/app9214678
Song SSanchez-Lozano ETellamekala MShen LJohnston AValstar M(2019)Dynamic Facial Models for Video-Based Dimensional Affect Estimation2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)10.1109/ICCVW.2019.00200(1608-1617)Online publication date: Oct-2019
https://doi.org/10.1109/ICCVW.2019.00200
Show More Cited By

Index Terms

Facial emotion recognition with expression energy
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Information systems
  1. Information systems applications

Recommendations

Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
Recognizing Action Units for Facial Expression Analysis

Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

October 2012

636 pages

ISBN:9781450314671

DOI:10.1145/2388676

General Chairs:
Louis-Philippe Morency
University of Southern California, USA
,
Dan Bohus
Microsoft Research, USA
,
Hamid Aghajan
Stanford University, USA
,
Program Chairs:
Justine Cassell
Carnegie Mellon University, USA
,
Anton Nijholt
University of Twente, Netherlands
,
Julien Epps
The University of New South Wales, Australia

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '12

Sponsor:

SIGCHI

ICMI '12: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 22 - 26, 2012

California, Santa Monica, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
400
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Li JYang CYang FHuang JWei WZhang SZuo XZhang S(2022)Face Detection and Tracking Based on Neural Network2022 3rd International Conference on Information Science, Parallel and Distributed Systems (ISPDS)10.1109/ISPDS56360.2022.9874114(257-260)Online publication date: 22-Jul-2022
https://doi.org/10.1109/ISPDS56360.2022.9874114
Canedo DNeves A(2019)Facial Expression Recognition Using Computer Vision: A Systematic ReviewApplied Sciences10.3390/app92146789:21(4678)Online publication date: 2-Nov-2019
https://doi.org/10.3390/app9214678
Song SSanchez-Lozano ETellamekala MShen LJohnston AValstar M(2019)Dynamic Facial Models for Video-Based Dimensional Affect Estimation2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)10.1109/ICCVW.2019.00200(1608-1617)Online publication date: Oct-2019
https://doi.org/10.1109/ICCVW.2019.00200
Jha PJoshi B(2017)Facial character detection in in real time2017 International conference of Electronics, Communication and Aerospace Technology (ICECA)10.1109/ICECA.2017.8212722(538-543)Online publication date: Apr-2017
https://doi.org/10.1109/ICECA.2017.8212722
Cruz AZhang ZCohen PBohus DHoraud RMeng H(2015)Quantification of Cinematography Semiotics for Video-based Facial Emotion Recognition in the EmotiW 2015 Grand ChallengeProceedings of the 2015 ACM on International Conference on Multimodal Interaction10.1145/2818346.2830592(511-518)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1145/2818346.2830592
An LYang SBhanu B(2015)Efficient smile detection by Extreme Learning MachineNeurocomputing10.1016/j.neucom.2014.04.072149:PA(354-363)Online publication date: 3-Feb-2015
https://dl.acm.org/doi/10.1016/j.neucom.2014.04.072
Cruz ABhanu BThakoor N(2015)Understanding of the Biological Process of Nonverbal Communication: Facial Emotion and Expression RecognitionVideo Bioinformatics10.1007/978-3-319-23724-4_18(329-347)Online publication date: 16-Dec-2015
https://doi.org/10.1007/978-3-319-23724-4_18
Fewzee PKarray F(2013)Continuous Emotion Recognition: Another Look at the Regression Problem2013 Humaine Association Conference on Affective Computing and Intelligent Interaction10.1109/ACII.2013.39(197-202)Online publication date: Sep-2013
https://doi.org/10.1109/ACII.2013.39

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Facial expression recognition with Convolutional Neural Networks

Recognizing Action Units for Facial Expression Analysis

Expression-invariant face recognition by facial expression transformations