research-article

Learning from More: Combating Uncertainty Cross-multidomain for Facial Expression Recognition

Authors:

Hui XiaoAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 5889 - 5898

https://doi.org/10.1145/3581783.3611702

Published: 27 October 2023 Publication History

Abstract

Domain adaptation has driven the progress of Facial Expression Recognition (FER). Existing cross-domain FER methods focus only on the domain alignment of a single source domain to the target domain, ignoring the importance of multisource domains that contain richer knowledge. However, Cross-Multidomain FER (CMFER)needs to combat the domain conflicts caused by the uncertainty of intradomain annotations and the inconsistency of interdomain distributions. To this end, this paper proposes a Domain-Uncertain Mutual Learning (DUML) method to deal with the more challenging CMFER problem. Specifically, we consider a domain-specific global perspective for domain-invariance representation and domain fusion for facial generic detail representation to mitigate cross-domain distribution differences. Further, we develop Intra-Domain Uncertainty (Intra-DU) and Inter-Domain Uncertainty (Inter-DU) to combat the large dataset shifts caused by annotation uncertainty. Finally, extensive experimental results on multiple benchmark across multidomain FER datasets demonstrate the remarkable effectiveness of DUML against CMFER uncertainty. All codes and training logs are publicly available at https://github.com/liuhw01/DUML.

Supplemental Material

MP4 File

Presentation Video

Download
48.22 MB

References

[1]

Wissam J Baddar and Yong Man Ro. 2019. Mode variational lstm robust to unseen modes of variation: Application to facial expression recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 3215--3223.

Digital Library

[2]

Zechen Bai, Zhigang Wang, Jian Wang, Di Hu, and Errui Ding. 2021. Unsupervised multi-source domain adaptation for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12914--12923.

[3]

Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016. Training deep networks for facial expression recognition with crowd-sourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 279--283.

Digital Library

[4]

Dongliang Chen, Peng Song, and Wenming Zheng. 2021b. Learning Transferable Sparse Representations for Cross-corpus Facial Expression Recognition. IEEE Transactions on Affective Computing (2021).

[5]

Shikai Chen, Jianfeng Wang, Yuedong Chen, Zhongchao Shi, Xin Geng, and Yong Rui. 2020. Label distribution learning on auxiliary label space graphs for facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 13984--13993.

[6]

Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, Lingbo Liu, and Liang Lin. 2021a. Cross-domain facial expression recognition: A unified evaluation benchmark and adversarial graph learning. IEEE transactions on pattern analysis and machine intelligence (2021).

[7]

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5203--5212.

[8]

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4690--4699.

[9]

Yangye Fu, Ming Zhang, Xing Xu, Zuo Cao, Chao Ma, Yanli Ji, Kai Zuo, and Huimin Lu. 2021. Partial Feature Selection and Alignment for Multi-Source Domain Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16654--16663.

[10]

Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning. PMLR, 1180--1189.

[11]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research, Vol. 17, 1 (2016), 2096--2030.

[12]

Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. 2013. Challenges in representation learning: A report on three machine learning contests. In International conference on neural information processing. Springer, 117--124.

[13]

Jihun Hamm, Christian G Kohler, Ruben C Gur, and Ragini Verma. 2011. Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. Journal of neuroscience methods, Vol. 200, 2 (2011), 237--256.

[14]

Yanli Ji, Yuhan Hu, Yang Yang, and Heng Tao Shen. 2021. Region Attention Enhanced Unsupervised Cross-Domain Facial Emotion Recognition. IEEE Transactions on Knowledge and Data Engineering (2021).

[15]

Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, Vol. 30 (2017).

[16]

Mengxue Li, Yi-Ming Zhai, You-Wei Luo, Peng-Fei Ge, and Chuan-Xian Ren. 2020. Enhanced transport distance for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13936--13944.

[17]

Ruihuang Li, Xu Jia, Jianzhong He, Shuaijun Chen, and Qinghua Hu. 2021c. T-SVDNet: Exploring High-Order Prototypical Correlations for Multi-Source Domain Adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9991--10000.

[18]

Shan Li, Weihong Deng, and JunPing Du. 2017. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2852--2861.

[19]

Yingjian Li, Yingnan Gao, Bingzhi Chen, Zheng Zhang, Lei Zhu, and Guangming Lu. 2021a. JDMAN: Joint Discriminative and Mutual Adaptation Networks for Cross-Domain Facial Expression Recognition. In Proceedings of the 29th ACM International Conference on Multimedia. 3312--3320.

Digital Library

[20]

Yingjian Li, Yingnan Gao, Bingzhi Chen, Zheng Zhang, Lei Zhu, and Guangming Lu. 2021b. Jdman: Joint discriminative and mutual adaptation networks for cross-domain facial expression recognition. In Proceedings of the 29th ACM International Conference on Multimedia. 3312--3320.

Digital Library

[21]

Yunsheng Li, Lu Yuan, Yinpeng Chen, Pei Wang, and Nuno Vasconcelos. 2021d. Dynamic transfer for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10998--11007.

[22]

Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. 2018. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Transactions on Image Processing, Vol. 28, 5 (2018), 2439--2450.

[23]

Yingjian Li, Zheng Zhang, Bingzhi Chen, Guangming Lu, and David Zhang. 2022. Deep margin-sensitive representation learning for cross-domain facial expression recognition. IEEE Transactions on Multimedia (2022).

Digital Library

[24]

Hanwei Liu, Huiling Cai, Qingcheng Lin, Xuefeng Li, and Hui Xiao. 2022. Adaptive multilayer perceptual attention network for facial expression recognition. IEEE Transactions on Circuits and Systems for Video Technology, Vol. 32, 9 (2022), 6253--6266.

[25]

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015. Learning transferable features with deep adaptation networks. In International conference on machine learning. PMLR, 97--105.

[26]

Patrick Lucey, Jeffrey F Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. 2010. The extended cohn-kanade dataset (ck): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, 94--101.

[27]

Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, Vol. 10, 1 (2017), 18--31.

Digital Library

[28]

Eduardo Fernandes Montesuma and Fred Maurice Ngole Mboula. 2021. Wasserstein barycenter for multi-source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16785--16793.

[29]

Van-Anh Nguyen, Tuan Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. 2021. Stem: An approach to multi-source domain adaptation with guarantees. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9352--9363.

[30]

Tongguang Ni, Cong Zhang, and Xiaoqing Gu. 2020. Transfer model collaborating metric learning and dictionary learning for cross-domain facial expression recognition. IEEE Transactions on Computational Social Systems, Vol. 8, 5 (2020), 1213--1222.

[31]

Bowen Pan, Shangfei Wang, and Bin Xia. 2019. Occluded facial expression recognition enhanced through privileged information. In Proceedings of the 27th ACM international conference on multimedia. 566--573.

Digital Library

[32]

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision. 1406--1415.

[33]

Delian Ruan, Yan Yan, Shenqi Lai, Zhenhua Chai, Chunhua Shen, and Hanzi Wang. 2021. Feature decomposition and reconstruction learning for effective facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7660--7669.

[34]

Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen, and Tao Mei. 2021. Dive into ambiguity: Latent distribution mining and pairwise uncertainty estimation for facial expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6248--6257.

[35]

Manogna Sreenivas, Sawa Takamuku, Soma Biswas, Aditya Chepuri, Balasubramanian Vengatesan, and Naotake Natori. 2023 a. Improved Cross-Dataset Facial Expression Recognition by Handling Data Imbalance and Feature Confusion. In Computer Vision--ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. Springer, 262--277.

Digital Library

[36]

Manogna Sreenivas, Sawa Takamuku, Soma Biswas, Aditya Chepuri, Balasubramanian Vengatesan, and Naotake Natori. 2023 b. Improved Cross-Dataset Facial Expression Recognition by Handling Data Imbalance and Feature Confusion. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V. Springer, 262--277.

Digital Library

[37]

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. 2017. Adversarial discriminative domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7167--7176.

[38]

Guoqing Wang, Hu Han, Shiguang Shan, and Xilin Chen. 2020a. Cross-domain face presentation attack detection via multi-domain disentangled representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6678--6687.

[39]

Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. 2020d. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In European Conference on Computer Vision. Springer, 727--744.

Digital Library

[40]

Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. 2020b. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6897--6906.

[41]

Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao. 2020c. Region attention networks for pose and occlusion robust facial expression recognition. IEEE Transactions on Image Processing, Vol. 29 (2020), 4057--4069.

Digital Library

[42]

Jiaxi Wu, Jiaxin Chen, Mengzhe He, Yiru Wang, Bo Li, Bingqi Ma, Weihao Gan, Wei Wu, Yali Wang, and Di Huang. 2022. Target-relevant knowledge preservation for multi-source domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5301--5310.

[43]

Yuanyuan Xu, Meina Kan, Shiguang Shan, and Xilin Chen. 2022. Mutual Learning of Joint and Separate Domain Alignments for Multi-Source Domain Adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 1890--1899.

[44]

Jiabei Zeng, Shiguang Shan, and Xilin Chen. 2018. Facial expression recognition with inconsistently annotated datasets. In Proceedings of the European conference on computer vision (ECCV). 222--237.

Digital Library

[45]

Kun Zhang, Mingming Gong, and Bernhard Schölkopf. 2015. Multi-source domain adaptation: A causal view. In Twenty-ninth AAAI conference on artificial intelligence.

[46]

Wenjing Zhang, Peng Song, and Wenming Zheng. 2022. Joint Local-Global Discriminative Subspace Transfer Learning for Facial Expression Recognition. IEEE Transactions on Affective Computing (2022).

[47]

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4320--4328.

[48]

Guoying Zhao, Xiaohua Huang, Matti Taini, Stan Z Li, and Matti PietikäInen. 2011. Facial expression recognition from near-infrared videos. Image and vision computing, Vol. 29, 9 (2011), 607--619.

[49]

Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. 2018. Adversarial multiple source domain adaptation. Advances in neural information processing systems, Vol. 31 (2018).

[50]

Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. 2020. Multi-source distilling domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12975--12983.

[51]

Yukun Zuo, Hantao Yao, and Changsheng Xu. 2021. Attention-based multi-source domain adaptation. IEEE Transactions on Image Processing, Vol. 30 (2021), 3793--3803.

Cited By

Yang YWen LZeng XXu YWu XZhou JWang YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Learning with Alignments: Tackling the Inter- and Intra-domain Shifts for Cross-multidomain Facial Expression RecognitionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680747(4236-4245)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680747
Liu HAn RZhang ZMa BZhang WSong YHu YChen WDing Y(2024)Norface: Improving Facial Expression Analysis by Identity NormalizationComputer Vision – ECCV 202410.1007/978-3-031-73001-6_17(293-314)Online publication date: 27-Nov-2024
https://doi.org/10.1007/978-3-031-73001-6_17

Index Terms

Learning from More: Combating Uncertainty Cross-multidomain for Facial Expression Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Biometrics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Virtual reality

Recommendations

Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Pose-robust feature learning for facial expression recognition

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods ...
Facial expression recognition with Convolutional Neural Networks

Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Shanghai Municipal Commission of Science and Technology Project
Shanghai Municipal Science and Technology Major Project
Shanghai Science and Technology Planning Project
Fundamental Research Funds for the Central Universities

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
263
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)9

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang YWen LZeng XXu YWu XZhou JWang YCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Learning with Alignments: Tackling the Inter- and Intra-domain Shifts for Cross-multidomain Facial Expression RecognitionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680747(4236-4245)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680747
Liu HAn RZhang ZMa BZhang WSong YHu YChen WDing Y(2024)Norface: Improving Facial Expression Analysis by Identity NormalizationComputer Vision – ECCV 202410.1007/978-3-031-73001-6_17(293-314)Online publication date: 27-Nov-2024
https://doi.org/10.1007/978-3-031-73001-6_17

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten