research-article

Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images

Authors:

Qinghua ZhengAuthors Info & Claims

Volume 236, Issue C

https://doi.org/10.1016/j.knosys.2021.107678

Published: 25 January 2022 Publication History

Abstract

Facial Expression Recognition (FER) for various low-resolution images is an important task and need in applications of analyzing crowd scenes (station, classroom, etc.). Due to the discriminative feature loss caused by reduced resolution, classifying various low-resolution facial images into the right category is still a challenging task. In this work, we proposed a novel generative adversarial network-based feature level super-resolution method for robust facial expression recognition (FSR-FER), which can reduce the chance of privacy leaking without restoring high-resolution facial images. In particular, a pre-trained FER model was employed as a feature extractor, and a generator network G and a discriminator network D are trained with features extracted from low-resolution and corresponding high-resolution images. Generator network G tries to transform features of low-resolution images to more discriminative ones by making them closer to the ones of corresponding high-resolution images. For better classification performance, we also proposed an effective classification-aware loss reweighting strategy based on the classification probability calculated by a fixed FER model to make our model focus more on samples that are prone to misclassification. Experimental results on the Real-World Affective Faces (RAF) Database and Static Facial Expressions in the Wild (SFEW) 2.0 dataset demonstrate that our method achieves satisfying results on various down-sample factors with a single model and has better performance on low-resolution images compared with methods using image super-resolution and expression recognition separately.

References

[1]

Tang J., Zhou X., Zheng J., Design of intelligent classroom facial recognition based on deep learning, J. Phys.: Conf. Ser. 1168 (2) (2019).

[2]

Lukas S., Mitra A.R., Desanti R.I., Krisnadi D., Student attendance system in classroom using face recognition technique, in: 2016 International Conference on Information and Communication Technology Convergence, ICTC, IEEE, 2016, pp. 1032–1035.

[3]

Hilles M.M., Naser S.S.A., Knowledge-based intelligent tutoring system for teaching mongo database, 2017.

[4]

K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

[5]

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1–9.

[6]

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, Y. Fu, Residual dense network for image super-resolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2472–2481.

[7]

B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee, Enhanced deep residual networks for single image super-resolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 136–144.

[8]

T. Dai, J. Cai, Y. Zhang, S.-T. Xia, L. Zhang, Second-order attention network for single image super-resolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11065–11074.

[9]

X. Hu, H. Mu, X. Zhang, Z. Wang, T. Tan, J. Sun, Meta-SR: a magnification-arbitrary network for super-resolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 1575–1584.

[10]

Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, Y. Fu, Image super-resolution using very deep residual channel attention networks, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018, pp. 286–301.

[11]

W.-S. Lai, J.-B. Huang, N. Ahuja, M.-H. Yang, Deep laplacian pyramid networks for fast and accurate super-resolution, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 624–632.

[12]

Liu Z., Li L., Wu Y., Zhang C., Facial expression restoration based on improved graph convolutional networks, in: International Conference on Multimedia Modeling, Springer, 2020, pp. 527–539.

[13]

Cheng B., Wang Z., Zhang Z., Li Z., Liu D., Yang J., Huang S., Huang T.S., Robust emotion recognition from low quality and low bit rate video: A deep learning approach, in: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction, ACII, IEEE, 2017, pp. 65–70.

[14]

W. Tan, B. Yan, B. Bare, Feature super-resolution: Make machine see more clearly, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3994–4002.

[15]

J. Noh, W. Bae, W. Lee, J. Seo, G. Kim, Better to follow, follow to be better: Towards precise supervision of feature super-resolution for small object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 9725–9734.

[16]

J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, S. Yan, Perceptual generative adversarial networks for small object detection, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1222–1230.

[17]

D. Acharya, Z. Huang, D. Pani Paudel, L. Van Gool, Covariance pooling for facial expression recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 367–374.

[18]

S. Li, W. Deng, J. Du, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2852–2861.

[19]

Ding H., Zhou S.K., Chellappa R., Facenet2expnet: Regularizing a deep face recognition net for expression recognition, in: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2017, IEEE, 2017, pp. 118–126.

[20]

H. Yang, U. Ciftci, L. Yin, Facial expression recognition by de-expression residue learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2168–2177.

[21]

F. Zhang, T. Zhang, Q. Mao, C. Xu, Joint pose and expression modeling for facial expression recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 3359–3368.

[22]

Vo T.-H., Lee G.-S., Yang H.-J., Kim S.-H., Pyramid with super resolution for in-the-wild facial expression recognition, IEEE Access 8 (2020) 131988–132001.

[23]

Wang K., Peng X., Yang J., Meng D., Qiao Y., Region attention networks for pose and occlusion robust facial expression recognition, IEEE Trans. Image Process. 29 (2020) 4057–4069.

Digital Library

[24]

B. Hasani, M.H. Mahoor, Facial expression recognition using enhanced deep 3D convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 30–40.

[25]

Zhao X., Liang X., Liu L., Li T., Han Y., Vasconcelos N., Yan S., Peak-piloted deep network for facial expression recognition, in: European Conference on Computer Vision, Springer, 2016, pp. 425–442.

[26]

Huang G.B., Mattar M., Berg T., Learned-Miller E., Labeled faces in the wild: A database forstudying face recognition in unconstrained environments, 2008.

[27]

Hesse N., Gehrig T., Gao H., Ekenel H.K., Multi-view facial expression recognition using local appearance features, in: Proceedings of the 21st International Conference on Pattern Recognition, ICPR2012, IEEE, 2012, pp. 3533–3536.

[28]

H. Jung, S. Lee, J. Yim, S. Park, J. Kim, Joint fine-tuning in deep neural networks for facial expression recognition, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2983–2991.

[29]

Z. Huang, L. Van Gool, A riemannian network for spd matrix learning, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017.

[30]

Yu K., Salzmann M., Second-order convolutional neural networks, 2017, arXiv preprint arXiv:1703.06817.

[31]

Dong C., Loy C.C., He K., Tang X., Image super-resolution using deep convolutional networks, IEEE Trans. Pattern Anal. Mach. Intell. 38 (2) (2015) 295–307.

Digital Library

[32]

Tuzel O., Porikli F., Meer P., Region covariance: A fast descriptor for detection and classification, in: European Conference on Computer Vision, Springer, 2006, pp. 589–600.

[33]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.

[34]

J. Wu, Z. Huang, J. Thoma, D. Acharya, L. Van Gool, Wasserstein divergence for gans, in: Proceedings of the European Conference on Computer Vision, ECCV, pp. 653–668.

[35]

Arjovsky M., Chintala S., Bottou L., Wasserstein gan, 2017, arXiv preprint arXiv:1701.07875.

[36]

Cao Y., Chen K., Loy C.C., Lin D., Prime sample attention in object detection, 2019, arXiv preprint arXiv:1904.04821.

[37]

Johnson J., Alahi A., Fei-Fei L., Perceptual losses for real-time style transfer and super-resolution, in: European Conference on Computer Vision, Springer, 2016, pp. 694–711.

[38]

T.-Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2980–2988.

[39]

X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, C. Change Loy, Esrgan: Enhanced super-resolution generative adversarial networks, in: Proceedings of the European Conference on Computer Vision, ECCV, 2018.

[40]

Kingma D.P., Ba J., Adam: A method for stochastic optimization, 2014, arXiv preprint arXiv:1412.6980.

[41]

A. Dhall, R. Goecke, J. Joshi, K. Sikka, T. Gedeon, Emotion recognition in the wild challenge 2014: Baseline, data and protocol, in: Proceedings of the 16th International Conference on Multimodal Interaction, 2014, pp. 461–466.

[42]

Dhall A., Goecke R., Lucey S., Gedeon T., Collecting large, richly annotated facial-expression databases from movies, IEEE Ann. Hist. Comput. 19 (03) (2012) 34–41.

[43]

J. Kim, J. Kwon Lee, K. Mu Lee, Accurate image super-resolution using very deep convolutional networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1646–1654.

[44]

Smilkov D., Thorat N., Nicholson C., Reif E., Viégas F.B., Wattenberg M., Embedding projector: Interactive visualization and interpretation of embeddings, 2016, arXiv preprint arXiv:1611.05469.

Cited By

Ying WDong TFan J(2024)An efficient multi-scale learning method for image super-resolution networksNeural Networks10.1016/j.neunet.2023.10.015169:C(120-133)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.neunet.2023.10.015
Tan YXia HSong S(2024)Learning informative and discriminative semantic features for robust facial expression recognitionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.10406298:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2024.104062
Zhang YSun Z(2024)Cross-domain facial expression recognition based on adversarial attack fine-tuning learningEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109014136:PBOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.109014
Show More Cited By

Index Terms

Feature super-resolution based Facial Expression Recognition for multi-scale low-resolution images
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Framework for reliable, real-time facial expression recognition for low resolution images

Automatic recognition of facial expressions is a challenging problem specially for low spatial resolution facial images. It has many potential applications in human-computer interactions, social robots, deceit detection, interactive video and behavior ...
Multi-pose Facial Expression Recognition Based on Unpaired Images
Image and Graphics
Abstract
Giving machines the ability to perceive human emotions and enable them to recognize our emotional states is one of the important goals to realize human-computer interaction. In the past decades, facial expression recognition (FER) has always been ...
Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems

Knowledge-Based Systems Volume 236, Issue C

Jan 2022

578 pages

ISSN:0950-7051

Issue’s Table of Contents

The Authors.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 25 January 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ying WDong TFan J(2024)An efficient multi-scale learning method for image super-resolution networksNeural Networks10.1016/j.neunet.2023.10.015169:C(120-133)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1016/j.neunet.2023.10.015
Tan YXia HSong S(2024)Learning informative and discriminative semantic features for robust facial expression recognitionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.10406298:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.jvcir.2024.104062
Zhang YSun Z(2024)Cross-domain facial expression recognition based on adversarial attack fine-tuning learningEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.109014136:PBOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.109014
Yang MWu CGuo YHe YJiang RJiang JYang Z(2024)A teacher–student deep learning strategy for extreme low resolution unsafe action recognition in construction projectsAdvanced Engineering Informatics10.1016/j.aei.2023.10229459:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.aei.2023.102294
Li XShao ZLi BShang YWu JDuan Y(2023)Residual shuffle attention network for image super-resolutionMachine Vision and Applications10.1007/s00138-023-01436-934:5Online publication date: 16-Aug-2023
https://dl.acm.org/doi/10.1007/s00138-023-01436-9
An LSun XZhang ZWang M(2023)Attention and Relative Distance Alignment for Low-Resolution Facial Expression RecognitionPattern Recognition and Computer Vision10.1007/978-981-99-8469-5_18(225-237)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1007/978-981-99-8469-5_18
Adyapady RAnnappa B(2022)A comprehensive review of facial expression recognition techniquesMultimedia Systems10.1007/s00530-022-00984-w29:1(73-103)Online publication date: 30-Jul-2022
https://dl.acm.org/doi/10.1007/s00530-022-00984-w

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents