Abstract
The robotic technology demands human-robot interaction to implement a real-time facial emotion detector. This system has a role in recognizing the expressions of the user. Therefore, this application is recommended to work quickly to support the robot’s capabilities. It helps the robot to analyze the customer’s face effectively. However, the previous methods weakly recognize non-frontal faces. It is caused by the facial pose variations only to show partial facial features. This paper proposes a multi-view real-time facial emotion detector based on a lightweight convolutional neural network. It offers a four-stage backbone as an efficient feature extractor that discriminates specific facial components. The convolution with Cross Stage Partial (CSP) approach was employed to reduce computations from convolution operations. The attention module is inserted into the CSP block. These modules also support the detector to work speedily on edge devices. The classification system learns the information about facial features from the KDEF dataset. As a result, facial emotion recognition achieves comparative performance to other methods with an accuracy of 97.10% on the KDEF, 73.95 on the FER-2013, and 84.91% on the RAFDB dataset. The integrated system using a face detector shows that the system obtains a data processing speed of 30 frames per second on the Jetson Nano.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Akhand, M.A.H., Roy, S., Siddique, N., Kamal, M.A.S., Shimamura, T.: Facial emotion recognition using transfer learning in the deep CNN. Electronics 10(9), 1036 (2021)
Calvo, M., Lundqvist, D.: Facial expressions of emotion (KDEF): identification under different display-duration conditions. Behav. Res. Methods 40, 109–115 (1998). http://www.kdef.se/
Dong, J., Zhang, L., Chen, Y., Jiang, W.: Occlusion expression recognition based on non-convex low-rank double dictionaries and occlusion error model. Signal Process.: Image Commun. 76, 81–88 (2019)
Ekman, P.: Facial expressions of emotion: new findings, new questions. Psychol. Sci. 3(1), 34–38 (1992)
Fareed, K., Sultan, F., Khan, K., Mahmood, Z.: A robust face recognition method for expression and pose variant images. In: 2020 14th International Conference on Open Source Systems and Technologies (ICOSST), pp. 1–6 (2020)
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. Neural Netw. 64, 59–63 (2015). Special Issue on “Deep Learning of Representations”
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Li, J., Jin, K., Zhou, D., Kubota, N., Ju, Z.: Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411, 340–350 (2020)
Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2019)
Nan, Y., Ju, J., Hua, Q., Zhang, H., Wang, B.: A-mobilenet: an approach of facial expression recognition. Alex. Eng. J. 61(6), 4435–4444 (2022). http://www.sciencedirect.com/science/article/pii/S1110016821006682
Nguyen, H.D., Kim, S.H., Lee, G.S., Yang, H.J., Na, I.S., Kim, S.H.: Facial expression recognition using a temporal ensemble of multi-level convolutional neural networks. IEEE Trans. Affect. Comput. 13(1), 226–237 (2022)
Pathak, R., Singh, Y.: Real time baby facial expression recognition using deep learning and IoT edge computing. In: 2020 5th International Conference on Computing, Communication and Security (ICCCS), pp. 1–6 (2020)
Putro, M.D., Jo, K.: Real-time face tracking for human-robot interaction. In: Proceedings of the International Conference on Information and Communication Technology Robotics (ICT-ROBOT), pp. 1–4 (2018)
Putro, M.D., Nguyen, D., Jo, K.: Lightweight convolutional neural network for real-time face detector on CPU supporting interaction of service robot. In: 2020 13th International Conference on Human System Interaction (HSI), pp. 94–99 (2020)
Qi, A., Wei, J., Bai, B.: Research on deep learning expression recognition algorithm based on multi-model fusion. In: 2019 International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI), pp. 288–291 (2019)
Rao, Q., Qu, X., Mao, Q., Zhan, Y.: Multi-pose facial expression recognition based on SURF boosting. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 630–635 (2015)
Rujirakul, K., So-In, C.: Histogram equalized deep PCA with ELM classification for expressive face recognition. In: 2018 International Workshop on Advanced Image Technology (IWAIT), pp. 1–4 (2018)
Santra, B., Mukherjee, D.P.: Local saliency-inspired binary patterns for automatic recognition of multi-view facial expression. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 624–628 (2016)
Sirithunge, C., Ravindu, H.M., Bandara, T., Buddhika, A.G., Jayasekara, P., Chandima, D.P.: Situation awareness for proactive robots in HRI. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7813–7820 (2019)
Sun, W., Zhao, H., Jin, Z.: A visual attention based ROI detection method for facial expression recognition. Neurocomputing 296, 12–22 (2018). http://www.sciencedirect.com/science/article/pii/S0925231218303266
Wang, C., Mark Liao, H., Wu, Y., Chen, P., Hsieh, J., Yeh, I.: CSPNet: a new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1571–1580 (2020)
Webb, N., Ruiz-Garcia, A., Elshaw, M., Palade, V.: Emotion recognition from face images in an unconstrained environment for usage on social robots. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020)
Ye, Y., Zhang, X., Lin, Y., Wang, H.: Facial expression recognition via region-based convolutional fusion network. J. Vis. Commun. Image Represent. 62, 1–11 (2019)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Acknowledgment
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the government (MSIT) (No. 2020R1A2C200897212).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Putro, M.D., Nguyen, DL., Priadana, A., Jo, KH. (2022). An Efficient Multi-view Facial Expression Classifier Implementing on Edge Device. In: Szczerbicki, E., Wojtkiewicz, K., Nguyen, S.V., Pietranik, M., Krótkiewicz, M. (eds) Recent Challenges in Intelligent Information and Database Systems. ACIIDS 2022. Communications in Computer and Information Science, vol 1716. Springer, Singapore. https://doi.org/10.1007/978-981-19-8234-7_40
Download citation
DOI: https://doi.org/10.1007/978-981-19-8234-7_40
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-19-8233-0
Online ISBN: 978-981-19-8234-7
eBook Packages: Computer ScienceComputer Science (R0)