Abstract
Facial expressions of learners are relevant to their learning outcomes. The recognition of their emotional status influences the benefits of instruction or feedback provided by the intelligent tutor in education. However, learners’ emotions expressed during interactions with the intelligent tutor are mostly detected by self-reports of learners or judges who observe them in manually. The automated Facial Emotion Recognition (FER) task has been a challenging problem for intelligent tutors. The state-of-art automated FER methods target six basic emotions instead of learning-related emotions (e.g., neutral, confused, frustrated, and bored). Thus our research contributes to training a machine learning (ML) model to recognise learning-related emotions for intelligent tutors automatically, based on an Affective Dynamics (AD) model. We implement the AD model into our loss function (AD-loss) to fine-tune the ML model. In the test scenario, the AD-loss method improves the performance of state-of-art FER algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
\(\mathcal {X}\) is a sequence of frames of facial emotion expressions of learners.
- 2.
We note \(p_{\theta }(\hat{y}_i)\) as the abbreviation of \(p_{\theta }(\hat{y}_i\mid x_i)\) in the rest of paper.
- 3.
The PUZZLED consists of 10 videos of students when they are watching educational videos (contains neutral, confused, frustrated, and boredom).
- 4.
To assess the performance, we utilized CE to measure the loss of the output from both AM and AF.
References
Akhand, M., Roy, S., Siddique, N., Kamal, M.A.S., Shimamura, T.: Facial emotion recognition using transfer learning in the deep CNN. Electronics 10(9), 1036 (2021)
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882 (2020)
Craig, S., Graesser, A., Sullins, J., Gholson, B.: Affect and learning: an exploratory look into the role of affect in learning with AutoTutor. J. Educ. Media 29(3), 241–250 (2004)
Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699 (2019)
D’mello, S., Graesser, A.: AutoTutor and affective AutoTutor: learning by talking with cognitively and emotionally intelligent computers that talk back. ACM Trans. Interact. Intell. Syst. (TiiS) 2(4), 1–39 (2013)
D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)
D’Mello, S., Kappas, A., Gratch, J.: The affective computing approach to affect measurement. Emot. Rev. 10(2), 174–183 (2018)
Graesser, A., Chipman, P., King, B., McDaniel, B., D’Mello, S.: Emotions and learning with auto tutor. Front. Artif. Intell. Appl. 158, 569 (2007)
Graesser, A.C.: Emotions are the experiential glue of learning environments in the 21st century. Learn. Instr. 70, 101212 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
ImageNet: ImageNet (2021). https://www.image-net.org/. Accessed 28 Dec 2022
Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. 13, 1195–1215 (2020)
Linson, A., Xu, Y., English, A.R., Fisher, R.B.: Identifying student struggle by analyzing facial movement during asynchronous video lecture viewing: Towards an automated tool to support instructors. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds.) AIED 2022. LNCS, vol. 13355, pp. 53–65. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-11644-5_5
Wang, F., Liu, W., Liu, H., Cheng, J.: Additive margin softmax for face verification. arXiv preprint arXiv:1801.05599 (2018)
Winne, P.H.: A cognitive and metacognitive analysis of self-regulated learning. Handbook of self-regulation of learning and performance, pp. 15–32 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ruan, X., Palansuriya, C., Constantin, A. (2023). Affective Dynamic Based Technique for Facial Emotion Recognition (FER) to Support Intelligent Tutors in Education. In: Wang, N., Rebolledo-Mendez, G., Matsuda, N., Santos, O.C., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2023. Lecture Notes in Computer Science(), vol 13916. Springer, Cham. https://doi.org/10.1007/978-3-031-36272-9_70
Download citation
DOI: https://doi.org/10.1007/978-3-031-36272-9_70
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36271-2
Online ISBN: 978-3-031-36272-9
eBook Packages: Computer ScienceComputer Science (R0)