In allusion to the problem that teachers not being able to timely grasp student dynamics during online classroom, resulting in poor teaching quality, this paper proposes an online learning status analysis method that combines facial emotions with fatigue status. Specifically, we use an improved ResNet50 neural network for facial emotion recognition and quantify the detected emotions using the pleasure-arousal-dominance dimensional emotion scale. The improved network model achieved 87.51% and 75.28% accuracy on RAF-DB and FER2013 datasets, respectively, which can better detect the emotional changes of students. We use the Dlib’s face six key points detection model to extract the two-dimensional feature points of the face and judge the fatigue state. Finally, different weights are assigned to the facial emotion and fatigue state to evaluate the students’ learning status comprehensively. To verify the effectiveness of this method, experiments were conducted on the BNU-LSVED teaching quality evaluation dataset. We use this method to evaluate the learning status of multiple students and compare it with the manual evaluation results provided by expert teachers. The experiment results show that the students’ learning status evaluated using this method is basically matched with their actual status. Therefore, the classroom learning status detection method based on facial expression recognition proposed in this study can identify students’ learning status more accurately, thus realizing better teaching effect in online classroom.

1. [1] S. Zel, “Engaging Learners in Synchronous Online Training Using Facial Expression Analysis Technology,” University of Bridgeport ProQuest Dissertation and Theses, Article No.28257240, 2020.
2. [2] H. Hu et al., “Emotion Enhances Learning via Norepinephrine Regulation of AMPA-Receptor Trafficking,” Cell, Vol.131, No.1, pp. 160-173, 2007. https://doi.org/10.1016/j.cell.2007.09.017
3. [3] D. Long, “A Facial Expressions Recognition Method Using Residual Network Architecture for Online Learning Evaluation,” J. Adv. Comput. Intell. Intell. Inform., Vol.25, No.6, pp. 953-962, 2021. https://doi.org/10.20965/jaciii.2021.p0953
4. [4] C. Udeh, L. Chen, S. Du, M. Li, and M. Wu, “Multimodal Facial Emotion Recognition Using Improved Convolution Neural Networks Model,” J. Adv. Comput. Intell. Intell. Inform., Vol.27, No.4, pp. 710-719, 2023. https://doi.org/10.20965/jaciii.2023.p0710
5. [5] Q. J. Liao, T. Liu, and X. Y. Zhang, “Research on Face Expression Recognition Based on VGGNet,” Software Engineering, Vol.26, No.11, pp. 59-62, 2023. https://doi.org/10.19644/j.cnki.issn2096-1472.2023.011.013
6. [6] S. B. Zhang and W. T. Wang, “Research on Facial Expression Recognition Based on Improved VGG Model,” Modern Information Technology, Vol.5, No.23, pp. 100-103, 2021. https://doi.org/10.19850/j.cnki.2096-4706.2021.23.026
7. [7] H. L. Zhang and X. Y. Bai, “Facial Expression Recognition Method Using Optimized Pruning GoogLeNet,” Computer Engineering and Applications, Vol.57, No.19, pp. 179-188, 2021.
8. [8] X. Wang, G. Wang, and Y. Cui, “Facial expression recognition based on improved ResNet,” The J. of China Universities of Posts and Telecommunications, Vol.30, No.01, pp. 28-38, 2023. https://doi.org/10.19682/j.cnki.1005-8885.2023.2003
9. [9] J. Yang et al., “A Facial Expression Recognition Method Based on Dlib, RI-LBP and ResNet,” J. of Physics, Conf. Series, Vol.1634, No.1, Article No.012080, 2020. https://doi.org/10.1088/1742-6596/1634/1/012080
10. [10] L. N. Qiu et al., “Attention Detection Based on Symmetrical Dual-Channel EEG Signals,” Computer Systems and Applications, Vol.32, No.05, pp. 1-10, 2023. https://doi.org/10.15888/j.cnki.csa.009086
11. [11] S. Gupta and P. Kumar, “Attention Recognition System in Online Learning Platform Using EEG Signals,” Emerging Technologies for Smart Cities, pp. 139-152, 2021. https://doi.org/10.1007/978-981-16-1550-4_15
12. [12] G. Tonguç and B. O. Ozkara, “Automatic recognition of student emotions from facial expressions during a lecture,” Computers and Education, Vol.148, No.2, Article No.103797, 2020. https://doi.org/10.1016/j.compedu.2019.103797
13. [13] Y. C. Jiang et al., “Research on Student Learning State Monitoring Technology for Library Based on Artificial Intelligence,” Chinese J. of Electron Devices, Vol.46, No.4, pp. 1070-1074, 2023.
14. [14] X. Y. Zhou, W. H. Ying, and S. R. Gong, “Design and implementation of classroom concentration real-time feedback system based on facial expression recognition,” Modern Electronics Technique, Vol.46, No.24, pp. 141-146, 2023. https://doi.org/10.16652/j.issn.1004-373x.2023.24.025
15. [15] P. Chakradhar and K. Praveen, “An intelligent system for monitoring students’ engagement in large classroom teaching through facial expression recognition,” Expert Systems: The Int. J. of Knowledge Engineering, Vol.39, No.1, Article No.e12839, 2022. https://doi.org/10.1111/exsy.12839
16. [16] K. Prabhu, S. Sathishkumar, M. Sivachitra, S. Dineshkumar, and P. Sathiyabama, “Facial Expression Recognition Using Enhanced Convolution Neural Network with Attention Mechanism,” Computer Systems Science and Engineering, Vol.41, No.1, pp. 415-426, 2022. https://doi.org/10.32604/CSSE.2022.019749
17. [17] G. Gao, “Survey on Attention Mechanisms in Deep Learning Recommendation Models,” Computer Engineering and Applications, Vol.59, No.9, pp. 9-18, 2022.
18. [18] L. Zhu, “Research Progress of Attention Mechanism in Deep Learning,” J. of Chinese Information Processing, Vol.33, No.6, pp. 1-11, 2019.
19. [19] A. Mehrabian, “Pleasure-Arousal-Dominance: A General Framework for Describing and Measuring Individual Differences in Temperament,” Current Psychology, Vol.14, No.4, pp. 261-292, 1996. https://doi.org/10.1007/BF02686918
20. [20] Y. Zhao et al., “Evaluating Users’ Emotional Experience in Mobile Libraries: An Emotional Model Based on the Pleasure-Arousal-Dominance Emotion Model and the Five Factor Model,” Frontiers in Psychology, Vol.13, Article No.942198, 2022. https://doi.org/10.3389/fpsyg.2022.942198
21. [21] M. Li et al., “An Analysis Model of Learners’ Online Learning Status Based on Deep Neural Network and Multi-Dimensional Information Fusion,” Computer Modeling in Engineering Sciences, Vol.135, No.3, pp. 2349-2371, 2022. https://doi.org/10.32604/cmes.2023.022604
22. [22] Y. Gao, “Learner Expression Recognition and Emotional Analysis in Natural Scenes,” North China Electric Power University, 2019. https://doi.org/10.27140/d.cnki.ghbbu.2019.000323
23. [23] M. Liu and J. Zhao, “SVM based facial image feature detection algorithm for fatigue driving,” Semiconductor Optoelectronics, Vol.41, No.2, pp. 278-282, 2020. https://doi.org/10.16818/j.issn1001-5868.2020.02.026
24. [24] H. Zhang, W. Su, J. Yu, and Z. Wang, “Identity–Expression Dual Branch Network for Facial Expression Recognition,” IEEE Trans. on Cognitive and Developmental Systems, Vol.13, No.4, pp. 898-911, 2021. https://doi.org/10.1109/TCDS.2020.3034807
25. [25] Z. Wang, F. Zeng, S. Liu, and B. Zeng, “OAENet: Oriented Attention Ensemble for Accurate Facial Expression Recognition,” Pattern Recognition, Vol.112, Article No.107694, 2021. https://doi.org/10.1016/j.patcog.2020.107694
26. [26] S. Jyoti, G. Sharma, and A. Dhall, “Expression Empowered ResiDen Network for Facial Action Unit Detection,” 2019 14th IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG2019), 2019. https://doi.org/10.1109/fg.2019.8756580
27. [27] Y. Li et al., “Occlusion Aware Facial Expression Recognition Using CNN with Attention Mechanism,” IEEE Trans Image Processing, Vol.28, No.5, pp. 2439-2450, 2019. https://doi.org/10.1109/TIP.2018.2886767
28. [28] W. Xie, L. Shen, and J. Duan, “Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition,” IEEE Trans. on Cybernetics, Vol.51, No.5, pp. 2787-2800, 2021. https://doi.org/10.1109/TCYB.2019.2925095
29. [29] J. Chen and Y. Xu, “Expression Recognition Based on Convolution Residual Network of Attention Pyramid,” Computer Engineering and Applications, Vol.58, No.22, pp. 123-131, 2022.
30. [30] H. Zhong et al., “Research on real-time teachers’ facial expression recognition based on YOLOv5 and attention mechanisms,” EURASIP J. on Advances in Signal Processing, Vol.2023, Article No.55, 2023. https://doi.org/10.1186/S13634-023-01019-W
31. [31] K. Lee, S. Kim, and E. Lee, “Fast and accurate facial expression image classification and regression method based on knowledge distillation,” Appl. Sci., Vol.13, No.11, Article No.6409, 2023. https://doi.org/10.3390/app13116409
32. [32] Q. Wei et al., “BNU-LSVED 2.0: Spontaneous multimodal student affect database with multi-dimensional labels,” Signal Processing: Image Communication, Vol.59, pp. 168-181, 2017. https://doi.org/10.1016/j.image.2017.08.012
33. [33] L. Wang and M. Lai, “Analysis of Students’ Concentration in Online Classroom Based on Facial Expression Recognition,” Computer Systems and Applications, Vol.32, No.02, pp. 55-62, 2023. https://doi.org/10.15888/j.cnki.csa.008970
34. [34] J. Shen, H. Yang, J. Li, and Z. Cheng, “Assessing learning engagement based on facial expression recognition in MOOC’s scenario,” Multimedia Systems, Vol.28, No.2, pp. 469-478, 2022. https://doi.org/10.1007/s00530-021-00854-x