Abstract
In UX evaluation, pragmatic criteria still prevail over hedonic ones. However, emotion is an essential part of the user experience and has particular importance to system acceptance, thus it should be assigned more value in such assessments. Emotion recognition based on facial expressions is one of the tools that can be used to assess user emotion and it can be performed during the interaction and in a less intrusive way than with other sensors. In this context, this paper presents a systematic literature review about user experience evaluation using facial emotion recognition addressing the following research questions: which kinds of user studies take advantage of facial emotion recognition and to what purpose; how emotion recognition is implemented; how user experience is evaluated using this data and what strategies are used to validate these results. From 372 unique papers identified by the search string, 332 were initially discarded and, of the remaining 40 remaining papers that were read in full, only 14 were included in the final analysis. We identified that this area is still relatively novel, with few works published and all of them in the last eight years. Facial images were the most frequent type of data used and comparisons with self-reported emotions were the prevalent strategy to validate automatic emotion recognition, but just as often no such strategy was discussed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
ACM Digital Library dl.acm.org.
- 2.
IEEE Xplore ieeexplore.ieee.org.
- 3.
Springer Link https://link.springer.com/.
References
Carrillo, I., Meza-Kubo, V., Morán, A.L., Galindo, G., García-Canseco, E.: Processing EEG signals towards the construction of a user experience assessment method. In: Bravo, J., Hervás, R., Villarreal, V. (eds.) AmIHEALTH 2015. LNCS, vol. 9456, pp. 281–292. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26508-7_28
Chang, H.-M., Díaz, M., Català, A., Chen, W., Rauterberg, M.: Mood boards as a universal tool for investigating emotional experience. In: Marcus, A. (ed.) DUXU 2014. LNCS, vol. 8520, pp. 220–231. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07638-6_22
Fabbri, S., Silva, C., Hernandes, E., Octaviano, F., Di Thommazo, A., Belgamo, A.: Improvements in the start tool to better support the systematic review process. In: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, EASE 2016, New York, NY, USA. Association for Computing Machinery (2016). https://doi.org/10.1145/2915970.2916013
Filho, J.F., Prata, W., Oliveira, J.: Where-how-what am I feeling: user context logging in automated usability tests for mobile software. In: Marcus, A. (ed.) DUXU 2016. LNCS, vol. 9748, pp. 14–23. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40406-6_2
Filho, J.F., Prata, W., Valle, T.: Emotions logging in automated usability tests for mobile devices. In: Marcus, A. (ed.) DUXU 2015. LNCS, vol. 9186, pp. 428–435. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20886-2_40
Filho, J.F., Valle, T., Prata, W.: Automated usability tests for mobile devices through live emotions logging. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pp. 636–643 (2015)
Ferreira, T.M., Costella, F.L., Zanetti, A.B., da Silva, S.E., Zanatta, A.L., De Marchi, A.C.B.: CrowdRec: a prototype recommendation system for crowdsourcing platforms using Google venture design: Google venture design sprint. In: Proceedings of the XV Brazilian Symposium on Information Systems, pp. 1–8 (2019)
Filho, J.F., Prata, W., Oliveira, J.: Affective-ready, contextual and automated usability test for mobile software. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pp. 638–644 (2016)
Huynh, S., Kim, S., Ko, J., Balan, R.K., Lee, Y.: EngageMon: multi-modal engagement sensing for mobile games. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 2(1), 1–27 (2018)
Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering - a systematic literature review. Inf. Softw. Technol. 51(1), 7–15 (2009)
Landowska, A.: Towards emotion acquisition in it usability evaluation context. In: Proceedings of the Multimedia, Interaction, Design and Innovation, pp. 1–9. Association for Computing Machinery (2015)
LaViola Jr., J.J., Kruijff, E., McMahan, R.P., Bowman, D., Poupyrev, I.P.: 3D User Interfaces: Theory and Practice. Addison-Wesley Professional, Boston (2017)
Liapis, A., Karousos, N., Katsanos, C., Xenos, M.: Evaluating user’s emotional experience in HCI: the PhysiOBS approach. In: Kurosu, M. (ed.) HCI 2014. LNCS, vol. 8511, pp. 758–767. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07230-2_72
Lyytinen, K., Grover, V.: Management misinformation systems: a time to revisit? J. Assoc. Inf. Syst. 18(3), 2 (2017)
Mennig, P., Scherr, S.A., Elberzhager, F.: Supporting rapid product changes through emotional tracking. In: 2019 IEEE/ACM 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion), pp. 8–12. IEEE (2019)
Munim, K.M., Islam, I., Khatun, M., Karim, M.M., Islam, M.N.: Towards developing a tool for UX evaluation using facial expression. In: 2017 3rd International Conference on Electrical Information and Communication Technology (EICT), pp. 1–6. IEEE (2017)
Schroeder, W.J., Lorensen, B., Martin, K.: The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics. Kitware (2004)
Shaker, N., Shaker, M.: Towards understanding the nonverbal signatures of engagement in Super Mario Bros. In: Dimitrova, V., Kuflik, T., Chin, D., Ricci, F., Dolog, P., Houben, G.-J. (eds.) UMAP 2014. LNCS, vol. 8538, pp. 423–434. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08786-3_38
Sharma, K., Niforatos, E., Giannakos, M., Kostakos, V.: Assessing cognitive performance using physiological and facial features: generalizing across contexts. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 4(3), 1–41 (2020)
Soleimani, S., Law, E.L.C.: What can self-reports and acoustic data analyses on emotions tell us? In: Proceedings of the 2017 Conference on Designing Interactive Systems, pp. 489–501 (2017)
Szajna, B.: Empirical evaluation of the revised technology acceptance model. Manage. Sci. 42(1), 85–92 (1996)
de Souza Veriscimo, E., Bernardes Junior, J.L., Digiampietri, L.A.: Evaluating user experience in 3D interaction: a systematic review. In: XVI Brazilian Symposium on Information Systems, pp. 1–8 (2020)
Xiang, N., Yang, L., Zhang, M.: Dynamic difficulty adjustment by facial expression. In: Du, W. (ed.) Informatics and Management Science V. Lecture Notes in Electrical Engineering, vol. 208, pp. 761–768. London, Springer (2013). https://doi.org/10.1007/978-1-4471-4796-1_97
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
de Souza Veriscimo, E., Bernardes Júnior, J.L., Digiampietri, L.A. (2021). Facial Emotion Recognition in UX Evaluation: A Systematic Review. In: Kurosu, M. (eds) Human-Computer Interaction. Theory, Methods and Tools. HCII 2021. Lecture Notes in Computer Science(), vol 12762. Springer, Cham. https://doi.org/10.1007/978-3-030-78462-1_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-78462-1_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78461-4
Online ISBN: 978-3-030-78462-1
eBook Packages: Computer ScienceComputer Science (R0)