article

Free access

A hierarchical probabilistic framework for recognizing learners' interaction experience trends and emotions

Authors:

Maher Chaouachi,

Claude FrassonAuthors Info & Claims

Advances in Human-Computer Interaction, Volume 2014

Article No.: 6, Page 6

https://doi.org/10.1155/2014/632630

Published: 01 January 2014 Publication History

PDF eReader Publisher Site

Abstract

We seek to model the users' experience within an interactive learning environment. More precisely, we are interested in assessing the relationship between learners' emotional reactions and three trends in the interaction experience, namely, flow: the optimal interaction (a perfect immersion within the task), stuck: the nonoptimal interaction (a difficulty to maintain focused attention), and off-task: the noninteraction (a dropout from the task). We propose a hierarchical probabilistic framework using a dynamic Bayesian network to model this relationship and to simultaneously recognize the probability of experiencing each trend as well as the emotional responses occurring subsequently. The framework combines three modality diagnostic variables that sense the learner's experience including physiology, behavior, and performance, predictive variables that represent the current context and the learner's profile, and a dynamic structure that tracks the evolution of the learner's experience. An experimental study, with a specifically designed protocol for eliciting the targeted experiences, was conducted to validate our approach. Results revealed that multiple concurrent emotions can be associated with the experiences of flow, stuck, and off-task and that the same trend can be expressed differently from one individual to another. The evaluation of the framework showed promising results in predicting learners' experience trends and emotional responses.

References

[1]

L. Alben, "Quality of experience: defining the criteria for effective interaction design," Interactions, vol. 3, no. 3, pp. 11-15, 1996.

Digital Library

[2]

C. Berka, D. J. Levendowski, M. N. Lumicao et al., "EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks," Aviation Space and Environmental Medicine, vol. 78, no. 5, pp. B231-B244, 2007.

[3]

S. Brave and C. Nass, "Emotion in human-computer interaction," in Handbook of Human-Computer Interaction, J. Jacko and A. Sears, Eds., pp. 81-96, Elsevier Science, 2002.

Digital Library

[4]

R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis et al., "Emotion recognition in human-computer interaction," IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32-80, 2001.

[5]

E. Hudlicka, "To feel or not to feel: the role of affect in human-computer interaction," International Journal of Human Computer Studies, vol. 59, no. 1-2, pp. 1-32, 2003.

Digital Library

[6]

S. H. Matthias, M. Uwe, and K. Thomas, Adaptive User Interfaces: Principles and Practice, Elsevier Science, New York, NY, USA, 1993.

Digital Library

[7]

R. Picard, Affective Computing, MIT Press, 1997.

Digital Library

[8]

A. Bechara, H. Damasio, and A. R. Damasio, "Emotion, decision making and the orbitofrontal cortex," Cerebral Cortex, vol. 10, no. 3, pp. 295-307, 2000.

[9]

A. Damasio, Descartes' Error: Emotion, Reason and the Human Brain, Grosset and Putnam, New York, NY, USA, 1994.

[10]

D. Goleman, Emotional Intelligence, Bantam Books, New York, NY, USA, 1995.

[11]

A. M. Isen, "Positive affect and decision making," in Handbook of Emotions, M. Lewis and J. M. Haviland-Jones, Eds., pp. 417-435, Guilford Press, New York, NY, USA, 2000.

[12]

I. Arroyo, D. G. Cooper, W. Burleson, B. P. Woolf, K. Muldner, and R. Christopherson, "Emotion sensors go to school," Frontiers in Artificial Intelligence and Applications, vol. 200, no. 1, pp. 17-24, 2009.

Digital Library

[13]

C. Conati and H. Maclaren, "Empirically building and evaluating a probabilistic model of user affect," User Modelling and User-Adapted Interaction, vol. 19, no. 3, pp. 267-303, 2009.

Digital Library

[14]

S. K. D'Mello, S. D. Craig, A. Witherspoon, B. McDaniel, and A. Graesser, "Automatic detection of learner's affect from conversational cues," User Modelling and User-Adapted Interaction, vol. 18, no. 1-2, pp. 45-80, 2008.

Digital Library

[15]

K. Forbes-Riley and D. Litman, "Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system," Computer Speech and Language, vol. 25, no. 1, pp. 105-126, 2011.

Digital Library

[16]

A. Kapoor, W. Burleson, and R. W. Picard, "Automatic prediction of frustration," International Journal of Human Computer Studies, vol. 65, no. 8, pp. 724-736, 2007.

Digital Library

[17]

S. W. McQuiggan, S. Lee, and J. C. Lester, "Early prediction of student frustration," in Proceedings of the 2nd international conference on Affective Computing and Intelligent Interaction, Springer, Lisbon, Portugal, 2007.

Digital Library

[18]

B. Woolf, W. Burleson, I. Arroyo, T. Dragon, D. Cooper, and R. Picard, "Affect aware tutors: recognising and responding to student affect," International Journal of Learning Technology, vol. 4, no. 3-4, pp. 129-164, 2009.

Digital Library

[19]

J. Allanson and S. H. Fairclough, "A research agenda for physiological computing," Interacting with Computers, vol. 16, no. 5, pp. 857-878, 2004.

[20]

J. T. Cacioppo, G. C. Berntson, K. M. Poehlmann, and T. A. Ito, "The psychophysiology of emotions," in Handbook of Emotions, M. Lewis and J. M. Haviland-Jones, Eds., The Guilford Press, 2nd edition, 2000.

[21]

S. H. Fairclough, "Fundamentals of physiological computing," Interacting with Computers, vol. 21, no. 1-2, pp. 133-145, 2009.

Digital Library

[22]

R. S. J. D. Baker, S. K. D'Mello, M. M. T. Rodrigo, and A. C. Graesser, "Better to be frustrated than bored: the incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments," International Journal of Human Computer Studies, vol. 68, no. 4, pp. 223-241, 2010.

Digital Library

[23]

S. Craig, A. Graesser, J. Sullins, and B. Gholson, "Affect and learning: an exploratory look into the role of affect in learning with autotutor," Journal of Educational Media, vol. 29, no. 3, pp. 241-250, 2004.

[24]

R. W. Picard, S. Papert, W. Bender et al., "Affective learning'a manifesto," BT Technology Journal, vol. 22, no. 4, pp. 253-269, 2004.

Digital Library

[25]

A. C. Graesser, S. K. D'mello, P. Chipman et al., "Exploring relationships between affect and learning with autotutor," in Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 16-23, 2007.

[26]

N. Hara, "Student distress in a web-based distance education course," Information, Communication & Society, vol. 3, no. 4, pp. 557-579, 2000.

[27]

K. O'Regan, "Emotion and E-learning," Journal of Asynchronous Learning Network, vol. 7, no. 3, pp. 78-92, 2003.

[28]

L. A. Muse, S. G. Harris, and H. S. Feud, "Has the inverted-u theory of stress and job performance had a fair test?," Human Performance, vol. 16, no. 4, pp. 349-364, 2003.

[29]

S. E. Sullivan and R. S. Bhagat, "Organizational stress, job satisfaction and job performance: where do we go from here?," Journal of Management, vol. 18, no. 2, pp. 353-374, 1992.

[30]

A. C. Graesser and B. A. Olde, "How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down," Journal of Educational Psychology, vol. 95, no. 3, pp. 524-536, 2003.

[31]

K. VanLehn, S. Siler, C. Murray, T. Yamauchi, and W. B. Baggett, "Why do only some events cause learning during human tutoring?," Cognition and Instruction, vol. 21, no. 3, pp. 209-249, 2003.

[32]

K. Hone, "Empathic agents to reduce user frustration: the effects of varying agent characteristics," Interacting with Computers, vol. 18, no. 2, pp. 227-245, 2006.

Digital Library

[33]

J. Klein, Y. Moon, and R. W. Picard, "This computer responds to user frustration: theory, design, and results," Interacting with Computers, vol. 14, no. 2, pp. 119-140, 2002.

[34]

J. Healey and R. Picard, "Smartcar: detecting driver stress," in Proceedings of the 15th International Conference on Pattern Recognition, pp. 218-221, 2000.

Digital Library

[35]

R. W. Picard and K. K. Liu, "Relative subjective count and assessment of interruptive technologies applied to mobile monitoring of stress," International Journal of Human Computer Studies, vol. 65, no. 4, pp. 361-375, 2007.

Digital Library

[36]

J. Zhai and A. Barreto, "Stress detection in computer users based on digital signal processing of noninvasive physiological variables," in Proceedings of the 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS '06), pp. 1355-1358, September 2006.

[37]

I. Jraidi and C. Frasson, "Student's uncertainty modeling through a multimodal sensor-based approach," Educational Technology & Society, vol. 16, no. 1, pp. 219-230, 2013.

[38]

H. Pon-Barry, K. Schultz, E. O. Bratt, B. Clark, and S. Peters, "Responding to student uncertainty in spoken tutorial dialogue systems," International Journal of Artificial Intelligence in Education, vol. 16, no. 2, pp. 171-194, 2006.

Digital Library

[39]

J. A. Caldwell, K. K. Hall, and B. S. Erickson, "EEG data collected from helicopter pilots in flight are sufficiently sensitive to detect increased fatigue from sleep deprivation," The International Journal of Aviation Psychology, vol. 12, no. 1, pp. 19-32, 2002.

[40]

A. Heitmann, R. Guttkuhn, U. Trutschel, and M. Moore-Ede, "Technologies for the monitoring and prevention of driver fatigue," in Proceedings of the 1st International Driving Symposium on Human Factors in Driving Assessment, Training, and Vehicle Design, pp. 81-86, 2001.

[41]

Q. Ji, P. Lan, and C. Looney, "A probabilistic framework for modeling and real-time monitoring human fatigue," IEEE Transactions on Systems, Man and Cybernetics A: Systems and Humans, vol. 36, no. 5, pp. 862-875, 2006.

Digital Library

[42]

K. H. Kim, S. W. Bang, and S. R. Kim, "Emotion recognition system using short-term monitoring of physiological signals," Medical and Biological Engineering and Computing, vol. 42, no. 3, pp. 419-427, 2004.

[43]

K. R. Scherer, T. Wranik, J. Sangsue, V. Tran, and U. Scherer, "Emotions in everyday life: probability of occurrence, risk factors, appraisal and reaction patterns," Social Science Information, vol. 43, no. 4, pp. 499-570, 2004.

[44]

M. Pantic and L. J. M. Rothkrantz, "Facial action recognition for facial expression analysis from static face images," IEEE Transactions on Systems, Man, and Cybernetics B: Cybernetics, vol. 34, no. 3, pp. 1449-1461, 2004.

Digital Library

[45]

A. B. Ashraf, S. Lucey, J. F. Cohn et al., "The painful face'pain expression recognition using active appearance models," Image and Vision Computing, vol. 27, no. 12, pp. 1788-1796, 2009.

Digital Library

[46]

L. Ma and K. Khorasani, "Facial expression recognition using constructive feedforward neural networks," IEEE Transactions on Systems, Man, and Cybernetics B: Cybernetics, vol. 34, no. 3, pp. 1588-1595, 2004.

Digital Library

[47]

V. A. Petrushin, "Emotion in speech: recognition and application to call centers," in Proceedings of the Artificial Neural Networks in Engineering Conference (ANNIE '99), pp. 1085-1092, November 1999.

[48]

J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke, "Prosody-based automatic detection of annoyance and frustration in human-computer dialog," in Proceedings of the 7th International Conference on Spoken Language Processing, pp. 2037-2040, Denver, Colo, USA, 2002.

[49]

D. J. Litman and K. Forbes-Riley, "Predicting student emotions in computer-human tutoring dialogues," in Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, Association for Computational Linguistics, Barcelona, Spain, 2004.

Digital Library

[50]

R. E. Kaliouby and P. Robinson, "Real-time inference of complex mental states from facial expressions and head gestures," in Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW '04), p. 154, 2004.

Digital Library

[51]

W. Liao, W. Zhang, Z. Zhu, Q. Ji, and W. D. Gray, "Toward a decision-theoretic framework for affect recognition and user assistance," International Journal of Human Computer Studies, vol. 64, no. 9, pp. 847-873, 2006.

Digital Library

[52]

M. Csikszentmihalyi, The Psychology of Optimal Experience, Harper & Row, New York, NY, USA, 1990.

[53]

W. Burleson and R. W. Picard, "Affective agents: sustaining motivation to learn through failure and a state of stuck," in Proceedings of the Social and Emotional Intelligence in Learning Environments, Workshop in conjunction with the 7th International Conference on Intelligent Tutoring Systems, Maceio, Alagoas, Brasil, 2004.

[54]

R. S. J. D. Baker, A. T. Corbett, I. Roll, and K. R. Koedinger, "Developing a generalizable detector of when students game the system," User Modelling and User-Adapted Interaction, vol. 18, no. 3, pp. 287-314, 2008.

Digital Library

[55]

J. E. Beck, "Engagement tracing: using response times to model student disengagement," in Artificial Intelligence in Education: Supporting Learning Through Intelligent and Socially Informed Technology, IOS Press, 2005.

Digital Library

[56]

M. M. T. Rodrigo, R. S. J. D. Baker, M. C. V. Lagud et al., "Affect and usage choices in simulation problem-solving environments," in Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, pp. 145-152, IOS Press, 2007.

Digital Library

[57]

B. Cowley, D. Charles, M. Black, and R. Hickey, "Toward an understanding of flow in video games," Computers in Entertainment, vol. 6, no. 2, article 20, 2008.

Digital Library

[58]

K. Jegers, "Pervasive game flow: understanding player enjoyment in pervasive gaming," Computers in Entertainment, vol. 5, no. 1, Article ID 1236238, 2007.

Digital Library

[59]

P. Sweetser and P. Wyeth, "Gameflow: a model for evaluating player enjoyment in games," Computers in Entertainment, vol. 3, no. 3, p. 3, 2005.

Digital Library

[60]

K. P. Murphy, Dynamic bayesian networks: representation, inference and learning [Ph.D. thesis], Computer Science Division, University of California, Berkeley, Calif, USA, 2002.

Digital Library

[61]

P. J. Lang, "The emotion probe: studies of motivation and attention," American Psychologist, vol. 50, no. 5, pp. 372-385, 1995.

[62]

P. J. Lang, M. K. Greenwald, M. M. Bradley, and A. O. Hamm, "Looking at pictures: affective, facial, visceral, and behavioral reactions," Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993.

[63]

J. Altimiras, "Understanding autonomic sympathovagal balance from short-term heart rate variations. Are we analyzing noise?," Comparative Biochemistry and Physiology: A Molecular and Integrative Physiology, vol. 124, no. 4, pp. 447-460, 1999.

[64]

R. W. Levenson, "Autonomic nervous system differences among emotions," Psychological Science, vol. 3, no. 1, pp. 23-27, 1992.

[65]

C. Berka, D. J. Levendowski, M. M. Cvetinovic et al., "Real-time analysis of EEG indexes of alertness, cognition, and memory acquired with a wireless EEG headset," International Journal of Human-Computer Interaction, vol. 17, no. 2, pp. 151-170, 2004.

[66]

M. B. Sterman and C. A. Mann, "Concepts and applications of eeg analysis in aviation performance evaluation," Biological Psychology, vol. 40, no. 1-2, pp. 115-130, 1995.

[67]

R. Stevens, T. Galloway, and C. Berka, "EEG-related changes in cognitive workload, engagement and distraction as students acquire problem solving skills," in User Modeling 2007, C. Conati, K. McCoy, and G. Paliouras, Eds., vol. 4511, pp. 187-196, Springer, Berlin, Germany, 2007.

Digital Library

[68]

A. T. Pope, E. H. Bogart, and D. S. Bartolome, "Biocybernetic system evaluates indices of operator engagement in automated task," Biological Psychology, vol. 40, no. 1-2, pp. 187-195, 1995.

[69]

F. G. Freeman, P. J. Mikulka, L. J. Prinzel, and M. W. Scerbo, "Evaluation of an adaptive automation system using three EEG indices with a visual tracking task," Biological Psychology, vol. 50, no. 1, pp. 61-76, 1999.

[70]

M. Chaouachi, P. Chalfoun, I. Jraidi, and C. Frasson, "Affect and mental engagement: towards adaptability for intelligent systems," in Proceedings of the 23rd International Florida Artificial Intelligence Research Society Conference (FLAIRS-23 '10), pp. 355-360, Daytona Beach, Fla, USA, May 2010.

[71]

R. Pekrun, A. J. Elliot, and M. A. Maier, "Achievement goals and discrete achievement emotions: a theoretical model and prospective test," Journal of Educational Psychology, vol. 98, no. 3, pp. 583-597, 2006.

[72]

M. Chaouachi, I. Jraidi, and C. Frasson, "Modeling mental workload using eeg features for intelligent systems," in User Modeling, Adaption and Personalization, J. Konstan, R. Conejo, J. L. Marzo, and N. Oliver, Eds., vol. 6787, pp. 50-61, Springer, Berlin, Germany, 2011.

Digital Library

[73]

"Thought Technology," 2007, http://www.thoughttechnology.com/.

[74]

H. H. Jasper, "The ten-twenty electrode system of the international federation," Electroencephalography and clinical neurophysiology, vol. 10, pp. 371-375, 1958.

[75]

O. P. John, L. P. Naumann, and C. J. Soto, "Paradigm shift to the integrative big-five trait taxonomy: history, measurement, and conceptual issues," in Handbook of Personality: Theory and Research, O. P. John, R. W. Robins, and L. A. Pervin, Eds., pp. 114-158, 2008.

[76]

S. L. Lauritzen, "The EM algorithm for graphical association models with missing data," Computational Statistics & Data Analysis, vol. 19, no. 2, pp. 191-201, 1995.

Digital Library

[77]

P. Domingos and M. Pazzani, "On the optimality of the simple bayesian classifier under zero-one loss," Machine Learning, vol. 29, no. 2-3, pp. 103-130, 1997.

Digital Library

[78]

J. R. Quinlan, "Induction of decision trees," Machine Learning, vol. 1, no. 1, pp. 81-106, 1986.

[79]

J. C. Platt, "Fast training of support vector machines using sequential minimal optimization," in Advances in Kernel Methods, pp. 185-208, MIT Press, 1999.

Digital Library

[80]

I. Jraidi, P. Chalfoun, and C. Frasson, "Implicit strategies for intelligent tutoring systems," in Intelligent Tutoring Systems, S. Cerri, W. Clancey, G. Papadourakis, and K. Panourgia, Eds., vol. 7315, pp. 1-10, Springer, Berlin, Germany, 2012.

Digital Library

Cited By

Abdessalem HChaouachi MBoukadida MFrasson C(2019)Toward Real-Time System Adaptation Using Excitement Detection from Eye TrackingIntelligent Tutoring Systems10.1007/978-3-030-22244-4_26(214-223)Online publication date: 3-Jun-2019
https://dl.acm.org/doi/10.1007/978-3-030-22244-4_26
Kadar MRestrepo EFerreira FCalado JArtifice ASarraipa JJardim-Goncalves R(2016)Affective Computing to Enhance Emotional Sustainability of Students in Dropout PreventionProceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion10.1145/3019943.3019956(85-91)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1145/3019943.3019956

A hierarchical probabilistic framework for recognizing learners' interaction experience trends and emotions
1. Human-centered computing

Recommendations

Recognizing Induced Emotions of Movie Audiences from Multimodal Information
Recognizing emotional reactions of movie audiences to affective movie content is a challenging task in affective computing. Previous research on induced emotion recognition has mainly focused on using audio-visual movie content. Nevertheless, the ...
Negative Emotions, Positive Experience: What Are We Doing Wrong When Evaluating the UX?
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Since its idealization more than 20 years ago, much research has been carried out on the User eXperience (UX) field, with several evaluation methods being proposed. However, studies have been pointing out conflicting results when evaluating the UX. ...
Mobile Game-Based Learning in Distance Education: A Mixed Analysis of Learners’ Emotions and Gaming Features
Learning and Collaboration Technologies: Games and Virtual Environments for Learning
Abstract
Although there are multiple approaches for making lectures more interactive, Game-Based Learning (GBL) tends to achieve the highest impact on students’ emotional engagement. To this end, this study seeks to implement a mobile GBL approach in a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Advances in Human-Computer Interaction

Advances in Human-Computer Interaction Volume 2014, Issue

January 2014

212 pages

ISSN:1687-5893

EISSN:1687-5907

Issue’s Table of Contents

Publisher

Hindawi Limited

London, United Kingdom

Publication History

Accepted: 03 March 2014

Published: 01 January 2014

Received: 24 June 2013

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
73
Total Downloads

Downloads (Last 12 months)27
Downloads (Last 6 weeks)6

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Abdessalem HChaouachi MBoukadida MFrasson C(2019)Toward Real-Time System Adaptation Using Excitement Detection from Eye TrackingIntelligent Tutoring Systems10.1007/978-3-030-22244-4_26(214-223)Online publication date: 3-Jun-2019
https://dl.acm.org/doi/10.1007/978-3-030-22244-4_26
Kadar MRestrepo EFerreira FCalado JArtifice ASarraipa JJardim-Goncalves R(2016)Affective Computing to Enhance Emotional Sustainability of Students in Dropout PreventionProceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion10.1145/3019943.3019956(85-91)Online publication date: 1-Dec-2016
https://dl.acm.org/doi/10.1145/3019943.3019956

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents