article

Toward computers that recognize and respond to user emotion

Author:

R. W. PicardAuthors Info & Claims

IBM Systems Journal, Volume 39, Issue 3-4

Pages 705 - 719

https://doi.org/10.1147/sj.393.0705

Published: 01 July 2000 Publication History

Abstract

For a long time emotions have been kept out of the deliberate tools of science; scientists have expressed emotion, but no tools could sense and respond to their affective information. This paper highlights research at the MIT Media Laboratory aimed at giving computers the ability to comfortably sense, recognize, and respond to the human communication of emotion, especially affective states such as frustration, confusion, interest, distress, anger, and joy, Two main themes of sensing--self-report and concurrent expression--are described, together with examples of systems that give users new ways to communicate emotion to computers and, through computers, to other people. In addition to building systems that try to elicit and detect frustration, our research group has built a system that responds to user frustration in a way that appears to help alleviate it. This paper highlights applications of this research to interface design, wearable computing, entertainment, and education, and briefly presents some potential ethical concerns and how they might be addressed.

References

[1]

1. B. Reeves and C. Nass, The Media Equation, Center for the Study of Language and Information, Stanford University (1996).

[2]

2. D. Goleman, Emotional Intelligence, Bantam Books, New York (1995).

[3]

3. P. Salovey and J. D. Mayer, "Emotional Intelligence," Imagination, Cognition, and Personality9, No. 3, 185-211 (1990).

[4]

4. R. W. Picard, Affective Computing, MIT Press, Cambridge, MA (1997).

Digital Library

[5]

5. "Workshop on Grounding Emotions in Adaptive Systems," D. Canamero, C. Numaoka, and P. Petta, Editors, 5th International Conference of the Society for Adaptive Behavior (SAB '98), Zurich (August 1998); http://www.ai.univie.ac.at/ ~paolo/conf/sab98/.

[6]

6. "Emotional and Intelligent: The Tangled Knot of Cognition" D. Canamero, Editor, AAAI Fall Symposium, Orlando, AAAI Press, Menlo Park, CA (October 1998).

[7]

7. Workshop on Emotion-Based Agent Architectures (EBAA '99), J. D. Velasquez, Editor, in conjunction with the Autonomous Agents Conference, Seattle (May 1999).

[8]

8. Emotions in Humans and Artifacts, R. Trappl and P. Petta, Editors, collected papers from Vienna workshop (August 1999); to be published by MIT Press, Cambridge, MA.

[9]

9. "Human-Computer Interaction: Ergonomics and User Interfaces," Volume 1, H.-J. Bullinger and J. Ziegler, Editors, special session on Affective Computing and HCI, Proceedings of HCI International '99, Munich, Lawrence Erlbaum Associates, Mahwah, NJ (August 1999).

[10]

10. Affect in Interactions: Towards a New Generation of Interfaces, A. Paiva, Editor, Springer Books, to appear; http://gaiva. inesc.pt/i3ws/i3workshop.html (1999).

[11]

11. J. D. Velasquez, "Modeling Emotions and Other Motivations in Synthetic Agents." Proceedings of AAAI 97 (1997), pp. 10-15.

Digital Library

[12]

12. J.D. Velasquez, "A Computational Framework for Emotion-Based Control," Workshop on Grounding Emotions in Adaptive Systems, Fifth International Conference on Simulation of Adaptive Behavior, Zurich (August 1998).

[13]

13. H. Ishii and B. Ullmer, "Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms," Proceedings of Conference on Human Factors in Computing Systems (CHI '97), Atlanta, ACM Press (March 1997), pp. 234-241.

Digital Library

[14]

14. M. Norwood, Affective Feedback Devices for Continuous Usability Assessment, S.M. thesis, MIT, Media Arts and Sciences, Cambridge, MA (2000).

[15]

15. J. Scheirer, R. Fernandez, and R. W. Picard, "Expression Glasses: A Wearable Device for Facial Expression Recognition," CHI '99 Short Papers, Pittsburgh, PA (1999).

Digital Library

[16]

16. T. B. Sheridan, "Community Dialog Technology," Proceedings of the IEEE63, No. 3, 463-475 (March 1975).

[17]

17. T. Starner, S. Mann, B. Rhodes, J. Levine, J. Healey, D. Kirsch, R. Picard, and A. Pentland, "Augmented Reality Through Wearable Computing," Presence6, No. 4, 386-398 (1997).

[18]

18. R.W. Picard and J. Healey, "Affective Wearables," Personal Technologies1, No. 4, 231-240 (1997).

[19]

19. E. Vyzas and R. W. Picard, "Online and Offline Recognition of Emotion Expression from Physiological Data," Workshop on Emotion-Based Agent Architectures at the International Conference on Autonomous Agents, Seattle, WA (1999).

[20]

20. J. Healey, J. Seger, and R. W. Picard, "Quantifying Driver Stress: Developing a System for Collecting and Processing Bio-Metric Signals in Natural Situations," Proceedings of the Rocky-Mt. Bio-Engineering Symposium, Boulder, CO (1999).

[21]

21. D. Kirsch, "The Sentic Mouse: Developing a Tool for Measuring Emotional Valence," S.M. thesis, MIT, Media Arts and Sciences (May 1997).

[22]

22. W. Ark, D. C. Dryer, and D. J. Lu, "The Emotion Mouse," Proceedings of HCI International '99, Munich, Germany (August 1999).

Digital Library

[23]

23. J. Healey and R. W. Picard, "Digital Processing of Affective Signals," Proceedings of the International Conference on Acous tics, Speech, and Signal Processing, Seattle, WA (May 1998).

[24]

24. J. Riseberg, J. Klein, R. Fernandez, and R. W. Picard, "Frustrating the User on Purpose: Using Biosignals in a Pilot Study to Detect the User's Emotional State," CHI '98 Short Papers, Los Angeles, CA (1998); a longer version of this paper has been accepted for publication in Interaction with Computers.

Digital Library

[25]

25. R. Fernandez and R. W. Picard, "Signal Processing for Recognition of Human Frustration," Proceedings of IEEE ICASSP '98, Seattle, WA (1997).

[26]

26. J. Klein, Y. Moon, and R. W. Picard, "This Computer Responds to User Frustration," CHI 99 Short Papers, Pittsburgh, PA (1999); a longer version has been accepted for publication in Interaction with Computers.

Digital Library

[27]

27. J. Klein, "Computer Response to User Frustration," S.M. thesis, MIT, Media Arts and Sciences, Cambridge, MA (1998).

[28]

28. J. Healey and R. W. Picard, "StartleCam: A Cybernetic Wearable Camera," Proceedings of the International Symposium on Wearable Computing, Pittsburgh, PA (1998).

Digital Library

[29]

29. J. Healey, F. Dabek, and R. W. Picard, "A New Affect-Perceiving Interface and Its Application to Personalized Music Selection," Proceedings of 1998 Workshop on Perceptual User Interfaces, San Francisco, CA (1998).

[30]

30. T. Marrin and R. W. Picard, "Analysis of Affective Musical Expression with the Conductor's Jacket," Proceedings of XII Colloquium on Musical Informatics, Gorizia, Italy (1998).

[31]

31. T. Marrin, Inside the Conductor's Jacket: Analysis. Interpretation, and Musical Synthesis of Expressive Gesture, Ph.D. thesis, MIT, Media Arts and Sciences, Cambridge, MA (1999).

Digital Library

[32]

32. K. Blocher, "Affective Social Quotient (ASQ): Teaching Emotion Recognition with Interactive Media and Wireless Expressive Toys," S.M. thesis, MIT, Cambridge, MA (May 1999).

Cited By

Dadebayev DGoh WTan E(2022)EEG-based emotion recognitionJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2021.03.00934:7(4385-4401)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2021.03.009
Genest PGoix LKhalafaoui YEgyed-Zsigmond EGrozavu N(2022)French translation of a dialogue dataset and text-based emotion detectionData & Knowledge Engineering10.1016/j.datak.2022.102099142:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.datak.2022.102099
Ahmed FCarrion JBellotti FLazzaroni LBarresi GBerta R(2022)A Serious Game for Using Socio-Economic and Trust Based Decision-Making Scenarios for Elicitation of Emotional ResponsesGames and Learning Alliance10.1007/978-3-031-22124-8_5(43-52)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-22124-8_5
Show More Cited By

Index Terms

Toward computers that recognize and respond to user emotion
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Adaptive intelligent user interfaces with emotion recognition
Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli
Special section on new and emerging technologies in bioinformatics and bioengineering

This paper proposes a methodology for the robust classification of neurophysiological data into four emotional states collected during passive viewing of emotional evocative pictures selected from the International Affective Picture System. The proposed ...
Using noninvasive wearable computers to recognize human emotions from physiological signals

We discuss the strong relationship between affect and cognition and the importance of emotions in multimodal human computer interaction (HCI) and user modeling. We introduce the overall paradigm for our multimodal system that aims at recognizing its ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IBM Systems Journal

IBM Systems Journal Volume 39, Issue 3-4

July 2000

517 pages

ISSN:0018-8670

Issue’s Table of Contents

Publisher

IBM Corp.

United States

Publication History

Published: 01 July 2000

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

36
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dadebayev DGoh WTan E(2022)EEG-based emotion recognitionJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2021.03.00934:7(4385-4401)Online publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2021.03.009
Genest PGoix LKhalafaoui YEgyed-Zsigmond EGrozavu N(2022)French translation of a dialogue dataset and text-based emotion detectionData & Knowledge Engineering10.1016/j.datak.2022.102099142:COnline publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1016/j.datak.2022.102099
Ahmed FCarrion JBellotti FLazzaroni LBarresi GBerta R(2022)A Serious Game for Using Socio-Economic and Trust Based Decision-Making Scenarios for Elicitation of Emotional ResponsesGames and Learning Alliance10.1007/978-3-031-22124-8_5(43-52)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1007/978-3-031-22124-8_5
Salutari ATarantino LDe Gasperis G(2022)BlocksBot: Towards an Empathic Robot Offering Multi-modal Emotion Detection Based on a Distributed Hybrid SystemHuman-Computer Interaction. Technological Innovation10.1007/978-3-031-05409-9_45(625-638)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-05409-9_45
Kim MKim GSim JJi Y(2020)How Motion Graphics Affect Emotional Quality: In the Context of an In-Vehicle Information SystemDesign, User Experience, and Usability. Interaction Design10.1007/978-3-030-49713-2_34(491-500)Online publication date: 19-Jul-2020
https://dl.acm.org/doi/10.1007/978-3-030-49713-2_34
Andrade PLaw E(2018)User-based evaluation of gamification elements in an educational applicationProceedings of the 32nd International BCS Human Computer Interaction Conference10.14236/ewic/HCI2018.27(1-13)Online publication date: 4-Jul-2018
https://dl.acm.org/doi/10.14236/ewic/HCI2018.27
Ng YKhong CNathan R(2018)Evaluating Affective User-Centered Design of Video Games Using Qualitative MethodsInternational Journal of Computer Games Technology10.1155/2018/37570832018Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1155/2018/3757083
Grd PTomičić IBaca M(2018)Privacy improvement model for biometric person recognition in ambient intelligence using perceptual hashingProceedings of the Central European Cybersecurity Conference 201810.1145/3277570.3277589(1-6)Online publication date: 15-Nov-2018
https://dl.acm.org/doi/10.1145/3277570.3277589
Salekin AChen ZAhmed MLach JMetz DDe La Haye KBell BStankovic J(2017)Distant Emotion RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/31309611:3(1-25)Online publication date: 11-Sep-2017
https://dl.acm.org/doi/10.1145/3130961
Markowski MNtalla IKamposiori CRoutsis VNg K(2015)Multidisciplinary perspectives on affective experiencesProceedings of the Conference on Electronic Visualisation and the Arts10.14236/ewic/eva2015.16(163-170)Online publication date: 7-Jul-2015
https://dl.acm.org/doi/10.14236/ewic/eva2015.16
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents