research-article

Public Access

Effect of Feedback on Users’ Immediate Emotions: Analysis of Facial Expressions during a Simulated Target Detection Task

Authors:

Md Abdullah Al Fahim,

Mohammad Maifi Hasan Khan,

Theodore Jensen,

Yusuf Albayram,

Ross BuckAuthors Info & Claims

ICMI '19: 2019 International Conference on Multimodal Interaction

Pages 49 - 58

https://doi.org/10.1145/3340555.3353732

Published: 14 October 2019 Publication History

All formats PDF

Abstract

Safety-critical systems (e.g., UAV systems) often incorporate warning modules that alert users regarding imminent hazards (e.g., system failures). However, these warning systems are often not perfect, and trigger false alarms, which can lead to negative emotions and affect subsequent system usage. Although various feedback mechanisms have been studied in the past to counter the possible negative effects of system errors, the effect of such feedback mechanisms and system errors on users’ immediate emotions and task performance is not clear. To investigate the influence of affective feedback on participants’ immediate emotions, we designed a 2 (warning reliability: high/low) × 2 (feedback: present/absent) between-group study where participants interacted with a simulated UAV system to identify and neutralize enemy vehicles under time constraint. Task performance along with participants’ facial expressions were analyzed. Results indicated that giving feedback decreased fear emotions during the task whereas warning increased frustration for high reliability groups compared to low reliability groups. Finally, feedback was found not to affect task performance.

References

[1]

[n.d.]. Affdex for Market Research (Cloud Based Emotion Recognition Software). https://www.affectiva.com/product/affdex-for-market-research/. [Online; accessed 28-April-2019].

[2]

[n.d.]. FaceReader Automatic Facial Expression Analysis Tool. https://www.noldus.com/human-behavior-research/products/facereader/. [Online; accessed 28-April-2019].

[3]

[n.d.]. iMotions Biometric Research Platform. https://imotions.com/. [Online; accessed 28-April-2019].

[4]

[n.d.]. Publications made with iMotions Biometric Research Platform. https://imotions.com/resources/publications/. [Online; accessed 28-April-2019].

[5]

Yusuf Albayram, Mohammad Maifi Hasan Khan, Theodore Jensen, Ross Buck, and Emil Coman. 2018. The Effects of Risk and Role on Users’ Anticipated Emotions in Safety-Critical Systems. In International Conference on Engineering Psychology and Cognitive Ergonomics. Springer, 369–388.

[6]

Anne Aula and Veikko Surakka. 2002. Auditory emotional feedback facilitates human-computer interaction. In People and Computers XVI-Memorable Yet Invisible. Springer, 337–349.

[7]

Laura H Barg-Walkow and Wendy A Rogers. 2016. The effect of incorrect reliability information on expectations, perceptions, and use of automation. Human factors 58, 2 (2016), 242–260.

[8]

Ann-Renee Blais and Elke U. Weber. 2006. A Domain-Specific Risk-Taking (DOSPERT) Scale for Adult Populations. Judgment and Decision Making 1, 1 (2006).

[9]

Ross Buck. 2014. Emotion: A biosocial synthesis. Cambridge University Press.

[10]

Ross Buck, Mohammad Khan, Michael Fagan, and Emil Coman. 2018. The User Affective Experience Scale: A Measure of Emotions Anticipated in Response to Pop-Up Computer Warnings. International Journal of Human–Computer Interaction 34, 1(2018), 25–34.

[11]

Gao-Yi Chao, Chun-Min Chang, Jeng-Lin Li, Ya-Tse Wu, and Chi-Chun Lee. 2018. Generating fMRI-Enriched Acoustic Vectors using a Cross-Modality Adversarial Network for Emotion Recognition. In Proceedings of the 2018 on International Conference on Multimodal Interaction. ACM, 55–62.

Digital Library

[12]

Jeffrey F Cohn, Laszlo A Jeni, Itir Onal Ertugrul, Donald Malone, Michael S Okun, David Borton, and Wayne K Goodman. 2018. Automated Affect Detection in Deep Brain Stimulation for Obsessive-Compulsive Disorder: A Pilot Study. In Proceedings of the 2018 on International Conference on Multimodal Interaction. ACM, 40–44.

Digital Library

[13]

Pasquale Dente, Dennis Küster, Lina Skora, and E Krumhuber. 2017. Measures and metrics for automatic emotion classification via FACET. In Proceedings of the Conference on the Study of Artificial Intelligence and Simulation of Behaviour (AISB). 160–163.

[14]

Munjal Desai, Poornima Kaniarasu, Mikhail Medvedev, Aaron Steinfeld, and Holly Yanco. 2013. Impact of robot failures and feedback on real-time trust. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction. IEEE Press, 251–258.

Digital Library

[15]

Mary T Dzindolet, Scott A Peterson, Regina A Pomranky, Linda G Pierce, and Hall P Beck. 2003. The role of trust in automation reliance. International journal of human-computer studies 58, 6 (2003), 697–718.

Digital Library

[16]

Mary T Dzindolet, Linda G Pierce, Hall P Beck, and Lloyd A Dawe. 2002. The perceived utility of human and automated aids in a visual detection task. Human Factors 44, 1 (2002), 79–94.

[17]

Paul Ekman and Wallace V Friesen. 1971. Constants across cultures in the face and emotion.Journal of personality and social psychology 17, 2(1971), 124.

[18]

Paul Ekman and Wallace V Friesen. 1976. Measuring facial movement. Environmental psychology and nonverbal behavior 1, 1 (1976), 56–75.

[19]

M Lance Frazier, Paul D Johnson, and Stav Fainshmidt. 2013. Development and validation of a propensity to trust scale. Journal of Trust Research 3, 2 (2013), 76–97.

[20]

Yuchi Huang and Saad Khan. 2018. A Generative Approach for Dynamically Varying Photorealistic Facial Expressions in Human-Agent Interactions. In Proceedings of the 2018 on International Conference on Multimodal Interaction. ACM, 437–445.

Digital Library

[21]

Theodore Jensen, Yusuf Albayram, Mohammad Maifi Hasan Khan, Ross Buck, Emil Coman, and Md Abdullah Al Fahim. 2018. Initial Trustworthiness Perceptions of a Drone System based on Performance and Process Information. In Proceedings of the 6th International Conference on Human-Agent Interaction. ACM, 229–237.

Digital Library

[22]

Theodore Jensen, Yusuf Albayram, Mohammad Maifi Hasan Khan, Md Abdullah Al Fahim, Ross Buck, and Emil Coman. 2019. The Apple Does Fall Far from the Tree: User Separation of a System from its Developers in Human-Automation Trust Repair. In Proceedings of the 2019 on Designing Interactive Systems Conference. ACM, 1071–1082.

Digital Library

[23]

Theodore Jensen, Mohammad Maifi Hasan Khan, Yusuf Albayram, Md Abdullah Al Fahim, Ross Buck, and Emil Coman. 2019. Anticipated Emotions in Initial Trust Evaluations of a Drone System Based on Performance and Process Information. International Journal of Human–Computer Interaction 0, 0(2019), 1–10. https://doi.org/10.1080/10447318.2019.1642616

[24]

Myounghoon Jeon, Bruce N Walker, and Jung-Bin Yim. 2014. Effects of specific emotions on subjective judgment, driving performance, and perceived workload. Transportation research part F: traffic psychology and behaviour 24 (2014), 197–209.

[25]

Yung-Tsan Jou, Tzu-Chung Yenn, Chiuhsiang Joe Lin, Wan-Shan Tsai, and Tsung-Ling Hsieh. 2011. The research on extracting the information of human errors in the main control room of nuclear power plants by using Performance Evaluation Matrix. Safety science 49, 2 (2011), 236–242.

[26]

Jonathan Klein, Youngme Moon, and Rosalind W Picard. 2002. This computer responds to user frustration: Theory, design, and results. Interacting with computers 14, 2 (2002), 119–140.

[27]

Tamar Kugler, Terry Connolly, and Lisa D Ordóñez. 2012. Emotion, decision, and risk: Betting on gambles versus betting on people. Journal of Behavioral Decision Making 25, 2 (2012), 123–134.

[28]

Iolanda Leite, Ginevra Castellano, André Pereira, Carlos Martinho, and Ana Paiva. 2012. Modelling empathic behaviour in a robotic game companion for children: an ethnographic study in real-world settings. In Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction. ACM, 367–374.

Digital Library

[29]

Jennifer S Lerner and Dacher Keltner. 2000. Beyond valence: Toward a model of emotion-specific influences on judgement and choice. Cognition & Emotion 14, 4 (2000), 473–493.

[30]

Minglei Li, Qin Lu, Yunfei Long, and Lin Gui. 2017. Inferring affective meanings of words from word embedding. IEEE Transactions on Affective Computing 8, 4 (2017), 443–456.

[31]

Edwin A Locke, Dong-Ok Chah, Scott Harrison, and Nancy Lustgarten. 1989. Separating the effects of goal specificity from goal level. Organizational Behavior and Human Decision Processes 43, 2 (1989), 270–287.

[32]

Edwin A Locke and Gary P Latham. 2002. Building a practically useful theory of goal setting and task motivation: A 35-year odyssey.American psychologist 57, 9 (2002), 705.

[33]

Joseph B Lyons and Charlene K Stokes. 2012. Human–human reliance in the context of automation. Human factors 54, 1 (2012), 112–121.

[34]

Derek McColl, Alexander Hong, Naoaki Hatakeyama, Goldie Nejat, and Beno Benhabib. 2016. A survey of autonomous human affect detection methods for social robots engaged in natural HRI. Journal of Intelligent & Robotic Systems 82, 1 (2016), 101–133.

Digital Library

[35]

Albert Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology 14, 4 (1996), 261–292.

[36]

Stephanie M Merritt. 2011. Affective processes in human–automation interactions. Human Factors 53, 4 (2011), 356–370.

[37]

Joachim Meyer. 2004. Conceptual issues in the study of dynamic hazard warnings. Human factors 46, 2 (2004), 196–204.

[38]

Douglas C Montgomery. 2017. Design and analysis of experiments(eighth ed.). John wiley & sons.

[39]

Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of social issues 56, 1 (2000), 81–103.

[40]

Clifford Nass, Jonathan Steuer, and Ellen R Tauber. 1994. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 72–78.

Digital Library

[41]

S Joon Park, Craig M MacDonald, and Michael Khoo. 2012. Do you care if a computer says sorry?: user experience design through affective messages. In Proceedings of the designing interactive systems conference. ACM, 731–740.

Digital Library

[42]

Timo Partala and Veikko Surakka. 2004. The effects of affective interventions in human–computer interaction. Interacting with computers 16, 2 (2004), 295–309.

[43]

Phuong Pham and Jingtao Wang. 2018. Adaptive Review for Mobile MOOC Learning via Multimodal Physiological Signal Sensing-A Longitudinal Study. In Proceedings of the 2018 on International Conference on Multimodal Interaction. ACM, 63–72.

Digital Library

[44]

Rosalind W Picard and Jonathan Klein. 2002. Computers that recognise and respond to user emotion: theoretical and practical implications. Interacting with computers 14, 2 (2002), 141–169.

[45]

Stephen Rice and David Keller. 2009. Automation reliance under time pressure.Cognitive Technology(2009).

[46]

Gayle Schwark and Stephen Rice. 2016. An Affect-Trust (AT) Model With Regards to Technological Errors. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 60. SAGE Publications Sage CA: Los Angeles, CA, 1150–1154.

[47]

Younho Seong and Ann M Bisantz. 2008. The impact of cognitive feedback on judgment performance and trust with decision aids. International Journal of Industrial Ergonomics 38, 7-8(2008), 608–625.

[48]

Sabrina Stöckli, Michael Schulte-Mecklenbeck, Stefan Borer, and Andrea C Samson. 2018. Facial expression analysis with AFFDEX and FACET: A validation study. Behavior research methods 50, 4 (2018), 1446–1460.

[49]

Charlene K Stokes, Joseph B Lyons, Kenneth Littlejohn, Joseph Natarian, Ellen Case, and Nicholas Speranza. 2010. Accounting for the human in cyberspace: Effects of mood on trust in automation. In Collaborative Technologies and Systems (CTS), 2010 International Symposium on. IEEE, 180–187.

[50]

Mohamed Yassine Tsalamlal, Michel-Ange Amorim, Jean-Claude Martin, and Mehdi Ammi. 2016. Combining facial expression and touch for perceiving emotional valence. IEEE Transactions on Affective Computing(2016).

[51]

Jeng-Yi Tzeng. 2004. Toward a more civilized design: studying the effects of computers that apologize. International Journal of Human-Computer Studies 61, 3 (2004), 319–345.

Digital Library

[52]

Elke U Weber, Ann-Renee Blais, and Nancy E Betz. 2002. A domain-specific risk-attitude scale: Measuring risk perceptions and risk behaviors. Journal of behavioral decision making 15, 4 (2002), 263–290.

[53]

Christopher D Wickens and Stephen R Dixon. 2007. The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Science 8, 3 (2007), 201–212.

[54]

Rebecca Wiczorek and Dietrich Manzey. 2014. Supporting attention allocation in multitask environments: effects of likelihood alarm systems on trust, behavior, and performance. Human factors 56, 7 (2014), 1209–1221.

[55]

Michael S Wogalter, Vincent C Conzola, and Tonya L Smith-Jackson. 2002. based guidelines for warning design and evaluation. Applied ergonomics 33, 3 (2002), 219–230.

[56]

Michael S Wogalter, Raheel Rashid, Steven W Clarke, and Michael J Kalsher. 1991. Evaluating the behavioral effectiveness of a multi-modal voice warning sign in a visually cluttered environment. In Proceedings of the Human Factors Society Annual Meeting, Vol. 35. SAGE Publications Sage CA: Los Angeles, CA, 718–722.

[57]

Jie Xu and Enid Montague. 2015. Affect and Trust in Technology in Teams: The Effect of Incidental Affect and Integral Affect. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 59. SAGE Publications Sage CA: Los Angeles, CA, 205–209.

[58]

Jie Xu and Enid Montague. 2018. An experimental study on individual and group affect in multi-tasking teams. Ergonomics (2018), 1–15.

Cited By

Al Fahim MKhan MJensen TAlbayram YComan EBuck R(2023)The Mediating Effect of Emotions on Trust in the Context of Automated System UsageIEEE Transactions on Affective Computing10.1109/TAFFC.2021.309488314:2(1572-1585)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3094883
Fahim MKhan MJensen TAlbayram Y(2022)Human vs. Automation: Which One Will You Trust More If You Are About to Lose Money?International Journal of Human–Computer Interaction10.1080/10447318.2022.207677239:12(2420-2435)Online publication date: 2-Jun-2022
https://doi.org/10.1080/10447318.2022.2076772
Fahim MKhan MJensen TAlbayram YComan E(2021)Do Integral Emotions Affect Trust? The Mediating Effect of Emotions on Trust in the Context of Human-Agent InteractionProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3461997(1492-1503)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3461997
Show More Cited By

Recommendations

Psychological responses to simulated displays of mismatched emotional expressions

Embodied agents are often designed with the ability to simulate human emotion. This paper investigates the psychological impact of simulated emotional expressions on computer users with a particular emphasis on how mismatched facial and audio ...
Manipulation of an emotional experience by real-time deformed facial feedback
AH '13: Proceedings of the 4th Augmented Human International Conference

The main goals of this paper involved assessing the efficacy of computer-generated emotion and establishing a method for integrating emotional experience. Human internal processing mechanisms for evoking an emotion by a relevant stimulus have not been ...
Cross-Cultural Design of Facial Expressions of Robots
Cross-Cultural Design. User Experience of Products, Services, and Intelligent Environments
Abstract
In this research, the authors succeeded in creating facial expressions made with the minimum necessary elements for recognizing a face. The elements are two eyes and a mouth made using precise circles, which are transformed to make facial ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICMI '19: 2019 International Conference on Multimodal Interaction

October 2019

601 pages

ISBN:9781450368605

DOI:10.1145/3340555

Editors:
Wen Gao
Peking University, China
,
Helen Mei Ling Meng
Chinese University of Hong Kong, China
,
Matthew Turk
Toyota Technological Institute at Chicago, USA
,
Susan R. Fussell
Cornell University, USA
,
Björn Schuller
Imperial College London / University of Augsburg, UK
,
Yale Song
Microsoft Research, USA
,
Kai Yu
Shanghai Jiao Tong University, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Air Force Office of Scientific Research

Conference

ICMI '19

ICMI '19: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 14 - 18, 2019

Suzhou, China

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
517
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)13

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Al Fahim MKhan MJensen TAlbayram YComan EBuck R(2023)The Mediating Effect of Emotions on Trust in the Context of Automated System UsageIEEE Transactions on Affective Computing10.1109/TAFFC.2021.309488314:2(1572-1585)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TAFFC.2021.3094883
Fahim MKhan MJensen TAlbayram Y(2022)Human vs. Automation: Which One Will You Trust More If You Are About to Lose Money?International Journal of Human–Computer Interaction10.1080/10447318.2022.207677239:12(2420-2435)Online publication date: 2-Jun-2022
https://doi.org/10.1080/10447318.2022.2076772
Fahim MKhan MJensen TAlbayram YComan E(2021)Do Integral Emotions Affect Trust? The Mediating Effect of Emotions on Trust in the Context of Human-Agent InteractionProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3461997(1492-1503)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3461997
Laffan KSunstein CDolan P(2021)Facing it: assessing the immediate emotional impacts of calorie labelling using automatic facial codingBehavioural Public Policy10.1017/bpp.2021.32(1-18)Online publication date: 11-Nov-2021
https://doi.org/10.1017/bpp.2021.32

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents