Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3064663.3064770acmconferencesArticle/Chapter ViewAbstractPublication PagesdisConference Proceedingsconference-collections
research-article

What Can Self-Reports and Acoustic Data Analyses on Emotions Tell Us?

Published: 10 June 2017 Publication History

Abstract

There are two approaches to measure people's experiences: memory-based and moment-based. Whereas the memory-based approaches are susceptible to the peak-end effect, the moment-based approaches appear to better reflect the experience of the present. In this paper, we propose that emotional assessment of think aloud verbalisations is a moment-based methodology for measuring UX. We conducted an empirical study with 46 participants in the domain of online shopping to evaluate their emotional experiences. Acoustic analysis of verbal data was used as the moment-based approach and self-report questionnaires as the retrospective or memory-based approach. Results of the study confirmed the previous finding that retrospective assessments did not reflect the actual experience. The results also suggested that retrospective evaluations of emotions were significantly correlated with the most frequently elicited emotion (i.e. modal emotion) during the interaction. In conclusion these results support the use of acoustic data analysis as an alternative approach to measuring UX.

References

[1]
Halis Altun and Gökhan Polat. 2007. New frameworks to boost feature selection algorithms in emotion detection for improved human-computer interaction. In International Symposium on Brain, Vision, and Artificial Intelligence. Springer, 533--541.
[2]
Panagiotis D Bamidis, Christos Papadelis, Chrysoula Kourtidou-Papadeli, Costas Pappas, and Ana B Vivas. 2004. Affective computing in the era of contemporary neurophysiology and health informatics. Interacting with Computers 16, 4 (2004), 715--721.
[3]
Javier A Bargas-Avila and Kasper Hornbæk. 2011. Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2689--2698.
[4]
Lisa Feldman Barrett and James A Russell. 1999. The structure of current affect: Controversies and emerging consensus. Current directions in psychological science 8, 1 (1999), 10--14.
[5]
Paulus Petrus Gerardus Boersma and others. 2002. Praat, a system for doing phonetics by computer. Glot international 5 (2002).
[6]
Daniel Bone, Chi-Chun Lee, and Shrikanth S Narayanan. 2012. A Robust Unsupervised Arousal Rating Framework using Prosody with Cross-Corpora Evaluation. In INTERSPEECH. 1175--1178.
[7]
Wolfram Boucsein. 2012. Electrodermal activity. Springer Science & Business Media.
[8]
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49--59.
[9]
Anders Bruun and Simon Ahm. 2015. Mind the gap! comparing retrospective and concurrent ratings of emotion in user experience evaluation. In Human-Computer Interaction. Springer, 237--254.
[10]
Anders Bruun, Effie Lai-Chong Law, Matthias Heintz, and Lana HA Alkly. 2016. Understanding the Relationship between Frustration and the Severity of Usability Problems: What can Psychophysiological Data (Not) Tell Us?. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3975--3987.
[11]
11. Anders Bruun and Jan Stage. 2015. An empirical study of the effects of three think-aloud protocols on identification of usability problems. In Human-Computer Interaction. Springer, 159--176.
[12]
Michael Burmester, Marcus Mast, Kilian Jäger, and Hendrik Homans. 2010. Valence method for formative evaluation of user experience. In Proceedings of the 8th ACM conference on Designing Interactive Systems. ACM, 364--367.
[13]
Brandon Burr. 2006. VACA: a tool for qualitative video analysis. In CHI'06 extended abstracts on Human factors in computing systems. ACM, 622--627.
[14]
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42, 4 (2008), 335.
[15]
Rafael A Calvo and Dorian Peters. 2014. Positive computing: technology for wellbeing and human potential. MIT Press.
[16]
Ching Hau Chan and Gareth JF Jones. 2005. Affect-based indexing and retrieval of films. In Proceedings of the 13th annual ACM international conference on Multimedia. ACM, 427--430.
[17]
Akhilesh Chandra and Thomas Calderon. 2005. Challenges and constraints to the diffusion of biometrics in information systems. Commun. ACM 48, 12 (2005), 101--106.
[18]
18. Andy Cockburn, Philip Quinn, and Carl Gutwin. 2015. Examining the peak-end effects of subjective experience. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, 357--366.
[19]
Fergus IM Craik and Robert S Lockhart. 1972. Levels of processing: A framework for memory research. Journal of verbal learning and verbal behavior 11, 6 (1972), 671--684.
[20]
John R Crawford and Julie D Henry. 2004. The Positive and Negative Affect Schedule (PANAS): Construct validity, measurement properties and normative data in a large non-clinical sample. British Journal of Clinical Psychology 43, 3 (2004), 245--265.
[21]
Pieter MA Desmet. 2012. Faces of product pleasure: 25 positive emotions in human-product interactions. International Journal of Design 6, 2 (2012).
[22]
Pieter MA Desmet and Anna E Pohlmeyer. 2013. Positive design: An introduction to design for subjective well-being. International Journal of Design 7, 3 (2013).
[23]
Sidney D'mello and Arthur Graesser. 2007. Mind and body: Dialogue and posture for affect detection in learning environments. Frontiers in Artificial Intelligence and Applications 158 (2007), 161.
[24]
Panteleimon Ekkekakis. 2013. The measurement of affect, mood, and emotion: A guide for health-behavioral research. Cambridge University Press.
[25]
Paul Ekman and Wallace V Friesen. 1978. Manual for the facial action coding system. Consulting Psychologists Press.
[26]
Paul Ekman, Robert W Levenson, and Wallace V Friesen. 1983. Autonomic nervous system activity distinguishes among emotions. Science 221, 4616 (1983), 1208--1210.
[27]
K Anders Ericsson and Herbert A Simon. 1980. Verbal reports as data. Psychological review 87, 3 (1980), 215.
[28]
Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, and others. 2016. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing 7, 2 (2016), 190--202.
[29]
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia. ACM, 1459--1462.
[30]
Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement 33, 3 (1973), 613--619.
[31]
Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD explorations newsletter 11, 1 (2009), 10--18.
[32]
Mark A Hall and Lloyd A Smith. 1999. Feature Selection for Machine Learning: Comparing a Correlation-Based Filter Approach to the Wrapper. In FLAIRS conference, Vol. 1999. 235--239.
[33]
Marc Hassenzahl. 2003. The thing and I: understanding the relationship between user and product. In Funology. Springer, 31--42.
[34]
Marc Hassenzahl. 2008. User experience (UX): towards an experiential perspective on product quality. In Proceedings of the 20th Conference on l'Interaction Homme-Machine. ACM, 11--15.
[35]
Christian Jones and Andrew Deeming. 2008. Affective human-robotic interaction. In Affect and emotion in human-computer interaction. Springer, 175--185.
[36]
Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan.
[37]
Daniel Kahneman, Barbara L Fredrickson, Charles A Schreiber, and Donald A Redelmeier. 1993. When more pain is preferred to less: Adding a better end. Psychological science 4, 6 (1993), 401--405.
[38]
Daniel Kahneman, D Kahneman, A Tversky, and others. 2003. Experienced utility and objective happiness: A moment-based approach. The psychology of economic decisions 1 (2003), 187--208.
[39]
Daniel Kahneman, Peter P Wakker, and Rakesh Sarin. 1997. Back to Bentham? Explorations of experienced utility. The quarterly journal of economics 112, 2 (1997), 375--406.
[40]
Paul R Kleinginna and Anne M Kleinginna. 1981. A categorized list of emotion definitions, with suggestions for a consensual definition. Motivation and emotion 5, 4 (1981), 345--379.
[41]
Peter J Lang. 1995. The emotion probe: Studies of motivation and attention. American psychologist 50, 5 (1995), 372.
[42]
Effie Lai-Chong Law. 2011. The measurability and predictability of user experience. In Proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems. ACM, 1--10.
[43]
Irene Lopatovska and Ioannis Arapakis. 2011. Theories, methods and current research on emotions in library and information science, information retrieval and human--computer interaction. Information Processing & Management 47, 4 (2011), 575--592.
[44]
Regan L Mandryk, Kori M Inkpen, and Thomas W Calvert. 2006. Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & information technology 25, 2 (2006), 141--158.
[45]
Albert Mehrabian and James A Russell. 1974. An approach to environmental psychology. the MIT Press.
[46]
Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes on in people's heads : reflections on the think-aloud technique. In Proceedings of the second Nordic conference on Human-computer interaction. ACM, 101--110.
[47]
Michael Nowak, Juho Kim, Nam Wook Kim, and Clifford Nass. 2012. Social visualization and negotiation: effects of feedback configuration and status. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1081--1090.
[48]
James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. Technical Report.
[49]
Helen Petrie and John Precious. 2010. Measuring User Experience of websites: Think aloud protocols and an emotion word prompt list. In CHI'10 Extended Abstracts on Human Factors in Computing Systems. ACM, 3673--3678.
[50]
Paolo Petta, Catherine Pelachaud, and Roddy Cowie. 2011. Emotion-oriented systems. The Humaine Handbook, ISBN (2011), 978--3.
[51]
Tomas Pfister and Peter Robinson. 2011. Real-time recognition of affective states from nonverbal features of speech and its application for public speaking skill analysis. IEEE Transactions on Affective Computing 2, 2 (2011), 66--78.
[52]
Rosalind W Picard and Roalind Picard. 1997. Affective computing. Vol. 252. MIT press Cambridge.
[53]
Talya Porat and Noam Tractinsky. 2012. It's a pleasure buying here: The effects of web-store design on consumers' emotions and attitudes. Human--Computer Interaction 27, 3 (2012), 235--276.
[54]
Jonathan Posner, James A Russell, and Bradley S Peterson. 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology 17, 03 (2005), 715--734.
[55]
Tauhidur Rahman and Carlos Busso. 2012. A personalized emotion recognition system using an unsupervised feature adaptation scheme. In Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on. IEEE, 5117--5120.
[56]
Fabien Ringeval, Florian Eyben, Eleni Kroupi, Anil Yuce, Jean-Philippe Thiran, Touradj Ebrahimi, Denis Lalanne, and Björn Schuller. 2015. Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data. Pattern Recognition Letters 66 (2015), 22--30.
[57]
Colin Robson. 2002. Real world research: a resource for social scientists and practitioner. Adapting Open Innovation in ICT Ecosystem Dynamics References Real World Research: A Resource for Social Scientists and Practitioner (2002), 270.
[58]
James A Russell. 1979. Affective space is bipolar. Journal of personality and social psychology 37, 3 (1979), 345.
[59]
James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review 110, 1 (2003), 145.
[60]
Andrew Ryan, Jeffery F Cohn, Simon Lucey, Jason Saragih, Patrick Lucey, Fernando De la Torre, and Adam Rossi. 2009. Automated facial expression recognition system. In Security Technology, 2009. 43rd Annual 2009 International Carnahan Conference on. IEEE, 172--177.
[61]
Klaus R Scherer. 1986. Vocal affect expression: a review and a model for future research. Psychological bulletin 99, 2 (1986), 143.
[62]
Klaus R Scherer. 2003. Vocal communication of emotion: A review of research paradigms. Speech communication 40, 1 (2003), 227--256.
[63]
Klaus R Scherer. 2005. What are emotions? And how can they be measured? Social science information 44, 4 (2005), 695--729.
[64]
Klaus R Scherer, Klaus R Scherer, and Paul Ekman. 1984. On the nature and function of emotion: A component process approach. Approaches to emotion 2293 (1984), 317.
[65]
Klaus R Scherer, Angela Schorr, and Tom Johnstone. 2001. Appraisal processes in emotion: Theory, methods, research. Oxford University Press.
[66]
Björn Schuller, Anton Batliner, Stefan Steidl, and Dino Seppi. 2011. Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech Communication 53, 9 (2011), 1062--1087.
[67]
Björn Schuller, Gerhard Rigoll, Michael Grimm, Kristian Kroschel, Tobias Moosmayr, and Günther Ruske. 2007. Effects of in-car noise-conditions on the recognition of emotion within speech. Fortschritte der Akustik 33, 1 (2007), 305.
[68]
Björn W Schuller, Stefan Steidl, Anton Batliner, and others. 2009. The INTERSPEECH 2009 emotion challenge. In Interspeech, Vol. 2009. 312--315.
[69]
Thomas Tullis, William Albert, Joseph S Dumas, and Beth A Loring. 2008. Measuring the User Experience: Collecting Analyzing, and Presenting Usability. (2008).
[70]
Asimina Vasalou and Tanja Bänziger. 2006. Using the *think aloud* for affective evaluations in the lab: a work-in-progress. (2006). http://emotion-research.net/projects/humaine/ws/wp10/ presentation-materials/VasalouAsimina-BaenzigerTanjaUsingThinkAloudForAffectiveEvaluations.pdf.
[71]
Dimitrios Ververidis and Constantine Kotropoulos. 2006. Emotional speech recognition: Resources, features, and methods. Speech communication 48, 9 (2006), 1162--1181.
[72]
Thurid Vogt, Elisabeth André, and Nikolaus Bee. 2008. EmoVoice? A framework for online recognition of emotions from voice. In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems. Springer, 188--199.
[73]
David G Wastell and Michael Newman. 1996. Stress, control and computer system design: a psychophysiological field study. Behaviour & Information Technology 15, 3 (1996), 183--192.
[74]
Margaret Wetherell. 2012. Affect and emotion: A new social science understanding. Sage Publications.
[75]
Carl E Williams and Kenneth N Stevens. 1972. Emotions and speech: Some acoustical correlates. The Journal of the Acoustical Society of America 52, 4B (1972), 1238--1250.
[76]
Zhihong Zeng, Maja Pantic, Glenn I Roisman, and Thomas S Huang. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence 31, 1 (2009), 39--58.
[77]
Philippe Zimmermann, Sissel Guttormsen, Brigitta Danuser, and Patrick Gomez. 2003. Affective computing a rationale for measuring mood with mouse and keyboard. International journal of occupational safety and ergonomics 9, 4 (2003), 539--551.

Cited By

View all
  • (2024)Ah-Aloud Method to Comprehend Time-Series Emotion Observation During Gameplay: An Initial Investigation with Japanese SpeakersProceedings of the ACM on Human-Computer Interaction10.1145/36770568:CHI PLAY(1-23)Online publication date: 15-Oct-2024
  • (2022)Post-Pandemic HCI—Living Digitally: Well-Being-Driven Digital TechnologiesInteracting with Computers10.1093/iwc/iwac02133:4(331-334)Online publication date: 11-Jul-2022
  • (2021)Are UX Evaluation Methods Providing the Same Big Picture?Sensors10.3390/s2110348021:10(3480)Online publication date: 17-May-2021
  • Show More Cited By

Index Terms

  1. What Can Self-Reports and Acoustic Data Analyses on Emotions Tell Us?

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    DIS '17: Proceedings of the 2017 Conference on Designing Interactive Systems
    June 2017
    1444 pages
    ISBN:9781450349222
    DOI:10.1145/3064663
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 June 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. acoustic analysis
    2. emotion
    3. evaluation
    4. user experience

    Qualifiers

    • Research-article

    Conference

    DIS '17
    Sponsor:
    DIS '17: Designing Interactive Systems Conference 2017
    June 10 - 14, 2017
    Edinburgh, United Kingdom

    Acceptance Rates

    DIS '17 Paper Acceptance Rate 107 of 487 submissions, 22%;
    Overall Acceptance Rate 1,158 of 4,684 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)24
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 22 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Ah-Aloud Method to Comprehend Time-Series Emotion Observation During Gameplay: An Initial Investigation with Japanese SpeakersProceedings of the ACM on Human-Computer Interaction10.1145/36770568:CHI PLAY(1-23)Online publication date: 15-Oct-2024
    • (2022)Post-Pandemic HCI—Living Digitally: Well-Being-Driven Digital TechnologiesInteracting with Computers10.1093/iwc/iwac02133:4(331-334)Online publication date: 11-Jul-2022
    • (2021)Are UX Evaluation Methods Providing the Same Big Picture?Sensors10.3390/s2110348021:10(3480)Online publication date: 17-May-2021
    • (2021)Facial Emotion Recognition in UX Evaluation: A Systematic ReviewHuman-Computer Interaction. Theory, Methods and Tools10.1007/978-3-030-78462-1_40(521-534)Online publication date: 24-Jul-2021
    • (2020)ML for UX? - An Inventory and Predictions on the Use of Machine Learning Techniques for UX ResearchProceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society10.1145/3419249.3420163(1-11)Online publication date: 25-Oct-2020
    • (2020)Automatic voice emotion recognition of child-parent conversations in natural settingsBehaviour & Information Technology10.1080/0144929X.2020.174168440:11(1072-1089)Online publication date: 17-Mar-2020
    • (2019)Negative Emotions, Positive ExperienceExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3313000(1-6)Online publication date: 2-May-2019
    • (2019)A Methodological Approach for Cross-Cultural Comparisons of Multimodal Emotional Expressions in Online Collaborative Learning EnvironmentsTransforming Learning with Meaningful Technologies10.1007/978-3-030-29736-7_58(645-649)Online publication date: 16-Sep-2019
    • (2018)Digital Educational GamesACM Transactions on Computer-Human Interaction10.1145/317788125:2(1-47)Online publication date: 11-Apr-2018
    • (2018)A Bermuda Triangle?Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3174035(1-16)Online publication date: 21-Apr-2018

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media