research-article

Investigating the Reliability of Self-report Data in the Wild: The Quest for Ground Truth

Authors:

Mohammad Saiedur Rahaman,

Flora D SalimAuthors Info & Claims

UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers

Pages 237 - 242

https://doi.org/10.1145/3460418.3479338

Published: 24 September 2021 Publication History

Abstract

Inferring human mental state (e.g., emotion, depression, engagement) with sensing technology is one of the most valuable challenges in the affective computing area, which has a profound impact in all industries interacting with humans. Self-report is the most common way to quantify how people think, but prone to subjectivity and various responses bias. It is usually used as the ground truth for human mental state prediction. In recent years, many data-driven machine learning models are built based on self-report annotations as the target value. In this research, we investigate the reliability of self-report data in the wild by studying the confidence level of responses and survey completion time. We conduct a case study (i.e., student engagement inference) by recruiting 23 students in a high school setting over a period of 4 weeks. Overall, our participants volunteered 488 self-reported responses and sensing data from smart wristbands. We find that the physiologically measured student engagement and perceived student engagement are not always consistent. The findings from this research have great potential to benefit future studies in predicting engagement, depression, stress, and other emotion-related states in the field of affective computing and sensing technologies.

References

[1]

Jorn Bakker, Mykola Pechenizkiy, and Natalia Sidorova. 2011. What’s Your Current Stress Level? Detection of Stress Patterns from GSR Sensor Data. In 2011 IEEE 11th International Conference on Data Mining Workshops. 573–580.

Digital Library

[2]

Stephen Barclay, Chris Todd, Ilora Finlay, Gunn Grande, and Penny Wyatt. 2002. Not Another Questionnaire! Maximizing the Response Rate, Predicting Non-response and Assessing Non-response Bias in Postal Questionnaire Studies of GPs. Family Practice 19, 1 (2002), 105–111.

[3]

Lee Anna Clark and David Watson. 2019. Constructing validity: New developments in creating objective measuring instruments.Psychological Assessment 31, 12 (2019), 1412–1427.

[4]

Elena Di Lascio, Shkurta Gashi, and Silvia Santini. 2018. Unobtrusive Assessment of Students’ Emotional Engagement during Lectures using Electrodermal Activity Sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–21.

Digital Library

[5]

Jennifer A Fredricks, Phyllis C Blumenfeld, and Alison H Paris. 2004. School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research 74, 1 (2004), 59–109.

[6]

Jennifer A Fredricks and Wendy McColskey. 2012. The Measurement of Student Engagement: A Comparative Analysis of Various Methods and Student Self-report Instruments. In Handbook of Research on Student Engagement. Springer, 763–782.

[7]

Kathryn A Fuller, Nilushi S Karunaratne, Som Naidu, Betty Exintaris, Jennifer L Short, Michael D Wolcott, Scott Singleton, and Paul J White. 2018. Development of a Self-report Instrument for Measuring in-class Student Engagement Reveals that Pretending to Engage is a Significant Unrecognized Problem. PLoS ONE 13, 10 (2018), e0205828.

[8]

Nan Gao, Max Marschall, Jane Burry, Simon Watkins, and Flora Salim. 2021. In-Gauge and En-Gage Datasets. Figshare. https://doi.org/10.25439/rmt.14578908. https://doi.org/10.25439/rmt.14578908

[9]

Nan Gao, Max Marschall, Jane Burry, Simon Watkins, and Flora D. Salim. 2021. Understanding Occupants’ Behaviour, Engagement, Emotion, and Comfort Indoors with Heterogeneous Sensors and Wearables. arxiv:2105.06637 [cs.HC]

[10]

Nan Gao, Wei Shao, Mohammad Saiedur Rahaman, and Flora D Salim. 2020. n-Gage: Predicting in-class Emotional, Behavioural and Cognitive Engagement in the Wild. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1–26.

Digital Library

[11]

Nan Gao, Wei Shao, and Flora D Salim. 2019. Predicting Personality Traits from Physical Activity Intensity. Computer 52, 7 (2019), 47–56.

[12]

Shkurta Gashi, Elena Di Lascio, and Silvia Santini. 2018. Using Students’ Physiological Synchrony to Quantify the Classroom Emotional Climate. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 698–701.

Digital Library

[13]

Shkurta Gashi, Elena Di Lascio, and Silvia Santini. 2019. Using Unobtrusive Wearable Sensors to Measure the Physiological Synchrony between Presenters and Audience Members. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (2019), 1–19.

Digital Library

[14]

Nathan W. Hudson, Ivana Anusic, Richard E. Lucas, and M. Brent Donnellan. 2020. Comparing the Reliability and Validity of Global Self-Report Measures of Subjective Well-Being With Experiential Day Reconstruction Measures. Assessment 27, 1 (2020), 102–116. https://doi.org/10.1177/1073191117744660 arXiv:https://doi.org/10.1177/1073191117744660PMID: 29254354.

[15]

Sinh Huynh, Seungmin Kim, JeongGil Ko, Rajesh Krishna Balan, and Youngki Lee. 2018. EngageMon: Multi-Modal Engagement Sensing for Mobile Games. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–27.

Digital Library

[16]

Stephanos Ioannou, Vittorio Gallese, and Arcangelo Merla. 2014. Thermal Infrared Imaging in Psychophysiology: Potentialities and Limits. Psychophysiology 51, 10 (2014), 951–963.

[17]

Robin C. Jackson, Hayley Barton, Kelly J. Ashford, and Bruce Abernethy. 2018. Stepovers and Signal Detection: Response Sensitivity and Bias in the Differentiation of Genuine and Deceptive Football Actions.Frontiers in Psychology 9 (2018), 2043.

[18]

Zachary D King, Judith Moskowitz, Begum Egilmez, Shibo Zhang, Lida Zhang, Michael Bass, John Rogers, Roozbeh Ghaffari, Laurie Wakschlag, and Nabil Alshurafa. 2019. Micro-stress EMA: A Passive Sensing Framework for Detecting in-the-wild Stress in Pregnant Mothers. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–22.

Digital Library

[19]

Kurt Kroenke, Robert L. Spitzer, Janet B.W. Williams, and Bernd Löwe. 2009. An Ultra-brief Screening Scale for Anxiety and Depression: The PHQ-4. Psychosomatics 50, 6 (2009), 613–621.

[20]

Kurt Kroenke, Tara W. Strine, Robert L. Spitzer, Janet B.W. Williams, Joyce T. Berry, and Ali H. Mokdad. 2009. The PHQ-8 as a Measure of Current Depression in the General Population. Journal of Affective Disorders 114, 1 (2009), 163–173.

[21]

Neil Malhotra. 2008. Completion Time and Response Order Effects in Web Surveys. Public Opinion Quarterly 72, 5 (2008), 914–934.

[22]

Andreas Möller, Matthias Kranz, Barbara Schmid, Luis Roalter, and Stefan Diewald. 2013. Investigating Self-reporting Behavior in Long-term Studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2931–2940.

Digital Library

[23]

Mehrab Bin Morshed, Koustuv Saha, Richard Li, Sidney K. D’Mello, Munmun De Choudhury, Gregory D. Abowd, and Thomas Plötz. 2019. Prediction of Mood Instability with Passive Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–21.

Digital Library

[24]

Richard V Palumbo, Marisa E Marraccini, Lisa L Weyandt, Oliver Wilder-Smith, Heather A McGee, Siwei Liu, and Matthew S Goodwin. 2017. Interpersonal Autonomic Physiology: A Systematic Review of the Literature. Personality and Social Psychology Review 21, 2 (2017), 99–141.

[25]

Mohammad Saiedur Rahaman, Jonathan Liono, Yongli Ren, Jeffrey Chan, Shaw Kudo, Tim Rawling, and Flora D Salim. 2020. An Ambient-physical System to Infer Concentration in Open-plan Workplace. IEEE Internet of Things Journal(2020), 1–1. https://doi.org/10.1109/JIOT.2020.2996219

[26]

Daniel C Richardson, Nicole K Griffin, Lara Zaki, Auburn Stephenson, Jiachen Yan, Thomas Curry, Richard Noble, John Hogan, Jeremy I Skipper, and Joseph T Devlin. 2020. Engagement in Video and Audio Narratives: Contrasting Self-report and Physiological Measures. Scientific Reports 10, 1 (2020), 1–8.

[27]

Ellen A Skinner, Thomas A Kindermann, and Carrie J Furrer. 2009. A Motivational Perspective on Engagement and Disaffection: Conceptualization and Assessment of Children’s Behavioral and Emotional Participation in Academic Activities in the Classroom. Educational and Psychological Measurement 69, 3 (2009), 493–525.

[28]

Eric van Sonderen, Robbert Sanderman, and James C. Coyne. 2013. Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain. PLoS ONE 8, 7 (2013).

[29]

Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell. 2014. StudentLife: Assessing Mental Health, Academic Performance and Behavioral Trends of College Students using Smartphones. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 3–14.

Digital Library

[30]

Rui Wang, Weichen Wang, Alex DaSilva, Jeremy F Huckins, William M Kelley, Todd F Heatherton, and Andrew T Campbell. 2018. Tracking Depression Dynamics in College Students using Mobile Phone and Wearable Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–26.

Digital Library

[31]

Weichen Wang, Gabriella M Harari, Rui Wang, Sandrine R Müller, Shayan Mirjafari, Kizito Masaba, and Andrew T Campbell. 2018. Sensing Behavioral Change over Time: Using Within-person Variability Features from Mobile Sensing to Predict Personality Traits. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–21.

Digital Library

[32]

Rick Wash, Emilee Rader, and Chris Fennell. 2017. Can People Self-Report Security Accurately? Agreement Between Self-Report and Behavioral Measures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2228–2232.

Digital Library

[33]

Xuhai Xu, Prerna Chikersal, Afsaneh Doryab, Daniella K Villalba, Janine M Dutcher, Michael J Tumminia, Tim Althoff, Sheldon Cohen, Kasey G Creswell, J David Creswell, 2019. Leveraging Routine Behavior and Contextually-filtered Features for Depression Detection among College Students. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 1–33.

Digital Library

[34]

Xiao Zhang, Wenzhong Li, Xu Chen, and Sanglu Lu. 2018. Moodexplorer: Towards Compound Emotion Detection via Smartphone Sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–30.

Digital Library

Cited By

Funk MOkada SAndré E(2024)Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on EngagementProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685757(224-233)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685757
Zhang PJung GAlikhanov JAhmed ULee U(2024)A Reproducible Stress Prediction Pipeline with Mobile Sensor DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785788:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678578
Suen HHung KLiu CSu YFan H(2024)Artificial Intelligence Can Recognize Whether a Job Applicant Is Selling and/or Lying According to Facial Expressions and Head Movements Much More Correctly Than Human InterviewersIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.337673211:5(5949-5960)Online publication date: Oct-2024
https://doi.org/10.1109/TCSS.2024.3376732
Show More Cited By

Recommendations

Stress recognition in human-computer interaction using physiological and self-reported data: a study of gender differences
PCI '15: Proceedings of the 19th Panhellenic Conference on Informatics

This paper investigates gender differences in stress recognition in human computer interaction (HCI) for both objective (i.e., skin conductance) and subjective (i.e., valence-arousal VA ratings) metrics. To this end, 31 healthy participants, 18 females, ...
Mediating individual affective experience through the emotional photo frame
Affect Aware Ubiquitous Computing

A photograph is considered a medium with emotional legibility and a means of expressing and exchanging emotional experience. This research presents the interactive emotional photo frame system focusing on mediating individual affective experience among ...
Towards a Ground Truth for Affective Classification in Movies
ISM '09: Proceedings of the 2009 11th IEEE International Symposium on Multimedia

Automatic affective movie classification can be a powerful technology that can facilitate searching, recommending and play listing of movies. In the process of developing a good classification system, the identification of suitable classes and the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers

September 2021

711 pages

ISBN:9781450384612

DOI:10.1145/3460418

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Australian Research Council

Conference

UbiComp '21

Sponsor:

UbiComp '21: The 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 21 - 26, 2021

Virtual, USA

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
319
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)9

Reflects downloads up to 17 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Funk MOkada SAndré E(2024)Multilingual Dyadic Interaction Corpus NoXi+J: Toward Understanding Asian-European Non-verbal Cultural Characteristics and their Influences on EngagementProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685757(224-233)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685757
Zhang PJung GAlikhanov JAhmed ULee U(2024)A Reproducible Stress Prediction Pipeline with Mobile Sensor DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785788:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678578
Suen HHung KLiu CSu YFan H(2024)Artificial Intelligence Can Recognize Whether a Job Applicant Is Selling and/or Lying According to Facial Expressions and Head Movements Much More Correctly Than Human InterviewersIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.337673211:5(5949-5960)Online publication date: Oct-2024
https://doi.org/10.1109/TCSS.2024.3376732
Synek SLohman HJewell V(2024)The Effectiveness of Upper Extremity Orthotic Interventions on Performance Skills and Performance of Occupations for Adults after Stroke: A Scoping ReviewOccupational Therapy In Health Care10.1080/07380577.2024.231080138:2(236-253)Online publication date: 7-Feb-2024
https://doi.org/10.1080/07380577.2024.2310801
Ahmed FBerta RBellotti FLazzaroni LFloris FBarresi GCarrion J(2023)Socio-Economic Decision Making and Emotion Elicitation with a Serious Game in the WildApplied Sciences10.3390/app1311643213:11(6432)Online publication date: 24-May-2023
https://doi.org/10.3390/app13116432
Sabuncuoglu ASezgin T(2023)Developing a Multimodal Classroom Engagement Analysis Dashboard for Higher-EducationProceedings of the ACM on Human-Computer Interaction10.1145/35932407:EICS(1-23)Online publication date: 19-Jun-2023
https://dl.acm.org/doi/10.1145/3593240
Gao NRahaman MShao WJi KSalim F(2022)Individual and Group-wise Classroom Seating ExperienceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35503356:3(1-23)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550335
Majethia RSharma VDwaraghanath R(2022)Mental Health Indices as Biomarkers for Assistive Mental Healthcare in University Students2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)10.1109/ACII55700.2022.9953847(1-8)Online publication date: 18-Oct-2022
https://doi.org/10.1109/ACII55700.2022.9953847
Gao NMarschall MBurry JWatkins SSalim F(2022)Understanding occupants’ behaviour, engagement, emotion, and comfort indoors with heterogeneous sensors and wearablesScientific Data10.1038/s41597-022-01347-w9:1Online publication date: 2-Jun-2022
https://doi.org/10.1038/s41597-022-01347-w

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents