research-article

Pay attention!: designing adaptive agents that monitor and improve user engagement

Authors:

Bilge MutluAuthors Info & Claims

CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 11 - 20

https://doi.org/10.1145/2207676.2207679

Published: 05 May 2012 Publication History

Abstract

Embodied agents hold great promise as educational assistants, exercise coaches, and team members in collaborative work. These roles require agents to closely monitor the behavioral, emotional, and mental states of their users and provide appropriate, effective responses. Educational agents, for example, will have to monitor student attention and seek to improve it when student engagement decreases. In this paper, we draw on techniques from brain-computer interfaces (BCI) and knowledge from educational psychology to design adaptive agents that monitor student attention in real time using measurements from electroencephalography (EEG) and recapture diminishing attention levels using verbal and nonverbal cues. An experimental evaluation of our approach showed that an adaptive robotic agent employing behavioral techniques to regain attention during drops in engagement improved student recall abilities 43% over the baseline regardless of student gender and significantly improved female motivation and rapport. Our findings offer guidelines for developing effective adaptive agents, particularly for educational settings.

References

[1]

Aleven, V., and Koedinger, K. R. An effective metacognitive strategy: learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science 26, 2 (2002), 147--179.

[2]

Alibali, M. W., Heath, D. C., and Myers, H. J. Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44, 2 (2001), 169--188.

[3]

Anderson, J. R., Corbett, A. T., Koedinger, K. R., and Pelletier, R. Cognitive tutors: Lessons learned. The Journal of the Learning Sciences 4, 2 (1995), 167--207.

[4]

Ayaz, H., Shewokis, P., Bunce, S., Schultheis, M., and Onaral, B. Assessment of cognitive neural correlates for a functional near infrared-based brain computer interface system. In Augmented Cognition, HCII 2009, D. Schmorrow, I. Estabrooke, and M. Grootjen, Eds., vol. 5638 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2009, 699--708.

Digital Library

[5]

Barenger, D. K., and McCroskey, J. Immediacy in the classroom: Student immediacy. Communication Education 49 (2000), 178--186.

[6]

Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., Olmstead, R. E., Tremoulet, P. D., and Craven, P. L. EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, Space, and Environmental Medicine 78, Supplement 1 (May), B231--B244.

[7]

Buller, D. B., and Aune, R. K. The effects of vocalics and nonverbal sensitivity on compliance a speech accommodation theory explanation. Human Communication Research, 14 (1988), 301--332.

[8]

Burgoon, J. K., and Dillman, L. Gender, immediacy, and nonverbal communication. In Gender, power, and communication in human relationships, P. J. Kalbfleisch and M. J. Cody, Eds. Psychology Press, 1995.

[9]

Cassell, J., Steedman, M., Badler, N., Pelachaud, C., Stone, M., Douville, B., Prevost, S., and Achorn, B. Modeling the interaction between speech and gesture. In Proc CogSci '94 (1994), 153--158.

[10]

Chesebro, J. L., and McCroskey, J. C. The relationship of teacher clarity and immediacy with student state receiver apprehension, affect, and cognitive learning. Communication Education 50, 1 (January 2001), 59--68.

[11]

Christophel, D. M. The relationships among teacher immediacy behaviors, student motivation, and learning. Communication Education 39, 4 (1990), 323--340.

[12]

Cutrell, E. Tan, D. BCI for passive input in HCI. In Proc CHI '07 (2007).

[13]

Ferrez, P. W., and Millan. You are wrong! automatic detection of interaction errors from brain waves. In Proc IJCAI '05 (2005).

Digital Library

[14]

Gentili, R. J., Hadavi, C., Ayaz, H., Shewokis, P. A., and Contreras-Vidal, J. L. Hemodynamic correlates of visuomotor motor adaptation by functional near infrared spectroscopy. In Proc IEEE '10 (2010).

[15]

George, L., and Lecuyer, A. An overview of research on 'passive' brain-computer interfaces for implicit human-computer interaction. In Proc ICABB '10 Workshop: "Brain-Computer Interfacing and Virtual Reality" (2010).

[16]

Gevins, A., and Smith, M. E. Neurophysiological measures of cognitive workload during human-computer interaction. Theoretical Issues in Ergonomics Science 4, 1 (2003), 113--131.

[17]

Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., and Rush, G. Monitoring working memory load during computer-based tasks with EEG pattern recognition methods. Human Factors 40 (1998), 79--91.

[18]

Girouard, A., Solovey, E. T., and Jacob, R. J. K. Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. International Journal of Autonomous and Adaptive Communications Systems (2010).

Digital Library

[19]

Gorham, J. The relationship between verbal teacher immediacy behaviors and student learning. Communication Education 37 (1988), 40--53.

[20]

Gorham, J., and Christophel, D. M. The relationship of teachers' use of humor in the classroom to immediacy and student learning. Communication Education 39 (1990), 46--62.

[21]

Gorham, J., and Christophel, D. M. Students' perceptions of teacher behaviors as motivating and demotivating factors in college classes. Communication Quarterly 40, 3 (1992), 239--252.

[22]

Gulz, A. Benefits of virtual characters in computer based learning environments: Claims and evidence. Int. J. Artif. Intell. Ed. 14 (December 2004), 313--334.

Digital Library

[23]

Harris, M., and Rosenthal, R. No more teachers' dirty looks: Effects of teacher nonverbal behavior on student outcomes. applications of nonverbal communication. In Applications of nonverbal communication, R. E. Riggio and R. S. Feldman, Eds. Lawrence Erlbaum, 2005, 157--192.

[24]

Kanda, T., Ishiguro, H., Ono, T., Imai, M., and Nakatsu, R. Development and evaluation of an interactive humanoid robot "robovie". In Proc ICRA '02, vol. 2 (2002), 1848--1855.

[25]

Kulik, C.-L. C., and Kulik, J. A. Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior 7, 1-2 (1991), 75--94.

[26]

Lee, J. C., and Tan, D. S. Using a low-cost electroencephalograph for task classification in HCI research. In Proc UIST '06 (2006), 81--90.

Digital Library

[27]

Lowe, J. Computer-based education: Is it a panacea? Journal of Research on Technology in Education 34, 2 (2002), 163--71.

[28]

McCroskey, J. C., Richmond, V. P., Sallinen, A., Fayer, J. M., and Barraclough, R. Nonverbal immediacy and cognitive learning: A cross-cultural investigation. Communication Education 45, 3 (July 1996), 200--211.

[29]

McCroskey, J. C., Richmond, V. P., Sallinen, A., Fayer, J. M., and Barraclough, R. A. A cross-cultural and multi-behavioral analysis of the relationship between nonverbal immediacy and teacher evaluation. Communication Education 44, 4 (October 1995), 281--291.

[30]

Mehrabian, A. Immediacy: An indicator of attitudes in linguistic communication. Journal of Personality 34, 1 (1966), 26--34.

[31]

Menzel, K. E., and Carrell, L. J. The impact of gender and immediacy on willingness to talk and perceived learning. Communication Education 48, 1 (1999), 31--40.

[32]

Molina, G., Tsoneva, T., and Nijholt, A. Emotional brain-computer interfaces. In Proc ACII '09, IEEE (2009), 1--9.

[33]

Mumm, J., and Mutlu, B. Human-robot proxemics: physical and psychological distancing in human-robot interaction. In Proc HRI '11 (2011), 331--338.

Digital Library

[34]

Nijholt, A., Bos, D. P.-O., and Reuderink, B. Turning shortcomings into challenges: Brain-computer interfaces for games. Entertainment Computing 1, 2 (2009), 85--94.

[35]

Nijholt, A., and Tan, D. Brain-computer interfacing for intelligent systems. IEEE Intelligent Systems 23, 3 (May-June 2008), 72--79.

Digital Library

[36]

Pfurtscheller, G., Neuper, C., Guger, C., Harkam, W., Ramoser, H., Schlogl, A., Obermaier, B., and Pregenzer, M. Current trends in Graz Brain-Computer Interface (BCI) research. IEEE Transactions on Rehabilitation Engineering 8, 2 (2000), 216--219.

[37]

Pope, A. T., Bogart, E. H., and Bartolome, D. S. Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology 40, 1-2 (1995), 187--195.

[38]

Richmond, V. P. Teacher nonverbal immediacy use and outcomes. In Communication for Teachers, J. Chesebro and J. McCroskey, Eds. Allyn & Bacon., 2001, 65--82.

[39]

Richmond, V. P., Gorham, J. S., and McCroskey, J. C. The relationship between selected immediacy behaviors and cognitive learning. Communication Yearbook, 10 (1987), 574--590.

[40]

Richmond, V. P., and McCroskey, J. C. Influencing teacher influence through immediacy. In Power in the classroom: communication, control, and concern, V. P. Richmond and J. C. McCroskey, Eds. Psychology Press, 1992, 101--119.

[41]

Richmond, V. P., McCroskey, J. C., and Johnson, A. D. Development of nonverbal immediacy scale (nis): Measures of self-and other-perceived nonverbal immediacy. Communication Quarterly 51, 4 (2003), 504--517.

[42]

Rosip, J. C., and Hall, J. A. Knowledge of nonverbal cues, gender, and nonverbal decoding accuracy. Journal of Nonverbal Behavior 28 (2004), 267--286.

[43]

Scherer, K., Ladd, D., and Silverman, K. Vocal cues to speaker affect: Testing two models. Journal of the Acoustical Society of America 76, 5 (1984), 1346--1356.

[44]

Schofield, J. W. Computers and Classroom Culture. Cambridge University Press, 1995.

Digital Library

[45]

Tan, D. Brain-computer interfaces: applying our minds to human-computer interaction. In Proc CHI Workshop: "What is the Next Generation of Human-Computer Interaction?" (2006).

[46]

Woolfolk, A. E., and Brooks, D. The influence of teachers' nonverbal behaviors on students' perceptions and performance. The Elementary School Journal 85 (1985), 513--528.

[47]

Zander, T., and Kothe, C. Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. Journal of Neural Engineering 8, 2 (2011), 025005.

[48]

Zander, T. O., Kothe, C., Jatzev, S., and Gaertner, M. Enhancing human-computer interaction with input from active and passive brain-computer interfaces. In Brain-Computer Interfaces, D. S. Tan and A. Nijholt, Eds., Human-Computer Interaction Series. Springer London, 2010, 181--199.

Cited By

Sorbello RCalí C(2024)Perspective Chapter: A Model for Measuring Trust Using BCI in Human-Humanoid InteractionNew Insights in Brain-Computer Interface Systems [Working Title]10.5772/intechopen.115094Online publication date: 1-Oct-2024
https://doi.org/10.5772/intechopen.115094
Verma AWani N(2024)Channeling Basic Sciences for Innovation and Growth Insights From AI Integration in Education and BusinessUnleashing the Power of Basic Science in Business10.4018/979-8-3693-5503-9.ch012(222-242)Online publication date: 12-Jul-2024
https://doi.org/10.4018/979-8-3693-5503-9.ch012
Salomons NScassellati B(2024)Time-dependant Bayesian knowledge tracing—Robots that model user skills over timeFrontiers in Robotics and AI10.3389/frobt.2023.124924110Online publication date: 26-Feb-2024
https://doi.org/10.3389/frobt.2023.1249241
Show More Cited By

Index Terms

Pay attention!: designing adaptive agents that monitor and improve user engagement
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Towards ambulatory brain-computer interfaces: a pilot study with P300 signals
ACE '09: Proceedings of the International Conference on Advances in Computer Entertainment Technology

Brain-Computer Interfaces (BCI) are communication systems that enable users to interact with computers using only brain activity. This activity is generally measured by ElectroEncephaloGraphy (EEG). A major limitation of BCI is the electrical ...
Predicting epileptic seizures from scalp EEG based on attractor state analysis

The seizure-precursor phenomenon of the epileptic seizure is extracted from an attractor-based analysis of the macroscopic dynamics of the brain.A novel seizure prediction method using scalp electroencephalogram (EEG) based on attractor state analysis ...
Feature Analysis of EEG Based Brain-Computer Interfaces to Detect Motor Imagery
Brain Informatics
Abstract
Brain-Computer Interfaces (BCI) is one of the alluring breakthroughs for mankind as it provides a new way of communication for the patients of neuro-muscular disorders. Electroencephalography (EEG) signals are the most studied type of signals to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

May 2012

3276 pages

ISBN:9781450310154

DOI:10.1145/2207676

General Chair:
Joseph A. Konstan
University of Minnesota
,
Program Chairs:
Ed H. Chi
Google
,
Kristina Höök
Mobile Life at KTH

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 May 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '12

Sponsor:

SIGCHI

CHI '12: CHI Conference on Human Factors in Computing Systems

May 5 - 10, 2012

Texas, Austin, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

227
Total Citations
View Citations
3,817
Total Downloads

Downloads (Last 12 months)245
Downloads (Last 6 weeks)23

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sorbello RCalí C(2024)Perspective Chapter: A Model for Measuring Trust Using BCI in Human-Humanoid InteractionNew Insights in Brain-Computer Interface Systems [Working Title]10.5772/intechopen.115094Online publication date: 1-Oct-2024
https://doi.org/10.5772/intechopen.115094
Verma AWani N(2024)Channeling Basic Sciences for Innovation and Growth Insights From AI Integration in Education and BusinessUnleashing the Power of Basic Science in Business10.4018/979-8-3693-5503-9.ch012(222-242)Online publication date: 12-Jul-2024
https://doi.org/10.4018/979-8-3693-5503-9.ch012
Salomons NScassellati B(2024)Time-dependant Bayesian knowledge tracing—Robots that model user skills over timeFrontiers in Robotics and AI10.3389/frobt.2023.124924110Online publication date: 26-Feb-2024
https://doi.org/10.3389/frobt.2023.1249241
McKie EChandrasekaran AVenkataraman S(2024)How Do Curbside Feedback Tactics Impact Households’ Recycling Performance? Evidence From Community ProgramsProduction and Operations Management10.1177/1059147824123499933:5(1064-1082)Online publication date: 3-May-2024
https://doi.org/10.1177/10591478241234999
Chang ZPai YCao JGupta KBillinghurst M(2024)EMiRAs-Empathic Mixed Reality AgentsProceedings of the 3rd Empathy-Centric Design Workshop: Scrutinizing Empathy Beyond the Individual10.1145/3661790.3661791(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3661790.3661791
Salehzadeh RBezawada HCrawford CGrollman DBroadbent EJu WSoh HWilliams T(2024)BCI Exploration of User Responses to Vulnerable and Expressive Robot BehaviorsCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3640574(925-929)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610978.3640574
Ng SLin TLi YSebo SGrollman DBroadbent EJu WSoh HWilliams T(2024)Role-Playing with Robot Characters: Increasing User Engagement through Narrative and Gameplay AgencyProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634941(522-532)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634941
Choi JLeung JFrahm NChristman MBertasius GSengupta R(2024)Building Secure and Engaging Video Communication by Using Monitor Illumination2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00441(4377-4386)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00441
Lewis MHolloman AHernández Cuevas BCrawford C(2024)Exploring Computational Thinking Perspectives in Black Communities with Physiological Computing2024 Black Issues in Computing Education (BICE)10.1109/BICE60192.2024.00013(27-32)Online publication date: 1-Feb-2024
https://doi.org/10.1109/BICE60192.2024.00013
John ASingh AGramann KLiu DLin C(2024)Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaborationJournal of Neural Engineering10.1088/1741-2552/ad249421:2(026010)Online publication date: 19-Mar-2024
https://doi.org/10.1088/1741-2552/ad2494
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents