research-article

See You See Me: The Role of Eye Contact in Multimodal Human-Robot Interaction

Authors:

Tian (Linger) Xu,

Chen YuAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 6, Issue 1

Article No.: 2, Pages 1 - 22

https://doi.org/10.1145/2882970

Published: 05 May 2016 Publication History

Abstract

We focus on a fundamental looking behavior in human-robot interactions—gazing at each other's face. Eye contact and mutual gaze between two social partners are critical in smooth human-human interactions. Therefore, investigating at what moments and in what ways a robot should look at a human user's face as a response to the human's gaze behavior is an important topic. Toward this goal, we developed a gaze-contingent human-robot interaction system, which relied on momentary gaze behaviors from a human user to control an interacting robot in real time. Using this system, we conducted an experiment in which human participants interacted with the robot in a joint-attention task. In the experiment, we systematically manipulated the robot's gaze toward the human partner's face in real time and then analyzed the human's gaze behavior as a response to the robot's gaze behavior. We found that more face looks from the robot led to more look-backs (to the robot's face) from human participants, and consequently, created more mutual gaze and eye contact between the two. Moreover, participants demonstrated more coordinated and synchronized multimodal behaviors between speech and gaze when more eye contact was successfully established and maintained.

References

[1]

Henny Admoni, Christopher Datsikas, and Brian Scassellati. 2014. Speech and gaze conflicts in collaborative human-robot interactions. In Proceedings of the 36th Annual Conference of the Cognitive Science Society (CogSci’14).

[2]

Henny Admoni, Bradley Hayes, David Feil-Seifer, Daniel Ullman, and Brian Scassellati. 2013. Are you looking at me?: Perception of robot attention is mediated by gaze type and group size. In Proceedings of the 8th ACM/IEEE International Conference on Human-Robot Interaction. IEEE Press, 389--396.

Digital Library

[3]

Sean Andrist, Xiang Zhi Tan, Michael Gleicher, and Bilge Mutlu. 2014. Conversational gaze aversion for humanlike robots. In Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 25--32.

Digital Library

[4]

M. Argyle. 1988. Bodily Communication. Methuen, New York, NY.

[5]

Michael Argyle and Jean Ann Graham. 1976. The central Europe experiment: Looking at persons and looking at objects. Journal of Nonverbal Behavior 1, 1 (1976), 6--16.

[6]

Dare A. Baldwin. 1993. Infants’ ability to consult the speaker for clues to word reference. Journal of Child Language 20, 02 (1993), 395--418.

[7]

Jean-David Boucher, Ugo Pattacini, Amelie Lelong, Gerard Bailly, Frederic Elisei, Sascha Fagel, Peter Ford Dominey, and Jocelyne Ventre-Dominey. 2012. I reach faster when I see you look: Gaze effects in human--human and human--robot face-to-face cooperation. Frontiers in Neurorobotics 6, 3 (2012), 1--11.

[8]

Tanya L. Chartrand and John A. Bargh. 1999. The chameleon effect: The perception--behavior link and social interaction. Journal of Personality and Social Psychology 76, 6 (1999), 893.

[9]

H. H. Clark and S. E. Brennan. 1991. Grounding in communication. Perspectives on Socially Shared Cognition 13 (1991), 127--149.

[10]

Jeffrey F. Cohn and Edward Z. Tronick. 1987. Mother--infant face-to-face interaction: The sequence of dyadic states at 3, 6, and 9 months. Developmental Psychology 23, 1 (1987), 68.

[11]

Adam Csapo, Emer Gilmartin, Jonathan Grizou, Jingguang Han, Raveesh Meena, Dimitra Anastasiou, Kristiina Jokinen, and Graham Wilcock. 2012. Multimodal conversational interaction with a humanoid robot. In Proceedings of the 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom). IEEE, 667--672.

[12]

K. De Barbaro, C. M. Johnson, D. Forster, and G. O. Deak. 2013. Methodological considerations for investigating the microdynamics of social interaction development. IEEE Transactions on Autonomous Mental Development 5, 258--270.

Digital Library

[13]

Katharine Graf Estes, Julia L. Evans, Martha W. Alibali, and Jenny R. Saffran. 2007. Can infants map meaning to newly segmented words? Statistical Segmentation and Word Learning. Psychological Science 18, 3 (2007), 254--260.

[14]

Kerstin Fischer. 2011. Interpersonal variation in understanding robots as social actors. In Proceedings of the 6th International Conference on Human-Robot Interaction. ACM, 53--60.

Digital Library

[15]

Alan Fogel. 1977. Temporal organization in mother-infant face-to-face interaction. In Studies in Mother-Infant Interaction. 119--152.

[16]

Megan Freeth, Tom Foulsham, and Alan Kingstone. 2013. What affects social attention? Social presence, eye contact and autistic traits. PLoS One 8, 1 (2013), e53286.

[17]

D. Fricker, H. Zhang, and C. Yu. 2011. Sequential pattern mining of multimodal data streams in dyadic interactions. In Proceedings of the IEEE Conference of Development and Learning. 1--6.

[18]

Alexandra Frischen, Andrew P. Bayliss, and Steven P. Tipper. 2007. Gaze cueing of attention: Visual attention, social cognition, and individual differences. Psychological Bulletin 133, 4 (2007), 694.

[19]

Lakshmi J. Gogate. 2010. Learning of syllable--object relations by preverbal infants: The role of temporal synchrony and syllable distinctiveness. Journal of Experimental Child Psychology 105, 3 (2010), 178--197.

[20]

Lakshmi J. Gogate, Lorraine E. Bahrick, and Jilayne D. Watson. 2000. A study of multimodal motherese: The role of temporal synchrony between verbal labels and gestures. Child Development 71, 4 (2000), 878--894.

[21]

Michael H. Goldstein, Jennifer A. Schwade, and Marc H. Bornstein. 2009. The value of vocalizing: Five-month-old infants associate their own noncry vocalizations with responses from caregivers. Child Development 80, 3 (2009), 636--644.

[22]

Charles Goodwin. 1980. Restarts, pauses, and the achievement of a state of mutual gaze at turn-beginning. Sociological Inquiry 50, 3--4 (1980), 272--302.

[23]

David Gouaillier, Vincent Hugel, Pierre Blazevic, Chris Kilner, Jerome Monceaux, Pascal Lafourcade, Brice Marnier, Julien Serre, and Bruno Maisonnier. 2008. The Nao humanoid: A combination of performance and affordability. CoRR Abs/0807.3223.

[24]

Z. M. Griffin and K. Bock. 2000. What the eyes say about speaking. Psychological Science 11, 4 (2000), 274--279.

[25]

Koh Hosoda, Hidenobu Sumioka, Akio Morita, and Minoru Asada. 2004. Acquisition of human-robot joint attention through real-time natural interaction. In Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’04). IEEE, 2867--2872.

[26]

Jukka Hyönä and Richard K. Olson. 1995. Eye fixation patterns among dyslexic and normal readers: Effects of word length and word frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition 21, 6 (1995), 1430.

[27]

Kristiina Jokinen and Graham Wilcock. 2014. Multimodal open-domain conversations with the Nao robot. In Natural Interaction with Robots, Knowbots and Smartphones. Springer, 213--224.

[28]

Masayuki Kamashima, Takayuki Kanda, Michita Imai, Tetsuo Ono, Daisuke Sakamoto, Hiroshi Ishiguro, and Yuichiro Anzai. 2004. Embodied cooperative behaviors by an autonomous humanoid robot. In Proceedings of the 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2506--2513.

[29]

A. Kendon and M. Cook. 1969. The consistency of gaze patterns in social interaction. British Journal of Psychology 60, 4 (1969), 481--494.

[30]

Adam Kendon. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26 (1967), 22--63.

[31]

C. L. Kleinke. 1986. Gaze and eye contact: A research review. Psychological Bulletin 100, 1 (1986), 78--100.

[32]

Patricia K. Kuhl, Feng-Ming Tsao, and Huei-Mei Liu. 2003. Foreign-language experience in infancy: Effects of short-term exposure and social interaction on phonetic learning. Proceedings of the National Academy of Sciences 100, 9096--9101.

[33]

Chaoran Liu, Carlos T. Ishi, Hiroshi Ishiguro, and Norihiro Hagita. 2012. Generation of nodding, head tilting and eye gazing for human-robot dialogue interaction. In Proceeding of the 2012 7th ACM/IEEE International Conference Human-Robot Interaction (HRI). IEEE, 285--292.

Digital Library

[34]

Simon P. Liversedge and John M. Findlay. 2000. Saccadic eye movements and cognition. Trends in Cognitive Sciences 4, 1 (2000), 6--14.

[35]

Caroline Lyon, Chrystopher L. Nehaniv, and Joe Saunders. 2012. Interactive language learning by robots: The transition from babbling to word forms. PLoS One 7, 6 (2012), e38236.

[36]

Davide Marocco, Angelo Cangelosi, Kerstin Fischer, and Tony Belpaeme. 2010. Grounding action words in the sensorimotor interaction with the world: Experiments with a simulated iCub humanoid robot. Frontiers in Neurorobotics 4 (2010), 7.

[37]

K. L. Marsh, M. J. Richardson, and R. C. Schmidt. 2009. Social connection through joint action and interpersonal coordination. Topics in Cognitive Science 1, 2 (2009), 320--339.

[38]

Ethel Matin. 1974. Saccadic suppression: A review and an analysis. Psychological Bulletin 81, 12 (1974), 899.

[39]

Raveesh Meena, Kristiina Jokinen, and Graham Wilcock. 2012. Integration of gestures and speech in human-robot interaction. In Proceedings of the 2012 IEEE 3rd International Conference on Cognitive Infocommunications (CogInfoCom). IEEE, 673--678.

[40]

P. Mundy and L. Newell. 2007. Attention, joint attention, and social cognition. Current Directions in Psychological Science 16, 5 (2007), 269--274.

[41]

B. Mutlu, T. Shiwa, T. Kanda, H. Ishiguro, and N. Hagita. 2009. Footing in human-robot conversations: How robots might shape participant roles using gaze cues. In Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction. ACM, 61--68.

Digital Library

[42]

Bilge Mutlu, Fumitaka Yamaoka, Takayuki Kanda, Hiroshi Ishiguro, and Norihiro Hagita. 2009. Nonverbal leakage in robots: Communication of intentions through seemingly unintentional behavior. In Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction. ACM, 69--76.

Digital Library

[43]

Yukie Nagai, Claudia Muhl, and Katharina J. Rohlfing. 2008. Toward designing a robot that learns actions from parental demonstrations. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’08). IEEE, 3545--3550.

[44]

Yukie Nagai and Katharina J. Rohlfing. 2007. Can motionese tell infants and robots what to imitate. In Proceedings of the 4th International Symposium on Imitation in Animals and Artifacts. 299--306.

[45]

Yukiko I. Nakano, Gabe Reinstein, Tom Stocky, and Justine Cassell. 2003. Towards a model of face-to-face grounding. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, -Volume 1. Association for Computational Linguistics, 553--561.

Digital Library

[46]

Tim Niem Ller, Alexander Ferrein, Gerhard Eckel, David Pirro, Patrick Podbregar, Tobias Kellner, Christof Rath, and Gerald Steinbauer. 2011. Providing ground-truth data for the Nao robot platform. In RoboCup 2010: Robot Soccer World Cup XIV. Springer, 133--144.

[47]

Karola Pitsch, Anna-Lisa Vollmer, Jannik Fritsch, Britta Wrede, Katharina Rohlfing, and Gerhard Sagerer. 2009. On the loop of action modification and the recipient’s gaze in adult-child interaction. In Proceedings of the International Conference on Speech and Gesture in Interaction (GESPIN).

[48]

Michael I. Posner. 1980. Orienting of attention. Quarterly Journal of Experimental Psychology 32, 3--25.

[49]

Charles Rich, Brett Ponsler, Aaron Holroyd, and Candace L. Sidner. 2010. Recognizing engagement in human-robot interaction. In Proceedings of the 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, 375--382.

Digital Library

[50]

M. Rolf, M. Hanheide, and K. J. Rohlfing. 2009. Attention via synchrony: Making use of multimodal cues in social learning. IEEE Transactions on Autonomous Mental Development 1, 55--67.

Digital Library

[51]

Brian Scassellati. 1999. Imitation and mechanisms of joint attention: A developmental structure for building social skills on a humanoid robot. Computation for Metaphors, Analogy, and Agents, 176--195.

Digital Library

[52]

Atsushi Senju and Mark H. Johnson. 2009. The eye contact effect: Mechanisms and development. Trends in Cognitive Sciences 13, 3 (2009), 127--134.

[53]

Sara C. Sereno and Keith Rayner. 2003. Measuring word recognition in reading: Eye movements and event-related potentials. Trends in Cognitive Sciences 7, 11 (2003), 489--493.

[54]

Syamimi Shamsuddin, Hanafiah Yussof, Luthffi Ismail, Fazah Akhtar Hanapiah, Salina Mohamed, Hanizah Ali Piah, and Nur Ismarrubie Zahari. 2012. Initial response of autistic children in human-robot interaction therapy with humanoid robot NAO. In Proceedings of the 2012 IEEE 8th International Colloquium on Signal Processing and Its Applications. IEEE, 188--193.

[55]

Candace L. Sidner, Christopher Lee, Cory D. Kidd, Neal Lesh, and Charles Rich. 2005. Explorations in Engagement for Humans and Robots. Artificial Intelligence 166, 140--164.

Digital Library

[56]

Maria Staudte and Matthew W. Crocker. 2011. Investigating joint attention mechanisms through spoken human--robot interaction. Cognition 120, 2 (2011), 268--291.

[57]

Fumihide Tanaka and Shizuko Matsuzoe. 2012. Children teach a care-receiving robot to promote their learning: Field experiments in a classroom for vocabulary learning. Journal of Human-Robot Interaction 1, 1 (Aug. 2012).

Digital Library

[58]

Adriana Tapus, Maja J. Mataric, and B. Scasselati. 2007. Socially assistive robotics [grand challenges of robotics]. Robotics & Automation Magazine, IEEE 14, 1 (2007), 35--42.

[59]

R. Vertegaal. 2003. Attentive user interfaces. Communications of the ACM 46, 3 (2003), 31--33.

Digital Library

[60]

Frances C. Volkmann. 1986. Human visual suppression. Vision Research 26, 9 (1986), 1401--1416.

[61]

Ning Wang and Jonathan Gratch. 2010. Don’t just stare at me&excl; In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1241--1250.

Digital Library

[62]

Tian Xu, Hui Zhang, and Chen Yu. 2013. Cooperative gazing behaviors in human multi-robot interaction. Interaction Studies 14, 3 (2013), 390--418.

[63]

A. Yamazaki, K. Yamazaki, Y. Kuno, M. Burdelski, M. Kawashima, and H. Kuzuoka. 2008. Precision timing in human-robot interaction: Coordination of head movement and utterance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 131--140.

Digital Library

[64]

Yuichiro Yoshikawa, Kazuhiko Shinozawa, Hiroshi Ishiguro, Norihiro Hagita, and Takanori Miyamoto. 2006. The effects of responsive eye movement and blinking behavior in a communication robot. In Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 4564--4569.

[65]

C. Yu, M. Scheutz, and P. Schermerhorn. 2010. Investigating multimodal real-time patterns of joint attention in an HRI word learning task. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction. ACM, 309--316.

Digital Library

[66]

C. Yu and L. B. Smith. 2012. Embodied attention and word learning by toddlers. Cognition 125, 2 (2012), 244--262.

Cited By

FANG YSHIOIRI S(2024)The Impact of Visual Field Size on Eye-Head Coordination During Cognitive ProcessingInterdisciplinary Information Sciences10.4036/iis.2024.A.0130:2(118-128)Online publication date: 2024
https://doi.org/10.4036/iis.2024.A.01
Wang XLi ZWang SYang YPeng YFu C(2024)Enhancing emotional expression in cat-like robots: strategies for utilizing tail movements with human-like gazesFrontiers in Robotics and AI10.3389/frobt.2024.139901211Online publication date: 15-Jul-2024
https://doi.org/10.3389/frobt.2024.1399012
Hoehl SKrenn BVincze M(2024)Honest machines? A cross-disciplinary perspective on trustworthy technology for childrenFrontiers in Developmental Psychology10.3389/fdpys.2024.13088812Online publication date: 28-Feb-2024
https://doi.org/10.3389/fdpys.2024.1308881
Show More Cited By

Index Terms

See You See Me: The Role of Eye Contact in Multimodal Human-Robot Interaction
1. Applied computing
  1. Law, social and behavioral sciences
    1. Psychology
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Laboratory experiments
  2. Interaction design
    1. Empirical studies in interaction design

Recommendations

Adaptive eye gaze patterns in interactions with human and artificial agents

Efficient collaborations between interacting agents, be they humans, virtual or embodied agents, require mutual recognition of the goal, appropriate sequencing and coordination of each agent's behavior with others, and making predictions from and about ...
I See What You See: Eye Movements in Real-World Scenes Are Affected by Perceived Direction of Gaze
Attention in Cognitive Systems. Theories and Systems from an Interdisciplinary Viewpoint

In this chapter, we report an investigation the influence of the saliency of another person's direction of gaze on an observer's eye movements through real-world scenes. Participants' eye movements were recorded while they viewed a sequence of scene ...
HMAGIC: head movement and gaze input cascaded pointing
PETRA '15: Proceedings of the 8th ACM International Conference on PErvasive Technologies Related to Assistive Environments

Augmentative and alternative communication tools allow people with severe motor disabilities to interact with computers. Two commonly used tools are video-based interfaces and eye trackers. Video-based interfaces map head movements captured by a camera ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 6, Issue 1

Special Issue on New Directions in Eye Gaze for Interactive Intelligent Systems (Part 2 of 2), Regular Articles and Special Issue on Highlights of IUI 2015 (Part 1 of 2)

May 2016

219 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/2896319

Editors:
Anthony Jameson
German Research Center for Artifi cial Intelligence (DFKI), Germany
,
Krzysztof Gajos
Harvard University, U.S.A.

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 May 2016

Accepted: 01 January 2016

Revised: 01 December 2015

Received: 01 December 2014

Published in TIIS Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSF BCS
NIH

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
690
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)17

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

FANG YSHIOIRI S(2024)The Impact of Visual Field Size on Eye-Head Coordination During Cognitive ProcessingInterdisciplinary Information Sciences10.4036/iis.2024.A.0130:2(118-128)Online publication date: 2024
https://doi.org/10.4036/iis.2024.A.01
Wang XLi ZWang SYang YPeng YFu C(2024)Enhancing emotional expression in cat-like robots: strategies for utilizing tail movements with human-like gazesFrontiers in Robotics and AI10.3389/frobt.2024.139901211Online publication date: 15-Jul-2024
https://doi.org/10.3389/frobt.2024.1399012
Hoehl SKrenn BVincze M(2024)Honest machines? A cross-disciplinary perspective on trustworthy technology for childrenFrontiers in Developmental Psychology10.3389/fdpys.2024.13088812Online publication date: 28-Feb-2024
https://doi.org/10.3389/fdpys.2024.1308881
Fang YPérez-Molerón JMerino LYeh SNishina SGomez R(2024)Enhancing social robot's direct gaze expression through vestibulo-ocular movementsAdvanced Robotics10.1080/01691864.2024.239855638:19-20(1457-1469)Online publication date: 10-Sep-2024
https://doi.org/10.1080/01691864.2024.2398556
Chan SChan CChan SZeng JZhong J(2023)A pilot study on factors of social attributes in desktop-size interactive robots2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309554(585-591)Online publication date: 28-Aug-2023
https://doi.org/10.1109/RO-MAN57019.2023.10309554
Clark AAhmad I(2023)Touchless and nonverbal human-robot interfaces: An overview of the state-of-the-artSmart Health10.1016/j.smhl.2022.10036527(100365)Online publication date: Mar-2023
https://doi.org/10.1016/j.smhl.2022.100365
Schmitz IEinhäuser W(2023)Gaze estimation in videoconferencing settingsComputers in Human Behavior10.1016/j.chb.2022.107517139:COnline publication date: 20-Jan-2023
https://dl.acm.org/doi/10.1016/j.chb.2022.107517
Bai KWang JWang HChen X(2022)A Study of Eye-Tracking Gaze Point Classification and Application Based on Conditional Random FieldApplied Sciences10.3390/app1213646212:13(6462)Online publication date: 25-Jun-2022
https://doi.org/10.3390/app12136462
Zhu XZhang XChen ZDong ZGu ZChang D(2022)The Trusted Listener: The Influence of Anthropomorphic Eye Design of Social Robots on User's Perception of TrustworthinessProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517670(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517670
de Haas MVogt Pvan den Berghe RLeseman POudgenoeg-Paz OWillemsen Bde Wit JKrahmer E(2022)Engagement in longitudinal child-robot language learning interactions: Disentangling robot and task engagementInternational Journal of Child-Computer Interaction10.1016/j.ijcci.2022.10050133(100501)Online publication date: Sep-2022
https://doi.org/10.1016/j.ijcci.2022.100501
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents