research-article

Semi-situated learning of verbal and nonverbal content for repeated human-robot interaction

Authors:

André Pereira,

Allison Funkhouser,

Jill Fain LehmanAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 13 - 20

https://doi.org/10.1145/2993148.2993190

Published: 31 October 2016 Publication History

Abstract

Content authoring of verbal and nonverbal behavior is a limiting factor when developing agents for repeated social interactions with the same user. We present PIP, an agent that crowdsources its own multimodal language behavior using a method we call semi-situated learning. PIP renders segments of its goal graph into brief stories that describe future situations, sends the stories to crowd workers who author and edit a single line of character dialog and its manner of expression, integrates the results into its goal state representation, and then uses the authored lines at similar moments in conversation. We present an initial case study in which the language needed to host a trivia game interaction is learned pre-deployment and tested in an autonomous system with 200 users "in the wild." The interaction data suggests that the method generates both meaningful content and variety of expression.

References

[1]

S. Al Moubayed, J. Beskow, G. Skantze, and B. Granström. Furhat: a back-projected human-like robot head for multiparty human-machine interaction. In Cognitive behavioural systems, pages 114–130. Springer, 2012.

Digital Library

[2]

T. Bickmore, D. Schulman, and L. Yin. Maintaining engagement in long-term interventions with relational agents. Applied Artificial Intelligence, 24(6):648–666, 2010.

Digital Library

[3]

C. Breazeal, N. DePalma, J. Orkin, S. Chernova, and M. Jung. Crowdsourcing Human-Robot interaction: New methods and system evaluation in a public environment. Journal of Human-Robot Interaction, 2(1):82–111, 2013.

Digital Library

[4]

A. Funkhouser. Annotation of utterances for conversational nonverbal behaviors. Master’s thesis, Robotics Institute, CMU, May 2016.

[5]

H. P. Graf, E. Cosatto, V. Strom, and F. J. Huang. Visual prosody: Facial movements accompanying speech. In Proc. of the 5th IEEE International Conference on Automatic Face and Gesture Recognition, pages 396–401. IEEE, 2002.

Digital Library

[6]

A. D. Jacq, S. Lemaignan, F. Garcia, P. Dillenbourg, and A. Paiva. Building successful long child-robot interactions in a learning context. In Proc. of the 11th ACM/IEEE International Conference on Human Robot Interaction, pages 239–246. IEEE Press, 2016.

Digital Library

[7]

F. Jurcıcek, S. Keizer, M. Gašic, F. Mairesse, B. Thomson, K. Yu, and S. Young. Real user evaluation of spoken dialogue systems using amazon mechanical turk. In Proc. of INTERSPEECH, volume 11, 2011.

[8]

A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proc. of the SIGCHI conf. on human factors in computing systems, pages 453–456. ACM, 2008.

Digital Library

[9]

W. S. Lasecki, R. Wesley, J. Nichols, A. Kulkarni, J. F. Allen, and J. P. Bigham. Chorus: a crowd-powered conversational assistant. In Proc. of the 26th annual ACM symposium on User interface software and technology, pages 151–162. ACM, 2013.

Digital Library

[10]

M. K. Lee, J. Forlizzi, P. E. Rybski, F. Crabbe, W. Chung, J. Finkle, E. Glaser, and S. Kiesler. The snackbot: documenting the design of a robot for long-term human-robot interaction. In Proc. of the 4th ACM/IEEE International Conference on Human-Robot Interaction, pages 7–14. IEEE, 2009.

Digital Library

[11]

I. Leite, G. Castellano, A. Pereira, C. Martinho, and A. Paiva. Empathic robots for long-term interaction. International Journal of Social Robotics, 6(3):329–341, 2014.

[12]

I. Leite, C. Martinho, and A. Paiva. Social robots for long-term interaction: a survey. International Journal of Social Robotics, 5(2):291–308, 2013.

[13]

B. Li, S. Lee-Urban, G. Johnston, and M. Riedl. Story generation with crowdsourced plot graphs. In Proc. of the AAAI Conference on Artificial Intelligence, 2013.

Digital Library

[14]

H. Li, P. Zhang, S. Al Moubayed, S. N. Patel, and A. P. Sample. Id-match: A hybrid computer vision and rfid system for recognizing individuals in groups. In Proc. of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 7–7, New York, NY, 2016. ACM.

Digital Library

[15]

M. Mitchell, W. Redmond, D. Bohus, and E. Kamar. Crowdsourcing language generation templates for dialogue systems. In Proc. of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 2014.

[16]

J. Orkin and D. Roy. The restaurant game: Learning social behavior and language from thousands of players online. Journal of Game Development, 3(1):39–60, 2007.

[17]

J. Orkin and D. Roy. Automatic learning and generation of social behavior from collective human gameplay. In Proc. of The 8th International Conference on Autonomous Agents and Multiagent Systems, pages 385–392. IFAAMAS, 2009.

Digital Library

[18]

B. Rossen and B. Lok. A crowdsourcing method to develop virtual human conversational agents. International Journal of Human-Computer Studies, 70(4):301–319, 2012.

Digital Library

[19]

O. Russakovsky, L.-J. Li, and L. Fei-Fei. Best of both worlds: human-machine collaboration for object annotation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2121–2131, 2015.

[20]

E. Short, K. Swift-Spong, J. Greczek, A. Ramachandran, A. Litoiu, E. C. Grigore, D. Feil-Seifer, S. Shuster, J. J. Lee, S. Huang, et al. How to train your dragonbot: Socially assistive robots for teaching children about nutrition through play. In Proc. of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 924–929. IEEE, 2014.

[21]

R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proc. of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 254–263, Stroudsburg, PA, USA, 2008.

Digital Library

[22]

Association for Computational Linguistics.

[23]

Z. Yu, Z. Xu, A. Black, and A. Rudnicky. Chatbot evaluation and database expansion via crowdsourcing. In Proc. of the chatbot workshop of LREC, 2016.

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Ravishankar JDoering MKanda TGrollman DBroadbent EJu WSoh HWilliams T(2024)Zero-Shot Learning to Enable Error Awareness in Data-Driven HRIProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634940(592-601)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634940
van Waveren SPek CTumova JLeite ISakamoto DWeiss AHiatt LShiomi M(2022)Correct Me If I'm Wrong: Using Non-Experts to Repair Reinforcement Learning PoliciesProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523825(493-501)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523825
Show More Cited By

Index Terms

Semi-situated learning of verbal and nonverbal content for repeated human-robot interaction
1. Human-centered computing
  1. Collaborative and social computing
    1. Collaborative and social computing systems and tools
  2. Interaction design
    1. Interaction design process and methods
2. Information systems
  1. World Wide Web
    1. Web applications
      1. Crowdsourcing

Recommendations

Changes in verbal and nonverbal conversational behavior in long-term interaction
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

We present an empirical investigation of conversational behavior in dyadic interaction spanning multiple conversations, in the context of a developing interpersonal relationship between a health counselor and her clients. Using a longitudinal video ...
The reliability of non-verbal cues for situated reference resolution and their interplay with language: implications for human robot interaction
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

When uttering referring expressions in situated task descriptions, humans naturally use verbal and non-verbal channels to transmit information to their interlocutor. To develop mechanisms for robot architectures capable of resolving object references ...
Comparison of Human-Human and Human-Robot Turn-Taking Behaviour in Multiparty Situated Interaction
UM3I '14: Proceedings of the 2014 workshop on Understanding and Modeling Multiparty, Multimodal Interactions

In this paper, we present an experiment where two human subjects are given a team-building task to solve together with a robot. The setting requires that the speakers' attention is partly directed towards objects on the table between them, as well as to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
360
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Oppenlaender JAbbas TGadiraju U(2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3641023
Ravishankar JDoering MKanda TGrollman DBroadbent EJu WSoh HWilliams T(2024)Zero-Shot Learning to Enable Error Awareness in Data-Driven HRIProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634940(592-601)Online publication date: 11-Mar-2024
https://dl.acm.org/doi/10.1145/3610977.3634940
van Waveren SPek CTumova JLeite ISakamoto DWeiss AHiatt LShiomi M(2022)Correct Me If I'm Wrong: Using Non-Experts to Repair Reinforcement Learning PoliciesProceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction10.5555/3523760.3523825(493-501)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.5555/3523760.3523825
Liu LSchoen AHenrichs CLi JMutlu BZhang YRadwin R(2022)Human Robot Collaboration for Enhancing Work ActivitiesHuman Factors: The Journal of the Human Factors and Ergonomics Society10.1177/0018720822107772266:1(158-179)Online publication date: 28-Mar-2022
https://doi.org/10.1177/00187208221077722
van Waveren SPek CTumova JLeite I(2022)Correct Me If I'm Wrong: Using Non-Experts to Repair Reinforcement Learning Policies2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)10.1109/HRI53351.2022.9889604(493-501)Online publication date: 7-Mar-2022
https://doi.org/10.1109/HRI53351.2022.9889604
Gvirsman OKoren YNorman TGordon GBelpaeme TYoung JGunes HRiek L(2020)PatriccProceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3319502.3374792(399-407)Online publication date: 9-Mar-2020
https://dl.acm.org/doi/10.1145/3319502.3374792
Feng DCarstensdottir ESeif El-Nasr MMarsella SElkind EVeloso MAgmon NTaylor M(2019)Exploring Improvisational Approaches to Social Knowledge AcquisitionProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems10.5555/3306127.3331804(1060-1068)Online publication date: 8-May-2019
https://dl.acm.org/doi/10.5555/3306127.3331804
Jonell PFallgren PDoğan FLopes JWennberg USkantze GCowan BClark L(2019)Crowdsourcing a self-evolving dialog graphProceedings of the 1st International Conference on Conversational User Interfaces10.1145/3342775.3342790(1-8)Online publication date: 22-Aug-2019
https://dl.acm.org/doi/10.1145/3342775.3342790
Doering MLiu PGlas DKanda TKulić DIshiguro H(2019)Curiosity Did Not Kill the RobotACM Transactions on Human-Robot Interaction10.1145/33264628:3(1-24)Online publication date: 23-Jul-2019
https://dl.acm.org/doi/10.1145/3326462
Robb DLopes JPadilla SLaskov AChiyah Garcia FLiu XScharff Willners JValeyrie NLohan KLane DPatron PPetillot YChantler MHastie HHarrison SBardzell SNeustaedter CTatar D(2019)Exploring Interaction with Remote Autonomous Systems using Conversational AgentsProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322318(1543-1556)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322318
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents