Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3568162.3576991acmconferencesArticle/Chapter ViewAbstractPublication PageshriConference Proceedingsconference-collections
research-article
Public Access

Sketching Robot Programs On the Fly

Published: 13 March 2023 Publication History

Abstract

Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. There is a need for robot programming solutions that reconcile the complexity of robotics with the on-the-fly goals of end-user development. In response to this need, we present a novel, multimodal, and on-the-fly development system, Tabula. Inspired by a formative design study with a prototype, Tabula leverages a combination of spoken language for specifying the core of a robot task and sketching for contextualizing the core. The result is that developers can script partial, sloppy versions of robot programs to be completed and refined by a program synthesizer. Lastly, we demonstrate our anticipated use cases of Tabula via a set of application scenarios.

Supplementary Material

MP4 File (hrifp1265.mp4)
Supplemental video
MP4 File (HRI21_fp1265.mp4)
This work presents Tabula, a novel, multimodal end-user development system for programming human-robot interactions. Tabula is motivated by the need for efficient, convenient, and minimally instrumented robot programming. In the pursuit of minimal instrumentation, Tabula's initial vision was to only allow verbal input. Feedback from a design study led Tabula to instead be designed as a multimodal application, in which end users sketch a rough path of the robot on a handheld tablet to contextualize a small set of verbal hints. Tabula then employs program synthesis to combine sketching and verbal input and produce an executable robot program. Tabula's capabilities were demonstrated using a set of test cases that covers multiple different ways in which Tabula can be used.

References

[1]
AirTable. 2022. Airtable | Everyone's app platform. https://airtable.com/.
[2]
Samir Alili, Rachid Alami, and Vincent Montreuil. 2009. A Task Planner for an Autonomous Social Robot. In Distributed Autonomous Robotic Systems 8, Hajime Asama, Haruhisa Kurokawa, Jun Ota, and Kosuke Sekiyama (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 335--344. https://doi.org/10.1007/978-3-642-00644-9_30
[3]
Ron Alterovitz, Sven Koenig, and Maxim Likhachev. 2016. Robot Planning in the Real World: Research Challenges and Opportunities. AI Magazine 37, 2 (Jul. 2016), 76--84. https://doi.org/10.1609/aimag.v37i2.2651
[4]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300233
[5]
Federico Boniardi, Abhinav Valada, Wolfram Burgard, and Gian Diego Tipaldi. 2016. Autonomous indoor robot navigation using a sketch interface for drawing maps and routes. In 2016 IEEE International Conference on Robotics and Automation (ICRA). 2896--2901. https://doi.org/10.1109/ICRA.2016.7487453
[6]
John Brooke et al. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.
[7]
Margaret Burnett, Curtis Cook, and Gregg Rothermel. 2004. End-User Software Engineering. Commun. ACM 47, 9 (sep 2004), 53--58. https://doi.org/10.1145/ 1015864.1015889
[8]
Yuanzhi Cao, Zhuangying Xu, Fan Li, Wentao Zhong, Ke Huo, and Karthik Ramani. 2019. V.Ra: An In-Situ Visual Authoring System for Robot-IoT Task Planning with Augmented Reality. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS '19). Association for Computing Machinery, New York, NY, USA, 1059--1070. https://doi.org/10.1145/3322276. 3322278
[9]
David Chen and Raymond Mooney. 2011. Learning to Interpret Natural Language Navigation Instructions from Observations. Proceedings of the AAAI Conference on Artifcial Intelligence 25, 1 (Aug. 2011), 859--865. https://doi.org/10.1609/aaai. v25i1.7974
[10]
Michael Jae-Yoon Chung and Maya Cakmak. 2022. Authoring Human Simulators via Probabilistic Functional Reactive Program Synthesis. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI). 727--730. https: //doi.org/10.1109/HRI53351.2022.9889630
[11]
Jonathan Connell. 2019. Verbal Programming of Robot Behavior. arXiv preprint arXiv:1911.09782 (2019).
[12]
Andrew Correa, Matthew R. Walter, Luke Fletcher, Jim Glass, Seth Teller, and Randall Davis. 2010. Multimodal interaction with an autonomous forklift. In 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI). 243--250. https://doi.org/10.1109/HRI.2010.5453188
[13]
Li Dong and Mirella Lapata. 2016. Language to Logical Form with Neural Attention. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - Long Papers, Vol. 1. Association for Computational Linguistics (ACL), 33--43. https://doi.org/10.18653/v1/P16-1004 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016; Conference date: 07-08-2016 Through 12-08-2016.
[14]
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. The MIT Press. https://doi.org/10.7551/mitpress/7287.001.0001
[15]
Maxwell Forbes, Rajesh P. N. Rao, Luke Zettlemoyer, and Maya Cakmak. 2015. Robot Programming by Demonstration with situated spatial language understanding. In 2015 IEEE International Conference on Robotics and Automation (ICRA). 2014--2020. https://doi.org/10.1109/ICRA.2015.7139462
[16]
Yuxiang Gao and Chien-Ming Huang. 2019. PATI: A Projection-Based Augmented Table-Top Interface for Robot Programming. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI '19). Association for Computing Machinery, New York, NY, USA, 345--355. https://doi.org/10.1145/3301275.3302326
[17]
Malik Ghallab, Dana Nau, and Paolo Traverso. 2016. Automated Planning and Acting. Cambridge University Press. https://doi.org/10.1017/CBO9781139583923
[18]
Javi F. Gorostiza and Miguel A. Salichs. 2011. End-User Programming of a Social Robot by Dialog. Robot. Auton. Syst. 59, 12 (dec 2011), 1102--1114. https: //doi.org/10.1016/j.robot.2011.07.009
[19]
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends® in Programming Languages 4, 1--2 (2017), 1--119. https: //doi.org/10.1561/2500000010
[20]
Justin Huang, Dieter Fox, and Maya Cakmak. 2019. Synthesizing Robot Manipulation Programs from a Single Observed Human Demonstration. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 4585--4592. https://doi.org/10.1109/IROS40897.2019.8968543
[21]
Justin Huang, Tessa Lau, and Maya Cakmak. 2016. Design and evaluation of a rapid programming system for service robots. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI). 295--302. https://doi.org/10.1109/ HRI.2016.7451765
[22]
Karin Kipper, Hoa Trang Dang, Martha Palmer, et al. 2000. Class-based construc- tion of a verb lexicon. AAAI/IAAI 691 (2000), 696.
[23]
James R. Kirk and John E. Laird. 2019. Learning Hierarchical Symbolic Repre- sentations to Support Interactive Task Learning and Knowledge Transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artifcial Intelligence, IJCAI-19. International Joint Conferences on Artifcial Intelligence Organization, 6095--6102. https://doi.org/10.24963/ijcai.2019/844
[24]
Hadas Kress-Gazit, Georgios E. Fainekos, and George J. Pappas. 2008. Translating Structured English to Robot Controllers. Advanced Robotics 22, 12 (2008), 1343--1359. https://doi.org/10.1163/156855308X344864 arXiv:https://doi.org/10.1163/156855308X344864
[25]
Stanislao Lauria, Guido Bugmann, Theocharis Kyriacou, and Ewan Klein. 2002. Mobile robot programming using natural language. Robotics and Autonomous Systems 38, 3 (2002), 171--181. https://doi.org/10.1016/S0921--8890(02)00166--5 Advances in Robot Skill Learning.
[26]
Nicola Leonardi, Marco Manca, Fabio Paternò, and Carmen Santoro. 2019. Trigger- Action Programming for Personalising Humanoid Robot Behaviour. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300675
[27]
Henry Lieberman, Fabio Paternò, Markus Klann, and Volker Wulf. 2006. End-User Development: An Emerging Paradigm. Springer Netherlands, Dordrecht, 1--8. https://doi.org/10.1007/1-4020-5386-X_1
[28]
Greg Little, Robert C. Miller, Victoria H. Chou, Michael Bernstein, Tessa Lau, and Allen Cypher. 2010. Sloppy Programming. In No Code Required, Allen Cypher, Mira Dontcheva, Tessa Lau, and Jefrey Nichols (Eds.). Morgan Kaufmann, Boston, 289--307. https://doi.org/10.1016/B978-0-12-381541-5.00015-8
[29]
Kexi Liu, Daisuke Sakamoto, Masahiko Inami, and Takeo Igarashi. 2011. Roboshop: Multi-Layered Sketching Interface for Robot Housework Assignment and Management. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). Association for Computing Machinery, New York, NY, USA, 647--656. https://doi.org/10.1145/1978942.1979035
[30]
Arnold M Lund. 2001. Measuring Usability with the USE Questionnaire. Usability and User Experience Newsletter of the STC Usability SIG 8, 2 (01 2001), 3--6.
[31]
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. 55--60. http://www.aclweb.org/anthology/P/P14/P14-5010
[32]
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to Parse Natural Language Commands to a Robot Control System. Springer International Publishing, Heidelberg, 403--415. https://doi.org/10.1007/978-3-319-00065-7_28
[33]
George A. Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM 38, 11 (nov 1995), 39--41. https://doi.org/10.1145/219717.219748
[34]
Shiwali Mohan and John Laird. 2014. Learning Goal-Oriented Hierarchical Tasks from Situated Interactive Instruction. Proceedings of the AAAI Conference on Artifcial Intelligence 28, 1 (Jun. 2014). https://doi.org/10.1609/aaai.v28i1.8756
[35]
Ronald Petrick and Mary Ellen Foster. 2013. Planning for Social Interaction in a Robot Bartender Domain. Proceedings of the International Conference on Automated Planning and Scheduling 23, 1 (Jun. 2013), 389--397. https://doi.org/ 10.1609/icaps.v23i1.13589
[36]
David Porfrio, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2018. Authoring and Verifying Human-Robot Interactions. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST '18). Association for Computing Machinery, New York, NY, USA, 75--86. https://doi.org/10.1145/3242587.3242634
[37]
David J Porfrio. 2022. Authoring Social Interactions Between Humans and Robots. Ph. D. Dissertation. UW-Madison.
[38]
David J. Porfrio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2021. Figaro: A Tabletop Authoring Environment for Human-Robot Interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 414, 15 pages. https://doi.org/10.1145/3411764.3446864
[39]
Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and Yasmin Kafai. 2009. Scratch: Programming for All. Commun. ACM 52, 11 (nov 2009), 60--67. https://doi.org/10.1145/1592761.1592779
[40]
Daisuke Sakamoto, Koichiro Honda, Masahiko Inami, and Takeo Igarashi. 2009. Sketch and Run: A Stroke-Based Interface for Home Robots. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI '09). Association for Computing Machinery, New York, NY, USA, 197--200. https://doi.org/10.1145/1518701.1518733
[41]
Emmanuel Senft, Michael Hagenow, Robert Radwin, Michael Zinn, Michael Gleicher, and Bilge Mutlu. 2021. Situated Live Programming for Human-Robot Collaboration. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 613--625. https://doi.org/10.1145/3472749.3474773
[42]
Danelle Shah. 2012. Towards Natural And Robust Human-Robot Interaction Using Sketch And Speech. Ph. D. Dissertation. Cornell.
[43]
Danelle Shah, Joseph Schneider, and Mark Campbell. 2010. A robust sketch interface for natural robot control. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems. 4458--4463. https://doi.org/10.1109/IROS.2010. 5649345
[44]
Maj Stenmark and Pierre Nugues. 2013. Natural language programming of industrial robots. In IEEE ISR 2013. 1--5. https://doi.org/10.1109/ISR.2013.6695630
[45]
Seth Teller, Matthew R. Walter, Matthew Antone, Andrew Correa, Randall Davis, Luke Fletcher, Emilio Frazzoli, Jim Glass, Jonathan P. How, Albert S. Huang, Jeong hwan Jeon, Sertac Karaman, Brandon Luders, Nicholas Roy, and Tara Sainath. 2010. A voice-commandable robotic forklift working alongside humans in minimally-prepared outdoor environments. In 2010 IEEE International Conference on Robotics and Automation. 526--533. https://doi.org/10.1109/ROBOT.2010. 5509238
[46]
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. 2011. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. Proceedings of the AAAI Conference on Artifcial Intelligence 25, 1 (Aug. 2011), 1507--1514. https://doi.org/10.1609/aaai.v25i1.7979
[47]
Jesse Thomason, Aishwarya Padmakumar, Jivko Sinapov, Nick Walker, Yuqian Jiang, Harel Yedidsion, Justin Hart, Peter Stone, and Raymond J. Mooney. 2019. Improving Grounded Natural Language Understanding through Human-Robot Dialog. In 2019 International Conference on Robotics and Automation (ICRA). 6934--6941. https://doi.org/10.1109/ICRA.2019.8794287
[48]
Blase Ur, Melwyn Pak Yong Ho, Stephen Brawner, Jiyun Lee, Sarah Mennicken, Noah Picard, Diane Schulze, and Michael L. Littman. 2016. Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT Recipes. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 3227--3231. https://doi.org/10.1145/2858036.2858556
[49]
Nick Walker, Yu-Tang Peng, and Maya Cakmak. 2019. Neural Semantic Parsing with Anonymization for Command Understanding in General-Purpose Service David Porfirio et al. Robots. In RoboCup 2019: Robot World Cup XXIII, Stephan Chalup, Tim Niemueller, Jackrit Suthakorn, and Mary-Anne Williams (Eds.). Springer International Publishing, Cham, 337--350. https://doi.org/10.1007/978-3-030-35699-6_26
[50]
Webfow. 2022. Create a custom website: No-code website builder. https: //webfow.com/.
[51]
W. A. Woods. 1973. Progress in Natural Language Understanding: An Application to Lunar Geology. In Proceedings of the June 4-8, 1973, National Computer Conference and Exposition (New York, New York) (AFIPS '73). Association for Computing Machinery, New York, NY, USA, 441--450. https://doi.org/10.1145/ 1499586.1499695
[52]
Marcelo Worsley, David Barel, Lydia Davison, Thomas Large, and Timothy Mwiti. 2018. Multimodal Interfaces for Inclusive Learning. In Artifcial Intelligence in Education, Carolyn Penstein Rosé, Roberto Martínez-Maldonado, H. Ulrich Hoppe, Rose Luckin, Manolis Mavrikis, Kaska Porayska-Pomsta, Bruce McLaren, and Benedict du Boulay (Eds.). Springer International Publishing, Cham, 389--393. https://doi.org/10.1007/978-3-319-93846-2_73
[53]
James Young, Kentaro Ishii, Takeo Igarashi, and Ehud Sharlin. 2012. Style by Demonstration: Teaching Interactive Movement Style to Robots. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (Lisbon, Portugal) (IUI '12). Association for Computing Machinery, New York, NY, USA, 41--50. https://doi.org/10.1145/2166966.2166976
[54]
Zapier. 2022. Automation that moves you forward. https://zapier.com/.
[55]
Luke S. Zettlemoyer and Michael Collins. 2005. Learning to Map Sentences to Logical Form: Structured Classifcation with Probabilistic Categorial Grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artifcial Intelligence (Edinburgh, Scotland) (UAI'05). AUAI Press, Arlington, Virginia, USA, 658--666.
[56]
Lefan Zhang, Weijia He, Jesse Martinez, Noah Brackenbury, Shan Lu, and Blase Ur. 2019. AutoTap: Synthesizing and Repairing Trigger-Action Programs Using LTL Properties. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada) (ICSE '19). IEEE Press, 281--291. https: //doi.org/10.1109/ICSE.2019.00043

Cited By

View all
  • (2024)End-User Development for Human-Robot Interaction: Results and Trends in an Emerging FieldProceedings of the ACM on Human-Computer Interaction10.1145/36611468:EICS(1-40)Online publication date: 17-Jun-2024
  • (2024)REX: Designing User-centered Repair and Explanations to Address Robot FailuresProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661559(2911-2925)Online publication date: 1-Jul-2024
  • (2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences
HRI '23: Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction
March 2023
631 pages
ISBN:9781450399647
DOI:10.1145/3568162
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2023

Check for updates

Author Tags

  1. end-user development
  2. human-robot interaction
  3. sketching

Qualifiers

  • Research-article

Funding Sources

Conference

HRI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 268 of 1,124 submissions, 24%

Upcoming Conference

HRI '25
ACM/IEEE International Conference on Human-Robot Interaction
March 4 - 6, 2025
Melbourne , VIC , Australia

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)328
  • Downloads (Last 6 weeks)46
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)End-User Development for Human-Robot Interaction: Results and Trends in an Emerging FieldProceedings of the ACM on Human-Computer Interaction10.1145/36611468:EICS(1-40)Online publication date: 17-Jun-2024
  • (2024)REX: Designing User-centered Repair and Explanations to Address Robot FailuresProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661559(2911-2925)Online publication date: 1-Jul-2024
  • (2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
  • (2024)Programming-by-Demonstration for Long-Horizon Robot TasksProceedings of the ACM on Programming Languages10.1145/36328608:POPL(512-545)Online publication date: 5-Jan-2024
  • (2024)OpenVP: A Customizable Visual Programming Environment for Robotics ApplicationsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3637477(944-948)Online publication date: 11-Mar-2024
  • (2024)Generative Expressive Robot Behaviors using Large Language ModelsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634999(482-491)Online publication date: 11-Mar-2024
  • (2024)Goal-Oriented End-User Programming of RobotsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634974(582-591)Online publication date: 11-Mar-2024
  • (2024)Alchemist: LLM-Aided End-User Development of Robot ApplicationsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634969(361-370)Online publication date: 11-Mar-2024
  • (2024)Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00020(89-95)Online publication date: 2-Sep-2024

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media