Cited By
View all- Stegner LHwang YPorfirio DMutlu B(2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
Goal-Oriented End-User Programming of Robots
Pages 582 - 591
Abstract
End-user programming (EUP) tools must balance user control with the robot's ability to plan and act autonomously. Many existing task-oriented EUP tools enforce a specific level of control, e.g., by requiring that users hand-craft detailed sequences of actions, rather than offering users the flexibility to choose the level of task detail they wish to express. We thereby created a novel EUP system, Polaris, that in contrast to most existing EUP tools, uses goal predicates as the fundamental building block of programs. Users can thereby express high-level robot objectives or lower-level checkpoints at their choosing, while an off-the-shelf task planner fills in any remaining program detail. To ensure that goal-specified programs adhere to user expectations of robot behavior, Polaris is equipped with a Plan Visualizer that exposes the planner's output to the user before runtime. In what follows, we describe our design of Polaris and its evaluation with 32 human participants. Our results support the Plan Visualizer's ability to help users craft higher-quality programs. Furthermore, there are strong associations between user perception of the robot and Plan Visualizer usage, and evidence that robot familiarity has a key role in shaping user experience.
Supplemental Material
MP4 File
Supplemental video
- Download
- 26.59 MB
ZIP File
SUPPLEMENTARY MATERIAL DESCRIPTION
- Download
- 512.96 KB
References
[1]
Angeline Aguinaldo, Jacob Bunker, Blake Pollard, Ankit Shukla, Arquimedes Canedo, Gustavo Quiros, and William Regli. 2022. RoboCat: A Category Theoretic Framework for Robotic Interoperability Using Goal-Oriented Programming. IEEE Transactions on Automation Science and Engineering, Vol. 19, 3 (2022), 2637--2645. https://doi.org/10.1109/TASE.2021.3094055
[2]
Gopika Ajaykumar, Maureen Steele, and Chien-Ming Huang. 2021. A Survey on End-User Robot Programming. Comput. Surveys, Vol. 54, 8, Article 164 (oct 2021), bibinfonumpages36 pages. https://doi.org/10.1145/3466819
[3]
Sonya Alexandrova, Zachary Tatlock, and Maya Cakmak. 2015. RoboFlow: A Flow-based Visual Programming Language for Mobile Manipulation Tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA '15'). IEEE, New York, NY, USA, 5537--5544. https://doi.org/10.1109/ICRA.2015.7139973
[4]
Samir Alili, Rachid Alami, and Vincent Montreuil. 2009. A Task Planner for an Autonomous Social Robot. Springer Berlin Heidelberg, Berlin, Heidelberg, 335--344. https://doi.org/10.1007/978--3--642-00644--9_30
[5]
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300233
[6]
Christel Baier and Joost-Pieter Katoen. 2008. Principles of Model Checking. MIT press, Cambridge, MA, USA.
[7]
Barbara Rita Barricelli, Fabio Cassano, Daniela Fogli, and Antonio Piccinno. 2019. End-user development, end-user programming and end-user software engineering: A systematic mapping study. Journal of Systems and Software, Vol. 149 (2019), 101--137. https://doi.org/10.1016/j.jss.2018.11.041
[8]
Sara Beschi, Daniela Fogli, and Fabio Tampalini. 2019. CAPIRCI: A Multi-modal System for Collaborative Robot Programming. In End-User Development (IS-EUD '19), Alessio Malizia, Stefano Valtolina, Anders Morch, Alan Serrano, and Andrew Stratton (Eds.). Springer International Publishing, Cham, 51--66. https://doi.org/10.1007/978--3-030--24781--2_4
[9]
David Brageul, Slobodan Vukanovic, and Bruce A. MacDonald. 2008. An intuitive interface for a cognitive programming by demonstration system. In 2008 IEEE International Conference on Robotics and Automation. IEEE, New York, NY, USA, 3570--3575. https://doi.org/10.1109/ROBOT.2008.4543757
[10]
Michael Bratman. 1987. Intention, Plans, and Practical Reason. Harvard University Press, Cambridge, MA, USA.
[11]
John Brooke. 1996. SUS-A `quick and dirty' usability scale. Usability Evaluation in Industry, Vol. 189, 194 (1996), 4--7. https://doi.org/10.1201/9781498710411
[12]
Yuanzhi Cao, Tianyi Wang, Xun Qian, Pawan S. Rao, Manav Wadhawan, Ke Huo, and Karthik Ramani. 2019a. GhostAR: A Time-Space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 521--534. https://doi.org/10.1145/3332165.3347902
[13]
Yuanzhi Cao, Zhuangying Xu, Fan Li, Wentao Zhong, Ke Huo, and Karthik Ramani. 2019b. V.Ra: An In-Situ Visual Authoring System for Robot-IoT Task Planning with Augmented Reality. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA) (DIS '19). Association for Computing Machinery, New York, NY, USA, 1059--1070. https://doi.org/10.1145/3322276.3322278
[14]
Colleen M. Carpinella, Alisa B. Wyman, Michael A. Perez, and Steven J. Stroessner. 2017. The Robotic Social Attributes Scale (RoSAS): Development and Validation. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (Vienna, Austria) (HRI '17). Association for Computing Machinery, New York, NY, USA, 254--262. https://doi.org/10.1145/2909824.3020208
[15]
Tathagata Chakraborti, Sarath Sreedharan, Sachin Grover, and Subbarao Kambhampati. 2019. Plan Explanations as Model Reconciliation -- An Empirical Study. In Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction (Daegu, Republic of Korea) (HRI '19). IEEE Press, New York, NY, USA, 258--266. https://doi.org/10.1109/HRI.2019.8673193
[16]
Michael Jae-Yoon Chung, Justin Huang, Leila Takayama, Tessa Lau, and Maya Cakmak. 2016. Iterative Design of a System for Programming Socially Interactive Service Robots. In Proceedings of Social Robotics: 8th International Conference (ICSR 2016), Arvin Agah, John-John Cabibihan, Ayanna M. Howard, Miguel A. Salichs, and Hongsheng He (Eds.). Springer International Publishing, Cham, 919--929. https://doi.org/10.1007/978--3--319--47437--3_90
[17]
Michael T. Cox and Chen Zhang. 2007. Mixed-Initiative Goal Manipulation. AI Magazine, Vol. 28, 2 (Jun. 2007), 62. https://doi.org/10.1609/aimag.v28i2.2040
[18]
Giuseppe De Giacomo, Alfonso Emilio Gerevini, Fabio Patrizi, Alessandro Saetti, and Sebastian Sardina. 2016. Agent planning programs. Artificial Intelligence, Vol. 231 (2016), 64--106. https://doi.org/10.1016/j.artint.2015.10.001
[19]
Emanuele De Pellegrin and Ronald P.A. Petrick. 2021. Automated Planning and Robotics Simulation with PDSim. In Proceedings of the ICAPS Workshop on Knowledge Engineering for Planning and Scheduling (KEPS).
[20]
Maxwell Forbes, Rajesh P. N. Rao, Luke Zettlemoyer, and Maya Cakmak. 2015. Robot Programming by Demonstration with Situated Spatial Language Understanding. In 2015 IEEE International Conference on Robotics and Automation (ICRA '15). IEEE, New York, NY, USA, 2014--2020. https://doi.org/10.1109/ICRA.2015.7139462
[21]
Maria Fox and Derek Long. 2003. PDDL2. 1: An Extension to PDDL for Expressing Temporal Planning Domains. Journal of Artificial Intelligence Research, Vol. 20 (2003), 61--124. https://doi.org/10.1613/jair.1129
[22]
Malik Ghallab, Dana Nau, and Paolo Traverso. 2016. Automated Planning and Acting. Cambridge University Press, Cambridge, England.
[23]
Javi F. Gorostiza and Miguel A. Salichs. 2011. End-user programming of a social robot by dialog. Robotics and Autonomous Systems, Vol. 59, 12 (2011), 1102--1114. https://doi.org/10.1016/j.robot.2011.07.009
[24]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. Elsevier B.V., Amsterdam, Netherlands, 139--183. https://doi.org/10.1016/S0166--4115(08)62386--9
[25]
Malte Helmert. 2006. The Fast Downward Planning System. Journal of Artificial Intelligence Research, Vol. 26, 1 (2006), 191--246. https://doi.org/10.1613/jair.1705
[26]
Malte Helmert. 2009. Concise Finite-Domain Representations for PDDL Planning Tasks. Artif. Intell., Vol. 173, 5--6 (apr 2009), 503--535. https://doi.org/10.1016/j.artint.2008.10.013
[27]
Minjie Hu, Michael Winikoff, and Stephen Cranefield. 2012. Teaching Novice Programming Using Goals and Plans in a Visual Notation. In Proceedings of the Fourteenth Australasian Computing Education Conference - Volume 123 (Melbourne, Australia) (ACE '12). Australian Computer Society, Inc., AUS, 43--52. https://dl.acm.org/doi/abs/10.5555/2483716.2483722
[28]
Justin Huang and Maya Cakmak. 2017. Code3: A System for End-to-End Programming of Mobile Manipulator Robots for Novices and Experts. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (Vienna, Austria) (HRI '17). Association for Computing Machinery, New York, NY, USA, 453--462. https://doi.org/10.1145/2909824.3020215
[29]
Justin Huang, Tessa Lau, and Maya Cakmak. 2016. Design and Evaluation of a Rapid Programming System for Service Robots. In The 11th ACM/IEEE International Conference on Human Robot Interaction (Christchurch, New Zealand) (HRI '16). IEEE Press, New York, NY, USA, 295--302. https://doi.org/10.1109/HRI.2016.7451765
[30]
Silvia Izquierdo-Badiola, Gerard Canal, Carlos Rizzo, and Guillem Alenyà. 2022. Improved Task Planning through Failure Anticipation in Human-Robot Collaboration. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, New York, NY, USA, 7875--7880. https://doi.org/10.1109/ICRA46639.2022.9812236
[31]
Charles C. Kemp, Aaron Edsinger, Henry M. Clever, and Blaine Matulevich. 2022. The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments. In 2022 International Conference on Robotics and Automation (ICRA). IEEE, New York, NY, USA, 3150--3157. https://doi.org/10.1109/ICRA46639.2022.9811922
[32]
Matthias Kovatsch, Yassin N. Hassan, and Simon Mayer. 2015. Practical semantics for the Internet of Things: Physical states, device mashups, and open questions. In 2015 5th International Conference on the Internet of Things (IOT). IEEE, New York, NY, USA, 54--61. https://doi.org/10.1109/IOT.2015.7356548
[33]
Alyssa Kubota, Emma I. C. Peterson, Vaishali Rajendren, Hadas Kress-Gazit, and Laurel D. Riek. 2020. JESSIE: Synthesizing Social Robot Behaviors for Personalized Neurorehabilitation and Beyond. In Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction (Cambridge, United Kingdom) (HRI '20). Association for Computing Machinery, New York, NY, USA, 121--130. https://doi.org/10.1145/3319502.3374836
[34]
Nicola Leonardi, Marco Manca, Fabio Paternò, and Carmen Santoro. 2019. Trigger-Action Programming for Personalising Humanoid Robot Behaviour. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3290605.3300675
[35]
Ying Siu Liang, Damien Pellier, Humbert Fiorino, and Sylvie Pesty. 2019. End-User Programming of Low-and High-Level Actions for Robotic Task Planning. In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, New York, NY, USA, 1--8. https://doi.org/10.1109/RO-MAN46459.2019.8956327
[36]
Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler. 2019. Synthesizing Environment-Aware Activities via Activity Sketches. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 6284--6292. https://doi.org/10.1109/CVPR.2019.00645
[37]
Arnold M Lund. 2001. Measuring Usability with the USE Questionnaire. Usability and User Experience Newsletter of the STC Usability SIG, Vol. 8, 2 (2001), 3--6.
[38]
Simon Mayer, Ruben Verborgh, Matthias Kovatsch, and Friedemann Mattern. 2016. Smart Configuration of Smart Environments. IEEE Transactions on Automation Science and Engineering, Vol. 13, 3 (2016), 1247--1255. https://doi.org/10.1109/TASE.2016.2533321
[39]
Mahda Noura, Sebastian Heil, and Martin Gaedke. 2018. GrOWTH: Goal-Oriented End User Development for Web of Things Devices. In Web Engineering, Tommi Mikkonen, Ralf Klamma, and Juan Hernández (Eds.). Springer International Publishing, Cham, 358--365. https://doi.org/10.1007/978--3--319--91662-0_29
[40]
Ronald P. A. Petrick and Mary Ellen Foster. 2013. Planning for Social Interaction in a Robot Bartender Domain. In Proceedings of the Twenty-Third International Conference on International Conference on Automated Planning and Scheduling (Rome, Italy) (ICAPS'13). AAAI Press, Washington, DC, USA, 389--397. https://doi.org/10.1609/icaps.v23i1.13589
[41]
David Porfirio, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2018. Authoring and Verifying Human-Robot Interactions. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST '18). Association for Computing Machinery, New York, NY, USA, 75--86. https://doi.org/10.1145/3242587.3242634
[42]
David Porfirio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2023. Sketching Robot Programs On the Fly. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (Stockholm, Sweden) (HRI '23). Association for Computing Machinery, New York, NY, USA, 584--593. https://doi.org/10.1145/3568162.3576991
[43]
David J. Porfirio, Laura Stegner, Maya Cakmak, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2021. Figaro: A Tabletop Authoring Environment for Human-Robot Interaction. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, 1--15. https://doi.org/10.1145/3411764.3446864
[44]
Emmanuel Pot, Jérôme Monceaux, Rodolphe Gelin, and Bruno Maisonnier. 2009. Choregraphe: a Graphical Tool for Humanoid Robot Programming. In The 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN '09). IEEE, New York, NY, USA, 46--51. https://doi.org/10.1109/ROMAN.2009.5326209
[45]
Morgan Quigley, Ken Conley, Brian Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Eric Berger, Rob Wheeler, and Andrew Y Ng. 2009. ROS: an open-source Robot Operating System. In ICRA Workshop on Open Source Software.
[46]
Anand S. Rao. 1996. AgentSpeak(L): BDI Agents speak out in a logical computable language. In Agents Breaking Away: 7th European Workshop on Modelling Autonomous Agents in a Multi-Agent World, Walter Van de Velde and John W. Perram (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 42--55. https://doi.org/10.1007/BFb0031845
[47]
Sebastian Sardina and Lin Padgham. 2011. A BDI Agent Programming Language with Failure Handling, Declarative Goals, and Planning. Autonomous Agents and Multi-Agent Systems, Vol. 23, 1 (jul 2011), 18--70. https://doi.org/10.1007/s10458-010--9130--9
[48]
Allison Sauppé and Bilge Mutlu. 2014. Design Patterns for Exploring and Prototyping Human-Robot Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 1439--1448. https://doi.org/10.1145/2556288.2557057
[49]
Andrew Schoen, Curt Henrichs, Mathias Strohkirch, and Bilge Mutlu. 2020. Authr: A Task Authoring Environment for Human-Robot Teams. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20). Association for Computing Machinery, New York, NY, USA, 1194--1208. https://doi.org/10.1145/3379337.3415872
[50]
Andrew Schoen, Nathan White, Curt Henrichs, Amanda Siebert-Evenstone, David Shaffer, and Bilge Mutlu. 2022. CoFrame: A System for Training Novice Cobot Programmers. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (Sapporo, Hokkaido, Japan) (HRI '22). IEEE Press, New York, NY, USA, 185--194. https://doi.org/10.1109/HRI53351.2022.9889345
[51]
Vikas Shivashankar, Ugur Kuter, Dana Nau, and Ron Alford. 2012. A Hierarchical Goal-Based Formalism and Algorithm for Single-Agent Planning. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 2 (Valencia, Spain) (AAMAS '12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 981--988. https://dl.acm.org/doi/abs/10.5555/2343776.2343837
[52]
David E Smith, Jeremy Frank, and William Cushing. 2008. The ANML Language. In The ICAPS-08 Workshop on Knowledge Engineering for Planning and Scheduling (KEPS).
[53]
Laura Stegner and Bilge Mutlu. 2022. Designing for Caregiving: Integrating Robotic Assistance in Senior Living Communities. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (Virtual Event, Australia) (DIS '22). Association for Computing Machinery, New York, NY, USA, 1934--1947. https://doi.org/10.1145/3532106.3533536
[54]
Unity Technologies. 2023. Unity Real-Time Development Platform. https://unity.com/.
[55]
Karthik Valmeekam, Sarath Sreedharan, Sailik Sengupta, and Subbarao Kambhampati. 2022. RADAR-X: An Interactive Mixed Initiative Planning Interface Pairing Contrastive Explanations and Revised Plan Suggestions. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 32. AAAI Press, Washington, DC, USA, 508--517. https://doi.org/10.1609/icaps.v32i1.19837 io
Index Terms
- Goal-Oriented End-User Programming of Robots
Recommendations
A Survey on End-User Robot Programming
As robots interact with a broader range of end-users, end-user robot programming has helped democratize robot programming by empowering end-users who may not have experience in robot programming to customize robots to meet their individual contextual ...
Assisted End-User Robot Programming
ICMI '21: Proceedings of the 2021 International Conference on Multimodal InteractionEnd-user programming methods that enable users who may not have programming or robotics experience to program robots have helped lower the barrier of entry for everyday people to use robots for their needs. However, popular end-user robot programming ...
Participatory Design and End-User Programming for Human-Robot Interaction
HRI '22: Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot InteractionThe Participatory Design and End-User Programming for Human-Robot Interaction (HRI) workshop aims to advance research on how to design systems that can be used by end users to program robots. There tends to be a fracture in HRI between the technical ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
982 pages
ISBN:9798400703225
DOI:10.1145/3610977
- General Chairs:
- Dan Grollman,
- Elizabeth Broadbent,
- Program Chairs:
- Wendy Ju,
- Harold Soh,
- Tom Williams
Copyright © 2024 ACM.
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
HRI '24: ACM/IEEE International Conference on Human-Robot Interaction
March 11 - 15, 2024
CO, Boulder, USA
Acceptance Rates
Overall Acceptance Rate 268 of 1,124 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 189Total Downloads
- Downloads (Last 12 months)189
- Downloads (Last 6 weeks)24
Reflects downloads up to 19 Nov 2024
Other Metrics
Citations
Cited By
View all- Stegner LHwang YPorfirio DMutlu B(2024)Understanding On-the-Fly End-User Robot ProgrammingProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660721(2468-2480)Online publication date: 1-Jul-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in