research-article

Supercharging Trial-and-Error for Learning Complex Software Applications

Authors:

George Fitzmaurice,

Justin MatejkaAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 381, Pages 1 - 13

https://doi.org/10.1145/3491102.3501895

Published: 29 April 2022 Publication History

Abstract

Despite an abundance of carefully-crafted tutorials, trial-and-error remains many people’s preferred way to learn complex software. Yet, approaches to facilitate trial-and-error (such as tooltips) have evolved very little since the 1980s. While existing mechanisms work well for simple software, they scale poorly to large feature-rich applications. In this paper, we explore new techniques to support trial-and-error in complex applications. We identify key benefits and challenges of trial-and-error, and introduce a framework with a conceptual model and design space. Using this framework, we developed three techniques: ToolTrack to keep track of trial-and-error progress; ToolTrip to go beyond trial-and-error of single commands by highlighting related commands that are frequently used together; and ToolTaste to quickly and safely try commands. We demonstrate how these techniques facilitate trial-and-error, as illustrated through a proof-of-concept implementation in the CAD software Fusion 360. We conclude by discussing possible scenarios and outline directions for future research on trial-and-error.

Supplemental Material

MP4 File

Talk Video

Transcript for: Talk Video

MP4 File

Video Figure

Transcript for: Video Figure

MP4 File

Video Preview

Transcript for: Video Preview

References

[1]

Oscar D. Andrade, Nathaniel Bean, and David G. Novick. 2009. The Macro-Structure of Use of Help. In Proceedings of the 27th ACM International Conference on Design of Communication - SIGDOC ’09. ACM Press, Bloomington, Indiana, USA, 143. https://doi.org/10.1145/1621995.1622022

Digital Library

[2]

Autodesk. 2021. Autodesk Screencast. https://knowledge.autodesk.com/community/screencast Retrieved September 5, 2021.

[3]

Meera M. Blattner, Denise A. Sumikawa, and Robert M. Greenberg. 1989. Earcons and Icons: Their Structure and Common Design Principles (Abstract Only). SIGCHI Bull. 21, 1 (Aug. 1989), 123–124. https://doi.org/10.1145/67880.1046599

Digital Library

[4]

John M. Carroll and Caroline Carrithers. 1984. Training Wheels in a User Interface. Commun. ACM 27, 8 (Aug. 1984), 800–806. https://doi.org/10.1145/358198.358218

Digital Library

[5]

John M. Carroll and Mary Beth Rosson. 1987. Paradox of the Active User. In Interfacing Thought: Cognitive Aspects of Human-Computer Interaction. MIT Press, Cambridge, MA, USA, 80–111.

[6]

Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014. Supporting Novice to Expert Transitions in User Interfaces. ACM Comput. Surv. 47, 2, Article 31 (Nov. 2014), 36 pages. https://doi.org/10.1145/2659796

Digital Library

[7]

Sjaak de Mul and Herre van Oostendorp. 1996. Learning User Interfaces by Exploration. Acta Psychologica 91, 3 (April 1996), 325–344. https://doi.org/10.1016/0001-6918(95)00060-7

[8]

Stephen W. Draper and Stephen B. Barton. 1993. Learning by Exploration and Affordance Bugs. In INTERACT ’93 and CHI ’93 Conference Companion on Human Factors in Computing Systems(CHI ’93). Association for Computing Machinery, New York, NY, USA, 75–76. https://doi.org/10.1145/259964.260084

Digital Library

[9]

Leah Findlater and Joanna McGrenere. 2004. A Comparison of Static, Adaptive, and Adaptable Menus. In Proceedings of the 2004 Conference on Human Factors in Computing Systems - CHI ’04. ACM Press, Vienna, Austria, 89–96. https://doi.org/10.1145/985692.985704

Digital Library

[10]

Leah Findlater, Karyn Moffatt, Joanna McGrenere, and Jessica Dawson. 2009. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Boston MA USA, 1655–1664. https://doi.org/10.1145/1518701.1518956

Digital Library

[11]

C. Ailie Fraser, Mira Dontcheva, Holger Winnemöller, Sheryl Ehrlich, and Scott Klemmer. 2016. DiscoverySpace: Suggesting Actions in Complex Software. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, Brisbane QLD Australia, 1221–1232. https://doi.org/10.1145/2901790.2901849

Digital Library

[12]

C. Ailie Fraser, Julia M. Markel, N. James Basa, Mira Dontcheva, and Scott Klemmer. 2020. ReMap: Lowering the Barrier to Help-Seeking with Multimodal Search. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 979–986. https://doi.org/10.1145/3379337.3415592

Digital Library

[13]

Tovi Grossman and George Fitzmaurice. 2010. ToolClips: An Investigation of Contextual Video Assistance for Functionality Understanding. In Proceedings of the 28th International Conference on Human Factors in Computing Systems - CHI ’10. ACM Press, Atlanta, Georgia, USA, 1515. https://doi.org/10.1145/1753326.1753552

Digital Library

[14]

Tovi Grossman, George Fitzmaurice, and Ramtin Attar. 2009. A Survey of Software Learnability: Metrics, Methodologies and Guidelines. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 649–658. https://doi.org/10.1145/1518701.1518803

Digital Library

[15]

Pushkar N. Kaul. 1998. Drug discovery: Past, present and future. Birkhäuser Basel, Basel, 9–105. https://doi.org/10.1007/978-3-0348-8833-2_1

[16]

Caitlin Kelleher and Randy Pausch. 2005. Stencils-Based Tutorials: Design and Evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’05. ACM Press, Portland, Oregon, USA, 541. https://doi.org/10.1145/1054972.1055047

Digital Library

[17]

Kimia Kiani, George Cui, Andrea Bunt, Joanna McGrenere, and Parmit K. Chilana. 2019. Beyond ”One-Size-Fits-All”: Understanding the Diversity in How Software Newcomers Discover and Make Use of Help Resources. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland UK, 1–14. https://doi.org/10.1145/3290605.3300570

Digital Library

[18]

Benjamin Lafreniere, Andrea Bunt, and Michael Terry. 2014. Task-Centric Interfaces for Feature-Rich Software. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (Sydney, New South Wales, Australia) (OzCHI ’14). Association for Computing Machinery, New York, NY, USA, 49–58. https://doi.org/10.1145/2686612.2686620

Digital Library

[19]

Benjamin Lafreniere, Parmit K. Chilana, Adam Fourney, and Michael A. Terry. 2015. TheseAren’t theCommands You’reLooking For: Addressing False Feedforward in Feature-Rich Software. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, Charlotte NC USA, 619–628. https://doi.org/10.1145/2807442.2807482

Digital Library

[20]

Frank Linton, Andy Charron, and Debbie Joy. 1998. OWL: A Recommender System for Organization-Wide Learning. Technical Report WS-98-08. The AAAI Press, Menlo Park, California.

[21]

Frank Linton and Hans-Peter Schaefer. 2000. Recommender Systems for Learning: Building User and Expert Models through Long-Term Observation of Application Use. User Modeling and User-Adapted Interaction 10, 2 (June 2000), 181–208. https://doi.org/10.1023/A:1026521931194

Digital Library

[22]

Aran Lunzer. 1998. Towards The Subjunctive Interface: General Support For Parameter Exploration By Overlaying Alternative Application States. Late Breaking Hot Topics Proceedings of IEEE Visualization 98 (1998), 45–48.

[23]

Aran Lunzer and Kasper Hornbæk. 2008. Subjunctive Interfaces: Extending Applications to Support Parallel Setup, Viewing and Control of Alternative Scenarios. ACM Trans. Comput.-Hum. Interact. 14, 4, Article 17 (Jan. 2008), 44 pages. https://doi.org/10.1145/1314683.1314685

Digital Library

[24]

Shareen Mahmud, Jessalyn Alvina, Parmit K. Chilana, Andrea Bunt, and Joanna McGrenere. 2020. Learning Through Exploration: How Children, Adults, and Older Adults Interact with a New Feature-Rich Application. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu HI USA, 1–14. https://doi.org/10.1145/3313831.3376414

Digital Library

[25]

Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2011. Ambient Help. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 2751–2760. https://doi.org/10.1145/1978942.1979349

Digital Library

[26]

Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2013. Patina: Dynamic Heatmaps for Visualizing Application Usage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 3227–3236. https://doi.org/10.1145/2470654.2466442

Digital Library

[27]

Justin Matejka, Wei Li, Tovi Grossman, and George Fitzmaurice. 2009. CommunityCommands: Command Recommendations for Software Applications. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST ’09. ACM Press, Victoria, BC, Canada, 193. https://doi.org/10.1145/1622176.1622214

Digital Library

[28]

Mathieu Nancel and Andy Cockburn. 2014. Causality: A Conceptual Model of Interaction History. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Toronto Ontario Canada, 1777–1786. https://doi.org/10.1145/2556288.2556990

Digital Library

[29]

Don Norman. 2013. The Design of Everyday Things: Revised and Expanded Edition. Basic Books, New York, New York, USA.

Digital Library

[30]

David G. Novick, Oscar D. Andrade, and Nathaniel Bean. 2009. The Micro-Structure of Use of Help. In Proceedings of the 27th ACM International Conference on Design of Communication - SIGDOC ’09. ACM Press, Bloomington, Indiana, USA, 97. https://doi.org/10.1145/1621995.1622014

Digital Library

[31]

Peter Pirolli and Stuart Card. 1999. Information Foraging. Psychological Review 106 (10 1999), 643–675. https://doi.org/10.1037/0033-295X.106.4.643

[32]

Peter G. Polson and Clayton H. Lewis. 1990. Theory-Based Design for Easily Learned Interfaces. Human–Computer Interaction 5, 2-3 (June 1990), 191–220. https://doi.org/10.1080/07370024.1990.9667154

[33]

Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F. Cohen. 2011. Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology(Santa Barbara, California, USA) (UIST ’11). Association for Computing Machinery, New York, NY, USA, 135–144. https://doi.org/10.1145/2047196.2047213

Digital Library

[34]

Atul Prakash and Michael J. Knister. 1994. A Framework for Undoing Actions in Collaborative Systems. ACM Trans. Comput.-Hum. Interact. 1, 4 (Dec. 1994), 295–330. https://doi.org/10.1145/198425.198427

Digital Library

[35]

Atul Prakash and Michael J Knister. 1994. A framework for undoing actions in collaborative systems. ACM Transactions on Computer-Human Interaction (TOCHI) 1, 4(1994), 295–330.

Digital Library

[36]

John Rieman. 1996. A Field Study of Exploratory Learning Strategies. ACM Transactions on Computer-Human Interaction 3, 3 (Sept. 1996), 189–218. https://doi.org/10.1145/234526.234527

Digital Library

[37]

Andrew Sears and Ben Shneiderman. 1994. Split Menus: Effectively Using Selection Frequency to Organize Menus. ACM Transactions on Computer-Human Interaction 1, 1 (March 1994), 27–51. https://doi.org/10.1145/174630.174632

Digital Library

[38]

Shneiderman. 1983. Direct Manipulation: A Step Beyond Programming Languages. Computer 16, 8 (Aug. 1983), 57–69. https://doi.org/10.1109/MC.1983.1654471

Digital Library

[39]

Herbert A. Simon and Peter A. Simon. 1962. Trial and error search in solving difficult problems: Evidence from the game of chess. Behavioral Science 7, 4 (1962), 425–429. https://doi.org/10.1002/bs.3830070402 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/bs.3830070402

[40]

Michael Terry and Elizabeth D. Mynatt. 2002. Side Views: Persistent, on-Demand Previews for Open-Ended Tasks. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (Paris, France) (UIST ’02). Association for Computing Machinery, New York, NY, USA, 71–80. https://doi.org/10.1145/571985.571996

Digital Library

[41]

Michael Terry, Elizabeth D. Mynatt, Kumiyo Nakakoji, and Yasuhiro Yamamoto. 2004. Variation in Element and Action: Supporting Simultaneous Development of Alternative Solutions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 711–718. https://doi.org/10.1145/985692.985782

Digital Library

[42]

Andries van Dam. 1997. Post-WIMP User Interfaces. Commun. ACM 40, 2 (Feb. 1997), 63–67. https://doi.org/10.1145/253671.253708

Digital Library

[43]

Wesley Willett, Jeffrey Heer, and Maneesh Agrawala. 2007. Scented Widgets: Improving Navigation Cues with Embedded Visualizations. IEEE transactions on visualization and computer graphics 13 (12 2007), 1129–36. https://doi.org/10.1109/TVCG.2007.70589

Digital Library

Cited By

Rezaie MPurdue SKhurana AChilana PTory MCarpendale S(2024)Summary of the Workshop on Interactions for Supporting Explanations and Promoting ComprehensionCompanion Proceedings of the 2024 Conference on Interactive Surfaces and Spaces10.1145/3696762.3698046(27-28)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3696762.3698046
Perraud RTabard AMalacria S(2024)Tutorial mismatches: investigating the frictions due to interface differences when following software video tutorialsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661511(1942-1955)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661511
Yang SVermeulen JFitzmaurice GMatejka J(2024)AQuA: Automated Question-Answering in Software Tutorial Videos with Visual AnchorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642752(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642752
Show More Cited By

Index Terms

Supercharging Trial-and-Error for Learning Complex Software Applications
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Learning by Exploration: New Challenges in Real-World Environments
KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Learning is a predominant theme for any intelligent system, humans, or machines. Moving beyond the classical paradigm of learning from past experience, e.g., offline supervised learning from given labels, a learner needs to actively collect exploratory ...
Federated Learning for Mobility Applications
The increasing concern for privacy and the use of machine learning on personal data has led researchers to introduce new approaches to machine learning. Federated learning is one such a novel privacy-preserving machine learning approach that “brings code ...
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

During training, model-free reinforcement learning (RL) systems can explore actions that lead to harmful or costly consequences. Having a human "in the loop'' and ready to intervene at all times can prevent these mistakes, but is prohibitively expensive ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
492
Total Downloads

Downloads (Last 12 months)85
Downloads (Last 6 weeks)11

Reflects downloads up to 09 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rezaie MPurdue SKhurana AChilana PTory MCarpendale S(2024)Summary of the Workshop on Interactions for Supporting Explanations and Promoting ComprehensionCompanion Proceedings of the 2024 Conference on Interactive Surfaces and Spaces10.1145/3696762.3698046(27-28)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3696762.3698046
Perraud RTabard AMalacria S(2024)Tutorial mismatches: investigating the frictions due to interface differences when following software video tutorialsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661511(1942-1955)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661511
Yang SVermeulen JFitzmaurice GMatejka J(2024)AQuA: Automated Question-Answering in Software Tutorial Videos with Visual AnchorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642752(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642752
Masson DMalacria SCasiez GVogel D(2024)DirectGPT: A Direct Manipulation Interface to Interact with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642462(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642462
Rezaie MTory MCarpendale S(2024)Struggles and Strategies in Understanding Information VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338856030:6(3035-3048)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3388560
Veuskens TRamakers RLeen DLuyten K(2023)History in Motion: Interactive 3D Animated Visualizations for Understanding and Exploring the Modeling History of 3D CAD DesignsProceedings of the 8th ACM Symposium on Computational Fabrication10.1145/3623263.3623358(1-13)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3623263.3623358
Masson DMalacria SVogel DLank ECasiez G(2023)ChartDetective: Easy and Accurate Interactive Data Extraction from Complex Vector ChartsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581113(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581113
Zhou TLiu CYang JHuang J(2023)filtered.ink: Creating Dynamic Illustrations with SVG FiltersProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581051(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581051
Gmeiner FYang HYao LHolstein KMartelaro N(2023)Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design ToolsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580999(1-20)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580999

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten