short-paper

Towards Sketch-based User Interaction with Integrated Software Development Environments

Authors:

Sigurdur Gauti Samuelsson,

Matthias BookAuthors Info & Claims

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

Pages 181 - 184

https://doi.org/10.1145/3387940.3392231

Published: 25 September 2020 Publication History

Abstract

Powerful software tools, such as software development environments, often have complex graphical user interfaces (GUIs) that are not intuitive to handle, especially when performing complex, multi-step operations. We hypothesize that sketching could be a more intuitive way of expressing user intentions than navigating nested menus or memorizing keyboard shortcuts to accomplish complex operations. Enabling this vision requires software capable of both allowing the user to sketch anywhere on a GUI, and interpreting those sketches as specific commands to be performed within the integrated development environment (IDE). In this paper, we report on preliminary results of an elicitation study performed to gather insights into how developers would use a sketch-based interface.

References

[1]

Caroline Appert and Shumin Zhai. 2009. Using Strokes As Command Shortcuts: Cognitive Benefits and Toolkit Support (CHI '09). ACM, New York, NY, USA, 2289--2298. https://doi.org/10.1145/1518701.1519052

[2]

Uday Athavankar. 1997. Mental Imagery as a Design Tool. Cybernetics and Systems 28, 1 (1997), 25--42. https://doi.org/10.1080/019697297126236 arXiv:https://doi.org/10.1080/019697297126236

[3]

Uday Athavankar and Arnab Mukherjee. 2003. Blindfolded Classroom: Getting Design Students to Use Mental Imagery. Springer Berlin Heidelberg, Berlin, Heidelberg, 111--120. https://doi.org/10.1007/978-3-662-07811-2_12

[4]

Matthias Book and André van der Hoek. 2018. Sketching with a purpose: moving from supporting modeling to supporting software engineering activities. In 2018 IEEE/ACM 11th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). IEEE, 93--96.

[5]

Mauro Cherubini, Gina Venolia, Rob DeLine, and Andrew J. Ko. 2007. Let's Go to the Whiteboard: How and Why Software Developers Use Drawings (CHI '07). Association for Computing Machinery, New York, NY, USA, 557--566. https://doi.org/10.1145/1240624.1240714

[6]

Ronald Chung, Petrut Mirica, and Beryl Plimmer. 2005. InkKit: A Generic Design Tool for the Tablet PC. In Proceedings of the 6th ACM SIGCHI New Zealand Chapter's International Conference on Computer-human Interaction: Making CHI Natural (CHINZ '05). ACM, New York, NY, USA, 29--30. https://doi.org/10.1145/1073943.1073950

Digital Library

[7]

Michael D. Good, John A. Whiteside, Dennis R. Wixon, and Sandra J. Jones. 1984. Building a User-derived Interface. Commun. ACM 27, 10 (Oct. 1984), 1032--1043. https://doi.org/10.1145/358274.358284

Digital Library

[8]

Kathryn Henderson. 1991. Flexible Sketches and Inflexible Data Bases: Visual Communication, Conscription Devices, and Boundary Objects in Design Engineering. Science, Technology, & Human Values 16, 4 (1991), 448--473. https://doi.org/10.1177/016224399101600402 arXiv:https://doi.org/10.1177/016224399101600402

[9]

Takeo Igarashi, Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 2007. Teddy: A Sketching Interface for 3D Freeform Design (SIGGRAPH '07). ACM, New York, NY, USA, Article 21, 8 pages. https://doi.org/10.1145/1281500.1281532

Digital Library

[10]

Nicolas Mangano, Alex Baker, and André van der Hoek. 2008. Calico: A Prototype Sketching Tool for Modeling in Early Design. In Proceedings of the 2008 International Workshop on Models in Software Engineering (MiSE '08). ACM, New York, NY, USA, 63--68. https://doi.org/10.1145/1370731.1370747

[11]

Elizabeth D. Mynatt, Takeo Igarashi, W. Keith Edwards, and Anthony LaMarca. 1999. Flatland: New Dimensions in Office Whiteboards. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 346--353. https://doi.org/10.1145/302979.303108

Digital Library

[12]

Shigeru Owada, Frank Nielsen, Kazuo Nakazawa, and Takeo Igarashi. 2003. A Sketching Interface for Modeling the Internal Structures of 3D Shapes. In Smart Graphics, Andreas Butz, Antonio Krüger, and Patrick Olivier (Eds.). Springer Berlin Heidelberg, 49--57. https://doi.org/10.1007/3-540-37620-8_5

[13]

Beryl Plimmer and Mark Apperley. 2004. INTERACTING with Sketched Interface Designs: An Evaluation Study (CHI EA '04). ACM, New York, NY, USA, 1337--1340. https://doi.org/10.1145/985921.986058

Digital Library

[14]

Huawei Tu, Xiangshi Ren, and Shumin Zhai. 2012. A Comparative Evaluation of Finger and Pen Stroke Gestures (CHI '12). ACM, New York, NY, USA, 1287--1296. https://doi.org/10.1145/2207676.2208584

[15]

Barbara Tversky. 2002. What do sketches say about thinking. In Sketch Understanding, papers from the 2002 AAAI Spring Symposium, March 25-27, 2002. 148--151.

[16]

Ilse M Verstijnen, Cees van Leeuwen, G Goldschmidt, Ronald Hamel, and JM Hennessey. 1998. Sketching and creative discovery. Design studies 19, 4 (1998), 519--546. https://doi.org/10.1016/S0142-694X(98)00017-9

[17]

Jacob O. Wobbrock, Meredith Ringel Morris, and Andrew D. Wilson. 2009. User-defined Gestures for Surface Computing (CHI '09). ACM, New York, NY, USA, 1083--1092. https://doi.org/10.1145/1518701.1518866

Digital Library

[18]

Wendy Yee. 2009. Potential Limitations of Multi-touch Gesture Vocabulary: Differentiation, Adoption, Fatigue. In Human-Computer Interaction. Novel Interaction Methods and Techniques, Julie A. Jacko (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 291--300. https://doi.org/10.1007/978-3-642-02577-8_32

[19]

Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 2007. SKETCH: An Interface for Sketching 3D Scenes (SIGGRAPH '07). ACM, New York, NY, USA, Article 19, 6 pages. https://doi.org/10.1145/1281500.1281530

[20]

S. Zhai, P. O. Kristensson, C. Appert, T. H. Andersen, and X. Cao. 2012. Foundational Issues in Touch-Surface Stroke Gesture Design: An Integrative Review.now. https://ieeexplore.ieee.org/document/8187096

Cited By

Samuelsson SBook M(2023)Towards a Visual Language for Sketched Expression of Software IDE Commands2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL-HCC57772.2023.00021(115-123)Online publication date: 3-Oct-2023
https://doi.org/10.1109/VL-HCC57772.2023.00021
Santana de Almeida EAhmed Ivan der Hoek A(2023)Let's Go to the Whiteboard (Again): Perceptions From Software Architects on Whiteboard Architecture MeetingsIEEE Transactions on Software Engineering10.1109/TSE.2023.331441049:10(4773-4795)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TSE.2023.3314410
García-Berná JSobrino-Duque RCarrillo de Gea JNicolás JFernández-Alemán J(2022)Automated Workflow for Usability Audits in the PHR RealmInternational Journal of Environmental Research and Public Health10.3390/ijerph1915894719:15(8947)Online publication date: 22-Jul-2022
https://doi.org/10.3390/ijerph19158947

Index Terms

Towards Sketch-based User Interaction with Integrated Software Development Environments
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens
    2. Interaction techniques
  2. Interaction design
    1. Empirical studies in interaction design
2. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

Evaluation of integrated software development environments: challenges and results from three empirical studies

Evidence shows that integrated development environments (IDEs) are too often functionality-oriented and difficult to use, learn, and master. This article describes challenges in the design of usable IDEs and in the evaluation of the usability of such ...
A visual language for a sketch-based UI prototyping tool
CHI EA '99: CHI '99 Extended Abstracts on Human Factors in Computing Systems

We describe the design of the visual language for SILK 2.0, a sketch-based tool for prototyping user interfaces. The new SILK visual language has been designed to allow user interface designers to quickly prototype the behavior of a user interface. This ...
Development Support of User Interfaces Adaptive to Use Environment
ICSCA '19: Proceedings of the 2019 8th International Conference on Software and Computer Applications

Many applications are used across various platforms, including personal computers (PCs) and mobile devices such as tablets and smartphones. User interfaces must consider various use environments to realize high quality and user-friendly applications. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

June 2020

831 pages

ISBN:9781450379632

DOI:10.1145/3387940

Copyright © 2020 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

ICSE '20

Sponsor:

SIGSOFT
KIISE

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
83
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Samuelsson SBook M(2023)Towards a Visual Language for Sketched Expression of Software IDE Commands2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL-HCC57772.2023.00021(115-123)Online publication date: 3-Oct-2023
https://doi.org/10.1109/VL-HCC57772.2023.00021
Santana de Almeida EAhmed Ivan der Hoek A(2023)Let's Go to the Whiteboard (Again): Perceptions From Software Architects on Whiteboard Architecture MeetingsIEEE Transactions on Software Engineering10.1109/TSE.2023.331441049:10(4773-4795)Online publication date: 1-Oct-2023
https://doi.org/10.1109/TSE.2023.3314410
García-Berná JSobrino-Duque RCarrillo de Gea JNicolás JFernández-Alemán J(2022)Automated Workflow for Usability Audits in the PHR RealmInternational Journal of Environmental Research and Public Health10.3390/ijerph1915894719:15(8947)Online publication date: 22-Jul-2022
https://doi.org/10.3390/ijerph19158947

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents