research-article

Sikuli: using GUI screenshots for search and automation

Authors:

Tsung-Hsiang Chang,

Robert C. MillerAuthors Info & Claims

UIST '09: Proceedings of the 22nd annual ACM symposium on User interface software and technology

Pages 183 - 192

https://doi.org/10.1145/1622176.1622213

Published: 04 October 2009 Publication History

Abstract

We present Sikuli, a visual approach to search and automation of graphical user interfaces using screenshots. Sikuli allows users to take a screenshot of a GUI element (such as a toolbar button, icon, or dialog box) and query a help system using the screenshot instead of the element's name. Sikuli also provides a visual scripting API for automating GUI interactions, using screenshot patterns to direct mouse and keyboard events. We report a web-based user study showing that searching by screenshot is easy to learn and faster to specify than keywords. We also demonstrate several automation tasks suitable for visual scripting, such as map navigation and bus tracking, and show how visual scripting can improve interactive help systems previously proposed in the literature.

Supplementary Material

JPG File (p183-yeh.jpg)

Download
20.65 KB

WMV File (p183-yeh.wmv)

Download
9.31 MB

References

[1]

St. Amant, R., H. Lieberman, R. Potter, and L. Zettlemoyer. Programming by example: visual generalization in programming by example. Commun. ACM 43(3), 107--114, 2003.

Digital Library

[2]

Bergman, L., V. Castelli, T. Lau, and D. Oblinger. DocWizards: a system for authoring follow-me documentation wizards. Proc. UIST '05, 191--200, 2005.

Digital Library

[3]

Bolin, M., M. Webber, P. Rha, T. Wilson, and R. C. Miller. Automation and customization of rendered web pages. Proc. UIST '05, 163--172, 2005.

Digital Library

[4]

Forsyth, D. and Ponce, J., Computer Vision: A Modern Approach, Prentice Hall, USA, 2002.

Digital Library

[5]

Harrison, S. M. A comparison of still, animated, or nonillustrated on-line help with written or spoken instructions in a graphical user interface. Proc. CHI '95, 82--89, 1995.

Digital Library

[6]

Hoffmann, R., J. Fogarty, and D. S. Weld. Assieme: finding and leveraging implicit references in a web search interface for programmers. Proc. UIST '07, 13--22, 2007.

Digital Library

[7]

Huang, J. and M. B. Twidale. Graphstract: minimal graphical help for computers. Proc. UIST '07, 203--212, 2007.

Digital Library

[8]

Kelleher, C. and R. Pausch. Stencils-based tutorials: design and evaluation. Proc. CHI '05, 541--550, 2005.

Digital Library

[9]

Knabe, K. Apple guide: a case study in user-aided design of online help. Proc. CHI '95, 286--287, 1995.

Digital Library

[10]

Leshed, G., E. M. Haber, T. Matthews, and T. Lau. CoScripter: automating&sharing how-to knowledge in the enterprise. Proc. CHI '08, 1719--1728, 2008.

Digital Library

[11]

Lowe, D. G. Object recognition from local scale-invariant features. Proc. International Conference on Computer Vision, 1150--1157, 1999.

Digital Library

[12]

Matas, J., O. Chum, U. Martin, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. Proc. British Machine Vision Conference '02, 2002.

[13]

Mikolajczyk, K., B. Leibe, and B. Schiele. Multiple object class detection with a generative model. Proc. Computer Vision and Pattern Recognition '06, 26--36, 2006.

Digital Library

[14]

Moriyon, R., P. Szekely, and R. Neches. Automatic generation of help from interface design models. Proc. CHI '94, 225--231, 1994.

Digital Library

[15]

Pangoli, S. and F. Paternó. Automatic generation of task-oriented help. Proc. UIST '95, 181--187, 1995.

Digital Library

[16]

Potter, R. L. Pixel data access: interprocess communication in the user interface for end-user programming and graphical macros. Ph.D. thesis, College Park, MD, USA, 1999.

Digital Library

[17]

Prabaker, M., L. Bergman, and V. Castelli. An evaluation of using programming by demonstration and guided walkthrough techniques for authoring and utilizing documentation. Proc. CHI '06, 241--250, 2006.

Digital Library

[18]

Sivic, J. and A. Zisserman. Video Google: A text retrieval approach to object matching in videos. Proc. International Conference on Computer Vision, 2003.

Digital Library

[19]

Sukaviriya, P. and J. D. Foley. Coupling a UI framework with automatic generation of context-sensitive animated help. Proc. UIST '90, 152--166, 1990.

Digital Library

[20]

Zettlemoyer, L. S. and St. Amant, R. A visual medium for programmatic control of interactive applications. Proc. CHI '99, 199--206, 1999.

Digital Library

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Liu SWang SSun K(2024)Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional VideosProceedings of the ACM on Human-Computer Interaction10.1145/36602458:EICS(1-19)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660245
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Show More Cited By

Index Terms

Sikuli: using GUI screenshots for search and automation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces

Recommendations

Combining usability research with documentation development for improved user support
SIGDOC '02: Proceedings of the 20th annual international conference on Computer documentation

As a usability research and information design firm, Tec-Ed has noted the advantages of evaluating a product's usability in conjunction with developing its print and online documentation. Usability research identifies the problems with a user interface. ...
Promoting universal usability with multi-layer interface design

Increased interest in universal usability is causing some researchers to study advanced strategies for satisfying first-time as well as intermittent and expert users. This paper promotes the idea of multi-layer interface designs that enable first-time ...
Human interface design at fidelity investments
CHI EA '97: CHI '97 Extended Abstracts on Human Factors in Computing Systems

This presentation describes the Human Interface Design department at Fidelity Investments. Although not in the computer hardware or software business, Fidelity develops an amazingly wide variety of systems in support of our business. The Human Interface ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '09: Proceedings of the 22nd annual ACM symposium on User interface software and technology

October 2009

278 pages

ISBN:9781605587455

DOI:10.1145/1622176

General Chair:
Andrew Wilson
Microsoft Research, USA
,
Program Chair:
François Guimbretière
Cornell University, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '09

Sponsor:

UIST '09: The 22nd Annual ACM Symposium on User Interface Software and Technology

October 4 - 7, 2009

BC, Victoria, Canada

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

265
Total Citations
View Citations
2,658
Total Downloads

Downloads (Last 12 months)137
Downloads (Last 6 weeks)15

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Liu SWang SSun K(2024)Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional VideosProceedings of the ACM on Human-Computer Interaction10.1145/36602458:EICS(1-19)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660245
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Liu ZChen CWang JChen MWu BHuang YHu JWang Q(2024)Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLMProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642939(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642939
Yang SVermeulen JFitzmaurice GMatejka J(2024)AQuA: Automated Question-Answering in Software Tutorial Videos with Visual AnchorsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642752(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642752
Masson DMalacria SCasiez GVogel D(2024)DirectGPT: A Direct Manipulation Interface to Interact with Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642462(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642462
Feng SMa SWang HKong DChen C(2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642350
Coppola RFulcini TArdito LTorchiano MAlègroth E(2024)On Effectiveness and Efficiency of Gamified Exploratory GUI TestingIEEE Transactions on Software Engineering10.1109/TSE.2023.334803650:2(322-337)Online publication date: Feb-2024
https://doi.org/10.1109/TSE.2023.3348036
De Luca MFasolino ATramontana P(2024)Investigating the robustness of locators in template-based Web application testing using a GUI change classification modelJournal of Systems and Software10.1016/j.jss.2023.111932210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111932
Wali AMahamad SSulaiman S(2023)Task Automation Intelligent Agents: A ReviewFuture Internet10.3390/fi1506019615:6(196)Online publication date: 29-May-2023
https://doi.org/10.3390/fi15060196
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents