research-article

Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces

Authors:

Alexander Nied,

James FogartyAuthors Info & Claims

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

Pages 221 - 230

https://doi.org/10.1145/2642918.2647412

Published: 05 October 2014 Publication History

Abstract

Pixel-based methods have the potential to fundamentally change how we build graphical interfaces, but remain difficult to implement. We introduce a new toolkit for pixel based enhancements, focused on two areas of support. Prefab Layers helps developers write interpretation logic that can be composed, reused, and shared to manage the multi-faceted nature of pixel-based interpretation. Prefab Annotations supports robustly annotating interface elements with metadata needed to enable runtime enhancements. Together, these help developers overcome subtle but critical dependencies between code and data. We validate our toolkit with (1) demonstrative applications and (2) a lab study that compares how developers build an enhancement using our toolkit versus state of the art methods. Our toolkit addresses core challenges faced by developers when building pixel based enhancements, potentially opening up pixel based systems to broader adoption.

References

[1]

Banovic, N., Grossman, T., Matejka, J., and Fitzmaurice, G. Waken: Reverse Engineering Usage Information and Interface Structure from Software Videos. UIST 2012. 83--92.

Digital Library

[2]

Bolin, M., Webber, M., Rha, P., Wilson, T., and Miller, R.C. Automation and Customization of Rendered Web Pages. UIST 2005. 163--172.

Digital Library

[3]

Chang, T.-H., Yeh, T., and Miller, R. Associating the Visual Representation of User Interfaces with Their Internal Structures and Metadata. UIST 2011. 245--254.

Digital Library

[4]

Chang, T.-H., Yeh, T., and Miller, R.C. GUI Testing Using Computer Vision. CHI 2010. 1535--1544.

Digital Library

[5]

Dixon, M., Fogarty, J., and Wobbrock, J. A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. CHI 2012. 3167--3176.

Digital Library

[6]

Dixon, M. and Fogarty, J. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. CHI 2010. 1525--1534.

Digital Library

[7]

Dixon, M., Laput, G., and Fogarty, J. Pixel-Based Methods for Widget State and Style in a Runtime Implementation of Sliding Widgets. CHI 2014. 2231--2240.

Digital Library

[8]

Dixon, M., Leventhal, D., and Fogarty, J. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. CHI 2011. 969--978.

Digital Library

[9]

Eagan, J.R., Beaudouin-Lafon, M., and Mackay, W.E. Cracking the Cocoa Nut: User Interface Programming at Runtime. UIST 2011. 225--234.

Digital Library

[10]

Edwards, W.K., Hudson, S.E., Marinacci, J., Rodenstein, R., Rodriguez, T., and Smith, I. Systematic Output Modification in a 2D User Interface Toolkit. UIST 1997. 151--158.

Digital Library

[11]

Fujima, J., Lunzer, A., Hornbæk, K., and Tanaka, Y. Clip, Connect, Clone: Combining Application Elements to Build Custom Interfaces for Information Access. UIST 2004. 175--184.

Digital Library

[12]

Grossman, T. and Balakrishnan, R. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. CHI 2005. 281--290.

Digital Library

[13]

Grossman, T., Matejka, J., and Fitzmaurice, G. Chronicle: Capture, Exploration, and Playback of Document Workflow Histories. UIST 2010. 143--152.

Digital Library

[14]

Hartmann, B., Wu, L., Collins, K., and Klemmer, S.R. Programming by a Sample: Rapidly Creating Web Applications with d.mix. UIST 2007. 241--250.

Digital Library

[15]

Hurst, A., Hudson, S.E., and Mankoff, J. Automatically Identifying Targets Users Interact with During Real World Tasks. IUI 2010. 11--20.

Digital Library

[16]

Kato, J., Mcdirmid, S., and Cao, X. DejaVu: Integrated Support for Developing Interactive Camera-Based Programs. UIST 2012. 189--196.

Digital Library

[17]

Meng, X., Zhao, S., Huang, Y., Zhang, Z., and Eagan, J.R. WADE: Simplified GUI Add-on Development for Third-party Software. CHI 2014. 2221--2230.

Digital Library

[18]

Moscovich, T. Contact Area Interaction with Sliding Widgets. UIST 2009. 13--22.

Digital Library

[19]

Myers, B., Hudson, S.E., and Pausch, R. Past, Present, and Future of User Interface Software Tools. TOCHI 7 (1). 3--28.

Digital Library

[20]

Nichols, J., Hua, Z., and Barton, J. Highlight: A System for Creating and Deploying Mobile Web Applications. UIST 2008. 249--258.

Digital Library

[21]

Nichols, J. and Lau, T. Mobilization by Demonstration: Using Traces to Re-author Existing Web Sites. IUI 2008. 149--160.

Digital Library

[22]

Olsen, D.R., Hudson, S.E., Verratti, T., Heiner, J.M., and Phelps, M. Implementing Interface Attachments Based on Surface Representations. CHI 1999. 191--198.

Digital Library

[23]

Olsen, D.R., Taufer, T., and Fails, J.A. ScreenCrayons: Annotating Anything. UIST 2004. 165--174.

Digital Library

[24]

Olsen, D.R. Evaluating User Interface Systems Research. UIST 2007. 251--258.

Digital Library

[25]

Pongnumkul, S., Dontcheva, M., Li, W., Wang, J., Bourdev, L., Avidan, S., and Cohen, M. Pause-and-Play: Automatically Linking Screencast Video Tutorials with Applications. UIST 2011. 135--144.

Digital Library

[26]

Savva, M., Kong, N., Chhajta, A., Fei-fei, L., Agrawala, M., and Heer, J. ReVision: Automated Classification, Analysis and Redesign of Chart Images. UIST 2011. 393--402.

Digital Library

[27]

St Amant, R., Riedl, R., Ritter, F.E., and Reifers, A. Image Processing in Cognitive Models with SegMan. HCII 2005.

[28]

Stuerzlinger, W., Chapuis, O., Phillips, D., and Roussel, N. User Interface Façades: Towards Fully Adaptable User Interfaces. UIST 2006. 309--318.

Digital Library

[29]

Waldner, M., Steinberger, M., Grasset, R., and Schmalstieg, D. ImportanceDriven Compositing Window Management: CHI2011.959--968.

Digital Library

[30]

Yeh, T., Chang, T.-H., and Miller, R.C. Sikuli: Using GUI Screenshots for Search and Automation. UIST 2009. 183--194.

Digital Library

[31]

Yeh, T., Chang, T.-H., Xie, B.,Walsh, G., Watkins, I., Wongsuphasawat, K., Huan, M., Davis, L.S., Bederson, B. Creating Contextual Help for GUIs Using Screenshots. UIST 2011. 145--154.

Digital Library

Cited By

Islam MBillah S(2023)SpaceX MagProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962537:2(1-36)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596253
Leiva LHota AOulasvirta A(2022)Describing UI Screenshots in Natural LanguageACM Transactions on Intelligent Systems and Technology10.1145/356470214:1(1-28)Online publication date: 9-Nov-2022
https://dl.acm.org/doi/10.1145/3564702
Jiang YStuerzlinger WLutteroth CKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445043
Show More Cited By

Index Terms

Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces
1. Human-centered computing

Recommendations

Content and hierarchy in pixel-based methods for reverse engineering interface structure
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

The rigidity and fragmentation of GUI toolkits are fundamentally limiting the progress and impact of interaction research. Pixel-based methods offer unique potential for addressing these challenges independent of the implementation of any particular ...
A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest ...
Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure
CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Current chasms between applications implemented with different user interface toolkits make it difficult to implement and explore potentially important interaction techniques in new and existing applications, limiting the progress and impact of human-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technology

October 2014

722 pages

ISBN:9781450330695

DOI:10.1145/2642918

General Chair:
Hrvoje Benko
Microsoft Research, USA
,
Program Chairs:
Mira Dontcheva
Adobe, USA
,
Daniel Wigdor
University of Toronto, Canada

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST '14

Sponsor:

UIST '14: The 27th Annual ACM Symposium on User Interface Software and Technology

October 5 - 8, 2014

Hawaii, Honolulu, USA

Acceptance Rates

UIST '14 Paper Acceptance Rate 74 of 333 submissions, 22%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
388
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Islam MBillah S(2023)SpaceX MagProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962537:2(1-36)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596253
Leiva LHota AOulasvirta A(2022)Describing UI Screenshots in Natural LanguageACM Transactions on Intelligent Systems and Technology10.1145/356470214:1(1-28)Online publication date: 9-Nov-2022
https://dl.acm.org/doi/10.1145/3564702
Jiang YStuerzlinger WLutteroth CKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445043
Li TMitchell TMyers B(2021)Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning AgentsArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_15(495-537)Online publication date: 5-Nov-2021
https://doi.org/10.1007/978-3-030-82681-9_15
Veuskens TLuyten KRamakers R(2020)RataplanProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/33973294:2(1-23)Online publication date: 15-Jun-2020
https://dl.acm.org/doi/10.1145/3397329
Maia RSilva JSilva J(2019)Towards Graphical User Interface Redefinition without Source Code AccessProceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3338286.3344398(1-6)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1145/3338286.3344398
Ledo DVermeulen JCarpendale SGreenberg SOehlberg LBoring SHarrison SBardzell SNeustaedter CTatar D(2019)AstralProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322329(711-724)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322329
Fraser CNgoon TDontcheva MKlemmer SBrewster SFitzpatrick GCox AKostakos V(2019)RePlayProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300527(1-13)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300527
Intharah TTurmukhambetov DBrostow G(2019)HILCACM Transactions on Interactive Intelligent Systems10.1145/32345089:2-3(1-27)Online publication date: 18-Mar-2019
https://dl.acm.org/doi/10.1145/3234508
Zhang XRoss AFogarty JBaudisch PSchmidt AWilson A(2018)Robust Annotation of Mobile Application Interfaces in Methods for Accessibility Repair and EnhancementProceedings of the 31st Annual ACM Symposium on User Interface Software and Technology10.1145/3242587.3242616(609-621)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3242587.3242616
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents