research-article

Public Access

GUIfetch: supporting app design and development through GUI search

Authors:

Farnaz Behrang,

Steven P. Reiss,

Alessandro OrsoAuthors Info & Claims

MOBILESoft '18: Proceedings of the 5th International Conference on Mobile Software Engineering and Systems

Pages 236 - 246

https://doi.org/10.1145/3197231.3197244

Published: 27 May 2018 Publication History

Abstract

A typical way to design and develop a mobile app is to sketch the graphical user interfaces (GUIs) for the different screens in the app and then create actual GUIs from these sketches. Doing so involves identifying which layouts to use, which widgets to add, and how to configure and connect the different pieces of the GUI. To help with this difficult and time-consuming task, we propose GUIFetch, a technique that takes as input the sketch for an app and leverages the growing number of open source apps in public repositories to identify apps with GUIs and transitions that are similar to those in the provided sketch. GUIFetch first searches public repositories to find potential apps using keyword matching. It then builds models of the identified apps' screens and screen transitions using a combination of static and dynamic analyses and computes a similarity metric between the models and the provided sketch. Finally, GUIFetch ranks the identified apps (or parts thereof) based on their computed similarity value and produces a visual ranking of the results together with the code of the corresponding apps. We implemented GUIFetch for Android apps and evaluated it through user studies involving different types of apps.

References

[1]

M. Akhin, N. Tillmann, M. Fahndrich, J. de Halleux, and M. Moskal. 2012. Search by example in TouchDevelop: Code search made easy. In 2012 4th International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation (SUITE). IEEE, New York, 5--8.

[2]

AndroidStudio. 2017. AndroidStudio. (2017). http://tools.android.com/build/studio.

[3]

Mikhail J. Atallah and Susan Fox (Eds.). 1998. Algorithms and Theory of Computation Handbook (1st ed.). CRC Press, Inc., Boca Raton, FL, USA.

Digital Library

[4]

Sushil Bajracharya, Trung Ngo, Erik Linstead, Yimeng Dou, Paul Rigor, Pierre Baldi, and Cristina Lopes. 2006. Sourcerer: A Search Engine for Open Source Code Supporting Structure-based Search. In Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Systems, Languages, and Applications (OOPSLA '06). ACM, New York, NY, USA, 681--682.

Digital Library

[5]

Sushil K. Bajracharya, Joel Ossher, and Cristina V. Lopes. 2010. Leveraging Usage Similarity for Effective Retrieval of Examples in Code Repositories. In Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '10). ACM, New York, NY, USA, 157--166.

Digital Library

[6]

Anabela Caetano, Neri Goulart, Manuel Fonseca, and Joaquim Jorge. 2002. JavaS-ketchIt: Issues in Sketching the Look of User Interfaces. In AAAI Spring Symposium on Sketch Understanding. AAAI Press, Menlo Park, 9--14.

[7]

Wing-Kwan Chan, Hong Cheng, and David Lo. 2012. Searching Connected API Subgraph via Text Phrases. In Proceedings of the ACMSIGSOFT 20th International Symposium on the Foundations of Software Engineering(FSE 12). ACM, New York, NY, USA, Article 10, 11 pages.

Digital Library

[8]

Shih-Chien Chou, Jen-Yen Chen, and Chyan-Goei Chung. 1996. A Behavior-based Classification and Retrieval Technique for Object-oriented Specification Reuse. Softw. Pract. Exper. 26 (1996), 815--832.

Digital Library

[9]

J.W. Creswell. 2013. Educational Research: Planning, Conducting and Evaluating Quantitative and Qualitative Research. Pearson, New Jersey. https://books.google.com/books?id=39LkngEACAAJ

[10]

Marco de Sá, Luís Carriço, Luís Duarte, and Tiago Reis. 2008. A Mixed-fidelity Prototyping Tool for Mobile Devices. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 225--232.

Digital Library

[11]

C. G. Drummond, D. Ionescu, and R. C. Holte. 2000. A learning agent that assists the browsing of software libraries. IEEE Transactions on Software Engineering 26, 12 (Dec 2000), 1179--1196.

Digital Library

[12]

Espresso. 2017. Espresso. (2017). http://developer.android.com/training/testing/ui-testing/espresso-testing.html.

[13]

Harleen K. Flora, Swati V. Chande, and Xiaofeng Wang. 2010. Adopting an Agile Approach for the Development of Mobile Applications. (2010).

[14]

GUIFetch. 2018. GUIFetch: Supporting App Design and Development through GUI Search. (2018). https://sites.google.com/view/guifetch/.

[15]

R. J. Hall. 1993. Generalized behavior-based retrieval {from a software reuse library}. In Proceedings of 1993 15th International Conference on Software Engineering. IEEE Computer Society Press, Los Alamitos, CA, USA, 371--380.

Digital Library

[16]

Shuai Hao, Bin Liu, Suman Nath, William G.J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for Large-scale Dynamic Analysis of Mobile Apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys 14). ACM, New York, NY, USA, 204--217.

Digital Library

[17]

David Hemer and Peter Lindsay. 2002. Supporting Component-based Reuse in CARE. Aust. Comput. Sci. Commun. 24, 1 (Jan. 2002), 95--104.

Digital Library

[18]

W. Janjic and C. Atkinson. 2012. Leveraging software search and reuse with automated software adaptation. In 2012 4th International Workshop on Search-Driven Development: Users, Infrastructure, Tools, and Evaluation (SUITE). IEEE, New York, 23--26.

[19]

W. Janjic, D. Stoll, P. Bostan, and C. Atkinson. 2009. Lowering the barrier to reuse through test-driven search. In 2009 ICSE Workshop on Search-Driven Development-Users, Infrastructure, Tools and Evaluation. IEEE, New York, 21--24.

Digital Library

[20]

Jun-Jang Jeng and Betty H. C. Cheng. 1995. Specification Matching for Software Reuse: A Foundation. In Proceedings of the 1995 Symposium on Software Reusability (SSR '95). ACM, New York, NY, USA, 97--105.

Digital Library

[21]

M. G. Kendall. 1938. A New Measure of Rank Correlation. Biometrika 30, 1/2 (1938), 81--93. http://www.jstor.org/stable/2332226

[22]

Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3083--3092.

Digital Library

[23]

Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011. Bricolage: Example-based Retargeting for Web Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2197--2206.

Digital Library

[24]

James A. Landay and Brad A. Myers. 1995. Interactive Sketching for the Early Stages of User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 43--50.

Digital Library

[25]

J. A. Landay and B. A. Myers. 2001. Sketching interfaces: toward more human interface design. Computer 34, 3 (Mar 2001), 56--64.

Digital Library

[26]

Otávio Augusto Lazzarini Lemos, Sushil Krishna Bajracharya, Joel Ossher, Ricardo Santos Morla, Paulo Cesar Masiero, Pierre Baldi, and Cristina Videira Lopes. 2007. CodeGenie: Using Test-cases to Search and Reuse Source Code. In Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering (ASE '07). ACM, New York, NY, USA, 525--526.

Digital Library

[27]

M. Linares-Vasquez, C. Vendome, Q. Luo, and D. Poshyvanyk. 2015. How developers detect and fix performance bottlenecks in Android apps. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, New York, 352--361.

Digital Library

[28]

D. Lucredio, A. F. Prado, and E. S. de Almeida. 2004. A survey on software components search and retrieval. In Proceedings. 30th Euromicro Conference, 2004. IEEE, New York, 152--159.

Digital Library

[29]

Y. S. Maarek, D. M. Berry, and G. E. Kaiser. 1991. An information retrieval approach for automatically constructing software libraries. IEEE Transactions on Software Engineering 17, 8 (Aug 1991), 800--813.

Digital Library

[30]

R. Mili, A. Mili, and R. T. Mittermeir. 1997. Storing and retrieving software components: a refinement based system. IEEE Transactions on Software Engineering 23, 7 (Jul 1997), 445--460.

Digital Library

[31]

Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI(T). In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE '15). IEEE Computer Society, Washington, DC, USA, 248--259.

Digital Library

[32]

David Ortinau. 2016. From Sketch to App: Mobile Design to Development Workflow. (2016). http://rendr.io/from-sketch-to-app-a-design-to-development-workflow.

[33]

Pencil. 2017. Pencil. (2017). http://pencil.evolus.vn.

[34]

Andy Podgurski and Lynn Pierce. 1993. Retrieving Reusable Software by Sampling Behavior. ACM Trans. Softw. Eng. Methodol. 2, 3 (July 1993), 286--303.

Digital Library

[35]

T. Pole and W. Frakes. 1994. An Empirical Study of Representation Methods for Reusable Software Components. IEEE Transactions on Software Engineering 20 (08 1994), 617--630.

Digital Library

[36]

Lutz Prechelt, Guido Malpohl, and Michael Philippsen. 2002. Finding Plagiarisms among a Set of Programs with JPlag. j-jucs 8, 11 (nov 2002), 1016--1038.

[37]

Steven P. Reiss. 2009. Semantics-based Code Search. In Proceedings of the 31st International Conference on Software Engineering(ICSE '09). IEEE Computer Society, Washington, DC, USA, 243--253.

Digital Library

[38]

Steven P. Reiss. 2014. Seeking the User Interface. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 103--114.

Digital Library

[39]

Eugene J. Rollins and Jeannette M. Wing. 1991. Specifications as search keys for software libraries. In Proceedings 8th International Conference on Logic Programming. Springer-Verlag Berlin Heidelberg, Heidelberg, Germany, 173--187.

[40]

Julian Seifert, Bastian Pfleging, Elba del Carmen Valderrama Bahamóndez, Martin Hermes, Enrico Rukzio, and Albrecht Schmidt. 2011. Mobidev: A Tool for Creating Apps on Mobile Phones. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 109--112.

Digital Library

[41]

C Spearman. 2010. The proof and measurement of association between two things. International Journal of Epidemiology 39, 5 (2010), 1137--1150.

[42]

J. Starke, C. Luce, and J. Sillito. 2009. Working with search results. In 2009 ICSE Workshop on Search-Driven Development-Users, Infrastructure, Tools and Evaluation. IEEE, New York, 53--56.

Digital Library

[43]

Vijayan Sugumaran and Veda C. Storey. 2003. A Semantic-based Approach to Component Retrieval. SIGMIS Database 34, 3 (Aug. 2003), 8--24.

Digital Library

[44]

Suresh Thummalapenta and Tao Xie. 2007. PARSEWeb: a programmer assistant for reusing open source code on the web. In Proceedings ASE'07. IEEE, New York, 204--213.

Digital Library

[45]

F. Thung, S. Wang, D. Lo, and J. Lawall. 2013. Automatic recommendation of API methods from feature requests. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, New York, 290--300.

Digital Library

[46]

Taciana A. Vanderlei, Frederico A. Durão, Alexandre C. Martins, Vinicius C. Garcia, Eduardo S. Almeida, and Silvio R. de L. Meira. 2007. A Cooperative Classification Mechanism for Search and Retrieval Software Components. In Proceedings of the 2007 ACM Symposium on Applied Computing (SAC '07). ACM, New York, NY, USA, 866--871.

Digital Library

[47]

S. Yang, H. Zhang, H. Wu, Y. Wang, D. Yan, and A. Rountev. 2015. Static Window Transition Graphs for Android (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, New York, 658--668.

[48]

Yunwen Ye and Gerhard Fischer. 2002. Supporting Reuse by Delivering Task-relevant and Personalized Information. In Proceedings of the 24th International Conference on Software Engineering (ICSE '02). ACM, New York, NY, USA, 513--523.

Digital Library

[49]

Clemens Zeidler, Christof Lutteroth, Wolfgang Stuerzlinger, and Gerald Weber. 2013. Evaluating Direct Manipulation Operations for Constraint-Based Layout. Springer Berlin Heidelberg, Berlin, Heidelberg, 513--529.

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Kolthoff KKretzer FBartelt CMaedche APonzetto S(2024)Interlinking User Stories and GUI Prototyping: A Semi-Automatic LLM-Based Approach2024 IEEE 32nd International Requirements Engineering Conference (RE)10.1109/RE59067.2024.00045(380-388)Online publication date: 24-Jun-2024
https://doi.org/10.1109/RE59067.2024.00045
Chen YZhen YShi CLi JChen LLi ZSun LZhou TChang Y(2023)UI layers merger: merging UI layers via visual learning and boundary priorUI图层合并器: 基于计算机视觉与边界先验的UI图层合并方法Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.220009924:3(373-387)Online publication date: 27-Mar-2023
https://doi.org/10.1631/FITEE.2200099
Show More Cited By

Index Terms

GUIfetch: supporting app design and development through GUI search
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Software design engineering

Recommendations

Code Search on Bytecode for Mobile App Development
ACMSE '19: Proceedings of the 2019 ACM Southeast Conference

Code search is an essential task in software development. Existing code search engines often rely on open-source software code repositories to build their search database. However, in mobile app development, high quality code is rarely available as open-...
The University of Alberta user interface management system

In this paper the design and implementation of the University of Alberta user interface management system (UIMS) is discussed. This UIMS is based on the Seeheim model of user interfaces, which divides the user interface into three separate components. ...
A prototype of google interfaces modified for simplifying interaction for blind users
Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility

In this study we present a SW prototype developed within the framework of a research project aiming at improving the usability of search engines for blind users who interact via screen reader and voice synthesizer. Following the eight specific ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MOBILESoft '18: Proceedings of the 5th International Conference on Mobile Software Engineering and Systems

May 2018

275 pages

ISBN:9781450357128

DOI:10.1145/3197231

General Chair:
Christine Julien
The University of Texas at Austin
,
Program Chairs:
Grace Lewis
Carnegie Mellon Software Engineering Institute
,
Itai Segall
Nokia Bell Labs, Israel

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - 28, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

35
Total Citations
View Citations
927
Total Downloads

Downloads (Last 12 months)157
Downloads (Last 6 weeks)20

Reflects downloads up to 05 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Kolthoff KKretzer FBartelt CMaedche APonzetto S(2024)Interlinking User Stories and GUI Prototyping: A Semi-Automatic LLM-Based Approach2024 IEEE 32nd International Requirements Engineering Conference (RE)10.1109/RE59067.2024.00045(380-388)Online publication date: 24-Jun-2024
https://doi.org/10.1109/RE59067.2024.00045
Chen YZhen YShi CLi JChen LLi ZSun LZhou TChang Y(2023)UI layers merger: merging UI layers via visual learning and boundary priorUI图层合并器: 基于计算机视觉与边界先验的UI图层合并方法Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.220009924:3(373-387)Online publication date: 27-Mar-2023
https://doi.org/10.1631/FITEE.2200099
Liang XQi JGao YPeng CYang PChandra SBlincoe KTonella P(2023)AG3: Automated Game GUI Text Glitch Detection Based on Computer VisionProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613867(1879-1890)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613867
Kim KGhatpande SKim DZhou XLiu KBissyandé TKlein JLe Traon Y(2023)Big Code Search: A BibliographyACM Computing Surveys10.1145/360490556:1(1-49)Online publication date: 26-Aug-2023
https://dl.acm.org/doi/10.1145/3604905
Chen SFan LChen CLiu Y(2023)Automatically Distilling Storyboard With Rich Features for Android AppsIEEE Transactions on Software Engineering10.1109/TSE.2022.315954849:2(667-683)Online publication date: 1-Feb-2023
https://doi.org/10.1109/TSE.2022.3159548
Liu ZChen CWang JHuang YHu JWang Q(2023)Nighthawk: Fully Automated Localizing UI Display Issues via Visual UnderstandingIEEE Transactions on Software Engineering10.1109/TSE.2022.315087649:1(403-418)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3150876
Wei JCourbis ALambolais TXu BBernard PDray G(2023)Boosting GUI Prototyping with Diffusion Models2023 IEEE 31st International Requirements Engineering Conference (RE)10.1109/RE57278.2023.00035(275-280)Online publication date: Sep-2023
https://doi.org/10.1109/RE57278.2023.00035
Huang YWang JLiu ZWang SChen CLi MWang Q(2023)Context-aware Bug Reproduction for Mobile Apps2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00196(2336-2348)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00196
Liu ZChen CWang JSu YHuang YHu JWang Q(2023)Ex pede Herculem: Augmenting Activity Transition Graph for Apps via Graph Convolution Network2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00168(1983-1995)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00168
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten