GuiEvo: Automated Evolution of Mobile App UIs
Pages 335 - 347
Abstract
With the increasing use of mobile applications in today's digital world, touch-based graphical user interfaces (GUIs) have become a crucial component of modern software by which end-users carry out computing tasks. As such, the tools involved in creating these GUIs are of fundamental importance. Due to the continuous pressure for frequent releases of mobile apps to keep pace with platform and device updates, the practice of evolving app GUIs is central to mobile app maintenance. Currently, developers manually introduce GUI changes to their apps as they evolve in a time-consuming process that involves creating mock-ups of updated GUIs and then implementing the changes stipulated by the mock-up.
To help ease the burden of implementing GUI changes, and to help free mobile app developers to focus on fixing bugs or adding features, this paper introduces an automated approach for GUI evolution, called GuiEvo. This approach aims to assist developers in the process of GUI evolution by detecting changes in GUIs between existing releases and proposed mock-ups using computer vision techniques, and automatically generating updated GUI metadata for the new release. We evaluate our approach's performance based on accuracy, precision, recall, and F1-score in detecting the GUI changes, and tree edit distance to measure the correctness of generated UI hierarchies. Our evaluation demonstrates that GuiEvo can detect GUI changes with over 85% accuracy, and the generated GUI hierarchies closely match the expected structure with an average tree edit distance of 5.9. This work points toward the promise of automated tool support for assisting in the evolution of GUIs.
References
[1]
Android Components. https://developer.android.com/design/ui/mobile/guides/components/material-overview.
[2]
Android Debug Bridge. https://developer.android.com/tools/adb.
[3]
Android UI/Application Exerciser Monkey. https://developer.android.com/studio/test/other-testing-tools.
[4]
Canny Edge Detection. https://docs.opencv.org/4.x/da/d22/tutorial_py_canny.html.
[5]
F-droid. https://f-droid.org/.
[6]
GitHub. https://www.github.org/.
[7]
Google Cloud Vision API. https://cloud.google.com/vision.
[8]
Google firebase test lab robo test. https://firebase.google.com/docs/test-lab/robo-ux-test.
[9]
Google Play Store. https://play.google.com/store.
[10]
GuiEvo GitHub Repository. https://github.com/SageSELab/GuiEvo.
[11]
GuiEvo Online Appendix. https://sagelab.io/guievo.
[12]
Image Contouring. https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html.
[13]
Python Imaging Library. https://pypi.org/project/Pillow/.
[14]
Structural Similarity Index. https://scikit-image.org/docs/stable/auto_examples/transform/plot_ssim.html.
[15]
Template Matching Algorithm. https://docs.opencv.org/3.4/d4/dc6/tutorial_py_template_matching.html.
[16]
UI Automator. https://developer.android.com/training/testing/ui-automator?hl=es-419.
[17]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Salvatore De Carmine, and Atif M. Memon. 2012. Using GUI Ripping for Automated Testing of Android Applications. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (Essen, Germany) (ASE '12). Association for Computing Machinery, New York, NY, USA, 258--261.
[18]
Domenico Amalfitano, Anna Rita Fasolino, Porfirio Tramontana, Bryan Dzung Ta, and Atif M. Memon. 2015. MobiGUITAR: Automated Model-Based Testing of Mobile Apps. IEEE Software 32, 5 (2015), 53--59.
[19]
Tanzirul Azim and Iulian Neamtiu. 2013. Targeted and Depth-First Exploration for Systematic Testing of Android Apps. SIGPLAN Not. 48, 10 (oct 2013), 641--660.
[20]
Gabriele Bavota, Mario Linares-Vásquez, Carlos Eduardo Bernal-Cárdenas, Massimiliano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. 2015. The Impact of API Change- and Fault-Proneness on the User Ratings of Android Apps. IEEE Transactions on Software Engineering 41, 4 (2015), 384--407.
[21]
Penelope A. Brooks, Brian P. Robinson, and Atif M. Memon. 2009. An Initial Characterization of Industrial Graphical User Interface Systems. In 2009 International Conference on Software Testing Verification and Validation. 11--20.
[22]
Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. In Proceedings - 2018 ACM/IEEE 40th International Conference on Software Engineering, ICSE 2018, Marsha Chechik and Mark Harman (Eds.). IEEE, Institute of Electrical and Electronics Engineers, United States of America, 665--676. International Conference on Software Engineering 2018, ICSE 2018 ; Conference date: 27-05-2018 Through 03-06-2018.
[23]
Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination? (ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1202--1214.
[24]
Sen Chen, Lingling Fan, Chunyang Chen, Ting Su, Wenhe Li, Yang Liu, and Lihua Xu. 2019. StoryDroid: Automated Generation of Storyboard for Android Apps. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada) (ICSE '19). IEEE Press, 596--607.
[25]
Wontae Choi, George Necula, and Koushik Sen. 2013. Guided GUI Testing of Android Apps with Minimal Restart and Approximate Learning. In Proceedings of the 2013 ACM SIGPLAN International Conference on Object Oriented Programming Systems Languages & Applications (Indianapolis, Indiana, USA) (OOPSLA '13). Association for Computing Machinery, New York, NY, USA, 623--640.
[26]
Adelina Ciurumelea, Andreas Schaufelbühl, Sebastiano Panichella, and Harald C. Gall. 2017. Analyzing reviews and code of mobile apps for better release planning. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). 91--102.
[27]
Brett Daniel, Qingzhou Luo, Mehdi Mirzaaghaei, Darko Marinov, and Mauro Pezzè. 2011. Automated GUI refactoring and test script repair. (01 2011).
[28]
Andrea Di Sorbo, Sebastiano Panichella, Carol V. Alexandru, Junji Shimagaki, Corrado A. Visaggio, Gerardo Canfora, and Harald C. Gall. 2016. What Would Users Change in My App? Summarizing App Reviews for Recommending Software Changes. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (Seattle, WA, USA) (FSE 2016). Association for Computing Machinery, New York, NY, USA, 499--510.
[29]
Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI Testing of Android Applications Via Model Abstraction and Refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). 269--280.
[30]
Gang Hu, Xinhao Yuan, Yang Tang, and Junfeng Yang. 2014. Efficiently, Effectively Detecting Mobile App Bugs with AppDoctor. In Proceedings of the Ninth European Conference on Computer Systems (Amsterdam, The Netherlands) (EuroSys '14). Association for Computing Machinery, New York, NY, USA, Article 18, 15 pages.
[31]
James A. Landay and Brad A. Myers. 1995. Interactive Sketching for the Early Stages of User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '95). ACM Press/Addison-Wesley Publishing Co., USA, 43--50.
[32]
Mario Linares-Vásquez, Gabriele Bavota, Carlos Bernal-Cárdenas, Massimiliano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. 2013. API Change and Fault Proneness: A Threat to the Success of Android Apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 477--487.
[33]
Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An Input Generation System for Android Apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 224--234.
[34]
Atif M. Memon and Bao N. Nguyen. 2010. Advances in Automated Model-Based System Testing of Software Applications with a GUI Front-End. In Advances in Computers, Marvin V. Zelkowitz (Ed.). Advances in Computers, Vol. 80. Elsevier, 121--162.
[35]
Kevin Moran. 2018. Fixing Bug Reporting for Mobile and GUI-Based Applications. (01 2018).
[36]
Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering PP (02 2018).
[37]
Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE '18). Association for Computing Machinery, New York, NY, USA, 165--175.
[38]
Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2017. CrashScope: A Practical Tool for Automated Testing of Android Applications. In Proceedings of the 39th International Conference on Software Engineering Companion (Buenos Aires, Argentina) (ICSE-C '17). IEEE Press, 15--18.
[39]
Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Christopher Vendome, and Denys Poshyvanyk. 2016. Automatically Discovering, Reporting and Reproducing Android Application Crashes. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). 33--44.
[40]
Kevin Moran, Cody Watson, John Hoskins, George Purnell, and Denys Poshyvanyk. 2018. Detecting and Summarizing GUI Changes in Evolving Mobile Apps. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE 2018). Association for Computing Machinery, New York, NY, USA, 543--553.
[41]
Brad A. Myers and Mary Beth Rosson. 1992. Survey on User Interface Programming. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, USA) (CHI '92). Association for Computing Machinery, New York, NY, USA, 195--202.
[42]
Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 248--259.
[43]
Tam The Nguyen, Phong Minh Vu, Hung Viet Pham, and Tung Thanh Nguyen. 2018. Deep Learning UI Design Patterns of Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results (Gothenburg, Sweden) (ICSE-NIER '18). Association for Computing Machinery, New York, NY, USA, 65--68.
[44]
Fabio Palomba, Mario Linares-Vásquez, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2015. User reviews matter! Tracking crowdsourced reviews to support evolution of successful apps. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 291--300.
[45]
Fabio Palomba, Pasquale Salza, Adelina Ciurumelea, Sebastiano Panichella, Harald Gall, Filomena Ferrucci, and Andrea De Lucia. 2017. Recommending and Localizing Change Requests for Mobile Apps Based on User Reviews. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). 106--117.
[46]
Lenin Ravindranath, Suman Nath, Jitendra Padhye, and Hari Balakrishnan. 2014. Automatic and Scalable Fault Detection for Mobile Applications. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (Bretton Woods, New Hampshire, USA) (MobiSys '14). Association for Computing Machinery, New York, NY, USA, 190--203.
[47]
Raimondas Sasnauskas and John Regehr. 2014. Intent Fuzzer: Crafting Intents of Death. In Proceedings of the 2014 Joint International Workshop on Dynamic Analysis (WODA) and Software and System Performance Testing, Debugging, and Analytics (PERTEA) (San Jose, CA, USA) (WODA+PERTEA 2014). Association for Computing Machinery, New York, NY, USA, 1--5.
[48]
Tiago Silva da Silva, Angela Martin, Frank Maurer, and Milene Silveira. 2011. User-Centered Design and Agile Methods: A Systematic Review. In 2011 Agile Conference. 77--86.
[49]
Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. 2018. Rewire: Interface Design Assistance from Examples. Association for Computing Machinery, New York, NY, USA, 1--12.
[50]
Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 470--483.
[51]
Qing Xie, Mark Grechanik, Chen Fu, and Chad Cumby. 2009. Guide: A GUI differentiator. In 2009 IEEE International Conference on Software Maintenance. 395--396.
[52]
Wei Yang, Mukul R. Prasad, and Tao Xie. 2013. A Grey-Box Approach for Automated GUI-Model Generation of Mobile Applications. In Proceedings of the 16th International Conference on Fundamental Approaches to Software Engineering (Rome, Italy) (FASE'13). Springer-Verlag, Berlin, Heidelberg, 250--265.
[53]
Razieh Nokhbeh Zaeem, Mukul R. Prasad, and Sarfraz Khurshid. 2014. Automated Generation of Oracles for Testing User-Interaction Features of Mobile Apps. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 183--192.
[54]
Sai Zhang, Hao Lü, and Michael D. Ernst. 2013. Automatically repairing broken workflows for evolving GUI applications. In Proceedings of the 22nd International Symposium on Software Testing and Analysis. ACM, 45--55.
[55]
Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P. Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In CHI. https://docs-assets.developer.apple.com/ml-research/papers/screen-recognition-chi-2021.pdf
[56]
Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 748--760.
Index Terms
- GuiEvo: Automated Evolution of Mobile App UIs
Recommendations
MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software EngineeringRecent research has begun to examine the potential of automatically finding and fixing accessibility issues that manifest in software. However, while recent work makes important progress, it has generally been skewed toward identifying issues that affect ...
An Explorative Study of the Mobile App Ecosystem from App Developers' Perspective
WWW '17: Proceedings of the 26th International Conference on World Wide WebWith the prevalence of smartphones, app markets such as Apple App Store and Google Play has become the center stage in the mobile app ecosystem, with millions of apps developed by tens of thousands of app developers in each major market. This paper ...
Inter-app communication between Android apps developed in app-inventor and Android studio
MOBILESoft '16: Proceedings of the International Conference on Mobile Software Engineering and SystemsCommunications between mobile apps are an important aspect of mobile platforms. Android is specifically designed with inter-app communication in mind and depends on this to provide different platform specific functionalities. Android Apps can either be ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2024
788 pages
ISBN:9798400705878
DOI:10.1145/3643991
- Chair:
- Diomidis Spinellis,
- Program Chair:
- Alberto Bacchelli,
- Program Co-chair:
- Eleni Constantinou
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
MSR '24
Sponsor:
MSR '24: 21st International Conference on Mining Software Repositories
April 15 - 16, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 101Total Downloads
- Downloads (Last 12 months)101
- Downloads (Last 6 weeks)32
Reflects downloads up to 14 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in