research-article

Automated reporting of GUI design violations for mobile apps

Authors:

Carlos Bernal-Cárdenas,

Denys PoshyvanykAuthors Info & Claims

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

Pages 165 - 175

https://doi.org/10.1145/3180155.3180246

Published: 27 May 2018 Publication History

Abstract

The inception of a mobile app often takes form of a mock-up of the Graphical User Interface (GUI), represented as a static image delineating the proper layout and style of GUI widgets that satisfy requirements. Following this initial mock-up, the design artifacts are then handed off to developers whose goal is to accurately implement these GUIs and the desired functionality in code. Given the sizable abstraction gap between mock-ups and code, developers often introduce mistakes related to the GUI that can negatively impact an app's success in highly competitive marketplaces. Moreover, such mistakes are common in the evolutionary context of rapidly changing apps. This leads to the time-consuming and laborious task of design teams verifying that each screen of an app was implemented according to intended design specifications.

This paper introduces a novel, automated approach for verifying whether the GUI of a mobile app was implemented according to its intended design. Our approach resolves GUI-related information from both implemented apps and mock-ups and uses computer vision techniques to identify common errors in the implementations of mobile GUIs. We implemented this approach for Android in a tool called Gvt and carried out both a controlled empirical evaluation with open-source apps as well as an industrial evaluation with designers and developers from Huawei. The results show that Gvt solves an important, difficult, and highly practical problem with remarkable efficiency and accuracy and is both useful and scalable from the point of view of industrial designers and developers. The tool is currently used by over one-thousand industrial designers & developers at Huawei to improve the quality of their mobile apps.

References

[1]

Adobe photoshop http://www.photoshop.com.

[2]

Android uiautomator http://developer.android.com/tools/help/uiautomator/index.html.

[3]

Apple app store https://www.apple.com/ios/app-store/.

[4]

Fuild-ui https://www.fluidui.com.

[5]

Google play store https://play.google.com/store?hl=en.

[6]

The marketch plugin for sketch https://github.com/tudou527/marketch.

[7]

Mobile apps: What consumers really need and want https://info.dynatrace.com/rs/compuware/images/MobileAppSurveyReport.pdf.

[8]

Mockup.io https://mockup.io/about/.

[9]

Proto.io https://proto.io.

[10]

The sketch design tool https://www.sketchapp.com.

[11]

Sketch extensions https://www.sketchapp.com/extensions/.

[12]

Why your app's ux is more important than you think http://www.codemag.com/Article/1401041.

[13]

E. Alégroth and R. Feldt. On the long-term use of visual gui testing in industrial practice: a case study. Empirical Software Engineering, 22(6):2937--2971, Dec 2017.

Digital Library

[14]

E. Alegroth, Z. Gao, R. Oliveira, and A. Memon. Conceptualization and evaluation of component-based testing unified with visual gui testing: An empirical study. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1--10, April 2015.

[15]

G. Bavota, M. Linares-Vásquez, C. Bernal-Cárdenas, M. Di Penta, R. Oliveto, and D. Poshyvanyk. The impact of api change- and fault-proneness on the userratings of android apps. Software Engineering, IEEE Transactions on, 41(4):384--407, April 2015.

Digital Library

[16]

J.Brooke. SUS: A quick and dirty usability scale. In P. W. Jordan, B. Weerdmeester, A. Thomas, and I. L. Mclelland, editors, Usability evaluation in industry. Taylor and Francis, London, 1996.

[17]

K. Charmaz. Constructing Grounded Theory. SAGE Publications Inc., 2006.

[18]

S. R. Choudhary, M. R. Prasad, and A. Orso. Crosscheck: Combining crawling and differencing to better detect cross-browser incompatibilities in web applications. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, ICST '12, pages 171--180, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[19]

A. Ciurumelea, A. SchaufelbÃČÂyhl, S. Panichella, and H. C. Gall. Analyzing reviews and code of mobile apps for better release planning. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 91--102, Feb 2017.

[20]

A. Di Sorbo, S. Panichella, C. V. Alexandru, J. Shimagaki, C. A. Visaggio, G. Canfora, and H. C. Gall. What would users change in my app? summarizing app reviews for recommending software changes. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, pages 499--510, New York, NY, USA, 2016. ACM.

Digital Library

[21]

K. Holl and F. Elberzhager. A mobile-specific failure classification and its usage to focus quality assurance. In 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications, pages 385--388, Aug 2014.

Digital Library

[22]

G. Hu, X. Yuan, Y. Tang, and J. Yang. Efficiently, effectively detecting mobile app bugs with appdoctor. In Proceedings of the Ninth European Conference on Computer Systems, EuroSys '14, pages 18:1--18:15, New York, NY, USA, 2014. ACM.

Digital Library

[23]

A. Issa, J. Sillito, and V. Garousi. Visual testing of graphical user interfaces: An exploratory study towards systematic definitions and approaches. In 2012 14th IEEE International Symposium on Web Systems Evolution (WSE), pages 11--15, Sept 2012.

Digital Library

[24]

N. Jones. Seven best practices for optimizing mobile testing efforts. Technical Report G00248240, Gartner.

[25]

K. Kuusinen and T. Mikkonen. Designing user experience for mobile apps: Long-term product owner perspective. In 2013 20th Asia-Pacific Software Engineering Conference, volume 1 of APSEC'13, pages 535--540, Dec 2013.

Digital Library

[26]

V. Lelli, A. Blouin, and B. Baudry. Classifying and qualifying gui defects. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1--10, April 2015.

[27]

M. Linares-Vásquez, G. Bavota, C. Bernal-Cárdenas, M. Di Penta, R. Oliveto, and D. Poshyvanyk. Api change and fault proneness: A threat to the success of android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE'13, pages 477--487, New York, NY, USA, 2013. ACM.

Digital Library

[28]

M. Linares-Vásquez, G. Bavota, M. D. Penta, R. Oliveto, and D. Poshyvanyk. How do API changes trigger Stack Overflow discussions? a study on the android SDK. In Proceedings of the 22nd International Conference on Program Comprehension, ICPC'14, pages 83--94, 2014.

Digital Library

[29]

M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome, C. Bernal-Cárdenas, and D. Poshyvanyk. Enabling mutation testing for android apps. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 233--244, New York, NY, USA, 2017. ACM.

Digital Library

[30]

S. Mahajan, A. Alameer, P. McMinn, and W. G. Halfond. Automated repair of layout cross browser issues using search-based techniques. In International Conference on Software Testing and Analysis, ISSTA'17, 2017.

Digital Library

[31]

S. Mahajan and W. G. J. Halfond. Detection and localization of html presentation failures using computer vision-based techniques. In Proceedings of the 8th IEEE International Conference on Software Testing, Verification and Validation, ICST'15, April 2015.

[32]

S. Mahajan, B. Li, P. Behnamghader, and W. G. Halfond. Using visual symptoms for debugging presentation failures in web applications. In Proceeding of the 9th IEEE International Conference on Software Testing, Verification, and Validation (ICST), ICST 16, April 2016.

[33]

T. McDonnell, B. Ray, and M. Kim. An empirical study of api stability and adoption in the android ecosystem. In Proceedings of the 2013 International Conference on Software Maintenance, ICSM'13, pages 70--79, 2013.

Digital Library

[34]

J. Meskens, K. Luyten, and K. Coninx. Plug-and-design: Embracing mobile devices as part of the design environment. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS '09, pages 149--154, New York, NY, USA, 2009. ACM.

Digital Library

[35]

K. Moran, B. Li, C. Bernal-Cárdenas, D. Jelf, and D. Poshyvanyk. Gvt online appendix http://www.android-dev-tools.com/gvt.

[36]

P. Morville. User experience design. http://semanticstudios.com/user_experience_design/.

[37]

B. Myers. Challenges of hci design and implementation. interactions, 1(1):73--83, Jan. 1994.

Digital Library

[38]

B. N. Nguyen, B. Robbins, I. Banerjee, and A. Memon. Guitar: An innovative tool for automated testing of gui-driven software. Automated Software Engg., 21(1):65--105, Mar. 2014.

Digital Library

[39]

T. A. Nguyen and C. Csallner. Reverse engineering mobile application user interfaces with REMAUI. In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering, ASE 15, pages 248--259, Washington, DC, USA, 2015. IEEE Computer Society.

Digital Library

[40]

F. Palomba, M. Linares-Vásquez, G. Bavota, R. Oliveto, M. D. Penta, D. Poshyvanyk, and A. D. Lucia. User reviews matter! tracking crowdsourced reviews to support evolution of successful apps. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 291--300, Sept 2015.

Digital Library

[41]

F. Palomba, P. Salza, A. Ciurumelea, S. Panichella, H. Gall, F. Ferrucci, and A. De Lucia. Recommending and localizing change requests for mobile apps based on user reviews. In Proceedings of the 39th International Conference on Software Engineering, ICSE '17, pages 106--117, Piscataway, NJ, USA, 2017. IEEE Press.

Digital Library

[42]

S. Roy Choudhary, M. R. Prasad, and A. Orso. X-pert: Accurate identification of cross-browser issues in web applications. In Proceedings of the 2013 International Conference on Software Engineering, ICSE '13, pages 702--711, Piscataway, NJ, USA, 2013. IEEE Press.

Digital Library

[43]

S. Roy Choudhary, H. Versee, and A. Orso. Webdiff: Automated identification of cross-browser issues in web applications. In Proceedings of the 2010 IEEE International Conference on Software Maintenance, ICSM '10, pages 1--10, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[44]

T. Silva da Silva, A. Martin, F. Maurer, and M. Silveira. User-centered design and agile methods: A systematic review. In Proceedings of the 2011 Agile Conference, AGILE '11, pages 77--86, Washington, DC, USA, 2011. IEEE Computer Society.

Digital Library

[45]

K.-J. Stol, P. Ralph, and B. Fitzgerald. Grounded theory in software engineering research: A critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering, ICSE '16, pages 120--131, New York, NY, USA, 2016. ACM.

Digital Library

[46]

A. B. Tucker. Computer Science Handbook, Second Edition. Chapman & Hall/CRC, 2004.

Digital Library

[47]

Q. Xie, M. Grechanik, C. Fu, and C. Cumby. Guide: A gui differentiator. In 2009 IEEE International Conference on Software Maintenance, ICSM'09.

[48]

H. Yee, S. Pattanaik, and D. P. Greenberg. Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments. ACM Trans. Graph., 20(1):39--65, Jan. 2001.

Digital Library

[49]

C. Zeidler, C. Lutteroth, W. Stuerzlinger, and G. Weber. Evaluating Direct Manipulation Operations for Constraint-Based Layout, pages 513--529. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Homocianu D(2024)Investigating Patterns in Mobile Phone Usage: An Empirical Exploration Using Multiple TechniquesSSRN Electronic Journal10.2139/ssrn.4707473Online publication date: 2024
https://doi.org/10.2139/ssrn.4707473
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Show More Cited By

Index Terms

Automated reporting of GUI design violations for mobile apps
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis
      2. Software design engineering

Recommendations

Analyzing GUI running fluency for Android apps
MSCC '16: Proceedings of the 3rd ACM Workshop on Mobile Sensing, Computing and Communication

Android as a free open platform has become increasingly popular and been widespread adopted in mobile, tablet, and other devices. However, a great number of issues, such as inadequate quality and the fragmentation phenomenon, have emerged, enhancing the ...
GUI Design for Android Apps
Serving Mobile Apps: A Slice at a Time
EuroSys '19: Proceedings of the Fourteenth EuroSys Conference 2019

End users wanting to do more and more with mobile apps has led to explosive growth in the number of available apps. This has widened the gap between developers making apps available and end users being able to install all the apps they want on their ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '18: Proceedings of the 40th International Conference on Software Engineering

May 2018

1307 pages

ISBN:9781450356381

DOI:10.1145/3180155

Conference Chair:
Michel Chaudron
Chalmers University of Technology, University of Gothenburg, Sweden
,
General Chair:
Ivica Crnkovic
Chalmers University of Technology, University of Gothenburg, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Mark Harman
Facebook and University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

76
Total Citations
View Citations
739
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)5

Reflects downloads up to 14 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Homocianu D(2024)Investigating Patterns in Mobile Phone Usage: An Empirical Exploration Using Multiple TechniquesSSRN Electronic Journal10.2139/ssrn.4707473Online publication date: 2024
https://doi.org/10.2139/ssrn.4707473
Wu JPeng YLi XSwearngin ABigham JNichols J(2024)UIClip: A Data-driven Model for Assessing User Interface DesignProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676408(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676408
Salma SMansur SZhang YMoran KSpinellis DConstantinou EBacchelli A(2024)GuiEvo: Automated Evolution of Mobile App UIsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644936(335-347)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644936
Baral KJohnson JMahmud JSalma SFazzini MRubin JOffutt JMoran KSpinellis DConstantinou EBacchelli A(2024)Automating GUI-based Test Oracles for Mobile AppsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644930(309-321)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644930
Takaffoli MLi SMäkelä V(2024)Generative AI in User Experience Design and Research: How Do UX Practitioners, Teams, and Companies Use GenAI in Industry?Proceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660720(1579-1593)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660720
Nie LSaid KHu M(2024)SoK: An Exhaustive Taxonomy of Display Issues for Mobile ApplicationsProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645193(537-548)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645193
Krishna Vajjala AMansur SJose JMoran KRoychoudhury APaiva AAbreu RStorey M(2024)MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIsProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639167(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639167
Chen LChen YXiao SSong YSun LZhen YZhou TChang YRoychoudhury APaiva AAbreu RStorey M(2024)EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal LearningProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623313(1-12)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623313
Hsueh YChen HChen T(2024)Design-Implementation Discrepancy Analysis for Mobile Application Interfaces Using Object Detection Methods2024 12th International Conference on Information and Education Technology (ICIET)10.1109/ICIET60671.2024.10542829(127-131)Online publication date: 18-Mar-2024
https://doi.org/10.1109/ICIET60671.2024.10542829
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents