article

To the attention of mobile software developers: guess what, test your app!

Authors:

David LoAuthors Info & Claims

Empirical Software Engineering, Volume 24, Issue 4

Pages 2438 - 2468

https://doi.org/10.1007/s10664-019-09701-0

Published: 01 August 2019 Publication History

Abstract

Software testing is an important phase in the software development lifecycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad hoc way, if tested at all. However, we show that, as in other types of software, testing increases the quality of apps (demonstrated in user ratings and number of code issues). Furthermore, we find evidence that tests are essential when it comes to engaging the community to contribute to mobile open source software. We discuss reasons and potential directions to address our findings. Yet another relevant finding of our study is that Continuous Integration and Continuous Deployment (CI/CD) pipelines are rare in the mobile apps world (only 26% of the apps are developed in projects employing CI/CD) --- we argue that one of the main reasons is due to the lack of exhaustive and automatic testing.

References

[1]

Amalfitano D, Amatucci N, Memon AM, Tramontana P, Fasolino AR (2017) A general framework for comparing automatic testing techniques of android mobile apps. J Syst Softw 125:322-343.

[2]

Bavani R (2012) Distributed agile, agile testing, and technical debt. IEEE Softw 29(6):28-33.

Digital Library

[3]

Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading.

Digital Library

[4]

Bird C, Rigby PC, Barr ET, Hamilton DJ, German DM, Devanbu P (2009) The promises and perils of mining git. In: 2009 MSR'09. 6th IEEE International Working Conference on Mining Software Repositories. IEEE, pp 1-10.

Digital Library

[5]

Brooks ME, Dalal DK, Nolan KP (2014) Are common language effect sizes easier to understand than traditional effect sizes? J Appl Psychol 99(2):332.

[6]

Choudhary SR, Gorla A, Orso A (2015) Automated test input generation for Android: Are we there yet? (E). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 429-440.

Digital Library

[7]

Coppola R (2017) Fragility and evolution of android test suites. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 405-408.

Digital Library

[8]

Coppola R, Morisio M, Torchiano M (2017) Scripted gui testing of android apps: A study on diffusion, evolution and fragility. In: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. ACM, pp 22-32.

Digital Library

[9]

Corral L, Fronza I (2015) Better code for better apps: a study on source code quality and market success of android applications. In: Proceedings of the Second ACM International Conference on Mobile Software Engineering and Systems. IEEE Press, pp 22-32.

Digital Library

[10]

Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, pp 137-141.

Digital Library

[11]

Cruz L, Abreu R (2018) Measuring the energy footprint of mobile testing frameworks. In: 2018 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-c). IEEE.

Digital Library

[12]

Das T, Di Penta M, Malavolta I (2016) A quantitative and qualitative investigation of performance-related commits in android apps. In: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 443-447.

[13]

de Langhe B, Fernbach PM, Lichtenstein DR (2016) Navigating by the stars: Investigating the actual and perceived validity of online user ratings. J Consum Res 42(6):817-833.

[14]

Gao J, Bai X, Tsai W-T, Uehara T (2014) Mobile application testing: a tutorial. Computer 47(2):46-55.

Digital Library

[15]

Gao Z, Chen Z, Zou Y, Memon AM (2016) Sitar: Gui test script repair. IEEE Trans Softw Eng 42(2):170-186.

Digital Library

[16]

Geiger F-X, Malavolta I, Pascarella L, Palomba F, Di Nucci D, Bacchelli A (2018) A graph-based dataset of commit history of real-world android apps. In: Proceedings of the 15th International Conference on Mining Software Repositories, MSR. ACM, New York.

Digital Library

[17]

Gousios G, Zaidman A, Storey M-A, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator's perspective. In: Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, pp 358-368.

Digital Library

[18]

Gousios G, Storey M-A, Bacchelli A (2016) Work practices and challenges in pull-based development: The contributor's perspective. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE). IEEE, pp 285-296.

Digital Library

[19]

Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, pp 426-437.

Digital Library

[20]

Hu C, Neamtiu I (2011) Automating GUI testing for android applications. In: Proceedings of the 6th International Workshop on Automation of Software Test. ACM, pp 77-83.

Digital Library

[21]

Joorabchi ME, Mesbah A, Kruchten P (2013) Real challenges in mobile app development. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, pp 15-24.

[22]

Kaasila J, Ferreira D, Kostakos V, Ojala T (2012) Testdroid: Automated Remote UI Testing on Android. In: Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia, ser. MUM '12. ACM, New York, pp. 28:1-28:4.

Digital Library

[23]

Karvonen T, Behutiye W, Oivo M, Kuvaja P (2017) Systematic literature review on the impacts of agile release engineering practices. Information and Software Technology.

Digital Library

[24]

Kerby DS (2014) The simple difference formula: An approach to teaching nonparametric correlation, vol 3. [Online]. Available:

[25]

Khalid H, Nagappan M, Shihab E, Hassan AE (2014) Prioritizing the devices to test your app on: A case study of android game apps. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering.

Digital Library

[26]

Kochhar PS (2013) An empirical study of adoption of software testing in open source projects. In: 2013 13th International Conference on Quality Software (QSIC). IEEE, pp 103-112.

Digital Library

[27]

Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) Adoption of software testing in open source projects -- A preliminary study on 50,000 projects. In: 2013 17th European Conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 353-356.

Digital Library

[28]

Kochhar PS, Thung F, Nagappan N, Zimmermann T, Lo D (2015) Understanding the test automation culture of app developers. In: 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 1-10.

[29]

Krutz DE, Mirakhorli M, Malachowsky SA, Ruiz A, Peterson J, Filipski A, Smith J (2015) A dataset of open-source android applications. In: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories (MSR). IEEE, pp 522-525.

Digital Library

[30]

Leech NL, Onwuegbuzie AJ (2002) A call for greater use of nonparametric statistics.

[31]

Li X, Chang N, Wang Y, Huang H, Pei Y, Wang L, Li X (2017) ATOM: Automatic maintenance of GUI test scripts for evolving mobile applications. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 161-171.

[32]

Linares-Vásquez M, Bernal-Cárdenas C, Moran K, Poshyvanyk D (2017a) How do developers test android applications?" In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME'17) page to appear.

[33]

Linares-Vásquez M, Moran K, Poshyvanyk D (2017b) Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing. In: 33rd IEEE International Conference on Software Maintenance and Evolution (ICSME'17) page to appear.

[34]

Maji AK, Hao K, Sultana S, Bagchi S (2010) Characterizing failures in mobile oses: A case study with android and symbian. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering (ISSRE). IEEE, pp 249-258.

Digital Library

[35]

Martin W, Sarro F, Jia Y, Zhang Y, Harman M (2017) A survey of app store analysis for software engineering. IEEE Trans Softw Eng 43(9):817-847.

Digital Library

[36]

McGraw KO, Wong S (1992) A common language effect size statistic. Psychol Bullet 111(2):361.

[37]

Moran K, Linares-Vásquez M, Poshyvanyk D (2017) Automated GUI testing of Android apps: from research to practice. In: Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, pp 505-506.

Digital Library

[38]

Muccini H, Di Francesco A, Esposito P (2012) Software testing of mobile applications: Challenges and future research directions. In: Proceedings of the 7th International Workshop on Automation of Software Test. IEEE Press, pp 29-35.

Digital Library

[39]

Nagappan M, Shihab E (2016) Future trends in software engineering research for mobile apps. In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 5, pp 21-32.

[40]

Nayebi M, Adams B, Ruhe G (2016) Release practices for mobile apps-what do users and developers think?" In: 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, vol 1, pp 552-562.

[41]

Nayebi M, Cho H, Ruhe G (2018) App store mining is not enough for app improvement. Empir Softw Eng 23(5):1-31.

Digital Library

[42]

Pascarella L, Geiger F-X, Palomba F, Di Nucci D, Malavolta I, Bacchelli A (2018) Self-reported activities of android developers. In: 5th IEEE/ACM International Conference on Mobile Software Engineering and Systems, New York.

Digital Library

[43]

Picco GP, Julien C, Murphy AL, Musolesi M, Roman G-C (2014) Software engineering for mobility: reflecting on the past, peering into the future In: Proceedings of the on Future of Software Engineering. ACM, pp 13-28.

Digital Library

[44]

Ruiz IM, Nagappan M, Adams B, Berger T, Dienst S, Hassan A (2017) An examination of the current rating system used in mobile app stores. IEEE Software.

[45]

Silva DB, Endo AT, Eler MM, Durelli VH (2016) An analysis of automated tests for mobile android applications. In: 2016 XLII Latin American Computing Conference (CLEI). IEEE, pp 1-9.

[46]

Stolberg S (2009) Enabling agile testing through continuous integration. In: 2009 AGILE'09 Agile Conference. IEEE, pp 369-374.

Digital Library

[47]

Van Deursen A, risks Program comprehension (2001) Opportunities in extreme programming. In: Eighth Working Conference on Reverse Engineering Proceedings. IEEE, pp 176-185.

Digital Library

[48]

Visser J, Rigal S, van der Leek R, van Eck P, Wijnholds G (2016) Building maintainable software, java edition: Ten Guidelines for Future-Proof Code. "O'Reilly Media Inc.".

Digital Library

[49]

Wang Y, Alshboul Y (2015) Mobile security testing approaches and challenges. In: 2015 First Conference on Mobile and Secure Services (MOBISECSERV). IEEE, pp 1-5.

[50]

Zaeem RN, Prasad MR, Khurshid S (2014) Automated generation of oracles for testing user-interaction features of mobile apps. In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation (ICST). IEEE, pp 183-192.

Digital Library

[51]

Zeng X, Li D, Zheng W, Xia F, Deng Y, Lam W, Yang W, Xie T (2016) Automated test input generation for android: Are we really there yet in an industrial case?" In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 987-992.

Digital Library

[52]

Zhao Y, Serebrenik A, Zhou Y, Filkov V, Vasilescu B (2017) The impact of continuous integration on other software development practices: A large-scale empirical study 32nd IEEE/ACM International Conference on Automated Software Engineering.

Digital Library

Cited By

Swearngin AWu JZhang XGomez ECoughenour JStukenborg RGarg BHughes GHilliard ABigham JNichols J(2024)Towards Automated Accessibility Report Generation for Mobile AppsACM Transactions on Computer-Human Interaction10.1145/367496731:4(1-44)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3674967
Sokolowski DSpielmann DSalvaneschi GSpinellis DConstantinou EBacchelli A(2024)The PIPr Dataset of Public Infrastructure as Code ProgramsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644888(498-503)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644888
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Show More Cited By

Recommendations

A Mobile Software Quality Model
QSIC '12: Proceedings of the 2012 12th International Conference on Quality Software

The technological progress of mobile devices, the relevance of Internet in today's society and the booming mobile market have revolutionized mobile software in recent years and significantly impact the software quality of mobile interactive devices. ...
A black-box based android GUI testing system
MobiSys '13: Proceeding of the 11th annual international conference on Mobile systems, applications, and services

In Android system, black box testing has risen to three key issues: S1: There is no source code and for tester to know the internal logic of the testing App. S2: There is no testing criterion for tester to know the correct behavior and testing scope of ...
Android apps and user feedback: a dataset for software evolution and quality improvement
WAMA 2017: Proceedings of the 2nd ACM SIGSOFT International Workshop on App Market Analytics

Nowadays, Android represents the most popular mobile platform with a market share of around 80%. Previous research showed that data contained in user reviews and code change history of mobile apps represent a rich source of information for reducing ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 24, Issue 4

August 2019

1171 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2019 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 August 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Swearngin AWu JZhang XGomez ECoughenour JStukenborg RGarg BHughes GHilliard ABigham JNichols J(2024)Towards Automated Accessibility Report Generation for Mobile AppsACM Transactions on Computer-Human Interaction10.1145/367496731:4(1-44)Online publication date: 19-Sep-2024
https://dl.acm.org/doi/10.1145/3674967
Sokolowski DSpielmann DSalvaneschi GSpinellis DConstantinou EBacchelli A(2024)The PIPr Dataset of Public Infrastructure as Code ProgramsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644888(498-503)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644888
Taeb MSwearngin ASchoop ECheng RJiang YNichols J(2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642777
Tan YChen JShang WZhang TFang SLuo XChen ZQi S(2023)STRE: An Automated Approach to Suggesting App Developers When to Stop Reading ReviewsIEEE Transactions on Software Engineering10.1109/TSE.2023.328574349:8(4135-4151)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3285743
Xue FWu JZhang T(2022)Learning-Replay Based Automated Robotic Testing for Mobile AppMobile Information Systems10.1155/2022/10846022022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/1084602
Fazzini MChoi CCopia JLee GKakehi YGorla AOrso ADwyer MDamian DZeller A(2022)Use of test doubles in Android testingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510175(2266-2278)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510175
Caldeira JBrito e Abreu FCardoso Jdos Reis J(2022)Unveiling process insights from refactoring practicesComputer Standards & Interfaces10.1016/j.csi.2021.10358781:COnline publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.csi.2021.103587
Figueira TGil A(2022)Youkai: A Cross-Platform Framework for Testing VR/AR AppsHCI International 2022 – Late Breaking Papers: Interacting with eXtended Reality and Artificial Intelligence10.1007/978-3-031-21707-4_1(3-12)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-21707-4_1
Chouchane MSoui MGhedira K(2021)The impact of the code smells of the presentation layer on the diffuseness of aesthetic defects of Android appsAutomated Software Engineering10.1007/s10515-021-00297-828:2Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1007/s10515-021-00297-8
Lin JSalehnamadi NMalek SGrundy JLe Goues CLo D(2020)Test automation in open-source Android appsProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416623(1078-1089)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416623

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents