Managing app testing device clouds: issues and opportunities
Abstract
Because creating and maintaining an in-house test lab is expensive and time-consuming, companies and app developers often use device clouds to test their apps. Because quality-assurance activities depend on such device clouds, it is important to understand possible issues related to their use. To this end, in this paper we present a preliminary study that investigates issues and highlights research opportunities in the context of managing and maintaining device clouds. In the study, we analyzed over 12 million test executions on 110 devices. We found that the management software of the cloud infrastructure we considered affected some test executions, and almost all the cloud devices had at least one security-related issue.
References
[1]
Amazon. 2020. AWS Device Farm. Retrieved August 31, 2020 from https://aws.amazon.com/device-farm
[2]
Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni García, Francisco Gortázar, Francesca Lonetti, and Eda Marchetti. 2019. A Systematic Review on Cloud Testing. ACM Comput. Surv. (2019).
[3]
L. Cheng, J. Chang, Z. Yang, and C. Wang. 2016. GUICat: GUI testing as a service. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE) (Singapore, Singapore). IEEE, 858--863.
[4]
CVE. 2016. CVE-2016-2507. Retrieved August 31, 2020 from https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2507
[5]
Zhen Dong, Abhishek Tiwari, Xiao Liang Yu, and Abhik Roychoudhury. 2020. Concurrency-related Flaky Test Detection in Android apps. arXiv:2005.10762 [cs.SE]
[6]
Mattia Fazzini and Alessandro Orso. 2017. Automated Cross-Platform Inconsistency Detection for Mobile Apps. In Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering (Urbana-Champaign, IL, USA). IEEE, 308--318.
[7]
Daniel Giordano. 2017. The Average Cost Of A Device Lab. Retrieved August 31, 2020 from https://crossbrowsertesting.com/blog/browsers/average-cost-device-lab
[8]
Google. 2020. Compatibility Test Suite. Retrieved August 31, 2020 from https://source.android.com/compatibility/cts
[9]
Google. 2020. Firebase Test Lab. Retrieved August 31, 2020 from https://firebase.google.com/docs/test-lab
[10]
P. Graubner, L. Baumgärtner, P. Heckmann, M. Müller, and B. Freisleben. 2015. Dynalize: Dynamic Analysis of Mobile Apps in a Platform-as-a-Service Cloud. In 2015 IEEE 8th International Conference on Cloud Computing (New York, NY, USA). IEEE, 925--932.
[11]
D. Han, C. Zhang, X. Fan, A. Hindle, K. Wong, and E. Stroulia. 2012. Understanding Android Fragmentation with Topic Analysis of Vendor-Specific Bugs. In 2012 19th Working Conference on Reverse Engineering (Kingston, Canada). IEEE, 83--92.
[12]
Dongjie He, Lian Li, Lei Wang, Hengjie Zheng, Guangwei Li, and Jingling Xue. 2018. Understanding and Detecting Evolution-induced Compatibility Issues in Android Apps. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, Montpellier, France, 167--177.
[13]
Software Testing Help. 2020. Best Cloud-Based Mobile App Testing Service Providers In 2020. Retrieved August 31, 2020 from https://www.softwaretestinghelp.com/cloud-mobile-testing-services
[14]
Andreas Holzinger, Peter Treitler, and Wolfgang Slany. 2012. Making apps useable on multiple different mobile platforms: On interoperability for business application development on smartphones. In International Conference on Availability, Reliability, and Security (Prague, Czech Republic). IEEE, 176--189.
[15]
J. Huang. 2014. AppACTS: Mobile App Automated Compatibility Testing Service. In 2014 2nd IEEE International Conference on Mobile Cloud Computing, Services, and Engineering (Oxford, UK). IEEE, 85--90.
[16]
Mona Erfani Joorabchi, Ali Mesbah, and Philippe Kruchten. 2013. Real Challenges in Mobile app Development. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Baltimore, MD, USA). IEEE, 15--24.
[17]
Taeyeon Ki, Chang Min Park, Karthik Dantu, Steven Y. Ko, and Lukasz Ziarek. 2019. Mimic: UI Compatibility Testing System for Android Apps. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada). IEEE Press, 246--256.
[18]
Huoran Li, Xuan Lu, Xuanzhe Liu, Tao Xie, Kaigui Bian, Felix Xiaozhu Lin, Qiaozhu Mei, and Feng Feng. 2015. Characterizing smartphone usage patterns from millions of Android users. In Proceedings of the 2015 ACM Conference on Internet Measurement Conference (Tokyo, Japan). ACM, 459--472.
[19]
Li Li, Tegawendé F. Bissyandé, Haoyu Wang, and Jacques Klein. 2018. CiD: Automating the Detection of API-related Compatibility Issues in Android Apps. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Amsterdam, Netherlands, 153--163.
[20]
C. Liu, W. Chen, and S. Chen. 2016. A Concurrent Approach for Improving the Efficiency of Android CTS Testing. In 2016 International Computer Symposium (ICS) (Chiayi, Taiwan). IEEE, 611--615.
[21]
Yepang Liu, Chang Xu, and Shing-Chi Cheung. 2014. Characterizing and detecting performance bugs for smartphone applications. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India). ACM, 1013--1024.
[22]
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An Empirical Analysis of Flaky Tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China). ACM, 643--653.
[23]
R. Mahmood, N. Esfahani, T. Kacem, N. Mirzaei, S. Malek, and A. Stavrou. 2012. A whitebox approach for automated security testing of Android applications on the cloud. In 2012 7th International Workshop on Automation of Software Test (AST) (Zurich, Switzerland). IEEE, 22--28.
[24]
A. Malini, N. Venkatesh, K. Sundarakantham, and S. Mercyshalinie. 2014. Mobile application testing on smart devices using MTAAS framework in cloud. In International Conference on Computing and Communication Technologies (Hyderabad, India). IEEE, 1--5.
[25]
Microsoft. 2020. Visual Studio App Center. Retrieved August 31, 2020 from https://appcenter.ms
[26]
Abhinav Pathak, Y Charlie Hu, and Ming Zhang. 2011. Bootstrapping energy debugging on smartphones: a first look at energy bugs in mobile devices. In Proceedings of the 10th ACM Workshop on Hot Topics in Networks (Cambridge, MA, USA). ACM, 1--6.
[27]
C. M. Prathibhan, A. Malini, N. Venkatesh, and K. Sundarakantham. 2014. An automated testing framework for testing Android mobile applications in the cloud. In 2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies (Ramanathapuram, India). IEEE, 1216--1219.
[28]
Isabel K. Villanes Rojas, Silvia Meireles, and Arilo Claudio Dias-Neto. 2016. Cloud-Based Mobile App Testing Framework: Architecture, Implementation and Execution. In Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing (Maringa, Parana, Brazil). ACM, 1--10.
[29]
Snigdha. 2020. Top Device Clouds for Mobile App Testing. Retrieved August 31, 2020 from https://www.appypie.com/top-device-clouds-for-mobile-app-testing
[30]
SYSTANGO. 2020. Everything You Need to know about Cloud-Based Device Testing Providers. Retrieved August 31, 2020 from https://medium.com/@systango/everything-you-need-to-know-about-cloud-based-device-testing-providers-3090fe465a58
[31]
Chuanqi Tao and Jerry Gao. 2017. On building a cloud-based mobile testing infrastructure service system. Journal of systems and software 124 (2017), 39--55.
[32]
C. Tao, J. Gao, and B. Li. 2015. Cloud-Based Infrastructure for Mobile Testing as a Service. In 2015 Third International Conference on Advanced Cloud and Big Data (Yangzhou, China). IEEE, 133--140.
[33]
S. Thorve, C. Sreshtha, and N. Meng. 2018. An Empirical Study of Flaky Tests in Android Apps. In 2018 IEEE International Conference on Software Maintenance and Evolution (Madrid, Spain). IEEE, 534--538.
[34]
Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2016. Taming Android Fragmentation: Characterizing and Detecting Compatibility Issues for Android Apps. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore). ACM, 226--237.
[35]
Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2019. Pivot: Learning API-Device Correlations to Facilitate Android Compatibility Issue Detection. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada). IEEE, 878--888.
[36]
Lei Wu, Michael Grace, Yajin Zhou, Chiachih Wu, and Xuxian Jiang. 2013. The impact of vendor customizations on android security. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security (Berlin, Germany). ACM, 623--634.
[37]
M. G. Xavier, K. J. Matteussi, G. R. França, W. P. Pereira, and C. A. F. De Rose. 2017. Mobile Application Testing on Clouds: Challenges, Opportunities and Architectural Elements. In 2017 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP) (St. Petersburg, Russia). IEEE, 181--185.
[38]
Samer Zein, Norsaremah Salleh, and John Grundy. 2016. A Systematic Mapping Study of Mobile Application Testing Techniques. J. Syst. Softw. (2016), 334--356.
[39]
S. Zhang and B. Pi. 2015. Mobile Functional Test on TaaS Environment. In 2015 IEEE Symposium on Service-Oriented System Engineering (San Francisco, CA, USA). IEEE, 315--320.
[40]
Xiaoyong Zhou, Yeonjoon Lee, Nan Zhang, Muhammad Naveed, and XiaoFeng Wang. 2014. The peril of fragmentation: Security hazards in android device driver customizations. In 2014 IEEE Symposium on Security and Privacy (San Jose, CA, USA). IEEE, 409--423.
- Managing app testing device clouds: issues and opportunities
Recommendations
Cloud-Based Mobile App Testing Framework: Architecture, Implementation and Execution
SAST '16: Proceedings of the 1st Brazilian Symposium on Systematic and Automated Software TestingThe growth in the use of mobile devices is notorious due to the multiple functionalities they offer. The time between the release of new device models and mobile platform updates is very short, and this has a direct influence on the quality of mobile ...
Managing appliance launches in infrastructure clouds
TG '11: Proceedings of the 2011 TeraGrid Conference: Extreme Digital DiscoveryInfrastructure cloud computing introduces a significant paradigm shift that has the potential to revolutionize how scientific computing is done. However, while it is actively adopted by a number of scientific communities, it is still lacking a well-...
Testing cross-platform mobile app development frameworks
ASE '15: Proceedings of the 30th IEEE/ACM International Conference on Automated Software EngineeringMobile app developers often wish to make their apps available on a wide variety of platforms, e.g., Android, iOS, and Windows devices. Each of these platforms uses a different programming environment, each with its own language and APIs for app ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2020
1449 pages
ISBN:9781450367684
DOI:10.1145/3324884
- General Chair:
- John Grundy,
- Program Chairs:
- Claire Le Goues,
- David Lo
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- IEEE CS
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 January 2021
Check for updates
Qualifiers
- Short-paper
Funding Sources
- Amazon
Conference
ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering
December 21 - 25, 2020
Virtual Event, Australia
Acceptance Rates
Overall Acceptance Rate 82 of 337 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 65Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)0
Reflects downloads up to 20 Nov 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in