research-article

Learning-to-rank vs ranking-to-learn: strategies for regression testing in continuous integration

Authors:

Antonia Bertolino,

Antonio Guerriero,

Roberto Pietrantuono,

Stefano RussoAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1 - 12

https://doi.org/10.1145/3377811.3380369

Published: 01 October 2020 Publication History

Abstract

In Continuous Integration (CI), regression testing is constrained by the time between commits. This demands for careful selection and/or prioritization of test cases within test suites too large to be run entirely. To this aim, some Machine Learning (ML) techniques have been proposed, as an alternative to deterministic approaches. Two broad strategies for ML-based prioritization are learning-to-rank and what we call ranking-to-learn (i.e., reinforcement learning). Various ML algorithms can be applied in each strategy. In this paper we introduce ten of such algorithms for adoption in CI practices, and perform a comprehensive study comparing them against each other using subjects from the Apache Commons project. We analyze the influence of several features of the code under test and of the test process. The results allow to draw criteria to support testers in selecting and tuning the technique that best fits their context.

References

[1]

Benjamin Busjaeger and Tao Xie. 2016. Learning for Test Prioritization: An Industrial Case Study. In 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York, NY, 975--980.

Digital Library

[2]

Borja Calvo and Guzman Santafe. 2015. scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems. The R Journal 8, 1 (2015), 248--256.

[3]

Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2008. An empirical study of the effect of time constraints on the cost-benefits of regression testing. In 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, 71--82.

Digital Library

[4]

Vinicius H. S. Durelli, Rafael S. Durelli, Simone S. Borges, Andre T. Endo, Marcelo M. Eler, Diego R. C. Dias, and Marcelo P. Guimarães. 2019. Machine Learning Applied to Software Testing: A Systematic Mapping Study. IEEE Transactions on Reliability 68, 3 (2019), 1189--1212.

[5]

Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York, NY, 235--245.

Digital Library

[6]

Salvador García, Alberto Fernández, Julián Luengo, and Francisco Herrera. 2010. Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences 180, 10 (2010), 2044 -- 2064.

Digital Library

[7]

Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. 2015. Practical Regression Test Selection with Dynamic File Dependencies. In 2015 International Symposium on Software Testing and Analysis (ISSTA). ACM, New York, NY, 211--222.

Digital Library

[8]

Alireza Haghighatkhah, Mika Mäntylä, Markku Oivo, and Pasi Kuvaja. 2018. Test prioritization in continuous integration environments. Journal of Systems and Software 146 (2018), 80 -- 98.

[9]

H. Hemmati, Z. Fang, and M. V. Mantyla. 2015. Prioritizing Manual Test Cases in Traditional and Rapid Release Environments. In IEEE 8th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 10.

[10]

H. Hotelling. 1947. Multivariate quality control. In Techniques of Statistical Analysis, Wallis W.A. Eisenhart C., Hastay M. (Ed.). McGraw-Hill, New York (1947), 111--184.

[11]

Ronald L. Iman and James M. Davenport. 1980. Approximations of the critical region of the fbietkan statistic. Communications in Statistics - Theory and Methods 9, 6 (1980), 571--595.

[12]

Muhammad Khatibsyarbini, Mohd Adham Isa, Dayang N.A. Jawawi, and Rooster Tumeng. 2018. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology 93 (2018), 74--93.

Digital Library

[13]

Eric Knauss, Miroslaw Staron, Wilhelm Meding, Ola Söder, Agneta Nilsson, and Magnus Castell. 2015. Supporting Continuous Integration by Code-churn Based Test Selection. In IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering (RCoSE). IEEE, 19--25.

Digital Library

[14]

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. 2004. Estimating mutual information. Phys. Rev. E 69 (2004), 066138. Issue 6.

[15]

Remo Lachmann, Sandro Schulze, Manuel Nieke, Christoph Seidl, and Ina Schaefer. 2016. System-level test case prioritization using machine learning. In 15th IEEE International Conference on Machine Learning and Applications. IEEE, 361--368.

[16]

Owolabi Legunsen, Farah Hariri, August Shi, Yafeng Lu, Lingming Zhang, and Darko Marinov. 2016. An Extensive Study of Static Regression Test Selection in Modern Software Evolution. In 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE). ACM, New York, NY, 583--594.

Digital Library

[17]

Alexandre R. Lenz, Aurora Pozo, and Silvia R. Vergilio. 2013. Linking software testing results with a machine learning approach. Engineering Applications of Artificial Intelligence 26, 5 (2013), 1631--1640.

Digital Library

[18]

Hang Li. 2011. Learning to Rank for Information Retrieval and Natural Language Processing. Morgan & Claypool, San Rafael, CA.

[19]

Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning 8, 3 (01 May 1992), 293--321.

Digital Library

[20]

Yijuan Lu, Ira Cohen, Xiang Sean Zhou, and Qi Tian. 2007. Feature selection using principal feature analysis. In 15th ACM International Conference on Multimedia (MM). ACM, New York, NY, 301--304.

Digital Library

[21]

Mike McGarr, Dianne Marsh, and the Developer Productivity team. 2017. Towards true continuous integration: distributed repositories and dependencies. https://medium.com/netflix-techblog/towards-true-continuous-integration-distributed-repositories-and-dependencies-2a2e3108c051

[22]

Atif Memon, Zebao Gao, Bao Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-Scale Continuous Testing. In IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 233--242.

[23]

Ade Miller. 2008. A Hundred Days of Continuous Integration. In Agile 2008 Conference. IEEE, 289--293.

Digital Library

[24]

Douglas C. Montgomery. 1997. Introduction to statistical quality control (3 ed.). Wiley, New York, NY.

[25]

Y. Pang, X. Xue, and A. S. Namin. 2013. Identifying Effective Test Cases through K-Means Clustering for Enhancing Regression Testing. In 12th International Conference on Machine Learning and Applications. IEEE, 78--83.

Digital Library

[26]

Naren Ramakrishnan. 2009. The Pervasiveness of Data Mining asnd Machine Learning. Computer 42, 8 (Aug 2009), 28--29.

Digital Library

[27]

John H. Skillings and Gregory A. Mack. 1981. On the Use of a Friedman-Type Statistic in Balanced and Unbalanced Block Designs. Technometrics 23, 2 (1981), 171--177. http://www.jstor.org/stable/1268034

[28]

Helge Spieker, Arnaud Gotlieb, Dusica Marijan, and Morten Mossige. 2017. Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration. In 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). ACM, New York, NY, 12--22.

Digital Library

[29]

Daniel Ståhl and Jan Bosch. 2013. Experienced benefits of continuous integration in industry software product development: A case study. In 12th IASTED International Conference on Software Engineering. ACTA Press, Calgary, 736--743.

[30]

Stephen W. Thomas, Hadi Hemmati, Ahmed E. Hassan, and Dorothea Blostein. 2014. Static Test Case Prioritization Using Topic Models. Empirical Software Engineering 19, 1 (2014), 182--212.

Digital Library

[31]

Michael E Tipping and Christopher M Bishop. 1999. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61, 3 (1999), 611--622.

[32]

Paolo Tonella, Paolo Avesani, and Angelo Susi. 2006. Using the Case-Based Ranking Methodology for Test Case Prioritization. In 22nd IEEE International Conference on Software Maintenance (ICSM). IEEE, 10.

Digital Library

[33]

Hado Philip van Hasselt. 2012. Reinforcement Learning in Continuous State and Action Spaces. In Reinforcement Learning. Adaptation, Learning, and Optimization, M. Wiering and M. van Otterlo (Eds.), Vol. 12. Springer, Berlin, Heidelberg, 207--251.

[34]

Ting Wang and Tingting Yu. 2018. A Study of Regression Test Selection in Continuous Integration Environments. In 29th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 135--143.

[35]

X. Yang, K. Tang, and X. Yao. 2015. A Learning-to-Rank Approach to Software Defect Prediction. IEEE Transactions on Reliability 64, 1 (2015), 234--246.

[36]

Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22, 2 (2012), 67--120.

[37]

Du Zhang and Jeffrey J.P. Tsai. 2003. Machine Learning and Software Engineering. Software Quality Journal 11, 2 (2003), 87--119.

Digital Library

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
ÖZER EBUZLUCA F(2024)Test Case Prioritization For Embedded SoftwareProceedings of the 2024 13th International Conference on Software and Computer Applications10.1145/3651781.3651794(81-89)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3651781.3651794
Wang GSun ZChen YZhao YLiang QHao DChristakis MPradel M(2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680356
Show More Cited By

Index Terms

Learning-to-rank vs ranking-to-learn: strategies for regression testing in continuous integration
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Optimizing test prioritization via test distribution analysis
ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Test prioritization aims to detect regression faults faster via reordering test executions, and a large number of test prioritization techniques have been proposed accordingly. However, test prioritization effectiveness is usually measured in terms of ...
Reinforcement learning for automatic test case prioritization and selection in continuous integration
ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Testing in Continuous Integration (CI) involves test case prioritization, selection, and execution at each cycle. Selecting the most promising test cases to detect bugs is hard if there are uncertainties on the impact of committed code changes or, if ...
Comparative study of machine learning test case prioritization for continuous integration testing
Abstract
There is a growing body of research indicating the potential of machine learning to tackle complex software testing challenges. One such challenge pertains to continuous integration testing, which is highly time-constrained, and generates a large ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

MIUR
FACEPE
CAPES
CNPq

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

53
Total Citations
View Citations
1,025
Total Downloads

Downloads (Last 12 months)160
Downloads (Last 6 weeks)17

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang JWang KNie PFilkov VRay BZhou M(2024)Efficient Incremental Code Coverage Analysis for Regression Test SuitesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695551(1882-1894)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695551
ÖZER EBUZLUCA F(2024)Test Case Prioritization For Embedded SoftwareProceedings of the 2024 13th International Conference on Software and Computer Applications10.1145/3651781.3651794(81-89)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1145/3651781.3651794
Wang GSun ZChen YZhao YLiang QHao DChristakis MPradel M(2024)Commit Artifact Preserving Build PredictionProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680356(1236-1248)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680356
Douglas Ferrari Mendonça WAssunção WVergilio S(2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672592
Khan MAzim ALiscano RSmith KChang YTauseef QSeferi GSaadatmand MLonetti FBudnik CLi JGuerriero A(2024)Machine Learning-based Test Case Prioritization using Hyperparameter OptimizationProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644467(125-135)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644467
Zeng ZXiao TLamothe MHata HMcintosh SSpinellis DConstantinou EBacchelli A(2024)A Mutation-Guided Assessment of Acceleration Approaches for Continuous Integration: An Empirical Study of YourBaseProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644914(556-568)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644914
Zhang ZChen JGu YLi ZSosu R(2024)Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression TestingIET Software10.1049/2024/99429592024(1-14)Online publication date: 4-Apr-2024
https://doi.org/10.1049/2024/9942959
Mendonça WAssunção WVergilio S(2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
https://doi.org/10.1016/j.jss.2024.112157
Roza EPrado Lima JVergilio S(2024)On the use of contextual information for machine learning based test case prioritization in continuous integration developmentInformation and Software Technology10.1016/j.infsof.2024.107444171:COnline publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107444
Altiero FCorazza ADi Martino SPeron ALibero Lucio Starace L(2024)Regression test prioritization leveraging source code similarity with tree kernelsJournal of Software: Evolution and Process10.1002/smr.2653Online publication date: 15-Feb-2024
https://doi.org/10.1002/smr.2653
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents