Abstract
Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions that work well within the context of the Integrated Development Environment (IDE) are in order. Consequently, we created two variants of a test selection heuristic which exploit fine-grained changes recorded during actual development inside the IDE. One variant only considers static binding of method invocations while the other variant takes dynamic binding into account. This paper investigates the tradeoffs between these two variants in terms of the reduction (i.e., How many tests could we omit from the test suite, and how much did we gain in runtime execution?) as well as the fault detection ability of the reduced test suite (i.e., Were tests omitted erroneously?). We used our approach on three distinct cases, two open source cases —Cruisecontrol and PMD— and one industrial case — Historia. Our results show that only considering static binding reduces the test suite significantly but occasionally omits a relevant test; considering dynamic binding rarely misses a test yet often boils down to running the complete test suite. Nevertheless, our analysis provides indications on when a given variant is more appropriate.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We consider a build successful when no build failures occurred, other than failing tests.
References
Andrews JH, Briand LC, Labiche Y (2005) Is mutation an appropriate tool for testing experiments? In: Proceedings Int’l conference on software engineering (ICSE), ACM, pp 402–411. doi:10.1145/1062455.1062530
Basili VR, Caldiera G, Rombach HD (1994) The goal question metric approach. Encyclopedia of Software Engineering
Beck K (2002) Test driven development: by example. Addison-Wesley
Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how and why developers (do not) test in their ides. In: Proceedings of the 10th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 179–190
Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test? In: Proceedings of the 37th international conference on software engineering (ICSE – vol 2). IEEE, pp 559–562
Bennett KH, Rajlich VT (2000) Software maintenance and evolution: a roadmap. In: Proceedings of the conference on the future of software engineering, ICSE ’00, ACM New York, pp 73–87. doi:10.1145/336512.336534
Binder R (1999) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley
Catal C, Mishra D (2013) Test case prioritization: a systematic mapping study. Softw Quality Control 21(3):445–478. doi:10.1007/s11219-012-9181-z
Daniel B, Jagannath V, Dig D, Marinov D (2009) Reassert: suggesting repairs for broken unit tests
Demeyer S, Tichelaar S, Steyaert P (1999) FAMIX 2.0 - the FAMOOS information exchange model. Tech. rep., University of Berne
Dösinger S, Mordinyi R, Biffl S (2012) Communicating continuous integration servers for increasing effectiveness of automated testing. In: 2012 Proceedings of the 27th IEEE/ACM international conference on Automated Software Engineering (ASE). IEEE, pp 374–377
Ebraert P, Vallejos J, Costanza P, Paesschen EV, D’Hondt T (2007) Change-oriented software engineering. In: Proceedings of the Int’l conference on dynamic languages (ICDL). ACM, pp 3–24. doi:10.1145/1352678.1352680
Engström E, Skoglund M, Runeson P (2008) Empirical evaluations of regression test selection techniques: a systematic review. In: Proceedings Int’l symposium empirical software engineering and measurement (ESEM). ACM, pp 22–31. doi:10.1145/1414004.1414011
Engström E, Runeson P, Skoglund M (2010) A systematic review on regression test selection techniques. J Inf Softw Technol 52(1):14–30
Fluri B, Wuersch M, PInzger M, Gall H (2007) Change distilling: tree differencing for fine-grained source code change extraction. IEEE Trans Softw Eng 33 (11):725–743. doi:10.1109/TSE.2007.70731
Fowler M (2006) Continuous integration. Tech. rep, http://www.martinfowler.com/, http://www.martinfowler.com/articles/continuousIntegration.html
Garousi V, Varma T (2010) A replicated survey of software testing practices in the canadian province of alberta: what has changed from 2004 to 2009?. J Syst Softw 83(11):2251–2262
Graves TL, Harrold MJ, Kim JM, Porter A, Rothermel G (2001) An empirical study of regression test selection techniques. ACM Trans Softw Eng Methodol 10(2):184–208
Hattori L, Lanza M (2010) Syde: a tool for collaborative software development. In: Proceedings of the Int’l conference on software engineering (ICSE). ACM, pp 235–238
Hunter JD (2007) Matplotlib: a 2d graphics environment. Comput Sci Eng 9 (3):90–95
Hurdugaci V, Zaidman A (2012) Aiding software developers to maintain developer tests. In: Proceedings of the 16th European conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 11–20
Mansour N, Bahsoon R, Baradhi G (2001) Empirical comparison of regression test selection algorithms. J Syst Softw 57(1):79–90
McGregor J (2007) Test early, test often. J Object Technol 6(4)
Meszaros G (2006) XUnit test patterns: refactoring test code. Prentice Hall PTR
Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99–118. doi:10.1145/227607.227610
Parsai A, Soetens QD, Murgia A, Demeyer S (2014) Considering polymorphism in change-based test suite reduction. In: Agile methods. Large-scale development, refactoring, testing, and estimation - XP 2014 international workshops. Rome, Italy, May 26-30, 2014, Revised Selected Papers, pp 166–181. doi:10.1007/978-3-319-14358-3_14
Qusef A, Bavota G, Oliveto R, De Lucia A, Binkley D (2011) Scotch: test-to-code traceability using slicing and conceptual coupling. In: Proceedings of the Int’l conference on software maintenance (ICSM). IEEE CS, pp 63–72. doi:10.1109/ICSM.2011.6080773
Robbes R, Lanza M (2007) A change-based approach to software evolution. Electron Notes Theor Comput Sci 166:93–109. doi:10.1016/j.entcs.2006.06.015
Robbes R, Lanza M (2008) Spyware: a change-aware development toolset. In: Proceedings of the Int’l conference in software engineering (ICSE). ACM Press, pp 847–850
Rothermel G, Untch R, Chu C, Harrold M (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27(10):929–948. doi:10.1109/32.962562
Runeson P (2006) A survey of unit testing practices. IEEE Softw 23(4):22–29
Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Eng 14(2):131–164
Saff D, Ernst MD (2004) An experimental evaluation of continuous testing during development. In: Proceedings Int’l symposium software testing and analysis (ISSTA). ACM, pp 76–85
Soetens Q D, Demeyer S (2012) ChEOPSJ: change-based test optimization. In: Proceedings of the European conference on software maintenance and reengineering (CSMR). IEEE CS, pp 535–538. doi:10.1109/CSMR.2012.70
Soetens QD, Demeyer S, Zaidman A (2013) Change-based test selection in the presence of developer tests. In: Proceedings of the 17th European conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 101–110
Tillmann N, Schulte W (2006) Unit tests reloaded: parameterized unit testing with symbolic execution. IEEE Softw 23(4)
Van Rompaey B, Demeyer S (2009) Establishing traceability links between unit test cases and units under test. In: Proceedigns of the conference on software maintenance and reengineering (CSMR). IEEE CS, pp 209–218. doi:10.1109/CSMR.2009.39
Venolia G, DeLine R, LaToza T (2005) Software development at microsoft observed. Tech. rep., Microsoft Research. http://research.microsoft.com/pubs/70227/tr-2005-140.pdf
Weijers J (2012) Extending project lombok to improve junit tests. Master’s thesis, Delft University of Technology, http://resolver.tudelft.nl/uuid:1736d513-e69f-4101-8995-4597c2a4df50
Yin RK (2002) Case study research: design and methods, 3 edn. Sage Publications
Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab 22(2):67–120. doi:10.1002/stvr.430
Yoo S, Nilsson R, Harman M (2011) Faster fault finding at Google using multi objective regression test optimisation. In: 8th European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE ’11). Szeged, Hungary
Zaidman A, Van Rompaey B, van Deursen A, Demeyer S (2011) Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Softw Eng 16(3):325–364
Acknowledgments
We express our gratitude to the SEAL team in the University of Zürich, Switzerland for releasing Changedistiller in the public domain; our ChEOPSJ tool is partly based on this release.
We would also like to express are gratitude to Yves Vandewoude at Qmino, for allowing us to conduct these analyses on the Historia codebase.
This work has been sponsored by (i) the Interuniversity Attraction Poles Programme - Belgian State Belgian Science Policy, project MoVES; (ii) the Institute for the Promotion of Innovation through Science and Technology in Flanders (IWT-Vlaanderen) under project number 120028 entitled “Change-centric Quality Assurance (CHAQ)”; (iii) the NWO TestRoots project, the Netherlands.
Figures 6, 7, 8, 9, 10, 11 and 12 were created with matplotlib (Hunter 2007).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Paolo Tonella
Rights and permissions
About this article
Cite this article
Soetens, Q.D., Demeyer, S., Zaidman, A. et al. Change-based test selection: an empirical evaluation. Empir Software Eng 21, 1990–2032 (2016). https://doi.org/10.1007/s10664-015-9405-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-015-9405-5