Article

Scaling regression testing to large software systems

Authors:

Alessandro Orso,

Mary Jean HarroldAuthors Info & Claims

SIGSOFT '04/FSE-12: Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering

Pages 241 - 251

https://doi.org/10.1145/1029894.1029928

Published: 31 October 2004 Publication History

Abstract

When software is modified, during development and maintenance, it is <i>regression tested</i> to provide confidence that the changes did not introduce unexpected errors and that new features behave as expected. One important problem in regression testing is how to select a subset of test cases, from the test suite used for the original version of the software, when testing a modified version of the software. Regression-test-selection techniques address this problem. Safe regression-test-selection techniques select every test case in the test suite that may behave differently in the original and modified versions of the software. Among existing safe regression testing techniques, efficient techniques are often too imprecise and achieve little savings in testing effort, whereas precise techniques are too expensive when used on large systems. This paper presents a new regression-test-selection technique for Java programs that is safe, precise, and yet scales to large systems. It also presents a tool that implements the technique and studies performed on a set of subjects ranging from 70 to over 500 KLOC. The studies show that our technique can efficiently reduce the regression testing effort and, thus, achieve considerable savings.

References

[1]

T. Apiwattanapong, A. Orso, and M. J. Harrold. A differencing algorithm for object-oriented programs. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering (ASE 2004), Linz, Austria, Sep. 2004.

Digital Library

[2]

T. Ball. On the limit of control flow analysis for regression test selection. Proceedings of the ACM-SIGSOFT International Symposium on Software Testing and Analysis, pages 134--142, Mar. 1998.

Digital Library

[3]

J. Bible, G. Rothermel, and D. S. Rosenblum. A comparative study of coarse- and fine-grained safe regression test selection techniques. ACM TOSEM, 10(2):149--183, Apr. 2001.

Digital Library

[4]

D. Binkley. Semantics guided regression test cost reduction. IEEE Transactions on Software Engineering, 23(8), Aug. 1997.

Digital Library

[5]

A. Chawla and A. Orso. A generic instrumentation framework for collecting dynamic information. In Online Proceeding of the ISSTA Workshop on Empirical Research in Software Testing (WERST 2004), Boston, MA, USA, Jul. 2004.

Digital Library

[6]

Y. Chen, D. Rosenblum, and K. Vo. Testtube: A system for selective regression testing. Proceedings of the 16th International Conference on Software Engineering, pages 211--222, May 1994.

Digital Library

[7]

M. J. Harrold, J. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. Spoon, and A. Gujarathi. Regression Test Selection for Java Software. Proceedings of OOPSLA, pages 312--326, Oct. 2001.

Digital Library

[8]

M. J. Harrold, G. Rothermel, D. S. Rosenblum, and E. J. Weyuker. Empirical studies of a prediction model for regression test selection. IEEE Transactions on Software Engineering, 27(3):248--263, Mar. 2001.

Digital Library

[9]

P. Hsia, X. Li, D. Kung, C.-T. Hsu, L. Li, Y. Toyoshima, and C. Chen. A technique for the selective revalidation of OO software. Software Maintenance: Research and Practice, 9:217--233, 1997.

Digital Library

[10]

H. K. N. Leung and L. J. White. A cost model to compare regression test strategies. In Proceedings of the Conference on Software Maintenance '91, pages 201--208, Oct. 1991.

[11]

A. Malishevsky, G. Rothermel, and S. Elbaum. Modeling the cost-benefits tradeoffs for regression testing techniques. In Proceedings of the International Conference on Software Maintenance (ICSM 02), pages 204--213, Oct. 2002.

Digital Library

[12]

C. E. McDowell and D. P. Helmbold. Debugging concurrent programs. ACM Computing Surveys, 21(4):593--622, Dec. 1989.

Digital Library

[13]

B. Meyer. Object-oriented Software Construction. Prentice Hall, New York, N.Y., second edition, 1997.

Digital Library

[14]

T. J. Ostrand and E. J. Weyuker. The distribution of faults in a large industrial software system. In Proceedings of the International Symposium on Software Testing and Analysis, pages 55--64, Jul. 2002.

Digital Library

[15]

X. Ren, F. Shah, F. Tip, B. G. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of java programs. Technical Report DCS-TR-551, Department of Computer Science, Rutgers University, Apr. 2004.

Digital Library

[16]

D. S. Rosenblum and E. J. Weyuker. Using coverage information to predict the cost-effectiveness of regression testing strategies. IEEE Transactions on Software Engineering, 23(3):146--156, Mar. 1997.

Digital Library

[17]

G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. ACM TOSEM, 6(2):173--210, Apr. 1997.

Digital Library

[18]

G. Rothermel, M. J. Harrold, and J. Dedhia. Regression test selection for C++ software. Journal of Software Testing, Verification, and Reliability, pages 77--109, Jun. 2000.

[19]

G. Rothermel and M.J.Harrold. Empirical studies of a safe regression test selection technique. IEEE TSE, pages 401--419, Jun. 1998.

Digital Library

[20]

F. Vokolos and P. Frankl. Pythia: A regression test selection tool based on text differencing. International Conference on Reliability, Quality, and Safety of Software Intensive System, May 1997.

Digital Library

[21]

Z. Wang, K. Pierce, and S. McFarling. BMAT -- a binary matching tool for stale profile propagation. The Journal of Instruction-Level Parallelism, 2, May 2000.

[22]

L. J. White and K. Abdullah. A firewall approach for regression testing of object-oriented software. In Proceedings of 10th Annual Software Quality Week, May 1997.

[23]

L. J. White and H. N. K. Leung. A firewall concept for both control-flow and data-flow in regression integration testing. Proceedings of the Conference on Software Maintenance, pages 262--270, Nov. 1992.

Cited By

Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Zhang GLiu LChen ZWang JFilkov VRay BZhou M(2024)Hybrid Regression Test Selection by Integrating File and Method DependencesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695525(1557-1569)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695525
Lou YYang JBenton SHao DTan LChen ZZhang LZhang L(2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3672450
Show More Cited By

Index Terms

Scaling regression testing to large software systems
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Scaling regression testing to large software systems

When software is modified, during development and maintenance, it is <i>regression tested</i> to provide confidence that the changes did not introduce unexpected errors and that new features behave as expected. One important problem in regression ...
Empirical Studies of a Safe Regression Test Selection Technique

Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression ...
Differential testing: a new approach to change detection
ESEC-FSE companion '07: The 6th Joint Meeting on European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering: companion papers

Regression testing, as it's commonly practiced, is unsound due to inconsistent test repair and test addition. This paper presents a new technique, differential testing, that alleviates the test repair problem and detects more changes than regression ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGSOFT '04/FSE-12: Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineering

October 2004

282 pages

ISBN:1581138555

DOI:10.1145/1029894

General Chair:
Richard N. Taylor
University of California, Irvine, CA
,
Program Chair:
Matthew B. Dwyer
University of Nebraska - Lincoln

ACM SIGSOFT Software Engineering Notes Volume 29, Issue 6
November 2004
275 pages
ISSN:0163-5948
DOI:10.1145/1041685
Issue’s Table of Contents

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGSOFT04/FSE-12

Sponsor:

SIGSOFT04/FSE-12: SIGSOFT 2004 -12th International Symposium on the Foundations of Software Engineering

October 31 - November 6, 2004

CA, Newport Beach, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

181
Total Citations
View Citations
2,168
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CBaz AShi AFilkov VRay BZhou M(2024)Reducing Test Runtime by Transforming Test FixturesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695541(1757-1769)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695541
Zhang GLiu LChen ZWang JFilkov VRay BZhou M(2024)Hybrid Regression Test Selection by Integrating File and Method DependencesProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695525(1557-1569)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695525
Lou YYang JBenton SHao DTan LChen ZZhang LZhang L(2024)When Automated Program Repair Meets Regression Testing—An Extensive Study on Two Million PatchesACM Transactions on Software Engineering and Methodology10.1145/367245033:7(1-23)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3672450
Liu YZhang JNie PGligoric MLegunsen OJust RFraser G(2023)More Precise Regression Test Selection via Reasoning about Semantics-Modifying ChangesProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598086(664-676)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598086
Hundsdorfer SElsner DPretschner A(2023)DIRTS: Dependency Injection Aware Regression Test Selection2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00046(422-432)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00046
Elsner DKacianka SLipp SPretschner AHabermann AGraber MReimer S(2023)BinaryRTS: Cross-language Regression Test Selection for C++ Binaries in CI2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00038(327-338)Online publication date: Apr-2023
https://doi.org/10.1109/ICST57152.2023.00038
Wang SLian XMarinov DXu T(2023)Test Selection for Unified Regression Testing2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00145(1687-1699)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00145
Baral TRahman SChanumolu BBalcı BTuncer TShi ALam W(2023)Optimizing Continuous Development by Detecting and Preventing Unnecessary Content Generation2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00216(901-913)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00216
Li CShi ARyu SSmaragdakis Y(2022)Evolution-aware detection of order-dependent flaky testsProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3533767.3534404(114-125)Online publication date: 18-Jul-2022
https://dl.acm.org/doi/10.1145/3533767.3534404
Eisner DWuersching RSchnappinger MPretschner AGarrido AWong WDe Angelis GDo HNguyen B(2022)Probe-based syscall tracing for efficient and practical file-level test tracesProceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test10.1145/3524481.3527239(126-137)Online publication date: 17-May-2022
https://dl.acm.org/doi/10.1145/3524481.3527239
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents