Empirical Studies of a Safe Regression Test Selection Technique

Reviewer: Charles Alan Wolfe

Beginning with a review of previous analytic results by the authors and others, with respect to (safe) test selection procedures and criteria for judging such procedures on several scales, this paper proves that one can select from among all available tests a safe subset that may also be cost effective. “Safe” means that no required test is omitted from the test set. The authors spend the bulk of the paper covering five empirical studies of test suite selection using their methodology. The studies begin with the “trivial” case of a short program that is easily modified and tested and concludes (in study 5) with a set of sequential versions of a commercial Microsoft Windows NT product. The model includes cost-effectiveness statistics that allow comparison of different test sets. While more research needs to be done, the authors have proven their thesis. Furthermore, they show that these procedures can have real-world applicability. This paper will interest software developers and managers. Quality assurance (QA) and testing practitioners will find ammunition here when arguing for better tools and for more funds. Managers who are willing to work through the more technical or academic portions will find an excellent review of the problems QA people face in properly testing new versions of software. They will also find that scientifically verified procedures—and tools that implement those procedures—can help improve the regression testing process and make it more cost-effective. The paper is fairly well written and, while it provides adequate treatment of the subject matter, it does not require that readers be highly versed in the subject. Its high dependence on two software products (Aristotle and de je vue) is the paper's only weakness.