Software Testing Tools Return On Investment (ROI) : Executive Summary
Software Testing Tools Return On Investment (ROI) : Executive Summary
Software Testing Tools Return On Investment (ROI) : Executive Summary
Executive Summary This paper addresses the question of Return on Investment in testing tools. It looks at the subject from two perspectives: the tangible and the intangible. Under tangible benefits, measurable short-term (single project) cost-savings are considered. Intangible benefits, which include longer-term gains such as reduced maintenance costs, corporate reputation and similar factors, are also discussed. IPL is an independent software house founded in 1979 and based in Bath. IPL was accredited to ISO9001 in 1988, and gained TickIT accreditation in 1991. Both Cantata++ and AdaTEST 95 have been produced to these standards.
Copyright IPL 2011. This document is the copyright of IPL Information Processing Ltd. It may not be copied or distributed in any form, in whole or in part, without the prior written consent of IPL. All trademarks acknowledged. Cantata and AdaTEST are registered trademarks of IPL Information Processing Limited.
Software Products Group IPL Eveleigh House Grove Street Bath BA1 5LR UK Phone: +44 (0) 1225 475000 Fax: +44 (0) 1225 444400 Email: tools@ipl.com
Last Update:18/03/2011 11:34:00
1.
1.1.
1.2.
2
IPL Information Processing Ltd 2004
ROI =
1.3.
1.4.
Subsequent Projects
For subsequent projects, the ROI greatly increases, as maintenance renewal costs of such tool licences would be only 15% of the original cost, but the savings would continue at the same rate assuming unit and/or integration testing with the tool continue at the same rates. So once the initial outlay is repaid, it is usual to see a very high ongoing ROI. Clearly the above arguments do not take into account the benefits derived from: - The ability to carry out regression testing easily and repeatably: reducing the cost of software changes. - The cost savings derived from not needing to develop or maintain homegrown testing solutions. - The tools ability to measure test coverage thus increasing the quality of testing, and the resultant quality of the software produced. These factors are further discussed below.
2.
3
IPL Information Processing Ltd 2004
The IPL tools play a further role in two key aspects of reducing long-term software support costs: 1. An important capability in maintaining software quality is the facility to run periodic regression tests. IPLs tools can be run in batch mode as well as interactively, making regression testing easy. The most general example of application of regression testing at the unit test level, would be regularly (nightly or weekly) rerunning of all unit tests, to ensure that all code modules work correctly. Such testing will detect problems in units which were seemingly unaffected by changes to other areas of the code. Similarly, regression testing is very important when porting applications to new environments 6. 2. The static analysis (code metrics) element of IPLs tools can help enforce coding rules that will help ensure that code units are maintainable. The key here is to ensure that code modules are of a manageable size and complexity, adequately commented, and containing no code constructs known to make code unreliable or hard to test 5. Further intangible benefits from the use of Cantata++ and AdaTEST 95: Increased staff motivation: Engineers are provided with appropriate tools to do what might otherwise be boring and time-consuming task7. It is a universally acknowledged truth, that few engineers like testing; they have to do it, so make it as palatable as possible by giving them the best tools for the job. Modern safety and quality standards demand not only that all levels of testing (including unit tests) be carried out, but also that evidence of the tests, including coverage analysis, be available8. Cantata++ and AdaTEST 95 have been specifically designed to provide the required evidence.
3.
Conclusions
A simplistic calculation of ROI shows that IPLs tools will typically repay their investment well inside the timescales of the first project on which they are used. These cost savings will be even larger for subsequent projects. In addition to these measurable savings, there are many other benefits. The most significant of these is the improvement in product quality and reliability achievable through more thorough testing at all stages of the development.
* These are generic names for the IPL products which include the older tools, Cantata and AdaTEST.
4.
References
These references can be found in the IPL http://www.iplbath.com/products/casestudies/pc000.shtml case studies, at (1) IPL Case Study 8. was able to report productivity metrics of around one manday per test script for in-house written software (Rolls Royce and Associates) (2) IPL Case Study 15. The other key Cantata++ facility was wrapping which allowed a class to be tested independently of all other interfaces. Prior says, 4
IPL Information Processing Ltd 2004
Without this, difficult external conditions not directly influenced by our code, such as memory allocation errors, may have been difficult to simulate. (Data Systems and Solutions) (3) IPL Case Study 11. We found errors using Cantata++ that we know we would otherwise have missed until later. (Marconi Communications) (4) IPL Case Study 12. AdaTEST has been instrumental in allowing the project to produce a lot of code with a very low residual bug level. (Thales Optronics) (5) Case Study 6. (AdaTEST) has also helped in the smooth running of code inspections by providing reports of source size, complexity, and the identification of prohibited Ada constructs. (Siemens Plessey System, now BAE Systems) (6) IPL Case Study 17.When switching to a new platform release, Cantata++ was a great relief for us. The regression test ability allows us to verify the success of the migration of our application. (Alcatel Transport) (7) IPL Case Study 8. AdaTEST was wholly reliable, robust and easy to use. Use of script templates allowed staff to be productive from an early stage (Rolls Royce and Associates) (8) Case Study 6. AdaTEST has allowed us to meet all contractual requirements on evidence of unit testing its greatest contribution has been the production of hard evidence relating to the adherence to standards (Siemens Plessey System, now BAE Systems)
5
IPL Information Processing Ltd 2004