TACO: An industrial case study of Test Automation for COverage

Timing analysis is an important part of the development of critical real-time systems. It stems from the need to provide evidence on the behaviour of the system, compliance to requirements and timing bounds. The formal testing process is complicated, and includes tests to achieve compliance with certification requirements. Where possible, testing should be performed on a host and then validated on the target. This is especially important for real systems where the target may not be available early in the project or target-based testing is expensive and time consuming. Meaningful host-based testing is difficult when it comes to timing analysis. Automation helps reduce the costs and move testing earlier in the application development cycle. Moving testing earlier in the development cycle not only enables the testing to scale to whole systems, it allows the risks of projects to be managed and software to be optimised before target-based testing is performed.

In this paper, we extend existing work achieving reliable coverage and High WaterMark (HWM) measurement, to scale its application to the analysis of a full system software build, automate the test process, and minimise the set of tests deployed on target. Our case study demonstrates the successful application of the approach on a large code base, i.e. an existing controls system software code. The paper ends with a position statement about how this work is instrumental for both future research but also as part of industry practically analysing the timing behaviour of systems automatically and certifying mixed-criticality systems.