Generating a test oracle from program documentation: work in progress

A fundamental assumption of software testing is that there is some mechanism, an oracle, that will determine whether or not the results of a test execution are correct. In practice this is often done by comparing the output, either automatically or manually, to some pre-calculated, presumably correct, output [17]. However, if the program is formally documented it is possible to use the specification to determine the success or failure of a test execution, as in [1], for example. This paper discusses ongoing work to produce a tool that will generate a test oracle from formal program documentation.

In [9], [10] and [11] Parnas et al. advocate the use of a relational model for documenting the intended behaviour of programs. In this method, tabular expressions are used to improve readability so that formal documentation can replace conventional documentation. Relations are described by giving their characteristic predicate in terms of the values of concrete program variables. This documentation method has the advantage that the characteristic predicate can be used as the test oracle -- it simply must be evaluated for each test execution (input & output) to assign pass or fail. In contrast to [1], this paper discusses the testing of individual programs, not objects as used in [1]. Consequently, the method works with program documentation, written in terms of the concrete variables, and no representation function need be supplied. Documentation in this form, and the corresponding oracle, are illustrated by an example.

Finally, some of the implications of generating test oracles from relational specifications are discussed.