Introduction to Software

Testing
(2nd edition)
Chapter 2

Model-Driven Test Design

Paul Ammann & Jeff Offutt

http://www.cs.gmu.edu/~offutt/softwaretest/

Update September 2013

First version, 28 August 2011
 Complexity of Testing Software
 No other engineering field builds products as complicated
as software

 The term correctness has no meaning

– Is a building correct?
– Is a car correct?
– Is a subway system correct?

 Like other engineers, we must use abstraction to manage

complexity
– This is the purpose of the model-driven test design process
– The “model” is an abstract structure

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 2

 Software Testing Foundations

Testing can only show the presence of failures

Not their absence

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 3

 Testing & Debugging

 Testing : Evaluating software by observing its execution

 Test Failure : Execution of a test that results in a software

failure

 Debugging : The process of finding a fault given a failure

Not all inputs will “trigger” a fault into causing

a failure
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 4
 Fault & Failure Model
Three conditions necessary for a failure to be observed

1. Reachability : The location or locations in the program

that contain the fault must be reached

2. Infection : The state of the program must be incorrect

3. Propagation : The infected state must cause some output

or final state of the program to be incorrect

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 5

 Software Testing Activities
 Test Engineer : An IT professional who is in charge of one
or more technical test activities
– Designing test inputs
– Producing test values
– Running test scripts
– Analyzing results
– Reporting results to developers and managers

 Test Manager : In charge of one or more test engineers

– Sets test policies and processes
– Interacts with other managers on the project
– Otherwise supports the engineers

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 6

 Traditional Testing Levels
 Acceptance testing : Is
main Class P the software acceptable
to the user?
 System testing : Test
the overall
functionality of the
Class A Class B system
 Integration testing :
method mA1() method mB1() Test how modules
interact with each
other
method mA2() method mB2()
 Module testing
(developer testing) :
Test each class, file,
module, component
 Unit testing (developer
This view obscures underlying similarities testing) : Test each unit
(method) individually
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 7
 Object-Oriented Testing Levels
 Inter-class testing :
Test multiple classes
together

Class A Class B
 Intra-class testing : Test
method mA1() method mB1() an entire class as
sequences of calls
method mA2() method mB2()
 Inter-method testing :
Test pairs of methods in
the same class
 Intra-method testing :
Test each method
individually

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 8

 Coverage Criteria
 Even small programs have too many inputs to fully test
them all
– private static double computeAverage (int A, int B, int C)
– On a 32-bit machine, each variable has over 4 billion possible
values
– Over 80 octillion possible tests!!
– Input space might as well be infinite
 Testers search a huge input space
– Trying to find the fewest inputs that will find the most problems
 Coverage criteria give structured, practical ways to search
the input space
– Search the input space thoroughly
– Not much overlap in the tests
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 9
 Advantages of Coverage Criteria
 Maximize the “bang for the buck”

 Provide traceability from software artifacts to tests

– Source, requirements, design models, …

 Make regression testing easier

 Gives testers a “stopping rule” … when testing is finished

 Can be well supported with powerful tools

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 10

 Test Requirements and Criteria
 Test Criterion : A collection of rules and a process that
define test requirements
̶ Cover every statement
̶ Cover every functional requirement
 Test Requirements : Specific things that must be satisfied
or covered during testing
– Each statement is a test requirement
– Each researchers
Testing functional requirement
have definedis a dozens
test requirement
of criteria, but they
are all really just a few criteria on four types of structures …

1. Graphs
2. Logic expressions
3. Input domains
4. Syntax descriptions
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 11
 Model-Driven Test Design
 Test Design is the process of designing input values that
will effectively test software

 Test design is one of several activities for testing software

– Most mathematical
– Most technically challenging

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 12

 Types of Test Activities
 Testing can be broken up into four general types of
activities
1. Test Design 1.a) Criteria-based
2. Test Automation 1.b) Human-based
3. Test Execution
4. Test Evaluation
 Each type of activity requires different skills, background
knowledge, education and training
 No reasonable software development organization uses the
same people for requirements, design, implementation,
Why do testand
integration organizations stillcontrol
configuration use the same people for all
four test activities??
This clearly wastes resources
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 13
 1. Test Design—(a) Criteria-Based
Design test values to satisfy coverage criteria
or other engineering goal
 This is the most technical job in software testing
 Requires knowledge of :
– Discrete math
– Programming
– Testing
 Requires much of a traditional CS degree
 This is intellectually stimulating, rewarding, and
challenging
 Test design is analogous to software architecture on the
development side
 Using people who are not qualified to design tests is a sure
way to get ineffective tests
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 14
 1. Test Design—(b) Human-Based
Design test values based on domain knowledge of
the program and human knowledge of testing
 This is much harder than it may seem to developers
 Criteria-based approaches can be blind to special situations
 Requires knowledge of :
– Domain, testing, and user interfaces
 Requires almost no traditional CS
– A background in the domain of the software is essential
– An empirical background is very helpful (biology, psychology,
…)
– A logic background is very helpful (law, philosophy, math, …)
 This is intellectually stimulating, rewarding, and
challenging
– But not to typical CS majors – they want to solve problems and
build things
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 15
 2. Test Automation
Embed test values into executable scripts
 This is slightly less technical
 Requires knowledge of programming
 Requires very little theory
 Often requires solutions to difficult problems related to
observability and controllability
 Can be boring for test designers
 Programming is out of reach for many domain experts
 Who is responsible for determining and embedding the
expected outputs ?
– Test designers may not always know the expected outputs
– Test evaluators need to get involved early to help with this

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 16

 3. Test Execution
Run tests on the software and record the results
 This is easy – and trivial if the tests are well automated
 Requires basic computer skills
– Interns
– Employees with no technical background
 Asking qualified test designers to execute tests is a sure
way to convince them to look for a development job
 If, for example, GUI tests are not well automated, this
requires a lot of manual labor
 Test executors have to be very careful and meticulous with
bookkeeping

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 17

 4. Test Evaluation
Evaluate results of testing, report to developers
 This is much harder than it may seem
 Requires knowledge of :
– Domain
– Testing
– User interfaces and psychology
 Usually requires almost no traditional CS
– A background in the domain of the software is essential
– An empirical background is very helpful (biology, psychology,
…)
– A logic background is very helpful (law, philosophy, math, …)
 This is intellectually stimulating, rewarding, and
challenging
– But not to typical CS majors – they want to solve problems and
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 18
 Other Activities
 Test management : Sets policy, organizes team, interfaces
with development, chooses criteria, decides how much
automation is needed, …
 Test maintenance : Save tests for reuse as software evolves
– Requires cooperation of test designers and automators
– Deciding when to trim the test suite is partly policy and partly
technical – and in general, very hard !
– Tests should be put in configuration control
 Test documentation : All parties participate
– Each test must document “why” – criterion and test requirement
satisfied or a rationale for human-designed tests
– Ensure traceability throughout the process
– Keep documentation in the automated tests

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 19

 Organizing the Team
 A mature test organization needs only one test designer to work with
several test automators, executors and evaluators
 Improved automation will reduce the number of test executors
– Theoretically to zero … but not in practice
 Putting the wrong people on the wrong tasks leads to inefficiency,
low job satisfaction and low job performance
– A qualified test designer will be bored with other tasks and look for a job in
development
– A qualified test evaluator will not understand the benefits of test criteria
 Test evaluators have the domain knowledge, so they must be free to
add tests that “blind” engineering processes will not think of
 The four test activities are quite different
Many test teams use the same people for ALL FOUR
activities !!
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 20
 Applying Test Activities

To use our people effectively

and to test efficiently
we need a process that

lets test designers

raise their level of abstraction

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 21

 Using MDTD in Practice
 This approach lets one test designer do the math
 Then traditional testers and programmers can do their parts
– Find values
– Automate the tests
– Run the tests
– Evaluate the tests
 Just like in traditional engineering … an engineer constructs models
with calculus, then gives direction to carpenters, electricians,
technicians, …

Test designers become the technical experts

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 22

 Model-Driven Test Design
refined
model / test
requirements /
structure requirements
test specs
test
requirements DESIGN
ABSTRACTION
LEVEL
IMPLEMENTATION
software
ABSTRACTION
artifact input
LEVEL
values

pass / test test test

fail results scripts cases

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 23

 Model-Driven Test Design – Steps
criterion refine refined
model / test
requirements /
structure requirements
test specs generate
analysis test
requirements
domain DESIGN
analysis ABSTRACTION
LEVEL
IMPLEMENTATION
software
ABSTRACTION
artifact input
LEVEL
k values
feedbac

evaluate execute prefix

automate
pass / test test test postfix
fail results scripts cases expected

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 24

 Model-Driven Test Design–Activities
refined
model / test
requirements /
structure requirements
test specs
Test Design DESIGN
ABSTRACTION
LEVEL
IMPLEMENTATION
Raising our abstraction level makes
ABSTRACTION
software test design MUCH easier
LEVEL input
artifact values
Test Automation

pass / test test test

fail results scripts cases
Test Test
Evaluation Execution
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 25
 Small Illustrative Example

Control Flow Graph

Software Artifact : Java Method
/**
* Return index of node n at the 1 i = 0
* first position it appears,
* -1 if it is not present
*/ 2 i < path.size()
public int indexOf (Node n)
{
for (int i=0; i < path.size(); i++) 3 if
if (path.get(i).equals(n))
return i;
return -1; 5 4
} return -1 return i

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 26

 Example (2)
Support tool for graph coverage
http://www.cs.gmu.edu/~offutt/softwaretest/
Edges 6 requirements for
Graph 12 Edge-Pair Coverage
Abstract version 23 1. [1, 2, 3]
1 32 2. [1, 2, 5]
34 3. [2, 3, 4]
25 4. [2, 3, 2]
2 Initial Node: 1 5. [3, 2, 3]
Final Nodes: 4, 5 6. [3, 2, 5]
3 Test Paths
[1, 2, 5]
[1, 2, 3, 2, 5] Find values …
5 4
[1, 2, 3, 2, 3, 4]
Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 27
 Types of Activities in the Book

Most of this book is on test design

Other activities are well covered elsewhere

Introduction to Software Testing, Edition 2 (Ch 2) © Ammann & Offutt 28