CH 11 Lect 1
CH 11 Lect 1
CH 11 Lect 1
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 2
Some Observations
It is impossible to completely test any nontrivial module or any
system
Theoretical limitations: Halting problem ??
Practial limitations: Prohibitive in time and cost
Testing can only show the presence of bugs, not their absence
(Dijkstra)
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 3
Testing Activities
Subsystem Requirements
Unit System
Code Test Analysis
Design Document
Tested Document User
Subsystem
Subsystem Manual
Unit
Code Test
Tested Integration Functional
Subsystem
Test Test
Integrated Functioning
Subsystems System
Tested Subsystem
Subsystem Unit
Code Test
All tests by developer
Cf. levels of testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 4
Testing Activities continued
Client’s
Global Understanding User
Requirements of Requirements Environment
Validated Accepted
Functioning
System
System PerformanceSystem Acceptance Installation
Test Test Test
Usable
Tests by client System
Tests by developer
User’s understanding
System in
Use
Tests (?) by user
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 5
Level of abstraction
Levels of Testing in V Model
system system
requirements integration
software acceptance
requirements test
preliminary software
design integration
detailed component
design test
Time
N.B.: component test vs. unit
Bernd Bruegge & Allen H. Dutoit
test; acceptance test vs. system integration
Object-Oriented Software Engineering: Using UML, Patterns, and Java 6
Test Planning [Pressman]
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 7
Fault Handling Techniques
Fault Handling
Fault Tolerance
Fault Avoidance Fault Detection
Configuration
Verification
Management
Testing Debugging
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 8
Quality Assurance encompasses Testing
Quality Assurance
Usability Testing
Atomic Modular
Configuration Transactions Redundancy
Verification
Management
Fault Detection
Reviews
Debugging
Walkthrough Inspection
Testing
Correctness Performance
Unit Integration System Debugging Debugging
Testing Testing Testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 9
Types of Testing
Unit Testing:
Individual subsystem
Carried out by developers
Goal: Confirm that subsystems is correctly coded and carries out
the intended functionality
Integration Testing:
Groups of subsystems (collection of classes) and eventually the
entire system
Carried out by developers
Goal: Test the interface among the subsystem
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 10
System Testing
Terminology:
System Testing: system testing here = validation testing
The entire system
Carried out by developers
Goal: Determine if the system meets the requirements (functional
and global)
Acceptance Testing: 2 kinds of Acceptance testing
Evaluates the system delivered by developers
Carried out by the client. May involve executing typical
transactions on site on a trial basis
Goal: Demonstrate that the system meets customer requirements
and is ready to use
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 11
Unit Testing
Informal:
Incremental coding Write a little, test a little
Static Analysis:
Hand execution: Reading the source code
Walk-Through (informal presentation to others)
Code Inspection (formal presentation to others)
Automated Tools checking for
syntactic and semantic errors
Dynamic Analysis:
Black-box testing (Test the input/output behavior)
White-box testing (Test the internal logic of the subsystem or object)
Data-structure based testing (Data types determine test cases)
Focus: I/O behavior. If for any given input, we can predict the
output, then the module passes the test.
Almost always impossible to generate all possible inputs ("test
cases") why?
Goal: Reduce number of test cases by equivalence partitioning:
Divide input conditions into equivalence classes
Choose test cases for each equivalence class. (Example: If an object
is supposed to accept a negative number, testing one negative
number is enough)
If x = 3 then …
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 14
White-box Testing
Focus: Thoroughness (Coverage). Every statement in the component is
executed at least once.
Four types of white-box testing
Statement Testing
Loop Testing
Path Testing
Branch Testing
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 15
White-box Testing (Continued)
Statement Testing (Algebraic Testing): Test single statements
Loop Testing:
Cause execution of the loop to be skipped completely. (Exception:
Repeat loops)
Loop to be executed exactly once
Loop to be executed more than once
Path testing:
Make sure all paths in the program are executed
Branch Testing (Conditional Testing): Make sure that each
possible outcome from a condition is tested at least once
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 16
White-Box Testing
Loop Testing
[Pressman]
Simple
loop
Nested
Loops
Concatenated
Loops Unstructured
Loops
Why is loop testing important?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 17
White-box Testing Example
FindMean(float Mean, FILE ScoreFile)
{ SumOfScores = 0.0; NumberOfScores = 0; Mean = 0;
Read(ScoreFile, Score); /*Read in and sum the scores*/
while (! EOF(ScoreFile) {
if ( Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
NumberOfScores++;
}
Read(ScoreFile, Score);
}
/* Compute the mean and print the result */
if (NumberOfScores > 0 ) {
Mean = SumOfScores/NumberOfScores;
printf("The mean score is %f \n", Mean);
} else
printf("No scores found in file\n");
}
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 18
White-box Testing Example: Determining the Paths
FindMean (FILE ScoreFile)
{ float SumOfScores = 0.0;
int NumberOfScores = 0;
1
float Mean=0.0; float Score;
Read(ScoreFile, Score);
2 while (! EOF(ScoreFile) {
3 if (Score > 0.0 ) {
SumOfScores = SumOfScores + Score;
4
NumberOfScores++;
}
5
Read(ScoreFile, Score); 6
}
/* Compute the mean and print the result */
7 if (NumberOfScores > 0) {
Mean = SumOfScores / NumberOfScores;
printf(“ The mean score is %f\n”, Mean); 8
} else
printf (“No scores found in file\n”); 9
}
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 19
Constructing the Logic Flow Diagram
Start
F
2
T
3
T F
4 5
7
T F
8 9
Exit
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 20
Finding the Test Cases
Start
1
a (Covered by any data)
2
b (Data set must contain at least one value)
(Positive score) d 3
e (Negative score)
c 4 5
(Data set must h (Reached if either f or
be empty) f g
6 e is reached)
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 21
Self reading
Comparison of White & Black-box Testing 25.1.2002
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 22
The 4 Testing Steps
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 24
Self reading
Unit-testing Heuristics
1. Create unit tests as soon as object 4. Desk check your source code
design is completed: Reduces testing time
Black-box test: Test the use 5. Create a test harness
cases & functional model Test drivers and test stubs are
White-box test: Test the needed for integration testing
dynamic model 6. Describe the test oracle
Data-structure test: Test the Often the result of the first
object model successfully executed test
2. Develop the test cases 7. Execute the test cases
Goal: Find the minimal Don’t forget regression testing
number of test cases to cover Re-execute test cases every time
as many paths as possible a change is made.
Big cost -> what should be done?
3. Cross-check the test cases to
eliminate duplicates 8. Compare the results of the test with the
test oracle
Don't waste your time! Automate as much as possible
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 25
OOT Strategy
[Pressman]
dead nonworking
acct close acct
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 29
NFRs: Reliability [Chung, RE Lecture Notes]]
Counting Bugs
Bebugging Process - based on a Monte Carlo technique for statistical analysis of random events.
1. before testing, a known number of bugs (seeded bugs) are secretly inserted.
2. estimate the number of bugs in the system
3. remove (both known and new) bugs.
# of detected seeded bugs/ # of seeded bugs = # of detected bugs/ # of bugs in the system
# of bugs in the system = # of seeded bugs x # of detected bugs /# of detected seeded bugs
But, deadly bugs vs. insignifant ones; not all bugs are equally detectable; ( Suggestion [Musa87]:
Testing is still a black art, but many rules and heuristics are
available
Testing consists of component-testing (unit testing, integration
testing) and system testing, and …
OOT and architectural testing, still challenging
User-oriented reliability modeling and evaluation not adequate
Testing has its own lifecycle
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 31
Additional Slides
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 32
Terminology
There are many different types of errors and different ways how
we can deal with them.
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 33
Examples of Faults and Errors
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 34
Dealing with Errors
Verification:
Assumes hypothetical environment that does not match real
environment
Proof might be buggy (omits important constraints; simply wrong)
Modular redundancy:
Expensive
Declaring a bug to be a “feature”
Bad practice
Patching
Slows down performance
Testing (this lecture)
Testing is never good enough
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 35
Another View on How to Deal with Errors
Error prevention (before the system is released):
Use good programming methodology to reduce complexity
Use version control to prevent inconsistent system
Apply verification to prevent algorithmic bugs
Error detection (while system is running):
Testing: Create failures in a planned way
Debugging: Start with an unplanned failures
Monitoring: Deliver information about state. Find performance bugs
Error recovery (recover from failure once the system is released):
Data base systems (atomic transactions)
Modular redundancy
Recovery blocks
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 36
What is this?
A failure?
An error?
A fault?
Need to specify
the desired behavior first!
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 37
Erroneous State (“Error”)
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 38
Algorithmic Fault
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 39
Mechanical Fault
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 40
How do we deal with Errors and Faults?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 41
Verification?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 42
Modular Redundancy?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 43
Declaring the Bug
as a Feature?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 44
Patching?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 45
Testing?
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 46
Testing takes creativity
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 47
Test Cases
Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 48