ITECH7409: Software Testing

Lecture 2. Software Development and Testing

This lecture
• Software Testing: When and How?
• Cost of Testing.
• Waterfall model.
• Requirements Specification.
• Software Reliability Specification.
• Spiral model.
• Agile approaches.
Software testing: When and How?
Software testing:
• Is a process of checking software to discover if it is suitable to the
task for which it is being developed. I.e., are the customer
requirements met correctly?
• Is the software free of defects/ bugs? The tester should find as many
defects as practical and have them corrected.

Or put more succinctly:

• Verification: Are we building the product right ?
• Validation: Are we building the right product?
Software testing: When?
Need to start with customer requirements. It is important to start
planning for testing when the functional requirements are formed.

• Testing is the most time consuming and expensive part of any

software development.
• The risk of not testing is even more expensive.
Cost of Testing.
• Note: Testing can be done at any stage of the software development,
but the cheapest cost is achieved at the earliest stage of
development, the Requirements stage.
• If there is too little testing effort early in the development, then the
cost of testing increases. In fact, testing after development is
prohibitively expensive.
• About two thirds of defects are introduced before the end of design.
Traditional testing can find only 80% of the defects.
Value of defects
Percentage of Defects
$14,000

85% % Defects
Introduced in
this phase

% Defects
42% found in
in this phase
$1000
$ Cost to
$130 $250 repair defect
$25
in this phase

Design Compile Pre-prod Post

& Code or Bind Integration Release
Source: Applied Software Measurement, Capers Jones, 1996
Software Development Life Cycle (SDLC).
Waterfall Model (Royce,1970)
• Requirements Specification: system’s services, constraints and goals
established in consultation with users.
• Analysis & Design: decides necessary hardware & software
components and overall architecture of the proposed system.
• Coding & Implementation: software design is realized as a set of
programs .
• Testing: all programs are tested independently, then with each other
on the relevant platforms so that software may be delivered.
• Support & Maintenance: system installed and used. Maintenance is
needed to correct any previously undiscovered errors.
Documenting the SDLC
The Waterfall model for software development is a very widely adopted model as it
splits the process into easily documented phases (Sommerville, 1996). The output
documents allow users and developers alike to track process of system generation.
Phase Output Documents
Requirement Specification Feasibility study. Requirements document.
Analysis & Design Functional specification
Architectural specification
Interface specification
Design specification
Coding & Implementation Program code
Testing Unit test report
Integration test plan
Integration test report
System test plan
Acceptance test plan
Support & Maintenance User manual
Final system documentation
V-model: Validation & Verification
Validation
Requirements Analysis Acceptance Test

System Analysis System Test

System Design Integration Test

Verification Implementation Module Test

V-Model
Requirements Specification
For large software systems, the Requirements Specification is usually
broken into up to 4 separate steps.
1. Feasibility study: exploration of the proposed system and the cost-
effectiveness of undertaking its development.
2. Requirements analysis: an analysis of the current business processes
and models of these for user and developer understanding.
3. Requirements definition: statement of exactly the nature of what
the proposed system must do
4. Requirements specification details: detailed description of the
proposed system to become the contract between customer and
the developer
Software Requirements Specification (SRS)
The system requirements are expressed in a Software Requirements
Specification document (SRS). The SRS is usually structured with
chapters to include:
1. Introduction
2. Glossary: listing of technical terms with definitions
3. System models: illustration of the relationship between the system
components and the system and the environment. These can be
object models, data-flow models and semantic data models
4. Non-functional requirements: listing of the constraints and the
restrictions imposed on the software which may include specific
data representation, response time and memory requirements. They
also include product and process standards that must be followed.
Software Requirements Specification (SRS)
5. Functional requirements: listing of services required by the user that
the software system will do.
6. Requirements specification: more detailed listing of the functional
requirements and will include descriptions of various routines
and sub-routines.
7. Other chapters may include descriptions of system evolution,
hardware and database requirements.
And sometimes:
8. Software reliability specification: identification of the types of failure
that may occur and a measure (e.g. 1 incident per 100 days) will be
tolerated.
Software Reliability
• In testing it is not possible to know if you have discovered all possible
inputs into a program or system that will result in faults or failures
(Sommerville, 1996).
• Reliability is about finding as many inputs as possible, however the
very high costs of reliability achievement often means that it may be
more cost effective to accept unreliability and pay for failure costs.
Software Reliability Specification
Many SRS documents have loose subjective statements concerning
software reliability specification, such as,
• “the software shall be as reliable as possible”.
• “the software will exhibit no more than N faults per 1000 lines of
code”.

In both cases, the meaning is not clear. For large systems, different
types of failure need to be identified and then quantified, that is have a
measure of acceptance.
Software Reliability Specification
• It is important for software systems to be reliable, particularly when
the costs of system failure can exceed the cost of developing the
system.
• Software reliability needs to be quantified, that is measured, as it is
possible for a system to contain faults whilst the software is still
considered by users to be reliable.
• The type of metric chosen depends on the type of system or sub-
system under examination.
• It is possible to statistically test systems to gauge metric values. Also,
there are models of reliability that can be used to estimate when
enough testing has been done.
Statistical Testing
Statistical testing is used for reliability assessment on the assumption that
test data set reflects the use of the software.
1. Determine operational profile of the software.
• May be straightforward for ‘normal’ inputs but it is difficult to predict ‘unlikely’
inputs and to create test data for them.
2. Generate a set of test data corresponding to this profile.
• Where possible automatically generate test data.
3. Apply tests, measuring amount of execution time between each failure.
4. After a statistically valid number of tests have been executed, reliability
can be measured.
Reliability models may be used to predict when a required level of
reliability will be achieved.
Software reliability measures
Software reliability measures
Waterfall Model: Advantages & Disadvantages
Advantages
• Works from definitions of requirements (broad descriptions) to more
detailed activities.
• Easily imported into project management tools.

Disadvantages
• Sequential nature precludes revisiting an earlier activity that is
considered finished. Each step finishes with a milestone so it is
probable that a defect generated in a particular stage could go
undetected until after the milestone has been reached.
• Testing is done quite late in project development.
• Cannot accommodate customer changes to requirements.
Boehm’s spiral model of software development
Boehm’s spiral model of software development
Each loop in the spiral is split into 4 sectors:
1. Objective setting for the loop where planning takes place,
constraints and risks are identified.
2. Risk assessment undertaken to reduce impact of risk, e.g. if
requirements are inappropriate develop a prototype to explore.
3. Development and validation by exploring prototype behaviour.
4. Review work done and plan future activities, continue to step 1.

Note: with each successive spiral the level of detail of the project
increases and becomes more complex
Boehm’s spiral model of software development
Key feature of this model is the use of prototypes, continuous
testing and risk evaluation.
• Throwaway prototyping refers to the creation of a model that will
eventually be discarded rather than becoming part of the final
delivered software.
• Evolutionary prototyping refers to the attempt to build a very
robust prototype in a structured manner and constantly refine it.
• Incremental prototyping is where the final product is built as
separate prototypes. At the end, the separate prototypes are
merged in an overall design.
• Extreme prototyping is a development process is used especially
for developing web applications.
 Prototyping
REQUIREMENTS
• A good approach for small to DETERMINATION
BY CUSTOMER
medium-sized projects. PROTOTYPE
• Very important: customer involvement. DESIGM

PROTOTYPE
IMPLEMENTATION

PROTOTYPE
EVALUATION
BY CUSTOMER

NO REQUIREMENTS FOR
REQUIREMENTS CORRECTIONS, CHANGES
FULFILLED ? AND ADDITIONS
YES
SYSTEM TESTS AND
ACCEPTANCE TESTS

SYSTEM CONVERSION

SYSTEM OPERATION
AND MAINTENANCE
Prototyping: Advantages

• Shorter development process.

• Substantial savings in development resources (time) resulting in
reduced costs.
• Improved and increased user involvement.
• Better fit to customer requirements and reduced risk of project
failure.
• Easier and faster user comprehension of new system.
Prototyping: Disadvantages

• User confusion of prototype and finished system.

• Developer misunderstanding of user objectives.
• Developer attachment to prototype.
• Excessive development time of the prototype.
• Expense of implementing prototyping.
•Less flexibility and adaptability to changes and additions.
•Reduced preparation for unexpected instances of failure.
Agile Models of Software Development

• Several methods including XP and Scrum. In common, they use cross-

functional teams of developers, testers and customers.
• All team members are able to take design or implementation
decisions in the process.
• Work involves numerous rapid iterations or Sprints, with user
feedback, instead of larger and less frequent deliveries.
• Each Sprint has high priority requirements to attend to and is released
to customers immediately.
Agile Software Development
Agile Software Development
• In multiple ‘software increments’ delivers an operational prototype to
collect customer feedback for adaption.
• Is driven by customer descriptions of what is required (scenarios or
stories). Some assumptions:
• Recognizes that plans are short-lived (some requirements will
persist, some will change. Customer priorities may change).
• Develops software iteratively with a heavy emphasis on
construction activities (design and construction are interleaved).
• Analysis, design, construction and testing are not predictable.
• Thus, has to adapt as changes occur due to unpredictability.
XP is an Example of Agile Development
• A set of user stories or business stories are used to define the
proposed system’s functionalities.
• Users/customers on the team give constant feedback to the
development team, describing stories and executing acceptance
testing of code.
• Development team uses pair programming and designs the test
system even before the code is designed.
• Frequent integration of components, even doing so several times a
day.
• Always opts for the simplest solution at hand.
XP: Advantages & Disadvantages
Advantages:
• Frequent interaction between customers, testers and developers.
• Quick delivery of working software even if it has a limited subset of
functionalities.
Disadvantages:
• Finding customers and team members who are always available for
interaction is most difficult aspect.
• Testing revolves around the “story” rather than being an independent
method of verification.
• Difficult to estimate overall cost of project and its likely duration.
• Coding can be ‘quick and dirty’ as developers opt for simplest version. This
makes code less likely to have documentation and impacts the
maintainability of the system.
Comparison of Development Cycles