How to Test More, Save Time, and Achieve Better Testing Results:

Software testing is an essential activity in the software development and maintenance life
cycles. It is a practice often used to decide and improve software quality.
Development is more systematic nowadays and organizations seek measures of testing
completeness and effectiveness to show test completion criteria. Of them all, coverage is
considered especially valuable.
Simply put, coverage is What are we testing and How much are we testing?
Test coverage helps monitor the quality of testing, and assists testers to create tests that
cover areas that are missing or not validated.

Most teams base their coverage calculations on functional requirements alone. It is also
fair because first and foremost an application should do what it is supposed to do. If not, its
speed or security or ease of use none of it matters.
However, if dedicated and independent non-functional testing teams are working on
performance, security, usability testing, etc., then they will have to track their requirements
all the way to execution through test coverage analytics.

Test Coverage and Code Coverage

Test coverage is often confused with Code Coverage. Even though the underlying principles
are the same, they are two different things.
Code Coverage really talks about unit testing practices that have to target all areas of the
code at least once and is done by developers.
Test Coverage, on the other hand, is testing every requirement at least once and is
obviously a QA team activity.

What really qualifies to be a covered requirement depends on the interpretation of each

team.
For example: Some teams call a requirement covered if there is at least one test case
against it. Sometimes, it is covered if at least one team member is assigned to it. Or, if all
the test cases associated with it are executed.
If there are 10 requirements and 100 tests created when these 100 tests target all

of the 10 requirements and dont leave out any we call this adequate test coverage
at the design level.
When only 80 of the created tests are executed and target only 6 of the

requirements. We say that 4 requirements are not covered even though 80% of
testing is done. This is coverage statistics at an execution level.
When only 90 tests relating to 8 requirements have assigned testers and the rest of

them are not, we say the test assignment coverage is 80% (8 out of 10
requirements).
It is also important as to when to calculate coverage.
If you do this too early in the process, you will see a lot of gaps because things are still
incomplete. So it is generally advised to wait until the Last Build i.e. Final Regression
Build. This will give a correct coverage of the Tests performed for the given Requirements.
Also read => What QA Tester Should Know About Release and Deployment
Management Process

How to adopt a proper Test Coverage method?

Awareness is everything:
First things first, the QA team must know how much work is involved and where the design
tasks are at. This way, they are going to be aware if more tests are to be added. To do this,
you could use an RTM as is the typical practice.
=> Check this article out to know more about it and how to use it:How to Create
Requirements Traceability Matrix Exact Process with Sample Template
Secondly, check resource assignment and test execution process to see if everything is
tested in the more effective manner.
-----------A table such as below can be helpful:
Test Type

Total
Cases

Executed
Cases

Functional

100

80

Status
50 pass ,
30 fail

Comments

Test Type

Total
Cases

Executed
Cases

Compatibility

100

50

45 pass,
5 fail

Usability

100

100

98 pass,
2 fail

Final
Regression

100

100

99 pass,
1 fail

Status

Comments

How to make sure everything is tested and in the best way

possible?

Every tester should be aware of the requirements and the testing methods
Prioritize your Requirements and focus your energy where it is most needed
Be informed about how a certain release is different from the previous one so you

can identify critical requirements more accurately and focus on maximum positive
coverage
Adapt Test Automation
Use Test Management tools to always stay in the know
Smart work assignment- Channel your best resources towards critical tasks and let

new testers explore more for a fresh perspective

Maintaining a checklist for all tasks and miscellaneous activities
Interact more with your Dev/Scrum/BA teams to get insights into the application

behaviour
Keep track of all your build cycles and fixes
Identify most impacting problems in the initial builds itself (when possible) so the
later ones can work for better stability and reach those areas blocked by prior
problems

Few critical areas and methods for effective testing are:

#1) Resource jumbling: Exchange tasks between your team members. This helps
improve engagement and prevent knowledge concentration.
#2) Compatibility coverage: Make sure you are aware and including the different
browsers and platforms to test your application.
#3) Ownership: Make testers accountable and give them a free rein (with monitoring and
support of course) for the module/task that they are working on. This helps build
responsibility and lets them try creative ways instead of following the beaten down road.
#4) Deadlines: Knowing the release deadlines prior to the commencement of testing
phase helps with effective planning
#5) Communication: Stay in touch with the dev and other teams in between release
cycles to know whats going on.
#6) Maintain an RTM: Acts as a good derivative to the Stakeholders or Clients, based on
which the release schedule can be confirmed

Advantages of Test Coverage awareness for a Tester :

It primarily helps with testing task prioritizing

It helps achieve 100% requirement coverage or in other words, it prevents

requirement leakage
Impacts Analysis becomes easier
Useful in determining the EXIT criteria
Enables a test lead to prepare a clear test closure report

Conclusion:
It is not always true that when you test more, the results are better. In fact, when you test
more with no apparent strategy, you probably will end up investing a lot of time.
With a more structured approach, an aim at 100% requirement coverage and effective
testing methods, you will not compromise on quality.

We hope the techniques outlined in this article will give you some pointers.
Traceability Matrix How to Create and Use It
Todays session is about an important QC tool, that is either over-simplified (read
overlooked) or over-emphasized Traceability Matrix(TM).
Most often, the making, reviewing or sharing of a Traceability Matrix is not one of the
primary QA process deliverables so it is not majorly concentrated on, thus causing the
under-emphasis. On the contrary, some clients expect a TM to reveal earth-shattering facets
about their product (under test) and are disappointed.
When used right, a Traceability Matrix can be your GPS for your QA journey.
As is a general practice at STH, we will see the What and How aspects about a TM in this
article.

What is a Traceability Matrix?

The focus of any testing engagement is and should be maximum test coverage. By
coverage, it simply means that we need to test everything there is to be tested. The aim of
any testing project should be 100% test coverage.
Requirements Traceability Matrix to begin with, establishes a way to make sure we place
checks on the coverage aspect. It helps in creating a snap shot to identify coverage gaps.

How to Create a Traceability Matrix?

To being with we need to know exactly what is it that needs to be tracked or traced.
Testers start writing their test scenarios/objectives and eventually the test cases based on
some input documents Business requirements document, Functional Specifications
document and Technical design document (optional).
Lets suppose, the following is our Business requirements document (BRD):
(Download this sample BRD in excel format)
(Click any image to enlarge)

The below is our Functional Specifications document (FSD) based on the interpretation of
the Business requirements document (BRD) and the adaptation of it to computer
applications. Ideally all the aspects of FSD need to be addressed in the BRD. But for
simplicitys sake I have only used the points 1 and 2.
Sample FSD from Above BRD: (Download this sample FSD in excel format)

Note: the BRD and FSD are not documented by QA teams. We are merely, the consumers
of the documents along with the other projects teams.
Based on the above two input documents, as the QA team we came up with the below list
high-level scenarios for us to test.

Sample Test Scenarios from the Above BRD and FSD: (Download this sample test
Scenarios file)

Once we arrive here, now would be a good time to start creating the requirements
traceability matrix.
I personally prefer a very simple excel sheet with columns for each document that we wish
to track. Since the business requirements and functional requirements are not numbered
uniquely we are going to use the section numbers in the document to track. (You can
choose to track based on line numbers or bulleted-point numbers etc. depending on what
makes most sense for your case in particular.)
Here is how a simple Traceability Matrix would look for our example:

The above document establishes a trace between, the BRD to FSD and eventually to the test
scenarios. By creating a document like this, we can make sure every aspect of the initial
requirements have been taken into consideration by the testing team for creating their test
suites.
You can leave it this way. However, in order to make it more readable, I prefer including the
section names. This will enhance understanding when this document is shared with the
client or any other teams. The outcome is as below:

------------

Again, the choice to use the former format or the later is yours.
This is the preliminary version of your TM but generally does not serve its purpose when you
stop here. Maximum benefits can be reaped from it when you extrapolate it all the
way to defects.
Lets see how.
For each test scenario that you came up with, you are going to have at least 1 or more test
cases. So, include another column when you get there and write the test case IDs as shows
below:

At this stage, the Traceability Matrix can be used to find gaps. For example, in the
above Traceability Matrix you see that there are no test cases written for FSD section 1.2.
As a general rule, any empty spaces in the Traceability Matrix are potential areas for
investigation. So a gap like this can mean one of the two things:
1. The test team has somehow missed considering the Existing user functionality.
2. The Existing user functionality has been deferred to later or removed from the
applications functionality requirements. In this case, the TM shows an inconsistency

in the FSD or BRD which means that an update on FSD and/or BRD documents
should be done.
If it is scenario 1, it will indicate the places where test team needs to work some more to
ensure 100% coverage.
In scenarios 2, TM not just shows gaps it points to incorrect documentation that needs
immediate correction.
Let us now expand the TM to include test case execution status and defects.
The below version of the Traceability Matrix is generally prepared during or after
test execution:

Download requirements traceability matrix template here:Traceability Matrix in

excel format

Important Points to Note About Traceability Matrix

The following are the important points to note about this version of the Traceability Matrix:
1) The execution status is also displayed. During execution, it gives a consolidated
snapshot of how work is progressing.
2) Defects: When this column is used to establish the backward traceability we can tell that
the New user functionality is the most flawed. Instead of reporting that so and so test
cases failed, TM provides a transparency back to the business requirement that has most
defects thus show casing the Quality in terms of what the client desires.
3) As a further step, you can color code the defect ID to represent their states. For
example, defect ID in red can mean it is still Open, in green can mean it is closed. When
this is done, the TM works as a health check report displaying the status of the defects
corresponding to a certain BRD or FSD functionality is being open or closed.

4) If there is a technical design document or use cases or any other artifacts that you would
like to track you can always expand the above created document to suit your needs by
adding additional columns.
To sum up, a requirements traceability Matrix helps in:
1. Ensuring 100% test coverage
2. Showing requirement/document inconsistencies
3. Displaying the overall defect/execution status with focus on business requirements.
4. If a certain business and/or functional requirement were to change, a TM helps
estimate or analyze the impact on the QA teams work in terms of
revisiting/reworking on the test cases.
Additionally,
1. A TM is not a manual testing specific tool, it can be used for automation projects as
well. For an automation project, the test case ID can indicate the automation test
script name.
2. It is also not a tool that can be used just by the QAs. The development team can use
the same to map BRD/FSD requirements to blocks/units/conditions of code created
to make sure all the requirements are developed.
3. Test management tools like HP ALM come with the inbuilt traceability feature.
An important point to note is that, the way you maintain and update your Traceability
Matrix determines the effectiveness of its use. If not updated often or updated incorrectly
the tool is a burden instead of being a help and creates the impression that the tool by itself
it not worthy of using.
Have you created a Traceability Matrix in your projects? How similar or different is
it from what we have created in this article? Please share your experiences,
comments, thoughts and feedback on this article through your comments.