CONTINUOUS TESTING

IN PRACTICE
Completing the Continuous
Delivery Process
Table of Contents

INTRODUCTION: 1
CONTINUOUS DELIVERY IS THE NORMAL

PART 1:
2
WHY CONTINUOUS DELIVERY?

PART 2:
4
TESTING: THE FINAL BARRIER FOR CONTINUOUS DELIVERY

PART 3:
5
CONTINUOUS DELIVERY WITHOUT CONTINUOUS TESTING

PART 4:
7
HOW CONTINUOUS TESTING FACILITIES CONTINUOUS DELIVERY

PART 5:
11
CONTINUOUS TESTING IN PRECTICE: REQUIREMENTS CHECKLIST

CONCLUSION 13

REFERENCES 14
 INTRODUCTION:
CONTINUOUS DELIVERY IS THE NEW NORMAL

Continuous Delivery (CD) is rapidly emerging as the new normal in software development, with

approximately 80% of SaaS companies and 51% of non-SaaS companies adopting this practice1.

Dominant online companies like Amazon, Facebook, and Google have been implementing

aggressive Continuous Delivery workows for years, and deploying release to production

numerous times a week or even multiple times a day. As such, companies failing to make the leap

to Continuous Delivery are falling behind and putting themselves at a signicant competitive

disadvantage.

Despite the widespread, rapid, and perpetually increasing adoption of this practice, it is

apparent that there are deep-rooted barriers preventing companies from the actualization of a

True Continuous Delivery process. Today, the toughest and most challenging barrier is testing.

This whitepaper covers the rise of Continuous Delivery, the key barriers and pain points leading

companies towards inevitable failure, and how to overcome these challenges through

Continuous Testing.

1
 ?
PART 1:
WHY CONTINUOUS DELIVERY?

Jez Humble, co-author of the Jolt Award winning book on Continuous Delivery, describes the
practice as follows:

Continuous delivery is a set of principles and practices to reduce the cost, time, and risk of
delivering incremental changes to users.2

Continuous Delivery is the Agile Ages answer to traditional waterfall methodologies - the
dominant mode just a few years ago.

In the Waterfall Age, product iterations were linear and sequential, moving from product
marketing specication, through to design, development, QA/testing, and nally deployment
and maintenance. As the name Waterfall indicates, the process can only move forward. Once
a phase in the lifecycle is completed, it proceeds to the next phase and its almost impossible to
turn back.

2
There are numerous aws with this approach. By the time an application gets to testing, its very
difcult to go back and resolve an issue that was caused in the design or implementation stage.
As such, delays are common and application releases and updates usually take many months to
complete.

In todays Agile Age, such timeframes are unacceptable.

Continuous Delivery plays a critical role in todays Agile Age. Without it, there is no agility. In an
ideal Continuous Delivery process, the test, support and development teams work closely and
collaboratively to automate and streamline the build, test and release process. Iterations are
short, manual handoffs are eliminated, risks are minimized, and releases are fast and frequent.

As a result, a successful Continuous Delivery process can reduce the time-to-market for new
releases from several months to a matter of days or even hours.

3
 PART 2:
TESTING: CONTINUOUS DELIVERYS FINAL BARRIER

The case for Continuous Delivery is undisputable.

However, a smooth, error-free and automated Continuous Delivery process is difcult to

actualize.

Key barriers have been overcome through the widespread adoption of Continuous Integration
(CI) and Continuous Deployment. Continuous Integration, enabled by tools like Jenkins,
TeamCity, and Travis, gave IT organizations the rst component of Continuous Delivery.
Developers integrate code into a shared repository and verify each check-in with an automated
build, enabling the rapid detection and resolution of problems.

The second component, Continuous Deployment, could be considered as an extension of

Continuous Integration. IT teams automate the deployment process, greatly improving
deployment time and reducing human error, while minimizing the need for manual work.

Despite these considerable developments, a crucial component in the Continuous Delivery

process is diminishing its effectiveness. To date, software testing remains a major source of
friction. It has failed to match other advancements made in the software development lifecycle,
yet its importance just keeps on increasing.

As software changes over time, the number of tests required to ensure its quality increases
signicantly. IT organizations do not have the time or manpower to run all the necessary tests
every time a new feature is added. As a result, most companies only execute a small fraction of
these required tests resulting in software quality compromises or an increased
time-to-release.

As such, any Continuous Delivery process is ultimately bound to fail.

4
 PART 3:
WITHOUT CONTINUOUS TESTING:
WHERE CONTINUOUS DELIVERY FAILS

Testing should be a cross-functional operation which involves the entire technical team
throughout the product lifecycle.

To demonstrate the pain that testing can bring at every step of the Continuous Delivery process,
this paper examines the product pipeline of a mid-sized organization adopting the practice.

A CONTINUOUS DELIVERY SCENARIO

Engineering teams develop software modules and commit their code regularly into a version
control tool like GitHub.

Devop teams use a Continuous Integration solution such as Jenkins to run daily builds,
including various post-build tests, to validate the quality of the build.

QA testers validate release candidates prior to their deployment in production with a handful of
automation tools (e.g. JMeter, Loadrunner).

Operations validate releases that were just deployed in production, and monitor the application
further in a post-production environment.

THE ABOVE PROCESS IS OPERATED BY NUMEROUS ROLES WITHIN THE ORGANIZATION

AND REQUIRE MANY TEST POINTS, INCLUDING:

Post-build tests through unit/module tests

QA tests through performance, functional, UI, soak, and regression tests
Deployment acceptance tests
Post-production monitoring/testing

5
Below is a typical example of how these tests points are executed by various roles throughout
the organization.

Developers
Developers perform
unit & module testing

DevOps
Operations/DevOps run a set
of tests on the build (such as
in Continuous Integration)

QA
QA run a more
comprehensive & holistic
set of pre-production tests

Operations
Operations run the same tests
upon deployment and in
post-production - and a monitoring
solution to test on an ongoing basis.

As the example above illustrates, different tools are used by each role in the organization to
perform a wide range of tests. When it comes to testing, the cohesiveness and consistency
between each role is little to none. Each tool needs a script to perform a test, and the script
usually requires the application to be reverse-engineered. For example: it may need to record
the operation and convert this recording into a script the testing tool can understand and
execute.

This issue is exacerbated when the application is updated regularly, as the scripts also need to be
updated. This workow is extremely hard to maintain and will ultimately result in inadequate
testing and a longer time to release.

The above process is incompatible with a successful Continuous Delivery Process.

6
 PART 4:
HOW CONTINUOUS TESTING FACILITATES
CONTINUOUS DELIVERY

Continuous Testing involves applying the methods and concepts of agile development to the
testing and QA process, resulting in a more efcient testing process.

For Continuous Testing to succeed, it requires an automated end-to-end testing solution that
integrates into the existing development process, eliminating errors and facilitating true
continuity throughout the development lifecycle.

This paper has already covered the roles and test points in a typical Continuous Delivery
process.

Now, we present these same roles and test points when Continuous Testing is properly
integrated into Continuous Delivery.

DEVELOPERS
Testing must start with the developers, testing the performance and functionality of their own
code by creating and running tests using open source tools such as JMeter or Selenium,
commercial tools like Perfecto mobile or code, i.e Ruby or Python. Tests include API, Functional,
Performance, UI, and more. Regression tests can also be added to the set to ensure functionality
is not compromised and specic scenarios are tested.

Test congurations need to be committed with every code commit. The test conguration
references all tests associated with a certain version of a software module.

To ensure this will be a widely adopted practice, the creation and running of such tests should be
easy, fast, and require minimum prior education and/or experience. Developers can successfully
create and execute these tests without help from consultants, professional services and support
development workows, such as version control, bug tracking systems etc.

DEVOPS (CONTINUOUS INTEGRATION)

DevOps do not need to understand the internals of each software module. As such, they only
need to congure the testing element of any project once. Any post-build test should consider
the modules test conguration and be able to run the right set of tests associated with the build
in the CI environment. Tests run in parallel and end as soon as possible with a clear pass/fail
indication, allowing the process to continue smoothly even when the number of tests increase.

7
Once the CI environment is congured to use the right set of tests, developers receive an
automatic message whenever a build fails. They receive sufcient test artifacts to identify the
cause of failure and the tools to be able to keep running the tests until a pass indication is given.
This process should be as fully automated as possible.

Another important item to consider is the speed in which execution needs to happen. Not only is
it important to ensure comprehensive coverage when testing, but it needs to be accomplished
within a reasonable timeframe. Of course, the denition of a reasonable timeframe is purely
subjective. Therefore, its best to attempt to complete the CI process as fast as possible.

QA (PRE-PRODUCTION)
QA needs to validate a release candidate prior to production. As such, they need to run
comprehensive tests to ensure a release candidate is ready for deployment. There are numerous
tests, which include: regression, functional, load, soak, and more. Today, QA testers build scripts
and use tools according to a predened test plan.

As there are sets of tests associated with the release candidates, the QA tester doesnt need to
understand the internals of each release. Any pre-production test should consider the set of
module test congurations and be able to run the right set of tests associated with the release
candidate in the pre-production environment. Again, tests should run in parallel and end as soon
as possible with a clear pass/fail indication, allowing the process to continue smoothly as the
number of tests increase. When congured correctly, they can run continuously and without
human intervention. When theres a failure, the developer should receive a message with the
artifacts and ability to keep running the tests until a pass indication is given.

OPERATIONS (CONTINUOUS DEPLOYMENT)

Every release needs to pass an acceptance test upon deployment. Once more, operations
managers can leverage an existing set of tests which have already been created, used and, most
importantly, are associated with a specic release.

OPERATIONS (POST-PRODUCTION MONITORING)

In a production environment, operations managers use a variety of monitoring tools to test the
application on an ongoing basis. As such, this process is often simply referred to as monitoring.

8
In a Continuous Delivery process, monitoring is yet another form of Continuous Testing. In a
post-production environment, Operations can leverage the set of tests associated with a specic
release to monitor the health of the application on an ongoing basis. There is no need to reverse
engineer, create a script or adapt the script with every change. A comprehensive set of tests can
run 24/7 and send automatic alerts on failures.

END TO END CONTINUOUS TESTING

When implemented correctly, Continuous Testing facilitates true continuity in the delivery
cycle. Even tasks previously considered as management tasks, such as multi-geographical load
tests, can now be performed as just another automated step in the delivery process.

Testing that was once cumbersome, complicated and time consuming can be completely
streamlined, ensuring a predictable time-to-release and superior software quality.

9
THE TESTING PROCESS - FROM DEVELOPMENT TO OPERATIONS

People Systems Testing Alert

Version Control
Repository
Commit Code +
Test Conguration

Developer

Continuous Integration Leverage Test

Congure Conguration

Devop

Re-use Build
Comprehensive Testing Test Conguration
Test / Automate

QA

Continuous Deployment
Re-use Release
Acceptance Test Test Conguration

OPs

Post-Production Re-use Release

Monitoring Test Conguration
Congure

OPs

Red Arrow = Test Failed Green Arrow = Test Succeeded Blue Arrow = Sent to Testing

10
 PART 5:
CONTINUOUS TESTING IN PRACTICE: REQUIREMENTS CHECKLIST

A successful continuous testing process requires the right tools and processes to be adopted
across the various teams within the organization. Without the proper tools in place, the process
will fail.

BlazeMeter developed the following checklist based on its experience helping clients adopt
continuous testing. The checklist provides the features and functionality required for the
continuous testing process to succeed, enabling teams to run tests at scale while also providing
smooth integrations with various CI environments.

TOOLS, SCRIPTING AND OVERALL TEST CREATION

Support for both manual and automated testing
The provision of libraries and APIs to create homegrown tests using common DSLs.
Ability to activate commercial tools (such as Perfecto Mobile)
Support of JMeter and API tests (BlazeMeter)
The ability to create browser-based tests (Selenium Builder, BlazeMeter Chrome Plugin)
Support of Selenium and Appium tests (Sauce Labs)

PROVISIONING & RESOURCES

The option to run the same sets of tests with different provisioning (i.e. a developer can run
a module test from nbehind the rewall with local resources, and the same test can run
in production from multiple geographical locations on the public cloud)
The ability to easily adopt a new provisioning scheme when a test conguration is executed
in a different environment (e.g. Dev, CI, Pre-Prod, Post-Prod). These schemes should include:
The developers laptop for debugging and troubleshooting
A private cloud of resources available on-premise and protected behind the rewall
A public cloud of unlimited resources available on multiple geographical locations
Sufcient resources in the public and private cloud, enabling organizations to run
multiple tests in parallel with zero time to test
The option for users to dene which OS and Browser combinations they want to execute
the test against
The ability to start testing as part of the CI process (i.e. Jenkins, Travis, TeamCity, CircleCI etc.)

11
AUTOMATION
The ability to run any test conguration, a test, or a set of tests using simple API calls
The option to run as many tests as required in parallel
Availability of all test data via a REST API

ON-DEMAND TESTING AND RE-RUNNING FAILED USE CASES

The ability to run any type of test on-demand. This is critical for test development,
debugging, troubleshooting and identifying the root cause of a failure
The option to automate test executions and run the same test on-demand

MODULES, BUILDS, RELEASE CANDIDATES, RELEASES AND PRODUCTION

The ability to combine different conguration fragments into one test conguration in order
to comprehensively test build, release, and production snapshots.

VERSION CONTROL FRIENDLY INCREMENTAL TESTING

The ability to support version control, incremental testing and associate test congurations,
sets of tests, and tests with versions (e.g code, build, RC, releases).

AUTOMATIC FAILURE ALERTS

Automatic indications of failures, with an alerting scheme per developer, module and project
The ability to gather all test artifacts and immediately send to relevant people
The option to run the failed tests again to identify the root cause

REPORTING
A seamless integration with existing reporting solutions (e.g. Jenkins Performance Trend)
The option to group tests by builds
The ability to give jobs a pass or fail status and group accordingly
Pass/fail trend reports
Deep dive reports for diagnostics and analysis, which can overlay data from third
party systems (e.g. NewRelic, CloudWatch) for a comprehensive picture.

12
 CONCLUSION

To compete in todays environment, its critical to act and react fast in order to keep up with the
rapid pace of change. As such, Continuous Delivery is more than just a new working process, its
a strategic capability that provides organizations with a strong competitive advantage. This
advantage is rapidly becoming the new benchmark; companies that dont adopt agile processes
like Continuous Delivery risk falling beneath todays standards of acceptability.

However, the path to a successful Continuous Delivery process is still incomplete. Continuous
Integration paved the way, rapidly followed by Continuous Deployment. Yet testing has
remained the roadblock in Continuous Deliverys path, thwarting attempts to reduce
time-to-market without quality compromises.

The only viable solution for IT organizations is to integrate a fully automated Continuous Testing
process into their software development lifecycle. There is no room in agile processes for
manual handoffs and quality compromises - they must be fully automated and predictable. As
such, Continuous Testing will become increasingly critical as an inherent piece of any successful
Continuous Delivery process.

FIND OUT MORE:

To learn more about how BlazeMeter can help you with Continuous Testing, contact us at:
info@blazemeter.com, request a demo, call us on 1.855.445.2285, at or
visit www.blazemeter.com

13
 REFERENCES

1. Perforce Software Study

2. Thoughtworks - The Case for Continuous Delivery

3. TechTarget - Waterfall Model

4. Wikipedia - Waterfall Model

5. StickyMinds - Testing in the Agile Age

6. BlazeMeter

7. Sauce Labs

2390 El Camino Real Palo Alto, CA, 94036, USA www.blazemeter.com