To improve their test environments and application quality, organizations are turning to the cloud for on-demand provisioning, as well as build and deployment automation.

1. • Cognizant Reports
Why On-Demand Provisioning Enables
Tighter Alignment of Test and Production
Environments
Executive Summary environment downtime and inaccurate con-
Steve Smith looks like a worried man, moving rest- figuration of test environments impact testing
lessly in his chair and making frantic phone calls effectiveness. These result in application roll-out
late on a Friday evening, inquiring about what delays and test managers signing off on projects
went wrong with the software his team had devel- with caveats, leading to applications misbehaving
oped. All seemed well during testing, but errors once put into production.
surfaced as soon as the application was deployed,
leaving Steve and his team in a quandary. Organizations are currently exploring various
ways to mitigate these risks. Attempts to use
Tales such as Steve’s abound in the software test- virtualization to overcome test environment
ing arena. From banking applications to mobile challenges have met with limited success.
phones and airport systems, software defects Also on the rise is “on-demand provisioning”
affect organizations’ business-critical and cus- via the cloud, as well as build and deployment
tomer-facing applications, impacting their daily automation. Our experience shows that
business operations and resulting in huge losses. cloud-based test environments offer bet-
According to the National Institute of Standards ter results by providing testing teams with
and Technology (NIST), software flaws cost the on-demand capacity. Cloud-based offerings
U.S. economy a whopping $60 billion annually. also enable organizations to generate standard
builds that are more closely and cost-effectively
One of the primary reasons for such unpleasant aligned with real production environments.
situations — in which errors are discovered in
production or only after end-users are affected On-demand provisioning has much to offer
— is low-quality testing, often impacted by organizations seeking to instill greater discipline
poorly built test environments. Test environ- in managing builds and elevating code quality.
ments rarely match production environments Application quality depends on test environ-
due to the huge cost and effort involved in mim- ments that support functional testing require-
icking them. Further, issues such as unplanned ments; moreover, it is also important to have
cognizant reports | december 2011

2. a production-like environment to complete the applications, it may not be possible to predict all
testing process without an exception. This syn- the likely scenarios in which the application will
chronization of test and production environ- be deployed.
ments is achieved by building a service frame-
work around the test environment, supported by The conventional approach of manually creating
cloud-based computing grids. This approach, in in-house testing environments that fully mirror
our view, has already delivered great benefits to these complexities and multiplicities consumes
organizations that are piloting and enacting full- huge capital and resources. This has caused
fledged on-demand test environment strategies. application developers to take the risky route
of conducting testing in their production envi-
ronments. However, the Sarbanes–Oxley Act, as
Test vs. Production Environments well as other regulations, restricts access to
According to a 2008 Aberdeen study, only 42% production systems, which has forced organi-
of over 200 organizations surveyed were happy zations across industries to conduct testing
with the performance of their business-critical in separate environments. Further, it is not
applications.1 Primary reasons for this include uncommon for organizations that cannot afford
the inability to identify issues before end-users large-scale test environments to use sizing/
are impacted and the challenges of testing appli- extrapolation techniques during performance
cation performance in pre-production stages testing to understand how an application behaves
(see Figure 1). Sadly, our experience with com- in the real world. This approach, however, does
panies across industries indicates that not much not guarantee success.
has changed. At a time when companies are
laser-focused on operational cost-cutting and Poorly built test environments often limit the
time-to-market acceleration, software defects ability of testing teams to test applications
that emerge in production environments can for various scenarios. Test environments differ
severely undercut financial performance and from production environments in terms of OS,
cause reputational damage. patch levels, software version, configuration,
etc. The wider the gap between test and produc-
With applications becoming increasingly complex, tion environments, the greater the chance of an
the scope of testing has expanded tremendously, application misbehaving once deployed or for a
creating unforeseen challenges for testing teams. defect to leak into live systems. The following are
For instance, to assess their likely performance among the most common issues that afflict test
in real-time, Web and mobile applications must environments:
be tested for multiple operating systems and
updates, multiple browser platforms and versions, Creating test environments can be
different types of hardware and a large number time-consuming and delay-prone. It can
of concurrent users. Further, in the case of new sometimes take weeks or month to set up
Top Challenges for Application Performance Management
Inability to identify issues before
57%
end-users are impacted
Increase in complexity of applications 51%
Inability to measure SLAs around
37%
application performance
Inability to test application 34%
performance in pre-production stages
Increase in complexity of
32%
network traffic
Source: Aberdeen Group
Figure 1
cognizant reports 2

3. a simple test environment due to delays in opportunity to maintain an up-to-date test
finding servers, procuring licenses for pro- environment.
grams and testing tools and installing them. Loosely managed asset control necessi-
Roughly 30% of defects arise due to tates undue investments in the infrastruc-
inaccurate test environment configurations, ture. Inadequate access control reduces
studies show. Further, configuration changes confidence in the state of the environment
made in response to errors often go untracked, and its configuration control. Together, this
causing errors when applications are moved adds to the cost of the project and to the risk-
into production. based sign-off of the test plan.
Scheduling of the test environment is
crucial, especially when environments
are shared by two projects. If not properly The Impact
managed, this can lead to unnecessary delays. Test environment issues can impact delivery
Testing teams often clone production schedules and result in the risk of subsequent
databases and write extract scripts to software failures, as well as additional business
create test data. This approach is resource expenditures. It is estimated that 40% of the
consumptive and error prone due to a lack of effort involved in a software development lifecy-
audit trails. cle goes toward managing defects that arise due
Simulating the production issue in the to environment-related issues.2 The ever-present
test environment to identify the root cause pressure to meet service level agreements (SLAs)
is challenging due to a lack of test environ- can force IT managers to resort to risk-based
ment alignment with production versions at all sign-off on a product/application release. This
system component layers. means that the product/application is tested for
Test environment downtime and changes only those risks that have a high probability of
are often not controlled and communicated occurrence.
properly. This poses challenges in managing
testing projects and achieving KPIs. This risked-based sign-off can have the
Lack of a single view makes the environ- unintended consequence of putting the
ment less flexible to support changes organization’s reputation and financial condi-
due to code modifications during the tion at risk (i.e., errors that cause customers to
test cycle. Further, it is difficult to tune a question product quality — or, worse, cause
complex test environment to suit the advance- monetary harm — can be grounds for lawsuits).
ments in software development methods such Further, with the cost of fixing defects in pro-
as Agile. duction being significantly greater than in the
The test environment refresh is not early stages (see sidebar below), testing teams
considered while planning for the test must ensure that defects do not pass through to
project execution phase. This narrows the later stages.
Cost of Fixing Defects
According to a study by dynaTrace on application performance management, about 50% of
software problems are discovered in production only after they have affected end users and
that resolving these issues can involve huge costs.3
Business losses from system downtime can exceed $1 million per hour. The majority of defects
creep into code during the early stages of the development lifecycle. The longer a defect goes
undetected, the greater the costs of fixing it. According to studies, the ratio of the cost to fix a
defect at different stages of a project is 1:10:100. This means the cost of fixing a coding defect in
production is 10 times the cost of fixing it during testing and 100 times the corresponding cost of
fixing it during the design or requirements phase (see Figure 2, next page). According to Gartner,
“The cost of fixing defects ranges from a low of approximately $70 (cost to fix a defect at the
requirements phase) to a high of $14,000 (cost to fix a defect in production).”4
cognizant reports 3

4. Index of the Relative Cost of Fixing Defects
110
90
70
50
30
10
0
Requirements Design Code Test Production
Source: “The Economics of Testing,” www.riceconsulting.com
Note: Analysis based on base cost of fixing a defect in the requirements and design stages, measured as one unit (hour, dollar, etc.).
Figure 2
On-Demand Provisioning Cloud-based testing leverages on-demand
The criticality of business applications for today’s computing grids that can be more easily and
organizations goes without saying. As such, flexibly deployed, eliminating the need for the
delivering quality applications requires a proper upfront capital expenditures (Cap-Ex) required
test strategy complemented by an efficient test to procure hardware and software. The pay-per-
environment. Testing helps minimize errors but use model provides several benefits, including
does not guarantee their removal. Organizations providing testing organizations with freedom
strive to deliver quality applications by reducing from owning assets; enhanced collaboration
the majority of defects through rigorous testing among testers, developers and managers;
while preparing themselves to fix minor bugs greater levels of efficiency; and, importantly,
during production. In the case of a new version reduced time-to-market for key business
of an existing application, organizations must applications. These benefits are delivered by
ensure that the new version works without dis- leveraging access to cloud-based resources over
turbing the current systems environment. This the Internet without compromising security
creates the need for a production-like test envi- requirements.
ronment, which might not offer the same process-
ing capacity but delivers the same functionality The pursuit to reduce time-to-market and the
and enables all required components to interact expense of application releases is often ham-
as they would in a production environment. pered by huge costs and long delays in setting up
an appropriate test environment. Production-like
Organizations have tried numerous options to environments at reduced costs can be achieved
improve the test process. Recently, system virtu- through on-demand provisioning, combined with
alization has provided some respite, but it is not efficient processes and governance that creates
a panacea. It has met base-level operational and the discipline to develop quality build images and
financial objectives by eliminating the need for version control.
large capital investments in hardware, software
and related infrastructure; however, given the To a large extent, the testing model can also
requisite set-up costs, many pioneering compa- eliminate configuration errors. This will help
nies have yet to achieve the operational flexibility organizations achieve their test plan objectives
and scalability required to deliver the forecasted and minimize defect leakages when applications
productivity and ROI on assets. enter production due to incorrect test environ-
ment setup/configurations. Organizations would
We believe that on-demand provisioning of test do well to combine on-demand provisioning
environments via the cloud offers an effective with industry best practices like shift-left (see
solution for achieving better ROI on assets. Cloud sidebar, next page) to improve the overall quality
computing minimizes the volume of issues that of software and reduce the cost of application
plague current approaches to test environments. development, maintenance and testing.
cognizant reports 4

5. Shift-Left Practice
Finding coding flaws early in the development lifecycle is one way to reduce the number of defects
that surface in production. Shift-left is a practice where efforts are made to identify defects early
and fix them before submitting the code to a testing team. It also includes sharing the project’s
progress with clients at regular intervals, which ensures that the project is heading in the right
direction. This relieves testing teams from incurring additional costs and effort required to fix
defects at later stages, when teams are hard-pressed for time.
The process can be time-consuming, and care must be taken to ensure it does not cause
unnecessary delays in product development. Automating tests and running automated tests with
every build can help in identifying bugs quickly.
Footnotes
1
“Application Performance Management: The Lifecycle Approach Brings IT and Business Together,”
Aberdeen Group, June 30, 2008.
2
Julian Brook, “Testing Environment Management Challenges With Virtualisation,” ComputerWorldUK,
Sept. 6, 2008.
3
Alois Reitbauer, “Is There a Business Case for Application Performance?” Dynatrace, May 26, 2010.
4
“Hype Cycle for Application Development, 2011,” Gartner, July 27, 2011.
Bibliography
Martin Perlin, "Notes from the Trenches: Obstacles and Challenges to IT Environment Stability,"
www.evolven.com, October 2011.
Wayne Ariola, "The Next Generation of Test Environment Management," Virtualization Journal,
July 12, 2011.
"Maximizing the Value of Good Testing Practice in an Agile Environment: Delivering On Time, In Scope,
On Budget and At the Tight Level of Quality," Sogeti, September 2010.
“Is Shift-Left Agile? And Death by Build,” AgileFall, March 22, 2010.
Vinaya Kumar Mylavarapu, “Taking Testing to the Cloud,” Cognizant, March 2011.
"Official Fix for the Zune 30 Fail," www.gizmodo.com, December 31, 2008.
Bojan Simic, "Application Performance Management: The Lifecycle Approach Brings IT and Business
Together," Aberdeen Group, June 2008.
"Improving Software Quality to Drive Business Agility," IDC, June 2008.
Matt Adamson, “Root Out Elusive Production Bugs with These Effective Techniques,” July 2005.
"The Economics of Testing," www.riceconsulting.com, 2001.
cognizant reports 5

6. Author and Analyst
Vinaya Kumar Mylavarapu, Cognizant Research Center
Subject Matter Expert
Inamdar Mahendra, Associate Director, IT IS, Cognizant Technology Solutions
The author and subject matter expert would like to acknowledge the inputs and support of Navin Rao,
a Senior Director in Cognizant’s Testing Practice.
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-
sourcing services. Cognizant’s single-minded passion is to dedicate our global technology and innovation know-how,
our industry expertise and worldwide resources to working together with clients to make their businesses stronger.
With over 50 global delivery centers and more than 130,000 employees as of September 30, 2011, we combine a unique
global delivery model infused with a distinct culture of customer satisfaction. A member of the NASDAQ-100 Index and
S&P 500 Index, Cognizant is a Forbes Global 2000 company and a member of the Fortune 1000 and is ranked among
the top information technology companies in BusinessWeek’s Hot Growth and Top 50 Performers listings.
Visit us online at www.cognizant.com for more information.
World Headquarters European Headquarters India Operations Headquarters
500 Frank W. Burr Blvd. Haymarket House #5/535, Old Mahabalipuram Road
Teaneck, NJ 07666 USA 28-29 Haymarket Okkiyam Pettai, Thoraipakkam
Phone: +1 201 801 0233 London SW1Y 4SP UK Chennai, 600 096 India
Fax: +1 201 801 0243 Phone: +44 (0) 20 7321 4888 Phone: +91 (0) 44 4209 6000
Toll Free: +1 888 937 3277 Fax: +44 (0) 20 7321 4890 Fax: +91 (0) 44 4209 6060
Email: inquiry@cognizant.com Email: infouk@cognizant.com Email: inquiryindia@cognizant.com
© Copyright 2011, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.