1. Advantageous of Implementing
Performance Testing
in Agile Process
White Paper
idexcel

2. Background
In many web or cloud applications, performance testing is critical part of application testing since it affects
business revenue, credibility, and customer satisfaction. Conventional software development models are known
to pushing the performance testing to the very end of project, with the expectations that, only minor tweaks
and tune up are required to meet the performance requirements from the business, however any major
performance bottlenecks found during this phase were major factors for delay in Go to Market. With more and
more companies are adapting the agile software development process which believes in performance testing
should never be an afterthought but it should tightly integrate from initial planning to production analysis of
software development lifecycle. This white paper explains how any company can integrate performance testing
into agile process, and key barriers for agile performance testing when team decides to adopt agile performance
testing.
idexcel
2 Page
Performance Testing in Agile Process

3. Performance Testing in
Traditional development
Agile Development
Practice
Performance engineering is an important software
Business Impact: Performance issues detected late may
not only end in small tuning, but may force business to
renegotiate contracts Conducting performance testing in
agile, iterative way may increase the efficiency of perfor-
mance testing. development discipline that ensures that
applications are architected, designed, built and tested
for performance. Conventional development models like
waterfall models are known to pushing the performance
testing to the very end of the project, because of many
reasons like last minute changes, bugs that prevent test
execution, infrastructure limitations, or lack of dedicated
testing resources. This flawed approach is having many
disadvantages like
Slow: It assumes that system is completely ready to
execute performance test cases; these practice results
pushing the performance testing at the end of the project
lifecycle. Any performance bottlenecks found during this
phase will postpone GTM date.
Late Changes: As late changes are always expensive to fix,
any performance bottlenecks found during this phase are
expensive to fix.
3 Page
idexcel
Performance Testing in Agile Process
Performance in all teams and all roles are missing: It is
mindset and comfort zone in traditional development
teams that, performance testing is carried out by special-
ized team at the end of development lifecycle. However
most of the teams fail to realize that performance
engineering must be architected, designed, and tested.
This mindset forces to miss the performance testing at
unit level, integration and functional level which leaks the
performance issues to last gate of development cycle
Analyzing the results: Running all scripts and/or large
tests, gives minimal information, about the system
behavior. Analyzing will become complicated and time
consuming when we run all scenarios and scripts all
together.
Business Impact: Performance issues detected late may
not only end in small tuning, but may force business to
renegotiate contracts
Conducting performance testing in agile, iterative way
may increase the efficiency of performance testing.
Growing need for quicker and adaptive solutions to tech
problems is pushing firms to adopt the agile methodolo-
gy. Today more and more companies are addressing
different technology issues by adopting this iterative
approach to software development and releasing high
quality software faster and more efficiently. Organization
sees agile software development as a faster way to create
products that would generate sales. Agile methodology
promotes self-organization, flexibility to midcourse
action, better alignment between the team, accountabili-
ty, open collaboration throughout the lifecycle of the
project.
One of the main goals of agile is to efficiently manage
change in both external and software functionality and
accelerate the development of quality software. Agile
puts development, QA, and support staff on the same
page and all stakeholders, including those on the
business side need to be tightly involved.

4. 4 Page
idexcel
them to react early to any detected performance or
scalability issue.
Performance Test at Every Level: In agile, team may not
get the complete features which are needed for perfor-
mance test at the end of every sprint. This forces teams
to conduct performance testing on portion of the code
or component which is ready. In agile Start performance
testing at component level of function and if all the
components work fine individually fine then hypotheti-
cally they also work well together. This knowledge will
help the team to analyze and debug performance
bottlenecks when team test on the system level later on.
Team need to look at the performance testing in agile as
testing of different performance layers that can be
approached in cadence as the sprints progress.
Develop and Execute Test Cases: As stated early, agile
may not generate workable product at the end of each
sprint. We may not be able to use the standard load
testing tools as not all components would be present
which standard load testing tool may require. So, in
most cases there is an inherent challenge in simulating
load and test for performance. Hence tools usage may
vary between the phases. For example a stub or a
harness might be used in the initial phase along with
JunitPerf, and extra effort for this need to be taken into
consideration during project planning phase. Once
created, development and execution of test cases would
follow, with the help of profiling tool; most of the
performance hotspots can be identified.
At later stages of the project, the traditional perfor-
mance scripting and execution will replace the harness
that was created in the earlier stage. So there will be a
big rework which again would be a challenge consider-
ing the sprint timelines. So these tasks should be consid-
ered in backlog and sprint should be planned according-
ly.
Integrating Performance
Testing in Agile
The concept of performance testing in agile is different
from conventional model. Performance testing is an
integral part of agile process. The goal is to test perfor-
mance early and often in the development effort, and to
test functionality and performance in the same sprint.
To remain efficient in an agile environment, team need
to change the way it used to managing performance
testing. One of the big challenge team face when
integrating performance testing is that agile movement
shifts the inter team process of software engineering to
one that is focused on software development, to the
exclusion of QA and the exclusion of operations Below
section describe the x step approach for integrating
performance testing into agile process in any organiza-
tion
Definition of done: In agile, scrum team along with the
product owners and scrum master, defines a definition
of done to claim a product backlog to be completed.
Usually it includes incremental product architecture
design, feature design, development, unit testing,
integration testing, functional testing, and documenta-
tion. It is highly recommended to list the performance
test requirement in the product backlog and include
performance as a definition of done for a sprint at com-
ponent, integration and system level.
Add Performance Test to Build Process: As continuous
improvements takes center stage in agile, so adding
performance tests into continuous integration process is
one important step to continuous performance
engineering. This entire process starts when developer
checks in code and continues through the build and test
process. I n
addition to the functional testing results, performance
test results are analyzed and recorded for each build.
this integration of CI performance testing with the agile
makes developers to collect feedback on changed com-
ponents in continuous feedback loop, which enables
Performance Testing in Agile Process

5. 5 Page
idexcel
Decide on Test environment: In Agile, although perfor-
mance testing starts early in development lifecycle,
there are certain performance testing aspect that are
hard to do until later in the delivery cycle like stress and
capacity testing. While development and test environ-
ment performance is not typically as big concern as
production, but can be used for performance smoke
testing, and once all components are ready, team can
move towards staging and production environment with
appropriate tools with valid test data
Although performance testing at an early stage with
iterative approach looks to be very good idea, while
integrating performance testing in agile practice, team
may face many challenges.
Unstable Builds: As agile process is adaptive, designed
to accommodate, and encourage inevitable changes.
This may lead to changes in system. But Performance
testing at system level is painful process. It is difficult to
get the application deployed and running the perfor-
mance testing, teams may get frustrated to test multiple
builds with changed requirements in same iteration. It is
responsibility of management to make these specialist
teams to understand the importance of iterative system
testing and motivate them in agile practice
Test Data Preparation: Because of the nature of agile
environment, sprints are short which shrinks testing
time in every sprint. This forces team to decide on test
data for performance testing, early in development
lifecycle preferably in planning stage this gives enough
time for performance testing during the sprint. Team
need to decide on whether to use the production data
for testing or dummy data, does any regulatory aspects
to use the production data. The earlier test data prepa-
ration takes place, the more time team will have for
testing in short agile sprints. So performance testers
should work with stake holders in the planning stage to
prepare the test data. Plan carefully to get the test data
early in the life cycle.
Performance Testing in Agile Process
Key Barriers of Agile
Performance Testing
High skilled Performance team: Any tester who could
record, generate the script, test and provides the perfor-
mance results using standard commercial tools may not
be suitable for performance testing in agile. The agile
methodology provides lot of challenges from perfor-
mance standpoint. Examples are, conducting perfor-
mance testing early in lifecycle at component level and
performance testing of incomplete components. This
makes it important to performance test only a portion
or a component of an application. Hence it is imperative
that team members should have the confidence to get
their hands dirty and learn the technology and program-
ming skills to develop stubbing which simulates parts of
an application that are either not written or not avail-
able. If an application uses outside third party integra-
tion then performance engineers need to develop stubs
which simulates 3rd party integration, as they may not
get the third party production server for performance
testing at every sprint.

6. 6 Page
idexcel
Stake holders buy in: Traditional QA departments segre-
gate performance test specialists to single group who
will perform the work for different development teams.
Contrary, performance testing in agile demands perfor-
mance testers early in the lifecycle of the project, having
an additional performance testing team from the early
phase of the project directly impacts the budget and
timelines of the project. Adding to it, advantages of
performance testing are more of an intangible kind.
Thus it becomes more difficult to get a stake holder
buy-in for budget and team for performance testing
early in lifecycle of agile
Project Management Vision: Although performance
testing is critical part of application testing as it affects
business revenue, credibility and customer satisfaction,
many project managers are still unaware of perfor-
mance test process and challenges performance team
faces. Within agile framework, this would cause more
conflicts between the project managers and perfor-
mance teams. Hence it is important to discuss and nego-
tiate a plan with project managers, performance team
and business sponsors, and come up with a plan which
will for “approach of performance testing in agile”, “
From which sprint to start the performance testing”,
“Transition of performance test tools as the application
development progress”, etc.
Look at the other side of the coin: Although perfor-
mance testing at an early stage with iterative approach
looks to be very good idea, there may be limited possi-
bility of presence of bottleneck in code base if we have
a proper reviews and architecture in place. The more
performance bottlenecks that may be present, would be
more on infrastructure side which are easier to fix and
with almost no functional/regression testing. Hence
before adopting early testing which consumes huge
effort and cost, have tradeoff between, talent in the
team, nature of application, business impact, business
and nature of contract and decide whether you can
address the performance challenges later in the devel-
opment life cycle or in successive releases.
Conclusion:
Performance engineering is an important software
development discipline that ensures that applications
are architected, designed and built and tested for
performance. With more and more companies are
adapting the agile software development process which
believes in performance testing should never be an
afterthought but it should tightly integrate from initial
planning to production analysis of software develop-
ment lifecycle, it is important to integrate performance
engineering to agile development process. Unlike
performance testing in conventional development
process, team has to start performance testing early in
development lifecycle which has a positive outcome in
code tuning, earlier rejection, performance bug which
appears in production.
Performance Testing in Agile Process

7. 7 Page
idexcel
In this competitive IT days, organizations are becoming
more and more conscious about the performance of their
Enterprise Solutions with in short time span along with
development like not spending separate time line for it, till
developers complete their coding for application and
making it functionally stable. There are several competi-
tive applications with better performance are coming with
high quality software in less time before one finish out
their project functionally ready , As the IT industry matures
and the technology evolves, so does the awareness about
expectations from an Enterprise Application. Focusing just
on the design / implementation and Zero-functional-de-
fect solutions then checking the performance of applica-
tion are things of the past. With increasing maturity in
technology and IT staff, the best solution is driving every-
thing in to agile process. As per this now days most of the
ITs moving towards agile, the challenge is how they imple-
ment performance testing in agile model.
Performance Testing: Performance testing of an applica-
tion / system is basically the process of understanding how
the application and its operating environment behave at
various user loads. In general, it is performed by simulating
virtual users to determine / validate the scalability,
availability, robustness, and hardware & software resource
utilization of the application thus ideally paving the way
for effective capacity planning of the system for future
usage. IT is one of the single biggest catalysts to significant
changes in architecture, code, hardware and environ-
ments. One can utilize this to benefit by exploring the
performance issues. Performance testing that usually
takes place at the end of the lifecycle in a waterfall model.
Performance testing in Agile? - The (in) famous sentence.
What is it? What does it mean? Why and how to imple-
ment and drive performance testing in to agile? In this
White paper, we'll touch upon Performance Project Phases
traditional waterfall approach to do performance testing,
what an agile performance testing advantages of imple-
menting performance testing in to agile model.
Performance Testing in Agile Process
The Performance
project Phases:
1. Sizing and Capacity Planning
• System usage scenarios
• Load during normal usage and peak hours
• Performance requirements
• Right-sizing systems to meet needs without wasting
money and creating unnecessary complexity.
2. Performance Engineering
• Proactively engineering for performance
• Technical design with performance in mind
• Tools Selection
• Test environment & data preparation
• Test script design and preparation

8. 8 Page
idexcel
Main Performance Types
Waterfall development methodology is a sequential
process in which development is seen as flowing steadily
downwards (like a waterfall) through the phases of
requirements analysis
Most of the cases performance testing phase will follow
traditional water fall approach. After development of
finished, application functionally tested and verified and
System is ready then the performance scripting starts for
requested scenarios. Preparing test runs with required
combinations and settings then gathering the results and
compare with the requirements will happen. If require-
ments not met team alerts the developer to get back for
PERFORMANCE TESTING
LOAD TESTING STRESS TESTING
CAPACITY TESTING
Performance Testing in Agile Process
3. Performance Testing
• Tests execution
• Result analysis
• Recommendations
• Fix problems
• Monitoring and Troubleshooting
• Understanding systems and solving problems in produc
tion
Main types of performance tests to cover repeatedly in
Performance Phases are:
Load testing: Part of performance testing and it applies a
“normal” amount of load, reflective of real-world num-
bers, Shows bottlenecks with different user levels and any
immediate problems when typical load is applied.
Stress testing: Performance testing beyond normal or
“peak” amounts of load and possibly for extended periods
of time. Shows the how application will behave under
stress, problems if any.
Capacity Testing: Performance test to determine how
many users and/or transactions a given system will
support and still meet performance goals.
All of this testing is essential to truly understand the
systems performance characteristics
Traditional water fall performance approach and
constraints:
Requirements analysis
Design
Develop
Deploy
Test
Maintain
Waterfall SDLC moduleTime taken to statt test

9. 9 Page
Agile SDLC Module:
idexcel
Performance Testing in Agile Process
work to meet the requirements. Though this methodology
has been proven success full approach there are some
constraints and problems mentioned below which will be
costlier when considering the time in to factor.
• No functional system until the very end and everything
must be ready to startup performance testing.
• It assumes that the system is completely ready
• Developers may not be available by the time the perfor
mance testing starts
• Late changes as per the requirements if not met across
the board will be the costlier.
• Performance scripting development may took long time
to debug modify as per the functionality.
• Analyzing will become complicated and time consuming
when we run all scenarios and scripts all together.
• Lot of repeated set of runs needed to build any Perfor
mance Model for future but after one successful run.
ROI: Return On Investment
What is exactly Agile Methodology and how it works?
The word agile here doesn't refer to any specific develop-
ment process or methodology, the meaning of word Agile
is “able to move quickly and easily “.
Agile software development is a group of software devel-
opment methods based on iterative and incremental
envelopment, where requirements and solutions evolve
through collaboration between self-organizing, cross-func-
tional teams.
Agile methodology promotes close teamwork, appropriate
reviews, more iterations and self-organization. This
approach enhances rapid delivery of high quality software
and enables a business practice that aligns software devel-
opment with customer needs and company goals.
Application stakeholders, programmers and testers must
make performance a primary consideration in all iterations
of agile development process.
Challenges in Implementing Agile Performance testing
and successful approach:
DESIGN
DEVELOP
TEST
REQ, ANALYSIS

10. idexcel
Performance Testing in Agile Process
Performance Testing
in Agile Process
• How to cover all performance project phases? (Sizing,
Capacity Planning, Performance Engineering, and Perfor
mance Testing)
• Do we need to Waite till the developers give code feature
by feature?
• If any application feature changes do we need to re test
the one that has changed?
• How we go ahead without scalability and performance
requirements?
• Below three steps approach will give complete solutions
for above challenges
Approach: As already explained how agile works above
let us see how we approach and face the challenges in
application performance testing in agile methodology.
The main reason of point and its advantage of doing the
performance testing in agile process are to save time
factor we have to start concentrate about performance
testing early life cycle of SDLC .thus have to determine
the performance potentials in main business scenarios as
early as possible during the project development. Test
the available, analyze and fix performance bottlenecks at
the code level. The three Step approach for implement-
ing agile performance testing is described below
• Test environment setup
• Tools selection
• Perfomance requirements
• Test data preparation
• Load tests
• Stress tests
• Capasity testing
• Resillience testing
Iter.0
Benchmark Test
Focus Test
SDLC + STLC SDLC + STLC
Focus Test Focus Test Focus Test
Benchmark Test Benchmark Test
Iter.1 Iter.2 Iter.3 .... Iter.x-1 Iter.x
....................................................
10 Page

11. idexcel
Performance Testing in Agile Process
Performance Testing
in Agile -Tree Steps approach:
First step: Test and find the potentials at unit code level as
it will be developed. Execute the tests to fix optimize
bottlenecks at unit/Code level. This is performed when the
scalability and performance requirements are specified at
method level.
Unit performance tuning proceeds along with the devel-
opment of the application and unit testing of the method
to tune is completed.
Application
Load Turning
Product
Released
Yes
Hardware/
Software
Optimization
Integration/
Functional
Testing
Hardware/
Software
Optimization
Perfomance
Bug Found?
Perfomance
Bug Found?
No
Yes
No
Application Load Turning
Component Level Turning
Component
Level
Perfomance
Testing
UAT Testing
Component
Level
Perfomance
Testing
Application
Load Turning
Yes
Hardware/
Software
Optimization
Perfomance
Bug Found?
No
Component Level Turning
Yes
Code
Optimization
Perfomance
Bug Found?
No
Unit Level Turning
Unit
Testing
Unit Level
Perfomance
Testing
Integration/
Functional
Testing
Second Step: The Components level testing (testing
components): Executing tests to isolate and fix bottlenecks
in application components level.
Third Step: Application flow level testing: Testing the
critical application flows for user experience under normal
and maximum high loads.
11 Page

12. Performance Testing in Agile Process
idexcel
Performance at
Unit Code Level
Goal:
• Identifying the methods.
• Scripts are prepared to invoke individual method.
• Methods are tested under normal load for response
time while measuring CPU and memory utilization.
• If method does not show acceptable level of perfor
mance, it is fine-tuned and test is re-executed
• Tests are focused on discovering poorly performing
methods.
• This is performed when the scalability and perfor
mance requirements are specified at method level.
When to do? Unit performance tuning proceeds along
with the development of the application and unit
testing of the method to tune is completed.
How to do?
• Identifying the methods.
• Scripts are prepared to invoke individual method.
• Methods are tested under normal load for response
time while measuring CPU and memory utilization.
• If method does not show acceptable level of perfor
mance, it is fine-tuned and test is re-executed.
Component Level
Tuning
Goal:
• Here the tests are targeted towards finding bottle
necks in application tasks and activities.
• The major objective is to evaluate the system behavior
when it is pushed beyond its breaking point
When to do? The components are deployed on their
respective application server.
How to do?
The components can be identified based on following
characteristics:
• High business criticality and usage.
• Can be deployed and invoked independently.
• Deployed at application or middle tier.
• Scalability and response time requirement of each
component is captured.
• Scripts are prepared to invoke individual component.
• Tests are executed for each component separately
• With respect to two major areas of bottlenecks, the
following two tests are executed:
1. Concurrency Test: Component is tested for simultane-
ous requests under incremental user load, until failure is
noticed.
2. Test for Throughput: Component is tested for high
number of hits per second with limited number of users.
With the help of above results, the component is tuned
by optimizing hardware/software configuration and test
is re-executed.
12 Page

13. idexcel
Performance Testing in Agile Process
Application Load
Tuning
Goal:
• Here, critical application flows are tested for user
experience under normal and maximum loads.
• The major objective is to capture the performance
metrics and verify whether the performance objectives
are met.
When to do? Acceptance Testing is completed and
application is stable.
How to do?
• The critical application flows are identified here.
• Scripts are prepared for the flows identified.
• Application is tested for normal load and maximum
load. Server and client metrics are captured.
• As per the results, the application is tuned by optimiz
ing hardware/software configuration and test is re-ex
ecuted.
Performance testing
like On Demand
The model is the functional equivalent of outsourcing to
an internal team, but this is where most companies that
are trying to integrate their performance testing into agile
processes start. This is because the performance testing
was actually an On Demand model before the agile transi-
tion. Some of the steps to be handled for this model are
mentioned below:
• You need to make use of On Demand services periodi
cally.
• The performance targets, goals and objectives should
be a standard module of each user story.
• Developers should take the responsibility for unit-lev
el, component level, integration-level testing and
tuning.
• A full time team member has to take overall responsi
bility for managing performance related tasks.
On Retainer:
Goal:
On Retainer model can be used as an interim step between
the On Demand and Full Immersion models. Some organi-
zations may not have the required testers, performance
testing environments and testing tools to support the Full
Immersion model.
Here, each development project is assigned a specific
performance tester. That performance tester is also
assigned to two or more development projects. Though
this model provides more performance testing expertise
to individual project and more domain knowledge to the
performance tester, the tester may fall short of becoming
a fully integrated contributor to the team. As a result, the
tester works independently, but provides guidance period-
ically. By following the same steps as listed in the above
model and with the help of two additional steps as
mentioned below, this model can provide more value.
• The performance tester should be available to the
team at times that is critical to the performance evolu
tion of the project.
• This model may require more performance testers or
tools than On Demand model.
13 Page

14. Full Immersion
idexcel
Performance Testing in Agile Process
If the performance is really important to the value of the product and reputation of the company then this model should be
the goal of every team in an agile process. Here, full time team members are specialized in both delivering and testing
performance and have responsibility for managing and coordinating performance related activities throughout the lifecycle
of product development.
Risk Mitigation
Benefits
Agile Performance Testing has positive outcome in risk mitigation with respect to:
Code Tuning: Unit level tuning is done early - so no need for code optimization at later stage on testing lifecycle.
Application Rejection: Reduces risk of the application being rejected because of issues such as memory leaks, hard coded
values in code and database locking.
Earlier Detection: Reduces the effort and duration for performance tuning and retesting.
Release Dates: Iterative testing leads to constant awareness of the application performance and therefore more confidence
in meeting release dates.
Performance Bugs: Less number of performance bugs in subsequent phases.
Performance Testing Efforts: Performance script can be re-used thus saving 60% of the efforts of performance cycle.
Conclusion
Hence, Performance testing is one of the single biggest catalysts to significant changes in architecture, code, hardware and
environments. You can utilize this to your benefit by exploring the performance issues. Performance testing in an agile
project environment allows you to manage the testing in a highly flexible way. Iterative testing approach leads to better
code that is optimized for performance. In particular, this approach allows you to revisit the project vision and reprioritize
tasks based on the value they add to the performance test at a given point in time.
14 Page

15. idexcel
Performance Testing in Agile Process
About the Author
Venkat Prasad works as QA Lead in the Mobility Division of Idexdcel. He has around 7years of experience in Automation
and manual testing of mobile applications. He is previously worked with IBM India Pvt Ltd before joining to Idexcel
technologies Pvt Ltd. He worked with IBM India for almost 6 years as QA Automaion tool consultant for mobile applica-
tions includes Billing systems, Sales applications, Commissions applications.
About Idexcel
Idexcel is an innovative provider of IT Products & Services focused on emerging technologies. We help world leading
companies build efficiencies and stronger businesses. With more than 15 years into existence Idexcel’s main focus is
client satisfaction and technology innovation. Our industry expertise and a global, collaborative workforce forms the
backbone of our services. We offer high degree of skills in Enterprise Applications, Cloud Services, Data-warehousing,
Big Data, Analytic, QA & Testing Services, IT consulting and Staffing. Idexcel product line includes: NDS, ERP, and
Cync - A revolutionary credit monitoring application for the manufacturing and financial management.
For more information log on to www.idexcel.com.
Global Head quarters
459 Herndon Parkway Suite 11
Herndon, VA 20170
Tel: 703-230-2600
Fax: 703-467-0218
Email: inquiry@idexcel.com
India Operations
“Crystal Plaza” 9, 10 ,11
Bhuvanappa Layout, Hosur Road
Bengaluru – 560 029
Karnataka
Tel: +91-80-2550 8830
Email: inquiry@idexcel.com
© Copyright, Idexcel. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any means,
electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Idexcel. The information contained herein is subject to
change without notice. All other trademarks mentioned herein are the property of their respective owners.