Load and Performance tests in Agile Scrum framework. Presented in Scrum Gathering India 2013 at Pune on July 26th 2013.
Presented by myself and S. Ravindra
Load and Performance tests in agile scrum framework SGI 2013
1. Load and Performance tests in Agile Scrum
framework
S Ravindra
S R V Subrahmaniam
Siemens Limited
An approach for complex product development context
Scrum Gathering India Regional 2013
3. Slide 3
New Product development – project context for our presentation
Enterprise grade
Not a typical web
application
Niche product
(domain / target
users / use cases)
Critical Non-
Functional
requirements
Multiple deployment
environments
Centralized communication platform for use by emergency
intervention forces
Operators in the Command center will use it for
communication with public and on-field intervention forces
Safety critical application - used in Public safety and fire
emergency situations
Platform integrates Analog fixed, ISDN, Analog
radio, digital radio (Tetra)
Integration of heterogenous technologies
4. Slide 4
The Product : Communication platform
Touchscreen –
Thick client
application
Linux server Web server
IP PBX Digital & Analog Radio
Database
Gateways for different
types of incoming /
outgoing calls
Voice traffic carried over IP
7. Slide 7
Uncertainty
over the
release dates
Traditional waterfall
methodology
Post code freeze, critical
Load and Performance test
defects are uncovered –
which sometime require fixes
with high impact
The calendar duration for
Load and Perfomance testing
took 200% more time,
compared to initial estimate
Technical challenges come to
the fore much later in the
project, when it is tested in a
close to real environment
Challenging environment
Complex interplay of different
technologies including Java,
.NET, messaging platform over
VoIP /SIP.
Dependencies with telecom
devices and equipments
Mixing audio from various
streams and verifying for voice
quality
Limited scope for test
automation (using conventional
tools) for this kind of a product
Multiple deployment
environments
Technology Process
Uncertainty over Product quality. Management wanted to strengthen
the tests given the criticality of the application
8. Slide 8
Traditional delivery model
System test
Design
Coding
Unit tests
Functional tests
Requirements
Code freeze
Critical bugs identified
Related to:
Long run stability
Load and Performance
Environment
Heterogenous end-to-
end operations
Prolongs
9. Slide 9
Nature of Load and Performance defects pose a challenge
Defects with high turn-around time
Occurs during
long run
May require
architecture fixes
Bug verification –
again takes time
Fixes may have
functional impact
Hard to reproduce
Defect occurrance
does not follow a
pattern
11. Slide 11
Agile Scrum is the right choice
Complex technology
New product & need for early
customer feedback
Positive Organizational mandate
Late testing needs to be avoided
Internal customer – less of contracting
hassles
12. Slide 12
Agile Scrum is fit for purpose
Niche
Generic
Simple /
Homegenous
Complex /
Heterogenous
Domain
Technical complexity
13. Slide 13
Key considerations in defining a methodology within Sprint framework
Thick client
application, more
susceptible to
stability issues
Used in emergency
call centers. Zero
tolerance for failure
Given nature of
application,
limited
automation
options
Device
integrations
are involved
Defect
turnaround time
high
14. Slide 14
Framing Agile testing for a Product in Safety critical environment
Sprint Test
Limited
feasibility
to do within
Sprint
Testing for
NFRs take
time
Accomodating the factors in the Agile test framework which is to be carried out
within a Sprint is the critical decision
NFRs need
to be
tested at
earliest
15. Slide 15
Sprint testing
IN Scope for Sprint Testing
User story working at a functional
level
User story working at an end-to-end
level
Horses for the courses
Load and Performance tests
Stability tests run over long duration in
multiple environments
Complex concurrent workflows in a full
load scenario
Exploratory tests
Negative tests
Production-like environment
To be handled out of Sprint
16. Slide 16
Testing model practiced in the project
Sprint 1 Sprint 2
Functional tested package
Sprint 3
Dedicated
test phases
NFR test
The Sprint build is taken up for long running stability, load and performance tests,
multiple enviornment tests, exploratory tests and negative tests.
Defects
NFR test
.....
Four weeks
Product
Backlog
17. Slide 17
Testing methodology adopted
Sprint Testing
Enable test
Monitor resource utilization patterns; Focus on
tooling
Do tests where feasible – even if it is partial; Use
test stubs where needed
Plan the tests which need to be run; actively
collaborate with Product Owner
Bugs / tasks for forthcoming sprints
Early detection will enable certainty in product release schedules
Gather data
Perform test
Act on results
18. Slide 18
Test Organization
Within Scrum teams
- Testers part of the Scrum teams
- Operationally reporting to Scrum Master;
Officially reporting to Test Manager
- Apprisal, Competency development taken
care by Test Manager
- Testers need to be good in inter-personal skills
- Testers need to be aware of white box
techniques, interpretation of logs, configuring
and setting up parameters etc.,
System test team
- Testers do not belong to Scrum teams
- Testers report to Test Manager
- Work on testing the software at System level
and for non-functional requirements
- Diversity in terms of technical skills is key.
Each tester has to be a stand-alone expert in a
chosen technical area as the breadth of testing
would typically involve multiple areas
19. Slide 19
Other measures to foster Agile way of working
Management focus for moving towards Agile1
Visibility of the project status and key metrics for all stakeholders2
A possible dip in productivity initially and a rebound afterwards3
Facilitating the transition with suitable trainings and coaching4
Transition with new processes and tools, in tandem5
Co-location and office layout to reflect the new work culture6
20. Slide 20
Benefits accrued..1
Team
Product
Methodology
Enhanced collaboration between the development and test teams
Less stress compared to fixing critical issues, late in the project lifecycle
Formulating Test organization for different test types
Re-skilling and Competency development
Riskier aspects of the Product tested upfront
Design refactoring for performance and stability is a continuous process
NFRs are given due focus - Enhanced visibility of the NFR related test cases
to development and product teams
Product is as close to being production ready
Clear separation between the different levels of testing
NFR Tests planned and executed with due rigour without getting influenced
by Sprint dynamics
Will support a move towards TDD
Compliance to schedule and lesser critical bugs after code freeze
21. Slide 21
Defect turnaround rate improved by 20%
50% defects found early – i.e. During the Sprint testing
NFR test cycles reduced by 25%
15% less of critical and major defects
Benefits accrued..2
23. Slide 23
Next improvements – to our project
Continous review
and
improvement of
tools &
techniques
Invest in developing an
automation framework at
end-to-end and
component Integration
levels
Use of White-box tools and techniques for Load and
Performance tests
Have specific user stories to improve performance
Enhance
Automation
Sprint development