Nitisak Mooltreesri from DST Worldwide Services spoke about automated load testing for continuous delivery. He discussed how load testing is important to find bugs under high user loads. His company performs daily automated performance tests using simulation approaches to test incomplete systems cheaply and reliably. This helps reduce performance issues by providing early feedback and catching problems before deployment.

1. BUGDAY BANGKOK 2014 SPEAKER
BugDay Bangkok 2014
Nitisak
Nitisak Mooltreesri
DST Worldwide Services
NMooltreesri@dstworldwideservices.com
Facebook: Nitisak Mooltreesri
Twitter: @theimagine
Automated Load Testing for Continuous
Delivery.
Consider how much time you spend finding and fixing problems in
production. Even with the most comprehensive functional testing
process, you still discover elusive bugs under simultaneous load of
thousands of users after going live. Early testing and rapid feedback, as
well as effective communication, are cornerstones of the Agile approach
to software development.
This talk covers how the BPMS performance team at DST have done the
Automated Load Testing for Continuous Delivery approach.
The Approach and techniques for early testing can also be applied to non
Agile projects, so even if you haven’t jumped on the Agile bandwagon,
you and your team can benefit from this session.

2. Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014

3. Why automated Load testing is
important ?
• Most users click away after 8 seconds of delay
• Business revenue loss due to poor web applications performance
• Failure of US Based Airlines computerized system to maintain flight
BugDay Bangkok 2014
plans in Nov 2009
• Aberdeen found that inadequate performance could impact revenue by
up to 9%
• Business performance begins to suffer at 5.1 seconds of delay in
response times of web applications and 3.9 for critical applications

4. Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014

5. What are the risks we are testing
for ? (1)
Scalability :
• Operation Problems ? Solve with hardware ?
• Tall Stacks --> Wirsth’s Law : “Software is getting slower more
BugDay Bangkok 2014
rapidly than hardware get faster”
• Subject to use patterns and user models.
What Does a Problem Look Like ?
• Longer response times is a clue
• “High” CPU/Memory/Storing/Network Utilization

6. What are the risks we are testing
for ? (2)
Capacity : System can’t support the expected load structurally/as
engineered.
• Response time very sensitive to load
• Growing Queues
• Hard or Soft Resource Limitations
BugDay Bangkok 2014
• High CPU Limitation
• Increasing I/O Latency
• Run out of database threads

7. What are the risks we are testing
for ? (3)
Concurrency : ( Race Conditions, Database locks,
contention points)
• Infrequent functional issues that seem to only occur under load.
• Process crashes
• Not easily reproducible.
BugDay Bangkok 2014

8. What are the risks we are testing
for ? (4)
Reliability: Degradation over time, system becomes
slower, less predictable, or eventually fails.
BugDay Bangkok 2014
• Memory or Object Leaks
• More frequent Garbage Collection
• Decaying response time

9. Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014

10. “Realism” vs “Simulation Tests”
Approaches.
Traditional testing we have to test realistic things!!!
• Will the completed, deployed system support :
• [a,b…) users
• Performing ( e,f…] activities
• At (j,k…] rates
• On mn… configuration Uder rs … external conditions,
• Meeting x,y… response time goals ?
BugDay Bangkok 2014

11. “Realism” vs “Simulation Tests”
Approaches.
Simulations Tests
• Maybe the answer is more control
• Horizontal Scalability makes assumptions let’s use them
• Test Subsets: Single servers, single components, cheaply and
BugDay Bangkok 2014
repeatedly
• Calibration tests.

12. “Realism” vs “Simulation Tests”
Approaches.
Suggestions: Build easily repeatable, reliable rapid
test.
• Login ( Measure session overhead/footprint)
• Simple Workflows, avoid data caching effects.
• Control variables
• Be ready to repeat/troubleshoot when you find anomalies.
BugDay Bangkok 2014

13. Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014

14. Performance Testing Incomplete
Systems.
Showcase : Daily performance testing in BMPS business
unit at DSTWS and DSTSystem.
BugDay Bangkok 2014

15. Approach: Simulation Approach
Suggestions: Build easily repeatable, reliable rapid test.
• Login ( Measure session overhead/footprint)
• Simple Workflows, avoid data caching effects.
• Control variables
• Be ready to repeat/troubleshoot when you find anomalies.
Cheap and Fast Performance Tests :
BugDay Bangkok 2014

16. BugDay Bangkok 2014
Automated Tools
Tools:
• HP Performance Center 11.52
• HP Vugen IDE
• Electric Cloud for CI
• In-house software for generating report.

17. Performance Testing Incomplete
Systems.
Process and Lifecycle:
BugDay Bangkok 2014

18. BugDay Bangkok 2014
HP Performance Scripting.

19. Create Performance Test Scenarios
BugDay Bangkok 2014

20. Create a Performance Test Workflow.
BugDay Bangkok 2014

21. Integrate to Continues Integration
BugDay Bangkok 2014

22. Reporting Early Performance
Results.
BugDay Bangkok 2014

23. The benefits of daily performance
testing
• Reduce performance tickets by 70% compare with the previous
BugDay Bangkok 2014
version.
• Increase performance test efficiency by 150%.
• Reduce performance test effort by 80+% (Compared with the previous
project)
• Increase the performance test machines utilization by 450%

24. Some Recommendation for Agile Load
Testing
• Establish performance goals
• Include performance tests in the build process
• Test incrementally
• Reuse functional tests
BugDay Bangkok 2014

25. BugDay Bangkok 2014
Q/A