This document discusses the importance of performance testing web applications and services. It explains that performance tests help locate issues before public release, define system limits, and find bottlenecks. The document then defines different types of performance tests, including load, stress, spike, and endurance tests. It provides examples of tools for performance testing, such as Apache JMeter, and how to analyze test results. Overall, the document makes a case for regularly performance testing software to ensure expected quality and response times.
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
Optimize Website Performance with Load Testing
1. 3 SECONDS OR LESS…
PERFORMANCE TESTS INTRODUCTION
AUTHOR: PIOTR LISS
2. WHY TEST PERFORMANCE?
Why NOT to test
It will work without it
Why YES
Locate performance downgrade before release
Define the limits
Find our bottleneck before user will
On commercial sites – clients loss
On internal sites – users frustration
DDoS attack simulation
3. WHY TEST PERFORMANCE?
GOOGLE RANKING
Source: http://www.monitis.com/blog/website-performance-its-impact-on-google-ranking/
4. WHY TEST PERFORMANCE?
3 SECONDS OR LESS
Source: https://www.soasta.com/blog/google-mobile-web-performance-study/
6. WHY TEST PERFORMANCE?
INCREASING SIZE OF AVERAGE PAGE
Source: http://httparchive.org/trends.php?s=All&minlabel=Nov+15+2010&maxlabel=Feb+1+2018
7. WHAT ARE PERFORMANCE TESTS?
DEFINITION
Test which result generates measurable value
regarding efficiency of testing product.
response time
number of users
request per
second
bits per second
speed
stability
reliability
capacity
application
web page
service
database
8. WHAT ARE PERFORMANCE TESTS?
HOW TO TEST
Requirements – examples:
can handle 1000 request per hour
response time not bigger than 2000ms with 100 concurrent users
can handle 6 hours with active 50 concurrent users
after spike load application returns to previous state
Parameters:
number of active users (connections)
response time
capacity (maximum load)
requests per second
9. WHAT ARE PERFORMANCE TESTS?
LOAD TEST
How application behave with predictable and safe load (good candidate for Continuous Integration process
and response time measurement)
0
10
20
30
40
50
0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00
Users
Time
LOAD TEST
10. WHAT ARE PERFORMANCE TESTS?
STRESS TEST
Usable to define border values (good test to define capacity and how infrastructure behave after crash)
0
10
20
30
40
50
60
0:00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00
Users
Time
STRESS TEST
11. WHAT ARE PERFORMANCE TESTS?
SPIKE TEST
Short and extremely high load to verify do application return to its before test state
0
10
20
30
40
50
60
0:00 1:00 2:00 3:00 4:00 5:00
Users
Time
SPIKE TEST
12. WHAT ARE PERFORMANCE TESTS?
SOAK/ENDURANCE TEST
Few hours load tests to verify does application on stable load will not lose its usability from beginning
(Memory Leak)
0
10
20
30
40
50
0:01:00 0:30:00 0:59:00 1:28:00 1:57:00 2:26:00 2:55:00 3:24:00 3:53:00
Users
Time
ENDURANCE TEST
13. HOW TESTING TOOL WORKS?
Testing tool
emulates user
story scenario
repeating it X
times during
Y seconds.
14. HOW TESTING TOOL WORKS?
Eliminate any unrelated with tested product element:
no User Interface if it’s not generated on our application (e.g. no web browser)
eliminate external application on test environment (e.g. in place of external authorization proxy server make a
plug that will always emulate same result)
Make as real as possible:
Think time
Cache as a browser
Embedded resources
Stable increasing load
Cookies
Different load regions (if possible)
Environment “warm up”
Divide test to steps
15. TOOLS
APACHE JMETER
since: 1998
free license: for all (Apache License Version 2.0)
protocols: web, webservices (SOAP, REST), FTP, Mail, DB,
TCP, OS command line and many more
programming experience: minimal (groovy, beanshell)
requirements: any operating system that runs Java
support: accessible community, keen to help, many
tutorials, trainings, courses, forum topics and discussions
main advantages:
dozens of free useful plugins that extend functionalities (e.g.
additional protocols, visualizations, webdriver)
User-friendly interface
Easy scalability
16. TOOLS
VISUAL STUDIO LOAD TESTS
since: 2010
free license: very limited (requires most expensive VS
version)
protocols: web, webservices (SOAP, REST)
programming experience: minimum-medium (C#, VB)
requirements: any based on Windows
support: official forum and support group, not to many
online tutorials
main advantages:
C# code gives possibility of advance test expansion
Azure integration
easy debugging and recording
17. TOOLS
AZURE PERFORMANCE TESTS
since: 2014?
free license: limited
protocols: web
programming experience: none
requirements: web browser + Visual Studio Team
Services account
support: not needed
main advantages:
free 20000 user-minutes/month tests on on-demand test
machine from desired world region
test import from Visual Studio or JMeter (and partly
Fiddler)
possibility to use own Azure infrastructure (additional
cost)
18. TOOLS
HP LOADRUNNER
since: 1991
free license: community edition (non-commercial)
protocols: web, webservices (SOAP, REST), FTP, Mail, DB,
RDP, SAP and many more
programming experience: medium (ANSI C or few other)
requirements: Windows system with 8 cores and 16GB
memory
support: many tutorials, trainings, courses, forum topics and
discussions
main advantages:
rich reports
integration with many external software (including CI)
good recorder
19. TOOLS
GATLING
since: 2011
free license: almost all (Apache License Version 2.0)
protocols: web, DB, Mail, webservices (REST)
programming experience: medium (DSL)
requirements: any operating system that runs Java
support: many web helpers and forum topics
main advantages:
can generate more load from single machine than other
tools
simple and user-friendly reports
easy to write plugins
21. PERFORMANCE TEST REPORT ANALYSIS
Summary of all filtered test steps with errors,
response time and throughput statistics
APDEX index
APDEX is from 0 to 1 scale value that represents
how many request finished in given time
toleration threshold - our target time
frustration threshold - acceptable maximum time
Helpful to compare results
22. PERFORMANCE TEST REPORT ANALYSIS
Example charts:
Response times over time
Active threads
Bytes sent/receive over time
Hits per second
Latency vs request
Response time percentiles
23. APACHE JMETER
EXAMPLE USAGE - RECORDER
JMeter Recorder
Requires some basic adjustments and web
browser configuration
Recorded records have to verified because
might contains some not related traffic
Other methods:
SAML recorder on any web browser (usually as
plugin)
JMeter request/response analyzys
Fiddler traffic recorder
24. APACHE JMETER
EXAMPLE USAGE – ASSERTIONS
Text response (XPath response)
Site header response
Response time
(not necessary) code response
26. APACHE JMETER
EXAMPLE USAGE – TEST RUN
Verification process – GUI mode
Run test using listeners
Run process – non-GUI mode
Run test using command line
27. APACHE JMETER
EXAMPLE USAGE – IS THAT ALL?
Extraction rules - passing tokens and IDs between requests
External files - e.g. other credential for every iteration
Loops and IFs - e.g. requesting every few milliseconds until page will get certain content
Scripts - e.g. exactly decide in which moment start timer
Continuous Integration - e.g. integration with Jenkins
Distributed environment - slave machines from different regions to generate load
28. PERFORMANCE TESTS PROBLEMS
Asynchronous request and JavaScript code
Increase performance by better code or by environment improvement?
Do performance tester has enough work for full-time contract?
Test environment vs production environment
What to analyze after performance test run?
29. LINKS AND THANKS
Author: Piotr Liss (www.linkedin.com/in/piotrLiss)
Consultant: Darek Kozon (https://www.linkedin.com/in/dkozon/)
Link: Blazemeter (https://www.blazemeter.com/)