3. OBJECTIVE
Put working software into production as quickly as possible,
whilst minimising risk of load-related problems:
› Bad response times
› Too small capacity
› Availability too low
› Excessive system resource use
4. PREVENTING RISK IS A BIG SUBJECT,
WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCE
RISK PREVENTION IS A BIG SUBJECT
Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
5. CONTINUOUS DELIVERY LITERATURE PROVIDES METHODS
THAT HELP REDUCE RISK
› Blue-green deployments
› Dark launching
› Feature toggles
› Canary releasing
› Production immune systems
Jez Humble, http://continuousdelivery.com
7. WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTS
TO HELP IDENTIFY THE BIGGEST RISKS
8. PRE-PROD LOAD TESTING IS NOT FREE
› Extra code to maintain
› Usually test runs last several hours
› A production-like environment is expensive
› Realistic testing is hard
› Not all developers like writing (performance) tests
9.
10. USE IT WISELY, WHERE PRODUCTION TESTING IS STILL
INAPPROPRIATE
› It provides no guarantee
› Use it to find any showstoppers you can
› Essentially, an optional service that teams can use
11. USE IT AS A PLAYGROUND TO TRY RISKY CHANGES
Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
12. Load tests
Functional
Build, unit
integration
test, etc.
tests
Very often Less often About once a day
(at night)
13. Load tests
Functional
Build, unit Load test
integration
test, etc. script check
tests
Very often Less often About once a day
(at night)
14. THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC AS
NEEDED”
15. SET UP TEST DATA IN THE WEEKEND, TO MINIMIZE
DISRUPTION
18. ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT IS
ACCEPTABLE
› Seen from the main stakeholders’ perspective
– Response time: users
– System resources: ops
– Capacity: business
› Specific
› Measurable
› Achievable
› Relevant
19. Concurrent users
Fail: Now: Target:
< 100k 150k 200k
Intention: The website should at least be Stakeholder: Business
able to manage our typical daily load, but
we would like some margin for growth
and marketing campaigns.
Scale: Maximum load in a day, while Meter: Session table row count.
response times are still according to
spec.
20. FOR RESPONSE TIMES TOO, FOCUS ON THE
MAIN STAKEHOLDER!
FOR RESPONSE TIMES TOO, FOCUS ON THE MAIN
STAKEHOLDER!
21. SO USE A REAL BROWSER TO TEST
A REAL USER’S EXPERIENCE
24. TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTS
DETERMINISTIC
Stub systems that you have no control over
25. LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THING
THAT COUNTS IS PRODUCTION!
› Your definition of done should reflect that
› The aim is to get early feedback from a safe environment
27. SO WHAT MONITORING IS TYPICALLY NEEDED?
› Be able to localise where latency is coming from!
– For every system, all incoming and outgoing calls (count and
time spent stats)
› Finite resources (pools, CPU, I/O, etc.)
› Number of active users
› Response size, where possible
› Add whatever you need
It should be identical on all environments!
28. SET CLEAR TARGETS, SO YOU KNOW YOUR SITUATION
› How many errors would be OK in production?
› What kind of errors do we care about?
29. Number of stale server session requests /
min
50
0
100
150
250
300
200
00:00
01:00
02:00
03:00
04:00
05:00
06:00
07:00
08:00
09:00
10:00
11:00
12:00
13:00
Other servers taken out of LB
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
back in LB
22:00
Other servers
23:00
00:00
30. CONCLUSION
Support your continuous delivery process with optional load
tests and strong specs.
Use the load tests to identify some pain points, so you can
modify the code and add monitoring, making it safer to do
dark releases and canary testing in production.
Constantly ask yourself: what would it take to only do this in
production? Is it adding value?
