The document discusses methods for minimizing performance risks when deploying software continuously to production. It recommends blue-green deployments, dark launching, feature toggles, canary releasing, and production immune systems to reduce risks. It also advocates for establishing load testing requirements, using real browsers for user experience testing, making tests deterministic, and monitoring key metrics in all environments to identify performance bottlenecks before production. The overall aim is to provide early feedback through optional and safe pre-production load testing in order to fix issues and gradually implement dark releases and canary testing for continuous delivery with minimized risks.
3. OBJECTIVE
Put working software into production as quickly as possible,
whilst minimising risk of load-related problems:
› Bad response times
› Too small capacity
› Availability too low
› Excessive system resource use
4.
5. PREVENTING RISK IS A BIG SUBJECT,
WHAT FOLLOWS IS TAKEN FROM OUR EXPERIENCE
RISK PREVENTION IS A BIG SUBJECT
Photo by chillihead: www.flickr.com/photos/chillihead/1778980935
6. CONTINUOUS DELIVERY LITERATURE PROVIDES METHODS
THAT HELP REDUCE RISK
› Blue-green deployments
› Dark launching
› Feature toggles
› Canary releasing
› Production immune systems
Jez Humble, http://continuousdelivery.com
7. BLUE-GREEN DEPLOYMENTS
Elastic Instances
Load
Balancer Version n
Amazon
Route 53
Elastic Instances
Load
Balancer Version n+1
17. WHILE WE WORK ON OUR RESILIENCE, WE USE LOAD TESTS
TO HELP IDENTIFY THE BIGGEST RISKS
18. PRE-PROD LOAD TESTING IS NOT FREE
› Extra code to maintain
› Usually test runs last several hours
› A production-like environment is expensive
› Realistic testing is hard
› Not all developers like writing (performance) tests
19.
20. USE IT WISELY, WHERE PRODUCTION TESTING IS STILL
INAPPROPRIATE
› It provides no guarantee
› Use it to find any showstoppers you can
› Essentially, an optional service that teams can use
21. USE IT AS A PLAYGROUND TO TRY RISKY CHANGES
Photo by vastateparksstaff: www.flickr.com/photos/vastateparksstaff/5330257235
22. Load tests
Functional
Build, unit
integration
test, etc.
tests
Very often Less often At least once a day
(at night)
23. Load tests
Functional
Build, unit Load test
integration
test, etc. script check
tests
Very often Less often At least once a day
(at night)
24. THE AIM IS NOT PERFECTION, GO FOR “AS REALISTIC AS
NEEDED”
25. SET UP TEST DATA IN THE WEEKEND, TO MINIMIZE
DISRUPTION
28. ESTABLISH REQUIREMENTS TO MAKE CLEAR WHAT IS
ACCEPTABLE
› Seen from the main stakeholders’ perspective
– Response time: users
– System resources: ops
– Capacity: business
› Specific
› Measurable
› Achievable
› Relevant
29. Concurrent users
Fail: Now: Target:
< 100k 150k 200k
Intention: The website should at least be Stakeholder: Business
able to manage our typical daily load, but
we would like some margin for growth
and marketing campaigns.
Scale: Maximum load in a day, while Meter: Session table row count.
response times are still according to
spec.
30. SO USE A REAL BROWSER TO TEST
A REAL USER’S EXPERIENCE
33. TO MAKE COMPARING SENSIBLE, MAKE YOUR TESTS
DETERMINISTIC
Stub systems that you have no control over
34. LOAD TESTING SHOULD BE OPTIONAL, THE ONLY THING
THAT COUNTS IS PRODUCTION!
› Your definition of done should reflect that
› The aim is to get early feedback from a safe environment
36. SO WHAT MONITORING IS TYPICALLY NEEDED?
› Be able to localise where latency is coming from!
– For every system, all incoming and outgoing calls (count and
time spent stats)
› Finite resources (pools, CPU, I/O, etc.)
› Number of active users
› Response size, where possible
› Add whatever you need
It should be identical on all environments!
37. CONCLUSION
In order to put code live without pre-prod load testing, at least
the following need to be in place:
› Culture
› State-of-the-art monitoring
› Resilience
Without these, support your continuous delivery process with
optional load tests and strong specs.
Use the load tests to identify some pain points, so you can
modify the code and add monitoring, making it safer to do
(incremental) dark releases and canary testing in production.