Adding Performance Verifications in Continuous Delivery

1. Adding Performance Verifications
in Continuous Delivery
by Federico Toledo
(COO at Abstracta)
federico@abstracta.us
@fltoledo

2. WHO IS ABSTRACTA?
Abstracta is a world leader in
software testing focused on
harnessing automation to improve
performance and reduce time to
market. Our mission is to make the
world a better place by co-creating
winning technologies of the
highest quality.
4 Offices Worldwide
80 Software Quality Engineers
10 Years in Business
95% Client Retention Rate
+300 Projects
Thought Leaders
Testing Educators

3. www.abstracta.us
Agenda
Introduction to the
typical approach to
performance
testing
Why it does
not work for
continuous
delivery
Conclusion
A different
approach with a
different goal
21 3 4

4. Introduction
1

5. www.abstracta.us
vs
Response Times
Resource Consumption
Performance

6. What is
performance
testing
about?

7. www.abstracta.us
Source:http://www.miniatur-wunderland.de/
Simulation

8. www.abstracta.us
Measurement

9. www.abstracta.us
Finding the Bottlenecks and the Breaking Point

10. www.abstracta.us
Types of Performance Testing

11. www.abstracta.us
Load Simulation Tools

12. www.abstracta.us
}
Simulation

13. www.abstracta.us
Simulation

14. www.abstracta.us
HTTP
HTTP
Server
Simulation

15. www.abstracta.us
Demo
Simple DEMO Using JMeter
Typical approach:
• Define bunch of test cases (user flows) and load scenario.
• Automate them with the tool (http level).
• Run tests, analyze, work with devs and ops to improve.
• Continue until you reach the SLAs or good enough
performance (response times and resource consumption).

16. www.abstracta.us
What’s special about Continuous Delivery?
We want to enable the business people to deliver the
product to the customers as soon as they want.
Continuous
Integration:
merge code at least
once a day.
Continuous
Deployment:
deliver continuously
to users.
Continuous
Delivery:
deliver continuously to
business people,
frequently to users
(production).
1 2 3
We need an automated pipeline with different checks.
Let’s add performance validations to this pipeline!

17. Do you think that the typical approach
for performance testing would work?
2

18. www.abstracta.us
Issues with the typical approach for CI/CD
The pipeline will run at least once a day
(probably more frequently)
End-to-End scripts are
fragile (need lots of
maintenance)
If there is an issue or
degradation, it’s hard to
find the cause
1 2

19. What can we do differently?
Different approach with different goals
3

20. www.abstracta.us
1 2
A different approach with a different goal
Frequent releases to production implies:
• Smaller difference between versions.
• Reduced risk (less uncertainty).
Detect any degradation in the
system as soon as possible.
Typical goal:
Know if the system will
support the expected load.
Goal for performance tests in
Continuous Delivery:

21. www.abstracta.us
Definition of
the test cases
to automate
Detecting degradations as soon as possible
Selection of
the tool
Design of the
load scenarios
1 2 3

22. www.abstracta.us
We love JMeter, but...

23. www.abstracta.us
We love JMeter, but...

24. www.abstracta.us
We love JMeter, but...

25. www.abstracta.us
Understanding what changed

26. www.abstracta.us
Takeaways
A Choose a CI-friendly tool.
www.abstracta.us

27. www.abstracta.us
Let’s talk about the test scripts
What is the simplest test we can
run in order to detect degradations?
www.abstracta.us

28. www.abstracta.us
What is the simplest test we can run in
order to detect degradations?
Hint:
We want to avoid false
negatives.
Specially because you
will run these tests on a
daily basis.
End-to-End tests (http
level) are fragile.
We want to test early
and frequently.
Another hint:Remember:

29. www.abstracta.us
Cohn’s Pyramid on Test Automation
Equivalent to:
HTTP traffic between user and server.
Combining different endpoints.
Test endpoints in isolation.

30. www.abstracta.us
Cohn’s Pyramid on Test Automation
User metrics
User centered design
More costly (scripting, infrastructure)
Cheaper (easier)
Early results
More Frequent
No idea about the user experience

31. www.abstracta.us
What we want to do:
Let’s add unit performance tests to our pipeline
This way we discover automatically any
degradation of the system as soon as possible.
• Independent tests.
• Each one running against one endpoint at a time.

32. www.abstracta.us
Detect degradations right after they are introduced
Requestspersecond
0
50.0
100.0
150.0
200.0
250.0
300.0
09/12 09/16 09/20 09/24 09/28 10/02 10/06
Means requests
per second
Performance
assert threshold
Release branch
created
Percent KOs
0
25
50
75
100
PercentageKOs

33. www.abstracta.us
Detect degradations right after they are introduced
ResponseTimes
0
100
200
300
400
500
600
09/12 09/16 09/20 09/24 09/28 10/02 10/06
Average response
times (ms)
Performance
assert threshold
Release branch
created
Percent KOs
0
25
50
75
100
PercentageKOs

34. www.abstracta.us
Define scenario and assertions

35. www.abstracta.us
Define scenario and assertions

36. www.abstracta.us
Define scenario and assertions

37. www.abstracta.us
Define scenario and assertions

38. www.abstracta.us
Define scenario and assertions

39. www.abstracta.us
Define scenario and assertions

40. www.abstracta.us
Define scenario and assertions
Load
● 350 threads
Assertions
Given
● Exclusive environment
● < 1% error
● P95 Response Times < 130ms + 10%
● Throughput >= 150 TPS – 10%

41. www.abstracta.us
You should also check Keptn and Pitometer

42. www.abstracta.us
Takeaways
A Choose a CI-friendly tool.
www.abstracta.us
B Unit performance tests to detect degradations
as soon as possible.

43. Is that enough?
4

44. www.abstracta.us
Fallacy of the composition
Testing the parts is not enough to know how the whole will behave.
You need to test the whole to see how the parts interacts.
Parts Parts
Parts Parts
Whole

45. www.abstracta.us
Testing Strategy
Every month?
Every week?
Every day?

46. www.abstracta.us
Testing Strategy
Similar to production
Scaled down

47. www.abstracta.uswww.abstracta.us
Takeaways
B
Choose a CI-friendly tool.
Unit performance tests to detect degradations as
soon as possible.
C Complement with integration and load tests.
A

48. Is that enough?
4

49. www.abstracta.us
What about the client side?
Web
- Google page speed
- Website.io
- APM
- Etc!
What about mobile?

50. www.abstracta.us
Apptim (www.apptim.com)

51. www.abstracta.us
Summary of my learnings
www.abstracta.us
B
C
Choose a CI-friendly tool.
Unit performance tests to detect degradations as
soon as possible.
Complement with integration and load tests.
Don’t forget the client side.
A
D

52. Adding Performance Verifications in
Continuous Delivery
by Federico Toledo (COO at Abstracta)
federico@abstracta.us
@fltoledo
¡Muchas
gracias!