This document discusses managing performance tests through a pipeline in Jenkins. It identifies issues with current approaches like the need for programming and infrastructure knowledge. It proposes automating the pipeline in Jenkins to dynamically create and delete test client servers, run tests in parallel using Gatling, stash and collect logs, generate reports from logs, and publish results. Sample pipeline code is provided. The goal is to improve the process by making it less manual and complex.

1. Manage Pipeline of Performance Test
with Jenkins
Vol.01 Mar/07/2017
Takeshi Takizawa
Ecosystem Service Department, Rakuten, Inc.

2. Self-introduction
2

3. DEMO
3
Jenkins project (https://jenkins.io/)
GatlingCorp (http://gatling.io)
Test
Client
Test
Client
Test
Client

4. Performance Test Pipeline
4

5. There are some issues
���
5

6. Performance Test Pipeline
6
• It’s difficult to estimate concrete requests �
• Need to clarify that requests come permanently
or instantly �

7. Performance Test Pipeline
7
• Need programming knowledge �

8. Need programming knowledge
• It’s not 10 QPS �
8

9. • Expected: Send requests in parallel
Need programming knowledge
9
Test Client APP

10. Need programming knowledge
• It’s not 10 QPS �
10

11. • Actual: Send requests sequentially
Need programming knowledge
11
Test Client APP

12. Performance Test Pipeline
12
• Need infrastructure knowledge �
• How to install packages
• How to configure number of file descriptor
• Sometimes Test ENG doesn’t have enough permission
to build server so need to ask INFRA ENG �

13. Performance Test Pipeline
13
• Managing distributed test clients is complicated �

14. When I want to send heavy requests
• 1 Test Client doesn’t make enough load
14
Client APP
I can send 1000 QPS
I can respond 2000 QPS

15. When I want to send heavy requests
• Scale Up Spec of Client Server?
15
Client APP
I can send 2500 QPS I can respond 2000 QPS

16. When I want to send heavy requests
• Scale Out is better because more flexible
16
Client
APPClient
Client
We can send 3000 QPS
I can respond 2000 QPS

17. But
17

18. Managing distributed test clients is complicated
• Concurrency �
– Kick test clients at the same time
– Only one Client is running on one Server
18

19. Performance Test Pipeline
19
• It’s boring to generate Test Report from
distributed logs manually �

20. Performance Test Pipeline
20
• Sometimes Test ENG doesn’t have enough permission
to delete server so need to ask INFRA ENG �
• People make mistakes �

21. Automate / Improve Test Pipeline
21
Not Covered
by this slide
Jenkins project (https://jenkins.io/)
GatlingCorp (http://gatling.io)
HashiCorp (https://www.hashicorp.com/)
Chef Software, Inc. (https://www.chef.io/)

22. Need programming knowledge
• Gatling code is writable & readable �
22

23. Test ENG doesn’t have enough permission
23
• Provide Jenkins jobs to create/delete
new Test Client Servers (Jenkins Slaves)

24. Manage Test Clients with Jenkins Pipeline
24

25. Pipeline Code Sample
25

26. 26

27. 27

28. 28

29. 29

30. 30

31. 31

32. Collect Test Logs
• Use stash / unstash DSL
32

33. Generate Report from Test Logs
• gatling command w/ --reports-only option
33

34. Generate Report from Test Logs
34

35. Publish Test Report
• Use other Jenkins plugins through pipeline
– HTML Publisher Plugin
– Compress Artifacts Plugin
35

36. 36

37. Automate / Improve Test Pipeline
37
Not Covered
by this slide
Jenkins project (https://jenkins.io/)
GatlingCorp (http://gatling.io)
HashiCorp (https://www.hashicorp.com/)
Chef Software, Inc. (https://www.chef.io/)

38. Manage Pipeline of Performance Test with Jenkins
38
Jenkins project (https://jenkins.io/)
GatlingCorp (http://gatling.io)

39. We’re Hiring Senior QA Engineer!!
Looking for a talented and experienced
Senior QA engineer
with over 3 years of experience in software QA
who can take on challenging tasks in running
full-scale testing on company’s worldwide services
https://jobs.rakuten.careers/careersection/rakuten_ext_cs/jobdetail.ftl?job=00004405
39