In this session, Oleksandr Metelytsa and Benjamin Magro will take a technical deep dive into Atlassian's new apps performance testing framework to see how it works, how it can be used to test your Data Center applications, where it is heading in the future, and what we will require for Data Center apps performance testing going forward.

1. Technical Deep Dive
Into Atlassian's New Apps Performance Testing Framework
BEN MAGRO | PRODUCT MANAGER | ATLASSIAN | @MAGRO
OLEKSANDR METELYTSIA | TEAM LEAD | ATLASSIAN

2. Background Our
Approach
Demo Questions
Agenda

4. BACKGROUND

6. FPO

7. FPO
QUICK SHOW
!"#$OF

8. THE PROGRAM’S GOALS

9. 1. DELIVER AN ENTERPRISE
GRADE ECOSYSTEM FOR OUR
CUSTOMERS MISSION CRITICAL
DEPLOYMENTS
THE PROGRAM’S GOALS

10. 1. DELIVER AN ENTERPRISE
GRADE ECOSYSTEM FOR OUR
CUSTOMERS MISSION CRITICAL
DEPLOYMENTS
2. PROVIDE A WAY FOR VENDORS
TO DIFFERENTIATE THEIR DC
APPS
THE PROGRAM’S GOALS

11. Enterprise
Grade

12. Enterprise
Grade
1. Architectural Review

13. Enterprise
Grade
1. Architectural Review
2. Performance Testing

14. Enterprise
Grade
1. Architectural Review
2. Performance Testing
Approval

15. 620+Data Center Approved Apps

16. 93%of the tops apps are approved

17. DC APPS LAUNCH
SEPTEMBER 2018

18. VENDOR FEEDBACK
ATLASSIAN CODE BARREL INNVALOG BRICKIT
JQL TRICKS BITVOODOO BEECOM
ADAPTAVIST SMARTBEAR SNAPBYTES APPFIRE SERVICEROCKET

19. VENDOR INTERVIEWS
15

20. VENDOR INTERVIEWS
15PERFORMANCE TESTING
TOOK WEEKS

21. VENDOR INTERVIEWS
15PERFORMANCE TESTING
TOOK WEEKS
CREATING DATASETS

22. VENDOR INTERVIEWS
15PERFORMANCE TESTING
TOOK WEEKS
CREATING DATASETS
LIMITED GUIDANCE

23. Existing Tools
PERFORMANCE TOOLS

24. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case

25. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case
• Missing Data Sets

26. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case
• Missing Data Sets
• Not Cross Product

27. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case
• Missing Data Sets
• Not Cross Product
• Tightly Coupled to Cloud Provider

28. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case
• Missing Data Sets
• Not Cross Product
• Tightly Coupled to Cloud Provider
• Time Consuming

29. Existing Tools
PERFORMANCE TOOLS
• Specific Use Case
• Missing Data Sets
• Not Cross Product
• Tightly Coupled to Cloud Provider
• Time Consuming
• Inconsistant Output

30. BETA

31. INTRODUCING
BETA
DATA CENTER APP
PERFORMANCE TOOLKIT

32. INTRODUCING
BETA
DATA CENTER APP
PERFORMANCE TOOLKIT

33. BETA
8.0.X

34. 8.0.X
CO
M
IN
G
SO
O
N
7.13.X
7.0.X
6.13.X
6.0.X
5.0.X

35. GOALS OF THE PERFORMANCE TOOLKIT

36. Easy
Quick to setup and
run. Detailed
walkthroughs.
GOALS OF THE PERFORMANCE TOOLKIT

37. Easy
Quick to setup and
run. Detailed
walkthroughs.
Ship with data
Ship with realistic
enterprise datasets
GOALS OF THE PERFORMANCE TOOLKIT

38. Easy
Quick to setup and
run. Detailed
walkthroughs.
Ship with data
Ship with realistic
enterprise datasets
Decoupled
Not tied to
deployment type or
dataset
GOALS OF THE PERFORMANCE TOOLKIT

39. Easy
Quick to setup and
run. Detailed
walkthroughs.
Ship with data
Ship with realistic
enterprise datasets
Decoupled
Not tied to
deployment type or
dataset
Extensible
Built to be
customised to test
your apps
GOALS OF THE PERFORMANCE TOOLKIT

40. OUR ASK WHAT IT’S NOT

41. Carved In StoneSeeking Feedback
OUR ASK WHAT IT’S NOT

42. Carved In Stone
Mandatory
Seeking Feedback
Recommended
OUR ASK WHAT IT’S NOT

43. Carved In Stone
Mandatory
Built only for Atlassian Apps
Seeking Feedback
Recommended
Try to use it for your apps
OUR ASK WHAT IT’S NOT

44. Carved In Stone
Mandatory
Built only for Atlassian Apps
Abandonware
Seeking Feedback
Recommended
Try to use it for your apps
Raise Improvements
OUR ASK WHAT IT’S NOT

45. bit.ly/DCAPTK
BETA

46. BETA

48. OUR
APPROACH

50. How hard is
performance
testing?

51. Hardware
Provisioned hardware, number of application nodes,
db power and instance configuration

52. Hardware
Provisioned hardware, number of application nodes,
db power and instance configuration
Configuration
Performance load profiles, datasets,
Product configuration

53. Hardware
Provisioned hardware, number of application nodes,
db power and instance configuration
Configuration
Performance load profiles, datasets,
Product configuration
Software
Performance tools, load generation scripts, product
and tools versions

54. Support App Developers
By providing tooling and guidance on
how to measure the performance and
scalability of apps developed for
Atlassian’s DC products.

55. Start: fix moving parts

56. Start: fix moving parts
Jira versions
Jira Software 8.0.x and ER 7.13.x

57. Start: fix moving parts
Jira versions
Jira Software 8.0.x and ER 7.13.x
Deployment
AWS Quickstart with defined instance
types and parameters

58. Start: fix moving parts
Datasets
Realistic enterprise-scale dataset
Jira versions
Jira Software 8.0.x and ER 7.13.x
Deployment
AWS Quickstart with defined instance
types and parameters

59. Start: fix moving parts
Datasets
Realistic enterprise-scale dataset
Scenarios
Performance regression and scalability
Jira versions
Jira Software 8.0.x and ER 7.13.x
Deployment
AWS Quickstart with defined instance
types and parameters

60. OUR GOALS THINGS TO AVOID

61. Reinventing the wheelUtilise existing tools/learnings
OUR GOALS THINGS TO AVOID

62. Reinventing the wheel
Custom wrapper
Utilise existing tools/learnings
JMeter for REST
OUR GOALS THINGS TO AVOID

63. Reinventing the wheel
Custom wrapper
Ignore UI
Utilise existing tools/learnings
JMeter for REST
Selenium for UI measures
OUR GOALS THINGS TO AVOID

64. Reinventing the wheel
Custom wrapper
Ignore UI
Hardcode/magic numbers
Utilise existing tools/learnings
JMeter for REST
Selenium for UI measures
Simple yml config
OUR GOALS THINGS TO AVOID

65. Reinventing the wheel
Custom wrapper
Ignore UI
Hardcode/magic numbers
Monolith
Utilise existing tools/learnings
JMeter for REST
Selenium for UI measures
Simple yml config
Extensible
OUR GOALS THINGS TO AVOID

66. Taurus
automation
tool
Free and open source
Under Apache 2.0 License

67. Thin layer
An abstraction layer over JMeter and Selenium
Taurus
automation
tool
Free and open source
Under Apache 2.0 License

68. Thin layer
An abstraction layer over JMeter and Selenium
Platform-independent
Runs where Python and Java can run
Taurus
automation
tool
Free and open source
Under Apache 2.0 License

69. Taurus
For orchestration and
configuration
Data Center Application Performance Toolkit

70. Taurus
For orchestration and
configuration
JMeter
JMeter scripts to
generate base load
Data Center Application Performance Toolkit

71. Taurus
For orchestration and
configuration
JMeter
JMeter scripts to
generate base load
Selenium
Selenium PyTest
scrips to measure UI
response
Data Center Application Performance Toolkit

72. Taurus
For orchestration and
configuration
JMeter
JMeter scripts to
generate base load
Selenium
Selenium PyTest
scrips to measure UI
response
Utils
Deployments scripts
Datasets upload
Reports and charts
generation
Data Center Application Performance Toolkit

74. DEMO

76. Instance Setup
How to easily set

77. Prepare ENV
Launch Jira DC
Upload Dataset
Install Toolkit
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.

78. Prepare ENV
Launch Jira DC
Upload Dataset
Install Toolkit
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.

79. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

80. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

81. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

82. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

83. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

84. Prepare ENV
Launch Jira DC
Upload Dataset
Configuration
Shape usage
Please only use circles, rectangles, and
rounded rectangles to call attention to a
particular part of a screenshot, for the
sake of consistency.
Install Toolkit

85. Launch Jira DC
Upload Dataset
Configuration
Prepare ENV
Install Toolkit

86. Launch Jira DC
Upload Dataset
Configuration
Prepare ENV
Install Toolkit

87. Upload Dataset
Configuration
Prepare ENV
Launch Jira DC
Generate Eval
Use the UI too generate an
evaluator license.
Contact Atlassian
We can provide you 2 time
bomb licenses for testing. Ask
on your DCHELP ticket.
License Setup
Install Toolkit

88. Launch Jira DC
Upload Dataset
Configuration
Prepare ENV
Install Toolkit

89. Launch Jira DC
Upload Dataset
Configuration
Restore DB from Postgres SQL Dump
By executing populate_db.sh script.
Prepare ENV
Install Toolkit

90. Launch Jira DC
Upload Dataset
Configuration
Restore DB from Postgres SQL Dump
By executing populate_db.sh script.
Upload attachments
By executing upload_attachments.sh script.
Prepare ENV
Install Toolkit

91. Launch Jira DC
Upload Dataset
Configuration
Restore DB from Postgres SQL Dump
By executing populate_db.sh script.
Upload attachments
By executing upload_attachments.sh script.
Perform Re-Index
Start Re-Index on Jira system preferences page.
Prepare ENV
Install Toolkit

92. $ wget https://dapt_repo/populate_db.sh
$ chmod +x populate_db.sh
$ ./populate_db.sh
Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index

93. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
--2019-08-26 05:22:25-- https://centaurus-
datasets.s3.amazonaws.com/jira/8.0.3/large/db.dump
Resolving centaurus-datasets.s3.amazonaws.com (centaurus-
datasets.s3.amazonaws.com)... 52.219.88.116
Connecting to centaurus-datasets.s3.amazonaws.com (centaurus-
datasets.s3.amazonaws.com)|52.219.88.116|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7340090453 (6.8G) [application/x-www-form-urlencoded]
Saving to: ‘db.dump’
31%
[================================================================>

94. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
pg_restore: creating FK CONSTRAINT "public.AO_E8B6CC_PR_TO_COMMIT
fk_ao_e8b6cc_pr_to_commit_request_id"
pg_restore: creating FK CONSTRAINT "public.AO_E8B6CC_REPO_TO_PROJECT
fk_ao_e8b6cc_repo_to_project_repository_id"
pg_restore: creating FK CONSTRAINT "public.AO_F1B27B_KEY_COMP_HISTORY
fk_ao_f1b27b_key_comp_history_timed_promise_id"
pg_restore: creating FK CONSTRAINT “public.AO_F1B27B_KEY_COMPONENT
fk_ao_f1b27b_key_component_timed_promise_id"
real 68m38.656s
user 1m43.237s
sys 0m11.933s
Step6: Start Jira

95. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
$ wget https://dapt_repo/upload_attachments.sh
$ chmod +x upload_attachments.sh
$ ./upload_attachments.sh

96. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index
[300000/2000000 entries] [1.7 G/11.1 G transferred]
[2289 entries/s] [13.1 M/s bw] [monq 0] [jq 469]
[600000/2000000 entries] [3.3 G/11.1 G transferred]
[1929 entries/s] [11.0 M/s bw] [monq 0] [jq 369]
[900000/2000000 entries] [5.0 G/11.1 G transferred]
[666 entries/s] [3.8 M/s bw] [monq 0] [jq 269]
[1059000/2000000 entries] [5.9 G/11.1 G transferred]
[457 entries/s] [2.6 M/s bw] [monq 0] [jq 216]
[2000000/2000000 entries] [11.1 G/11.1 G transferred]
[385 entries/s] [2.2 M/s bw] [monq 1] [jq 1]

97. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index

98. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index

99. Upload
Dataset
Restore DB
Upload
attachments
Perform Re-Index

100. Launch Jira DC
Upload Dataset
Configuration
Clone git repository
$ git clone dcapt.git
Prepare ENV
Install Toolkit

101. Launch Jira DC
Upload Dataset
Configuration
Clone git repository
Follow README.md instructions
$ git clone dcapt.git
$ pip install bzt
Prepare ENV
Install Toolkit

102. Launch Jira DC
Upload Dataset
Configuration
jira.yml file
Prepare ENV
Install Toolkit

103. Launch Jira DC
Upload Dataset
Configuration
jira.yml file
Prepare ENV
Install Toolkit

104. Launch Jira DC
Upload Dataset
Configuration
jira.yml file
env:
application_hostname: jira.atlassian.com
application_protocol: http
application_port: 80
admin_login: admin
admin_password: admin
concurrency: 200
test_duration: 45m
WEBDRIVER_VISIBLE: False
Prepare ENV
Install Toolkit

105. Scenario 1: Performance Testing
With No App Data

106. Scenario 1:
Performance
Testing
Run 1: Without
App
Install App
Run 2: With App
Base line
Performance run for fixed configuration without an
app:
$ bzt jira.yaml
Analyse Report

107. Scenario 1:
Performance
Testing
Run 1: Without
App
Install App
Run 2: With App
Base line
Performance run for fixed configuration without an
app:
JMeter
Constant JMeter load.
$ bzt jira.yaml
Analyse Report

108. Scenario 1:
Performance
Testing
Run 1: Without
App
Install App
Run 2: With App
Base line
Performance run for fixed configuration without an
app:
JMeter
Constant JMeter load.
Selenium
Measure timings for UI actions.
$ bzt jira.yaml
Analyse Report

113. Install App
Load Your App
Run 1: Without
App
Run 2: With App
Analyse Report
Scenario 1:
Performance
Testing

114. Install App
Load Your App
INSTALL
Load App
DATA
Setup Data
Run 1: Without
App
Run 2: With App
Analyse Report
Scenario 1:
Performance
Testing

115. Run 2: With App
Run
The same run with app installed.
$ bzt jira.yaml
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

116. Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

117. column_name: 90% Line
title: Performance Testing
runs:
- runName: without app
resultsPath: results/run1
- runName: with app
resultsPath: results/run2
performance_profile.yml
Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

118. column_name: 90% Line
title: Performance Testing
runs:
- runName: without app
resultsPath: results/run1
- runName: with app
resultsPath: results/run2
performance_profile.yml
Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

119. column_name: 90% Line
title: Performance Testing
runs:
- runName: without app
resultsPath: results/run1
- runName: with app
resultsPath: results/run2
performance_profile.yml
Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

120. column_name: 90% Line
title: Performance Testing
runs:
- runName: without app
resultsPath: results/run1
- runName: with app
resultsPath: results/run2
performance_profile.yml
$ python csv_chart_generator.py performance_profile.yml
Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

121. Run 2: With App
Run 1: Without
App
Install App
Analyse Report
Scenario 1:
Performance
Testing

122. INDEX TIMING TEST
Now is a great
time to run your
re-index test

123. Extending The Tool
To Test Your Apps

124. How to extend the Performance Toolkit
Modify JMeter
Test your app-specific
endpoint at high load
(e.g. REST APIs).
Modify Selenium
Test your app with a real
browser in parallel
(e.g. load times)

125. EXTENSION FILES

126. Modifying JMeter script
main script
(jira.jmx)
extension script
(extension.jmx)

127. (extension.jmx)
More on modifying JMeter

128. Modifying Selenium script
main script
(jira-ui.py)
extension script
(extension-ui.py)
...
def test_1_selenium_view_issue
(webdriver, datasets):
view_issue(webdriver, datasets)
def test_1_selenium_custom_action
(webdriver, datasets):
custom_action(webdriver, datasets)
...
def custom_action(webdriver, datasets):
@print_timing
def measure(webdriver, interaction):
webdriver.get(f'{URL}/plugin')
# WebDriverWait wait condition
measure(webdriver, 'selenium_view_report')

129. Running tests with your modification
main config
(jira.yml)
bzt jira.yml
settings:
...
concurrency: 10
test_duration: 5m
scenarios:
jmeter:
script: jmeter/jira.jmx
properties:
# Workload model
total_actions_per_hr: 54500
perc_create_issue: 100

130. With App Data
Scenario 2: Scalability Testing

131. Scenario 2:
Scalability
Load and Setup Your App
Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyze report

132. Scenario 2:
Scalability
Load and Setup Your App
INSTALL
Load App
DATA
Setup Data
Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyze report

133. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyze report
Base line
Performance run for fixed configuration with app-
specific actions on one node Jira Data Center.
$ bzt jira.yaml
Scenario 2:
Scalability

134. Run 3: One node
Run 5: Four nodes
Analyze report
Scenario 2:
Scalability
Run 4: Two nodes

135. Run 3: One node
Run 5: Four nodes
Analyze report
Scenario 2:
Scalability
Run 4: Two nodes

136. Run 3: One node
Run 5: Four nodes
Analyze report
Scale DC to two nodes
$ bzt jira.yaml
Scenario 2:
Scalability
Run 4: Two nodes

137. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyze report
Scale DC to four nodes
Set 4 as Min and Max number of cluster nodes in Jira
AWS Quickstart template and update Stack.
$ bzt jira.yaml
Scenario 2:
Scalability

138. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
column_name: 90% Line
title: Scale Testing
runs:
- runName: one node
resultsPath: results/run3
- runName: two nodes
resultsPath: results/run4
- runName: four nodes
resultsPath: results/run5
Report yml
Scenario 2:
Scalability

139. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
column_name: 90% Line
title: Scale Testing
runs:
- runName: one node
resultsPath: results/run3
- runName: two nodes
resultsPath: results/run4
- runName: four nodes
resultsPath: results/run5
Report yml
Scenario 2:
Scalability

140. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
column_name: 90% Line
title: Scale Testing
runs:
- runName: one node
resultsPath: results/run3
- runName: two nodes
resultsPath: results/run4
- runName: four nodes
resultsPath: results/run5
Report yml
Scenario 2:
Scalability

141. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
column_name: 90% Line
title: Scale Testing
runs:
- runName: one node
resultsPath: results/run3
- runName: two nodes
resultsPath: results/run4
- runName: four nodes
resultsPath: results/run5
Report yml
Scenario 2:
Scalability

142. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
column_name: 90% Line
title: Scale Testing
runs:
- runName: one node
resultsPath: results/run3
- runName: two nodes
resultsPath: results/run4
- runName: four nodes
resultsPath: results/run5
Report yml
$ python csv_chart_generator.py report.yml
Scenario 2:
Scalability

143. Run 3: One node
Run 4: Two nodes
Run 5: Four nodes
Analyse report
Scenario 2:
Scalability

144. Performance Testing 101
Congrats you’ve graduated
performance testing 101 with the Data
Center App Performance Toolkit

145. FEEDBACK

146. SLACK GROUP
bit.ly/dcapt_slack
ISSUE TRACKER
bit.ly/dcapt_issue
FEEDBACK

148. QUESTIONS

150. Thank you!
BEN MAGRO | PRODUCT MANAGER | ATLASSIAN | @MAGRO
OLEKSANDR METELYTSIA | TEAM LEAD | ATLASSIAN