2. Pulling a rabbit from a hat
How to make your apps run faster
George Barnett, Performance Engineer, Atlassian
2
2
3. Why are we here?
⢠How to make your Atlassian software run faster
⢠What doesnʟt work
⢠What Atlassian does to improve performance
⢠No development HOWTOs or code!
3
3
4. Who can improve performance?
⢠Atlassian
⢠YOU! (Thanks for watching!)
4
4
7. Performance Testing Scripts
⢠Apache JMeter
⢠In beta for JIRA
⢠Ask support@atlassian.com
⢠Available for ConďŹuence
⢠http://conďŹuence.atlassian.com/display/DOC/Performance+Testing+Scripts
7
7
8. Performance Testing
⢠Real world data and load
⢠Public Atlassian instances
⢠Performance test suite
⢠JMeter & scripts
⢠Automation
⢠Maven & Bamboo
8
8
9. Performance Testing
⢠Daily performance tests
⢠Most products & libraries
⢠Weekly soak tests
⢠Run for much longer than daily tests
⢠Reporting for developers
⢠Build screens, dashboards, notiďŹcations
9
9
12. Upgrade your Atlassian Software!
⢠Dozens of Performance issues ďŹxed
⢠Release criteria includes performance
12
12
13. You are here
⢠How I did the testing
⢠Hardware & Software
⢠Must have improvements
⢠Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
⢠Worth doing
⢠Gigabit vs 100mbit, Heap size tuning, use an SSD
⢠Sadly, not useful
⢠Tune MySQL, Replace MySQL
13
13
14. The contenders
⢠Dell PE 850 - âWALL-Eâ
⢠Xeon X3220 2.4Ghz Quad Core
⢠Quad Core, 2 x 4Mb L3
⢠8Gb DDR2 RAM
⢠2 x 7.2K SAS Disks
⢠1 x Intel X25-E 32GB SSD
⢠~$1500
14
14
15. The contenders
⢠Dell PE 2950 - âJohnny 5â
⢠2 x Xeon E5405 2.0Ghz
⢠Quad Core, 2 x 6M L3
⢠32Gb RAM
⢠2 x 72Gb 15K Drives
⢠~$4000
15
15
16. The contenders
⢠Dell PE R610 - âEVEâ
⢠2 x Xeon E5520 2.2Ghz
⢠Quad Core, 8M âSmartCacheâ
⢠32Gb Ram
⢠2 x 146G 15K drives
⢠~$4000
16
16
17. The Workload
⢠JIRA 4, MySQL 5, RHEL 5
⢠~70,000 Issues, ~80 projects
⢠~30 reqs/sec, JMeter 2.3.4 (patched, JSR-223)
⢠TrafďŹc modelled on http://jira.atlassian.com with higher writes
17
17
18. About the graphs
⢠Most show Average Response Time in Milliseconds
⢠Includes time to generate AND deliver the response
⢠Does not include any browser time
⢠Arrows show if lower or higher is better!
18
18
19. You are here
⢠How I did the testing
⢠Hardware & Software
⢠Must have improvements
⢠Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
⢠Worth doing
⢠Gigabit vs 100mbit, Heap size tuning, use an SSD
⢠Sadly, not useful
⢠Tune MySQL, Replace MySQL
19
19
21. Upgrade Java 1.5 to 1.6
⢠Advances in compiler / VM technology
⢠Produces better runtime code
⢠Able to use modern instructions, eg SSE
⢠Able to use modern hardware, eg NUMA
21
21
22. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5.0_18 Java 1.6.0_16
1500
1125
750
375
0
Average Response Time (ms)
6.3X
22
22
23. Java 1.5 vs 1.6 (Johnny 5)
Java 1.5.0_18 Java 1.6.0_16
380
285
190
95
0
Average CPU (%)
26%
23
23
24. Upgrade Java to 1.6
⢠3 x HotSpot VM updates in 1.6.0_XX
⢠Advancing technology
⢠Compressed OOPS on 64bit
⢠Escape Analysis
⢠G1 Garbage Collector
⢠NUMA
24
24
25. Java 1.6 vs 1.6 (Eve)
Java 1.6.0_7 Java 1.6.0_16
148
146
143
141
138 7%
Average Response Time (ms)
25
25
26. Hardware Upgrades
⢠Hyper-Threading
⢠QuickPath Interconnect
⢠Nehalem E5520 (Eve) have 3X on die DDR3 Memory Controllers
⢠Memory throughput improvements - good for Java!
Technology Speed
DDR2-800 (dual channel) 6.4 GB/sec
DDR3-1333 (dual channel) 21.3 GB/sec
26
26
27. FSB architecture
⢠Shared bus
⢠1333 MT/s = 1333M Transfers / sec
⢠4 transfers/clock = 266Mhz.
RAM
CPU
FSB
North
Bridge
CPU South
I/O: SATA, USB, etc
Bridge
27
27
32. Garbage Collection
⢠Javaʟs Virtual Machine manages memory
⢠New objects are continually created during runtime
⢠GC is the process of collecting dead objects to reclaim memory
32
32
33. Tune Garbage Collection
⢠Getting your GC tuning wrong is bad
⢠How bad?
⢠Are the defaults sane?
33
33
34. Tune Garbage Collection
CMS ParNew ParOld
4000
3000
2000
1000
0
Wall-E (ms) Johnny 5 (ms)
Average Response Time
Eve (ms)
2X
34
34
35. Tune Garbage Collection
CMS ParNew ParOld
500
375
250
125
0
Johnny 5 (ms)
Average Response Time
Eve (ms)
2X
35
35
36. Tune Garbage Collection
⢠Defaults are sane
⢠Parallel New + Parallel Old is fastest
⢠CMS prioritises low pauses over CPU usage
⢠CMS does not compact - can lead to heap fragmentation
36
36
38. Remove VMWare
Real ESX 3.5 ESX 4i
170
128
85
43
0
Average Response Time (ms)
43%
38
38
39. Remove VMWare
⢠On average ~30% slower
⢠Under extreme resource starvation, up to 100% slower
⢠Avoid VMWare for high trafďŹc instances
⢠See the Atlassian docs for recommendations
⢠http://conďŹuence.atlassian.com/display/DOC/Running+ConďŹuence+in+a+Virtualised+Environment
⢠http://conďŹuence.atlassian.com/display/JIRA/Running+JIRA+in+a+Virtualised+Environment
39
39
40. Upgrade your browsers!
⢠Use a modern browser
⢠IE8, Safari 4 or Firefox 3.6
⢠Pick the browser thats right for you
⢠Avoid IE6 (unsupported and slow)
⢠http://sixrevisions.com/infographics/performance-comparison-of-major-web-browsers/
40
40
41. You are here
⢠How I did the testing
⢠Hardware & Software
⢠Must have improvements
⢠Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
⢠Worth doing
⢠Gigabit vs 100mbit, Heap size tuning, use an SSD
⢠Sadly, not useful
⢠Tune MySQL, Replace MySQL
41
41
43. Gigabit Network vs 100Mbit
⢠Gigabit is faster, sure
⢠But 100Mb is OK if youʟre not doing big transfers, RIGHT?!
43
43
44. Use Gigabit
100mb/sec 1000mb/sec
1900
1825
1750
1675
1600
Average Response Time (ms)
10%
44
44
45. Use Gigabit
⢠Lower RTT on Gigabit
⢠Unlikely to hit throughput limit
⢠If you have enough CPU power, keep the DB on localhost
45
45
46. Use Gigabit
100Mbit 1000Mbit
200
188
175
163
150
ReIndex Time (seconds) 26%
46
46
47. Heap size tuning
⢠Memory starvation is bad, but can we drown the system with too much
memory
47
47
48. Heap size - Wall-E
1.5G 3G 6G
2000
1500
1000
500
0
Average Response Time
23%
48
48
49. Heap size - Johnny 5
1.5G 3G 6G 15G
290
218
145
73
0
Average Response Time 49%
49
49
50. Heap size - Eve
1.5G 3G 6G 15G
170
128
85
43
0
Average Response Time 28%
50
50
51. Heap size tuning
⢠Heap too small - bad
⢠Heap too big - not bad!
⢠Bigger heaps give the JVM more room to work
⢠Diminishing returns from really big heaps
51
51
52. Buy an SSD
⢠SSDs are FAST!
⢠SSDs are EXPENSIVE!
⢠SSDs are all the rage!
⢠DBAs are ďŹnally happy (almost!)
52
52
53. Buy an SSD
SATA SSD
800
600
400
200
0
Browse Issue Create Issue 33%
53
53
54. Buy an SSD
⢠Fast writes
⢠0.1ms Avg Service Time
⢠Most improvement comes from SSD on the DB
⢠Writing to the Lucene index means re-opening IndexReaders
⢠Much faster on SSD
⢠Lucene Search Term Cache ďŹts in OS buffers
54
54
55. You are here
⢠How I did the testing
⢠Hardware & Software
⢠Must have improvements
⢠Upgrade Java, Hardware, Remove VMWare and tune Garbage Collection
⢠Worth doing
⢠Gigabit vs 100mbit, Heap size tuning, use an SSD
⢠Sadly, not useful
⢠Tune MySQL, Replace MySQL
55
55
57. Replace MySQL with MySQL
⢠MySQL 5.0.45 - RHEL 5
⢠MySQL 5.0.84 Percona B18.
⢠Percona HighPerf patches ďŹx contention points
⢠Improved I/O
⢠Better scaling
57
57
58. Replace MySQL with MySQL
MySQL 5.0.45 Percona MySQL 5.0.85
1687
1515
1344
1172
1000 0.4%
Average Response Time (ms)
58
58
59. Replace MySQL with MySQL
MySQL 5.0.45 Percona MySQL 5.0.84
25
19
13
6
0 0%
Average CPU(%)
59
59
60. Replace MySQL with MySQL
⢠JIRA doesnʟt do enough DB queries
⢠JIRA uses Lucene for many queries
⢠Percona Highperf MySQL really shines at high query rates
⢠Try this if you share your DB server with other apps!
60
60
62. Tune MySQL
Defaults Tuned
1710
1533
1355
1178
1%
1000
Average Response Time (ms)
62
62
63. Tune MySQL
MyISAM ALL Engines
join_buffer_size - Large joins
without Indexes
key_cache_age_threshold,
key_cache_block_size,
sort_buffer_size - Used by
key_cache_division_limit,
ORDER BY / GROUP BY
key_buffer_size
read_buffer_size,
query_cache_size
read_rnd_buffer_size
query_cache_limit
bulk_insert_buffer_size
query_cache_min_res_unit
63
63
64. Tune MySQL
⢠Be sure to set innodb_buffer_pool_size
⢠Note! innodb_log_ďŹle_size will affect shutdown speed
64
64
65. Remember!
⢠Upgrade!
⢠Atlassian software
⢠Java to 1.6
⢠Your server
⢠Use Parallel Garbage Collection with a BIG heap
⢠Remove VMWare if you can
⢠Use 1000mbit network
⢠Buy an SSD for write workloads
65
65
67. Thanks! Questions?
⢠Come chat to me in the hallway!
⢠Charlie Lounge, 10 June, 3pm - 4pm
⢠http://blogs.atlassian.com/developer
Flickr Attributions:
Siesta in Denmark - http://www.flickr.com/photos/jakecaptive/27123533/
Day 11 - http://www.flickr.com/photos/ivanomak/4057632458/
Blue Streak - http://www.flickr.com/photos/jettajet/2061715292/
67
67