Monster Builds and How to Tame Them Chris Mountford, Atlassian

1. Monster Builds
and How to Tame Them
Chris Mountford
chris@atlassian.com
1

2. Source Code
Build
Working Software
2

3. Truck Build
Small
Monster
x
you
are
here
3

4. The Monster Build Threat
• Shipping
• Unit Testing
• Integration Testing
• Functional Testing
• Selenium Testing
• Synergy Harnessing
4

5. 5

6. Just JIRA
6

7. Just JIRA
(not the biggest)
7

8. Poland
Hosted
Just JIRA Plugins
Multi
8

9. Poland
Hosted
Just JIRA Plugins
Multi
9

10. Wallboards
10

11. Build Duration
• release build took 30-60 CPU hours
• supported platform matrix combinations
• 3.5 hour functional test runs
JIRA
11

12. Long Builds Cause Low Velocity
• Context switching overhead
• Task entanglement
• Build breakage confusion
• Agile?
12

13. 13

14. Taming Monster Build Times
•optimization
•matrix culling
•maximize test power
•minimize wait time
14

15. 15

16. Optimization
• measure first
• reduce network & disk
• RAM disks, proxies, colocation
• cut rework
• measure again
16

17. Example: JIRA Func Tests
•smarter admin navigation
•20% speedup
17

18. Matrix Culling
•browsers
•operating systems
•databases
•libraries
18

19. Example: JIRA Matrix Culling
• user based editions
• dropped ancient app servers
• dropped java 1.4
• release down to 18 hours
19

20. Power to Wait Ratio
functional tests
unit tests
20

21. Be Selectively Continuous
•common customer configurations
•branch vs trunk
•nightly builds
•sunday builds
21

22. tests
features
platforms
browsers
versions
ajax
22

23. 23

24. 24

25. Herb Sutter http://www.gotw.ca/publications/concurrency-ddj.htm
25

26. 26

27. 27

28. Parallelism
28

29. Parallelism
time
agent
Checkout,
Compile Testing
29

30. time
Parallelism
agent
agent
agent
agent
agent
agent
agent
Checkout, Testing
agent
Compileagent
agent
agent
30

31. The Embarrassingly
Parallel Problem
vs.
The Elastic Batches
of Bamboo!
31

32. 15
Inelegant Yet Working
32

33. Inelegant Yet Working
• parametrize your test suite:
• number of batches
• current batch number
• make the suite only run subset of tests in the
requested batch
33

34. Inelegant Yet Working
34

35. Inelegant Yet Working
35

36. Need More
Computers!
36

37. 37

38. 38

39. Reliability
39

40. svn
maven 2
nfs
repo
release
maven 1
build
proxy
agent
JIRA
license
internet
server
test db bamboo
java
blogs
40

41. serve
test db bamboo
java
blogs
41

42. $
False Negatives
42

43. 43

44. canary death
44

45. 45

46. 46

47. 47

48. Maven vs. Bash
48

49. My technology is
better
This ENDS NOW
49

50. Monster Counter-tactics
• measure, optimize, measure
• use power to wait ratio
• elastic parallelization
• canary builds measure reliability
• careful tool evaluation
50

51. Eternal Vigilance
51

52. Chris Mountford
JIRA Dude
chris@atlassian.com
52