A Build Engineering Team’s Journey of scaling a CI/CD platform that is used within a fast growing company

1. PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV
Build Engineering @ Atlassian:
Providing CI / CD as a Service
AWS Summit Sydney - April 2015

2. SCALING THE BUILD GRID
INTRODUCTION
Introduction
AWS ADVANTAGES
FUTURE STATE

3. Build platform & services
used internally within
Atlassian to
build, test & deliver
software

4. Developers expect a
reliable infrastructure &
fast CI feedback

5. Build Engineering today @ Atlassian
• 10 Bamboo Servers
• maven.atlassian.com / 6 Nexus instances
• Monitoring - opsview / graphite / statsd
• 1000 build agents (own hardware + EC2 instances)
• include SCM clients, JDKs, JVM build tools, databases,
headless browser testing, python builds, NodeJS, installers
& more
• Maintain 20 AMIs of various build configurations
• Nexus proxies

6. 3 years ago:
21kBuilds per month

7. Last month:
127kBuilds per month

8. Build Engineering today @ Atlassian

10. JIRA alone has
47kAutomated tests

11. SCALING THE BUILD GRID
INTRODUCTION
Scaling the Build Grid
AWS ADVANTAGES
FUTURE STATE

12. Build Agents

13. Own Hardware
using KVM guests
+
EC2 instances

15. Differences between KVM
guests & elastic build agents
+
the need for
Functional Parity

16. 2 Problems

17. Problem 1:
Network connectivity
differences

18. Solution:
AWS Direct Connect
+ VPC enabled EC2
instances

19. Problem 2:
Configuration
differences between
KVM guests & elastic
build agents

20. Solution:
Removing tech debt using
https://www.packer.io/ https://www.atlassian.com/software/bamboohttps://puppetlabs.com/ https://cukes.info/

21. SCALING THE BUILD GRID
INTRODUCTION
AWS Advantages
AWS ADVANTAGES
FUTURE STATE

22. Handle the waves
of demand

23. Prices always decreasing
Product range expanding
https://aws.amazon.com/blogs/aws/category/price-reduction/

24. Hardware procurement
API + Credit Card

25. ‘Do Not Resuscitate’
order on existing
build agent hardware

26. Spot instances

27. Cheap!

28. Slower startup
times

29. Not always
available

30. Price Volatility

31. Expect Failure
Embrace It

32. Prebake at AMI
burn time rather
than at EC2
instance startup
time

33. Faster startup time

34. More reliable
instance startup
(+ retry on failure!)

35. Pre-caching of
git repositories,
docker images

36. “RequestLimitExceeded.
The maximum request rate
permitted by the Amazon
EC2 APIs has been exceeded
for your account”

37. Handle failure,
retry with backoff
& recover

38. “We currently do not have
sufficient m3.large capacity in
the Availability Zone you
requested”

39. Multiple-AZ support
in Bamboo 5.8

40. “we are experiencing
increased error rates
and latencies for the
EC2 APIs”

41. SCALING THE BUILD GRID
INTRODUCTION
Future State
AWS ADVANTAGES
FUTURE STATE

42. Moving all Build
Eng Infra to AWS

43. EC2 to ECS for
build agents

44. Come help us!

45. Thank you!
PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETER LESCHEV