An overview of the Netflix build, bake and deploy infrastructure, and the tools used to enable continuous delivery.

1. Engineering Tools at Netflix
Enabling Continuous Delivery
Mike McGarr
mmcgarr@netflix.com
@SonOfGarr
http://www.MikeMcGarr.com

2. about me
• J. Michael (Mike) McGarr
• 13 years as a software engineer
• Manager of Build Tools, Netflix
• Founder, DC Continuous Delivery Meetup
• Former Director of EngOps, Blackboard Inc.
10/10/14 @SonOfGarr 2

3. 10/10/14 @SonOfGarr 3

4. freedom and responsibility
10/10/14 @SonOfGarr 4

5. cloud native
10/10/14 @SonOfGarr 5

6. microservices (SOA)
10/10/14 @SonOfGarr 6

7. highly aligned, loosely coupled
10/10/14 @SonOfGarr http://bit.ly/1uM62Wi 7

8. open source
10/10/14 @SonOfGarr 8

9. continuous delivery
“Continuous delivery is a set of
practices and principles aimed at,
building, testing and releasing
software faster and more frequently.”
10/10/14 @SonOfGarr 9

10. confidence
10/10/14 @SonOfGarr http://bit.ly/1nV4szx 10

11. teams should
focus on
delivering
business value
10/10/14 @SonOfGarr http://bit.ly/1sb829y 11

12. Engineering Tools
• Simplify build automation
• Codify proven practices
• Enable self-service
• Automate deployments
Not an Ops team!
10/10/14 @SonOfGarr 12

13. Paved Road
10/10/14 @SonOfGarr http://bit.ly/1oIqYGA 13

14. Paved Road
10/10/14 @SonOfGarr 14

15. BUILD
10/10/14 http://bit.ly/1vMALQ8 @SonOfGarr 15

16. Netflix Build Language (NeBuLa)
• Gradle plugins
• Netflix OSS
http://github.com/nebula-plugins
10/10/14 @SonOfGarr 16

17. build script
10/10/14 @SonOfGarr 17

18. building OS packages?
10/10/14 @SonOfGarr 18

19. locking dependencies?
10/10/14 @SonOfGarr 19

20. generate lock file
build.gradle
dependencies.lock
10/10/14 @SonOfGarr 20

21. Jenkins
• Continuous integration
• ‘nuff said
10/10/14 @SonOfGarr 21

22. multiple masters
Billing Merch
OpsEng
PPD ProdEng
MCE
Memb
Platform
10/10/14 @SonOfGarr 22

23. Jenkins Job DSL
• Groovy DSL
• Define job config in
source
https://github.com/jenkinsci/job-dsl-plugin
10/10/14 @SonOfGarr 23

24. jobs.dsl.groovy
10/10/14 @SonOfGarr 24

25. BAKE
10/10/14 http://bit.ly/1CTzPNO @SonOfGarr 25

26. immutable servers
• Avoid configuration drift
• Change a line of code = build a new OS
10/10/14 @SonOfGarr 26

27. Baking Process
10/10/14 @SonOfGarr 27

28. Aminator
• Python library
• Build application AMI
• NetflixOSS
https://github.com/Netflix/aminator
10/10/14 @SonOfGarr 28

29. Bakery
• REST API’s for Aminator
• Single API for all regions
• Not OSS
10/10/14 @SonOfGarr 29

30. DEPLOY
10/10/14 @SonOfGarr http://1.usa.gov/1uBl4wF 30

31. AWS “applications”
10/10/14 @SonOfGarr 31

32. Asgard
• Netflix AWS console
• Provides “app”
abstraction
• Hides AWS keys
• NetflixOSS
https://github.com/Netflix/asgard
10/10/14 @SonOfGarr 32

33. deployments
10/10/14 @SonOfGarr 33

34. deployments
10/10/14 @SonOfGarr 34

35. red/black deployment
10/10/14 @SonOfGarr 35

36. inventing clusters
@SonOfGarr 36
10/10/14

37. canary releases
10/10/14 @SonOfGarr 37

38. Mimir
• Deployment
workflows
• Integrates
Jenkins/Bakery/Asg
ard
• Prototype, not OSS
10/10/14 @SonOfGarr 38

39. we are evolving
• Unifies Asgard & Mimir
• Netflix needs have evolved
• Highly available
10/10/14 @SonOfGarr 39

40. Simian Army
• Kill/inspect running instances
• Chaos Monkey
• Janitor Monkey
• Security Monkey
• Conformity Monkey
• Chaos Gorilla*
• Chaos Kong*
https://github.com/Netflix/SimianArmy
https://github.com/Netflix/security_monkey
10/10/14 @SonOfGarr 40

41. what I didn’t cover
• Ribbon
• EvCache
• Hystrix
• Governator
• Karyon
• Edda
• Archaius
• Eureka
• Zuul
• Data/Big Data tools
10/10/14 @SonOfGarr 41

42. Zero to the Cloud
• Stand-up your own
NetflixOSS cloud
• Aminator
• BaseAMI
• Asgard
• Edda
• Eureka
https://github.com/Netflix-Skunkworks/zerotocloud
10/10/14 @SonOfGarr 42

43. Thank you!
• Mike McGarr
• mmcgarr@netflix.com
• @jMichaelMcGarr
• http://www.mikemcgarr.com
We are Hiring!
netflix.com/jobs
10/10/14 @SonOfGarr 43