Netflix uses continuous delivery practices powered by open source tools to deploy code rapidly and reliably across multiple AWS regions. Teams deploy their own code using tools like Nebula/Gradle and Jenkins Job DSL for automated builds. The Aminator creates AMIs and Asgard deploys them using red/black deployment. Simian Army monkeys like Chaos Monkey test resiliency. Self-service, awareness of regions, and rollback ability are key to Netflix's approach.

2. Continuous Delivery at Netflix:
Powered by Open Source
Dianne Marsh
San Francisco, May 6, 2014
http://www.linkedin.com/in/diannemarsh

3. Building Continuous Delivery

4. Netflix Goals
High Availability
But … Move fast
Tools encourage Best Practices
But … Freedom to do the right thing

5. Teams Deploy Their Own Code
Run What You Wrote
• Rapid Innovation
• Rapid Detection
• Rapid Response
= Freedom +
Responsibility

6. Nebula/Gradle
• Netflix Build Language
• Gradle plugins
• Supports internal and open source
builds
https://github.com/nebula-plugins
BUILD

7. BUILD
Jenkins Job DSL
Configuration as Code
Groovy Script
Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code

8. BUILD BAKE

9. Aminator
• Create AMI from Base AMI
• Image contains service and
everything needed to run it
• Unit of Deployment for Test and
Prod
• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html
BAKE

10. BUILD DEPLOYBAKE

11. DEPLOY
Asgard: AWS Deployment Tool
Deploys Netflix to the Cloud
Red/Black push
http://www.infoq.com/presentations/asgard

12. CANARY ANALYSIS
Test, Int, Prod
Choose where to deploy
Run canary analysis
Scale up new instances
Turn on traffic to new ASG
Turn off traffic to old ASG
Wait … analyze … continue

13. Asgard Developer Portal

14. GLISTEN Extending Asgard’s Workflow
Automated Red/Black Push
Run canary/analysis
Scale up new instances
Turn on traffic/Run Tests/Turn off traffic
Wait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html

15. One Button Deployment?

16. Regional Isolation
Limit Impact of Human Error
• Stagger deployments?
• Canary testing per region?
Know your service!

17. Multi-Region Consistency
Build Tooling to:
• Schedule Deployments
• Prefer off peak
• Choose next available
region automatically
• Provide high visibility
per region

18. BUILD
DEPLOY
BAKE

19. Simian Army
• Chaos Monkey
• Latency Monkey
• Janitor Monkey
• Conformity Monkey (and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

20. Send in the Conformity Monkey
Have deployments diverged?
 Balance regional consistency
with regional isolation
 Provide meaningful
thresholds
 Build best practices into
tooling and reporting

21. Clean up with the Janitor Monkey
• Disassociate unused
EIPs
• Delete unassociated
Amazon EBS volumes
• Delete older Amazon
EBS snapshots
• Leverage Amazon S3
Object Expiration

22. Key Elements for Netflix
• Value Self-service
• Test Everywhere
• Build Awareness of Multiple Regions
• Avoid peak times
• Roll back quickly and easily
• Be Cloud Native

23. Netflix OSS
More at netflix.github.com

24. Thanks!
Dianne Marsh (@dmarsh)
dmarsh@netflix.com
Thanks!