Every customer-facing project aims to achieve high uptime. We, as developers, write thorough unit and integration tests to catch as many bugs as possible. We even set up staging environments to simulate production as well as we can. Yet, downtime still happens and not only once we had to revert a change. Blue-Green deployment is a technique that allows us to ship code with high confidence, minimizing the risk of errors. We will go through the why and what of Blue-Green,as well as show how you can start doing it using Spinnaker. BDEVOPS @ Adobe 11.04.2017

1. Safe deployments with
Blue-Green and Spinnaker
BDEVOPS
11.04.2017

2. Hi
Mihnea Dobrescu-Balaur
@mihneadb
https://www.mihneadb.net

3. Deployment
• getting product out to clients
• agility matters - first to market
• speed vs risk

4. Rolling deployment
• classic/easiest way of doing things
• on every server: git pull; restart
• you are probably doing some variation of this

5. http://searchitoperations.techtarget.com/definition/rolling-deployment

6. Rolling deployment (cont.)
• Advantages:
• simple
• no extra hardware
• can do partial deploys

7. Rolling deployment (cont.)
• Disadvantages:
• might drop requests
• slow rollback
• finding the "last good state"
• testing in staging only

8. Blue-Green deployment
• two identical production environments
• only one serves requests
• deploy to the new one; test; route traffic

9. http://fbrnc.net/blog/2016/05/green-blue-deployments-with-aws-lambda-and-cloudformation

10. Blue-Green deployment
(cont.)
• Advantages:
• testing in production as well
• no downtime
• quick rollback
• predictable release time

11. Blue-Green deployment
(cont.)
• Disadvantages
• cost - 2x servers
• databases require extra work
• development time to implement

12. Fixing the database problem

13. Optimizations
• connection draining
• canary
• sticky sessions
• automated tests on green
• reproducible deploys using AMIs/Docker images

14. Real world
• using it at Hootsuite since summer 2015
• 1K deploys in prod with no issues*
• has not affected our velocity
• deploy to prod is a mundane task
* trivial rollback in case we had any

15. Real world (cont.)
• Tech stack:
• Docker
• Asgard
• Jenkins
• ELB
• Hubot

16. Our deployment pipeline
http://code.hootsuite.com/blue-green-deployments/

17. Our deployment pipeline
(cont.)
• cluster check
• conflict check
• merge tests
• live tests
• git merge
• manual canary release
• manual blue-green switch

18. Spinnaker
• successor of Asgard (thanks, Netflix)
• custom build and deploy pipelines
• multiple cloud providers supported
• multi-zone, multi-region

19. Why Spinnaker?
• easy trigger integration with Jenkins
• handles all the "cloud" stuff
• server groups
• load balancers
• security groups
• useful templates built-in

20. https://cloud.google.com/solutions/spinnaker-on-compute-engine

21. Spinnaker stack
• Jenkins
• Packer
• .deb repo
• Spinnaker core

22. Deploying .debs
• not everybody used to this
• lifesaver: fpm
• running your stuff - upstart

23. How you'd do it
http://www.spinnaker.io/docs/from-source-to-prod

24. Test pipeline
• trigger
• Jenkins, manual, git, Docker registry etc.
• bake
• deploy to test
• destroy last server group

25. Deploy pipeline
• validate test deploy
• manual, script, another pipeline etc.
• find image
• deploy to prod
• using red/black strategy

26. http://www.spinnaker.io/docs/from-source-to-prod

27. Spinnaker
• orchestrates your deploy pipeline
• use Jenkins only for building artifacts
• useful templates for everything you'd need
• cloud agnostic
• friendly UI with auth and logging