This document discusses principles and practices of continuous deployment at Etsy. It describes how Etsy moved from deploying code changes every 2-3 weeks with stressful release processes, to deploying over 30 times per day. The key principles that enabled this are innovating continuously, resolving scaling issues quickly, minimizing recovery time from failures, and prioritizing employee well-being over stressful releases. Automated testing, deployment to staging environments, dark launches, and extensive monitoring allow for frequent, low-risk deployments to production.
5. Five years ago…
2-3 weeks of code changes
Release and rollback plans
Traffic and infrastructure management (Ops)
6-14 hours
6. Five years ago…
“Deployment Army”
Stressful, especially when things go wrong
Long days and late nights
Scheduled downtime
7. pro·duc·tion [pruh-duhk-shuhn] (n)
1. This complex system of application code,
distributed services, servers, networking gear, etc.,
upon which we’re going to try to carefully apply a
complicated set of changes and hope that nothing
goes wrong. Cross your fingers… here goes.
8. Software for large-scale web sites has been
traditionally written by one group of people, then
released and operated by a different group.
These two groups have very different levels of
visibility into how the software works.
35. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
36. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
37. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Source: http://en.wikipedia.org/wiki/Continuous_delivery
Continuous Delivery release pipeline
38. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
39. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Continuous Delivery release pipeline
Dev / Integration Staging Production
Assumptions:
Staging is a perfect reflection of
Production, with respect to
hardware, configurations, data,
overall load, capacity, etc.
Deploy process is infallible.
40. “What do you mean, ‘it’s not working in
production?’ I TESTED IT BEFORE WE
RELEASED!”
41. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
42. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Dev / Integration Staging Production
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
43. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
"Because you’re integrating
so frequently, there is
significantly less back-
tracking to discover where
things went wrong , so you
can spend more time
building features.”
!
—ThoughtWorks
!
!
http://www.thoughtworks.com/continuous-integration
Dev / Integration Staging Production
44. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
Where’s the bug?
!
In one of the numerous check-ins?
Missing unit tests?
Missing automated UA tests?
Missing manual UA tests?
45. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
Where’s the bug?
!
In one of the numerous check-ins?
Missing unit tests?
Missing automated UA tests?
Missing manual UA tests?
!
Data out of sync?
Server configurations out of sync?
Capacity vs. current load?
Deployment script?
46. Dev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Release
Check in
Trigger
Trigger
Approval
Continuous Delivery release pipeline
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Check in
Trigger
Trigger
Approval
Approval
Check in
Trigger
Trigger
Approval
Feedback
Dev / Integration Staging Production
How will we know when
something is wrong in
production?
!
How long will it take to
resolve the issue?
Check in
Trigger
47. We aim to reduce fundamental surprise in
every release.
49. Pre-production validation
Code deployed to de-pooled application (web) servers
touching prod services and databases.
Smoke tests
Integration tests
Functional tests
User-Acceptance (ad hoc)
50. Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce application
code to live traffic.
51. Production validation
Exactly the same server configs, services and data as
pre-prod, but this is where we introduce application
code to live traffic.
Smoke tests (esp. over public hostnames)
User-Acceptance testing behind config flags
Gratuitous monitoring
Customer support and forums
52. Single
release
Many
releases
50K LOC/month
Few opportunities for failure
Wide surface area (50,000 LOC)
High MTTR
!
All of the bugs we’ve written
More opportunities for failure
Narrow surface area (< 100 LOC)
Low MTTR
!
A fraction of the bugs we’ve
written per release
Imagine that we’ll write
57. Check in
Trigger
Feedback Trigger
Feedback Approval
Approval
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
Feedback
58. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
59. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback
60. Dev Pre-Production
(“Princess”)
Production
Check in
Trigger
Feedback Trigger
Feedback Approval
ApprovalFeedback
Smoke Tests
User Acceptance!
Tests
ReleaseDev Team Version Control
Build & Unit
Tests
Automated
Acceptance Tests
User Acceptance
Tests
Deploy (Prod)
Monitoring and
Automated Alerts
Continuous Deployment release pipeline
CI
Approval
Approval
Feedback
Feedback
Feedback Approval
Feedback Approval
Feedback
91. $1.35 Billion Goods sold in 2013
60+ Million Unique visitors per month
!
175+ Committers, everyone deploys
http://www.etsy.com/blog/news/2013/etsy-statistics-december-2012-weather-report/Items by anjaysdesigns, betwixxt, OneStarLeatherGoods, mediumcontrol, TheDesignPallet