Envato runs one of the world’s highest traffic websites, Themeforest.net, which is 1 of the 8 sites served by a 2-sided e-commerce platform called Envato Market. The Envato Market engineering team juggles serving around 30,000 requests per minute whilst automatically deploying changes once every 30 minutes across a team of 50 developers that contribute to the one codebase.
This presentation describes how we do this and what our the foundational engineering and organisational practices require to pull this off.
3. We run a two-sided Marketplace for creative assets
that has paid out $450 million to our “Authors” in the
last 10 years
7 million registered users
10 million items for sale
The Business
4. 25,000 requests a minute
160m Page Views a Month
1 sale a second at Peak
120,000 lines of code (Ruby on Rails)
25,000 tests
75 contributors, 10 years old
The Tech
5. Continuous Deployment vs Continuous Delivery
“Continuous Delivery is a software development
discipline where you build software in such a way
that the software can be released to production at
any time.”
“Continuous Deployment is a software development
practice in which every code change is deployed to
production, automatically.”
6. Foundational Engineering Capabilities
Visible Operational
Monitoring
Feature Toggle Tooling
Rollback Capability
Automated Build and Deploy Toolset
Comprehensive Automated Test Suite
Quality Code
Continuous Deployment
7. Foundational Organisational Behaviours
MTTR Focus
No QA / Approval Gates
Access to Production
Blameless Post Incident Reviews
Developers on First Line Support
Pairing and/or Peer Review
Continuous Deployment
8. The Lifecycle of a Change at Envato
Pull Request (PR) for all changes - Github
Developer Merges PR with at least 1 review
Automated Tests executed (no perf tests) - Buildkite
Automated Deploy executed on passed test - Samson + CodeDeploy
Rolling Deploy of Application with Zero Downtime
Schema Migrations with Zero Downtime
11. The Numbers for June 2016
~ 40 developers
3000 total builds (2.5 builds p/dev/day)
500 Pull Requests Closed
400 deployments (20 p/day, 0.5 p/dev/day, peak 25 per day)
4.5 hour cycle time (from first commit to deploy - working hours)
8000 deployments in the last 2 Years
Pull Request Size - 37 lines changed , 3 files changed
14. The Benefits for Envato
1. Reduced Risk.
• Small Changes are Easier to Identify and Rollback (or forward)
2. Increases Speed to Market
• Small Batch Efficiency - Faster Throughput
3. Increases Developer Happiness
• Code in Production makes for happy devs
• Supporting Production increases empathy with customer
15. How to Get Started
• Start on a Small System
• Encourage a Culture of Developer Ownership
• Invest in Test Automation
• Automate Your Build and Deployment Pipeline
• Invest in Feature Toggles
• Measure Your Cycle Time
• Optimise for Mean Time to Recovery (not Failure)