A presentation to the BankWest Solution Delivery team and the Perth DevOps Meetup describing the delivery processes at Envato that enable us to deploy 10 times a day.

1. 10 Deploys a Day
a case study of
continuous
delivery at
envato
AT
john viner
PRESENTATION TEMPLATE from Envato’s Graphic River - http:/
/graphicriver.net/item/karbon-keynote-presentation-template/2580765

2. envato ABOUT US
7 years ago - a designer scratching an itch
creates a flash-plugin marketplace
and now …
‣
‣
‣
3.2m members
4.6m Items
10 Authors sold > $1m
‣
‣
‣
130m PVs p/m
20M Visits p/M
Item sold every 10 sec

3. Envato
our marketplace Sites

4. Envato
wordpress - our biggest market
> 3400+ Themes for Sale
> Most Popular Theme has sold 37,500 times and generated $2m in
gross revenue
> Average price of a Theme is $42
> Average sales per TF author is 2,075
> Provides a Passive Income source
!
19% of the web runs on Wordpress! (4B monthly page view)

5. Envato
we’ve grown - and continue to grow
> Themeforest is now Ranked ~ #180 on the Alexa Rankings
> My development team was 8, 15 months ago and is now 24

6. Envato
the marketplace team
40 - in melbourne
‣ 30 - remote around the world
‣
‣
delivery team
- 19 Back End Developers
- 2 Front End Developers
- 1 Operations Developer
- 5 Product Managers and UX Designers
6 Teams
‣ finance,
‣ back office,
‣ front end,
‣ maintenance
‣ search
‣ 10x
‣

7. Envato
marketplace tech stack
ruby on rails web app
‣ mysql database
‣ elastic Search
‣ rackspace managed hosting in USA
‣
> 90M App Server Requests per Week
> Backend Response time of 148 ms
> Front end Response time of 2.5sec
> 45 Virtual Machines
> 4 Physical Machines (DBs, NFS)
> Avg 6 deploys a day - Peak of 10
> Average 100 commits a day
> Peak Load of 15,000 req per minute
> 4 Physical Machines (DBs etc…)

8. THIS IS HOW WE ROLL

9. Envato
pull work, talk to Product, write code, test code
deploy code, verify production
repeat … about twice a week per developer
> Story on the wall
> flip or no flip?
> Create a Branch
> write failing test
> write code
> Run Local Tests
> Run full personal build
> create Pull request
> advertise pull request (IRC)
!
optional
> deploy to staging
> show product owner
> manual test
> wait for +1s (code review)
> Merge Pull request
> Full Master Build Green
> notify team in IRC
> deploy master to Production
> watch System - monitoring tools
> rollback if failed
> Rinse and Repeat

10. ENVATO
in pictures
Story
Tests
Code

11. ENVATO
flippin good
>Flip allows us to roll out changes to % of users
> Something might be flipped off for months
> But we Always Deploy It

12. ENVATO
quality
Code Review
CI Build
Deploy

13. coordinating reviews
and deploys .. by irc
You Need Deep Dev and
Operational Experience right?
> Luke - Graduate with 4 months Exp
> Rakesh - Swinburne Industry Student
> Ben - Front End Developer
> Emmanuel - 2 yrs Dev Exp
4 Other Developers Mentioned
Across 4 Separate Teams

14. ENVATO
now check the systems
New Relic
Deploy
With Change
Traceability

15. things can break …
And Do

16. ENVATO
deploys gone bad
Rollbar Error Logging

17. Envato
the post incident review recommends?
>1 - Improve our ability to Automatically rollback on monitored 500 errors
!
> 2 - NO Recommendation to Improve Preventative Testing
!
> Why? - Too Costly and complex to simulate production
> Why? - Would slow development flow

18. so who operates this
site anyway?

19. Envato
1 developer all developers
you build it
you run it
> 16 Developers on a Weekly Rotation
> They come out of their team
> They are Trained, have a Buddy for First Week
> Supported by a Secondary
> They are First Line of Support for 168 hours straight
Tooling is:
> Scout , New Relic and Pingdom monitoring and generating alerts
> Pagerduty forwarding to them On-Call Developer
> Rollbar to track Errors
> IRC to find out what went wrong
> ALL Alerts are also pumped into IRC so everyone knows whats going on and to get Help!

20. Envato
we reflect on failures
and act to correct

21. Envato
but we aren’t great at operations
> We are just Good Enough right now, Just
> The rotation leaves without a continuous view of system health
> We don’t always clean up behind us, root causes often come back
> Monitoring, Metrics, Performance insights are all Operational Skills
that many developers don’t have
> We are hiring Operational Specialists for the first time
> Our Single DevOps guy is actively training all the devs - AJs Cafe

22. Envato
so what do we not do
> No - testing role or formal testing gate
> NO - change control gate
> No - formal comms of every change
> NO - release management
> No - separate deployment team
> NO - Separate operations team
all of these are not feasible
at 10 deploys a day

23. Envato
how do we solve the problems that these practices solve ?
‣
No Testing Capability
> TDD and 1000s of automated tests
> Use Production Users as your test team
> Flips help us control the rollout
> We test major infrastructure and architecture
changes in production as well … using Flips
> We Fail Fast and Loud
‣
No release management
> Releases are SMALL, no need to define
or document
> Release self-document and radiate
through IRC
‣
No change control
> Operations (i.e. Developers) know of changes
before they happen because they review them as PRs
> There are changes we cannot “signoff” as testing in
pre-prod environments
> A green build is our sign off
> Devs have skin-in-game so are incentivised for the
code to work!
> Small Batch efficiency (small deltas are easier to
identify and rollback)

24. Envato
OK How did we do it?
‣
it was our startup dNA
> The Team started and stayed very small (less
than 8 devs) for 4 years
> They started with this structure and process
and have never changed.
‣
it scales
> We’ve tripled the team, and split into 5 streams
> We’ve been able to scale this model
‣
but with a lag
> But it took 3 months to see the results

25. Envato
so do you want to drink the kooL-aid?
‣
it might not be for you
‣
> We have a tolerant savvy users
> We have users that come back to
complete a purchase if it fails
> We are in CONTROL of our full stack
‣
If you want it, you have to change
> people
‣ > processes
‣ > assumptions
‣‣
> Tech is the easy part
> You can’t optimise your “gatekeeping” processes to 10 deploys
a leap is required
> Existing processes can get you close, say once a fortnight or a week
> A leap is required to get to many times a day

26. Envato
distributed collaboration vs central control
quality
‣ release planning
‣ deployment
‣ operations
‣
!

27. did I say Continuous
Delivery or deployment?
meet the real
rockstars at
Envato …

28. Envato
the microlancer team
automatically deploy every
commit to master … always

29. Envato
whats the difference?

30. Envato
what our next challenges?
continuous automated reactive deployments
‣ flips + split testing and with every product feature
‣ flexibility of our infrastructure
‣ improved operations
‣
!

31. envato THANK YOU
@envato
@veztek
webuild.envato.com