We will go over the motivations for wix.com R&D to move to a CI/CD/TDD model, how the model was implemented and the impact on Wix R&D. We will cover the tools used (developed in-house and 3rd party), change in methodologies, what we have learned during the transformation and the unexpected change in working with product and the rest of the company. Presented in the Continuous Delivery track at DevOps Con Israel 2013

1. Continuous Delivery @ Wix
Yaniv Even-Haim
VP R&D
DevOpsCon
2013

2. Wix Intro

3. Wix in numbers
• 29,000,000 registered users from 233 countries
• ~1,200,000 new users every month
• ~ 30,000 new websites every day
• 26,000,000 web sites
• 800,000 mobile sites
• 1Tbyte of media uploaded every day
• More than 1 Billion users media files
• 250+ servers in 3 data centers
• 350 employees
• 100 R&D people

4. A story on Wix time machine

5. CD is business concern
Revenue per month
2010-…
2010-…
2010-…
2011-…
2011-…
2011-…
2012-2
2010-8
2010-9
2011-1
2011-2
2011-3
2011-4
2011-5
2011-6
2011-7
2011-8
2011-9
2012-1
2012-3
2012-4
Traditional “Economies of scale” threaten by cloud technology
Be faster than small startups
CD “Economies of Agility “ = Velocity X Size

6. CD is culture & mindset
• Trust the developers
– Empower developers to change production
– Developer knows his system best

7. CD is culture & mindset
• Automation as a default choice
– no more “is it worth to automate ? ”
– Everything should be automated

8. Welcome to the twilight zone
• Product/Dev/QA/Ops boundaries are going down
– Everyone responsible for everything

9. Welcome to the twilight zone
• Move fast - Less formality
– Corridor - IN , Meeting Room - Out

10. Get out of thought land
make sure you building the right it
before build it right
• The Law of Failure
– Most new “its” will fail even if they are flawlessly
executed
• Invest less, in-touch less , better ability to admit it fail
– Data beats opinions - Let the market decide

11. Some Best Practices
• Everyone develops on the Trunk/Master
– Release frequent – small pieces of functionality
– It’s the developer responsibility not to break anything
• Code can get to production at anytime
– TDD & integration test
• Use Feature Toggle
– Unbaked new code can go to production – no harm done
– New code goes with a guard – use new or old code

12. Some Best Practices
• Backwards and Forwards compatible
– Each component has to function with latest, next or prior
version of other components (including DBs)
• Gradual Deployment & Self Test
– Deploy new version to one server and perform self test. If
it passes, continue deployment to other servers
• A/B Testing
– Open a new feature to a percent of your users. Is it better?
– Deployment is an engineering decision

13. CD – prepare to invest…..
• Dev infrastructure - Refactor , Refactor, Refactor
• Testing infrastructure & know how
• Deployment infrastructure & tools
• Automation , Automation , Automation
• Monitoring (business and technical)
– hundreds of aspects
– thresholds use is a Must
– Monitor business KPIs
– Internal & external
– Endless Tuning & learning

14. Scaling challenges
• Deployment complexity
– More services
– More dependencies
– Multi data centers
• Development complexity
– Very hard to work on master/trunk with 30+ dev on same
code base
– backward and forward comptability is not easy
• Wider set of technologies ,tools & languages
– Higher level of infrastructure investments
– Harder to have all around players
– More communication/sync required

15. Scaling challenges – Quality
– Testing coverage challenge
• Use cases increases
• Browsers wild west
• Tablets and mobile variety
– Feature toggle combinations
• Dozens of parallel feature toggles
– Production frequent changes
• Services are updated daily
• Toggles switch on/off on daily basis

16. Scaling challenges - Availability
• Service Availability
• Cost of fault/downtime increasing as business grow
• Service downtime during deployment is unacceptable
• Human mistakes are inventible when changing production
daily

17. Scaling challenges – Monitoring
• Monitoring in multi technology env
• Unified dashboard require major effort
• Reduce alert noises resulted by deployments
• Identify root cause is not simple
– Some faults does not have immediate impacts
– Cascading effects
• Reliable business KPI without false positive
– Marketing changes
– Special events (holidays, sport event etc)

18. Scaling challenges – Product
• Product MVP practices
– Does MVP meet your product standards?
• What about tooltip,help,first time ux, etc.. ?
– How to define a product that can be developed in a day ?
– And that can win in a/b test …
To Be
Implemented

19. Scaling challenges – Org
• Complex Matrix Management
- People moving between teams frequently
- One can have several managers on different tasks
- Hard both to managers and employees
- Everyone should master the matrix management skills

20. Wix Developer Lifecycle

21. Wix Developer Lifecycle

22. New Relic - Monitoring

23. Application dashboard

24. Chix – Staging Deployment

25. It is not all or nothing game - it is
continuous journey