1. How to continuously deploy with
confidence. Success story of
wego.here.com
Vadim Ivanov
Event Title | August 31, 2016

2. Source Control Server
… …
Year Commits Lines
added
Lines
removed
2016 2116 733408 326301
2015 4477 253713 254151

3. Agile
You're only as #agile as your ability to ship frequently, and without drama.

4. PR deployments
589 in 2015
721 in 2016

5. How do we do it
• Development rules + code review
• Feature flags + progressive rollouts
• Tests: FE UT, BE UT, Integration tests, UI tests, Smoke tests
• A/B tests
• KPI gate: performance regression tests
• 0 downtime blue green deployments
• Analytics + monitoring
• Continuous deployment

6. Development rules + code review
• New feature only under Feature flag
• Every new feature has a rollout schedule starting from 1%
• Feature can not be deployed without tests written
• Code style + pre-commit git hooks
• No code review for bug fixes without UT

7. Tests + performance regression
• FE UT: 3552, 87% code coverage
• BE UT: 690
• Integration tests: 195
• UI tests: 495 Scenarios
• Smoke tests: a subset of UI tests running against real API on dev, staging, pr
envs
• Web Page Performance: Load time, fully loaded, start render, speed index

8. A/B tests
• Currently running 15 A/B tests
• NPS scoring to evaluate test success

9. Deployment
• Zero downtime blue green deployment
• Continuous Deployment:
~47 mins from commit to production

10. Build pipeline (Jenkins Pipeline)

12. Analytics + monitoring
• Keep an eye after deployment
• Real-time metrics for: system, application, business
• Quality alerts
Bug introduced Bug fixed

13. Conclusion
• Stimulates a culture of quality, the mindset to always keep the product in
releasable state
• Low risk releases: small change -> small risk; zero-downtime deployments
that are undetectable to users
• Faster time to market: Value delivered immediately to the user when it's
ready; Quick feedback on product changes
• Higher quality: automated tools to discover a regression; easy to rollback;
easy to identify bugs
• Better products: A/B testing enable us to take a hypothesis-driven approach
to product development
• Happier teams

14. Thank you
Contact
Vadim Ivanov
vadim.ivanov@here.com
HERE