A DevOps Journey - An experience report after 6 years of implementing DevOps and Continuous Delivery in Frende Forsikring, a small insurance company in Norway.
2. 2
Why DevOps?
• Release planning and management
• Handoffs (queue time)
• Manual labor and coordination to get a application into production
• Frustrating long delivery times (both for features and bug fixes)
• Added risk of releasing too many changes at once
«The gap between Dev and Ops will in most organizations result in a major bottleneck,
bringing with it a significant (hidden) cost way outside of IT.»
— Jon Arild Tørresdal
What are these costs?
4. 4
Gains of DevOps
• Freeing time for highly educated and paid employees to focus on
continuous improvement
• Common competence between Dev and Ops results in better efficiency
and innovation
• Faster time to marked
• Fast feedback loops from users, allowing quick adjustments
6. 6
Reform - Ops
• External IT operations
• No access to servers in production
• No Load Balancing during deployment
• Everything virtualized except Database Servers
• Changes and deployments was executed by
hosting provider
• “No” monitoring
• New (still external) IT operations
• Clear separation between Dev and Ops responsibilities
• Dev has full access to all environments, incl. production
• No changes allowed on servers - all changes are
coded as “Infrastructure as Code”
• Everything virtualized, incl. database servers
• Sticky load balancing to ensure optimum scaling
deployment
• Monitoring in production by Dev
• Hosting provider delivers bare OS
• Bootstrapping and configuration of servers +
deployment is fully automated
Before: Now:
7. 7
Reform - Development
• Team Foundation Server for source control and
build
• Some unit testing, but not structured
• Code freeze before each release
• Database schema changes was generated by
diffing test and production
• Bug-tracking
• TeamCity for build (build agents runs at Amazon)
• Git for source control
• Unit tests and integration tests when needed
• Never code freeze
• Extreme focus on low complexity and micro
services
• Every push to Git can potentially end up in
production at any time
• No branching
• Little or no bug tracking
Before: Now:
8. 8
Reform - Deployment
• Deploy packages was handed off to hosting provider
• Release approx. once/mont
• Rollback of code and databases on errors
• Executed early morning or late nigh
• All deployments automated
• Can deploy at any time without any downtime
• Multiple deployments to production every day
• Deployment Pipelines
• Canary releasing (e.g. Customer Center can try out
features before the customers)
• Schema changes in databases are forward and backward
compatible
• No rollback, forward only - fix error, redeploy
• Continuous Deployment in Test and Pre-Prod, Continuous
Delivery in Production
• Full audit of who and when deployments is executed
Before: Now:
9. 9
Reform - Process
• Startup company in rapid growth
• Scrum like
• Business expected release on fixed
date
• Long feedback loops
• Manual testing of the "entire"
application before release
• Company with high cost focus (combined
ratio in insurance)
• Kanban/Lean
• Stakeholders and upper management
participate in daily standups
• Aware of the increased risk with infrequent
releases
• Actively uses canary releasing
• Continuous testing and verification by
stakeholders
Before: Now:
13. 13
Principal #2 - Deploy vs Release
• You can Deploy to production…
• …without Releasing functionality to users
14. 14
Principal #3 - Feature toggles
Image from: http://labs.qandidate.com
15. 15
Principal #4 - NO branching!
• When you branch, you opt out of Continuous Integration!
• No branch, no merge :-)
• The alternative to branching is feature toggles
• If you insist on branching, note the following:
• Reduced delivery capacity
• Reduced quality
• Increased risk of errors
• Added complexity
16. 16
Principal #5 - No manual changes on servers
Image from: http://www.spreadshirt.co.uk
“Admin GUI’s on servers are poison!”
- Build 2015
— Jeffrey Snover, Distinguished Engineer and
Lead Architect for Windows Server
17. 17
Principal #6 - New project, deploy Hello World
Image from: http://fringe.chrisay.com
18. 18
Principal #7 - Version Control Infrastructure
Image from: http://www.bhmpics.com/
20. 20
Principal #9 - Done == In Production!
• No Done Done
• No Done Done Done
• No Done Done Done Done Done….
• Done == In Production!!!
Image from: http://www.definition-of-done.nl
21. 21
The 9 principals
1. Deployment Pipeline
2. Deploy vs Release
3. Feature toggles
4. No branching
5. No manual changes on servers
6. New project, deploy Hello World
7. Version Control Infrastructure
8. Canary Releasing
9. Done == In Production Success!
22. 22
Improvement points
• Container adoption
• Smoke testing
• Performance testing/monitoring
• Automating routers, firewalls and other peripheral network equipment
• Smart monitoring and AI agents
• …and continuous improvement in general
