The DevOps movement provides guidance on better ways to deliver working software to production in a fast, safe and automated manner. Is releasing new features every few weeks still a good approach? How quickly can you get a critical bug fix into production? Are you continually learning and improving? This talked is based on books such as The DevOps Handbook, Release It, and from real world experience delivering "mission critical" microservices in high volume production environments. Come learn how to increase profitability, improve culture, and exceed productivity goals through DevOps practices. We've all messed up releases. Learn from it. Embrace. Improve!
9. 9
Build a fast, reliable pyramid of tests
Test first; automate always
Integrate non-functional
requirements into tests
Pull the andon cord!
Automated
Testing
The First Way: The Technical Practices of Flow
10. 10
Create a
deployment
pipeline
Create a deployment pipeline
Small batch development
Short lived feature branches
Trunk-based development
The First Way: The Technical Practices of Flow
11. 11
Standardize
Environments
Create a single repo of truth
On-demand creation of envs
Rebuild, not repair
Done = running in prod-like env
The First Way: The Technical Practices of Flow
14. 14
Low-Risk
Releases
The First Way: The Technical Practices of Flow
Automated, self-service
deployments
Decouple deployments from
releases
Blue-Green Deployment
Pattern
Canary release pattern
15. 15
The First Way: The Technical Practices of Flow
Limit WIP, make it visible
Reduce Batch Sizes
& handoffs
17. 17
Use telemetry Use peer
reviews and
inspections
Get feedback
from
deployments
The Second Way: The Principles of Feedback
fast and continuous feedback from Operations to Development
18. 18
Use telemetry Use peer
reviews and
inspections
Get feedback
from
deployments
The Second Way: The Principles of Feedback
fast and continuous feedback from Operations to Development
• See problems as they occur
• Log useful metrics
• Overlay other relevant information e.g. releases
• Analyze: Means, SDs
19. 19
Use telemetry Use peer
reviews and
inspections
Get feedback
from
deployments
The Second Way: The Principles of Feedback
fast and continuous feedback from Operations to Development
For deployment and releases:
• Actively monitoring feature metrics
• Have devs initially self-manage prod releases
• Then dev and ops should share pager duty
20. 20
Use telemetry Use peer
reviews and
inspections
Get feedback
from
deployments
The Second Way: The Principles of Feedback
fast and continuous feedback from Operations to Development
• Avoid formal approval processes, manual testing
• Get feedback closer to the source
• Favor peer reviews and inspections
e.g,. GitHub PR, pair programming
• Fearlessly cut bureaucratic processes
22. Enable organizational learning
22
Errors Happen! Blameless
post-mortems
local -> global
learnings
How does your org
respond?
Pathological,
bureaucratic or
generative?
- blame
- fear
+ honesty & learning
share RCAs, code,
best practices
THE AGILE MANIFESTO :A lightweight set of values and principles against heavyweight software development approaches : small batch sizes, incremental releases and small, trusted, motivated teams.
THE LEAN MOVEMENT
a) the best predictor of quality is lead time (time to get something into production)
b) the best predictors of short lead times is small batch sizes of work
CONTINUOUS DELIVERY: using a “deployment pipeline” to ensure that code and infrastructure are always in a deployable state.
TOYOTA KATA: a practice of daily and continuous improvement
White board diagram
Dev -> QA -> Ops -> Production -> Customers
WIP: Queue size s the leading indicators of lead time. Context switching kills!
Stop starting. Start finishing.
2) REDUCE BATCH SIZES Large batch sizes = high levels of WIP and high levels of variability in flow, long lead times and poor quality. For example, the larger the change going into production, the more problems are likely to arise.
3) REDUCE THE NUMBER OF HANDOFFS
Each handoff to another team involves communication, loss of knowledge and delays. Aim to increase flow by reducing handoffs and the time work spends in queues, either by automating or by reorganizing & empowering teams.
4) MAKE OUR WORK VISIBLE: Unlike manufacturing, impeded value streams may not be easily seen in technology. Use visual work boards (e.g., kanban) to visualize flow across the entire value stream.
Create a deployment pipeline
that not just builds and runs tests, but that deploys and runs acceptance tests:
commit -> build -> unit test -> integration tests -> package -> deploys -> acceptance tests.
Goal = get feedback that, at any stage, a change has taken us out of a deployable state. As a result, our deployment pipeline infrastructure becomes as foundational as our version control infrastructure.
CREATE A SINGLE REPO OF TRUTH - Using version control for our environments is even more important than using version control for our code. The use of version control by Ops is a high predictor of both IT performance and organizational performance. We need to be able to repeatedly and reliably reproduce all components of our working software system.
Ensure that we always use production-like environments at every stage of the value stream, ideally created in an automated manner from version control: DB scripts, Tests, Docs, All scripts and config
ENABLE ON DEMAND CREATION OF ALL ENVSTo ensure fast lead times, and consistent environments, provide on-demand/self-service creation of environments.
CONCLUSIONFast flow from Dev to Ops requires production-like environments on demand, from a “single source of truth”, used even at the earliest stages of a software project.
Log useful metrics
Business level e.g., # sales, revenue, user signups
Application level e.g., transaction times, response times, faults
Infrastructure level e.g., server traffic, CPU load, disk usage, etc.
Deployment pipeline level: e.g., failing builds, deployment frequencies
How to think about feedback, before during and after.
Reserve: explicitly reserve time to pay down technical debt e.g., kaizen blitzes