Finding bottlenecks in our software delivery processes is often pretty easy. But once we squash one bottleneck, another team becomes the limiting factor. This presentation looks how bottlenecks work, and how to predict the next bottleneck you'll need to work on.
2. Lead Consultant & Tech Evangelist
Eric is Lead Consultant at IBM
UrbanCode Products where I help
customers get the most out of their
build, deploy and release processes.
Today he works with customers and
industry leaders to figure out this
DevOps thing.
Eric Minick
eminick@us.ibm.com
@EricMinick
3. The plan
Theory of constraints in a nutshell
Finding bottlenecks
Predicting the next bottleneck
Common bottleneck pushing patterns
Q&A
6. Only by increasing flow through the constraint
can overall throughput be increased*
Making the bottle
wide down here does
not help
* The Goal: a process of ongoing improvement. Goldratt eta al
8. Your system must have a measureable goal
My goal is to empty a
wine bottle quickly
9. Simplified five step plan
1. Identify the system’s most severe constraint
2. Decide how to get the most out of the constraint
3. Subordinate everything else to the above constraint
4. Make changes to expand constraint’s capacity
5. Once constraint is relieved, return to step 1
10. Example: Slow QA cycles
1. Identify the system’s most
severe constraint
2. Decide how to get the most
out of the constraint
3. Subordinate everything else
to the above constraint
4. Make changes to expand
constraint’s capacity
5. Once constraint is relieved,
return to step 1
1. It takes too long to test our
changes. Everyone else waits
2. Testers should focus on
exploratory testing
3. Devs help with regression
tests. Ops prioritizes QA.
4. Dev & QA work together to
automate regression tests
5. Find the next bottleneck
11. Wait, what’s our goal in software?
Generally: turn ideas into business
value
Measuring “business value” hard
Emptying wine bottles only relevant if dev speed is the constraint
and you believe in the Ballmer Peak (http://xkcd.com/323/)
?
12. Wait, what’s our goal in software?
Generally: turn ideas into business
value
Measuring “business value” hard
Features delivered minus bugs is a
decent approximation
– …but rewards building useless features.
Emptying wine bottles only relevant if dev speed is the constraint
and you believe in the Ballmer Peak (http://xkcd.com/323/)
?
13. Three key measures
Lag time: how long from idea to value?
Throughput: how much delivered value
per unit time?
Cost: what does it cost to deliver value?
?
15. Most teams can feel the constraint
What are you waiting on?
Where’s the pain?
Constraints before you in a
process feel like not enough
work.
Constraints after you in a
process are annoying or
painful
16. If it hurts, do it more often
Painful processes often grow
exponentially worse with large
batch sizes.
Examples
– Integration work
– Releases
– Bug Triage
– Updating databases
– Visiting the dentist
http://martinfowler.com/bliki/FrequencyReducesDifficulty.html
Time between doing it
Pain
17. Use Lean techniques to measure
What does it take to get a change from idea to production?
–At each phase measure wait time and work time
Long wait times indicate large batch sizes or backlogs
image credits: http://commons.wikimedia.org/wiki/File:Diagram_spaghetti_kilka_produktow.PNG
http://www.michaelnygard.com/blog/2008/02/outrunning_your_headlights.html
22. Dev
produces a
nightly build
Twice weekly,
2 hour deploy
to Test Lab
1.5 Days to test
each drop
If we improve test speed, our constraint moves.
23. Examining a constraint
• Manual process
• Limited staff
• Production releases have priority
Why can we only deploy twice a
week to QA?
24. Tackling the constraint
• Manual process
• Limited staff
• Production releases have priority
Why can we only deploy twice a
week to QA?
• Automate processes
• Hire more staff
• Prioritize QA Releases
Options
25. Imagine a 1 day test cycle
Dev
produces a
nightly build
2 hour
deploy to
Test Lab
1.0 Days to
test each
drop
¼ day deploy downtime becomes turns 1 day test cycle into
two days.
26. Tackling the constraint
• Manual process
• Limited staff
• Production releases have priority
Why can we only deploy twice a
week to QA?
• Automate processes
• Hire more staff
• Prioritize QA deploys
Options
Short term approach
Long term approach
27. Measuring utilization helps with this prediction
“Feeling” the pain isn’t enough to predict the next constraint
There may be no pain at the next constraint today
28. When something is free, it is used more
Example: Amazon Prime. For $79/yr, customers get free 2
day shipping on everything.
“…Customers spent as much as 150% more at
Amazon after they became Prime members.
Subscribers not only ordered more often … they
started buying things at Amazon that they
probably wouldn’t have in the past” *
* http://business.time.com/2013/03/18/amazon-prime-bigger-more-powerful-more-profitable-than-
anyone-imagined/
29. Consider implications of making something free
If builds, deploy, regression tests are free…
We’re going to be
testing lots more.
Better buy some
hardware.
31. After build, deploy is next
Build guy & deploy guy
used to be in sync
A CI server can do
hundreds of builds per day
Agile tends to make
Operations a constraint
Knowing my stuff
compiles at all times is
great. I want to know if
it passes functional
tests too.
32. As tests shift left, expensive tests constrain
Developers run more integration tests
Some tests use expensive resources
–pay per use web services, mainframes, production systems…
Stubbing those resources becomes important
– known as “service virtualization”
33. Concurrent agile dev requires more test labs
More code is compiling, and set to be released “soon”
Need more environments to test changes in
– Pressure for Platform as a Service
34. Frequent releases demand fast feedback
Frequent releases enable experimentation
Monitoring of business outcomes required
Architectural pressures to support A/B testing
♫
Stop ignoring difference
between features delivered
and value delivered
35. If we keep chasing constraints,
where does it end?
36. You may end up with Continuous Deployment
Build
Run thousands of tests
Deploy to some servers
Monitor
Deploy to remaining
servers
37. Summary
Optimizations other than at the constraint don’t help
“Breaking” one constraint will expose the next
Patterns or analysis can be used to predict next constraint
Continual Improvement, Continual Improvement, Continual
Improvement, Continual Improvement…
38. IBM will collaborate with you to
understand your current situation,
goals and constraints. The
Assessment and Planning
Workshop will aim to capture
sufficient information to make
specific recommendations for
improvement and implementation.
Intended Audience:
Key leadership from practice areas and
stakeholder organizations
Value Proposition
Clear recommendations for capability
improvements aligned to your business goals
Initial Architecture
Adoption roadmap based on proven best
practices
Activities
Workshop planning
Assessment and Planning Workshop
Collaborative discussion on current status, future
goals and adoption requirements
Produce Deliverable
Deliverables
Current Status and Improvement
Recommendations
Architecture
Adoption Roadmap
Assessment and Planning Workshop
39. More resources
Urbancode.com/resources
Continuous Delivery Maturity Model
Deployment Automation Basics
Applying Lean Principals to Software Delivery
Blogs.urbancode.com
Twitter.com/UrbanCode
Facebook.com/IBMUrbanCodeProducts
SlideShare.net/Urbancode