This was presented by Roger Brown and Peter Green at the Seattle Scrum Gathering on 5/17/11. Slides have been annotated with some discussion notes to provide additional context.
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
Lean Strategies for IT Support Organizations
1. LEAN STRATEGIES FOR
IT SUPPORT ORGANIZATIONS
Scrum Gathering 2011
Seattle
Roger Brown
CSC, CST Moonrise Consulting, San Jose,
CA
Peter Green
Agile Coach and Trainer, Adobe Systems,
Inc.
With assistance from
Jonathan Snyder, Adobe Systems,
Inc.
3. LEAN PRINCIPLES
3
Minimize the time from order to cash
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
The five-step thought process for
guiding the implementation of lean
techniques is easy to remember,
but not always easy to achieve
- lean.org
4. IDENTIFY VALUE
Specify value from the
standpoint of the end customer
by product family.
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
5. SOURCES OF VALUE FOR ENTERPRISE SYSTEMS
$ Useful functionality
$ High system reliability
$ Quick system
response
$ High quality
$ Ease of use
$ Good support
6. MAP THE VALUE STREAM
Identify all the steps in the value
stream for each product family,
eliminating whenever possible
those steps that do not create
value.
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
8. EXPANDING THE VALUE STREAM
Where does the
Product Vision
come from?
Scrum
Where does the
Product go after
delivery?
Product
Discovery
Product
Definition
Product
Development
Product
Delivery
Product
Operation
Innovation Games
Pragmatic Marketing
Customer
Development
DevOps
Who is missing?
Leading edge Agile approaches
Mainstream
10. COMPLETING THE VALUE STREAM
Product
Discovery
Product
Definition
Product
Development
Product
Delivery
Product
Operation
Support
Support is the
interface to the
customer
Now we can start thinking
about optimizing the entire
value stream
Bleeding edge for Agile Enterprises
What Lean/Agile
opportunities an
we find?
11. Product
WHAT IS SUPPORT?
What is a Service?
Activities, not tangibles
Produced and consumed at the same
time
Customer is a co-producer
Utility + Warranty
Service
12. WHAT LEAN PRACTICES HAS YOUR ORG TRIED?
Lean Production Practices Often Applied to Services:
• Reduce average activity time (stop watches!)
• Heavy specialization (silos!)
• Resource Management (offshoring!)
• Stepwise forwarding (your incident record has 10 entries
• Standardization (support scripts!)
Focus is on activity and cost.
Customers are frustrated.
Workers are de-motivated.
13. THE NEW PERSPECTIVE
Treat Service as a system
and focus on capacity and capability
to achieve flow.
Economies of Scale
Economies of Flow
14. FINDING FLOW
Make the value-creating steps occur
in tight sequence so the product
will flow smoothly toward the
customer.
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
20. ABOUT VARIABILITY
In general, it is better to reduce the economic consequences of
variability than to try to reduce variability.
- Reinertsen
Manufacturing Development Support
Unit
Unit
Unit
Unit
Unit
Unit
Story
Story
Story
Story
Story
Story
Ticket
Ticket
Ticket
Ticket
Ticket
21. ESTABLISH PULL
As flow is introduced, let customers
pull value from the next upstream
activity.
Note: customer is the next
downstream process, not just end
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
22. PULL
22
Push systems overwhelm capacity,
creating turbulence, waste and delay
Pull systems have a steady flow that
provides predictability
Push
♫
23. Normal Urgent Process Improvement
WI Types:
Design
WIP=2
Test
WIP = 3 Done
Develop
WIP=4
(Prioritized
Backlog)
Doing DoneDoing Done
Bottleneck
Station
Workflow
WIP
Limi
t
WIP
Limi
t
WIP
Limi
t
SIMPLE SOFTWARE KANBAN BOARD
To Do
23
KANBAN
26. SEEK PERFECTION
As value is specified, value streams
are identified, wasted steps are
removed, and flow and pull are
introduced, begin the process again
and continue it until a state of
perfection is reached in which
2. Map
the
Value
Stream
3.
Create
Flow
4.
Establish
Pull
5. Seek
Perfectio
n
1.
Identify
Value
27. ABOUT PERFECTION
Perfection is never actually
achieved. The notion of perfection
is itself subject to a process of
continuous improvement.
- Jonathan Snyder
When does our process
reach perfection?
28. REDUCING WASTE
Manufacturing Enterprise System Support
Inventory Stale support requests, planned process improvements,
unreleased fixes
Extra processing Heavy process steps, meetings, work assignments,
manual reporting
Overproduction Standardization of responses, speculative process
changes
Transportation Task switching, issue triage, offshoring, issue
forwarding
Waiting Specialist bottlenecks, batch fixes for a hot patch,
reproducing environments and configurations, queue
escalations
Motion Emergency fixes, handoffs due to specialization, log in
to multiple systems to test or research
Defects Lost knowledge, mis-applied fixes, out-of-date scripts,
Addressing systems instead of root causes, bugs
The Seven Deadly Wastes
31. EXISTING FEEDBACK LOOPS TO IMPROVE
Product
Discovery
Product
Definition
Product
Development
Product
Delivery
Product
Operation
Support
Help
Desk
Reliability
Configuration
Performance
Compliance
Bugs
Release
Frequency
32. NEW FEEDBACK LOOPS TO ADD
Product
Discovery
Product
Definition
Product
Development
Product
Delivery
Product
Operation
Support
Learning
Support viewpoint, tools
Low value features
Inefficient features
Supportability features
Feature ideas from customers
Usability issues
Wrong features
Missing features
Customer desires
Emerging problems
Help
Desk
34. AGILE ENTERPRISE MANIFESTO
We are uncovering better ways of developing enterprise business
services by doing it and helping others do it. Through this work we
have come to value:
Incentives for quality and value over time and cost
Agile organization over agile project methodology
Knowledge management over tribal memory
Economies of flow over economies of scale
That is, while there is value in the items on the right, we value the
items on the left more.
- A work in progress by Jonathan Snyder,
Sr. Manager, IT Application Support,
Adobe Systems, Inc.
35. REFERENCES
Anderson, D. J. (2010). Kanban:
Successful Evolutionary Change for
Your Technology Business. Sequim,
WA: Blue Hole Press.
Beck, K., & al., e. (2001). Manifesto for
Agile Software Development.
Retrieved from agilemanifesto.org:
http://agilemanifesto.org/
Bell, S. C., & Orzen, M. A. (2011). Lean
IT: Enabling and Sustaining Your
Lean Transformation. New York:
Productivity Press.
Grönroos, C. (2007). Service
Management and Marketing:
Customer Management in Service
Competition, 3rd Edition. Hoboken:
J. Wiley.
Humble, J., & Farley, D. (2010).
Continuous Delivery: Reliable
software releases through build, test,
and deployment automation. Boston: 35
Reinertsen, Donald G. (2009). The
Principles of Product Development
Flow: Second Generation Lean
Product Development. Redondo
Beach, CA: Celeritas Publishing.
Seddon, J., & O’Donovan, B. (2009).
Rethinking Lean Service.
http://www.systemsthinking.co.uk/6-
brendan-jul09.asp
Womack, J. P., & Jones, D. T. (1993).
Lean Thinking. New York: Free
Press.
Womack, J. P., Jones, D. T., & Roos, D.
(1990). The Machine that Changed
the World. New York: Macmillian
Publishing Company.
Editor's Notes
Take audience pulse on
How much do you know about Lean?
Are you interested in IT, Ops or Support?
Elements of Value to an Enterprise system customer:
solve business problems
deliver products and service to their customers
reduce costs of work
reduce effort required to perform their jobs
Ask: What are some sources of value that you can think og?
$Useful functionality
High system reliability
Quick system response
High quality
Ease of use
Good support
DevOps is one name for the growing field of Lean/Agile inspired
operations practices. It seeks to break down the wall between
Development and Operations so that new product does not pile
up unused and the challenges of change risk and compliance can
still be addressed. It leverages automation, virtualization and Agile
Practices for better communication and continuity between Dev and
Ops.
Practices include:
Automated provisioning
Infrastructure as code
Continuous deployment
Model Driven Automation
Deployment rollback
Incremental database development
Consider thinking of DevOps as a lean approach to Software Developing that solves problems in software development analogous to the Supply Chain Distribution in traditional product development. That is, once you have a product, how do you get it to market? The techniques for producing a product during design and development are usually very different than the techniques used to get the finished product to market. Mostly, the techniques in getting a product to market involve how to efficiently mass produce and deliver the product.
In software as a service (including web applications), we have underestimated the complexity of our own supply chain distribution mechanisms. Underestimating this complexity creates waste. DevOps solves much of the complexity by integrating operations practices into the development process itself. Solving problems with automated testing and deployment throughout the software development and delivery process reduces bad variability between software server landscapes and environments, increases consistency and predictability, largely through automation. This way, by the time a solution is ready to go live, the mechanisms to deploy to production are already themselves well tested and proven on the non-production landscapes.
Ask: What are some things that support organizations do?
Ask: What are some challenges they have?
Ask: What are some opportunities for improvement?
Activities:
Help Desk
Failure Analysis
Code updates
System Monitoring
System Configuration
Bug fixing
Incident Tracking
Challenges:
Users expect rapid response to problems
More people using more technology means more demand for help
More products and versions to support
Quarterly $ goals drive tight timelines
Fragile, debt-ridden systems
Management by time and budget, not value and quality
Knowledge gained during emergencies is not retained
Staff works in expertise silos
Opportunities:
Responsive support pleases customers leading to more sales
More “supportable” products have lower support costs
Higher quality products have lower support costs
More efficient and reduced demand saves people cost
Fewer production disruptions escalated to development team
Demand flows in unpredictably.
How can we process it in a smooth flow?
Reduce waste
Reduce batch size
Process single unit
Decentralize control
Limit work in progress
Be smart about variability
What are some examples?
Competitor features
New technologies
New ideas for products and features
Customer requests for new functionality
Payback of technical debt
Cup holder story
What are some examples:
Help requests
Code defects
Usability problems
Building the wrong features
Insufficient security, speed, uptime
Technical debt to hurry shipment
Another tool you can use to visualize your queues is the Cumulative Flow Diagram… it’s essentially a time lapse Kanban board.
WIP = Work in Process
LT = Lead Time: average time for an item to go through the development process
*click*
What you see in the 1 – 9 week range is a lot of work getting developed and tested, and some getting approved, but notice that the slopes of the transition lines are very different.
*click*
That’s a warning sign that work in process is building up, and you’re getting into a long queue state
*click*
Then it looks like some WIP limits got hit at week 9, so the rate of things being pulled off the backlog went way down, and everyone worked through their existing queues… note how the work in process got cut by 1/3rd
*click*
At about week 27 or so, all the slopes are about equal, and the amount of time for an item to make it through the development process is vastly reduced.
This is really the goal for your process: items flowing through the system at a consistently high rate, with no build up of queues or work in process.
Hire the right people
Respect what they know and how they work
Enable continual learning
Give individuals autonomy to make decisions
Use cross-functional teams where re-work occurs
Align decentralized authority with centralized strategy
Trust that uncertainty will be met more quickly by knowledgeable, capable people
Use explicit policies (team-defined and org-defined) to aid trust in self-organization of teams
In Lean manufacturing, we work hard to eliminate it.
In product development we encourage it to spawn innovation.
In services, it just is. So we try to make the most of it.
Look for patterns to leverage in prioritization and problem solving
Know the payoff function and the probability of success
Cut your losses
Pull improves work life.
Predictability
Cadence
Flow
Quality
Learning
Pride
Here we have a simple Kanban board. The team has visualized their current work process, where they do some design, some coding, and then some testing. Don’t worry, the scrum coaches are teaching them about TDD, we’ll get to that later in the project! . We can also use the Kanban board to visualize work types such as Urgent work items and Process Improvement items.
We have established WIP limits for each of the phases based on the team’s estimate of what skill sets are present on the team. The team has a queue of work to do, which is equivalent to a product backlog.
*click*
As the team starts working, they pull items from the product backlog onto the board and start doing the design work. Since they’ve established a WIP limit of 2 for design, they’ll hit this pretty quickly. No other work can proceed until the design on these items is done. The team can’t pull in more design items without violating WIP limits.
NOTE: In scrum, the iteration is a primary constraint. In Kanban, the WIP limit is the primary constraint. If your scrum team has a hard time adhering to the iteration constraints, it’s likely they’ll struggle adhering to WIP limits.
*click*
Ok, so the team has gotten the design work done and so they can pull items into the next stage, the develop stage. As we watch items move across the board, eventually we hit another WIP limit in the develop stage, and then quickly hit the WIP limit in Design as well. Again we can’t move any items into these stages to work on them until Develop finishes something. Even if Design finishes one of their items, they can’t push it into the Develop stage because the WIP is full there, and because it’s not a push system. So let’s assume we get one of the items Developed, and test pulls it into their queue. Now the system can begin moving again.
*click*
Items move through the system until we hit another WIP limit, this time in Test. We are below WIP limits in Develop, and so once these items are done in Design, we can pull them in and start working again.
*click*
We’re getting close to full capacity for each of our stations,
*click*
Everybody’s pretty busy, and then
*click*
Everyone’s queue is full! Traditional managers are ecstatic! 100% utilization! . This also means that we can’t start working on the next item until something gets completed from test and moved into Done. This may be a time of temptation for the team, but they stay strong and help reduce the testing load together and move things into Done.
*click*
This creates a pull and items quickly move back into the board, and we get to full again. With this simple Kanban, we don’t have a lot of visibility into what the real bottleneck is, and so we are going to add some lanes for actual WIP in a stage and Done in a stage (sometimes referred to as a buffer).
*click*
We can do the same with Design
*click*
Now when a little more progress is made in the Develop stage, we see very clearly where the bottleneck is:
*click*
Test. Now the team can use our agile & lean toolsets to start to tackle the bottleneck.
Lean cadence supports variability in delivery cadence. Development problems are large and need to be decomposed. Lean supports problems are already small but have different expectations of resolution (SLA). This is not to say that Lean cannot be adapted to development.
Goals for an Agile Organization
Optimal value delivered to customer
Consistent processes
Measurable processes
Collect usable knowledge
Focus
Trust
Continuous improvement
Inventory: completed bug fixes piling up due to infrequent releases; pushing support staff to 100% utilization leaves no slack for burst capacity or process/system improvement
Waiting: support staff that did not get knowledge of new changes can’t help customers efficiently; offshoring goes here too
Not sure where these go:
* over-specialization in support leads to multiple handoffs within support to get issues resolved. Risk of infinite loops (e.g. ecommerce support -> middleware support -> CRM support -> ecommerce support)
* too many issues in the system all at once leads to breakdowns in environment management. Environments get locked down, but the amount of work expected to flow through the system isn’t limited. Therefore, teams rush to get changes done in limited release windows, reducing quality and increasing production issues.
Allows smaller queues
Learning is faster and more efficient
Reduces noise, revealing weaker signals
Improves communications
Enhances decentralized control
Reduces sense of urgency
Promotes alignment