- The document discusses measuring and improving the return on investment (ROI) of an agile organization. - It notes that few organizations accurately measure ROI by evaluating projects after completion, yet many report their evaluations are somewhat accurate. - The key to increasing ROI is decreasing the cycle time for delivering valuable working features to customers by focusing on competent product ownership, effective backlog grooming, trained and empowered teams, mature engineering practices, and robust delivery capabilities. Frequent releases can also increase revenue and ROI.

1. Gauging the ROI of
Your Agile Organization
Lee Cunningham
Director, Enterprise Agile Enablement
VersionOne
June 2014
UnderstandingImproving

2. © 2014 VersionOne 2
About me…
• Lee Cunningham
• Lean-agile practitioner, coach,
advisor
• Some traditional hats along the way:
Software Engineer, Development
Manager, QA Manager, Release
Manager, Project/Program Manager
• USAF
• SPC, PMP, PMI-ACP, CSM
• lee.cunningham@versionone.com

3. © 2014 VersionOne 3
14% ROI standards “well defined”
ROI? Compared to what?
33% ROI standards “not defined” or
“poorly-defined”
Standish Group, DARTS 2007

4. © 2014 VersionOne 4
Is ROI even being measured?
64% “rarely” or “never” perform
post-project value evaluation
Standish Group, DARTS 01-2013

5. © 2014 VersionOne 5
…yet
72% report that their project
evaluations are
“somewhat accurate”
Standish Group, DARTS 01-2013

6. © 2014 VersionOne 6
So, what they have told us is…
I HARDLY EVER
EVALUATE MY
PROJECTS --
BUT WHEN I DO,
MY EVALUATIONS
ARE SOMEWHAT
ACCURATE.
STAY CLUELESS, MY
FRIENDS…

7. © 2014 VersionOne 7
A word about statistics
http://tylervigen.com/

8. © 2014 VersionOne 8
The Product Owner?

9. © 2014 VersionOne 9
Takeaways
• Agile is about business -- not software.
• We need to be able to talk about agile
practices in terms that senior organizational
leadership would appreciate.

10. © 2014 VersionOne 10
“Agile Organization”?
For our discussion, we’re defining “Agile Organization” as a system
formed around an agile value stream, whether a portfolio, program,
or a team.

11. © 2014 VersionOne 11
Cycle Time & Lead Time

12. © 2014 VersionOne 12
ROI =
Revenue – [Operating Expense + Total Variable Expenses]
Investment
How much we are paid for itHow much we are paid for it – How much it cost to make itHow much we are paid for it – How much it cost to make it
What it cost for the raw materials used to make it

13. © 2014 VersionOne 13
Why projecting ROI is hard to do
Revenue – [Operating Expense + Total Variable Expenses]
Investment

14. © 2014 VersionOne 14
Challenges to calculating Revenue
• New Revenue: new revenue from new customers
• Incremental Revenue: new revenue from existing
customers
• Retained Revenue: revenue from existing customers
we would have lost had we not undertaken this
• Operational Efficiencies: reduced costs resulting from
having undertaken this

15. © 2014 VersionOne 15
Basic “Concept to Cash” Value Stream
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features

16. © 2014 VersionOne 16
The essence of throughput economics
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features
-$ -$ -$ -$ +$
Until we’ve gotten paid, all we’ve done is spend money.
Frequency
and Value

17. © 2014 VersionOne 17
And we can’t forget….
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features
-$ -$ -$ -$ +$
rework
-$
Defects
-$

18. © 2014 VersionOne 18
Mapping to ROI components
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features
-$ -$ -$ -$ +$
rework
-$
Defects
-$
Investment RevenueOperating and Variable Expenses

19. © 2014 VersionOne 19
If we want to increase ROI…
Revenue – [Operating Expense + Total Variable Expenses]
Investment
Increase the Numerator
…and/or
Decrease the Denominator

20. © 2014 VersionOne 20
If we want to increase ROI…
Revenue – [Operating Expense + Total Variable Expenses]
Investment
Focus here
…and/or
Decrease the Denominator
Expenses and Investment can only be reduced so far until Revenue is no longer possible.
Increase the Numerator
Revenue, at least hypothetically, has no ceiling.

21. © 2014 VersionOne 21
“Revenue? But
we don’t sell our
software – it’s for
an internal
customer…”

22. © 2014 VersionOne 22
Factors affecting ROI components
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features
-$ -$ -$ -$ +$
rework
-$
Defects
-$
Investment RevenueOperating and Variable Expenses
• Effort expended in ideation
• Effort expended in
developing Epics, Stories
• Quality of Epics, Stories
• Effort expended in analysis & design
• Quality of Epics, Stories
• Degree of team empowerment
• Maturity of engineering practices & discipline
• Delivery/deployment capability
• Volume of escaped defects
• Quality of support process*
• Release frequency
• Customer-perceived
value
• Quality of Epics,
Stories
• Volume of escaped
defects

23. © 2014 VersionOne 23
What to
measure and
where

24. © 2014 VersionOne 24
What to Measure What that indicates
“Concept to team-ready” cycle time • How much the development, grooming, and
maintenance of features and stories is costing us
• How long we’re allowing the value of the
features and stories to decay
Story/Feature acceptance rate The effectiveness of the backlog generation and
grooming process
What to measure: Investment

25. © 2014 VersionOne 25
What to Measure What that indicates
“Team-ready-to-Deployed” cycle time • How much the process of taking a team-ready
story or feature to the customer is costing us
• How long we’re allowing the value of the
features and stories to decay
Escaped Defect rate • How much it is directly costing us to fix the
defects
• The opportunity cost of fixing those defects
What to measure: Expenses

26. © 2014 VersionOne 26
What to Measure What that indicates
“Concept to cash” cycle time (total lead
time)
How much the complete ideation-to-payment
value stream is costing us
Cycle-time by story/feature value How long the more valuable stories & features are
being allowed to decay in value
Release frequency • How long revenue-generating opportunities are
being deferred
• How long direct customer feedback is being
deferred
Escaped defect rate Risk exposure to retained, incremental, and new
revenue
What to measure: Revenue

27. © 2014 VersionOne 27
Cycle time shows up a lot as something to measure. Why?
Key: Cycle Time
• Revenue
• Inventory (epics, stories in the backlog)
• Feedback
The value of all of these decays with time:

28. © 2014 VersionOne 28
The best single indicator of system ROI is the trend in lead time
associated with that value stream…
…because all of the lower-level things we measure (e.g., defect
rate, internal cycle times) directly affect lead time.
Key: Cycle Time

29. © 2014 VersionOne 29
No matter what we might wish, the capability of the system today is what it is.
Key: Measure trends, not snapshots in time
Instead of just asking “Why is it taking us x days to deliver a feature?”
ask
“Are we delivering things that are truly valuable to our customers faster now
than we were 3 months ago?”

30. © 2014 VersionOne 30
So how can we improve ROI?
Idea
Design &
Code
Test
Deliver/
Deploy
Valuable,
Working,
Delivered
Features
Decrease
Investment
Increase
Revenue
Decrease
Operating and Variable Expenses
• Competent and
committed Product
Ownership
• Effective and efficient
backlog development
and grooming skills
• Trained and Empowered teams
• Mature and disciplined agile
engineering practices
• Robust delivery/deployment capability
• Solid definition of “Done”
• Robust support process*
• Value focus
• Frequent Releases
to customer
Decrease Lead Time!

31. © 2014 VersionOne 31
Suggested reading for thought leadership aggregation

32. © 2014 VersionOne 32
Q&A

33. © 2014 VersionOne 33
On Behalf of Everyone at VersionOne…
Thanks!