Companion slides to my Southern Fried Agile Talk on Wed, Nov 14th, 2018 on forecasting release dates with lean agile methodologies

1. The Agile Forecast
Fun with Probabilities
Joe Tristano
© 2018 Premier Inc. All rights reserved.
Premier and the Premier Mobius logo are
registered trademarks of Premier Healthcare
Alliance, L.P. All other trademarks are the
property of their respective owners.

2. Advertisement
https://stnektariostakingsteps5k.racesonline.com/
Register for St. Nektarios Taking Steps 5K on Saturday, Dec 8th 2018 @ 8:00 AM
Proceeds benefit Shelter Health Services, the region’s largest free medical clinic caring
for the 425+/night homeless women and children who stay at the Salvation Army
Center of Hope

3. The Agile Forecast: Outline
● Introduction
● Background
● Inspiration
● Innovation
● Nuts & Bolts
● Making it work
● Wrap up

4. Introduction
®

5. About Joe
● Passion for making high quality software on
time that makes users AWESOME at what they
do
● Mechanical Engineering background
● Software Development for 25 years
● Agile for 17
○ Learned XP in 2001 from Joshua Kerievsky
○ CSM from Net Objectives in 2010
○ Brought scrumban to ANSYS shortly thereafter
● Inspiration
○ Poppendieck
○ Modern Agile
○ etc
®

6. Background
● New job!
● Product “promises” & a fixed date
● How are we going to make the date?
○ Cut scope!
○ Still overloaded
○ Make Small Stories!
○ We need to leverage our historical data!

7. When Will This Be DONE?

8. Time to Delivery
● Set clear milestones and objectives
● Release often
● Measure cycle time to ensure people are moving their work across the board
● Increase throughput

9. Radical Departure from Velocity

10. Inspiration

11. Statistical Methods
“a slight study of statistical methods and
their application to social phenomena affords
one of the simplest examples of the
application of algebraic ideas.”
- Alfred North Whitehead, The Aims of Education

12. What about Little’s Law?
● Average Throughput
● Average Lead Time
● Average Work in Progress
Works great for:
● Assembly Line Production ....
● Little Variance ...

13. JIRA Control Chart
● Cycle Times Do NOT have a normal distribution
● Nor does WIP
● We are not stamping widgets!
YIKES!!!!

14. We Write Code

15. Where to Now?

16. Research Findings
● Others confirm that software cycle times do not have a normal distribution
● They also found that a naive use of Little’s Law did not fare well in practice
● Troy Magennis and Alexei Zheglov arrived at similar conclusions
○ Use historical data
○ Fit a Probability Distribution Function to your data
○ Both used a Weibull distribution

17. What’s a Weibull distribution
● Weibull distribution is a continuous probability
density function which uses a constant shape
parameters to alter the shape of the distribution
● It is commonly used in reliability failure analysis
of mechanical parts to forecast failure
SO WHAT?

18. Weibull & Software
● Magennis found that as a team becomes more
lean the β becomes smaller
● Smaller stories, smaller cycle times
● The magic is finding β for your team
● Lots of curve fitting
● Still dissatisfied with the hoops to jump through
to estimate β and make an automated process

19. Innovation

20. Data Science {Statistics} to the Rescue!

21. Data Science and the KDE
In need of a better fit, data scientists Mike Korvink and Katie Pratt suggested the use of
a KDE (Kernel Density Estimator) to estimate my distribution in a non parametric way
building it from my historical data
EUREKA!!!!
Perfect Fit

22. Hypothesis
Given KDE Probability Distributions of Cycle Time and WIP along with a release date,
I should be able to determine our chances of meeting said date
BUT HOW?

23. Monte Carlo Analysis
● Sample PDF over and over again and determine probability to meet date
● Additional uses of the data also reveal themselves

24. Nuts & Bolts

25. Process
WIP KDE
CYCLE
TIME KDE
Stories Remaining
WIP

26. Pseudocode
Num Buckets = Num
Stories/WIP
Pull from JIRA
Calcs:
Cycle Time/ Story
WIP/Day
Compute KDE for
Cycle Time and WIP
based on lookback
period
Sample WIP KDE
For WIP in Sample
Sample cycle times
For each bucket
For each sample
Total Days += sample
For each sample
End Date = today + Total Days
If End Date <= Release Date
Match++
Probability =
Match / Total Samples
Build CDF of sampled
probabilities

27. Complete Automation
● Using this method with various lookback periods, I can have a completely
automated process that runs everyday without fail.
● Everyone knows where we are, everyday

28. Failsafe
● Use a Fibonacci sequence to
look-back in weeks for samples
○ [1,2,3,5,8,13,21]
● X-Axis: How far do our
samples look back in time
● Y-Axis: What is our probability
● Bubble Size: Number of
Samples

29. Samples and More Samples
● 7 Lookback Samples [1,2,3,5,8,13,21]
● 1000 WIP Samples
● 1000 Cycle Time Samples
● Up to 7 Million Samples
○ Samples are excluded if outside real range of parameters
since KDE’s are not bound to discrete values

30. Mean Probability
● Linear combination of all
probabilities divided by their
sample size

31. Results
Daily result of what our
probability is and where
it has been...

32. Inverse Relationship

33. Bonus!
● When pinpointing our next
release date, we can leverage
the CDF to guide us on picking
a date
● The X- Axis is number of days
out from today
● The Y-Axis is the probability
that we will finish our work by
that date

34. Heuristics
lookback_weeks:<How many wks back>
longest_cycle_allowed:<outliers>
Stories_remaining:<jql or value>
monte_carlo_samples:<# samples>
wip_exclude:<exclude dates>
in/exclude: <exclude people>
blocked_status: <blocked>
start_status: <start timer>
closed_status: <end timer>
status_for_cycle_time:
<working kanban column
holidays: <holidays>
zero_cycle_time:
<time to allot
for stories closed
without being opened>

35. Multiple Development Streams in a Single Team
● Using inclusion and exclusion of people and stories bucketed to those people,
multiple development streams can be tracked and monitored
Production Timeline
Feature 2
Feature 1

36. Making it Work

37. The Scale of Probability
Unlikely Likely
0% 20% 50% 80% 100%
impossible certain
very likely but not certainvery unlikely but not impossible

38. ● Teams must get into a habit of making very small stories
● Heuristics account for larger stories
● Daily probability tells us when to step on the gas
● Since starting this method in July 2016, the method has predicted release dates
with 95%
○ One of 20+ large releases slipped by 1 week due to Murphy’s law
How does it work in practice

39. When to Worry

40. Philosophy
● It’s a marathon from release to release
● The sprint is there as a mental time box
along with level setting
● Assess release scope change as it impacts
probability.
○ If you can absorb it, do it
26.2

41. ● Estimates become complexity & doubt estimates,
time is taken into account from results
● No longer compare teams on story points
● Keep your eye on adding value
● We have experimented with no estimation
Stop Point Madness
“I’m not sure if I was the inventor of story
points, but if I am, I’m sorry”
-Ron Jeffries
Complexity
Doubt
Effort

42. What about Points?
● Use Points for High Level, relative, estimation
● Use historical data to guide you along the way
● How does “new work” compare to “old work”?
● Get a feel for what fits
● Use this method to determine date
● Don’t use points for day to day
○ Track throughput
Completed
Stories
Completed
Points

43. What about new teams?
● Start with Weibull, fix your distribution
● After accumulating data start this method
● You can start with less than 100 closed stories but 100 closed stories will get you
into the 90% accuracy range

44. Key Behaviors
● Regardless of story size, team must move stories across the board accurately
● Trust your team, if the numbers don’t sit well talk to the team and investigate

45. Enhancements
● Dependent stories and their placement into the WIP buckets
● Better point heuristics
● Performance improvements
○ Backed by a database to cook less WIP and cycle time numbers

46. Wrap Up

47. Wrap Up
● Ability to leverage wealth of information in JIRA to predict future release dates
and probability of meeting that release date using KDE and Monte Carlo analysis
● Team knows where they are everyday
● Stakeholders are aware of teams confidence in meeting release
● Trust is built between stakeholders and team for consistently meeting
commitments with high quality on time, every time

48. Thank You!
Questions?