My keynote at Agile Canvass conference in Chennai today. In this talk, I also talk about my "Crawl-Walk-Run-Fly" metrics framework

1. Agile
Metrics
Tathagat Varma
Sr Director,
Strategy & Ops,
WalmartLabs

2. Disclaimer
These are
my personal
views!
Pic: https://motaen.com/wallpapers/source/id/51908

3. In today’s
world, mere
opinions are
not enough
anymore!

4. Why
measure?

5. Metrics help
improve
visibility, and
plan more
realistically

6. Unfortunately,
most software
metrics are
often
“watermelon
metrics”

7. “How does
a project get
to be a
year late?
…One day at
a time.”
Fred Brooks,
The Mythical
Man Month,
1975

8. “When one hears of disastrous schedule slippage in a project, he
imagines that a series of major calamities must have befallen it.
Usually, however, the disaster is due to termites, not tornadoes; and
the schedule has slipped imperceptibly but inexorably. Indeed, major
calamities are easier to handle; one responds with major force,
radical reorganization, the invention of new approaches. The whole
team rises to the occasion. But the day-by-day slippage is harder to
recognize, harder to prevent, harder to make up. Yesterday a key
man was sick, and a meeting couldn't be held. Today the machines
are all down, because lightning struck the building's power
transformer. Tomorrow the disk routines won't start testing, because
the first disk is a week late from the factory. Snow, jury duty, family
problems, emergency meetings with customers, executive audits—
the list goes on and on. Each one only postpones some activity by a
half-day or a day. And the schedule slips, one day at a time.”

9. Most common
approach is
over-
measurement
which leads to
no meaningful
decision-
making

10. So how to
design,
capture and
leverage
metrics
effectively?

11. Models are…
well, just
models!

12. Metrics are
not panacea!

13. People will
invariably
(always?)
interpret
metrics
to suit
themselves!

14. Metrics
can be
(and most
probably
will be!)
gamed!!!

15. Good metrics
(often look
bad but)
drive
improvement

16. Simplicity
is the
key!

17. Metrics
without
business
alignment
=
ignorance
is bliss!

18. A great
metrics
dashboard
works just
like an
Air Traffic
Controller
Pic: https://www.telegraph.co.uk/travel/travel-truths/Are-airlines-exaggerating-flight-times-so-theyre-never-late/

19. Correlation
doesn’t
imply
causation!

20. Horses for
courses
it is!

21. Metrics
need to
show the
trajectory
to be
useful

22. Thinking in
systems
might be a
good idea!
• Measure adoption efforts
• Examples: #trainings
•

23. Unit of
measurement
is almost
as important
as the
purpose for
which
it is
used!

24. Know
your
audience!

25. Identify the
“vital few”
instead of the
“trivial many”
Pic: https://www.financialexpress.com/industry/sme/what-startups-can-learn-from-paretos-8020-rule/1122669/

26. Bad news is
sometimes
(often?)
good news!

27. What is your
“North Star
Metric”?

28. Great metrics
help identify
the root cause
and lead us
towards the
right solution!

29. Know the
difference
and use it
accordingly:
Accuracy
vs
Precision

30. Right-size
your
metrics

31. Without a
closed-loop
feedback
system, even
the best of
the metrics
are rather
useless!
Pic: https://instrumentationtools.com/open-loop-and-closed-loop-animation/

32. “Crawl
Walk
Run
Fly”
Metrics
Framework

33. Crawl
• Measure inputs or efforts
• Encourage participation at the starting
point without pre-conditions or judgment
• Examples include:
• # trainings done
• # teams adopting agile
• # scrummasters identified
• etc.

34. Walk
• Measure internal efficiencies achieved
• Encourage behaviors without looking at
results
• Examples include
• (yeah, the favorite) velocity
• % test automation
• % code coverage
• etc.

35. Run
• Measure effectiveness delivered
• Look at the business results!
• Examples:
• Revenue $$$
• Proxy to revenue - engagement ?
• Dave McClure’s AARRR Pirate Metrics
• etc.

36. Fly
• Measure impact made or the value delivered
to the customers or the society at large
• Serve the higher purpose beyond business
goals
• Examples:
• People lifted out of poverty since 80s
• Net impact on planet earth!
• Improve quality of life
• etc.

37. Recap

38. One last
thing…
The most
important
metric is…
Pic: https://www.thehindu.com/opinion/open-page/india-is-happy-and-the-rankings-just-dont-match/article18190887.ece