Why you need technical debt and ideas on how to measure it.

1. Dr StrangeTech or:
How I learned to Stop Bitching and Embrace Technical Debt

2. Who am I?
Old man yells at the Cloud (engineers)
@behemphi
● Former DBA (MySQL, Postgres, Oracle)
● Former Developer (PHP, Python, Java)
● Former Evangelist (Stackengine Jesus!)
● Former Director of Operations, Infra, Devops,
Cloud Engineering
● Current Engineering Manager
● Sometime community contributor
Tackle is the onramp to cloud marketplaces. Those
are like app stores for the cloud. If you aren’t selling
there you should ask me about it. I was amazed by
the market opportunities!

3. Outline
So you know WTF is going on …
● Tech debt is a metaphor,
○ Let’s reason about it like a CFO
● Flow is a metaphor
○ Let’s reason about it as the
currency of a dev team or
engineering organization
● Use Debt to fund better flow
● Pay down debt responsibly
● Know the terms of your debt

4. Debt is a Tool
A tool is neither good nor bad until it is put to use
Stand up if -
● You have a mortgage on a home
● Sit down if you don’t know the interest rate
and length of the note.
Stand up if -
● You have a loan on a car
● Sit down if you don’t know the interest rate
and length of the note.
Stand up if -
● You have one or more credit cards
● Sit down if you don’t know the interest rate
on the card(s).

5. Terms of a Debt
Asks the question - Is this debt worth the price?
● Cost of Credit
○ Typical Mortgage - borrow $240k
at 7.759% over 30 years
■ $379k is the cost of credit
■ This is a BET that the home
will appreciate at least this
much over 30 years.
● Stand up if you’ve ever considered cost
of credit or terms for the shortcuts you
took in your last project.

6. Psssst … all y’all sittin’ down?
You might be doing it wrong …

7. Consumer Debt - Can be good
Real examples from my personal finance
I recently borrowed $30k for a car b/c the
cost of credit was 2%. I’ll make more on
investments during the length of the loan.
Car - Depreciating Asset
Borrowed $160k in 2012 on 15 years. Owe about
$70k today. Home worth $540k. Debt allowed me
to purchase this asset. Cost of credit $39k
Home #1 - Appreciating Asset
Pay it off every month. Costs built into the
price of goods, so points are a way to recoup
this.
Credit Card - Short term debt
Borrowed $540k in 2021. Owe about $520k
today. Home appreciated 15%. Debt allowed
purchase of dream much cheaper.
Home #2 - Market Opportunity

8. Metrics of Debt
Cost of
Credit
Unlike consumer debt,
we don’t have solid ways
to measure what we are
getting ourselves into …
We also don’t have
scoring (Transunion, et
al) to tell us if we can
afford more debt.
In simple terms, if I
borrow 100 tests not
written today and that
saves me 10 days of
work now …
what are the costs in
time one year from
now?
You got one with
every loan. This is the
structure of the debt.
Is there an origination
fee? Late payment
penalty? Fixed interest
or variable? Balloon
payment?
What is gained by
taking on the debt?
● Immediate
term?
● Long term?
Term
sheet
ROI

9. Let’s build a Term Sheet for Tech Debt
A good debt should increase flow now but only marginally
impact it later
● Flow - For now let’s think of “flow” as
coin-of-the-realm for a team.
● ROI - Gets us that new feature critical to
business success
● Cost of Credit - After the feature is
delivered, how much of our existing flow
must now be tasked to debt service
(extra maintenance effort)?
● Term Sheet - How do we make this
evident to the business?

10. But everything is “critical to the business”
Of course it is, because you allow them to believe it is free
● Because we don’t measure and reason
about “flow”, the short cuts we take have
no cost to the business.
● So when we say “tech debt” they hear,
“free, except for the griping by
engineering.”
● Don’t wait for someone else to value your
team’s time and effort.

11. Business scenario
Business opportunity
Choosing Debt
A major reseller proposes that if we can offer a
specific capability they will enter into a $4m biz
agreement with us.
We have to get this capability to market in 3
months due to certain marketing and revenue
cycles with the reseller.
We decide to do 20% (rather than 60%) test
coverage while using dead easy tech (MySQL)
rather than what we anticipate to be more
appropriate tech (Kafka + Elasticsearch)
The Essential Conflict

12. Negotiating the Nature of the Debt
Lets talk testing …
We decided to do 20% (rather than 60%) test coverage
Assume that to get to 60% on initial development it was 1000 hours of work.
● Assume it’s 1500 hours after the fact.
● Assume 2 Sev1 and 4 Sev2 incidents resulting from lack of coverage.
● Assume an average of 2 escaped issues per sprint
Can getting to 60% coverage (or better) in three months while keeping up with current
maintenance burden and new project work even happen?
Time as
Opportunity Cost
1500 hr + 600 hr + 240 hr = 2200 hr
Roadmap shifts 3 weeks later.
Treasure
$60k + $120k + $30k = $210,000
Opportunity
$4,000,000

13. Example Term Sheet
Scenario Time as
Opportunity Cost
1500 hr + 600 hr + 240 hr = 2200 hr
Roadmap shifts 4 weeks later.
● 500 hr @ $120/hr = ($60,000)
● Cost of SEV1/SEV2 is $20k and 100 hrs
○ 6 @ $20k = ($120,000)
○ 6 @ 100 hrs = 600 hr
● Cost of escaped issue = 10 hours @ $120/hr = $1200
○ Assume next project is projected for 12 weeks.
■ = 240 hr
■ = ($30,000)
● 1500* hr + 600hr + 240 hr = 2200 hr
○ Team of 4 => 1.1 person taken with debt service
○ 28% reduction in flow while we pay the debt
○ This is (~4 week) shift in the roadmap
Treasure
$60k + $120k + $30k = $210,000
Opportunity
$4,000,000

15. Q&A - 2 or 3 questions
Have I established that Tech Debt is a tool that is neither good nor bad?

16. But…<throws-up-hands>REALITY!</throws-up-hands>
We are already drowning in debt
● How much?
● What is the impact on your flow?
● How much does it cost?
● What is the opportunity cost?
● Did you take on that debt intentionally
or is it a function of your not paying
attention?
WOAH, there buddy! That got dark fast!

17. Harsh
We don’t know how much debt service we have.
● We aren’t drowning in debt, we are drowning
in
○ debt service
○ ignorance
● If you don’t know you have debt how can you
know you are making the payments (service)
● It is our lack of making implicit assumptions of
debt explicit that causes it to accumulate.
● We actually were not paying attention in any
way that matters.
○ Our teams suffer because of this.

18. Remember - Example Term Sheet?
Let’s pull out the assumptions in this - Time as
Opportunity Cost
1500 hr + 600 hr + 240 hr = 2200 hr
Roadmap shifts 4 weeks later.
● 500 hr @ $120/hr = ($60,000)
● Cost of SEV1/SEV2 is $20k and 100 hrs
○ 6 @ $20k = ($120,000)
○ 6 @ 100 hrs = 600 hr
● Cost of escaped issue = 10 hours @ $120/hr = $1200
○ Assume next project is projected for 12 weeks.
■ = 240 hr
■ = ($30,000)
● 1500* hr + 600hr + 240 hr = 2200 hr
○ Team of 4 => 1.1 person taken with debt service
○ 28% reduction in flow while we pay the debt
○ This is (~4 week) shift in the roadmap
Treasure
$60k + $120k + $30k = $210,000
Opportunity
$4,000,000

19. Assumptions About the Info we have on hand
● 500 hr @ $120/hr = ($60,000)
● Cost of SEV1/SEV2 is $20k and 100 hrs
○ 6 @ $20k = ($120,000)
○ 6 @ 100 hrs = 600 hr
● Cost of escaped issue = 10 hours @ $120/hr = $1200
○ Assume next project is projected for 12
weeks.
■ = 240 hr
■ = ($30,000)
● 1500* hr + 600hr + 240 hr = 2200 hr
○ Team of 4 => 1.1 person taken with debt
service
○ 28% reduction in flow while we pay the debt
○ This is (~4 week) shift in the roadmap
● We know how long it will take to write tests
○ Velocity
○ Specifically velocity of test writing
● We are able to reliably estimate the cost of
an incident
● We know the cost of a bug in terms of time
per bug on average.
● We have the next project up well defined
enough to say 12 weeks
● We have a reasonably accurate way to turn
story points in to time.

20. Stand up if you know two or more of these
things about your team.
Predicting maybe a dozen folks at most will rise.
Those are the people who should be speaking at next years Longhorn PHP and the
Austin PHP, Cloud Austin and Austin Automation Prof Meetups !

21. This problem is entirely within our ability
to influence!
In other words, stop complaining about tech debt and start managing it.

22. YASS - Yet another standing survey.
How many of you believe your team is not going as fast as it _should_
OR
You are getting this message from your leadership?

23. Indicators of impending bankruptcy
If you cannot keep up with debt service then 7, 11, 13 …
● Me and my co-founder could create a
feature on a plane ride, now …
● Mgr: “We just gave you 2 new engineers,
why are you not delivering faster?”
● JuniorEng: “Why did the service break
when I added a log message?”
● Staff Eng: “Our deployment frequency
continues to drop.”
● DevOp: “Our change failure rate
continues to grow”

24. Include this question in your sprint retros
Reject perfection. Embrace experimentation. Progress
isn’t Linear - Rissa yesterday
“What is the most important short cut we took this
sprint? Why?”
● Create an actionable ticket(s) to pay down
this debt.
● Ensure the ticket is groomed/sized.
● Tag tickets like this with
`tech-debt-payments`.
● Build a report on this backlog that you can
watch (maybe accumulated story points)
Agile Manifesto - “At regular intervals, the team
reflects on how to become more effective, then tunes
and adjusts its behavior accordingly.”

25. Measure Velocity
Reject perfection. Embrace experimentation. Progress
isn’t Linear - Rissa yesterday
● Get in your ticketing system and measure
velocity.
○ Points, ticket count, doesn’t matter
● Show your work (b/c it is wrong, I promise -
Rissa again, avoid perfectionism)
● Call for review.
● Go back many months/sprints
● Do you see a pattern?
Agile Manifesto - “Agile processes promote
sustainable development. The sponsors, developers,
and users should be able to maintain a constant pace
indefinitely.”

26. Invent a way to turn velocity into time
Reject perfection. Embrace experimentation.
Good enough - Devtime = (P/2)(1.2 + 0.1R)
● P = number of story points
● R = number of research tasks representing
something we don’t know
● 2 - tunable - single point story takes about
half a day for one dev
● 1.2 - tunable - 20% safety for
unknown-unknowns
● 0.1 - tunable - 10% bump to delivery date for
each unknown b/c they usually turn into
more work.

27. Practice quantifying debts from your retro.
Reject perfection. Embrace experimentation
● Add some debt service tickets to your project
and measure how it moves the date.
● Realize it moves the whole roadmap.
● Communicate this to your leadership and
product team and see what they say.
● Improve your message
● Try again.
Agile Manifesto - “Simplicity--the art of maximizing
the amount of work not done--is essential.”

28. Include this question in your sprint retros
Evolve from super villain to super hero.
“What bugs/incidents were caused that we can tie
to these debt service ticket?”
● Build a spreadsheet (you are talking to biz
people, speak _their_ language)
● Determine the cost of each issue.
● Communicate this to your leadership and
product team and see what they say.
● Improve your message
● Try again.
Agile Manifesto - “Business people and developers must
work together daily throughout the project.”

29. Q&A - 2 or 3 questions
Do you see a way to make small changes, frequently to improve your understanding of
debt service and go on to improve your leaderships’ understanding of the same?

30. Coin of the Product Realm

31. Slow is smooth, smooth is fast.
Single piece flow - Lean manufacturing
Small changes done frequently - DevOps
Slow down to speed up.

32. The story of the story that took too long.
A common tail of woe, despair and agony on teams.
● Do you find your team has large tasks
that sit for many days or even weeks in
progress?
● During grooming do you stop and break
down the 8 point story?
○ Did the story take way too long b/c
the person working the problem
kept discovering unknowns?
● Mike Lehan from yesterday - “We are too
eager to start coding”

33. WIP - The Silent Killer
Inventory is cash not working for you.
● ● WIP Perspective
○ The more work in process your team
has the slower it goes. This is math.
● Inventory Perspective
○ All the written code building up in VCS
is cash not working for the business.
○ Think about what is in git right now
that is not yet in production

34. Breaking Work Down
Stop trying to code right away and invest in understanding
what _to_ code. - Mike Lehan yesterday
● When a story is large and we don't break it
down
○ We miss the identifiable unknowns.
○ We miss opportunities to deliver small
value to production sooner
○ We give estimates that are wildly
optimistic with full confidence.
● We miss places where it is easy to take on
technical debt as deadline pressure
mounts.
● We are forced to take “payday loan” types of
debts because WE failed to break large
stories down with maniacal focus.

35. Make Work Visible
It’s what the cool kids do.
● If the work is not visible, then you do not
know what work is being skipped.
● If the work is not visible you don’t know
where your team is taking on debt.
You are doing this to yourself!
STOP!
Maybe practice a bit of that self care Rissa
mentioned yesterday?

36. Slow is smooth, smooth is fast.
Single piece flow - Lean manufacturing
Small changes done frequently - DevOps
Slow down to speed up.

37. A simple Heurstic for Grooming and Retros
Software is invisible. It is hard to see WIP and Inventory.
● In grooming simply declare that if a story is
greater than n points, it should be broken
down until it is 1 and 2 point tasks and some
research tasks (things you learned you didn’t
know!)
● In retro if a story took too long, stop and
break it down based on what you now know.
Use what you learn to get better at
grooming.
○ Don’t keep making the same mistake!

38. By slowing down and breaking down work …
Lean says throughput (flow) and predictability increase …
● ● When batch sizes are small.
○ A ticket in a sprint is a batch
○ A sprint is a batch
○ A deployment is a batch
● Break down your tickets
● But also consider if your sprint size
could actually change too!
● Consider the power of identifying where
you can dial “batch size” and it is all in
your control.

39. What? Need more Convincing?
Application of logic …
● When it is one ticket it is common that only
one engineer is working on it. Where it 5
smaller tickets others could work in parallel
○ Result - Improved flow in the team
● When only one engineer can work a large
task managers think, "Oh look, I have idle
engineers," and allow more work into your
team.
○ Result - Decreased flow in the team

40. Slow is smooth, smooth is fast.
Single piece flow - Lean manufacturing
Small changes done frequently - DevOps
Slow down to speed up.

41. Tie it together
Flow
Maniacally
track flow
in your
team.
Identify tech
debt intake
areas with
better work
break down.
Measure tech
debt time, toil
and treasure.
Report to
your biz
stakeholders.
Detect Measure

42. With what you’ve learned these last days
Go forth and make exactly one imperfect step towards finding the sources of your debt
Develop a simple - if imperfect - measure of your team’s current debt service
Make one change to exert control on the debt you have