Shazam has been growing very fast recently. A lot of this growth happened in the engineering department at London HQ where our mobile apps are being developed. This talk will go into technicalities of project management techniques we use to keep 100+ MAU a month happy while maintain agility and respond to rapidly changing market situation. We will talk about data driven Kanban, flow, visual standups, changing requirements and structure of our teams. We will show what metrics we care about, how we measure them and what we do with the results. You should expect some observations about development of state of art apps useful from product/project manager’s and developer’s perspective.

1. Shazam mobile apps -
Data Driven Project Management
Tomasz Kustrzynski, October 2014

2. © 2014 Shazam
Entertainment
About Shazam
music recognition service and apps
founded in 1999
launched in the UK in 2002
one of the most popular mobile apps
500M+ downloads

3. © 2014 Shazam
Entertainment
Shazam technology - then
1. call 2580
2. play the music
3. wait for SMS with results
(phone you most likely used in 2002)

4. © 2014 Shazam
Entertainment
Shazam technology - now
since 2008 - iPhone and Android apps
since 2011 - integrated with Spotify
since 2014 - integrated with iOS8’s Siri
music streaming, TV ads, live TV,
and more

5. Shazam is now a part of popular culture...
© 2014 Shazam
Entertainment

6. © 2014 Shazam
Entertainment
...with lots of uses...

7. © 2014 Shazam
Entertainment
...but there’s still a lot to do

8. Agenda
organisation
embrace the change
spiral development
why and what do we measure
estimation (and causation bias)
handling dependencies

9. Organisation

10. © 2014 Shazam
Entertainment
The teams

11. © 2014 Shazam
Entertainment
Technical Project Management
the scientific method
hypothesis experiment
analyse
data
draw
conclusions
improve continuously
(Kaizen)

12. © 2014 Shazam
Entertainment
We use Kanban
visualise workflow

13. © 2014 Shazam
Entertainment
We use Kanban
visualise workflow
limit Work In Progress
(WIP)

14. © 2014 Shazam
Entertainment
We use Kanban
visualise workflow
limit Work In Progress
manage the flow

15. © 2014 Shazam
Entertainment
We use Kanban
visualise workflow
limit Work In Progress
manage the flow
define explicit policies

16. © 2014 Shazam
Entertainment
We use Kanban
visualise workflow
limit Work In Progress
manage the flow
define explicit policies
improve collaboratively

17. © 2014 Shazam
Entertainment
Why Kanban
Our work is complex, requirements and circumstances change.
(fixed scope iterations won’t work)
We can be flexible with release cadence.
(we release when we’re ready)
Visibility and metrics.
(value stream mapped on the board)

18. © 2014 Shazam
Entertainment
This talk is not about theory
why limit work in progress?
why focus on finishing, not starting?
why slack is good for efficiency, ‘busyness’ is not?
why early feedback loops are necessary?
effectiveness vs efficiency?

19. Embrace the change

20. © 2014 Shazam
Entertainment
What do we care about
100+
Users. We now have millions monthly active users.

21. © 2014 Shazam
Entertainment
Types of work
new functionalities commercial deals
deadlines
(quality still matters)
defined scope
(flexible interpretation)
no deadlines
(quality most important)
flexible scope
(UX most important)

22. © 2014 Shazam
Entertainment
Types of work
new functionalities commercial deals
requirements change rapidly
in both cases

23. © 2014 Shazam
Entertainment
Hack time (3rd type of work)
some time can be used to work on any project
Shazam for Mac was
one of them

24. © 2014 Shazam
Entertainment
Hack time - results

25. © 2014 Shazam
Entertainment
Last minute changes

26. © 2014 Shazam
Entertainment
Real Options model

27. © 2014 Shazam
Entertainment
Real Options model
● options have value
(you’ll have more information/make better decision later)
● options expire
(ability to execute option ceases to exist at some point)
● never commit early unless you know why
(sometimes no decision is the best decision)

28. © 2014 Shazam
Entertainment
Real Options in practice
● no long queues
● designs and specs refined as we go
● A/B testing
● shorter release cycles
● irreversible commitments

29. © 2014 Shazam
Entertainment
Real Options - example
early commitment
backing out of early commitment
backlog too long
late commitment

30. Spiral development

31. © 2014 Shazam
Entertainment
Ceremonies
ceremonies limited to a minimum
meetings often ad hoc, here and now
● morning standups
● retrospectives
● visual reviews
● bug triage meetings
● detailed planning
● estimation

32. © 2014 Shazam
Entertainment
Iterative/spiral development
1. update requirements
2. implement
3. visually review new
functionalities
(usually work in progress)
4. collect feedback
5. repeat until signed off

33. © 2014 Shazam
Entertainment
Visual reviews
in the breakout area for everyone to join

34. © 2014 Shazam
Entertainment
Visual reviews
in the breakout Terminal 6 area for everyone to join

35. Why and what do we measure?

36. © 2014 Shazam
Entertainment
T6 - taking the metaphor further
flight board
shows status of epics
all teams included
we track more than this...

37. © 2014 Shazam
Entertainment
Data driven
decisions

38. © 2014 Shazam
Entertainment
Data gathering
"In God we trust; all others must bring data."
Edward Deming
"What's measured improves."
Peter F. Drucker

39. © 2014 Shazam
Entertainment
Data gathering
information radiators across
the office

40. Estimation

41. © 2014 Shazam
Entertainment
Estimates are dangerous

42. © 2014 Shazam
Entertainment
Estimation scale we use
1
TFB - too f* big
NFC - no f* clue
[give me a minute]
cards in the photo by
Lunar Logic (@Lunar_Logic)

43. FAQ 1
As a Product Manager, when can I expect this Story
to be completed?

44. © 2014 Shazam
Entertainment
Disney/Cycle Times
time for a Story to get
implemented and tested
(or time from entering queue until
finishing the ride in Disneyland)
having lots of samples we
can use statistics
(and talk about probabilities)

45. © 2014 Shazam
Entertainment
Disney/Cycle Times
parallel
streams of
work
Cycle Times
t
work done
start of test
start of development

46. © 2014 Shazam
Entertainment
Disney/Cycle Times - statistics
histogram of Disney
Times (Cycle Times) density function
50th, 85th percentile
(50/85% items get finished within this time)

47. © 2014 Shazam
Entertainment
When a Story will get done?
1. You can get this story within X days with 85%
probability.
2. Probability of you getting it within a week is Y%.

48. © 2014 Shazam
Entertainment
Scientific method in action
we track median (50th percentile) Cycle Times:
tracking metrics in time allows us to use scientific method

49. © 2014 Shazam
Entertainment
Hypothesis
limiting Work In Progress will shorten Cycle Times
according to Little’s Law:
avg Cycle Time =
avg Work In Progress
avg Throughput

50. © 2014 Shazam
Entertainment
Correlation
Work In Progress limit for
Development lowered
correlated results
next day

51. Does eating ice cream make you more
© 2014 Shazam
Entertainment
prone to drowning?

52. © 2014 Shazam
Entertainment
Causation?
we had a glitch in the system, but over the next
month results matched our hypothesis, and we
could replicate the experiment

53. FAQ 2
Ok, so when can we get all these 10 stories done?

54. © 2014 Shazam
Entertainment
Modelling parallel work
parallel work is difficult to model with cycle times
it’s easier with Takt Times
Takt Times
parallel
streams of
work
Cycle Times
t

55. © 2014 Shazam
Entertainment
Distribution of average Takt Times
create set of average Takt Time samples based on similar project
average
resample and
Σmany times
(>1000)
average Takt distribution of average Takt Times
Times
Takt Times

56. Predicting delivery time for N items
© 2014 Shazam
Entertainment
average TT
* N
repeat many
times (>1000)
average TT
average TT
average TT
average expected delivery
distribution of expected delivery times
Takt Times
times
85% probability to deliver within this time

57. © 2014 Shazam
Entertainment
When will all the Stories get done?
1. You can get these Stories within X days with 85%
probability.
2. Probability of you getting them within a week is Y%.

58. FAQ 3
Why did we end up with large number of stories to
test at once?
Do some types of items take longer to develop than others?
Are there times when teams are more productive?
Did we split Stories properly or we missed splitting some big
ones?

59. © 2014 Shazam
Entertainment
Patterns and improvement
Being aware of patterns enables us to act on them:
● introduce better policies
(e.g. all UI assets for a Story done before dev commences)
● change development cycle
(e.g. release more often to prevent batching)
● treat some types of work differently
(e.g. allow extra dev time)

60. © 2014 Shazam
Entertainment
Control chart

61. © 2014 Shazam
Entertainment
Control chart

62. © 2014 Shazam
Entertainment
Control chart

63. © 2014 Shazam
Entertainment
Control chart
weekends

64. Dependencies and blockers
As a Project Manager, what should I focus on next?

65. © 2014 Shazam
Entertainment

66. © 2014 Shazam
Entertainment
Blockers/dependencies
● visualise
● prioritise
(what is blocked, for how long)
● act
● collect statistics and review them
(have any recent changes to the process change amount
of blocked items/how long they’re blocked for?)

67. © 2014 Shazam
Entertainment
What should I do
first? - answers
Team A: finally doing well
Team B: keeps getting
blocked
4 blockers including 1 old
Team C: doing well
Team D: 1 new blocker

68. Cumulative Density
Function
(80% blocked items
unblocked in ~3 days)
© 2014 Shazam
Entertainment
Blockers statistics
trend of increasing
median time to
unblock

69. Summary
● use Kanban for flexibility, continuous improvement
● embrace change, use Real Options rules
● daily visual reviews allow fast feedback cycle
● make decisions based on data
● make estimates based on statistical analysis
● handle dependencies and blockers

70. Thank you
tomasz.kustrzynski@shazam.com

71. © 2014 Shazam
Entertainment
References
● Real Options: commitment-thebook.com, Olav Maassen, Chris Matts
● Real Options: http://www.infoq.com/articles/real-options-enhance-agility
● Real Options: http://decision-coach.com/lean-and-real-options/
● “High level planning using Monte Carlo simulation”, Dimitar Bakardzhiev
● “Thinking, fast and slow”, Daniel Kahneman
● http://en.wikipedia.org/wiki/Takt_time