How to maximize learning and minimize risk. All new products start as a series of unvalidated assumptions. The most critical assumptions are usually implicit and relate to the purpose of the product and the value it is intended to deliver. The more key assumptions involved, the greater the risk. It is enough to have 7 key assumptions about which you are 90% certain for the combined odds of success to be below 50%. Contrary to popular belief, when we know very little about a situation, it only takes a small amount of new data to realise significant insights. Unfortunately, people often underestimate the value of information and misunderstand risk. As Product Owners we are often afraid to test our assumptions. We routinely pile on additional risk without a second thought. Do we have a death wish or are we simply masochists? Risk management is the bread and butter of the finance and insurance industries. Isn’t it time we evolved? In this fast paced and practical session we will explore answers to the following questions: - What is risk and how do we quantify and manage it? - How do we assess the value of information? - How can experimentation reduce risk and where does it fit in the product development cycle? - What makes a good experiment? - How to run experiments in a cost effective manner? - What are good metrics? - How to obtain Zen like focus and prioritisation? New concepts will be introduced, examples will be given and we will then point out where to seek further information. Hold onto your hats.

1. Business Model Innovation by
Experimentation
Yoav Aviram
Energized Work
@yobo

2. IT Projects
60% don’t meet schedule, budget or quality goals - IBM
50% failed to achieve what they set out to achieve - KPMG
17% go so badly that they can threaten the very existence of the
company. - Calleam Consulting

3. Startups
95% of technology startups fail - Allmand Law
93% of the Angel investments never achieve expected ROI - University
of Washington
80% of the VC investments never achieve expected ROI - National VC
Association

4. WHY?

5. Too late
Poor quality
Missing functionality
Software: Tip of the Iceberg

6. An iterative and incremental approach to development, where
requirements and solutions evolve through collaboration between selforganizing, cross-functional teams
- Wikipedia
Agile

7. ...the Product Iceberg
Who is the customer?
How do we acquire new users?
How do we convert users to paying customers?
How should the product look and behave?
How is our product used?
What problems does the product attempt to solve?
What features provide the greatest value?

8. A combination of business-hypothesis-driven experimentation,
iterative product releases, and "validated learning"
- The Lean Startup, Eric Ries
Lean Startup

9. ...the Business Model Iceberg
Who are our customers?
Who is the competition?
How do we reach our customers?
What are the revenue streams?
What is the investment needed?
What is the operational cost?
What is our unique selling point?
What team members to recruit?
What are our business goals?

10. Business Model Innovation
- Business Model Generation, Alexander Osterwalder

11. There is a poorly met need a reachable market faces
There is a solution people are happy to pay for
There is a way to package and deliver the solution in a cost effective
manner
Product-Market Fit

12. This is all About Risk

13. The lack of complete certainty, that is, the existence of more than one
possibility. The "true" outcome/state/result/value is not known.
- How to Measure Anything, Douglas Hubbard
Uncertainty

14. A set of probabilities assigned to a set of possibilities
Quantifying Uncertainty
“There is a 60% chance this market will double in five years”

15. A state of uncertainty where some of the possibilities involve a loss,
catastrophe, or other undesirable outcome.
- How to Measure Anything, Douglas Hubbard
Risk

16. A set of possibilities each with quantified probabilities and quantified
losses
Quantifying Risk
“There is a 40% chance the proposed oil well will be dry, with a loss of
$12 million in exploratory drilling costs”

17. The Problem with Risk & Technology
Technology is used to solve complex problems
Our mind is wired to oversimplify complexity (we are not very good at
understanding probabilities)
We are solution oriented
We are optimistic (overconfident) by nature
People (grossly) underestimate risk & uncertainty

18. Combination of two or more related risks
Compound Risk
Compound probabilities are (very) counter intuitive

19. Contestants on a game show are given the choice of three doors.
Behind one door is a car, behind the others, goats.
After a contestant picks a door, the host, who knows what's behind all
the doors, opens an unchosen door, which reveals a goat.
He then asks the contestant, "Do you want to switch doors?"
Monty Hall

20. Should the contestant switch doors?

21. Yes!
The odds of winning are 2 out of 3 if you
switch

22. This answer is so counter intuitive, most people get it wrong
It’s why they made a game show off of it: the trick works every time
Compound Probabilities
The lesson:
We need to give serious consideration to risk
We must not rely on our intuition when factoring risk & uncertainty
We need a better accounting system

23. Risk Management for Techies
Avoid by eliminating the situation or activity that presents it
Transfer through insurance or through other types of contracts
Reduce by hedging your bets or reducing uncertainty
Retain, because some risks are worth assuming

24. Risk Reduction

25. Spread your bets on a portfolio of investments instead of one
Decide on an investment strategy
Split the investment into small incremental bets: abort bad products
quickly and ramp up investment in the good ones
Investor Mindset

26. Uncertainty Reduction

27. Value of Information
Information reduces uncertainty
Reduced uncertainty improves decisions
Improved decisions have observable consequences with measurable
value
- Information Theory (1948)

28. Sources of Information
Existing within the company
Other’s research
Experiments
Live product

29. A scientific procedure undertaken to make a discovery, test
a hypothesis, or demonstrate a known fact
- Wikipedia
Experiment

30. Experimental Discovery

31. Prioritising Experiments
Uncertainty around making a decision is high (little existing
information)
The value of the opportunity, or the cost of a mistake, is high
Experimentation is least expensive in terms of cost and time

32. Attempt to falsify a hypothesis
Are objective, measurable and repeatable
Are controlled (variables tested in isolation)
Are cost effective
Influence a decision
Designing Good Experiments
Experiments aren't free
It's an investment decision
Information may be cheaper elsewhere

33. A measurement is an observation that results in information (reduction
of uncertainty) about a quantity
Measurement

34. The soft stuff
Descriptive, subjective, messy and hard to quantify
Customer surveys, focus groups, interviews
But … provide insights about Perceived Value
Qualitative Measurements

35. Good Qualitative Questions
Avoid leading phrases such as "do you agree that..."
Make customers part with money or sign up, rather than asking them
whether they would
Use follow up questions to get to the ‘why?’
Qualitative data can easily be influence by cognitive biases

36. Start by decide on the right Metric (what to measure)
Then set lines in the sand: what success and failure look like
Always validate results with qualitative data
Quantitative Measurements

37. Good Metric
Influences a decision
Comparative
Rates and Ratios are easier to compare and act upon
Measure what’s important to your customers

38. Putting it all Together
Invest wisely, through small incremental bets to reduce risk and
discover opportunities
Information reduces uncertainty & risk
Experiments are not the only source of information
Experiments have the biggest impact in a high value, high uncertainty
and little data situations
Disprove hypotheses, don’t confirm them
Think in terms of Risk and account for it
The distinction between business and technology is anachronistic

39. The Lean Startup by Eric Ries
Business Model Generation by Alexander Osterwalder
How To Measure Anything by
Douglas W. Hubbard
Lean Analytics by Alistair Croll and Benjamin Yoskovitz
Reading List

40. Questions?
Yoav Aviram
Energized Work
yoav@energizedwork.com
@yobo

41. Frame your business and product as a set of hypotheses
Declare your assumptions and how you can prove them wrong
(falsifiable)
Evaluate your results ruthlessly, and be prepared to change course
Accounting for Risk

42. Assumption Backlog
Assumption
Certainty
Risk ($)
Critical
People like shiny apps
90%
$100K (cost of beta)
Yes
Market size for shiny apps at least 200
million people
50%
$1M (cost of a full product)
Yes
I can build a shiny app
95%
$50K (cost of development)
Yes
People share shiny apps with friends
45%
$100K (cost of beta)
No
5% of users will upgrade to shiny+ for
$5 a month
20%
$1M (cost of a full product)
Yes
10% of active users will spend $2 a
month on shiny accessories
75%
$100K (cost of beta)
No