Weitere ähnliche Inhalte Ähnlich wie Why Some Companies Are More Innovative Than Others (20) Kürzlich hochgeladen (20) Why Some Companies Are More Innovative Than Others1. Copyright © 2018 by Stefan Thomke
Stefan Thomke
Professor of Business
Administration
Harvard Business School
Ashish
Chopra
Director, Product Marketing
Optimizely
Business Experimentation:
The Engine of Innovation
2. Copyright © 2018 by Stefan Thomke
Housekeeping • We are recording this webinar
• You’ll receive the recording and slides
• We’ll answer all questions at the end
4. Copyright © 2018 by Stefan Thomke 4
Business Experimentation:
The Engine of Innovation
Stefan Thomke
William Barclay Harding Professor of Business Administration
Harvard Business School
5. Copyright © 2018 by Stefan Thomke 5
What Is Innovation?
33,528 Number of times “innovation” was mentioned
in quarterly & annual reports (2011)*
255 Books published in last 90 days with
“innovation” in title*
Source: * Wall Street Journal, May 23, 2012
6. Copyright © 2018 by Stefan Thomke
What Is Innovation?
Innovation
(Novelty + Value)
• Incremental (can be powerful)
• Breakthrough
• Disruptive
Products & Services
Channels & Markets
Processes
Technologies
Business Models
What is Innovation @ Your Organization?
Invention
(Patents)
6
7. Copyright © 2018 by Stefan Thomke
Why And How Do Some Companies
Develop Great Innovations?
7
9. Copyright © 2018 by Stefan Thomke 9
…While Some Firms Have Surprising
Solutions To Customer Problems
10. Copyright © 2018 by Stefan Thomke 10
Growing Through Innovation:
Management’s Dream
• A steady and predictable stream of breakthrough
products, services, and businesses.
• High margins and growth due to first-mover
advantages.
• High speed and productivity.
• Investments in disruptive technologies address
new markets and customer needs.
• …
12. Copyright © 2018 by Stefan Thomke 12
Growing Through Innovation:
Management’s Reality
• Management faces strong incentives to focus on
short term.
• Innovation activities can be highly variable and
difficult to plan and predict.
• Customer behavior is very difficult to predict.
• Organizational incentives encourage discourage
risk taking, experimentation and change.
• Core Challenge: the ability to manage uncertainty
is fundamental to innovation.
13. Copyright © 2018 by Stefan Thomke 13
Why Business Experimentation
Drives Innovation
Source of uncertainty How can uncertainty
be resolved?
• R&D
• Does “it” work as intended?
• Production/Scale-Up
• Can “it” be effectively produced?
• Customer Needs
• Does “it” address actual needs?
• Market/Business
• Does the opportunity justify the
(resource) investment?
1. Experience?
14. Copyright © 2018 by Stefan Thomke 14
How can uncertainty
be resolved?
• R&D
• Does “it” work as intended?
• Production/Scale-Up
• Can “it” be effectively produced?
• Customer Needs
• Does “it” address actual needs?
• Market/Business
• Does the opportunity justify the
(resource) investment?
1. Experience
2. Data?
Source of uncertainty
Why Business Experimentation
Drives Innovation
15. Copyright © 2018 by Stefan Thomke 15
But What Can We Learn From Data?
More Novelty = Less Data
(Is It Really Novel If Someone Has Done It Before?)
Context Matters
(Does Experience Transfer to Other Contexts?)
Correlation ≠ Causation
(Does Management Action Lead to Desired Outcome?)
16. Copyright © 2018 by Stefan Thomke 16
Famous Correlations:
Don’t Trust Observational Studies
Palm size correlates with life expectancy
Ice cream sales correlates with drowning deaths
More cleaning by men correlates with shorter lives
17. Copyright © 2018 by Stefan Thomke 17
Famous Correlations:
Don’t Trust Observational Studies
Palm size correlates with life expectancy
(Common cause: women have smaller palms and live longer)
Ice cream sales correlates with drowning deaths
(Common cause: warmer temperatures)
More cleaning by men correlates with shorter lives
(Common cause: unknown, many factors possible)
18. Copyright © 2018 by Stefan Thomke 18
Why Business Experimentation
Drives Innovation
Source of uncertainty How can uncertainty
be resolved?
• R&D
• Does “it” work as intended?
• Production/Scale-Up
• Can “it” be effectively produced?
• Customer Needs
• Does “it” address actual needs?
• Market/Business
• Does the opportunity justify the
(resource) investment?
1. Experience
2. Data
3. Experiments
19. Copyright © 2018 by Stefan Thomke 19
Recipes for Innovation:
Past and Present
“ The real measure of success is the number of experiments
that can be crowded into twenty-four hours. ”
– Thomas Alva Edison, Inventor
“ Our success at Amazon is a function of how many
experiments we do per year, per month, per week, per
day...”
– Jeff Bezos, CEO, Amazon
“ We try things … we celebrate our failures. This is a company
where it’s absolutely okay to try something that’s very hard,
have it not be successful, and take the learning from that. ”
– Eric Schmidt, ex-CEO, Google
20. Copyright © 2018 by Stefan Thomke
The Digital Opportunity:
Competing Through Online Experiments
20
21. Copyright © 2018 by Stefan Thomke
Experimentation Organization Needed:
How Microsoft (And Others) Do It
Key Data at Bing
• > 15,000 online
experiments per year.
• Experiments run on
millions of users (billions
of variations).
• Only 10-20% of
experiments generate
positive results.
• 80% of proposed changes
are tested as controlled
experiments.
Source: The Surprising Power of Online Experiments (Harvard Business Review, 2017).
21
22. Copyright © 2018 by Stefan Thomke
Scale and Velocity Pays
0
100
200
300
400
500
600
2013 2014 2015 2016 2017
01/01/2013 = 100
Experimenters S&P 500
Source: Bloomberg (01/02/2018)
Experimenters: Amazon, ETSY, Facebook, Google, Microsoft, Netflix, Priceline (Booking.com)
22
23. Copyright © 2018 by Stefan Thomke
Advice 1: Scale and Velocity
Could You Run >10x Experiments?
23
• Capacity: How many experiments can you run
simultaneously (infrastructure, user population)?
• Creativity: Can you generate enough hypotheses
that are worth testing?
• Communication: Can you manage large test
programs and absorb the results? Do they really
impact decision-making?
24. Copyright © 2018 by Stefan Thomke
Experimentation Capacity
Infrastructure Should Not Be the Bottleneck
when people are not working 100% of the time—and
therefore, a busy development organization will be
faster and more efficient than one that is not as good
at utilizing its people.
But in practice that logic doesn’t hold up. We
have seen that projects’ speed, efficiency, and out-
put quality inevitably decrease when managers com-
take 5% more time to complete.
Processes with high variability behave very dif-
ferently. As utilization increases, delays lengthen
dramatically. (See the exhibit “High Utilization
Leads to Delays.”) Add 5% more work, and complet-
ing it may take 100% longer. But few people under-
stand this effect. In our experience with hundreds
High Utilization
Leads to Delays
The curve below is calculated using Queuing
Theory, the mathematical study of waiting
lines. It shows that with variable processes,
the amount of time projects spend on hold,
waiting to be worked on, rises steeply as
utilization of resources increases. Though
the curve changes slightly depending on the
project work, it always turns sharply upward
as utilization nears 100%.
30x
0
RESOURCE UTILIZATION
WAITINGTIME
Waiting times more
than double as
utilization moves
from 80% to 90%
and double again as
it moves from 90%
to 95%.
100%20% 40% 60% 80%
1802 Harvard Business Review May 2012From: “Six Myths of Product Development”, Harvard Business Review, May 2012. 24
25. Copyright © 2018 by Stefan Thomke
Actions That Can Increase Velocity
• Invest in capacity: non-linear returns (speed)
• Align objectives: focus on response time
• No idle time: make “deviations” easier to see
• Expectations: Create strategic slack (3M, Google)
25
26. Copyright © 2018 by Stefan Thomke
Advice 2: Culture
Create Learning Organization
• Learning from failure is central to innovation.
• Failure ≠ mistakes; mistakes produce no new or
useful information and are without value.
• Not winning ≠ losing; failed experiments are the
source of new hypotheses and iterations.
• Biases, incentives, and governance can slow down
experimentation.
26
27. Copyright © 2018 by Stefan Thomke
Advice 3: Quality
Pay Attention to Discipline and Rigor
• Invest in testable hypotheses that have well-
defined, measureable variables
• Strong: “Opening stores one hour later to reduce
operating costs will not lead to a significant drop in sales”
• Weak: “We can extend our brand upmarket”
• Build trust in the data
• Safeguards: A/A tests, outlier detection, simplicity, etc.
• Understand the statistics
• Follow Twyman’s law: “Any figure that looks interesting or
different is usually wrong”
27
28. Copyright © 2018 by Stefan Thomke 28
Disciplined Business Experiments:
Some Essential Questions
• Does the experiment have a clear purpose?
• Specific management decision, clear learning objective.
• Will the stakeholders abide by the results?
• Agreed changes/actions, results aren’t ignored.
• Is the experiment doable?
• Testable prediction, required sample size, minimum duration.
• How can we ensure reliable results?
• Control group, systemic biases, randomization.
• Have we maximized the experiment’s value?
• Targeted rollout, components with highest ROI, causality.
29. Copyright © 2018 by Stefan Thomke 29
Management Decisions by Experiment
“It doesn’t matter how beautiful your
theory is, it doesn’t matter how smart
you are. If it doesn’t agree with
experiment(s), it’s wrong.”
Richard Feynman
Scientist, Teacher, Storyteller, Musician
and Nobel Laureate (1918-1988)
31. Copyright © 2018 by Stefan Thomke 31
optimize.ly/opticon18
Promo code: THOMKE550