Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
'Test & learn goes mainstream' (iq, fall 2013) pdf
1.
‘Test & Learn’ Goes Mainstream
By
Brian
Christian
M
uch
of
the
strategic
innovation
work
that
Inovo
has
done
over
the
years
has
been
at
the
front
end
of
innovation,
or
in
what
we
call
the
Discovery
phase.
Oftentimes,
having
helped
our
clients
to
discover
a
set
of
attractive
opportunities,
we
are
asked
to
stick
around
and
assist
with
post-‐
Discovery
phase
work.
For
strategic
innovations,
the
post-‐Discovery
phases
are
commonly
referred
to
as
Incubation
and
Commercialization.
Recently,
several
of
our
clients
have
inquired
about
an
approach
to
Incubation
and
Commercialization
called
‘Test
&
Learn.’
In
this
Innovation
Quarterly
article,
we
would
like
to
offer
our
perspective
on
the
emerging
Test
&
Learn
movement.
Before
we
dive
in,
it
may
be
helpful
to
remind
our
readers
what
we
mean
by
strategic
innovation.
(For
a
more
detailed
understanding,
please
see
our
white
paper
on
strategic
innovation
at
www.theinovogroup.com.)
Others
refer
to
this
type
of
innovation
as
breakthrough,
disruptive
or
radical.
At
Inovo,
we
prefer
the
descriptor
‘strategic’
because
it
connotes
the
degree
of
long-‐term
importance
this
type
of
innovation
has
internally
for
the
innovating
organization.
Not
all
strategic
innovations
are
radical
or
disruptive
and
neither
do
they
need
to
be.
Others
(such
as
Govindarajan
and
Trimble
in
10
Rules
for
Strategic
Innovators)
have
also
chosen
the
label
‘strategic
innovation,’
and
the
world
has
come
to
recognize
that
strategic
innovation
requires
a
very
different
set
of
principles,
methods
and
skills
than
does
the
more
incremental,
sustaining
type
of
innovation.
It
is
also
appropriate
to
view
a
strategic
innovation
as
the
large
company
version
of
a
startup.
Chief
among
the
proponents
of
Test
&
Learn
are
Steve
Blank
(‘Customer
Development’),
Eric
Ries
(‘The
Lean
Startup’),
and
Alexander
Osterwalder
(‘Business
Model
Generation’).
For
the
purposes
of
this
article,
let’s
call
their
collective
methodologies
the
Test
&
Learn
Approach,
or
TLA.
TLA
goes
something
like
this:
The
startup
entrepreneur
has
a
vision
of
a
specific
innovative
offering
and/or
business
model
in
mind,
as
well
as
an
approach
for
achieving
that
vision.
The
precise
manifestation
of
The Innovation Quarterly (Fall 2013) | 1
2. the
actual
offering
and
its
related
business
model
emerges
through
an
iterative
process
of
testing
with
potential
customers
in
an
approximation
of
the
eventual
marketplace.
The
entrepreneur
learns
from
these
tests
and
adapts
(or
‘pivots’)
as
necessary
to
optimize
the
‘product/market
fit.’
In
essence,
each
test
is
aimed
at
one
or
more
assumptions
or
hypotheses,
and
the
startup
learning
journey
will
succeed
or
fail
according
to
how
well
one
tests
these
assumptions.
The
entrepreneur
does
not
declare
failure
until
the
vision
has
been
proven
wrong
–
or
the
clock
or
funding
runs
out.
This
is
contrasted
against
the
traditional
startup
approach
in
which
more
thorough
planning
and
analysis
are
conducted
with
the
expectation
that
the
first
version
tested
in
the
market
is
essentially
the
right
one.
In
the
highly
uncertain
world
of
startups,
this
traditional
approach
often
leads
to
disappointment,
missed
expectations
and
investor
fatigue.
To
provide
a
deeper
perspective
on
TLA,
let’s
conduct
a
brief
‘Lean’
primer.
One
of
the
foundational
principles
underlying
Lean
Thinking
is
the
elimination
of
waste
(muda
in
Japanese).
The
original
application
of
Lean
Thinking
at
Toyota
was
in
manufacturing.
In
Lean
Manufacturing,
waste
is
reduced
with
a
variety
of
specific
methods
and
tools
such
as
value
stream
mapping,
kanban
inventory
control,
and
cellular
manufacturing.
The
TLA
proponents
point
to
Lean
Manufacturing
as
the
inspiration
for
their
thinking.
For
some
reason,
they
have
chosen
not
to
cite
Lean
Product
Development
(as
explored
by
Kennedy,
Liker,
et
al.),
which
is
a
more
recent
application
of
Lean
Thinking.
This
is
also
sometimes
referred
to
as
Lean
Design
or
Lean
Innovation.
In
Lean
Product
Development,
waste
is
reduced
with
methods
such
as
customer-‐defined
value
and
set-‐based
concurrent
engineering.
In
the
Lean
set-‐based
approach,
multiple
design
options
(a
set)
are
tested
in
parallel,
and
options
are
closed
out
only
as
knowledge
is
gained
and
assumptions
are
confirmed
or
refuted.
Eric
Ries
calls
this
‘validated
learning.’
In
Lean
Product
Development,
testing
occurs
according
to
the
iterative
Plan-‐Do-‐Check-‐Act
or
Shewhart/Deming
cycle.
Ries
has
labeled
this
the
Build-‐Measure-‐Learn
cycle.
In
Lean
Product
Development,
a
critical
means
for
eliminating
waste
is
to
include
end
user
or
customer
testing
in
the
PDCA
cycle
(customer-‐defined
value)
to
ensure
design
fit
to
purpose.
Steve
Blank
calls
this
Customer
Development.
As
you
can
see,
the
foundation
for
TLA
is
more
directly
found
in
the
principles
and
methodologies
of
Lean
Product
Development
than
in
those
of
Lean
Manufacturing.
While
waste
elimination
would
not
seem
to
be
an
essential
element
of
a
strategic
innovation
process,
in
fact
it
is
quite
critical.
Its
value
is
less
in
cost
reduction
in
the
normal
operational
sense
and
more
in
the
conservation
of
precious
money
and
people
resources
allocated
to
strategic
innovation
projects.
When
confronted
with
a
vast
white
space
of
opportunity,
an
organization
must
be
able
to
discover
and
exploit
its
hidden
opportunities
within
restrictive
money
and
people
constraints.
This
would
not
be
possible
without
a
Lean
set-‐based
approach.
Also,
as
these
opportunities
are
often
new
to
the
world
and/or
new
to
the
company,
uncertainty
remains
high
right
through
to
commercialization
of
the
new
offering
or
business
model.
The
greater
the
degree
of
uncertainty,
the
more
important
is
a
The Innovation Quarterly (Fall 2013) | 2
3. Lean
set-‐based
approach.
This
is
in
contrast
to
highly
predictable
situations
typical
of
sustaining
innovations,
for
which
the
traditional
linear
stage-‐gate
model
is
a
reasonable
approach.
The
proponents
of
TLA
have
smartly
recognized
the
relevance
of
Lean
Thinking
(as
addressed
by
Ohno,
Womack,
et
al.)
for
startups
and
entrepreneurs.
By
doing
so,
they
have
broadened
the
application
of
Lean
Thinking
and
made
it
understandable
to
an
audience
that
probably
has
not
been
much
exposed
to
Lean
concepts.
In
addition,
through
Osterwalder’s
contributions,
they
have
broadened
the
application
of
Lean
beyond
the
offering
to
encompass
the
entire
business
model.
In
addition,
the
TLA
proponents
have
asserted
that
the
mindset
and
methodologies,
originally
developed
for
startups,
are
translatable
to
the
world
of
large-‐company
strategic
innovation.
As
Blank
stated
in
an
article
in
the
May
2013
edition
of
the
Harvard
Business
Review,
“…despite
the
methodology’s
name
[‘The
Lean
Startup’],
in
the
long
term
some
of
its
biggest
payoffs
may
be
gained
by
the
large
companies
that
embrace
it.”
We
agree
with
this
assertion
and
are
happy
to
see
that
large
companies
are
starting
to
adopt
and
adapt
TLA
principles
and
methods
for
Incubation
and
Commercialization.
However,
there
are
also
some
caveats
to
be
made,
in
our
opinion,
regarding
the
application
of
TLA
for
large-‐company
strategic
innovation.
1. Business
model
diversity:
As
we
move
from
intangible
products
distributed
through
virtual
channels
to
tangible
products
distributed
through
physical
channels,
the
methods
and
tools
required
to
support
TLA
vary
greatly.
In
fact,
business
models
are
becoming
increasingly
complex
with
a
blend
of
intangible,
tangible,
virtual
and
physical.
The
strategic
innovator
must
be
able
to
select
the
right
methods
and
tools
for
each
situation
from
a
vast
array
of
options.
2. Large
company
complexity:
TLA
emerged
as
a
framework
for
startups,
either
VC-‐
funded
or
bootstrapped.
Startups
are
relatively
simple
organizations.
In
translating
TLA
to
large-‐company
strategic
innovation,
it
is
important
to
complement
TLA
with
other
methods
and
tools
that
help
to
manage
a
complex
process
in
a
complex
organization.
The
first
caveat
relates
to
the
vast
difference
across
business
model
types
and
how
this
affects
the
way
that
one
designs
and
conducts
a
TLA
program.
Consider
the
differences
between
the
Incubation
and
Commercialization
of
a
mobile
device
gaming
app
like
Fruit
Ninja
and
that
of
a
human
drug
like
Lipitor,
or
that
of
a
new
commercial
jetliner
like
the
Boeing
787
Dreamliner.
As
business
models
move
from
intangible
to
tangible
offerings
and
from
virtual
to
physical
channels,
the
types
of
methods
and
tools
to
be
used
to
support
a
TLA
program
differ
greatly.
The Innovation Quarterly (Fall 2013) | 3
4.
In
addition,
how
these
methods
and
tools
are
assembled
into
an
overall
TLA
program
design
will
vary
greatly
across
these
diverse
business
model
types.
Here
are
some
of
the
factors
that
vary
across
business
model
type
and
which
have
an
important
impact
on
the
design
of
a
TLA
program
for
Incubation
and
Commercialization.
1. Degree
of
invention:
When
the
required
invention
level
is
high,
iterations
on
the
product
development
side
take
a
long
time.
However,
market-‐side
assumption
testing
need
not
come
to
a
halt
during
these
periods.
Valuable
input
into
the
invention
process
can
be
gained
while
waiting
for
the
minimum
viable
product
(MVP)
or
a
more
advanced
prototype.
The
ecosystem
that
determines
adoption
is
much
broader
than
the
customer
or
end
user,
and
valuable
information
about
the
ecosystem
(e.g.,
regulatory
constraints)
can
be
gained
during
these
extended
invention
periods.
Finally,
the
expected
invention
time
can
often
be
short-‐circuited
by
scanning
the
landscape
for
available
technologies
or
technology
development
partners.
Recently,
Inovo
helped
a
large
CPG
client
(which
typically
spends
years
developing
its
strategic
innovations)
to
identify
a
material
technology
that
significantly
advanced
the
client’s
development
of
a
very
promising
new
product.
(This
high-‐potential
offering
is
launching
in
the
marketplace
this
month,
so
we
will
share
more
about
it
in
the
near
future.)
2. Cost
of
prototype
construction:
When
the
cost
of
prototype
construction
is
high,
as
with
complex
systems
like
electric
vehicles,
it
is
often
practical
to
gain
customer
and
ecosystem
feedback
in
advance
of
a
full
system
investment
to
reduce
investment
risk.
In
these
situations,
it
can
make
sense
to
develop
in
silico
models
of
adoption
to
correlate
design
variations
to
customer
and
ecosystem
needs
and
desires.
A
few
years
The Innovation Quarterly (Fall 2013) | 4
5. ago,
Inovo
built
an
adoption
model
for
the
electric
vehicle
market
to
support
the
product
design
efforts
of
an
electric
vehicle
battery
manufacturer.
It
is
also
possible
to
test
individual
elements
of
a
complex
system
and
piece
together
the
market
responses
to
build
a
complete
picture.
This
has
obvious
risks
but
can
be
superior
to
the
alternative
of
taking
one
large
risk
with
a
complete
system
prototype.
This
is
not
so
much
a
minimum
viable
prototype
as
a
set
of
minimum
viable
subsystems.
You
can
be
sure
that
for
its
Dreamliner,
Boeing
was
market-‐testing
seat
design
with
passengers,
cockpit
design
with
pilots,
galley
design
with
flight
attendants,
and
cargo
bay
design
with
baggage
handlers
all
while
developing
and
testing
the
aircraft
itself
for
flight
worthiness.
3. Cost
of
in-‐market
testing:
Sometimes
in-‐market
testing
is
difficult,
costly
or
even
impossible.
This
is
true
in
many
business-‐to-‐business
situations
as
well
as
in
the
life
sciences.
In
these
situations,
it
is
often
practical
and
sometimes
necessary
to
alleviate
investment
risk
by
getting
end
user
and
broader
ecosystem
input
in
lieu
of
a
minimum
viable
product.
Several
years
ago,
Inovo
helped
a
Fortune
500
client
test
assumptions
related
to
the
use
of
a
proprietary
plastic
composite
to
replace
wooden
railroad
ties.
By
engaging
with
experts
across
the
railroad
industry
ecosystem,
we
were
able
to
help
the
client
discover
that
several
large
railway
operators
were
potentially
interested
in
this
new
composite
offering
for
certain
use
cases.
More
importantly,
however,
we
also
helped
them
discover
that
their
proposed
offering
was
insufficient
along
a
key
performance
dimension
and
therefore
would
not
be
adopted.
The
client
quickly
made
the
decision
to
cease
its
product
development
efforts,
and
we
were
told
that
this
saved
them
a
large
investment.
One
effective
way
to
manage
the
risk
associated
with
the
three
factors
discussed
above
is
to
seek
financial,
product
development,
and/or
commercialization
partners
who
can
share
the
risk
and
perhaps
quicken
one’s
time
to
market.
For
example,
Inovo
is
just
winding
down
a
project
in
which
we
are
helping
a
carbon
fiber
part
manufacturer
build
an
entirely
new
business
in
the
automotive
sector
by
identifying,
screening
and
developing
partnerships
with
OEMs
and
Tier
1
suppliers
who
are
well
positioned
to
co-‐develop
their
first
trial
offerings.
The
second
caveat
regarding
the
translation
of
TLA
to
the
world
of
corporate
strategic
innovation
relates
to
the
real
cultural
and
business
process
differences
between
a
true
startup
(VC-‐funded
or
bootstrapped)
and
a
large,
established
company.
As
Blank
likes
to
say,
“A
startup
is
not
a
small
version
of
a
big
company.”
Here
are
some
important
factors
that
explain
why
TLA
alone
may
not
be
adequate
to
the
task
of
large-‐company
strategic
innovation.
1. Coordination:
Startups
are
simple
organizations
with
few
layers
and
shallow
functional
divisions.
Large
companies
are
complex
structures
with
many
layers
and
The Innovation Quarterly (Fall 2013) | 5
6. deep
functionalization.
Orchestrating
a
‘startup’
activity
within
a
large
company
is
a
complicated
task,
and
doing
it
quickly
enough
to
achieve
adequate
speed-‐to-‐
market
is
a
significant
challenge.
Inovo
has
found
that
constructing
cross-‐
functional
and
multi-‐level
teams
with
clear
accountabilities
and
active
involvement
is
the
key,
but
it
does
require
a
lot
of
effort
and
patience
to
manage
this
team
structure.
2. Decision-‐making:
Startups
have
one
or
few
decision-‐makers,
and
they
are
well
aligned
around
a
simple
goal.
Large
companies
have
many
decision-‐makers,
often
in
a
matrix
consensus
structure,
and
goals
are
not
always
well
aligned.
Decisions
to
adjust,
pivot
or
proceed
within
a
large
company
are
difficult
to
make
effectively
while
maintaining
alignment.
At
Inovo,
we
have
found
that
the
use
of
online
voting
tools
to
gather
opinions
and
preferences
in
advance
of
decision-‐making
events
are
a
great
way
to
cut
through
politics
and
posturing
and
also
to
broaden
the
conversation.
We
believe
that
for
strategic
innovation,
the
reliance
on
the
judgment
of
senior
executives
or
scientific
gurus
is
dangerous,
because
their
intuition
is
derived
from
the
existing
business.
It
helps
to
apply
decision
meeting
facilitation
tools
that
interrupt
the
normal
way
in
which
administrative
or
technical
seniority
tends
to
bias
the
outcome
of
conversations.
3. Portfolio
management:
Startups
generally
have
one
offering
and
one
business
model
to
manage.
Large
companies
have
many
offerings
and
possibly
multiple
business
models.
Large
companies
are
not
just
making
decisions
about
one
opportunity
but
must
make
‘go/no
go’
decisions
across
multiple
opportunities
on
an
ongoing
basis.
Inovo
uses
an
objective
assessment
tool
to
rate
opportunities
along
two
axes
–
‘should’
we
do
it
and
‘could’
we
do
it
–
to
assist
in
making
portfolio
decisions
across
multiple
opportunities.
We
also
use
opportunity
canvases
to
monitor
and
maintain
a
healthy
distribution
of
opportunities
across
the
white-‐space
domains
that
we
are
exploring.
As
with
the
business
model
caveat,
this
organizational
caveat
does
not
invalidate
the
application
of
TLA
for
large-‐company
strategic
innovation.
However,
it
does
mean
that
TLA
alone
is
probably
inadequate.
In
summary,
having
a
Lean
set-‐based
approach
such
as
TLA
is
an
important
component
of
a
large-‐
company
strategic
innovation
capability,
but
it
is
not
sufficient.
It
requires
an
appreciation
for
the
wide
variety
of
business
model
types
that
exist
in
the
world
and
the
need
to
tailor
one’s
Test
&
Learn
program
with
the
proper
methods
and
tools.
It
also
requires
additional
methods
and
tools
to
allow
TLA
to
be
effective
in
a
complex,
large-‐company
setting.
The Innovation Quarterly (Fall 2013) | 6
