2. A bit about meâŚ
⢠Originally from Northern California, lived in Silicon Valley
for almost 10 years
⢠2007-2012: Silicon Valley Bank â financing early stage
startups all over world alongside top VC firms
⢠2012-2014: Received MBA, taught by Steve Blank (father of
Lean Startup), raised venture fund for student-run startups,
and taught self to code
⢠2012-2014: Worked for Okta (IPO), New Relic (IPO), Stripe
⢠2014-2016: Mixpanel (worked with unicorns on analytics
strategy)
⢠2016-2017: PM at Capital One (manager is VP Product for
Alexa)
6. Major Mistakes Companies Make
⢠Premature scaling
⢠Not taking the time to understand users
⢠Biased testing
⢠No clear plan
7. Premature Scaling
⢠Do not assume âI know more than the
customerâ
â You might, but if you are too smart for them, they
wonât pay you
â Taking the time to understand customers will
inform marketing, sales, design, and engineering
architecture
8. A Silicon Investor once said to meâŚ
⢠2 types of questions I get asked by
entrepreneurs:
â âHow do I find more paying customers?â
â âMy phone wonât stop ringing with people
wanting to buy my product. I canât get enough
people to close sales. Where do I find them?â
9. Assuming Everyone is Like You
⢠We replace actual understanding with
assumptions
⢠99.9% of the time, we miss critical facts or
donât get to the level of understanding needed
⢠You have to do the hard work to deeply
understand your users to build a product they
love (and there is no substitute for hard work)
11. Biased Testing
⢠Startups are a series of hypotheses
⢠We donât want to invest a lot of time building
on unproven hypotheses
⢠We know as a startup we need to move fast
⢠To test our hypotheses, we often focus on
data that confirms what we believe and
dismiss data that contradicts what we think
12. No clear plan
⢠Because we want our company to succeed,
believe in it, and need money, it distorts our
motivations
⢠We chase easy dollars without digging deeply
into âwhy?â
⢠Have to have a disciplined way of assessing,
prioritizing, and reducing risk
15. Advice for avoiding these mistakes
⢠Use user personas
⢠Learn how to set up tests
⢠Focus, then expand
⢠Have a clear plan and stick to it
We are going to discuss these, but I want
to put this in a framework first
16. What is Needed?
⢠A framework for mapping out our
assumptions
⢠Then, we can assess what our greatest risks
are
⢠Form hypotheses on these risks
⢠Systematically test these risks, build user
personas, and validate solutions
19. 3 Stages of Startup
⢠Stage 1: Problem / Solution Fit
â Do I have a problem worth solving?
â âYou canât leave here without me buying thatâ
â You should be interviewing at least 10 customers per week
⢠Stage 2: Product / Market Fit
â Have I built something people want?
â âIf this product were to go away, Iâd be very disappointedâ
â You should be interviewing at least 10 customers per week
⢠Stage 3: Scale
â How do I accelerate growth?
â âHow can I grow this business fast enough to keep up with demand?â
20. Process for Each Type of Risk
Product Risk: Getting product right ⢠First make sure you have a problem worth
solving
⢠Define smallest possible MVP
⢠Build and validate MVP at small scale
⢠Verify it at scale
Customer Risk: Building a path to
customer
⢠Identify who has pain
⢠Narrow this down to early adopters who
really want product now
⢠Build scalable inbound channel
Market Risk: Building a viable
business
⢠Identify competition through existing
alternatives and pick price for solution
⢠Test pricing by measuring what customers
say
⢠Then test pricing by what they do (i.e. paying
you)
⢠Optimize cost structure to make business
model work
21. Which stage are we in?
⢠What is the problem we are trying to solve?
⢠What are our assumptions that we need to
validate?
23. Bowling Balls Online
⢠$6 billion industry in US
⢠Separation between
traditional bowling alleys
(basic services) and
upscale alleys (fancy food,
drinks, dance music)
⢠Upscale alleys are trying
to appeal to younger
generation
24. User Personas
âBudâ
⢠50 years old
⢠Plays in bowling league with
friends for past 15 years
⢠At least 3 games per week
⢠Owns equipment
⢠Usually only purchases a
Coca-Cola on bowling nights
⢠Average annual spend: $300
âRickyâ
⢠27 years old
⢠Social activities with friends
including restaurants,
sporting events, gym
⢠New bowling alley in city
⢠Go 2x per year
⢠Buy food, drinks, etc
⢠Average annual spend: $150
We eventually want these to be 3-4 pages with LOTS of detail
25. What should we do?
⢠Bud
â Loyal, but not big spender
â Can we get him to spend more with us?
⢠Ricky
â Infrequent visitor, but spends lot of money
â Can we get him to visit more often?
26. Assumptions audit
⢠At least 2x per week using Lean Canvas
⢠What must be true for our business to work?
⢠What do we not have data to prove?
27. How to Validate
⢠As we go through Lean Canvas in order, we need to
define what are our assumptions
⢠Turn these assumptions into falsifiable hypotheses,
then start with the null
⢠We want to prove our assumptions wrong
⢠Example:
â For us to sell bowling balls online, people prefer buying
bowling balls at the bowling alley to going online
â Bud prefers his ball which he has had for a long time to
newer technologies; Ricky doesnât want a ball because he
bowls so infrequently
28. Testing
⢠Null Hypothesis: For us to sell bowling balls
online, people prefer buying bowling balls at
the alley to going online
⢠Focus on Budâs problem:
â Wants to play better, but doesnât have more time
to devote
â Wants to have a cool new toy among his friends
â Ball will have to last at least 15 years
âYour opinion, while interesting, is
irrelevantâ
29. On InterviewingâŚ
⢠Donât ask leading questions
â âWhat are some of your favorite cuisines?â
â NOT âDo you like Indian food?â
⢠Relevance
â 4 out of 10 (not leading) â statistically relevant
â 100 out of 1000 (leading) â statistically relevant
⢠Get to the âwhy?â
â Ask 5 times and donât assume you know
30. Tests and Experiments
⢠Maximize for speed and learning -> requires
us to focus on single metric or goal
⢠Validate qualitatively, verify quantitatively
Understand problem Define solution
Validate qualitativelyVerify quantitatively
32. Testing with Small Samples
⢠From Andy Johns: http://firstround.com/review/indispensable-growth-
frameworks-from-my-years-at-facebook-twitter-and-wealthfront
33. Controlling Variables
⢠Test one variable per time
⢠Run several tests quickly on different variables
⢠Always understand the size of your market when evaluating
test results
⢠Donât test multiple variables at once!!
⢠Example (4 different tests with 9 variants; null is 9th
variants):
â Customers: Bud / Ricky
â Channel: Email / Google Ads
â Solution: Bowl Master 1000 / Strike Maker 200
34. Focus, then expand
⢠In terms of controlling variables, we want to
understand a small segment of users
extremely well
⢠Validate this group by showing much higher
conversion rates
⢠Based on testing, we then expand once we
have extraordinary demand from initial
segment
35. Focus, then expand examples
San Francisco riders after events or bad
weather
Swedish music lovers who didnât want to
pirate music; by invitation-only
Busy, tech-savvy people in SF at the gym
who needed to display their status
36. Online Bowling Ball Business
⢠Letâs say through our testing, weâve identified
that we can sell the Bowl Master 1000 through
email to lots of âBudsâ
â Test this in single market (Milwaukee)
⢠There is huge demand for this and we canât keep
enough inventory in stock
⢠Now we want to pivot
â Geographic pivot (Chicago â 90 mins away)
â Channel pivot (Google ad sales focused in Milwaukee)
â Customer pivot (go after more âRickysâ)
â Solution pivot (try to sell Strike Maker 200 to âBudsâ)
37. Key Lessons for Focus, then Expand
⢠Use initial segment to work out kinks
â Make sure market is large enough
â Identify marketing messaging that works
â Identify channels that work
â Identify any product issues
â Understand what the retention mechanism is
â Figure out how to create barriers to entry
⢠Figure out what the challenges will be for scaling
larger
â Start to address these
â Understand what costs are and if scale will fix these
38. Summary
⢠Because this environment is resource-constrained, you
need to start with a plan and stick to it
⢠Use Lean Canvas to clarify assumptions
⢠Create user personas from interview data, not
assumptions
⢠Structure your tests to fight your own biases
⢠Start small, and expand outward