Main Takeaways: - How to set the company for growth and success through KPIs - How to learn, iterate and grow through A/B testing - How to use the dashboard to focus and succeed with your product

1. JM Coaching & Training © 2020
www.productschool.com
Data-Driven Product Management by Shutterfly
Director of Product

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS
40 HOURS
40 HOURS

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

5. Data Driven
Product
Management
By Roman Levin, Director of Product @ Shutterfly
5
For Product School - February 2021

6. 6

7. Only 60+
slides to go...

8. Roman Levin
Director of Product @
Shutterfly
Linkedin:
www.linkedin.com/in/levinroman/
Email:
levinrom@gmail.com

9. Serial Entrepreneur
Software Engineer
About Me
Director of Product
@ Shutterfly
9
At Shutterfly for 7 years - moving from Engineering
to Product
?

10. My Entrepreneurial past
▸ Been in an Incubator
▸ One of the first Microsoft Accelerator
participants
▸ Many Many MANY failed startups...
10
…

11. 1.
AGENDA
1. Shutterfly
2. My team - APC
3. What is a PM
4. Defining KPIs
5. A/B testing
6. Dashboards
7. Q&A
11

12. Won’t cover
Strategy
Roadmap
UX
Working with Engineering
Working with Marketing
Communication
Empathy
Time Management
... 12

13. Will Cover
Thinking about & defining KPIs
A/B testing & iterations
Dashboards
13

14. 2.
BACKGROUND
14

15. Shutterfly
Shutterfly is the market leader in
personalized gifts in the US with
over $2.5B market value
(before COVID)
15
www.shutterfly.com

16. 16

17. APC Team
Automatic Product Creation
Combining Machine Learning, Web
and Mobile to create personalized
shopping and product creation
experience
17

18. 18

19. 3.
PRODUCT
MANAGER
19

20. 20

21. ▸ Understand Customers
▸ Set Strategy
▸ Releases
▸ Ideation
▸ Features / Prioritization
▸ And more...
21
Product managers are responsible for guiding the
success of a product and leading the cross-functional
team that is responsible for improving it

22. 22
Actually depends on the company

23. 23
You’re the
Voice of the
Customer

24. Secret power?
24
DATA
Defining, Collecting, Validating, Analyzing and making learnings

25. 25
B2C vs. B2B

26. 3.
KPIs
26

27. What is a KPI
A Key Performance Indicator is a measurable value
that demonstrates how effectively a company is
achieving key business goals.
▸ Product Level KPI
▸ Feature Level KPI
▸ Primary KPI
▸ Secondary KPI
27

28. SMART KPI (Goal)
Specific - Increase Clicks on X by 20%
Measurable - Can we collect the data?
Attainable - Can we actually achieve 20% increase?
Relevant - Are the clicks the correct KPI?
Time Constrained - By when to increase?
28

29. 29

30. Example for a good KPI goal
KPI - User Coverage
Goal - Improve user coverage to 80% in 2020
Specific - Increase to 80%
Measurable - We measure the UC KPI
Attainable - Believe 90% is possible
Relevant - More users = More impact
Time Constrained - End of 2020
30

31. Example for a Bad KPI goal
Increase Revenue Per Visitor (RPV)
Too far in the funnel
Has time delay (for eCommerce)
Hard to optimize in iterations
31

32. Your North Star KPI
All else works for that KPI
Examples:
- Daily Active Users
- Weekly Active Users
- Monthly Active Users
Maybe - Weekly Paying Customers?
Or… Daily Revenue?
Or number of installs?
32

33. APC Tile Example
3 Main Daily KPIs
- Coverage
- Project Start
- Load time
1 Main Weekly KPI
- % of total SFLY sales
1 Main Quarterly KPI
- Revenue 33

34. How to move a KPI?
(1) Goal - I want to improve Conversion
(2) Analyze - “Do we have supporting data?”
(3) Hypothesis - “I think X will move the needle”
(4) Define and Run a test
34

35. 35
Step 1
Step 2
Step 3
Step 4

36. 36
Visit page / experience
Engage
Add to Cart
Buy
Basic eCommerce funnel

37. 37
Visit page / experience
Engage
Add to Cart
Buy
Basic eCommerce funnel
Funnel
Conversion

38. 38
Where is the opportunity?

39. 39
Where is the opportunity?

40. 40
VS.

41. Example
Goal: “I want to improve CTR by 5%”
Analyze: Does Photo changes have correlation
with Click-through? (it does) (Graph)
Hypothesis: Improving Photo-selection will result
in better conversion
Test: Tap vs. Swipe to change
What KPIs?
41

42. 42

43. KPIs for the test
43
Total photo changes
Primary KPI
CTR, Unique Photo changes
Secondary KPI

44. 3.
A/B Testing
44

45. 45

46. What is an A/B test
Control and Treatment (Test)
Strive for statistical significance
Can be A/B/C/.. Test (why you shouldn’t)
Can be Multivariate test
46
AB testing is essentially an experiment where two or
more variants of a page are shown to users at random,
and statistical analysis is used to determine which
variation performs better for a given KPI(s).

47. 47

48. 48

49. Startups
Don’t do A/B/C/… or Multivariate testing
49
Analysis is harder
You need much more traffic
You need better systems to run the test

50. 50
1
5.5%
Conversion
NEXT
3
13.2%
Conversion
>>
4
7.8%
Conversion
>>
2
16.7%
Conversion
WINNER
Bad test design

51. Growth through testing
51
Come up with as many hypotheses as possible
Do many small tests
Iterate on success
Learn from failures

52. Proper test design
1. Hypothesis
2. Define what KPI you want to move
3. You should have a baseline and define a
treatment(s)
4. Define actions on possible outcomes
5. Calculate desired sample size
6. Run test until sample size reached
7. Observe results and confidence
8. Decide on next steps (see #4)
52

53. What is a StatSig result
A result has statistical significance when it is
very unlikely to have occurred given the null
hypothesis (wikipedia)
In human terms:
We’re confident if we do this test again, it
will turn out exactly the same
Confidence: 1 - p-value
53

54. How many users you need?
You would use an online calculator
Example: https://www.optimizely.com/sample-size-calculator/
54

55. How to define correct KPIs
What would happen if you measure the
wrong KPI?
Good test KPI
Simple
Test has direct effect
Is “close” to the test in the funnel
Relevant
55

56. Best Practices
(1) Have a clear and strong hypothesis
(2) Run an A/A test on your system
(3) Remove outliers, flippers and
unqualified users
(4) Test only ONE variable for A/B tests
(5) Run for enough time and include
enough traffic
(6) Holidays and Promotions - be careful.
Run specific tests
(7) Test to learn
56

57. 57
Example for an A/B test

58. Caution!
There is a price to pay for too much or too
little A/B testing
Too much - you move slower
Too little - you don’t learn enough
Run your tests when you have something
to learn or need to make a decision
58

59. 3.
Dashboards
59

60. Why you need a dashboard
Daily Dashboard - Health of business
Weekly Dashboard - More high level health
Deep Dive Dashboard - For analysis
Start with your North Star metric
dashboard only
60

61. What is a good dashboard
61
Tells you a story
Is not overwhelming
Allows to get insights and decide on next steps
Shareable
Explainable

62. Bad Dashboard
62

63. Good Dashboard
63

64. Tips (mainly for startups / teams)
64
Focus on your main KPI(s)
Look at it daily
Optimize everything you do for that KPI
Every other KPI is to improve your Primary KPI

65. 65

66. Keep in touch!
Roman Levin
levinrom@gmail.com
www.linkedin.com/in/levinroman/
66

67. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training