Main takeaways: -Learn how to instill customer obsession in your metrics -Tips for setting up an effective A/B test -Influencing partner teams with insights

1. www.productschool.com
Wielding Data to Drive Product
Improvements by Amazon Sr PM

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™
20 HOURS40 HOURS40 HOURS

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

6. WIELDING DATA TO DRIVE
PRODUCT IMPROVEMENTS
Corbett Hobbs
Senior Product Manager, Amazon

7. DREAM: DATA RULES EVERYTHING AROUND ME
Being comfortable with data provides…
• An analytical mindset
• Transferable skills
• The ability to influence

8. WHAT WE’LL COVER
1. Background
2. Grocery pickup 101
3. Creating customer-obsessed metrics
4. Setting realistic goals and prioritizing features
5. Designing pilots
6. Influencing through data

9. MY BACKGROUND IS IN POLITICS AND DATA
1. BSFS in International Politics from
Georgetown University
2. Press Intern in the US Senate
3. Sr. Product Manager at Graphiq
4. Knowledge Engineer at Amazon Alexa
5. Business Analyst/Sr. PM at Whole
Foods on Amazon

10. SO HOW DID I GET INTO GROCERY BUSINESS
MANAGEMENT?
• Wanted to explore other orgs at Amazon, especially in more ‘physical’ (i.e.
non-software) disciplines
• Started off as a Business Analyst delivering insights, but ultimately my goal was to move
into PM.
• Now as a PM, I help steer the business at the intersection of product, operations,
marketing, and launch

11. HOW DOES GROCERY PICKUP WORK?
ORDER
Customer places their order
online and picks it up at the
store (vs. delivery to their
home)

12. HOW DOES GROCERY PICKUP WORK?
CHECK IN
Customer can check in on their way to
the store to reduce their wait time

13. HOW DOES GROCERY PICKUP WORK?
PICK UP
In 95% of cases, associate brings
order out to the customer’s car and
loads it into their trunk (vs. walking
in to the store)

14. WHY DO CUSTOMERS LIKE GROCERY PICKUP?
• Cheaper than delivery: No tip, lower <$35 order fee
• Faster than delivery: Don’t need to wait for a delivery driver; can pick up as soon as it’s
ready
• Different item selection: You choose which store you pick up from (unlike w/delivery)

15. WE STAY CUSTOMER-OBSESSED BY FOCUSING ON
THE P95
• The 95th
percentile highlights the worst
customer experiences
• If you optimize for the P95, you keep the
overwhelming majority of your customers
happy
• For pickup, we track P95 as a time (e.g. 95%
of orders experience an onsite wait time of
less than X mins) and as a percent (e.g. X%
of orders experience a wait time within our
Y min P95 goal).
P95 as a time P95 as a %

16. WE USE ‘ENTITLEMENTS’ TO HELP PRIORITIZE
FEATURES
• Use your team’s KPI (e.g. PDOR) or a
common business metric (e.g. revenue) as
the basis for your calculation
• Estimate the extent of improvement via
heuristics and/or observational studies
• Define an achievable goal based on the
aggregation of your entitlements
• Prioritize roadmap based on entitlement
ranking (and level of effort)
Feature/Initiative PDOR
Improvement
(bps)
Estimated
Completion
Date
Add in-app map to
help associates locate
customers
+250 bps Q1
Move grocery staging
area to Front of House
+100 bps Q2
Estimate customer
ETA to store more
accurately
+75 bps Q3

17. EXPERIMENT OFTEN! BUT WHEN YOU DO,
CLEARLY DEFINE YOUR PLAN UPFRONT
- Rationale: why should we run this pilot
- Risks: why should we not run this pilot
- Proposal: what is this pilot
- Implementation: how should we run this pilot
- Expectation: what do we think this pilot will show us
- Evaluation criteria: what metrics will we track, and what will
they tell us. How do we define success?
At the end, review your results and make a recommendation

18. DATA FLUENCY HELPS BUILD INFLUENCE
- Dive deep to understand business drivers
- Ensure business needs don’t get lost in
translation with BI team
- Build credibility with peers
- Motivate partner teams without authority

19. RESOURCES THAT HELPED
DATA
- Vertabelo Academy: SQL Basics
- Dataquest: Python and SQL
- Real world practice
PRODUCT MANAGEMENT
- PM blogs and books (e.g. Decode &
Conquer by Lewis Lin)
- Industry news and articles
- Daily Google alerts about [insert
business here]

20. THANK YOU
/IN/CORBETTHOBBS

21. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training