It's a Product Manager's perspective to building AI/ML products.

1. Being a Data Science PM
Ram Narayan Subudhi

2. Agenda
● My Journey as a PM
● Introduction to Data Science
● Applications of AI/ML in E-commerce
● Journey of an AI/ML product
● PM Responsibilities
● When to use AI/ML
● Mini Case Study
● Challenges & learnings
● Q&A

3. Who am I?
Joined
GlobalScholar
(EdTech)
Built e-learning
applications
2011
2017
Started working on
engagement constructs
UGC, Wishlists &
Collections
Graduated as a
CS Engineer
Joined Amdocs,
built Billing & CRM
software
2009
Started my MBA
from IIM Bangalore
Majors in Finance
2013
Moved to the
Marketplace Team
(Seller side)
Leading Selection
Design
2019
Started my PM
journey at Flipkart
Led Catalog
Platformization
2015
● Engineer turned Product Manager
● Experience across domains - E-commerce, EdTech, Telecom
● Experience of building both consumer facing products as well as platform products
● Have worked extensively with Data Scientists to solve for key customer & business problems at scale

4. My DS Journey
Identifying fraudulent
activity
(Building intelligence)
Auto answering
of customer queries
(Creating customer
delight)
Content Quality &
Ranking
(Building
intelligence)
Demand - supply
gap analysis
(Solving for
business needs)
Feedback
summarisation
(Creating customer
delight)
Selection
Benchmarking &
Assessment
(Solving for business
needs)
Auto moderation of
content
(Automation)
Highlights -

5. Data Science is Everywhere today

6. ● Vast & complex
● Fast evolving
● Lot of misconceptions
Common Queries
● How much of Data Science should a PM know?
● What are the artefacts that a PM produces while building a data science product?
● How much time does it take to build a Data Science product?
● What does your typical day look it?
● How does data science form a part of product strategy?
Introduction to Data Science

7. Applications of AI/ML in E-commerce
Stakeholders Needs04
● Monetisation - Ads Platform, Brand insights
● Marketing spends optimisation
● Private Labels Design
Fulfilment Journey03
● Procurement - vendor selection, lot management
● Warehouse - inventory placement, w/h space optimisation, w/h automation (bots)
● Logistics - route planning, delivery SLA prediction & optimisation
● Reverse logistics - returned product grading
● Fraud prevention - theft management
Seller Journey02
● Onboarding - lead generation, document verification
● Listing - catalog enrichment, listing quality measurement
● Inventory mgmt - demand forecasting & planning
● Growth insights - listing ads, pricing recommendations, selection insights
● Seller support - chat assistant, ticket allocation & prioritisation
Buyer Journey01
● Discovery - home page optimisation, search, recommendations
● Decision making - catalog (size reco), UGC (moderation, feedback summarisation)
● Checkout - address verification, reseller identification, payments (COD eligibility)
● Post order - customer support (chat bot), fraud detection
● Core services - user profiling

8. Journey of an AI/ML Product
Problem
Definition
Define the business
problem
Formulate
hypotheses
Translate it into one
or more DS problem
statements
Identify the right
metrics & establish
clear success criteria
(PM)
Data
Exploration
Identify/create the
underlying datasets
Identify feature sets
(domain knowledge
comes in handy)
Identify different user
segments, corner
cases, etc.
(PM/DS)
Modelling &
Optimisation
Modularise
Explore various
models/techniques
Train the models &
iterate
Measure the model
metrics, make
tradeoffs
Validate through
business/ops
(P - DS, S - PM)
Scale up &
Maintenance
Enable logging &
debuggability
Setup alerts &
dashboards for health
metrics
Perform periodic
checks to identify the
need to retrain the
models
(P - Engg, S - PM/DS)
Deployment &
Experimentation
Integrate with the
core Tech stack
Implement a fallback
flow & have feature
flags
Instrument the
necessary data
signals
Perform an A/B
experiment
Design the UX
(P - DS/Engg, S - PM)

9. Problem Definition - Mini Case Study
Aspect Ratings
● Define the business problem & the product to be built
○ User Need - to research about the product & its features
○ Business Problem - enable faster & convenient decision making for
the user
○ Target product - summarise customer feedback at an aspect level
● Translate it into DS problems
○ Identify relevant aspects automatically
○ Tag feedback to aspects
○ Grade feedback from positive to negative
○ Summarise feedback at product level
● Business metrics vs. DS metrics
○ Business metrics - engagement, conversion
○ DS metrics - coverage, accuracy
● Challenges
○ Linguistic complexities - incorrect grammar, use of hinglish, etc.
○ Optimising for category nuances

10. Applying your domain knowledge - |
● Identify relevant feature sets
○ Proxy data
○ Signals from the broader ecosystem
● Handling data anomalies
○ Filtering out noise
○ Handling data quality issues
● Ensuring data quality
○ Check the checker flows
● Critique the Models
○ Question assumptions
○ Play the devil’s advocate
○ Test out both happy & unhappy flows

11. Applying your domain knowledge - ||
● Make the right trade-offs
○ Accuracy vs. interpretability
○ Precision vs. recall
● Identify commonalities across products
○ Modularise & re-use

12. 1. Filtering out profanity from user generated content - what do you prioritise?
a. Precision
b. Recall
2. Aiding law & court proceedings - what do you prioritise?
a. Accuracy
b. Explainability
Examples - Tradeoffs

13. Capture user preferences
● Design your onboarding
experience
● Ask explicitly
● Allow users to update their
preferences
● Allow users to blacklist
Create feedback loops
● Validate your output regularly
● Respond to new data
User Experience Design - I

14. Collect data intelligently
● Make it playful
● Solve a customer need
Communicate effectively
● Build trust with the user
● Tell the why part
Interact naturally
● Make it look & sound humane
● Simulate interactions & review
User Experience Design - II
Google Duplex phone calls -

15. Examples - Whether to use AI/ML?
1. Shortlisting resumes for a job profile?
a. Yes
b. No
2. Taking decisions during a medical surgery?
a. Yes
b. No

16. When to use AI/ML?
Guiding principles:
● Recurring needs which are too costly or time consuming to do manually (eg. content moderation)
● When rules are not enough - either they are too many or they are too complex to define objectively (eg.
address intelligence)
● Scale of data is huge to analyse & predict (eg. recommendations)
● Underlying data keeps changing over time (eg. user preferences)
How to get started:
● Manual -> Rules -> DS
When not to use DS?
● Rules work reasonably well
● Mission critical systems with no scope for errors, where decisions are irreversible
● Explainability is crucial

17. Mini Case Study
Recommendation Engines & Personalisation
● Why do we need AI/ML?
○ Too many choices
○ Too many users with varying preferences
● Dimensions of Personalisation
○ Level: No -> Cohort level -> User level -> User * context
level
○ Aspects: language, genre, content format, etc.
● Data Inputs
○ Implicit signals - browse/watch history, completion rates
○ Explicit signals - selected interests during onboarding
● Context
○ User context vs. session context
● Techniques
○ Collaborative Filtering
○ Facet Similarity

18. ● DS is non-deterministic
○ Stakeholders want predictability
○ Predicting the likelihood & extent of success is almost
impossible in the beginning
○ Start small, iterate and scale up
● Business decisions
○ Build vs. buy vs. license
○ TTM, strategic importance, skill availability
○ Time bounding the research & development process
● DS models lack explainability
○ Prioritise between accuracy & explainability
○ Communicate & create transparency
○ Leave scope for manual overrides
● User data is no more secure
○ Put the control in the hands of the user
○ Anonymise the data
○ Build organisational firewalls to restrict access
○ Communicate the benefits of using the data
Challenges & Learnings

19. Examples - PM Decisions
1. You built a feature using AI/ML. The model has a pretty high accuracy of 95%. The feature when
launched led to a drop in conversion. What you do?
a. Launch the feature
b. Kill the feature
c. Reimagine the feature

20. Q&A
Thank You