Product Management in the Era of Data Science

1. Product Management in the era of Data Science

2. Mandar Parikh @mandarparik h VP Product & Engineering @ Entytle Ex- Aviso Zuora Salesforce

3. Today’s Webinar • What does it take to build killer products in the “AI-first” era? • What makes for a great Data Science-driven product and how do great Product Managers leverage Data Science to drive value for customers? • Find out how to avoid the pitfalls of hype-chasing Data Science tactics. • Learn how to work with Data Science and Engineering to build a compelling product and solve real problems. Product Data Science Engineering

4. Agenda 1. What is Data Science? What are Data Science-enabled Products? 2. A Mini-Primer on Data Science 3. Designing Products with Data Science: Strategies for Success 4. Data Science Pitfalls 5. Engaging with DS Teams 6. Product Marketing & Data Science 7. Data Science Essentials for Product Managers

5. 1. What is Data Science? What are Data Science-enabled Products? 2. A Mini-Primer on Data Science 3. Designing Products with Data Science: Strategies for Success 4. Data Science Pitfalls 5. Engaging with DS Teams 6. Product Marketing & Data Science 7. Data Science Essentials for Product Managers

6. What is “Data Science”? • Not really science: not falsifiable, often untested, no explanatory power • An approach that analyzes large amounts of data (where large is relative) and extracts patterns and insights and makes predictions from it, to impact some business driver • New term for statistical techniques that have existed for decades

7. • Data Science is at the “core” of the offering; differentiating IP • Tons of examples • LinkedIn • Uber • Salesforce • I won’t talk about other applications of AI including NLP, Image Recognition et al. 1 What are “Data Science-enabled Products”?

8. What are “Data Science-enabled Products”? LinkedIn • Early adopters of Data Science • Graph search • Skills • Reputation • Recruiting • “People You May Know” 2

9. What are “Data Science-enabled Products”? Uber • Driver Positioning • Ride Demand Forecasting • Surge Pricing • Estimating ETA 3

10. • Started as a Contact Manager, no Data Science • Salesforce Einstein • “Provides Insights to Users” 4What are “Data Science-enabled Products”? Salesforce

12. Data Science Basics • Features & Feature Selection • ”Art of Data Science is to figure out which features to use” • Unsupervised Learning • “What patterns exist in the data?” • Supervised Learning • “Make predictions using existing patterns in the data” • Popular algorithms: Random Forest, Gradient Boosted Machines, Kernel Regression, Logistic Regression, Nearest Neighbors, Clustering • Deep Learning

13. A Word On Machine Learning • DS, ML and AI used interchangeably • I am not here to settle the debate, but I think of it as: • DS model extracts patterns and makes predictions • ML automatically calibrates the model and improves predictions over time

14. Different Types of Analyses Descriptive: What happened? 01 Predictive: What will happen? 02 Prescriptive: What should I do (to achieve some outcome)? 03

15. DESCRIPTIVE ANALYTICS

16. PREDICTIVE ANALYTICS

17. Prescriptive Analytics Price Optimization 1 Expert Systems / Decision Support Systems 2 What-if Systems 3

18. Accuracy v. Recall v. Precision v….. Business Metrics N=170 Predicted: YES Predicted: NO Actual: YES 70 5 Actual: NO 45 50 Accuracy: How many times were you right? (70+50)/(70+50+5+45) = 70% Recall: Of all the positives, how many times were you right? 70/(70+5) = 93% Precision: How many times did you correctly predict YES? 70/(70+45) = 61% But none are Business Metrics!

20. Product Management Fundamentals Don’t Change • Customer Empathy • Data-driven Salesforce Lead Scoring Tell me why a lead is an 83 v. #1

21. Cleanliness is Next to Godliness • The data you get will not be clean – guaranteed • Data cleansing is the single best thing you can do for your data, beating incremental improvements you’ll get with a ”better” model • Cleaning data is HARD! Don’t underestimate the effort involved. #2

22. • Don’t over think it, and don’t let your Data Scientists tell you otherwise • Start simple; you might be surprised how far you can get • Don’t get stuck in Analysis – Paralysis; be skeptical of incremental improvements #3The Best Model is Often the Simplest One

23. Get the AI Out of the Way Opportunity Score Opportunity 42929 48% Opportunity 23427 68% Opportunity 55620 23% Opportunity Score Opportunity 42929 Medium Opportunity 23427 High Opportunity 55620 Low #4

24. Make it Actionable Opportunity Score Why? Opportunity 42929 Medium Time-in-stage, Meeting/email sentiment Opportunity 23427 High High level of activity, External factors Opportunity 55620 Low Close date pushed AKA Interpretability #5

25. Your Users Are NOT Data Scientists No need to show the confidence level or margin of error Users like plain-speak explanations. #6

26. Design is AI’s Best Friend Cadence is key Progressive Disclosure Drill-down Flexible & uncluttered Trend arrows are great Name the segments #7

27. • Less is more • Show key insights, skip the details (but be able to drill-in) • Articulate the insights, don’t keep the user guessing • Anomaly Detection is powerful at scale and is great for ”management by exception” • If you’re using Clustering, it’s often insightful to expose the output to managers • Managers love Prescriptive Analytics; consider What-If modeling http://www.tatvic.com/blog/wp- content/uploads/2017/01/fetured.jpg Prescriptive Analytics Anomaly Detection #8 Managers’ Deserve Less!

28. Tools ■ BITools LikeTableau, Domo ■ Excel, R, Jupyter for quick prototyping ■ D3.js – very popular for building rich, interactive visualizations Techniques ■ “TheVisual Display of Quantitative Information”, by EdwardTufte ■ … Don’t Visualize Data: Tell Stories #9

29. Be The QA You Want To See KPIs & Metrics 01 Backtesting 02 #10

30. Backtesting: The Basic Idea 1. Take a big slice of valid data 2.Split into training and testing data sets a. Training data set should be several multiples of “business cycles” b. Testing data set should probably be a few “business cycles” 3. Train the model on the training data set 4.Run the model on the testing data set and generate predictions over the testing period 5. Compare actuals to predictions in the testing period

31. Backtesting: Motivations • Basis for comparison for new stuff from Data Science; baseline for measuring marginal value added • Regression test for Engineering • Quantifiable quality test for Product & Customer Success • Aid in Sales and Marketing

32. 1. What is Data Science?What are Data Science-enabled Products? 2. A Mini-Primer on Data Science 3. Designing Products with Data Science: Strategies for Success 4. Data Science Pitfalls 5. Engaging with DSTeams 6. Product Marketing & Data Science 7. Data Science Essentials for Product Managers

33. All Big Data Is Not Equal • Particularly in Enterprise systems, the amount of data is not “Big” • Can make useful predictions or provide useful insights on relatively small amount of data • E.g.Years worth of Opportunity Data <<A month’s worth of Censor Data • More data does not necessarily mean better predictions • E.g.Very old data may be “out-of-date” and may impact accuracy negatively

34. • You don’t need Data Science ALL the time • Don’t shy away from a “simple" solution just because there is no DS involved Data Science is NOT a Silver Bullet

36. Engaging with DS Teams • Integral part of the Product & Engineering Team • Much of Data Science is (also) Software Engineering • Should be part of the “normal” product cadence and processes • Bring in Data Science early into Customer conversations (a la Design) • Drive use-case-based initiatives with Data Science 1

37. Engaging with DS Teams • Help DS understand and define the problem • Help with the art, leave the science to DS • Help DS understand the context, assist in feature-selection • DS picks the algorithms • Much like API design, agree on the contract: inputs and outputs • Don’t forget the UX part of Data Science • Reinforce Business Metrics 2

39. • As the spokesperson for your product, learn to speak Data Science • You will have to grapple with: • When to mention DS and when not to? • When you speak about DS, how much do you give-away? • How do you explain algorithms to the lay user? • Understand your audience, understand the context 1 Product Marketing with Data Science

40. Context • Consumer App: “Powered by sophisticated AI” works • Business App: “Powered by sophisticated AI” could back-fire with users who have been following a process for decades 2 Product Marketing with Data Science

41. What to say & How to say it • Check with your DS team and Legal/IP on what NOT to say • General approach to solving the problem is probably okay • IP is usually in the end-to-end solution, not the choice of model • Stick with simple explanations that your grandmother could understand • Be prepared to back up your claims (proof-points) 3 Product Marketing with Data Science

43. Data Science Essentials for Product Managers • A background in basic statistics is an absolute must • Pick up a book with a “high-level” intro to Data Science; my favorites are: • Numsense! Data Science for the Layman: No Math Added; by Annalyn Ng, Kenneth Soo • Data Science for Dummies, Lillian Pierson • An “Intro to Data Science” online class could be useful; here’s one: Intro to Data Science (Udemy) • Unlike Balsamiq or Sketch for UX Design, there are no “starter” toolkits for Data Science-related prototyping • Understanding Jupyter (or similar) output (metrics, curves, distributions) is a good skill to develop • Excel or analytic tools like Looker are great arsenal to have • Ability to collaborate closely with your Data Scientists is key

44. Thank You @mandarparikh

Product Management in the Era of Data Science

Product Management in the Era of Data Science

Recomendados

Más contenido relacionado

Was ist angesagt?

Was ist angesagt?(20)

Similar a Product Management in the Era of Data Science

Similar a Product Management in the Era of Data Science(20)

Último

Último(20)

Product Management in the Era of Data Science