Product Camp Silicon Valley (March 31st, 2018) Prasad Velamuri

1. Managing AI Products
Prasad Velamuri

2. Prasad Velamuri
• AI Product Fellow @ Insight Data Science
• Product Manager (+ Co-founder) @Voicy.ai
• Product Manager @Cisco
• Technology Strategy @Samsung
• Product Manager, SW Developer @Juniper

3. Agenda
• Intelligent Systems
• OODA
• AI/ ML/ DL
• AI Product Management
• Strategy
• Analyze, Decide, Build –––
• Influence across matrix
• Other considerations
• Driverless AI
• Monetizing AI

4. Observe
Orient
Decide
Act
Intelligent Systems
LEARN

5. Decision Making
Insight:
Speeding through the loop
is more important than
quality of the decisions
Sir John Boyd
Distinguished
fighter pilot,
developed military
theories in ‘60s

6. The field of AI

7. Economic Value Created by AI
99% of the EVC by
AI today is through
Supervised Learning
Input Output
Picture Is it you or not
Loan application Will you repay (%)
Ad User Will the user click?
Speech Recognition Text Transcript
Translation (English) French
Image/ Lidar Position

8. AI Product Management
Why is it different?
•Non-deterministic Product (F1 score)
•Atypical Product Testing
• Output changes with use
•Data science is not engineering
• AI models aren’t like databases
• Significant time spent on data prep
•Still very ‘Researchy’

9. Strategic AI Product Manager
• Corporate Strategy
• PM should drive the Emergent strategy
• Few companies adopting Deliberate strategy
• Data Strategy
• Centralized and secure
• Avoid inaccurate, incomplete, out-of-date
• Kickstart “Data network effects”

10. Data Network Effects
Data network effects occur when your
product, generally powered by machine
learning, becomes smarter as it gets more
data from your users.
http://mattturck.com/the-power-of-data-network-effects/
Network Effect
+
Data Network Effect

11. Strategic AI Product Manager
• Corporate Strategy
• PM should drive the Emergent strategy
• Few companies adopting Deliberate strategy
• Data Strategy
• Centralized and secure
• Avoid inaccurate, incomplete, out-of-date
• Kickstart “Data network effects”
• Data acquisition strategies

12. Data acquisition strategies
• Manual work at least till data network effect kicks in
• Crawling the web (e.g., text summarization/ simplification)
• Narrow the domain (e.g., vertical chatbots)
• Crowdsourcing/ Outsourcing (e.g., Crowdflower, Amazon Mechanical Turk)
• Gamification/ Incentivizing user-in-the loop
• Data capture SDKs in third party apps
• Build “Data trap” (create/sell something valuable to gather data- Tesla?)
• Publicly available datasets
https://www.kdnuggets.com/2016/06/10-data-acquisition-strategies-startups.html

13. Strategic AI Product Manager
• Corporate Strategy
• PM should drive the Emergent strategy
• Few companies adopting Deliberate strategy
• Data Strategy
• Unified data warehouse and secure
• Avoid inaccurate, incomplete, out-of-date
• Kickstart “Data network effects”
• Data acquisition strategies
• Get over “Cold start” problem

14. AI Product Management
• Analyze – What to build
• Decide – How to build
• Build – The building Process

15. Observe Product Trends in the AI Market
• Develop market insights and macro trends
• McKinsey Global Institute (MGI):
• Only 12% use cases progressed beyond experimentation stage
• Adoption limited outside technology sector
• Best-practice is to adopt agile test and learn approach
• Free research from MGI, Gartner, CB Insights

16. Follow trends in Applied AI research
• Your true competitive advantage
• Not from expertise in algorithms
• Ability to shorten time-to-market of products
• Have good handle on latest algorithm advances
• Andrej Karpathy’s arxiv-sanity summarizing latest research
• Follow influencers

17. Cut through hype- focus on practical use cases
• Have insights into practical use cases
• Identify problem
• Perception, Prediction, Personalization

18. Identify Problem
• Perception
• If a typical person can do a mental task with < 1 sec of thought, we can probably
automate it using AI now or in the near future (Andrew Ng, HBR, Nov 2016)
• Prediction
• For any concrete, repeated event that we observe, we can reasonably try to predict
the outcome of the next such event (Andrew Ng, NIPS 2016)
• Personalization
• Serving content desired by users in a personalized manner (Spotify/ Netflix)

19. Cut through hype- focus on practical use cases
• Have insights into practical use cases
• Identify problem
• Perception, Prediction, Personalization
• The PAC Framework to build use cases
• Automate, Classify, Predict
• Customers, Product, Operations

20. The PAC Framework*
Customers Product Operations
Predict • Which customer will buy
• Which user will churn
• Sales Forecast
• Infrastructure Usage
• Employee Attrition
Classify • Who might upgrade
• Micro segmentation
• Customer Input
• Bug Classification
• Manufacturing
Automate • Lead Generation
• Call Follow-Up
• Bug resolution workflows
• Product Training
• Operational Workflows
• Supply chain
* Rob May

21. Cut through hype- focus on practical use cases
• Have insights into practical use cases
• Identify problem
• Perception, Prediction, Personalization
• The PAC Framework to build use cases
• Automate, Classify, Predict
• Customers, Product, Operations
• AI hierarchy of opportunities
• Building on Maslow’s hierarchy of needs

22. AI hierarchy of opportunities*
Superpowers
for humans
Customer Service,
Conversation
Analytics
Retail self-checkout, supply
chain optimization, Pricing
predictions
Security, Predictive Analytics,
Autonomous Vehicles, eDiscovery
Agricultural monitoring, Disease prevention,
Medical Imaging, Smart Home, Geospacial
Analytics, Drug Discovery
* Ankit Jain
(Gradient Ventures)
Transcendence
Esteem and Education
Operational Efficiency
Safety Needs
Physiological Needs

23. Cut through hype- focus on practical use cases
• Have insights into practical use cases
• Identify problem
• Perception, Prediction, Personalization
• The PAC Framework to build use cases
• Automate, Classify, Predict
• Customers, Product, Operations
• AI hierarchy of opportunities
• Focus on use cases that improve EBIT
• RoI, data network effects, data set, drift, tools required

24. Customer and Data obsession
• Customer obsession
• Going beyond product features & benefits
• Understanding meaning for customer’s jobs, their purpose, motivations and
the conscious choices they make
• Data obsession
• Being a champion of digitization while quantifying problems customers care
• Build comprehensive datasets needed for quality AI models
• Fetching data that reflects user’s jobs, behaviors & interaction patterns.

25. Build usable products with simple AI model
• Don’t be over obsessed with complexity of AI models
• Accuracy improvements vs user experience improvements
• AI MVP pyramid (adapted from Jussi Pasanen’s MVP pyramid)

26. • Be familiar with tools and techniques
• Influence AI Engineers, Data Scientists and Data Engineers
• API ecosystem that help serve end users
• Data ingestion tools (Kafka)
• Data processing systems (Spark)
• NoSQL DBMS (Cassandra)
• Commercial alternatives on AWS & GCP (cost structures)
• Avoid reinventing the wheel for commoditized AI techniques
Breadth first approach (Data/ Pipeline/ Model)

27. • Some crucial applications involve high liability
• Law, medicine and safety
• Output requires clear explanation for compliance purposes
• Use the approaches to explaining predictions from deep learning
• Eliminate Bias*
• Articulate organizational values of fairness and equality
• Communicate this to all employees (data scientists)
• Benchmark training data
• Validate algorithms periodically
Consider Model Explainability
* SAP Design Center

28. • Use validated learning loops for quick iterations
• Conceive use cases and map to capabilities of ML, Deep Learning
• Classification (Binary/ Multiclass)
• Regression (prediction)
• Clustering
• Universal approximation of Deep Learning
• Tie to a small set of metrics that matter
• Challenges of end-to-end AI models optimizing multiple objectives
• Agile deep learning
Iteratively build use cases with mapped AI models

29. Agile Deep Learning
Have a sub-loop of “Explore, Experiment and Prioritize”

30. • Gracefully handle low performance scenarios
Ensure product fails gracefully

31. • Technical language of AI researchers and data scientists.
• Artificial Intelligence, deep learning, machine learning — whatever you’re doing if
you don’t understand it — learn it. Because otherwise you’re going to be a
dinosaur within 3 years! (Mark Cuban)
Understand the fundamentals
Monica Rogati

32. AI Product Management
• Analyze – What to build
• Decide – How to build
• Build – The building Process

33. Influencing across the matrix
• Data scientists and AI Engineers
• Influencing-Up
• SCIPAB model
• Key Assertions based on realized benefits from AI Products
• Establish credibility
• Build Trust

34. Other Considerations
• Driverless AI/ Auto ML
• Automate laborious tasks- Feature Engineering, Model tuning
• Ensembling, Automatic cross-validation, Detect time-series
• AI Monetizing 101

35. AI Monetization models*
• Subscription models
• Freemium through monetizing data network effects
• Outcome-based
• Pay for the outcome (benefit) provided by the product/service
• Asset-Sharing
• Maximize utilization of product across multiple customers
• Revenue-sharing
• Sell product at cost, earn a percentage of client’s product sales
• Data monetization
• Product serves as a vehicle to collect and monetize quality data
• Win-win-win models
* Heiko Schmidt

36. Summary
• The current phase of AI is very promising
• Several opportunities to
• make elegant products that create tremendous value,
• delight customers and significantly transform the business.
• AI Product Manager is a catalyst in this transformation

37. Resources
• This slide deck available at bit.ly/managingAIproducts
• Blog covering salient points in this deck:
• blog.insightdatascience.com/moving-towards-managing-ai-products-5268c5e9ecf2
• Follow me:
• Twitter @prasadvsd
• Linkedin.com/in/pvelamuri