The document discusses how companies can use data and experimentation to improve consumer products and business metrics. It recommends that companies (1) collect extensive customer data, (2) instrument all customer touchpoints to measure key metrics, and (3) analyze the data to identify correlations and opportunities for improvement. Regular experimentation is important to continuously innovate and optimize the customer experience.

1. MONITOR, MEASURE, EXCEL
DATA & CONSUMER
PRODUCT DEVELOPMENT

2. WHO AM I?
JIYO NATURAL
FOUNDER AND FOODIE
BAHUTBADHIYA LABS
CHIEF CONSULTANT
PAST
FLIPKART
ENGINEERING FOR CUSTOMER
AND MARKETING PLATFORMS
RETREVO, KOSMIX,…

3. OUTLINE
• Data Driven Development?
• Knowing and monitoring your customer
• Measure everything that makes your offering (brand,
product, technology, hiring, etc.)
• Learnings from building an experimentation platform

4. “Consumers are increasingly recognised as
important co-developers of innovations solving
unforeseen problems and demanding innovative
solutions.”

5. DATA DRIVEN DEVELOPMENT
• If you are building a consumer product, align your
efforts with what consumers want.
• How?
• Use data to make informed decisions
• Use data to evaluate efficacy of products

6. WHERE IS DATA
• Data is everywhere, collect everything you know about
your customer, brand or the product
• Build/Buy capabilities to store and analyse all this data
What you can control
What you observe
Identify patterns/anomalies

7. HOW DO WE USE DATA
• Identify what is the goal
• Identify data points that are measurable
• Connect measurements to know how you are doing on
the goals

8. MONITOR/MEASURE
for a typical web product

9. KNOW YOUR CUSTOMER (1)
• Demographics: Age, Gender
• Location : City, Home/Office
• Acquisition Channel: Direct, Paid, Organic, Social,
Affiliate
• Device: Web, Mobile, Browser Type
Multiple other raw/derived attributes

10. INSTRUMENT TOUCHPOINTS (2)
• Identify and instrument opportunities to collect user
data
explicit :
feedback,
ratings, etc.

11. Implicit : CTRs, time spent, bounces
Usually through
Javascript
embedded on
the page
Beacons sent to
backend server
• Collect aggregate statistics : Google analytics, Omniture
• Collect request level data : Usually home grown
platforms

12. IDENTIFY CORRELAT I O N S ( 3 )
• Identify control variables that drive particular user
responses (shipping date accuracy, widget placement,
logistics provider)
• can even be differences on the tech stack - e.g. a
faster API, or a new search algorithm used
• Connect and correlate with results (ratings, CTRs, sales,
time spent, etc.) to identify what works or what users
like/don’t like

13. Adding review summary increases the add to cart rate !

14. LETS SEE MORE
EXAMPLES

15. NPS : Tool to gauge the loyalty of a firm's customer
relationships
How likely is it that you will recommend our company to a friend or a colleague?

16. • Collect this information across all or representative set of your customers
• You are doing great if you are above 60%
• Identify segments where NPS is low, drill against a set of control
variables
CATEGORY NPS
BOOKS 65
ELECTRONICS 45
CATEGORY NPS
LARGE
APPLIANCES
30
CAMERA 60
MOBILES 61
COMPUTERS 65
Electronics is not performing well Something going wrong with appliances
Large appliances have different delivery/installation challenges

17. CITY NPS
METROS 60
T I E R - 2 45
LOGISTICS NPS
SELF 70
3RD PARTY 35
Something going wrong in smaller towns Third party deliveries suck
Higher proportion of deliveries in smaller towns is through 3rd party

18. AFTER THE HOLIDAY SALE

19. FOOD QUALITY
• How do you know if people
liked a particular item today?
• Are there specific taste
preferences of an individual?

20. • Ask questions (surveys, profile preferences)
• Most don’t have incentive to do this
• Look for sales data if it’s a cash and carry
• Users don’t know the taste before buying
• Allow them to submit feedback
• Usually biased towards those that did not like

21. Better still - look at leftovers !
• Isn’t this what our moms did? :-)

22. HIRING FUNNEL
• Goal: Recruit the best in less time
• Method: Create a recruitment funnel and obtain
data at each stage
Candidates Interviews Offers Hires

23. • Monitor and act on anomalies
• Slice & Dice on multiple control variables (roles, consultant)
Table 1-1
Role Resumes
Received
Screened Interviews Offers Hires
SDE 82 40 23 15 10
Tech Lead 31 14 10 3 2
Manager 19 16 12 3 2
Architect 9 5 2 1 0
Data Scientist 2 0 0 0 0
58% 65% 67%
Candidates Interviews Offers Hires
Low number across roles indicates
scheduling, branding issues
Low numbers indicate either of
- problem with hiring bar
- need for additional gating : e.g. online
coding
- quality of candidates ingested
Low number across roles indicates
salary, role clarity, engagement
issues

24. ECOMMERCE FUNNEL
• Goal: Get People to Buy more on the site
• Method:
1. Break the buying process into stages (think of each
stage as a milestone culminating finally in an order)
2. Measure & Maximise the number of users reaching
any of these stages
3. Identify factors that drive transitions from one
stage to next (also ways of leapfrogging stages)

25. Landing Product Selection Cart/Wishlist Checkout Order
CS contact
Submit Reviews
Repeat
60%
15%
80% 60%
15% 40%
Healthy Funnel : Conversion (Landing to Order) is High

26. USES OF THE FUNNEL

27. IDENTIFY FUNCTIONAL CHARTERS
A. Traffic Acquisition
• Get more users with high conversion
B. Discovery
• Users find what they are looking for
C. Product Details
• Sufficient Information to decide fitment
D. Checkout
• Clutter free with no distractions

28. DISCOVER
Get users to find product they want quicker
Browse into
relevant
categories
Most popular
products

29. CHECKOUT
Increase the ratio of cart to order
ExpressBuy to
reduce number of
steps

30. PRODUCT DETAILS
Get user to commit fast Viewed ended
up buying
Review summary
at the top

31. DETECT ANOMALIES
• Users follow different paths (hence different funnels)
• Anomalies can help uncover issues and areas of
improvement
E.g. low conversion for a particular affiliate channel
Low cart additions from Safari browser post a release

32. IDENTIFY GAPS IN EXPERIENCE
• Funnels looking very different across categories
• E.g. Watches: Footwear/Watches — higher volume on
product selection
Cluster similar
products on sizes
and colors

33. Show available
sizes and color
on the browse
pages

34. FUNNEL CONTINUES BEYOND THE
WEBSITE
Customer support : at one point very large number of
customers would call immediately after placing an order
• Analyze and segment the call conversations
• Large number of anxiety calls
• Proactive updates necessary

35. EXPERIMENTATION
PLATFORM

36. • Anyone building a consumer product needs
to have an experimentation platform
• Otherwise you are running blind
• Rapid experimentation is the key to rapid
innovation

37. BUILDING BLOCKS

38. INPUT, OUTPUT VARIABLES
• Ability to create user buckets
• Ability to segment the users consistently (A/B, cohorts on website,
shipping warehouse on backend)
• Control Variables
• Things that you want to vary in the experiments (search algo,
widget placement, shipping partner)
• Result Variable
• Goals that you are trying to drive - (sales, engagement, user
signups)

39. INSTRUMENTATION
• Add rich instrumentation both on your site and your
backend
• Capture all variables, context, request/response properties
• Have a way to connect everything to current request
• Need a crisp and correct definition of the touchpoints
• User session
• Visit attribute for a traffic channel

40. MANAGEABILITY CONTROLS
• Usable tools/interfaces to create and monitor the
experiments
• Integration with various result/performance
dashboards in the company
• browser plugins to put yourself into an experiment
bucket

41. BACKEND STORAGE/ANALYTICS
• Store all your data, possibly as much of it realtime
• Pipe data points for various experiments to the funnel
you use
• Identify when an experiment has achieved statistical
significance
• Ability to slice and dice on demand

42. SOME LEARNINGS

43. UNDERSTAND THE DATA YOU MEASURE
Search : CTR
as a measure
of search quality
• Sometime’s clicks can be bad and no clicks can be good
• Account for corner cases (e.g. users opening results in
Tabs)

44. MEASURE AND MONITOR RELEASES
• Look out for anomalies post a release (usually quick)
• For changes that take time always do an A/B to understand
if things have been made better or worse
• Some times you will find that despite what you do sales
don’t drop (can happen in the current ecosystem in India)

45. USE EXPERIMENTATION FOR MAJOR
TECH ROLLOUTS
• Do an A/B even for non functional rollouts
• checkout re-architecture
• rolling into the new supply chain backend
• Functionality finds convoluted ways of hiding in code

46. MEASURE WHAT USERS SEE
• Important to measure what your users are seeing (End
user latencies)
• as opposed to server side
• as opposed to our agents sitting in different
geographies
• You’ll be surprised how external factors can affect your
performance

47. PACE OUT THE CHANGES
• Kill the dropdown : Avoid big bang releases

48. INCREMENTAL STEPS
• Default to all items
• Rich autosuggest
• Suggest categories and stores in item results
• Search bars on category and store pages

49. NOT EVERYTHING CAN BE TESTED
• There is engineering cost to implementing a feature.
• Use experiments to validate hypothesis
• Prefer incremental redesign and plan on being wrong

50. RUN A LIMITED # OF TESTS
• Created a pool of A/B buckets, re-use once the
experiment is over
• browser plugin to assign oneself to an A/B bucket
• Have well defined exit criteria before an experiment is
admitted

51. FINALLY : D ATA CAN BE MISLEADING
• Sample set may not be representative
• e.g. price elasticity/shipping charges have different
effect on different bands (head, body, tail, etc.)
• Interpretation of the data is key, many times further drill
downs are necessary

52. THANK YOU!