Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Netflix_Controlled Experimentation_Panel_The Hive

  • Loggen Sie sich ein, um Kommentare anzuzeigen.

Netflix_Controlled Experimentation_Panel_The Hive

  1. 1. Experimentation Panel 3-20-13Some Insights from NetflixExperimentation 1
  2. 2. Experimentation at Netflix Core to our culture Goal is to maximize our customers’ viewing enjoyment New and existing global members participate in multiple tests We experiment in all areas (personalization algorithms, product features, acquisition, streaming optimization, etc.) 2
  3. 3. Clarity on key metric(s) is critical  Netflix’s goal with our members: Continually improve member enjoyment  Retention  Netflix’s goal with our visitors: Optimize visitor experience to entice people to try Netflix  Free trial conversion 3
  4. 4. What about other great metricsthat you believe to be a positive measure? 4
  5. 5. Determining the appropriate use of a metric Predictive modeling (of core metric) Brainstorm Vet any “winners”potential metrics, with PMs and pastcollect new data experiments Productize successful metrics 5
  6. 6. Example ranking of some possible metrics 0.012 0.01Variable Importance Measure 0.008 0.006 0.004 0.002 0 6
  7. 7. Streaming hours is a key secondary metricVoluntary Cancel Rate 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 Customers‟ Stream Hours in the past 28 days
  8. 8. We predict customer tenure from streaming hours Probability of retaining at each future billing cycle based on streaming S hoursRetention at N days of tenure Total hours consumed during 22 days of membership Cume % in Test Cell Leverage the retention-hours curves above to measure the full distribution of hours in each test cell and predict tenure Streaming Hours 8
  9. 9. „Search-based Rows‟ Experiment 9
  10. 10. Percent of streaming hours from search-based rows 10
  11. 11. Filtered measurement Activity filtering: Filter to a subset of activity – e.g. streaming hours from one row  Controversial for decision-making; risk increases as the interaction potential (or cannibalization potential) increases Allocation filtering: Filter to a subset of members in the test – e.g streaming hours for the subset of customers who performed a search  Good for decision-making as long as: 1. The segment incorporates the full set of members who were exposed to the experience being tested 2. Segment is large enough to care about (or strategically important) 3. The segment holds up to a controlled experiment (members comprising the segment are not selected in a way that could 11 have been influenced by the test experience)
  12. 12. Unintended threats to controlled experiment  Engineering bug (A and B don’t work as intended)  Control cell is not engineered like a true test cell (“fixed”), and instead uses the standard production experience  Unplanned interaction with other experiments, campaigns, etc. that is differential across test cells 12
  13. 13. Experiment on minimum number ofqualifying titles in order for a “genre row” toappear 13
  14. 14. Negative results appeared immediately 14
  15. 15. Discovered that the test cells were notworking properly Cumulative distribution of page views by test cell Customers in the test cell using 15 as the minimum were seeing fewer rows altogether Number of genre rows on the page 15

×