Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.
Mining Member
Feedback to
Improve
The Customer
Experience
Nishant Hegde, Sr. Analytics Engineer
analytics @ Netflix
Data Engineering And Analytics
The “Content Quality” Problem
Buying Sourcing
Assets
Studio
Content
Partners
Encoding
Package
How is
our
Content
Quality?
FAIL
PASS
Encoding Video Hit Audio Distortion Hit
Some Examples of Content Quality Issues
● Value
● Easily Integrated
v1
v1
Event Producer
Ingest Pipeline
v1
Week 1 Bojack ugh
Week 1 Bojack Bojack is awesome
Week 1 Bojack No audio
Week
2
Bojack When is season 4
coming out?
Wee...
v1
Outcome
● Accuracy = 25% ● signal : noise
○ “ i didnt watch this piece of crap dumb
people call movie.”
○ “I DID NOT WA...
● Language Specific Models
● Daily Processing
● Integrate Twitter
v2
Local Language +
English Translated Models +
Separate Model for Tweets
v2
Day 1 Bojack ugh 0.7
Day 1 Bojack Bojack is
awes...
Measuring Performance...
v2
● Recall = TP / (TP + FN)
● Precision = TP / (TP + FP)
● Compliance = (TP + FP) / (TP + FN + F...
v2
Russian Dialogue IssueSome Big Finds...
Dubbed Audio Issue
Out of Sync Subtitle Issue
The Future ...
● Abandonment Time in Movie
● # of Seek Operations
● Average Time between Sessions
● Unsupervised Learning
...
plata o plomo?
Mining Member Feedback to Improve the Customer Experience by Nishant Hegde of Netflix
Nächste SlideShare
Wird geladen in …5
×

Mining Member Feedback to Improve the Customer Experience by Nishant Hegde of Netflix

1.339 Aufrufe

Veröffentlicht am

Abstract:-

At Netflix we strive to go beyond the user’s expectations for a streaming experience. A key component to achieving this is ensuring that the best quality of digital assets is made available. These assets are video, audio, subtitle and closed captioning files that collectively contribute towards the viewing experience. Having a rich catalog with the freshest content becomes inconsequential if a user experiences issues like the timing of the audio and video being off, or if the subtitles are positioned poorly. Moreover, asset quality can have a direct effect on member satisfaction and ultimately retention.

Netflix sets a high bar on content quality, and has a thorough Quality Control (QC) process in place to ensure that this bar is met. Our recent global launch has necessitated having a broad catalog with a wide variety of audio and subtitle languages across countries. In order to retain a lean operation as we scale, we based our QC process on a supervised model that predicts the likelihood of an asset having an issue. We then perform a QC only on assets that are predicted with a likelihood beyond a threshold.

Since we intelligently select what to QC and do not check every asset that goes on our service, there may be instances when a bad asset slips through. We needed a facility on the back end that could catch these undetected issues. Our member feedback channels were an obvious area to tap. Member feedback comes in two forms: explicit and implicit. Explicit feedback is received from sources like the “Report a Problem” section on the site, social media (Twitter, Facebook) and customer service calls. Implicit feedback can be derived from user viewing behavior, such as sharp drop-offs at certain points during the playback.

This talk will focus on

• How we mine explicit member feedback, in particular from the “Report a Problem” section on our site

• The challenges posed with identifying what is relevant due to the variety of context in the feedback obtained

• The analysis framework to monitor feedback and manage workflow

• Areas for improvement / future work

Bio:-

Nishant Hegde is a Senior Analytics Engineer at Netflix. He focuses on data engineering, analysis and data visualization in the Digital Supply Chain Analytics team. Nishant was previously an analyst and managed teams in the Forensic Analytics practices at Deloitte and Price Waterhouse Coopers.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

Mining Member Feedback to Improve the Customer Experience by Nishant Hegde of Netflix

  1. 1. Mining Member Feedback to Improve The Customer Experience Nishant Hegde, Sr. Analytics Engineer
  2. 2. analytics @ Netflix Data Engineering And Analytics
  3. 3. The “Content Quality” Problem Buying Sourcing
  4. 4. Assets Studio Content Partners Encoding Package How is our Content Quality? FAIL PASS
  5. 5. Encoding Video Hit Audio Distortion Hit Some Examples of Content Quality Issues
  6. 6. ● Value ● Easily Integrated v1
  7. 7. v1 Event Producer Ingest Pipeline
  8. 8. v1 Week 1 Bojack ugh Week 1 Bojack Bojack is awesome Week 1 Bojack No audio Week 2 Bojack When is season 4 coming out? Week 2 Bojack ... Random Forest Classifier Week 1 Bojack ugh 0.7 Week 1 Bojack Bojack is awesome 0.3 Week 1 Bojack No audio 0.9 Week 2 Bojack When is season 4 coming out? 0.5 Week 2 Bojack ... Act / Do Not Act Retrain Spreadsheets Assign likelihood that comment needs action Collect comments per show on a weekly basis
  9. 9. v1 Outcome ● Accuracy = 25% ● signal : noise ○ “ i didnt watch this piece of crap dumb people call movie.” ○ “I DID NOT WATCH THIS! THEREFORE, MY NETFLIX ACCOUNT HAS BEEN HACKED....AGAIN!!! WOULD YOU PLEASE DO SOMETHING ABOUT THIS???” ○ “Someone else is accessing Netflix on my account, probably my EX wife, I'd like to stop any other use then myself and have tried to change my account email and password but they still have access, this movie was not ordered by me nor the one previous.” ○ “Frame appears to be windowboxed 4:3-in-16:9-in-4:3 with some sort of white line border artifact on the right side. All other episodes are normal.” ● Class Imbalance ● Buy-in
  10. 10. ● Language Specific Models ● Daily Processing ● Integrate Twitter v2
  11. 11. Local Language + English Translated Models + Separate Model for Tweets v2 Day 1 Bojack ugh 0.7 Day 1 Bojack Bojack is awesome 0.3 Day 1 Bojack pas de hd 0.6 Day 2 Bojack Nao fica em hd de forma alguma 0.5 Day 2 Bojack ... Expanded to foreign languages Act / Do Not Act Cloud App Assign likelihood that comment needs action Collect comments per show on a daily basis Retrain Day 1 Bojack ugh Day 1 Bojack Bojack is awesome Day 1 Bojack pas de hd Day 2 Bojack Nao fica em hd de forma alguma Day 2 Bojack ... Measure Performance
  12. 12. Measuring Performance... v2 ● Recall = TP / (TP + FN) ● Precision = TP / (TP + FP) ● Compliance = (TP + FP) / (TP + FN + FP + TN) ● % looked at = # acted on / # served ● % accurate = # issues found/ # acted on ● % thrash = # non-issues found / # acted on 1. Models 2. Workflow
  13. 13. v2
  14. 14. Russian Dialogue IssueSome Big Finds...
  15. 15. Dubbed Audio Issue
  16. 16. Out of Sync Subtitle Issue
  17. 17. The Future ... ● Abandonment Time in Movie ● # of Seek Operations ● Average Time between Sessions ● Unsupervised Learning Problem?!
  18. 18. plata o plomo?

×