The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality
•
2 gefällt mir•603 views
FIS and Databricks worked together to develop a conversational analytics system using Spark and machine learning. Their initial approach using pre-trained NLP models had problems, so they built an LSTM model trained on conversation data to predict sentiment. A random forest classifier using the LSTM output and other features achieved 80% accuracy. The full pipeline was deployed on Databricks to continuously learn from streaming conversation data and provide sentiment analysis at scale.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality
Ähnlich wie The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality (20)
Mehr von Databricks
Mehr von Databricks (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
The Pursuit of Happiness: Building a Scalable Pipeline Using Apache Spark and NLP to Measure Customer Service Quality
- 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
- 2. Jaime Woodfin, FIS Global Brooke Wenig, Databricks Special thanks to Amir Issaei, Kevin Mellott, and Aaron Colcord #UnifiedAnalytics #SparkAISummit
- 3. Outline • FIS & Databricks Intro • Business Problem & Motivation • Approach v1 – Problems encountered along the way • Approach v2 • Production 3#UnifiedAnalytics #SparkAISummit
- 4. Who is FIS Global? • The Global leader in financial services technology • Customers: banks and credit unions • Ecosystem of products and services built around core banking • FIS Digital Finance, Digital Data and Analytics 4#UnifiedAnalytics #SparkAISummit
- 5. Accelerate innovation by unifying data science, engineering and business • Founded by the original creators of Apache Spark • Contributes 75% of the open source code, 10x more than any other company • Trained 100k+ Spark users on the Databricks platform VISION WHO WE ARE Unified Analytics Platform powered by Apache SparkPRODUCT
- 7. Conversational Analytics • Measure in support conversations • Follow-up on support conversation 7#UnifiedAnalytics #SparkAISummit
- 9. Conversational Channels Developments 9#UnifiedAnalytics #SparkAISummit Face-to-face Human support chat Support chatbots Conversational Banking
- 10. Goals • Score overall user conversation satisfaction • Question: What contributed to their satisfaction? • AND how to do it at scale?? 10#UnifiedAnalytics #SparkAISummit
- 11. Approach v1
- 12. Approach v1 • Apply open-source NLP libraries to each turn in conversation 12#UnifiedAnalytics #SparkAISummit
- 13. Library Comparison • Different Scales – TextBlob: [-1, 1] – NLTK: [-1, 1] – John Snow Labs (sparknlp): Negative or Positive – Stanford CoreNLP: 0, 1, 2, 3, 4 13#UnifiedAnalytics #SparkAISummit
- 14. Demo
- 15. Problems Encountered • Stanford CoreNLP gave predictions per sentence, not turn • Performed poorly on neutral sentences – John Snow Labs had no “neutral” category • Didn’t do well with banking domain Can we do better? 15#UnifiedAnalytics #SparkAISummit
- 16. Approach v2
- 17. Approach v2 • No pre-trained sentiment analysis models! • Model: – Build LSTM model on all conversation text to predict sentiment – Augment with additional features (e.g. # of turns, time of day, etc.) – Pass features through end classifier • Positive/Negative 17#UnifiedAnalytics #SparkAISummit X
- 18. Transfer Learning • Distributed training of LSTM on open-source sentiment dataset with HorovodRunner • Transfer learning on banking data 18#UnifiedAnalytics #SparkAISummit
- 19. LSTM Stats • Basic stats for performance: – Accuracy: 73% – FPR: 11% – FNR: 32% 19#UnifiedAnalytics #SparkAISummit 𝐹𝑃𝑅 = 𝐹𝑃 𝑁 = 𝐹𝑃 𝐹𝑃 + 𝑇𝑁 𝐹𝑁𝑅 = 𝐹𝑁 𝑁 = 𝐹𝑁 𝑇𝑃 + 𝐹𝑁
- 20. Features • LSTM output • Conversational Features – User average turn length – Agent average turn length – # Turns – Duration • Temporal Features – Day of Week – Time of Day • Others 20#UnifiedAnalytics #SparkAISummit
- 21. Classifier • Logistic Regression – Accuracy: 77% – FPR: 24% – FNR: 22% • Random Forests – Accuracy: 80% – FPR: 10% – FNR: 51% • Others 21#UnifiedAnalytics #SparkAISummit
- 22. Random Forest • Chose the Random Forest b/c: – Lowest FP rate & Highest Accuracy – Good model interpretability – Part of SparkML and can use with Pipeline API (easy to switch to Scala) 22#UnifiedAnalytics #SparkAISummit
- 23. Production
- 24. Production Requirements • Fit into dev pipeline, largely Scala/Java based – But a lot of data science is done in Python • Close the feedback loop - constantly learning • Automated deployments • Streaming instead of batch 24#UnifiedAnalytics #SparkAISummit
- 27. Python & Scala • Train LSTM in Python (Keras) • Save Model • Load model via UDF • Apply using Scala! 27#UnifiedAnalytics #SparkAISummit
- 30. Thank you!
- 31. Life is but a Stream @ 3:30 pm Thursday Aaron Colcord & Kevin Mellott (FIS Global) Streaming Track, ROOM 2007
- 32. Questions?
- 33. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT