Transforming the call center with Text mining and Deep learning: 1. Text ming tool to unlock user insights 2. Artificial Intelligence revolution in call centers: deep learning-based bot

1. Paul Lo
Data Analytics Manager @ Uber, Asia-Pacific Community Operation Central team
paullo0106@gmail.com | paul.lo@uber.com | http://paullo.myvnc.com/blog/
Transforming the Call Center with Text Mining
and Deep Learning for Better User Experience
PythonPH Sep. 2018 (https://www.meetup.com/pythonph/events/254444065/)

2. Project #1
Text ming tool to unlock user insights
Python lib: natural language processing,
topic modeling
Self-introduction
Who am I?
What does our analytics team do for
Asia-Pacific?
Project #2
Artificial Intelligence revolution in call
centers: deep learning-based bot
Python lib: machine learning related
such as tensorflow, keras, sklearn,
numpy, and etc.
Transforming the Call Center with Text Mining and Deep Learning for Better User Experience
Table of contents
Transforming the Call
Center with Text Mining
and Deep Learning for
Better User Experience

3. Self-introduction
Skills: Full stack software engineer (Java/ Python) → Data Analyst (R/ Python, databases, machine learning)
Journey: Taipei → Shanghai → Manila

4. Self-introduction
Uber Shanghai → Uber Manila (APAC Community Operation Central Analytics team)

5. Scope of Community Operation in Uber APAC
Scope
10+ languages in ~20 locations
Central Team
In
Manila
India
Singapore (South East and North Asia)
Australia

6. APAC A&I
2017 Year-end
Analytics & Insights is the team responsible for building the analyses,
models, and tools to aid operational and strategic decision making for the
APAC Region. We are also dedicated to furthering Uber’s collective
analytical capability.

7. Self-introduction
Uber still has awesome team (Analytics, S&P, PM, and etc) based in Manila!!

8. Improving user experience is one of our core mission
Improve user experience
Drive down defect rate
Optimize operational efficiency
Manage the cost of business operation

9. Project #1:
Text mining and NLP for use experience
enhancement
Acknowledgement: Troy James Palanca, Lorenzo Ampil

10. Value proposition
Speed up the workflow on user experience enhancement
Defect rate and issue type
Leaderboard
Community
Operation
Product,
Engineering,
and etc.
User
feedback
database
Root cause analysis
and recommended
feature or policy
changes
Review
customer
feedback in
tickets
User experience
enhancement

11. Value proposition
Speed up the workflow on user experience enhancement
Defect rate and issue type
Leaderboard
Community
Operation
Product,
Engineering,
and etc.
User
feedback
database
Root cause analysis
and recommended
feature or policy
changes
Review
customer
feedback in
tickets
User experience
enhancement
Making this process more efficient

12. Issue type dashboard as a high-level data source
Mockup
Dashboard

13. Problem
How can we quickly get the insights from users’ feedback?
Problem
Reviewing tickets
manually to diagnose
the root cause is not
scalable and
unsystematic
Ticket dataset
Driver > Trips > Fare … > … > Technical issue
ticket
ticket ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket
ticket

14. Problem
How can we quickly get the insights from users’ feedback?
Solution
Use topic modeling
techniques to
efficiently group tickets
and assign them to
reasonably named
topics.
Ticket dataset
Driver > Trips > Fare … > … > Technical issue
App stuck/ crash
(35%)
Fare calculation
Dispute
(15%)
GPS issue
(55%)

15. Key features of our solution
Using Topic modeling based tool to learn pain points from our users
Ticket snippet with user profile: respective ticket
samples are displayed when clicking on a keyword
Word cloud view: user can switch to
this view to see most relevant (tf-idf
score) keywords in each topic
>>DEMO

16. Sample results
“Fare Disputes” in one of the city we operate are
mainly about payments, airport issues, and wrong
riders:
● Credit cards and other modes of payment
(18%)
● Overcharging (28.8%)
● Wrong profiles being billed (12.8%)
● Airport terminal issues (12.9%)
● Someone else taking the trip (12.5%)

17. Sample results
Lots of “rude”, “loud music”, “drunk”, and “slam door” keywords
were detected as the pain points of our NY driver partners

18. Sample results
More than 10% of driver cancellation
tickets in Singapore are related to car
seat rules for child safety: many
sample tickets show that drivers want to
reimburse their cancellation fee due to
their riders bringing children without prior
notice.

19. Tool architecture
Computing node
(any Uber servers)
Data collection
Data preparation
LDA model training
Web server
(AWS node)
Html and json
files from
training results
User Interface
(d3js)
Train the model for each country with top issues
monthly
Web 1.0 design with the focus on computing node

20. Workflow overview
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category

21. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Text processing library: nltk, BeautifulSoup, re, TextBlob
LDA library: gensim.ldamodel.LdaModel and pyLDAvis

22. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words:
● Numbers
● Html tags
● Custom dictionary
Stemming and lemmatization
Tokenization
TFIDF (Term Frequency Inverse Document
Frequency)

23. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words:
● Numbers
re.sub(r'd+', '', text)
● Html tags
BeautifulSoup(document).get_text()
BeautifulSoup(document).find_all(‘b’)
● Custom dictionary
Stemming and lemmatization
Tokenization
TFIDF (Term Frequency Inverse Document
Frequency)

24. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words
Stemming and lemmatization: Reduce inflectional
forms and sometimes derivationally related forms of a
word to a common base form. For instance:
○ cancel, cancels, cancelled -> cancel
○ riders, rider -> rider
Tokenization
TFIDF (Term Frequency Inverse Document
Frequency)

25. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words
Stemming and lemmatization
Tokenization: Part-of-speech based word
detection
TFIDF (Term Frequency Inverse Document
Frequency)

26. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words
Stemming and lemmatization
Tokenization: Part-of-speech based word
detection
TFIDF (Term Frequency Inverse Document
Frequency) Common practice to score each term
with weighted frequency and relevance

27. Data Preparation (Natural Language Processing)
Using TFIDF to filter the most important keywords
Machine Learning
Model

28. Data Preparation (Natural Language Processing)
Using TFIDF to filter the most important keywords
Machine Learning
Model
Term frequency
Inverse Document
Frequency

29. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Data preparation for text processing can be very time-consuming
Sample ~50,000 tickets for
each training in each issue
category
Remove invalid words:
Stemming and lemmatization
Tokenization
TFIDF (Term Frequency Inverse Document
Frequency)

30. Speed up data processing
Pandas runs on a single thread by default
A pandas DataFrame with 50k+ rows
Data Preparation
text_processing() is a heavy function
contains many things:
● Tokenization
● Removal of numbers, html tags, and
other invalid words
● Stemming and lemmatization
● TFIDF
df['content'].apply(text_processing)
→ single thread by default

31. Speed up data processing
Pandas runs on a single thread by default
Worker 1
Worker 2
Worker N
keywords

32. Data processing speedup trick in Pandas
Pandas runs on a single thread by default
1
2
3
4
5
6
7
8
9
10

33. Many handy text processing libraries
TextBlob and spaCy
Tokenization Sentence correction
.correct()
Part of speech
.tags
Sentiment analysis
.sentiment.polarity
NLP Library
(TextBlob)
(spaCy)

34. Workflow overview
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Unlocking support insights from textual content - but how?
Sample ~50,000 tickets for
each training in each issue
category
LDA:
- Unsupervised learning
- Bag of words
- “topic distribution”
Usage:
lda = LdaModel(corpus=corpus,
id2word=dictionary,
num_topics=4,
random_state=some_number)
lda.show_topics()

35. Latent Dirichlet Allocation model
General concept of this model
Unsupervised learning method - does not
require any class labels; similar to clustering
‘Bag of words’ model - uses word counts in
messages without regard for its order
(Peter owe Alice money = Alice owe Peter
money)
Estimated iteratively - Starts with random
initialization then adjusts probabilities to
reduce perplexity / increase fit
Doc 1 Doc 2 Doc 3 Doc n...
(topic) FruitsFruits
document-topic
probabilities
30% health (topic
1)
60% fruits
(topic 2)
10% disease
(topic 3)

36. Latent Dirichlet Allocation model
Model implementation and visualization
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Data input: ticket text as raw
data
Output: topic model clusters
Sample ~50,000 tickets for
each training in each issue
category
Usage:
lda = LdaModel(corpus=corpus,
id2word=dictionary,
num_topics=4,
random_state=some_number)
lda.show_topics()
from pyLDAvis.gensim import prepare, save_html
from gensim.models import LdaModel

37. Future work and learnings
Data Preparation
(text processing)
Extract useful information and
transform corpus to a sparse
matrix
Data Modeling
(Latent Dirichlet
Allocation)
Main computation to perform
topic modeling
Customization is needed
● Not suited for
specific issue
category
● Build own
dictionary for the
removal of
irrelevant words
Data input: ticket text as raw
data
Output: topic model clusters
How to make the results more “actionable”?
● # of topic for convergence
● Time and performance
tradeoff
● Other ”Deep NLP” model ?
Bad result
examples

38. Project #1
Text ming tool to unlock user insights
Python lib: natural language processing,
topic modeling
Self-introduction
Who am I?
What does our analytics team do for
Asia-Pacific?
Project #2
Artificial Intelligence revolution in call
centers: deep learning-based bot
Python lib: machine learning related
such as tensorflow, keras, sklearn,
numpy, and etc.
Transforming the Call Center with Text Mining and Deep Learning for Better User Experience
Table of contents
Transforming the Call
Center with Text Mining
and Deep Learning for
Better User Experience

39. Product owner: Huaixiu Zheng and Yichia Wang in Uber’s Applied Machine Learning team
Project #2:
Artificial Intelligence revolution in call centers

40. CSR’s sample workflow for user in a call center
How does our users submit an issue?

41. CSR’s sample workflow for user in a call center
Online support via in-app-help
User
CSRContact
Ticket
Response
Select
Issue Category
Write Message
Confirm
Issue Category
Lookup info. &
Knowledge Base
Select Action
Write response
using a Reply
Template

42. The issue for call center operation: scalability and cost
The growth comes at a price again….

43. Solution? Let’s start from a basic sample
“I want to change my rating for a rider”

44. API-less solution to the basic sample
We can ‘program’ the pre-defined logic for certain tickets with Selenium or Chrome Script
element mapping
element mapping

45. End-to-end solution
Web interaction
Read and Write (click and input text)
Knowledge base
● Keyword recognition
● Web element id dictionary
● (Natural Language Processing)
Policy engine
Program the flow aligning with policy/ SOP
Monitoring and logging
● Real-time gsheet API logging
● Monitoring and alert trigger
Ticket
answering
bot

46. The business impact of a simple bot-solving solution
3k+ weekly solves
A team of
18 CSR
28k USD
monthly

47. What’s the problem with this solution?

48. What’s the problem with this solution?
“Scalability”

49. The difference between Programming and Machine Learning
Outputs =
Agents’
responses
Inputs =
Contact
Ticket

50. Our machine learning solution design
Why go with “Semi-automated” assistance rather than real robot?
Pros:
- Scalable solution to all (+ new) ticket-types
- Flexible and safer application as human
can still evaluate it and make the final call
Cons: Not fully automated to replace the agent
workforce completely.
Product designed by Hugh Williams, Huaixiu Zheng, Yi-Chia Wang in Applied Machine Learning team

51. Our machine learning solution design
‘Assistant to CSR’ - Provide suggestions for reply and actions
Issue category suggestion
Action suggestion
10M+ tickets
Correct response from
agents to these 10M+
tickets
Technical model training Product design

52. Typical Machine Learning process
Note: picture from “Mark Peng’s “General Tips for participating Kaggle Competitions” on Slideshare

53. Typical Machine Learning process
Model selection
ML 101:
Start with simple model first
Data source: https://eng.uber.com/cota-v2/

54. Deep Learning Architecture
Reference: Uber AML Lab: http://eng.uber.com/cota
Sample code with Keras for a simple CNN

55. Deep Learning Architecture
Reference: Uber AML Lab: http://eng.uber.com/cota
Essay: COTA: Improving the Speed and Accuracy of Customer Support through Ranking and
Deep Networks

56. Development environment for Deep learning model training
How does model training look like?
>> DEMO
Main codebase + data set

57. Feature engineering and feature importance
Trade off between capacity and interpretability
“Capacity” “Interpretability”

58. Feature engineering and feature importance
What are the important features? Very easy to learn that in simpler model

59. Feature engineering and feature importance
What are the important features? Very easy to get explanation in simpler models

60. Feature engineering and feature importance
What are the important features? NN model is like our brain’s intuition … blackbox

61. Feature engineering and feature importance
What are the important features?
Sklearn: Recursive feature elimination
(sklearn.feature_selection.RFE)
Mockup
dataset

62. Feature engineering and feature importance
What are the important features?
Time on model training >>> prediction
Shuffle each feature to create noise…. on the testing set
Mockup
dataset

63. Python tips: be cautious about the underlying “copy implementation”
np.random.shuffle
What’s the value of
my_list2?
A. [1, 2, 3, 4, 5]
B. [2, 5, 1, 4, 3]

64. np.random.shuffle
What’s the value of
my_list2?
A. [1, 2, 3, 4, 5]
B. [2, 5, 1, 4, 3]
Python tips: be cautious about the underlying “copy implementation”

65. np.random.permutation
Python tips: be cautious about the underlying “copy implementation”

66. np.random.permutation
from copy import deepcopy
mylist2 = deepcopy(my_list)
Python tips: be cautious about the underlying “copy implementation”

67. Feature engineering and feature importance
What are the important features?
Shuffle each feature to create noise…. on the testing set
Mockup
example

68. Issue category suggestion
Action suggestion
Product design
Last stop: making business Impact
Ensure KPI measurement is well-planned in the beginning
User
CSRContact
Ticket
Response
Select
Issue Category
Write Message
Confirm
Issue Category
Lookup info. &
Knowledge Base
Select Action
Write response
using a Reply
Template

69. Last stop: making business Impact
Identify key business metrics, and cautiously conduct and monitor A/A and A/B testing
Source: https://eng.uber.com/cota-v2/

70. Look forward to collaborating! http://careers.uber.com

71. Paul Lo
Data Analytics Manager @ Uber
paul.lo@uber.com | paullo0106@gmail.com | | http://paullo.myvnc.com/blog/
Q&A