Types of customer feedback, how easy are they to collect, analyse and how insightful are they? Why analyzing customer feedback is important? Why is it hard to analyze free-text customer feedback? What approaches are there to make sense of customer feedback (manual coding, word clouds, text categorization, topic modeling, themes extraction) -- and what are their limitations? Which AI methods can help with the challenges in customer feedback analysis.

1. Using AI to Make Sense of
Customer Feedback
Alyona Medelyan
@zelandiya

2. Correct Understanding of Customer Feedback
Can Save Millions

3. 2015: Tensof Thousandsof New Zealanders
wereSurveyedAbout the new Flag

4. GovernmentReported
the Resultsof Manual FeedbackAnalysis

5. Actual Responses
Two costly & unnecessary referendum followed. Outcome: NZ kept the current flag
Millions could have been saved!
People wanted to ”keep the current flag”

6. 1. Types of customer feedback
2. Why analyzing customer feedback is important
3. Why is it hard
4. Approaches
5. Applying AI to customer feedback analysis
6. Demo

7. Different Types
of Customer Feedback

8. Types of Customer Feedback
one-on-one interviews / focus groups
call centre logs / complaints
social media
open-ended survey questions / reviews
quantitate survey questions
UX tests / analytics
unstructured
structured

9. Collection Analysis Insight
one-on-one interviews / focus groups hard hard good
call centre logs / complaints easy hard limited
social media easy hard limited
open-ended survey questions / reviews easy medium good
quantitate survey questions easy easy limited
UX tests / analytics medium easy limited
unstructured
structured
Comparing Types of Customer Feedback

10. Why Understanding
Customer Feedback
is More Important than Ever

11. Customer Experience
is the New Marketing
It’s Measured Using
Net Promoter Score Surveys
Image credit

13. The number of “Net Promoter Score”
searches on Google since 2004
1. Growing Number of
Satisfaction Surveys and Reviews

14. v
¯_(ツ)_/¯
2. The Need to Explain
the Why’s Behind the Scores
Net Promoter Score by month over time

15. 3. Scores can be Cheated
Unstructured Feedback, not so Much

16. Why Analyzing
Customer Feedback is Hard

17. Common Misconception:
Sarcasm MakesAnalysis Hard
One of Many Sarcastic Tui Beer Adverts

18. Sarcasm is Hard:
Even People Struggle
I’ll keep it in
mind
They’ll do it
I’ve
forgotten
already

19. Sarcasm is Rarer Than You Think
Dataset Sarcasm Example
NPS Survey 1%
I’m so disappointed! What a great
customer service you have!
Social Media
comments
5% Very helpful answer. Troll.

20. TheActual Challenges
With Customer Feedback

21. Challenge 1: Messy Data

22. How many ways there are to say
‘wet paper’?
Challenge 2: Synonyms and Paraphrases

23. Hundreds of
possible variations
of the same theme
wet
dripping
soaking
soaked
damp
drenched
paper
papers
newspaper
news paper
newspapers
news papers
+
Paraphrasing the Same Theme

24. Challenge 3: Negation
Positive or Negative?
My coffee was great positive
My coffee was awful negative
My coffee was not great negative
My coffee was not that great neutral?
I did not think my coffee was great negative
I did not expect my coffee to be this great positive
I was disappointed with the quality of the coffee negative
I was not disappointed with the quality of the coffee positive

25. Approaches to
Customer Feedback Analysis

26. Manual Coding
1.

27. Figure out the Code Frame, Apply, Repeat
What is the meaning of life?
1 2 3 4 5
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
What is the meaning of life?
42
Friends and family
Making a difference in the world
Happiness
Finding happiness
To achieve, to conquer
Family
…
1
2
3
4
4
5
2

28. Sentiment in a Manual Code Frame
Customer Service
Positive Negative
Timely Nice Helpful Didn’t fix issue Rude

29. Word Clouds
2.
“Every time I see a word cloud presented as insight,
I die a little inside.”
– J. Harris, journalist

30. Word Clouds Lack
Interpretation, Context, Meaning
“Overall the language
focuses on sweeping
statements focusing on
the state of the nation.”
Kalev Leetaru (Forbes)

31. You wouldn’t create a Word Cloud from your Numbers,
why is it ok from Text?

32. Rule-based Approaches
3.

33. It’s Hard to Find a Rule That Works Well
I was impressed by how friendly the person
on the other end of the line was
Staff friendliness ✔
The lady who helped me was friendly Staff friendliness ✔
Friendliness of staff Staff friendliness ✔
Your website is very user friendly Staff friendliness ✘
The young man on the phone was very pleasant Other ✘
friendly OR friendliness –> Staff friendliness

34. Text Categorization
4.

35. old
customer
responses
categories
new
customer
responses
Machine
Learning
Algorithm
Predictive
Model categories
Need for Sufficient Training Data,
and Clear Categories

36. Customer FeedbackAnalysis
Needs to be ‘Unsupervised’

37. Thanks to an unsupervised approach, Facebook found
Candi Crash Saga causes low App Store reviews

38. Topic Modeling
5.

39. 2
1
3
ATopic can be Hard to Interpret
2
???ok
Source: Ben Fields

40. Sentiment

41. 1. Rule-based (dictionary)
2. Text categorization (positive / negative)
Two Sentiment Detection Approaches

42. Advances in AI > Customer Feedback

43. Messy Data
Paraphrases
Negation
AI > Challenges
Word2vec*
Deep Learning
*See also: Conceptnet.io
Knowledge Representation

44. Word2Vec
Image source: ericbern.com
Best Intro: Word2Vec Udacity Youtube

45. Knowledge Representation

46. Deep Learning
Precision Recall F-Measure Errors
People 84 73 75 <1
Dictionaries 61 57 54 8
Linear Regression 65 56 47 3
Deep Learning 62 57 49 2
Sentiment Analysis is not about maximizing F-Measure,
it’s about reducing true Errors: positive confused with negative

47. Theme Extraction
6.

48. From Words to Complex Themes

49. Applying Customer Feedback Analysis

50. Google: Sentiment by Theme

51. Thematic Demo

52. Thanks
@zelandiya
alyona@getthematic.com
getthematic.com
linkedin.com/in/medelyan