I developed the Customer Sentiment Index, a measure based on a single word that customers use to describe a company. Using the following question, "What one word best describes this company?", Results show that the customer-generated words can be reliably scaled along a sentiment continuum and that these scores (CSI) are logically related to important customer experience metrics, like customer loyalty and satisfaction with the customer experience.

1. Developing the Customer
Sentiment Index
Bob. E. Hayes, PhD
bob@analyticsweek
@bobehayes

2. 2Copyright © 2015 Business Over Broadway and AnalyticsWeek
Who I am
Scientist /
Blogger
Author
Consultant

3. 3Copyright © 2015 Business Over Broadway and AnalyticsWeek
Business Over Broadway
Solve problems, primarily business problems,
through the use of the scientific method
Owner
Using data and analytics to help make decisions that
are based on fact, not hyperbole.
AnalyticsWeek
Help businesses optimize their data/analytics
Chief Research Officer
Improving talent/technology recruitments, facilitating
deeper community engagement with the power of
online/offline channels and conducting research
What I do

4. 4Copyright © 2015 Business Over Broadway and AnalyticsWeek
Structured
• Data generated to measure
specific construct
How satisfied are you with
company?
• Customer-generated
Unstructured
• Data not generated to
measure specific construct
emails, social media, support
calls, movie reviews, tweet
content, transcripts of
comments
• Algorithm-generated
Approaches to Measuring AttitudesIntentSourceScore

5. 5Copyright © 2015 Business Over Broadway and AnalyticsWeek
Intentionally Unstructured Measurement
• Intentional measurement
– Text needs to be generated for the purpose of
measuring attitude
• Sentiment lexicons
– list of words that reflect an opinion on a negative-
positive continuum
• What one word best describes this company?
• What one word best describes this company’s
products / services?

6. 6Copyright © 2015 Business Over Broadway and AnalyticsWeek
What one word best describes this company?

7. 7Copyright © 2015 Business Over Broadway and AnalyticsWeek
Developing a Sentiment Lexicon
• Judgment-based
– Subject matter experts’ classification/scaling of
words
• Empirically-based
– User-supplied textual metadata (e.g., star ratings
on reviews)

8. 8Copyright © 2015 Business Over Broadway and AnalyticsWeek
Judgment-Based
• Two subject matter experts independently
rate list of words from customer survey
– 0 (negative sentiment) to 10 (positive sentiment)
• High agreement between experts (rater)
Lexicon Mean SD 1 2 3 4
1. Rater 1 First Rating 6.49 2.32
2. Rater 2 First Rating 6.44 1.93 .87
3. Rater 1 Second Rating 6.35 2.39 .98
4. Rater 2 Second Rating 6.42 1.95 .99 .90
5. Average Sentiment 6.39 2.11 .96 .96 .98 .97
N = 251. All correlations are statistically significant at the p < .01 level.
Average sentiment: Based on based on the average second sentiment ratings of each rater.
Bold correlations represent inter-rater agreement.
Italic correlations represent intra-rater agreement.
Descriptive
Statistics
Correlations

9. 9Copyright © 2015 Business Over Broadway and AnalyticsWeek
Empirically-Based
• Examine four corpora (a collection of written
text) with accompanying ratings
• Data from four review sites*
– OpenTable, IMDB, Goodreads, Amazon/TripAdvisor
* See Christopher Potts: http://web.stanford.edu/~cgpotts/talks/potts-wordnetmods.pdf

10. 10Copyright © 2015 Business Over Broadway and AnalyticsWeek
Description of Four Lexicon Sources

11. 11Copyright © 2015 Business Over Broadway and AnalyticsWeek
Calculating Sentiment for Each Word
• Re-scale values from 1 to 5 -> 0 to 10
• Calculate sentiment of each word (adjective)
– “excellent” sentiment value of 8.39
– “good” sentiment value of 6.69

12. 12Copyright © 2015 Business Over Broadway and AnalyticsWeek
Distribution of Words’ Sentiment Values

13. 13Copyright © 2015 Business Over Broadway and AnalyticsWeek
Descriptive Statistics and Correlations of
Sentiment Values of Words

14. 14Copyright © 2015 Business Over Broadway and AnalyticsWeek
Judgment and Empirical Approach
• Classification/Scaling of words are comparable
across methods
Lexicon Mean SD N 1 2 3 4
1. OpenTable 7.07 1.48 164
2. AmazonTripadvisor 7.52 1.17 168 .72
3. Goodreads 6.48 1.29 168 .58 .60
4. IMDB 6.95 .82 169 .72 .62 .73
5. Expert 6.39 2.11 251 .71 .60 .52 .67
All correlations are statistically significant at the p < .01 level.
Expert sentiment was the average of the experts' second/adjusted sentiment ratings.
CorrelationsDescriptive Statistics

15. 15Copyright © 2015 Business Over Broadway and AnalyticsWeek
Difference among Lexicons

16. 16Copyright © 2015 Business Over Broadway and AnalyticsWeek
Reliability of Customer Sentiment Index
• B2B Technology Company
– Customer Survey – one word, ratings
• Context is important
Mean SD N 1 2 3 4
1. CSI - Expert 7.09 1.84 894
2. CSI - OpenTable 7.12 1.18 766 .77
3. CSI - IMDB 6.78 .86 786 .60 .78
4. CSI - Goodreads 6.30 1.23 757 .62 .74 .93
5. CSI - Amason/Tripadvisor 7.65 .97 623 .65 .83 .77 .68
Correlations among CSI scores
Mean SD N
CSI
Expert
CSI
OT
CSI
IMDB
CS
GR
Overall Satisfaction 7.60 1.99 1595 .57 .48 .33 .30
Recommend 7.91 1.96 1585 .56 .49 .35 .31
Purchase same / similar 8.22 2.26 1527 .34 .31 .24 .24
Purchase additional / different 6.32 2.83 1508 .18 .16 .10 .09
Expand use 6.80 2.59 1523 .22 .22 .18 .16
Renew service contract 7.90 2.53 1159 .30 .25 .19 .18
Ease of doing business 7.37 2.17 1204 .55 .49 .37 .35
Account Management 6.98 2.32 1189 .42 .35 .26 .25
Product Quality 7.93 1.89 1297 .50 .44 .27 .27
Service / Repair 7.67 2.15 1092 .41 .37 .29 .30
Technical Support 7.73 2.29 1253 .43 .37 .29 .29
Communications from
Company
7.37 2.13 1282 .51 .45 .34 .32
Direction and future
products/services
7.45 1.96 1165 .48 .41 .26 .26
All correlations statistically significant at the p < .05 level.
All measures are on a scale from 0 (low loyalty/satisfaction) to 10 (high loyalty/satisfaction).
Based on respondents (N = 1619) of annual customer survey of a B2B technology company.
Loyalty
Satisfactionwiththe
CustomerExperience
Correlations of CSI scores with Loyal

17. 17Copyright © 2015 Business Over Broadway and AnalyticsWeek
Validity of Customer Sentiment
Index
Mean SD N
CSI
Expert
CSI
OT
CSI
IMDB
CSI
GR
CSI
A/TA
Overall Satisfaction 7.60 1.99 1595 .57 .48 .33 .30 .43
Recommend 7.91 1.96 1585 .56 .49 .35 .31 .42
Purchase same / similar 8.22 2.26 1527 .34 .31 .24 .24 .21
Purchase additional / different 6.32 2.83 1508 .18 .16 .10 .09 .12
Expand use 6.80 2.59 1523 .22 .22 .18 .16 .18
Renew service contract 7.90 2.53 1159 .30 .25 .19 .18 .21
Ease of doing business 7.37 2.17 1204 .55 .49 .37 .35 .45
Account Management 6.98 2.32 1189 .42 .35 .26 .25 .36
Product Quality 7.93 1.89 1297 .50 .44 .27 .27 .42
Service / Repair 7.67 2.15 1092 .41 .37 .29 .30 .30
Technical Support 7.73 2.29 1253 .43 .37 .29 .29 .33
Communications from
Company
7.37 2.13 1282 .51 .45 .34 .32 .39
Direction and future
products/services
7.45 1.96 1165 .48 .41 .26 .26 .37
All correlations statistically significant at the p < .05 level.
All measures are on a scale from 0 (low loyalty/satisfaction) to 10 (high loyalty/satisfaction).
Based on respondents (N = 1619) of annual customer survey of a B2B technology company.
Loyalty
Satisfactionwiththe
CustomerExperience
Correlations of CSI scores with Loyalty/CX Metrics

18. 18Copyright © 2015 Business Over Broadway and AnalyticsWeek
Relationship between CSI and
Recommend
B2B Survey B2C Survey

19. 19Copyright © 2015 Business Over Broadway and AnalyticsWeek
Applications
• Mobile Surveys
• Extract more information from single word
– Apply different lexicons (e.g., anxiety, strength)
• Simplify feedback process
– Shorter surveys benefit customers
– Fewer dashboard metrics facilitate executive
reports

20. 20Copyright © 2015 Business Over Broadway and AnalyticsWeek
Bob E. Hayes, Ph.D.
Email: bob@analyticsweek.com and bob@businessoverbroadway.com
Web: www.analyticsweek.com and www.businessoverbroadway.com
Blog: www.businessoverbroadway.com/blog
Twitter: www.twitter.com/bobehayes