1. Social Text, Sentiment and Tone
Analysis
Advanced Analytics and Insights
Ruben Pertusa Lopez, SolidQ, Business Intelligence DPE & MAP 2013 (rpertusa@solidq.com)
Paco Gonzalez, SolidQ, Mentor (paco@solidq.com)
April 10-12 | Chicago, IL

2. Please silence
cell phones
April 10-12 | Chicago, IL

3. Paco & Ruben
• Conference Speakers
• Based in Europe
• Book Authors
• PHD Candidate on Data Mining
• Project Managers
• Microsoft Certified Professionals
3

4. Goals
• This session will help us understand how to analyze
sentiment / tone using cutting edge Microsoft
Technologies
• This is NOT a research session on NLP (Natural Language
Processing). Don’t be scared! ☺
4

5. Agenda
• Overview of Sentiment Analysis
• Gathering and Storing Data
• Sentiment Analysis Techniques
• Business Analytics for Sentiment & Structured data,
together
5

6. Overview of Sentiment
Analysis
April 10-12 | Chicago, IL

7. What is Sentiment?
Feelings, Opinions, Emotions
• Like
• Dislike
• Good
• Bad
• …
7

8. Some examples
paulo @paulomors
Barcelona Beating Milan 4-0 and a gallon of Coke, best day of the year ☺ ☺
αυѕтιη ✰ @AustinJohnsto
A 2 hour delay is like when a restaurant only has Pepsi products. It'll work but ill still be
very disappointed.
Trace @trace_haf
Drinking more Coke than water every day does no good for you health.
☺?
?
DONALD @donald150
My budget decides whether coke or pepsi, always buy the cheaper
8

9. Not only Twitter!
The Walking Dead Season 3 (3300 customer reviews)
Awesome season opener! October 15, 2012 By K. Erwin
After sticking through season two, which was alot of looking for a little girl and
standing around on a farm waiting for something to happen, I hoped that they'd
pick up the pace a little with season three. The premiere doesn't dissapoint!
The Walking Dead Season 3 Mid-Premiere (72K Likes, 7K comments)
Timothy Berteau You guys are crazy. One boring episode and you think the show sucks,
did you forget how awesome the two episodes before it were? Not every episode is
going to be action packed. Season 1 and 2 had some very boring moments as well.
18 February at 00:58 via mobile · 5 likes
9

10. DEMO
Looking at some Twitter Sentiment
10

11. Surrounded by opinions
12 Tb 21 Pb
Hadoop
day
cluster
7 Pb
Month
(search queries info)
1 Tb 7 Tb
Tweets
Data day
day
75 Mi 4B
Scores Graph
day edg/day
Millions of
opinions
11

12. Valuable Information for our Business
Questions
• Is this review positive or negative?
• Do this Twitter User like or dislike my new show?
• What are they saying about our company or services?
• How are Facebook User’s Attitudes about the next
election?
Is that the only valuable info?
12

13. What is Sentiment Analysis?
Text Categorization (Opinion Mining)
• Positive / Negative / Neutral
• 1, 2, 3, 4, 5 Stars
• For / Against
Sentiment Analysis
Text IN OUT Category
Techniques
Photos
Videos, etc.
13

14. DEMO
Twitter for Analytics
14

15. Gathering and Storing Data
April 10-12 | Chicago, IL

16. Gathering data from sources
Public/Private APIs
• Limitations
• Privacy
• Format & Structure of source data
• Updates
Webcrawlers? Call the cops!
16

17. How may it look?
JSON Example (1 tweet)
17

18. Storing the gathered data
Why?
• Valuable data!
• Store now. Figure out later
How?
• Structured (Relational Database – SQL Server)
• Semi-Structured (Hadoop Cluster – Microsoft HDInsight)
18

19. Doctor, We need some help
The 4 V’s
Volume
Velocity
Variety
Variability
19

20. Doctor, We need some help
The 4 V’s
Volume
Velocity
Variety
Variability
20

21. Start building our system
Extract
Extract
Transform
Load
Load
21

22. DEMO
Let’s gather some Twitter data ☺
22

23. Sentiment Analysis
Techniques
April 10-12 | Chicago, IL

24. Understanding the problem
Data != Sentiment Data
Identify relevant parts
• Nouns
• Adjectives/Adverbs
• Verbs
Drop anything else
24

25. Sentiment Analysis Techniques
Techniques
• Natural Language Processing
• Basic Statistics
• Clustering
• Fuzzy Components
• Classification
• Estimation
Tools
HDInsight, SSIS, SSAS DM, FullText Search (Semantic Search)
25

26. Dictionaries
Tone Index Dictionary
Positive 10 amazing
Positive 9 awesome
Positive 8 best
Match words Positive 7 excellent
Positive 6 exciting
“Coke is the best & coolest drink Positive 5 great
on the market” Positive
Positive
4
3
good
rocks
Positive 2 cool
Positive 1 :)
Neutral 0
Negative -1 :(
Negative -2 poor
Negative -3 bad
Negative -4 criticized
Some Other Database Example Negative -5 attacked
Negative -6 humiliated
SentiWordNet Negative -7 sucks
http://sentiwordnet.isti.cnr.it/ Negative
Negative
-8
-9
terrible
horrible
Negative -10 worthless
26

27. A closer look at the example
“Coke is the best & coolest drink on the market”
Tone Index Dictionary
Positive 10 amazing
Fuzzy Match Positive 9 awesome
Positive 8 best
Positive 7 excellent
“Best” matches “best” =+ 8 Positive 6 exciting
Positive 5 great
“coolest” matches “cool” =+ 2 Positive
Positive
4
3
good
rocks
Positive 2 cool
Total = + 10 Positive 1 :)
Neutral 0
☺ Negative
Negative
-1
-2
:(
poor
27

28. DEMO
Give me your sentiment!
28

29. Sentiment Analysis Challenges
Phrase level polarity
• “The US fears happy caffeine consumers”.
Context polarity
• Context: The date that Steve Jobs died.
• Opinion 1: “This is the worst day of my life”.
• Opinion 2: “The world is a better place today”.
Irony, Sarcasm, Irregular forms, Pragmatic information…
29

30. It looks like…
Extract
Extract
Transform
Load
Load
SA
Techniques
☺
30

31. Combining Sentiment
Analysis with Structured
Data
April 10-12 | Chicago, IL

32. MORE Valuable Information for our
Business
Questions
• Is this positive/negative review affecting my sales?
• Have we increased our sales because of positive tweets? Can we
learn anything from opinions to improve our products?
• How are people responding to this campaign?
32

33. Bring our cubes into a new world
New ways of analysis
Better decisions
• Drive next campaign, new wave of products, etc.
Estimate future figures
☺ +
33

34. DEMO
Our cubes have feelings ☺
34

35. Final picture
Extract
Extract
Transform
Load
Load
SA
Techniques
☺ + 35

36. Brief Summary
During this session..
• Business Insights from social opinions
• Sentiment Analysis loves Big Data ☺
• Microsoft Technologies can help us with some SA Techniques
• There is a huge competitive advantage by using Sentiment Analysis
36

37. QUESTIONS
Positive / Negative / Neutral ones ☺
37

38. Contact us!
Rubén Pertusa López (rpertusa@solidq.com)
@rpertusa
Data Platform Engineer, SolidQ
Microsoft Active Professional 2013
Paco González (paco@solidq.com)
@pacosql
Mentor, SolidQ
38

39. Win a Microsoft Surface Pro!
Complete an online SESSION EVALUATION
to be entered into the draw.
Draw closes April 12, 11:59pm CT
Winners will be announced on the PASS BA
Conference website and on Twitter.
Go to passbaconference.com/evals or follow the QR code link displayed on
session signage throughout the conference venue.
Your feedback is important and valuable. All feedback will be used to improve
and select sessions for future events.

40. Platinum Sponsor
Thank you!
Diamond Sponsor
April 10-12, Chicago, IL