What Business Innovators Need to Know about Sentiment Analysis, presented by Prof. Claire Cardie of Cornell University.

1. What Business Innovators Need to
Know about Sentiment Analysis
Claire Cardie
Department of Computer Science
Chair, Information Science Department
Cornell University
Co-founder
Chief Scientist

2. Plan for the Talk
Subjectivity and sentiment in language
Continuum of capabilities
– Surface-level in-depth understanding
– Document-level phrase-level
Next steps…

3. Subjective Language
Subjective text expresses speculations,
beliefs, emotions, evaluations, goals,
opinions, judgments, …
• Jill said, "I hate Bill."
• John thought about whom to vote for.
• Seth knew his symposium would go well.

4. Subjectivity vs. Sentiment
Sentiment-bearing text expresses
positive and negative speculations, beliefs,
emotions, evaluations, goals, opinions,
judgments,…
• Jill said, "I hate Bill." -
• John thought about whom to vote for. ~
• Seth knew his symposium would go well.
+
sentiment analysis tome [Pang & Lee, 2008]

5. A Word on Polarity (tone, valence)
Positive “I love NY.”
Negative “I hate NY.”
Neither positive nor negative
– Objective?
“I thought about NY.”
– Neutral?
“I’m ambivalent about NY.”
– Mixed polarity?
“Sometimes I love NY; other times I hate it.”

6. And What About Intensity?
Strength/intensity
“I love NY.”
“I absolutely adore NY!”
– Low, medium, high, very high, extreme
– ratings
– rotten tomatoes

7. Plan for the Talk
Subjectivity and sentiment in language
Continuum of capabilities
– Surface-level in-depth understanding
– Document-level phrase-level
Next steps…

8. Document-level Sentiment Analysis
Is the overall
sentiment in the
Document document
positive?
negative?
neutral?

9. Identifying Tone of a Collection
Sentiment (w.r.t. a topic)
– Example: Tone on “economic stimulus”

10. Detecting “chatter” or “buzz”
Chatter (w.r.t. a topic)
– Example: Buzz on “economic stimulus”

11. Keyword-based Approaches
Search the text for the presence of
specific terms from a manually created
“sentiment lexicon”
– +: “great”, “praise”, “peace”, “superb”, …
– -: “war”, “dull”, “messy”, “criticize”, …
Sentiment is based on the counts
– E.g.,
If more positive terms than negative terms,
then return +,
else return –

12. Keyword-based Approaches
Complications
– Inherent ambiguities of language…
– This laptop is a great deal.
– A great deal of media attention surrounded the
release of the new laptop model.
– If you think this laptop is a great deal, I’ve got
a nice bridge for you to buy.
[Examples from Lillian Lee]
[Pang & Lee, 2008]

13. Machine-learning Approaches
Learn from training data
Are better able to take advantage of
context to disambiguate terms
examples
ML Algorithm
statistical model
(novel) examples class
(program)

14. Measuring Performance
Precision: #correct / #attempted
Recall: #correct / #possible
F-measure: harmonic mean of P and R
1. _______
P = 3 / 4 = .75
2. _______
P = 3 / 3 = 1.00
3. _______
R = 3 / 4 = .75
4. _______
accuracy

15. Measuring Performance
How well do document-level sentiment
analysis systems work?
It depends…
– Product reviews easier than Movie reviews,
easier than News/editorials
– Shorter documents harder than longer ones
– Messy documents harder than clean ones
~75 F - ~85 F

16. This is actually quite good…
Comparison is not vs. 100% P/R…but vs.
human sentiment analysis accuracy
– Cohen’s kappa
Machine-learning methods for sentiment
analysis approach human agreement
levels
– ~85 F: for positive/negative
– ~75 F: when neutrals are included

17. Sentiment Analysis at Passage Level
Passage tone The suggestion that the White
House never took seriously an
– Optionally w.r.t. a issue that infuriated millions of
topic Americans was supported by
– E.g., AIG or Geithner Senator Robert Menendez, a
New Jersey Democrat who
claimed that several weeks
earlier he warned Timothy
Geithner, the Treasury
secretary, that AIG was
planning to use taxpayer funds
to pay out $165m in bonuses…
speculation that Obama will
have to replace him, despite
the president’s insistence to
Leno that Geithner is doing "an
outstanding job“.

18. Sentiment Analysis at Phrase Level
Fine-grained opinion analysis
Identify who is saying what about what

20. Fine-Grained Sentiment Extraction
The suggestion that the White House never took
seriously an issue that infuriated millions of Americans
was supported by Senator Robert Menendez, a New
Jersey Democrat who claimed that several weeks
earlier he warned Timothy Geithner, the Treasury
secretary, that AIG was planning to use taxpayer
funds to pay out $165m in bonuses… speculation that
Obama will have to replace him, despite the
president’s insistence to Leno that Geithner is doing
"an outstanding job".

21. Fine-Grained Sentiment Extraction
…the president insisted to Leno that Geithner is doing "an
outstanding job".
– Opinion trigger
– Polarity Opinion Frame
– Intensity Polarity: positive
– Opinion holder Intensity: high
Opinion Holder: “the president”
– Target (topic) Target: “Geithner”

22. Example – fine-grained opinions
opinion frame
opinion frame
opinion frame
opinion frame opinion frame
opinion frame
The suggestion that the White House never took
seriously an issue that infuriated millions of
Americans was supported by Senator Robert
Menendez, a New Jersey Democrat who claimed
that several weeks earlier he warned Timothy
Geithner, the Treasury secretary, that AIG was
planning to use taxpayer funds to pay out $165m in
bonuses…the president insisted to Leno that
Geithner is doing "an outstanding job".
opinion frame

23. Example – Opinion Summary
AIG
Obama
Geithner
Americans
Menendez

24. Example – Opinion Summary
Summarize thoughts and views across
documents
– Critical addition: opinion holder
AIG

25. What makes this hard?
Same issues of ambiguity as before plus…
Need to associate opinion with topic and
with opinion holder
Requires different machine learning
methods
Requires many language-processing
modules

26. Noun Phrase Coreference Resolution
The suggestion that the White House never took
seriously an issue that infuriated millions of Americans
was supported by Senator Robert Menendez, a New
Jersey Democrat who claimed that several weeks
earlier he warned Timothy Geithner, the Treasury
secretary, that AIG was planning to use taxpayer
funds to pay out $165m in bonuses…speculation that
Obama will have to replace Geithner, despite
the president’s insistence to Leno that he is
doing "an outstanding job".
Ng & Cardie [2002, 2003]; Stoyanov & Cardie [2006, 2008]

27. Performance
82F opinion OH 79F
extraction extractor
69F link
classifier
–<opinion holder> expresses <opinion>
Choi, Breck & Cardie [2006, 2007]

28. Plan for the Talk
Subjectivity and sentiment in language
Continuum of capabilities
– Surface-level in-depth understanding
– Document-level phrase-level
Next steps…

29. Next Steps…
Predicting business outcomes from
opinions
– Doable in some settings
Determining the key influencers

30. Thank you!
Questions?