Micropinion Generation: An Unsupervised Approach to Generating Ultra-Concise Summaries of Opinions, WWW '12 This work presents a new unsupervised approach to generating ultra-concise summaries of opinions focusing on three aspects: compactness, representativeness and readability. By: Kavita Ganesan

1. Kavita Ganesan, ChengXiang Zhai & Evelyne Viegas
Go to project page

2. Opinion Summary for iPod Touch
Current methods: Focus on
generating aspect based ratings
for an entity
[Lu et al., 2009; Lerman et al., 2009;..]

3. Opinion Summary for iPod Touch
To know more: read many
redundant sentences

4. Opinion Summary for iPod Touch
To know more: read many
redundant sentences
Structured summaries useful, but
insufficient!

5. Textual opinion summaries:
• big and clear screen
• battery lasts long without wifi
• great selection of apps
………………
Textual summaries can provide more
information…

6.  Represent the major opinions
 Key complaints/praise in text
 Important critical information
 Readable/well formed
 Easily understood by readers
 Compact/Concise
 Can be viewed on all screen sizes
▪ e.g. phone, pda, tablets, dekstops
 Maximize information conveyed

7.  Generate a set of non-redundant phrases:
 Summarizing key opinions in text
 Short (2-5 words) Micropinions
 Readable
Micropinion summary for a restaurant:
“Good service”
“Delicious soup dishes”
 Ultra-concise nature of phrases to allow flexible
adjustment of summaries according to display
constraints

9.  Has been widely studied
[Radev et al.2000; Erkan & Radev, 2004; Mihalcea & Tarau, 2004…]
 Fairly easy to implement
 Problem: Not suitable for concise summaries
 Bias – with limit on summary size
▪ selected sentence/phrase may have missed critical info
 Verbose - may contain irrelevant information
▪ not suitable for smaller devices

10.  Understand the original text and “re-tell” story in a
fewer words  Hard to achieve!
 Some methods require manual effort
[DeJong 1982; Radev & McKeown 1998; Finley & Harabagiu 2002]
 Need to define templates
 Later filled with info using IE techniques
 Some methods rely on deep NL understanding
[Saggion & Lapalme 2002; Jing & McKeown 2000]
 Domain dependent
 Impractical – high computational costs

11.  Goal is to extract important phrases from text
 Traditionally used to characterize documents
 Can be potentially used to select key opinion phrases
 Closest to our goal!
 Problem:
 Only topic phrases may be selected
▪ E.g. battery life, screen size  candidate phrases
▪ Enough to characterize documents, but we need more info!
 Readability aspect is not much of a concern
▪ E.g. “Battery short life” == “Short battery life”
 Most methods are supervised – need training data
[Tomokiyo & Hurst 2003; Witten et al. 1999; Medelyan & Witten 2008]

12.  Unsupervised, lightweight & general approach to
generating ultra-concise summaries of opinions
 Idea is to use existing words in original text to
compose meaningful summaries
 Emphasis on 3 aspects:
 Compactness
▪ summaries should use as few words as possible
 Representativeness
▪ summaries should reflect major opinions in text
 Readability
▪ summaries should be fairly well formed

13. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
subject to
k
mi ss
i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

14. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
Micropinion Summary, M subject to
2.3 very clean rooms
2.1 friendly service k
1.8 dirty lobby and pool mi ss
1.3 nice and polite staff i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

15. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
Micropinion Summary, M subject to
m1 2.3 very clean rooms
m2 2.1 friendly service k
m3 1.8 dirty lobby and pool mi ss
mk 1.3 nice and polite staff
i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

16. Representativeness score of mi
k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
subject to Readability score of mi
k
mi ss
i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

17. Objective function
k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
subject to
k
mi ss
i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

18. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
constraints
subject to
k
mi ss
i 1
Srep (mi) rep
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

19. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
Objective function: Optimize
subject to m
1 2.3 very clean rooms
representativeness & k m2 2.1 friendly service
readability scores mi m
ss 3 1.8 dirty lobby and pool
Ensure: summaries reflect key opinions & mk 1.3 nice and polite staff
i 1
reasonably well formed
S (m ) rep i rep Objective function value
Sread (mi) read
sim( mi , mj ) sim i, j 1, k

20. k
M arg max mi...mk Srep (mi) Sread (mi)
Constraint 1: Maximum
i 1
length of summary.
subject to •User adjustable
k •Captures compactness.
mi ss
i 1
Srep (mi) rep
Sread (mi) read
sim(mi , mj ) sim i, j 1, k

21. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1Constraint 2 & 3: Min
representativeness &
subject to
readability.
k •Helps improve efficiency
mi •Does not affect performance
ss
i 1
Srep (mi) rep
Sread (mi) read
sim(mi , mj ) sim i, j 1, k

22. k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
subject to
k
mi Constraint 4: Max
ss
i 1 similarity between phrases
Srep (mi) • User adjustable
rep
• Captures compactness by
Sread (mi) minimizing redundancies
read
sim(mi , mj ) sim i, j 1, k

23. Ensures representativeness &
readability k
M arg max mi...mk Srep (mi) Sread (mi)
i 1
subject to Capture compactness
k
mi ss
i 1
Srep (mi) rep
Sread (mi) Capture compactness
read
sim( mi , mj ) sim i, j 1, k

24.  Now, we need to know how to compute:
 Similarity scores: sim(mi, mj) ?
 Readability scores: Sread(mi, mj) ?
 Representativeness scores: Srep(mi, mj) ?

25.  Score similarity between 2 phrases
 User adjustable parameter
 Why important?
 Allows user to control redundancy
 E.g. With small devices users may desire summaries with
good coverage of information less redundancies !
 Measure used:
 Standard Jaccard Similarity Measure
 Can use other measures (e.g. cosine) – not our focus

26.  Purpose: Measure how well a phrase
represents opinions from the original text?
 2 properties of a highly representative phrase:
1. Words should be strongly associated in text
2. Words should be sufficiently frequent in text
 Captured by a modified pointwise mutual
information (PMI) function

27. Captures property 1:
Measures strength of
association between
 Original PMI function, pmi(wi,wj)
words
p( wi, wj )
pmi ( wi , wj ) log 2
p( wi ) p( wj )

28.  Original PMI function, pmi(wi,wj)
p( wi, wj )
pmi ( wi , wj ) log 2 Captures property 2:
p( wi ) p( wj ) Rewards well
associated words with
 Modified PMI function, high co-occurrences
pmi’(wi,wj)
Add frequency of
p( wi, wj ) c( wi, wj ) occurrence within
pmi ' ( wi , wj ) log2
p( wi ) p( wj ) a window

29. To compute representativeness of a phrase:
n Take average strength of
1
Srep (w1.. wn) association (pmi’) of each
pmilocal ( wi )
n i 1 word, wi in phrase with C
neighboring words
i C
1
pmilocal ( wi ) [ pmi ' ( wi, wj )]
2C j i C

30. Gives a good estimate of
how strongly associated
To compute representativeness of
the words are in a phrase a phrase:
n
1
Srep (w1.. wn) pmilocal ( wi )
n i 1
i C
1
pmilocal ( wi ) [ pmi ' ( wi, wj )]
2C j i C

31.  Purpose: Measure well-formedness of phrases
 Phrases are constructed from seed words
▪ can have new phrases not in original text
 No guarantee phrases would be well-formed
chain rule to compute
joint probability in terms of
 Our readability scoring: conditional probabilities
(averaged)
 Based on Microsoft's Web N-gram model (publicly available)
 N-gram model used to obtain conditional probabilities of phrases
n
1
S read ( wk ... wn ) log 2 p ( wk |wk q 1... wk )
1
K k q
 Intuition: A phrase is more readable if it occurs more frequently on
the web

32. Example: Readability scores of phrases using tri-gram LM
Ungrammatical Grammatical
“sucks life battery” -4.51 “battery life sucks” -2.93
“life battery is poor” -3.66 “battery life is poor” -2.37

33.  Scoring each candidate phrase is not practical!
 Why?
 Phrases composed using words from original text
 Potential solution space would be Huge!
 Our solution: greedy summarization algorithm
 Explore solution space with heuristic pruning
 Touch only the most promising candidates

34. Input Unigrams Seed Bigrams
….
very very + nice
nice very + clean
very + dirty
place
clean + place Srep > σ
clean clean + room rep
Text to be problem
dirty
dirty + place …
summarized
room …
Step 1: Shortlist Step 2: Form seed
high freq unigrams bigrams by pairing
(count > median) unigrams. Shortlist
by Srep. (Srep > σrep)

35. Higher order n-grams Summary
Candidates + Seed Bi-grams = New Candidates
clean rooms = very clean rooms 0.9 very clean rooms
very clean + clean bed very clean bed 0.8 friendly service
dirty room very dirty room 0.7 dirty lobby and pool
very dirty + = 0.5 nice and polite staff
dirty pool very dirty pool
…..
very nice + nice place = very nice place …..
nice room very nice room
Sorted Candidates
Srep<σrep ; Sread<σread
Step 3: Generate higher order n-grams. Step 4: Final summary.
• Concatenate existing candidates + seed bigrams Sort by objective
• Prune non-promising candidates (Srep & Sread) function value. Add
• Eliminate redundancies (sim(mi,mj)) phrases until |M|< σss
• Repeat process on shortlisted candidates
(until no possbility of expansion)

37.  Product reviews from CNET (330 products)
 Each product has minimum of 5 reviews
CNET REVIEW
xxxxxx
PROS
Content Summarized
CONS
FULL REVIEW

38. Given
Textual reviews about a product
Constraints
Micropinion Summary, M
m1 2.3 easy to use summary size, σss
Task m2 2.1 lense is not clear redundancy, σsim
m3 1.8 too big for pocket
mk 1.3 expensive batteries representativeness, σrep
readability , σread
argmax Srep(mi) + Sread(mi)

39.  Human composed summaries
 Two human summarizers
▪ Each summarize 165 product reviews (total 330)
 Top 10 phrases from pros & cons provided as hints
▪ Hints help with topic coverage  reduce bias
 Summarizers asked to compose a set of short
phrases (2-5 words) summarizing key opinions on
the product

40.  3 representative baselines:
 Tf-Idf – unsupervised method commonly used for key phrase
extraction tasks
▪ Selected only adjective containing n-grams (performance reasons)
▪ Redundancy removal used to generate non-redundant phrases
 KEA – state of the art supervised key phrase extraction
model [witten et al. 1999]
▪ Uses a Naive Bayes model
▪ Trained using 100 review documents withheld from dataset
 Opinosis – unsupervised abstractive summarizer designed to
generate textual opinion summaries [ganesan et al. 2010]
▪ Shown to be effective in generating concise opinion summaries
▪ Designed for highly redundant text

41.  ROUGE - to determine quality of summary
(ROUGE-1 & ROUGE-2)
 Standard measure for summarization tasks
 Measures precision & recall of overlapping units
between computer generated summary & human
summaries

42.  Questionnaire – to assess potential utility of generated
summaries to users
 Answered by 2 assessors
 Original reviews provided as reference
Questionnaire
[1] Very Poor - [5] Very Good
Grammaticality Non-redundancy Informativeness
[DUC 2005] [DUC 2005] [Filippova 2010]
Do the phrases
Are the phrases Are the phrases in convey important
readable? the summary information about
unique? the product?

43.  Our approach is referred to as WebNGram

44. WebNgram: Performs
0.09 the best for this task
0.08
0.07 Opinosis: much better
ROUGE-2 RECALL
0.06 than KEA & tfidf
0.05 KEA: slightly KEA
0.04 better than tfidf Worst
Tfidf: Tfidf
0.03 performance
Opinosis
0.02 WebNGram
0.01
0.00
5 10 15 20 25 30
Summary Size (max words)

45.  Intuition: Well-formed phrases tend to be
longer in general
very clear screen vs. very clear
good battery life vs. good battery
screen is bright vs. screen is
 A few longer phrases is more desirable than
many fragmented (i.e. short) phrases

46. KEA: Generates most # of
12 phrases (i.e. favors short
11.75
phrases)
11 WebNGram: Generates
fewest phrases on average.
10
Average # of Phrases
9.86 (each phrase is longer)
9 WebNgram
8.74
8 Tfidf
7.36 Opinosis
7
Kea
6 6.04
5
4.53
4 WebNGram phrases are generally more
15 well-formed 30
25
Summary Size (max words)

47.  In our algorithm: n-grams are generated from
seed words
 potential of forming new phrases not in original text
 Why not use existing n-grams?
 With redundant opinions, using seen n-grams may
be sufficient
 Performed a run by forcing only seen n-grams to
appear as candidate phrases.

48. 0.09 System generated
0.08 n-grams
0.07
Seen n-grams
ROUGE-2 Recall
0.06
0.05
0.04 WebNGram
0.03 WebNGramseen
0.02
0.01
0.00
Our search 15 20 25 30 discover
5 10 algorithm helps
Summary Size (max words)
useful new phrases

49. Example:
Unseen N-Gram (Acer AL2216 Monitor)
“wide screen lcd monitor is bright”
readability : -1.88
representativeness: 4.25
“…plus the monitor is very bright…” Related
“…it is a wide screen, great color, great quality…” snippets in
“…this lcd monitor is quite bright and clear…” original text

50.  Purpose of σrep & σread:
 Control minimum representativeness & readability
 Helps prune non-promising candidates  improves
efficiency of algorithm
 Without σrep & σread - we would still arrive at a
solution, however:
 Time to convergence would be much longer
 Results could be skewed
 These parameters need to be set correctly!

51. ROUGE-2 scores at ROUGE-2 scores at
various σread settings various σrep settings
0.06 0.040
0.04
ROUGE-2
ROUGE-2
0.035
0.02
0.00 0.030
-4 -3 -2
σread -1 1 2 3
σrep 4 5
Performance is stable except in extreme
conditions - thresholds are too restrictive

52. ROUGE-2 scores at ROUGE-2 scores at
various σread settings various σrep settings
0.06 0.040
0.04
ROUGE-2
ROUGE-2
0.035
0.02
0.00 0.030
-4 -3 -2
σread -1 1 2 3
σrep 4 5
Ideal setting:
σread : between -2 and -4
σrep : between 1 and 4

53. Questionnaire [Score 1-5]
Score < 3  poor
Score > 3  good
Grammaticality Non-redundancy Informativeness
[DUC 2005] [DUC 2005] [Filippova 2010]
WebNgram 4.2/5 WebNgram 3.9/5 WebNgram 3.2/5
TfIDF 2.0/5 TfIDF 2.3/5 TfIDF 1.7/5
Human 4.7/5 Human 4.5/5 Human 3.6/5

54. Questionnaire [Score 1-5]
Score < 3  poor
Score > 3  good
Grammaticality Non-redundancy Informativeness
[DUC 2005] [DUC 2005] [Filippova 2010]
WebNgram 4.2/5 WebNgram 3.9/5 WebNgram 3.2/5
TfIDF 2.0/5 TfIDF 2.3/5 TfIDF 1.7/5
Human 4.7/5 Human 4.5/5 Human 3.6/5
TFIDF: Poor scores on all 3 aspects
(all below 3.0)

55. Questionnaire [Score 1-5]
Score < 3  poor
Score > 3  good
Grammaticality Non-redundancy Informativeness
[DUC 2005] [DUC 2005] [Filippova 2010]
WebNgram 4.2/5 WebNgram 3.9/5 WebNgram 3.2/5
TfIDF 2.0/5 TfIDF 2.3/5 TfIDF 1.7/5
Human 4.7/5 Human 4.5/5 Human 3.6/5
WebNGram: All scores above 3.0
Close to human scores

56. Questionnaire [Score 1-5]
Score < 3  poor
Score > 3  good
Grammaticality Non-redundancy Informativeness
[DUC 2005] [DUC 2005] [Filippova 2010]
WebNgram 4.2/5 WebNgram 3.9/5 WebNgram 3.2/5
TfIDF 2.0/5 TfIDF 2.3/5 TfIDF 1.7/5
Human 4.7/5 Human 4.5/5 Human 3.6/5
Subjective aspect. Human
summaries scored slightly
better than WebNgram

57.  Semantic redundancies
“Good sound quality” ≠ “Excellent audio”  semantically similar
 Due to directly using surface words
 Solution: opinion/phrase normalization before
summarizing
 Some phrases are not opinion phrases
“I bought this for Christmas”  grammatical & representative
 Due to our candidate selection strategy
▪ Plus Side: Very general approach, can summarize any text
▪ Down Side: Summaries may be too general
 Solution: stricter selection of phrases
▪ E.g. Select only opinion containing phrases

58. Canon Powershot SX120 IS
Easy to use
Good picture quality
Crisp and clear
Good video quality
Useful for pushing opinions
to devices where the screen
is small
E-reader/ Smart Cell Phones
Tablet Phones

59.  Optimization framework to generate ultra-concise
summaries of opinions
 Emphasis: representativeness, readability & compactness
 Evaluation shows our summaries are:
 Well-formed and convey essential information
 More effective than other competing methods
 Our approach is unsupervised, lightweight &
general
 Can summarize any other textual content
(e.g. news articles, tweets, user comments, etc.)