Discovering Themes and Groups in Political Text Data Using Clustering

© 2015 Lexalytics Inc. All rights reserved
Discovery++
Clustering + Text Analytics
Seth Redmore; CMO, Lexalytics, Inc.
@sredmore
Paul Barba, Senior Architect, Lexalytics, Inc.

Agenda
 Who is Lexalytics
 What’s our stack looks like
 How to fit Machine Learning and Text Analytics together
 Text and its annoying challenges
 Clustering and extraction process
 Interesting results
 What else could we have done?
 Human/Computer Partnership
2

Who is Lexalytics? 3
• Founded in 2003
• Text Analytics Engine
– Entities, Sentiment, Themes, Summaries, Intentions, Categories
• On-Premise, SaaS, Desktop
• Popular in Social Listening, Customer Experience Mgmt.
• Billions of documents/day processed across our customers
• Hybrid approach to text analytics using machine learning, natural language processing algorithms, pattern files,
and dictionaries
• Fun fact: We maintain almost 40 different machine learning models

Layers of Interpretation: Transparent Deep Learning
Sentence
Breaking
Tokenization Lexical
Chaining
PoS Chun
k
Syntax
Base Knowledge
Syntax
Matrix
i
Vertical
Optimization
,
Concept
Matrix
Multi-layered
Text Deconstruction
(Text Preparation)
IntentionsThemesEntities
Feature Extraction
Sentiment
+/-
Summaries
3
Categories
4

The Discovery Problem vs. The Prediction Problem 5
• Two obvious ways to integrate NLP and
Machine Learning
• Learn, then NLP  Discovery
• NLP first, then Learn Predictions
• We decided to give the first one a try, as
that’s often the first question an analyst
needs to know about text.
• “Ok, I just got 500k tweets dumped on
me and I need to understand what’s up.”
• Once some degree of “importance” is
measured, then easy to integrate into
predictive models
vs.

Text and why it’s annoying 6
• Medium dimensionality
– As compared to:
• Video: gazillions of dimensions
• Netflix rating data:
– Lots of users, bunch of movies dimensions, also sparse
– Users * Movies
• But very sparse across those dimensions
– Of the say ~100,000-200,000 lemmas that come up with reasonable
frequency, how many are you getting in your corpus?
http://www.oxforddictionaries.com/us/words/the-oec-facts-about-the-language

Discovery Process – Cluster then Extract 7
• Clustering allows us to discover naturally occurring
groupings of text
• Post-clustering, we will then extract the features from the
clusters to see what’s in them
– Terms
– Bigrams
– Trigrams
– Themes
– Entities
– Sentiment
Themes EntitiesSentiment
+/-
+/-
+/-

Themes
House and Senate leaders hatched their plans Thursday to
avoid a politically risky shutdown next week, moving to
separate an acrimonious battle over abortion from a must-
pass bill to keep government agencies open.
After Pope Francis addressed a joint meeting of Congress,
Speaker John Boehner told his leadership team he would
immediately put a plan to defund Planned Parenthood into
legislative vehicle known on Capitol Hill as "reconciliation,"
which cannot be filibustered in the Senate.
The speaker's team argues that by putting the provision in a
reconciliation bill, there's a good chance it will be approved
in both chambers of Congress and it will force Obama to use
his veto pen. It would also allow them to pass a stop-gap
measure free of Planned Parenthood restrictions before the
Oct. 1 deadline to keep the government open.
The move is bound to anger conservatives, and Boehner will
pitch the plan Friday morning to a closed-door conference
meeting.
Extracted themes Sentiment
anger conservatives -2.07
risky shut down -3.82
acrimonious battle -4.50
must-pass bill -2.32
good chance +3.00
Themes example 8

Themes 9
Algorithm
Scoring
Patterns
Candidate
Themes
Tuning
Theme Candidate PoS
Patterns
Scored
ThemesT
Text PrepText

Clustering 10
• H2O supports k-means clustering
• k-means clustering:
– Find n centerpoints upon which the distance between members of the cluster are minimized
(“Within Cluster Sum of Squares” – WCSS)
• k-means can be solved in reasonable time with fixed dimensionality
and number of clusters
• 3 steps:
– Decide what you’re going to cluster on
– Initialize the set – k-means++
– Run some sort of optimized algorithm

Datasets 11
• 2 test datasets:
– ~10k tweets from New Hampshire
that talk about the current
election cycle
– 20,000 tweets from a
Samsung® announcement
• We want to know if there are any
interesting, natural groupings in the
content that we should be aware of.

Challenges in Clustering 12
• Dimensionality vs. Sparseness
• We tried clustering on:
– Terms (single words) (stemmed +
unstemmed)
– Bigrams (stemmed + unstemmed)
– Themes (stemmed and unstemmed)
• Each one got a single mega-cluster
• Data is too high-dimensional and
sparse
– need to reduce dimensions

Reducing Dimensions (and improving sparseness) 13
• Principal Component Analysis (PCA) is native to H2O, so we tried that first.
– PCA reduces dimensionality by first finding the “principle component” that accounts for
the most variability.
– Then, it finds the component that has the next largest variability – with the constraint that
this component must be orthogonal to the first component
– Lather, rinse, repeat
• PCA ran for over a week on the fairly hefty cluster we were given to use, then
went down.
• PCA is thus too slow for this problem

Word2Vec 14
• Word2Vec is an open-source toolset for
– calculating the cosine distance between words
– categorizing words based on a training corpus
• You can train it yourself on your own corpora, or can
use some of the pre-trained Word2Vec models out
there already (see below)
• The cosine distance can be used to reduce the
dimensionality by grouping words into an arbitrary
number of dimensions
• We used 300, because This Is SPARTAAAAA!
– Actually because we used the pre-existing Google model that
had 300 vectors in it already
– https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21p
QmM/edit?usp=sharing
https://code.google.com/p/word2vec/

Clustering on Word2Vec processed content 15
• Yay! We’re not getting one big cluster any more.
• Now we need to figure out how many clusters are optimal
– Remember, we’re just doing discovery here, so, we don’t have to spend a lot of time optimizing
– We tried 8, 30, and 100 clusters

16Politics-30-split: Cluster 14 Size = 305, Sentiment = -0.31
Bigrams
• #alpolitics #iacaucus
• #alpolitics #tennessee
• #alpolitics #ukip
• #alpolitics @anncoulter
• #alpolitics @realdonaldtrump
• #gopdebate #nhpolitics
• #iacaucus #alpolitics
• #iacaucus #ukip
• #iacaucus @anncoulter
• #iacaucus @realdonaldtrump
Trigrams
• #alpolitics @anncoulter #tennessee
• #alpolitics @anncoulter @vdare
• #alpolitics @realdonaldtrump #ukip
• #iacaucus #alpolitics #tennessee
• #iacaucus #alpolitics #ukip
• #iacaucus #alpolitics @anncoulter
• #iacaucus #alpolitics @realdonaldtrump
• #iacaucus @anncoulter #alpolitics
• #iacaucus @anncoulter @vdare
• #iacaucus @realdonaldtrump #ukip
Terms
#GOPDebate
#Immigration
#NHGOP
#TPP
#UKIP
#alpolitics
#fitn
#iacaucus
#immigration
#nhpolitics

17Politics-30: Cluster 14 Size = 305, Sentiment = -0.31
Entities
• Bush(-9.89)
• Marco Rubio(-16.83)
• Hillary Clinton(-10.42)
• Mexico(-0.63)
• @AnnCoulter(-4.44)
• AMNESTY(-8.94)
• @realDonaldTrump(-4.89)
• @BruceBourgoine(-0.23)
• Mass(-0.88)
• Libya(-0.23)
Themes
• open borders mass immigration(1.87500047684)
• wage-reducing mass immigration(-6.61054801941)
• nation-wrecking mass immigration(-4.53758764267)
• alien invaders(-18.000005722)
• legal immigration(-7.35771656036)
• rancid whores(-8.327501297)
• job-killing trade deal scams(-3.09999990463)
• Trans-Pacific Partnership trade deal scam(-0.490000009537)
• multicultural mayhem(-3.75)
• treasonous rat(-3.75)

18Politics-100: Cluster 37, Size = 213, Sentiment = -0.38
Entities
• AMNESTY(-9.45)
• @realDonaldTrump(-5.75)
• @elraymer(0.00)
• @MartinOMalley(0.00)
• Martin O'Malley(-1.14)
• Maryland(-1.00)
• @RefugeeWatcher(0.00)
• @vaughnFNC(-1.20)
• Hillary Clinton(-0.72)
• @CampaignReg(-0.13)
Themes
• illegal aliens(-4.00000047684)
• wage-reducing mass immigration(-2.23903226852)
• nation-wrecking mass immigration(-1.57310771942)
• multicultural mayhem(-2.25)
• alien invaders(-1.20000004768)
• open border(0.450000017881)
• big money donors(0.882178008556)
• sanctuary state(-0.483333349228)
• immigrant crime(-1.20319712162)
• visa foreigners(-0.600000023842)

19Politics-30: Cluster 25 Size = 407, Sentiment = +0.27
Terms
• #11
• #2016election
• #3
• #603forHRC
• #911Anniversary
• #ACEs
• #Bernie2016
• #BernieAtUNH
• #Brooklyn
• #CNN
Bigrams
• #11 candidate
• #603forhrc #hillary2016
• #603forhrc #newhampshire
• #603forhrc @hillaryfornh
• #603forhrc together
• #bernie2016 #feelthebern
• #brooklyn today
• #carly2016 #fitn
• #carly2016 #nhgop
• #carly2016 listen
Trigrams
• #11 candidate i
• #603forhrc #hillary2016 http
• #carly2016 #fitn #nhgop
• #carly2016 #fitn #nhpolitics
• #climateactionnow thank @berniesanders
• #cnndebate stage tonight
• #delay #nh #nhpoli
• #feelthebern #climateactionnow #stopthenhpipeline
• #feelthebern #fitn #nhpolitics
• #fitn #bernie2016 #feelthebern

20Politics-30-split: Cluster 25 Size = 407, Sentiment = +0.27
Entities
• @ThisWeekABC(0.00)
• @donnabrazile(0.27)
• Senator Bernie Sanders(1.81)
• @Women4Bernie(0.00)
• NH(5.36)
• @BernieSanders(4.17)
• @CornelWest(0.00)
• RI Gov Lincoln Chafee(0.00)
• 4(1.10)
• Wheeler Hall(0.00)
Themes
• race car start(0.0)
• town hall meeting(0.0)
• 2nd day(0.487500011921)
• inviting folks(0.24375000596)
• great day(0.40000000596)
• Convention crowd cheers(0.980000019073)
• clear winner(0.490000009537)
• 17 town hall(0.0)
• Living room(0.0)
• state convention(0.0)

Samsung-30 Interesting Clusters (Themes Only)
Cluster 5, Size = 50, Sentiment =+0.28
• Android smartphone profits(8.37637424469)
• filling pre-orders(-2.89171385765)
• supply issues(-2.91585707664)
• global supply shortages(-0.980000019073)
• global rollout(-1.94809389114)
• initial supplies(-1.94912362099)
• mobile device market(5.76794099808)
• global rollout(-0.895333886147)
• Android profits(4.18818712234)
• lion share(3.84502887726)
Cluster 6, Size = 657, Sentiment =+0.32
• Limited edition(2.96772003174)
• 2 cover case leather sleeve brown(0.0)
• Rechargeable Power(0.147000014782)
• waxed leather(0.0)
• Cheap price(0.32262301445)
• Soft Skin(0.475291997194)
• S line(0.237645998597)
• Assorted Colors(0.118822999299)

What else could we have done? 22
• Different cluster sizes
• Semantic meaning of the themes for associations
• Pre-sorting based on queries of candidates, or topical queries
• Gathering other examples for comparison
• Building queries to pull out common items.
– Look to see which clusters it’s appearing in, is it across all the clusters?
• Demographic data, Klout scores

Human/Computer Partnership 23
Loop if broken
Text Content
Entities
Sentiment
Themes
Categories
Reduce
Dimensions
Cluster Extract Examine
Pick
one
lens
Loop though to dive into one area or segment text by feautures, then classify
E.G. “What are the clusters for each of the candidates
or
“I built classifiers for – Solar Power
– Fossil Fuels
– Wind”

Summary 24
• Text Analytics relies heavily on machine learning to do its job
• Text Analytics can come before other Machine Learning for predictive analysis
• Machine Learning can come before Text Analytics for discovery processes
• Reducing dimensionality of the text is an important step because of the sparse nature of the
matrix
• PCA was unsuitable (we used Word2Vec, but Sparse PCA might work as well)
• For discovery, an interesting process is to loop, taking a lens built from the first run (entities,
categories, etc), and then going back to step one and looking at the related clusters for that
lens

Thanks!
• H2O for providing us with all the processing power we needed and excellent technical
support and tools. We were very impressed with their responsiveness and
professionalism
• Paul Barba for doing the heavy lifting
• Y’all for listening 
• Happy Diwali Everyone!

Discovering Themes and Groups in Political Text Data Using Clustering

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (19)

Ähnlich wie Discovering Themes and Groups in Political Text Data Using Clustering

Ähnlich wie Discovering Themes and Groups in Political Text Data Using Clustering (20)

Mehr von Sri Ambati

Mehr von Sri Ambati (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Discovering Themes and Groups in Political Text Data Using Clustering

Hinweis der Redaktion