Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Trend detection and analysis on Twitter

3.908 Aufrufe

Veröffentlicht am

By Henning Muszynski, Benjamin Räthlein & Lukas Masuch

The popularity of social media services has increased exponentially in the last few years. The combination of big social data and powerful analytical technologies makes it possible to gain highly valuable insights that otherwise might not be accessible. The Twitter Analyzer comprises several components to collect, analyze and visualize Twitter data. Therefore, we explored various related technologies to implement this tool. We collected about 38 million english tweets related to various and analyzed those data with machine learning techniques to compute the respective sentiment and detect common topics. Furthermore, we visualized the results using varying visualization techniques to emphasize different aspects such as a wordcloud, several chart-types and geospatial visualizations. Used technologies: MongoDB, Python, Twython, Python NLTK, wordcloud2.js, wordfreq, amCharts, Google BigQuery, Google Cloud Storage, CartoDB, EtcML.

Veröffentlicht in: Daten & Analysen
  • @竞择 毕 we gathered a huge amount of tweets over several weeks by using the twitter streaming API
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier
  • Perfect job! May I know how you got historical data from Twitter, while there is 7-days limitation?
       Antworten 
    Sind Sie sicher, dass Sie …  Ja  Nein
    Ihre Nachricht erscheint hier

Trend detection and analysis on Twitter

  1. 1. Trend Detection and Analysis on Twitter
  2. 2. 2 Agenda Motivation Architecture Data Preparation Trend Analysis Analyzed Trends Conclusion
  3. 3. 3 Motivation Predict the stock market in real time source source Detecting influenza epidemics Automatic crime prediction source “Successful results of mainly research-based projects helped to open up new business opportunities”
  4. 4. 4 Twitter
  5. 5. 5 Early Trend Detector Bag-of-words (Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Bag of Words Bags Count #newyear 7 #christmas 6 @bigdata 2 @sap 3
  6. 6. 6 Statistical MeasurementEarly Trend Detector Bag-of-words (Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Statistical Measurement (growth, average usage, retweets, participating users…) Report statistics (every 20 minutes): • Total hashtags & user mentions • Hashtag/mentions count • Usage growth per hashtag/mention • Participating users per hashtag/mention • Retweet count per hashtag/mention
  7. 7. 7 Early Trend Detector Bag-of-words (Hashtags, Mentions) Twitter Streaming API (Twython) Architecture Statistical Measurement (growth, average usage, retweets, participating users…) Anomaly Detection Time Series Analysis Calculated for every hashtag / user mention Every 2 / 4 hours based on reports Anomaly detection using: • Relative & absolute fluctuation • Total occurrences (sum) • Minimum occurrences • Maximum occurrences • Average occurrences Time Series Analysis
  8. 8. 8 Twitter Streaming API (Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) Lowercasing & tokenizing URL & stopword removal Stop Word Removal This sample text shows which words will be removed when applying stop word removal. Mostly words like the, a or and. This sample text shows which words will be removed when applying stop word removal. Mostly words like the, a or and.
  9. 9. 9 Twitter Streaming API (Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Stemming Amazing Amazement Amazed amaze
  10. 10. 10 Twitter Streaming API (Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Sentiment Analysis I love cookiesI hate cookies
  11. 11. 11 Twitter Streaming API (Twython) Architecture Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Topic Modeling (LDA) Topic Modeling Topics • … • … • … Trend Classification
  12. 12. 14 Trend Analyzer Text Preprocessing (Python NLTK) URL & stopword removal Lowercasing & tokenizing Word stemming Sentiment Analysis Topic Modeling (LDA) Wordcloud Visualization Wordfreq.js Wordcloud2.js GeoSpatial Visualization CartoDB Early Trend Detector Bag-of-words (Hashtags, Mentions) Anomaly Detection Statistical Measurement (growth, average usage, retweets, participating users…) Time Series Analysis Trend Classification Twitter Streaming API (Twython) Architecture
  13. 13. 15 Analyzed Trends
  14. 14. 16 Limitations Tweets collected: 38 million (70GB) Only English tweets from the USA Twitter Streaming API
  15. 15. 17 New Year Time Series
  16. 16. 18 New Year Word Cloud
  17. 17. 19 New Year Geospatial Analysis Midnight Los Angeles Midnight New York
  18. 18. 20 New Year Sentiment Analysis Positive Neutral Negative Home sick on #nye. Horrible timing stupid cold. Ugh. My date is my couch & pillow watching. #HappyNewYear everyone. #HappyNewYear from the Youth for Astronomy and Engineering Program at Space Telescope Science Institute! Happy New Year! Last year was amazing, and here’s to another great year of love & happiness! #NYE2015
  19. 19. 21 Air Asia Tragedy
  20. 20. 22 Air Asia Tragedy Time Series
  21. 21. 23 Air Asia Tragedy Word Cloud
  22. 22. 24 Air Asia Tragedy Topic Modeling News airasia, missing, flight, air, Indonesia, singapore, asia Search for the Plane airasia, missing, plane, find, plane, world, technology Sympathy Prayers, families, thoughts, airasia, crash, thought, airfrance Cause airasia, weather, flight, pilots, fly, bad, path International Help raaf, butterworth, china, australia, Russia, trndnl, trending
  23. 23. 25 Air Asia Tragedy Sentiment Analysis Neutral Negative Positive Prayers are USELESS! Stop repeating meaningless crap, pretending that you care … #PrayForAirAsia #QZ8501 #GrowABrain #ReligousNonsense #BREAKING #AirAsia Flight #8501 likely “at the bottom of the sea” rescue officials says. May God’s great love shine on the families and loved ones of all passengers and crew #AirAsia #8501
  24. 24. 26 Air Asia Tragedy Google Trends Comparison Google Trends Twitter Sample
  25. 25. 27 Air Asia Tragedy Google Trends Comparison Google Trends Twitter Sample
  26. 26. 28 Sony Hack
  27. 27. 29 Sony Hack Time Series
  28. 28. 30 Sony Hack Word Cloud
  29. 29. 31 Sony Hack Topic Modeling Christmas Release theinterview, christmas, day, theaters, freedom, theater, showing Reviews theinterview, jamesfrancotv, sethrogen, movie, interview, funny, hilarious Suspicions northkorea, sonyhack, korea, north, internet, sony, amp News theinterview, sonypictures, sony, movie, korea, north, interview Insider Joke theinterview, aint, hate, cuz, jealous, anus, peanutbutter
  30. 30. 32 Sony Hack Geospatial Analysis
  31. 31. 33 Sony Hack Sentiment Analysis Neutral Negative Positive #TheInterview SUCKS!!! @sethrogen Like I knew it would #Stupid #NotFunny #Sony says #TheInterview made more than $1 million at the box office on in 1 single day on Dec. 25. Happy I joined my fellow Americans in the great #TheInterview Christmas Day Viewing. Plus it was pretty funny, truth be told.
  32. 32. 34 Network Outage
  33. 33. 35 Network Outage Time Series
  34. 34. 36 Network Outage Word Cloud
  35. 35. 37 Network Outage Topic Modeling Network Error xbox, psn, sign, connect, live, error, account, issues Connection between Hacks xbox, playstation, watch, movie, fuckcrucifix, north, korea, interview Xbox Down xbox, christmas, play, xboxlivedown, live, xboxlive, xboxsupport, day Caused Damage playstation, dollar, psn, company, lizardsquad, sony, billion, multi Hacker Group fuckcrucifix, lizardmafia, lizardsquad, fuck,lizard, squad, finestsquad, stop Restored psn, back, playstation, online, askplaystation, network, psndown, working
  36. 36. 38 Network Outage Sentiment Analysis Neutral Negative Positive @XboxSupport f*** your servers, a big ass company like you should handle these teenage kids, terrible @AskPlayStation when will the service be back online because it says there’s maintenance? @PlayStation thanks for the great year. I am sure this new year will be amazing. Don’t allow yourselves to be hacked ever again.
  37. 37. 39 Conclusion High quality insights into world’s interest Twitter is very good for detecting and predicting trends Maintaining a high data quality is important
  38. 38. 40 #Questions Benjamin Räthlein @B3nRa Henning Muszynski @henningmus Lukas Masuch @LukasMasuch

×