The document discusses techniques for identifying, characterizing, and selecting social media content related to events. It presents methods for identifying trending events using online clustering and classification, learning similarity metrics for clustering social media documents, identifying content for planned events using query formulation strategies, and selecting high-quality event content using centrality-based techniques. The techniques were shown to effectively identify trending events, leverage cross-site content to find planned events, and pick relevant and useful documents for events. Areas of future work include optimizing the clustering framework and exploring blocking techniques and topic models.

1. Identification and Characterization of Events in Social Media Hila Becker, Thesis Defense

2. Social Media is Changing the World 2 Lady Gaga, Justin Bieber, and Britney Spears have more Twitter followers than the entire populations of some countries (e.g., Israel, Greece) YouTube is the second largest search engine in the world Every minute, 24 hours of video are uploaded to YouTube Over the past five years people uploaded 6,000,000,000 images to Flickr

3. 3 Source: http://www.searchenginejournal.com/the-growth-of-social-media-an-infographic/

4. Event Content in Social Media 4

5. 5 MIKE CLARKE/AFP/Getty Images

6. 6 Source: Tweets from Tahrir, edited by Nadia Idle and Alex Nunns

7. 7

8. Event Identification, Characterization, and Content Selection Identify events and their associated social media documents In a timely manner Across different social media sites Characterize events along different dimensions Select high-quality, relevant, useful event documents 8

9. Event Content in Social Media Challenges: Wide variety of topics, not all related to events (e.g., personal status updates, every-day mundane conversations) Unconventional text: abbreviations, typos Large-scale, rapidly produced content Opportunities: Content generated in real-time, as events happen Rich context features (e.g., time, location) Users’ perspective 9

10. Event Content in Social Media 10 Timeliness Real-time Retrospective Twitter new event detection [Petrović et al. NAACL’10] Event detection on Flickr[Chen and Roy CIKM’09] Unknown Content Discovery Organization of YouTube concert videos [Kennedy and Naaman WWW’09] Earthquake prediction using Twitter [Sakaki et al. WWW’10] Known

11. Event Content in Social Media 11 Trending Event isa real-world occurrence described by: One or more terms and a time period Volume of messages posted for the terms in the time period exceeds some expected level of activity Unknown Content Discovery Planned Event is a real-world occurrence with corresponding published event record consisting of: Title, describing the subject of the event The time at which the event is planned to occur Known

12. Contributions Trend (and trending event) study, for characterizing and differentiating between different types of trends Online clustering framework with an event classification step for identifying trending events and their associated documents in social media Social media document similarity metric learning approaches Query formulation strategies for identifying social media documents for planned events Selection techniques for identifying high quality, relevant, and useful event content 12 Unknown Known Unknown/Known

13. Contributions Trend (and trending event) study, for characterizing and differentiating between different types of trends Online clustering framework with an event classification step for identifying trending events and their associated documents in social media Social media document similarity metric learning approaches Query formulation strategies for identifying social media documents for planned events Selection techniques for identifying high quality, relevant, and useful event content 13 Known Unknown/Known

14. Contributions Trend (and trending event) study, for characterizing and differentiating between different types of trends Online clustering framework with an event classification step for identifying trending events and their associated documents in social media Social media document similarity metric learning approaches Query formulation strategies for identifying social media documents for planned events Selection techniques for identifying high quality, relevant, and useful event content 14 Unknown/Known

15. Contributions Trend (and trending event) study, for characterizing and differentiating between different types of trends Online clustering framework with an event classification step for identifying trending events and their associated documents in social media Social media document similarity metric learning approaches Query formulation strategies for identifying social media documents for planned events Selection techniques for identifying high quality, relevant, and useful event content 15

16. Identification and Characterization of Events in Social Media 16 Characterizationof trending events Identification of trending events Similarity metric learning for trending events Identification of content for planned events Selection of event content

17. What Types of Trends Exist in Social Media? Taxonomy of trends Characterization of each trend Manually assigned categories Automatically computed features Analysis of differences between trend types according to each characteristic 17 Trending Events Non-Event Trends

18. Trends Trend: One or more terms and a time period Volume of messages posted for the terms in the time period exceeds some expected level of activity May or may not reflect a real-world occurrence A trending event is a type of trend 18

19. Twitter Content Streams of textual messages Brief content (140 characters) Communicated to network of followers Provide timely reflection of thoughts and interests 19

20. Characterizing Trends on Twitter Collect a set of Twitter trends Burst detection Twitter’s “trending topics” Qualitative analysis: trend taxonomy Quantitative analysis Automatically compute features of each trend and corresponding messages Manually label each trend according to categories introduced by the taxonomy Identify differences between trend categories according to automatically computed features 20

21. Affinity Diagram Method 21

22. Endogenous vs. Exogenous Trends Endogenous Trends: Twitter-centric activities that do not correspond to external events (e.g., a popular post by a celebrity) Exogenous Trends: trending eventsthat originated outside of the Twitter system (e.g., an earthquake) Do exogenous and endogenous trends exhibit different characteristics? 22

23. Characterization of Trends and Trending Events Automatically computed features Content Features Interaction Features Time-based Features Participation Features Social Network Features Compared differences between categories Hypotheses guided by differences in categories according to feature types Performed t-tests for significance analysis 23

24. Contributions of the Study Trends fall into two main categories: exogenous (i.e., trending event) and endogenous (i.e., platform-centric trend) There are significant differences between exogenous and endogenous trends Proportion of messages with URLs Unique hashtag in top 10% of messages Proportion of retweets Reciprocity 24

25. Identification and Characterization of Events in Social Media 25 Characterizationof trending events Identification of trending events Similarity metric learning for trending events Identification of content for planned events Selection of event content

26. Identifying Trending Events Event Clusters Documents Document Clusters 26

27. Identifying Trending Events in Real-Time Order documents by post time Use tf-idf vector representation of textual content Stop word elimination Stemming idf computed over past data Separate tweets by location Focus on tweets from NYC Different locations can be processed in parallel 27

28. Clustering Algorithm Many alternatives possible! [Berkhin 2002] Single-pass incremental clustering algorithm Scalable, online solution Using centroid representation Used effectively for Event identification in textual news [Allan et al. 1998] News event detection on Twitter [Sankaranarayanan et al. 2009] Does not require a priori knowledge of number of clusters Parameters: Similarity Function σ Threshold μ 28

29. Overview of Cluster-based Approach Group similar documents via online clustering Compute statistics of cluster content Top terms (e.g., [earthquake, japan]) Number of documents per hour … Use cluster-level features to identify trendingeventclusters Single feature with threshold (e.g., increase in volume over time-window [Petrovićet al. 2010]) Trained classification model 29

30. Event Classification on Twitter Cluster-level features Social interaction Topic coherence Trending behavior Platform-centric Event classifier Human-annotated training data SVM model (selected during training phase) 30

31. Experimental Setup Classification accuracy Baseline: Naïve Bayes text classification (NB-Text) [Sankaranarayanan et al. 2009] 10-fold cross validation Blind test set of randomly chosen tweets Event surfacing: select top event clusters per hour Baselines Fastest-growing clusters per hour (Fastest) [Petrović et al. 2010] Randomly selected clusters per hour (Random) 5 hours, top-20 clusters per hour 31

32. Identified Events 32 A sample of events identified by our classifiers on the test set

33. Classification Performance (F-measure) RW-Event event classifier is more effective at discriminating between real-world events and rest of Twitter data 33

34. NDCG@K Evaluation 34 Performance of event classifier and baselines for event surfacing task.

35. Identification and Characterization of Events in Social Media 35 Characterizationof trending events Identification of trending events Similarity metric learning for trending events Identification of content for planned events Selection of event content

36. Social Media Document Representation Title Description Tags Date/Time Location All-Text 36 36

37. Social Media Document Similarity Text: cosine similarity of tf-idf vectors (tf-idf version?; stemming?; stop word elimination?) 37 Title A A A B B B Description Time: proximity in minutes Tags time Date/Time Location: geo-coordinate proximity Location All-Text 37

38. Clustering Algorithm Many alternatives possible! [Berkhin 2002] Single-pass incremental clustering algorithm Scalable, online solution Using centroid representation Used effectively for Event identification in textual news [Allan et al. 1998] News event detection on Twitter [Sankaranarayanan et al. 2009] Does not require a priori knowledge of number of clusters Parameters: Similarity Function σ Threshold μ 38

39. Cluster Representation and Parameter Tuning Centroid cluster representation Average tf-idf scores Average time Geographic mid-point Parameter tuning in supervised training phase Clustering quality metrics to optimize: Normalized Mutual Information (NMI) [Amigó et al. 2008] B-Cubed [Strehl et al. 2002] 39

40. Learning a Similarity Metric for Clustering Ensemble-based similarity Training a cluster ensemble Computing a similarity score by: Combining individual partitions Combining individual similarities Classification-based similarity Training data sampling strategies Modeling strategies 40

41. Overview of a Cluster Ensemble Algorithm Ctitle Ensemble clustering solution Consensus Function: combine ensemble similarities Wtitle f(C,W) Wtags Ctags Wtime Ctime Learned in a training step 41

42. Overview of a Cluster Ensemble Algorithm: Combining Partitions Wtitle Ctitle f(C,W) Wtags Ctags Wtime Ctime 42

43. Overview of a Cluster Ensemble Algorithm: Combining Similarities For each document di and cluster cj σCtitle(di,cj)>μCtitle Wtitle f(C,W) Wtags σCtags(di,cj)>μCtags Wtime σCtime(di,cj)>μCtime 43

44. Learning a Similarity Metric for Clustering Ensemble-based similarity Training a cluster ensemble Computing a similarity score by: Combining individual partitions Combining individual similarities Classification-based similarity Training data sampling strategies Modeling strategies 44

45. Classification-based Similarity Metrics Classify pairs of documents as similar/dissimilar Feature vector Pairwise similarity scores One feature per similarity metric (e.g., time-proximity, location-proximity, …) Modeling strategies Document pairs Document-centroid pairs 45

46. Experiments: Alternative Similarity Metrics Ensemble-based techniques Combining individual partitions (ENS-PART) Combining individual similarities (ENS-SIM) Classification-based techniques Modeling: document-document vs. document-centroidpairs Logistic Regression (CLASS-LR), Support Vector Machines (CLASS-SVM) Baselines Title, Description, Tags, All-Text, Time-Proximity, Location-Proximity 46

47. Experimental Setup Datasets: Upcoming >270K Flickr photos Event labels from the “upcoming” event database (upcoming:event=12345) Split into 3 parts for training/validation/testing LastFM >594K Flickr photos Event labels from last.fm music catalog (lastfm:event=6789) Used as an additional test set 47

48. Clustering Accuracy over Upcoming Test Set All similarity learning techniques outperform the baselines Classification-based techniques perform better than ensemble-based techniques 48

49. NMI: Clustering Accuracy over Both Test Sets Upcoming LastFM 49 NMI Similarity learning models trained on Upcoming data show similar trends when tested on LastFM data

50. Identification and Characterization of Events in Social Media 50 Characterizationof trending events Identification of trending events Similarity metric learning for trending events Identification of content for planned events Selection of event content

51. Identifying Content for Planned Events Identify planned event documents given known event information User-contributed planned event records LastFM Events EventBrite Facebook Events Structured features (e.g., title, time, location) Challenging identification scenario Known event information is often inaccurate or incomplete Social media documents are brief and noisy 51

52. Planned Event Record 52 Title Description Date/Time Venue City

53. Approach for Known Identification Scenario Two-step query formulation strategy Precision-oriented queries using known event features Recall-oriented queries using retrieved content from precision-oriented queries Leverage cross-site content Identify event documents on each site individually Use event documents on one site to retrieve additional event documents on a different site 53

54. Query Formulation Strategies Precision-oriented Queries: Combined event record features Phrase, bag-of-words, stop word elimination Examples: [“title”+”venue”], [title-no-stopwords+”city”] Recall-oriented Queries Frequency Analysis Frequent terms in the event’s retrieved content Infrequently found in Web documents Term Extraction 54

55. Leveraging Cross-Site Content Build precision-oriented queries using planned eventfeatures Use precision-oriented queries to retrieve data from: Twitter Flickr YouTube Build recall-oriented queries using data from: Each site individually All sites collectively 55 [title+city] [title+venue] … tweet1 tweet2 tweetn photo1 photo2 photon video1 video2 videon

56. Experimental Settings 60 planned events from EventBrite, LastFM, LinkedIn, and Facebook Corresponding social media documents Retrieved from Twitter, Flickr, and YouTube Ranked according to similarity to event record Techniques Precision: only precision-oriented queries MS: precision- and recall-oriented queries selected using Microsoft n-gram probability score RTR: precision- and recall-oriented queries selected using ratio of document frequency around the time of the event to document frequency in larger time window 56

57. NDCG Performance on Twitter 57 NDCG scores for top-k Twitter documents retrieved by Precision-oriented queries (Precision), and query strategies using Twitter data (Twitter-RTR, Twitter-MS).

58. Cross-Site NDCG Performance 58 NDCG scores for top-k YouTube documents retrieved by Precision-oriented queries (Precision), and query strategies using data from Twitter (Twitter-MS) and YouTube (YouTube MS).

59. Identification and Characterization of Events in Social Media 59 Characterizationof trending events Identification of trending events Similarity metric learning for trending events Identification of content for planned events Selection of event content

60. Event Content Selection 60 Tiger Woods to make a public apology Friday and talk about his future in golf. Tiger woods y'all,tiger woods y'all,ah tiger woods y'all Tiger Woods Apology Tiger Woods Hugs: http://tinyurl.com/yhf4uzw Tiger Woods Returns To Golf - Public Apology http://bit.ly/9Ui5jx Wedge wars upstage Watson v Woods: BBC Sport (blog)

61. Event Content Selection Challenges: Document clusters contain noise Relevant documents might have poor quality text Relevant, high quality documents might not be interesting For each document and a given event evaluate Quality Relevance Usefulness 61

62. Centrality Based Document Selection Centroid Cosine similarity of each document to cluster centroid Degree Documents are nodes Documents are connected if their similarity is above a threshold Compute degree centrality of each node LexRank[Erkan and Radev 2004] Same graph structure as Degree method Central documents are similar to other central documents 62

63. Experimental Methodology: Content Selection 50 event clusters Randomly selected 5 top tweets per event for each: Centroid, Degree, LexRank Labeled on a 1-4 scale Quality: excellent (4) poor (1) Relevance: clearly relevant (4)  not relevant (1) Usefulness: clearly useful (4)  not useful (1) 63

64. Content Selection Results Average scores over all events (out of 4) High quality and relevance (>3) for both Degree and Centroid Centroid only method with high usefulness 64

65. Conclusions Techniques for identifying, characterizing, and selecting social media content for events There are significant differences between types of trends in social media, specifically trending events and non-event trends Trending events and their associated social media documents can be effectively identified using online clustering with: A classification step to separate event and non-event content Social media document similarity metrics for documents with rich context features A two-step query formulation technique is useful for identifying planned events across different social media sites Centrality-based techniques can be used to select high quality, relevant, and useful social media event content 65

66. Future Work Clustering framework optimization Blocking techniques Topic models Identify unknown events with learned similarity metrics across sites Improve breadth of event content Rank events for search and presentation Extension of content selection techniques Learned ranking models 66

67. Publications Hila Becker, Dan Iter, MorNaaman, Luis Gravano, “Identifying Content for Planned Events Across Social Media Sites,” under submission. Hila Becker, Mor Naaman, Luis Gravano, “Beyond Trending Topics: Real-World Event Identification on Twitter,” in Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM’11), short paper. Hila Becker, Mor Naaman, Luis Gravano, “Selecting Quality Twitter Content for Events,” in Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM’11), short paper. Hila Becker, Feiyang Chen, Dan Iter, Mor Naaman, Luis Gravano, “Automatic Identification and Presentation of Twitter Content for Planned Events,” in Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (ICWSM’11), demo paper. Mor Naaman, Hila Becker, Luis Gravano, “Hip and Trendy: Characterizing Emerging Trends on Twitter,” in Journal of the American Society for Information Science and Technology. Hila Becker, MorNaaman, Luis Gravano, “Learning Similarity Metrics for Event Identification in Social Media,” in Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM '10), 291-300. Hila Becker, Bai Xiao, MorNaaman and Luis Gravano, “Exploiting Social Links for Event Identification in Social Media,” in Proceedings of the 3rd Annual Workshop on Search in Social Media (SSM '10), poster paper. Hila Becker, MorNaaman, Luis Gravano, “Event Identification in Social Media,” in Proceedings of the ACM SIGMOD Workshop on the Web and Databases (WebDB '09), 2009. 67

68. Thank You! 68

69. 69