This document discusses extracting useful information from social media messages during disasters. It outlines filtering disaster-related tweets, classifying them by type (e.g. caution/advice, casualties), and extracting key information within tweets (e.g. locations, needs). The approach is demonstrated on datasets from the 2011 Joplin tornado and 2012 Hurricane Sandy. Automatic classification achieves over 80% accuracy for some classes. Information extraction obtains up to 90% precision. Ongoing work includes providing these tools as a machine learning service to help during crises.

1. Extracting Information Nuggets from
Disaster-Related Messages in Social Media
Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, Patrick Meier

2. Outline
• Social media response to disaster
• Finding tactical and actionable information
• Disaster ontologies
• Filtering, classification and extraction
• Ongoing work
• Discussion

3. Disaster and Social Media
2.3 million tweets reflecting the words “Haiti”
or “Red Cross” from Jan 12 to Jan 14, 2010
http://www.sysomos.com

4. Disaster and Social Media

5. Why Social Media?
• Virtual Collaboration, Information Sharing
• Highly valuable information
• Contribute to situational awareness
• Highly useful, if analyzed timely and
effectively

6. Sandy Tweets
@NYGovCuomo orders closing of NYC bridges. Only Staten Island
bridges unaffected at this time. Bridges must close by 7pm. #Sandy
#NYC.
rt @911buff: public help needed: 2 boys 2 & 4 missing nearly 24 hours
after they got separated from their mom when car submerged in si.
#sandy #911buff
freaking out. home alone. will just watch tv #Sandy #NYC.
400 Volunteers are needed for areas that #Sandy destroyed.

7. Sandy Tweets
@NYGovCuomo orders closing of NYC bridges. Only Staten Island
bridges unaffected at this time. Bridges must close by 7pm. #Sandy
#NYC.
rt @911buff: public help needed: 2 boys 2 & 4 missing nearly 24 hours
after they got separated from their mom when car submerged in si.
#sandy #911buff
freaking out. home alone. will just watch tv #Sandy #NYC.
400 Volunteers are needed for areas that #Sandy destroyed.
Personal
Informative

8. Sandy Tweets
@NYGovCuomo orders closing of NYC bridges. Only Staten Island
bridges unaffected at this time. Bridges must close by 7pm. #Sandy
#NYC.
rt @911buff: public help needed: 2 boys 2 & 4 missing nearly 24 hours
after they got separated from their mom when car submerged in si.
#sandy #911buff
freaking out. home alone. will just watch tv #Sandy #NYC.
400 Volunteers are needed for areas that #Sandy destroyed.
Personal
Informative
Caution and Advice
Casualties and Damage
Donations

9. Sandy Tweets
@NYGovCuomo orders closing of NYC bridges. Only Staten Island
bridges unaffected at this time. Bridges must close by 7pm. #Sandy
#NYC.
rt @911buff: public help needed: 2 boys 2 & 4 missing nearly 24 hours
after they got separated from their mom when car submerged in si.
#sandy #911buff
freaking out. home alone. will just watch tv #Sandy #NYC.
400 Volunteers are needed for areas that #Sandy destroyed.
Personal
Informative
Caution and Advice
Casualties and Damage
Donations

10. Finding Tactical & Actionable Information
Personal
Informative
(Direct & Indirect)
Other
Caution and advice
Casualties and damage
Donations
People missing, found, or seen
Information source
Siren heard, warning issued/lifted etc.
People dead, injured, damage etc.
Money, shelter, blood, goods, or services
Webpages, photos, videos information sources
…

11. Our Approach
3.
Extraction
2.
Classification
1.
Filtering

12. Our Datasets
Joplin Dataset
• 206,764 tweets collected during Joplin tornado
that hit Joplin, Missouri on May 22, 2011
• Collected by researchers at the university of
Colorado at Boulder
• Collected through Twitter API by monitoring the
tweets with hashtags #joplin or #tornado

13. Our Datasets
Sandy Dataset:
• 140,000 tweets collected during hurricane Sandy
that hit northeastern USA on Oct 29, 2012
• Collected through Twitter API by monitoring the
tweets with hashtag #sandy or #nyc

14. 1. Filtering
Is disaster-
related?
Contributes to
situational
awareness?
Yes Yes
No No

15. 1. Filtering: Training Data
32%
60%
8%
4406 tweets sampled uniformly from the
Joplin dataset Annotated using CrowdFlower
Personal
Informative
Other

16. 2. Classification
Caution &
Advice
Information
Sources
Damage &
Casualties
Donations
Health
Shelter
Food
Water
Logistics
...
...
Filtered
tweets

17. Distribution of Tweet Types
50%
18%
16%
10%
6%
Caution/Advice
Info Source
Donations
Casualties/Damage
Unknown
Joplin Tornado (2011)

18. Automatic Classification
Class Prec Rec F-Measure AUC
Caution and advice 0.85 0.76 0.80 0.91
Information source 0.54 0.58 0.56 0.76
Donations 0.72 0.71 0.72 0.89
Casualties/damage 0.52 0.65 0.58 0.87
• Binary (hashtags, URL, emotion etc.)
• Scalar (tweet length)
• Text features (Unigram, bigram, POS tags, Verbnet etc.)
Features:

19. 3. Extraction
...
Classified
tweets
@JimFreund: Apparently we have no choice.
There is a tornado watch in effect
tonight.

20. Labels for Extraction: Training Data
• Type-dependent instruction
• Ask evaluators to copy-paste a word/phrase
from each tweet

21. Tool
• CMU ARK Twitter NLP
– Tokenization
– Feature extraction
– CRF learning
• Very easy to use: simply change the training
set (part-of-speech tags) into anything, and re-
train

22. Extraction Evaluation
Setting Rec Prec
Train 2/3 Joplin, Test 1/3 Joplin 78% 90%
Train 2/3 Sandy, Test 1/3 Sandy 41% 79%
Train Joplin, Test Sandy 11% 78%
Train Joplin + 10% Sandy, Test 90% Sandy 21% 81%
• Precision is: one word or more in common with
what humans extracted (Imran et al., 2013)

23. Ongoing work

24. Self-service for crisis-related classification
• Machine learning software can be provided as
a service
– e.g. Google Prediction API
• Can we provide crisis-related tweet
classification as a service?
– Automatic collection of tweets
– Re-usable ontologies / default training sets
– Active learning

25. Request Labeled / Unlabeled Datasets
Contact us at: mimran@qf.org.qa

26. References
• K. Starbird, L. Palen, A. Hughes, and S. Vieweg (2010) Chatter on the red: what hazards
threat reveals about the social life of microblogged information. In Proceedings of the 2010
ACM conference on Computer supported cooperative work, pages 241–250. ACM.
• Latonero, Mark, and Irina Shklovski. "“Respectfully Yours in Safety and Service”: Emergency
Management & Social Media Evangelism." Proceedings of the 7th International ISCRAM
Conference–Seattle. Vol. 1. 2010.
• Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz and Patrick Meier.
Practical Extraction of Disaster-Relevant Information from Social Media. WWW-2013
SWDM, May 2013

27. Thank you!
Muhammad Imran
mimran@qf.org.qa
With thanks to Carlos Castillo for several slides