SlideShare ist ein Scribd-Unternehmen logo
1 von 28
Downloaden Sie, um offline zu lesen
Using	
  Twi)er	
  Data	
  to	
  Predict	
  Flu	
  Outbreak	
  
Son	
  Doan	
  

Division	
  of	
  Biomedical	
  Informa2cs	
  
University	
  of	
  California	
  San	
  Diego	
  

	
  
BigData@UCSD	
  workshop	
  
Nov	
  25,	
  2013	
  
Seasonal	
  influenza	
  and	
  influenza-­‐like	
  illness	
  
•  Seasonal	
  influenza	
  is	
  a	
  major	
  public	
  health	
  concern:	
  
•  3-­‐5	
  million	
  cases	
  of	
  severe	
  illness	
  	
  
•  250,000	
  to	
  500,000	
  deaths	
  worldwide	
  	
  each	
  year	
  
	
  
•  Seasonal	
  influenza	
  has	
  main	
  syndrome	
  called	
  Influenza-­‐
Like	
  Illness	
  (ILI)	
  
•  During	
  the	
  peak	
  of	
  a	
  major	
  outbreak	
  of	
  influenza,	
  more	
  
cases	
  of	
  ILI	
  are	
  observed	
  
à	
  Monitoring	
  ILI	
  can	
  help	
  in	
  predict	
  flu	
  outbreak	
  	
  	
  
	
  
Tradi?onal	
  system	
  to	
  monitor	
  ILI:	
  ILINet	
  
	
  
•  ILINet:	
  CDC’s	
  U.S.	
  Outpa2ent	
  ILI	
  Surveillance	
  
Network	
  

–  consists	
  of	
  	
  >3,000	
  outpa2ent	
  healthcare	
  providers	
  	
  
–  all	
  50	
  US	
  states	
  and	
  area	
  
–  reports	
  more	
  than	
  30	
  million	
  pa2ent	
  visits	
  each	
  year	
  

•  ILINet	
  monitors	
  influenza	
  through	
  ILI	
  rate	
  	
  
–  ILI	
  rate	
  is	
  percentage	
  of	
  pa2ents	
  with	
  ILI	
  among	
  all	
  
pa2ents	
  
–  Average	
  na2onal	
  baseline	
  ILI	
  rate	
  for	
  2013	
  is	
  2.0%	
  
Source:	
  hVp://www.cdc.gov/flu/weekly/index.htm	
  
Let’s	
  revisit	
  the	
  process	
  
	
  	
  	
  Pa2ent	
  1	
  

	
  	
  	
  Pa2ent	
  2	
  

visits	
  

Healthcare	
  
provider	
  

Check	
  if	
  ILI	
  

visits	
  

Healthcare	
  
provider	
  

Check	
  if	
  ILI	
  

Healthcare	
  
provider	
  

Check	
  if	
  ILI	
  

…	
  

	
  	
  	
  Pa2ent	
  n	
  

visits	
  

ILINet	
  gather	
  
data	
  and	
  then	
  
calculate	
  ILI	
  rate	
  
ILINet	
  issue	
  
ILINet	
  needs	
  1-­‐2	
  weeks	
  to	
  gather	
  and	
  process	
  
data	
  

Can	
  we	
  leverage	
  other	
  data	
  sources	
  
to	
  predict	
  ILI	
  rate	
  faster?	
  
Nowadays,	
  users	
  tend	
  to	
  find	
  informa?on	
  in	
  Internet	
  

	
  	
  User	
  1	
  

	
  	
  	
  User	
  2	
  

searches	
  

searches	
  

Internet	
  
…	
  

	
  	
  	
  User	
  n	
  

searches	
  
…	
  or	
  tweet	
  their	
  personal	
  health	
  condi?ons	
  
	
  	
  User	
  1	
  

	
  	
  	
  User	
  2	
  

tweets	
  

tweets	
  

Internet	
  
…	
  

	
  	
  	
  User	
  n	
  

tweets	
  
Es?mate	
  ILI	
  rate	
  using	
  user-­‐generated	
  data	
  
	
  
•  Models	
  

–  Linear	
  model	
  [1]:	
  

ILI	
  rate	
  =	
  (ILI-­‐related	
  data)Ÿα	
  +	
  error	
  

–  Logis2c	
  regression	
  [2]:	
  	
  

logit(ILI	
  rate)	
  =	
  logit(ILI-­‐related	
  data)Ÿα	
  +	
  error	
  

	
  

•  Key	
  point:	
  How	
  to	
  iden2fy	
  ILI-­‐related	
  data?	
  
	
  
•  Hint:	
  ILI	
  is	
  defined	
  as	
  fever	
  (temperature	
  of	
  100°F	
  
[37.8°C]	
  or	
  greater)	
  and	
  cough	
  and/or	
  sore	
  throat	
  
[1]	
  Polgreen	
  et	
  al.	
  “Using	
  internet	
  searches	
  for	
  influenza	
  surveillance”,	
  Clinical	
  Infec2ous	
  Disease,	
  
2008,	
  47(11):1443-­‐8.	
  
[2]	
  Ginsberg	
  et	
  al.	
  “Detec?ng	
  influenza	
  epidemics	
  using	
  search	
  engine	
  query	
  data.”,	
  Nature.	
  2009	
  
Feb	
  19;457(7232):1012-­‐4	
  
GFT	
  es?mates	
  based	
  on	
  flu-­‐related	
  queries	
  
are	
  highly	
  correlated	
  to	
  ILI	
  rate	
  

Repor2ng	
  lag	
  of	
  about	
  1	
  day	
  
Source:	
  hVp://www.google.org/flutrends/about/how.html	
  	
  
GFT	
  is	
  good,	
  however…	
  
•  Researchers	
  cannot	
  access	
  original	
  data	
  
•  GFT	
  does	
  not	
  disclose	
  search	
  queries	
  

Source:	
  Ginsberg	
  et	
  al,	
  Nature	
  457,	
  1012-­‐1014	
  (19	
  February	
  2009)	
  
SOURCES:	
  GOOGLE	
  FLU	
  TRENDS	
  
(WWW.GOOGLE.ORG/FLUTRENDS);	
  
CDC;	
  FLU	
  NEAR	
  YOU	
  
Twi)er	
  corpus	
  
Timeline:	
  36	
  weeks	
  for	
  the	
  US	
  2009	
  influenza	
  season	
  
(Aug	
  30,	
  2009	
  to	
  May	
  8,	
  2010)	
  
	
  
Name	
   Total	
  
25 mil

Tweets	
   587,290,394	
  
Unique	
   23,571,765	
  
users	
  
	
  
URL	
  

136,034,309	
  

Hash	
  
Tags	
  

20 mil
15 mil

10 mil

96,399,587	
  

5 mil

Thanks	
  to	
  Brendan	
  O’Connor	
  (CMU)	
  and	
  TwiVer	
  Inc.	
  
Related	
  work	
  
Twi)er	
  
corpus	
  

ILI-­‐related	
  
tweets	
  

Culo)a4	
  

Signorini3	
  

Chew3	
  

flu	
  

swine	
  

h1n1	
  

cough	
  

flu	
  

swine	
  flu	
  

headache	
  

influenza	
  

swineflu	
  

sore	
  throat	
  
[3]	
  A.	
  CuloVa,	
  “Detec2ng	
  influenza	
  epidemics	
  by	
  analyzing	
  twiVer	
  messages,”	
  arXiv:1007.4748v1	
  
[4]	
  A.	
  Signorini,	
  A.	
  M.	
  Segre,	
  and	
  P.	
  M.	
  Polgreen,	
  “The	
  Use	
  of	
  TwiVer	
  to	
  Track	
  Levels	
  of	
  Disease	
  Ac2vity	
  and	
  Public	
  Concern	
  in	
  the	
  U.S.	
  during	
  
the	
  Influenza	
  A	
  H1N1	
  Pandemic,”	
  PLoS	
  ONE,	
  vol.	
  6,	
  no.	
  5,	
  p.	
  e19467,	
  05	
  2011.	
  	
  
[5]	
  C.	
  Chew	
  and	
  G.	
  Eysenbach,	
  “Pandemics	
  in	
  the	
  Age	
  of	
  TwiVer:	
  Content	
  Analysis	
  of	
  Tweets	
  during	
  the	
  2009	
  H1N1	
  Outbreak,”	
  PLoS	
  ONE,	
  
vol.	
  5,	
  no.	
  11,	
  p.	
  e14118,	
  11	
  2010.	
  
Our	
  approach:	
  two-­‐step	
  filtering	
  
Twi)er	
  
corpus	
  

Respiratory	
  
syndrome-­‐related	
  
tweets	
  
Filter	
  1	
  
Knowledge-­‐
based	
  approach	
  

Respiratory	
  syndrome	
  
only	
  
Respirator	
  syndrome	
  	
  +	
  
“flu”	
  
Respiratory	
  syndrome	
  	
  +	
  
“flu”	
  -­‐	
  URL	
  

Seman?c	
  filtered	
  
tweets	
  
Filter	
  2	
  
Seman?c	
  level	
  	
  	
  

Nega?on	
  

Emo?con	
  

HashTags	
  

Humor	
  
Geo	
  
Correla?on	
  to	
  ILI	
  rate	
  (CDC	
  data)	
  
Method	
  
Google	
  Flu	
  Trends	
  

Pearson	
  corr	
  
with	
  ILI	
  rate	
  
0.9912	
  

Related	
  work	
  

CuloVa4	
  

0.9485	
  

Filter	
  1	
  

Respiratory	
  syndrome	
  +	
  “flu”	
  -­‐	
  URL	
  

0.9752	
  

Filter	
  1+2	
  

Nega2on	
  +	
  Emo2con	
  +	
  HashTags	
  +	
  
Humor	
  +	
  Geo	
  

0.9846	
  
%

Correla?on	
  to	
  ILI	
  rate	
  (CDC	
  data)	
  

S.	
  Doan,	
  L.Ohno-­‐Machado,	
  N.	
  Collier,	
  "Enhancing	
  TwiVer	
  Data	
  Analysis	
  with	
  Simple	
  Seman2c	
  Filtering:	
  Example	
  in	
  Tracking	
  Influenza-­‐	
  Like	
  
Illnesses",	
  Proc.	
  of	
  the	
  2nd	
  IEEE	
  HISB	
  2012,	
  pp.62-­‐71,	
  2012.	
  
Big	
  Data	
  challenge	
  
Twi)er:	
  140	
  millions	
  ac?ve	
  users	
  
340	
  millions	
  tweets/day	
  

Twitter API sampling rate is
small (1-5% data)

Filtered tweets: 0.2% of samples

Is	
  sampling	
  
data	
  	
  enough?	
  
DIZIE:	
  system	
  for	
  syndromic	
  surveillance	
  using	
  Twi)er	
  

Syndromic	
  surveillance	
  for	
  gastrointes?nal,	
  
respiratory,	
  neurological,	
  dermatological,	
  
haemorrhagic,	
  musculoskeletal	
  from	
  Tweets	
  in	
  40	
  
world	
  ci2es.	
  
Use	
  cases	
  
•  DIZIE	
  was	
  integrated	
  to	
  BioCaster,	
  our	
  news	
  media	
  
biosurveillance	
  system	
  
•  DIZIE	
  was	
  used	
  by	
  European	
  Centre	
  for	
  Disease	
  
Preven2on	
  and	
  Control	
  (ECDC)	
  to	
  track	
  syndromes	
  in	
  
the	
  London	
  2012	
  Summer	
  Olympics	
  
Poten?al	
  applica?ons	
  using	
  Twi)er	
  in	
  public	
  health	
  
•  Mental	
  Heath	
  Analysis	
  
•  Tobacco	
  surveillance	
  
•  Medica2on	
  use	
  in	
  social	
  media	
  
Acknowledgements	
  
•  Nigel	
  Collier,	
  European	
  Bioinforma2cs	
  Ins2tute	
  
•  Mike	
  Conway,	
  UCSD	
  
•  Lucila	
  Ohno-­‐Machado,	
  UCSD	
  
Using Twitter Data to Predict Flu Outbreak
Data	
  source	
  for	
  influenza	
  surveillance	
  
• 
• 
• 
• 
• 

Data	
  provided	
  by	
  physicians	
  and	
  laboratory	
  
Over-­‐the-­‐counter-­‐drug	
  sales	
  
School	
  absentee	
  records	
  
Health-­‐related	
  phone	
  calls	
  
Internet-­‐based	
  data:	
  
–  News	
  media	
  
–  Mailing	
  list	
  
–  Social	
  media	
  
Extract	
  respiratory	
  syndrome	
  keywords	
  
achy	
  chest	
  

cold	
  symptom	
  

respiratory	
  failure	
  

apnea	
  

cough	
  

runny	
  nose	
  

asthma	
  

dyspnea	
  

short	
  of	
  breath	
  	
  

asthma?c	
  

dyspnoea	
  

shortness	
  of	
  breath	
  

blocked	
  nose	
  

gasping	
  for	
  air	
  

sinusi?s	
  

breathing	
  difficul?es	
  

lung	
  sounds	
  

sore	
  throat	
  

breathing	
  trouble	
  

pneumonia	
  

stop	
  breathing	
  

bronchi?s	
  

rales	
  

stuffy	
  nose	
  

…	
  

…	
  

…	
  

We	
  have	
  a	
  total	
  of	
  37	
  keywords	
  	
  
Knowledge-­‐based	
  approach	
  

Name	
  

Example	
  

Respiratory	
  syndrome	
  
only	
  

tweets	
  containing	
  
syndrome	
  keywords	
  

Barber just coughed
on me in the chair.

Respiratory	
  syndrome	
  
+	
  “flu”	
  

tweets	
  containing	
  
syndrome	
  keywords	
  and	
  
“flu”	
  
	
  

I got flu n coughed a
lot.

Respiratory	
  	
  syndrome	
   tweets	
  containing	
  
+	
  “flu”	
  -­‐	
  URL	
  
syndrome	
  keywords	
  and	
  
“flu”,	
  remove	
  links	
  
	
  

7-year-old boy dies of
flu,pneumonia < URL>
Seman2c	
  level	
  filtering	
  
Name	
  

Examples	
  

Nega?on	
   Remove	
  nega?on	
  in	
  tweets	
  

I don’t have flu

Emo?con	
   Remove	
  tweets	
  containing	
  
smiley	
  emo?cons,	
  
e.g.,	
  :-­‐),,:D	
  	
  

Glad to hear that you’re beating
the flu. :-) Hope you don’t get the
nasty cough that everyone’s
getting this year

HashTags	
   Keeps	
  tweets	
  containing	
  
keyword	
  “flu”	
  

Still coughing smh #swineflu
#h1n1

Humor	
  

Remove	
  humor	
  features	
  in	
  
tweets,	
  e.g.,	
  “haha”,”hihi”,	
  
“***cough	
  …	
  cough***”	
  

Hm Im kinda wanting to go to
NYC really soon ***cough …
cough*** @Ctmomofsix =)

Geo	
  

Tweets	
  from	
  graphical	
  
loca?ons	
  (e.g.,	
  US)	
  
Seman2c-­‐level	
  filtered	
  tweets	
  
Types	
  

Tweet	
  samples	
  

Influenza	
  confirma?on	
  

I got flu n coughed a lot. Now my voice is like
monster’s voice. Rrr

Influenza	
  symptoms	
  

My day: flu-like symptoms (headache, body aches,
cough, chills, 100.9 fever). Swine flu not ruled out.
#H1N1

Flu	
  shots	
  

I’m still getting flu shots, nothing is worth flu turning
into bronchitis into pneumonia

Self	
  protec?on	
  

Cover your mouth if coughing, use a tissue, wash
your hands often & get a flu shot - protect and
defend your community from #H1N1

Medica?on	
  

Wondering why I didn’t take the flu shot, laying in
bed with cough drops, medicine, and the remote

Weitere ähnliche Inhalte

Andere mochten auch

Social media and it's use in disease surveillance
Social media and it's use in disease surveillanceSocial media and it's use in disease surveillance
Social media and it's use in disease surveillanceDan Aronne
 
Izobrazevanje za data-mining
Izobrazevanje za data-miningIzobrazevanje za data-mining
Izobrazevanje za data-miningbutest
 
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...Application of Data Warehousing & Data Mining to Exploitation for Supporting ...
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...Gihan Wikramanayake
 
Educational Data Mining in relation to education statistics of Nepal
Educational Data Mining in relation to education statistics of NepalEducational Data Mining in relation to education statistics of Nepal
Educational Data Mining in relation to education statistics of NepalRaj Subit
 
Data Mining and Text Mining in Educational Research
Data Mining and Text Mining in Educational ResearchData Mining and Text Mining in Educational Research
Data Mining and Text Mining in Educational ResearchQiang Hao
 
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
Learning Analytics in Education:  Using Student’s Big Data to Improve TeachingLearning Analytics in Education:  Using Student’s Big Data to Improve Teaching
Learning Analytics in Education: Using Student’s Big Data to Improve TeachingRafael Scapin, Ph.D.
 

Andere mochten auch (6)

Social media and it's use in disease surveillance
Social media and it's use in disease surveillanceSocial media and it's use in disease surveillance
Social media and it's use in disease surveillance
 
Izobrazevanje za data-mining
Izobrazevanje za data-miningIzobrazevanje za data-mining
Izobrazevanje za data-mining
 
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...Application of Data Warehousing & Data Mining to Exploitation for Supporting ...
Application of Data Warehousing & Data Mining to Exploitation for Supporting ...
 
Educational Data Mining in relation to education statistics of Nepal
Educational Data Mining in relation to education statistics of NepalEducational Data Mining in relation to education statistics of Nepal
Educational Data Mining in relation to education statistics of Nepal
 
Data Mining and Text Mining in Educational Research
Data Mining and Text Mining in Educational ResearchData Mining and Text Mining in Educational Research
Data Mining and Text Mining in Educational Research
 
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
Learning Analytics in Education:  Using Student’s Big Data to Improve TeachingLearning Analytics in Education:  Using Student’s Big Data to Improve Teaching
Learning Analytics in Education: Using Student’s Big Data to Improve Teaching
 

Ähnlich wie Using Twitter Data to Predict Flu Outbreak

Facebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemFacebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemChen Luo
 
Facebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemFacebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemChen Luo
 
When Swine Flew: Embracing Innovation
When Swine Flew: Embracing InnovationWhen Swine Flew: Embracing Innovation
When Swine Flew: Embracing InnovationAndre Blackman
 
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...Gunther Eysenbach
 
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docx
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docxDiscussion Question 1Outbreak of Cyclosporiasis in the United St.docx
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docxelinoraudley582231
 
Detecting influenza epidemics using search engine query data
Detecting influenza epidemics using search engine query dataDetecting influenza epidemics using search engine query data
Detecting influenza epidemics using search engine query databenj_2
 
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...Muhammad Habibi
 
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...Healthcare consultant
 
Introduction to Epidemiology and Surveillance
Introduction to Epidemiology and SurveillanceIntroduction to Epidemiology and Surveillance
Introduction to Epidemiology and SurveillanceGeorge Moulton
 
Eysenbach: Infodemiology and Infoveillance
Eysenbach: Infodemiology and InfoveillanceEysenbach: Infodemiology and Infoveillance
Eysenbach: Infodemiology and InfoveillanceGunther Eysenbach
 
Mandatory Influenza Vaccine Essay
Mandatory Influenza Vaccine EssayMandatory Influenza Vaccine Essay
Mandatory Influenza Vaccine EssayJessica Stapleton
 
Extracting interesting concepts from large-scale textual data
Extracting interesting concepts from large-scale textual dataExtracting interesting concepts from large-scale textual data
Extracting interesting concepts from large-scale textual dataVasileios Lampos
 
6-005-1430-Eyenbach
6-005-1430-Eyenbach6-005-1430-Eyenbach
6-005-1430-Eyenbachmed20su
 
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...Flávio Codeço Coelho
 
Informatics for Disease Surveillance – New Technologies
Informatics for Disease Surveillance – New TechnologiesInformatics for Disease Surveillance – New Technologies
Informatics for Disease Surveillance – New TechnologiesDr Wasim Ahmed
 
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...RIFF - A Social Network and Collaborative Platform For Public Health Disease ...
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...InSTEDD
 
Riff: A Social Network and Collaborative Platform for Public Health Disease S...
Riff: A Social Network and Collaborative Platform for Public Health Disease S...Riff: A Social Network and Collaborative Platform for Public Health Disease S...
Riff: A Social Network and Collaborative Platform for Public Health Disease S...Taha Kass-Hout, MD, MS
 
WSDM2015Tutorial.pdf
WSDM2015Tutorial.pdfWSDM2015Tutorial.pdf
WSDM2015Tutorial.pdfssuser2c7393
 

Ähnlich wie Using Twitter Data to Predict Flu Outbreak (20)

Facebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemFacebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning System
 
Facebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning SystemFacebook: An Innovative Influenza Pandemic Early Warning System
Facebook: An Innovative Influenza Pandemic Early Warning System
 
When Swine Flew: Embracing Innovation
When Swine Flew: Embracing InnovationWhen Swine Flew: Embracing Innovation
When Swine Flew: Embracing Innovation
 
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...
Infodemiology, Infoveillance, Twitter- and Google-based Surveillance: The Inf...
 
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docx
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docxDiscussion Question 1Outbreak of Cyclosporiasis in the United St.docx
Discussion Question 1Outbreak of Cyclosporiasis in the United St.docx
 
Detecting influenza epidemics using search engine query data
Detecting influenza epidemics using search engine query dataDetecting influenza epidemics using search engine query data
Detecting influenza epidemics using search engine query data
 
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...
Tweeting fever can twitter be used to Monitor the Incidence of Dengue-Like Il...
 
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...
Social media for tracking disease outbreaks–way of the future By.Dr.Mahboob a...
 
Introduction to Epidemiology and Surveillance
Introduction to Epidemiology and SurveillanceIntroduction to Epidemiology and Surveillance
Introduction to Epidemiology and Surveillance
 
Eysenbach: Infodemiology and Infoveillance
Eysenbach: Infodemiology and InfoveillanceEysenbach: Infodemiology and Infoveillance
Eysenbach: Infodemiology and Infoveillance
 
Mandatory Influenza Vaccine Essay
Mandatory Influenza Vaccine EssayMandatory Influenza Vaccine Essay
Mandatory Influenza Vaccine Essay
 
Enhancing Twitter Data Analysis with Simple Semantic Filtering: Example in Tr...
Enhancing Twitter Data Analysis with Simple Semantic Filtering: Example in Tr...Enhancing Twitter Data Analysis with Simple Semantic Filtering: Example in Tr...
Enhancing Twitter Data Analysis with Simple Semantic Filtering: Example in Tr...
 
Extracting interesting concepts from large-scale textual data
Extracting interesting concepts from large-scale textual dataExtracting interesting concepts from large-scale textual data
Extracting interesting concepts from large-scale textual data
 
6-005-1430-Eyenbach
6-005-1430-Eyenbach6-005-1430-Eyenbach
6-005-1430-Eyenbach
 
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...
Carl koppeschaar: Disease Radar: Measuring and Forecasting the Spread of Infe...
 
Informatics for Disease Surveillance – New Technologies
Informatics for Disease Surveillance – New TechnologiesInformatics for Disease Surveillance – New Technologies
Informatics for Disease Surveillance – New Technologies
 
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...RIFF - A Social Network and Collaborative Platform For Public Health Disease ...
RIFF - A Social Network and Collaborative Platform For Public Health Disease ...
 
Riff: A Social Network and Collaborative Platform for Public Health Disease S...
Riff: A Social Network and Collaborative Platform for Public Health Disease S...Riff: A Social Network and Collaborative Platform for Public Health Disease S...
Riff: A Social Network and Collaborative Platform for Public Health Disease S...
 
Ijsea04031004
Ijsea04031004Ijsea04031004
Ijsea04031004
 
WSDM2015Tutorial.pdf
WSDM2015Tutorial.pdfWSDM2015Tutorial.pdf
WSDM2015Tutorial.pdf
 

Kürzlich hochgeladen

Male Infertility Panel Discussion by Dr Sujoy Dasgupta
Male Infertility Panel Discussion by Dr Sujoy DasguptaMale Infertility Panel Discussion by Dr Sujoy Dasgupta
Male Infertility Panel Discussion by Dr Sujoy DasguptaSujoy Dasgupta
 
Female Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyFemale Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyMedicoseAcademics
 
Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .Mohamed Rizk Khodair
 
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio..."Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio...Sujoy Dasgupta
 
Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Vaikunthan Rajaratnam
 
EXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionEXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionkrishnareddy157915
 
AORTIC DISSECTION and management of aortic dissection
AORTIC DISSECTION and management of aortic dissectionAORTIC DISSECTION and management of aortic dissection
AORTIC DISSECTION and management of aortic dissectiondrhanifmohdali
 
Basic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxBasic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxkomalt2001
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfHongBiThi1
 
Red Blood Cells_anemia & polycythemia.pdf
Red Blood Cells_anemia & polycythemia.pdfRed Blood Cells_anemia & polycythemia.pdf
Red Blood Cells_anemia & polycythemia.pdfMedicoseAcademics
 
ANATOMICAL FAETURES OF BONES FOR NURSING STUDENTS .pptx
ANATOMICAL FAETURES OF BONES  FOR NURSING STUDENTS .pptxANATOMICAL FAETURES OF BONES  FOR NURSING STUDENTS .pptx
ANATOMICAL FAETURES OF BONES FOR NURSING STUDENTS .pptxWINCY THIRUMURUGAN
 
How to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyHow to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyZurück zum Ursprung
 
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...Shubhanshu Gaurav
 
Trustworthiness of AI based predictions Aachen 2024
Trustworthiness of AI based predictions Aachen 2024Trustworthiness of AI based predictions Aachen 2024
Trustworthiness of AI based predictions Aachen 2024EwoutSteyerberg1
 
Unit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptUnit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptPradnya Wadekar
 
blood bank management system project report
blood bank management system project reportblood bank management system project report
blood bank management system project reportNARMADAPETROLEUMGAS
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptPradnya Wadekar
 

Kürzlich hochgeladen (20)

Male Infertility Panel Discussion by Dr Sujoy Dasgupta
Male Infertility Panel Discussion by Dr Sujoy DasguptaMale Infertility Panel Discussion by Dr Sujoy Dasgupta
Male Infertility Panel Discussion by Dr Sujoy Dasgupta
 
Female Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before PregnancyFemale Reproductive Physiology Before Pregnancy
Female Reproductive Physiology Before Pregnancy
 
Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .
 
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio..."Radical excision of DIE in subferile women with deep infiltrating endometrio...
"Radical excision of DIE in subferile women with deep infiltrating endometrio...
 
Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.
 
Rheumatoid arthritis Part 1, case based approach with application of the late...
Rheumatoid arthritis Part 1, case based approach with application of the late...Rheumatoid arthritis Part 1, case based approach with application of the late...
Rheumatoid arthritis Part 1, case based approach with application of the late...
 
EXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionEXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung function
 
AORTIC DISSECTION and management of aortic dissection
AORTIC DISSECTION and management of aortic dissectionAORTIC DISSECTION and management of aortic dissection
AORTIC DISSECTION and management of aortic dissection
 
American College of physicians ACP high value care recommendations in rheumat...
American College of physicians ACP high value care recommendations in rheumat...American College of physicians ACP high value care recommendations in rheumat...
American College of physicians ACP high value care recommendations in rheumat...
 
Basic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptxBasic structure of hair and hair growth cycle.pptx
Basic structure of hair and hair growth cycle.pptx
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
 
Biologic therapy ice breaking in rheumatology, Case based approach with appli...
Biologic therapy ice breaking in rheumatology, Case based approach with appli...Biologic therapy ice breaking in rheumatology, Case based approach with appli...
Biologic therapy ice breaking in rheumatology, Case based approach with appli...
 
Red Blood Cells_anemia & polycythemia.pdf
Red Blood Cells_anemia & polycythemia.pdfRed Blood Cells_anemia & polycythemia.pdf
Red Blood Cells_anemia & polycythemia.pdf
 
ANATOMICAL FAETURES OF BONES FOR NURSING STUDENTS .pptx
ANATOMICAL FAETURES OF BONES  FOR NURSING STUDENTS .pptxANATOMICAL FAETURES OF BONES  FOR NURSING STUDENTS .pptx
ANATOMICAL FAETURES OF BONES FOR NURSING STUDENTS .pptx
 
How to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyHow to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturally
 
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...
FDMA FLAP - The first dorsal metacarpal artery (FDMA) flap is used mainly for...
 
Trustworthiness of AI based predictions Aachen 2024
Trustworthiness of AI based predictions Aachen 2024Trustworthiness of AI based predictions Aachen 2024
Trustworthiness of AI based predictions Aachen 2024
 
Unit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptUnit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.ppt
 
blood bank management system project report
blood bank management system project reportblood bank management system project report
blood bank management system project report
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologyppt
 

Using Twitter Data to Predict Flu Outbreak

  • 1. Using  Twi)er  Data  to  Predict  Flu  Outbreak   Son  Doan   Division  of  Biomedical  Informa2cs   University  of  California  San  Diego     BigData@UCSD  workshop   Nov  25,  2013  
  • 2. Seasonal  influenza  and  influenza-­‐like  illness   •  Seasonal  influenza  is  a  major  public  health  concern:   •  3-­‐5  million  cases  of  severe  illness     •  250,000  to  500,000  deaths  worldwide    each  year     •  Seasonal  influenza  has  main  syndrome  called  Influenza-­‐ Like  Illness  (ILI)   •  During  the  peak  of  a  major  outbreak  of  influenza,  more   cases  of  ILI  are  observed   à  Monitoring  ILI  can  help  in  predict  flu  outbreak        
  • 3. Tradi?onal  system  to  monitor  ILI:  ILINet     •  ILINet:  CDC’s  U.S.  Outpa2ent  ILI  Surveillance   Network   –  consists  of    >3,000  outpa2ent  healthcare  providers     –  all  50  US  states  and  area   –  reports  more  than  30  million  pa2ent  visits  each  year   •  ILINet  monitors  influenza  through  ILI  rate     –  ILI  rate  is  percentage  of  pa2ents  with  ILI  among  all   pa2ents   –  Average  na2onal  baseline  ILI  rate  for  2013  is  2.0%  
  • 5. Let’s  revisit  the  process        Pa2ent  1        Pa2ent  2   visits   Healthcare   provider   Check  if  ILI   visits   Healthcare   provider   Check  if  ILI   Healthcare   provider   Check  if  ILI   …        Pa2ent  n   visits   ILINet  gather   data  and  then   calculate  ILI  rate  
  • 6. ILINet  issue   ILINet  needs  1-­‐2  weeks  to  gather  and  process   data   Can  we  leverage  other  data  sources   to  predict  ILI  rate  faster?  
  • 7. Nowadays,  users  tend  to  find  informa?on  in  Internet      User  1        User  2   searches   searches   Internet   …        User  n   searches  
  • 8. …  or  tweet  their  personal  health  condi?ons      User  1        User  2   tweets   tweets   Internet   …        User  n   tweets  
  • 9. Es?mate  ILI  rate  using  user-­‐generated  data     •  Models   –  Linear  model  [1]:   ILI  rate  =  (ILI-­‐related  data)Ÿα  +  error   –  Logis2c  regression  [2]:     logit(ILI  rate)  =  logit(ILI-­‐related  data)Ÿα  +  error     •  Key  point:  How  to  iden2fy  ILI-­‐related  data?     •  Hint:  ILI  is  defined  as  fever  (temperature  of  100°F   [37.8°C]  or  greater)  and  cough  and/or  sore  throat   [1]  Polgreen  et  al.  “Using  internet  searches  for  influenza  surveillance”,  Clinical  Infec2ous  Disease,   2008,  47(11):1443-­‐8.   [2]  Ginsberg  et  al.  “Detec?ng  influenza  epidemics  using  search  engine  query  data.”,  Nature.  2009   Feb  19;457(7232):1012-­‐4  
  • 10. GFT  es?mates  based  on  flu-­‐related  queries   are  highly  correlated  to  ILI  rate   Repor2ng  lag  of  about  1  day   Source:  hVp://www.google.org/flutrends/about/how.html    
  • 11. GFT  is  good,  however…   •  Researchers  cannot  access  original  data   •  GFT  does  not  disclose  search  queries   Source:  Ginsberg  et  al,  Nature  457,  1012-­‐1014  (19  February  2009)  
  • 12. SOURCES:  GOOGLE  FLU  TRENDS   (WWW.GOOGLE.ORG/FLUTRENDS);   CDC;  FLU  NEAR  YOU  
  • 13. Twi)er  corpus   Timeline:  36  weeks  for  the  US  2009  influenza  season   (Aug  30,  2009  to  May  8,  2010)     Name   Total   25 mil Tweets   587,290,394   Unique   23,571,765   users     URL   136,034,309   Hash   Tags   20 mil 15 mil 10 mil 96,399,587   5 mil Thanks  to  Brendan  O’Connor  (CMU)  and  TwiVer  Inc.  
  • 14. Related  work   Twi)er   corpus   ILI-­‐related   tweets   Culo)a4   Signorini3   Chew3   flu   swine   h1n1   cough   flu   swine  flu   headache   influenza   swineflu   sore  throat   [3]  A.  CuloVa,  “Detec2ng  influenza  epidemics  by  analyzing  twiVer  messages,”  arXiv:1007.4748v1   [4]  A.  Signorini,  A.  M.  Segre,  and  P.  M.  Polgreen,  “The  Use  of  TwiVer  to  Track  Levels  of  Disease  Ac2vity  and  Public  Concern  in  the  U.S.  during   the  Influenza  A  H1N1  Pandemic,”  PLoS  ONE,  vol.  6,  no.  5,  p.  e19467,  05  2011.     [5]  C.  Chew  and  G.  Eysenbach,  “Pandemics  in  the  Age  of  TwiVer:  Content  Analysis  of  Tweets  during  the  2009  H1N1  Outbreak,”  PLoS  ONE,   vol.  5,  no.  11,  p.  e14118,  11  2010.  
  • 15. Our  approach:  two-­‐step  filtering   Twi)er   corpus   Respiratory   syndrome-­‐related   tweets   Filter  1   Knowledge-­‐ based  approach   Respiratory  syndrome   only   Respirator  syndrome    +   “flu”   Respiratory  syndrome    +   “flu”  -­‐  URL   Seman?c  filtered   tweets   Filter  2   Seman?c  level       Nega?on   Emo?con   HashTags   Humor   Geo  
  • 16. Correla?on  to  ILI  rate  (CDC  data)   Method   Google  Flu  Trends   Pearson  corr   with  ILI  rate   0.9912   Related  work   CuloVa4   0.9485   Filter  1   Respiratory  syndrome  +  “flu”  -­‐  URL   0.9752   Filter  1+2   Nega2on  +  Emo2con  +  HashTags  +   Humor  +  Geo   0.9846  
  • 17. % Correla?on  to  ILI  rate  (CDC  data)   S.  Doan,  L.Ohno-­‐Machado,  N.  Collier,  "Enhancing  TwiVer  Data  Analysis  with  Simple  Seman2c  Filtering:  Example  in  Tracking  Influenza-­‐  Like   Illnesses",  Proc.  of  the  2nd  IEEE  HISB  2012,  pp.62-­‐71,  2012.  
  • 18. Big  Data  challenge   Twi)er:  140  millions  ac?ve  users   340  millions  tweets/day   Twitter API sampling rate is small (1-5% data) Filtered tweets: 0.2% of samples Is  sampling   data    enough?  
  • 19. DIZIE:  system  for  syndromic  surveillance  using  Twi)er   Syndromic  surveillance  for  gastrointes?nal,   respiratory,  neurological,  dermatological,   haemorrhagic,  musculoskeletal  from  Tweets  in  40   world  ci2es.  
  • 20. Use  cases   •  DIZIE  was  integrated  to  BioCaster,  our  news  media   biosurveillance  system   •  DIZIE  was  used  by  European  Centre  for  Disease   Preven2on  and  Control  (ECDC)  to  track  syndromes  in   the  London  2012  Summer  Olympics  
  • 21. Poten?al  applica?ons  using  Twi)er  in  public  health   •  Mental  Heath  Analysis   •  Tobacco  surveillance   •  Medica2on  use  in  social  media  
  • 22. Acknowledgements   •  Nigel  Collier,  European  Bioinforma2cs  Ins2tute   •  Mike  Conway,  UCSD   •  Lucila  Ohno-­‐Machado,  UCSD  
  • 24. Data  source  for  influenza  surveillance   •  •  •  •  •  Data  provided  by  physicians  and  laboratory   Over-­‐the-­‐counter-­‐drug  sales   School  absentee  records   Health-­‐related  phone  calls   Internet-­‐based  data:   –  News  media   –  Mailing  list   –  Social  media  
  • 25. Extract  respiratory  syndrome  keywords   achy  chest   cold  symptom   respiratory  failure   apnea   cough   runny  nose   asthma   dyspnea   short  of  breath     asthma?c   dyspnoea   shortness  of  breath   blocked  nose   gasping  for  air   sinusi?s   breathing  difficul?es   lung  sounds   sore  throat   breathing  trouble   pneumonia   stop  breathing   bronchi?s   rales   stuffy  nose   …   …   …   We  have  a  total  of  37  keywords    
  • 26. Knowledge-­‐based  approach   Name   Example   Respiratory  syndrome   only   tweets  containing   syndrome  keywords   Barber just coughed on me in the chair. Respiratory  syndrome   +  “flu”   tweets  containing   syndrome  keywords  and   “flu”     I got flu n coughed a lot. Respiratory    syndrome   tweets  containing   +  “flu”  -­‐  URL   syndrome  keywords  and   “flu”,  remove  links     7-year-old boy dies of flu,pneumonia < URL>
  • 27. Seman2c  level  filtering   Name   Examples   Nega?on   Remove  nega?on  in  tweets   I don’t have flu Emo?con   Remove  tweets  containing   smiley  emo?cons,   e.g.,  :-­‐),,:D     Glad to hear that you’re beating the flu. :-) Hope you don’t get the nasty cough that everyone’s getting this year HashTags   Keeps  tweets  containing   keyword  “flu”   Still coughing smh #swineflu #h1n1 Humor   Remove  humor  features  in   tweets,  e.g.,  “haha”,”hihi”,   “***cough  …  cough***”   Hm Im kinda wanting to go to NYC really soon ***cough … cough*** @Ctmomofsix =) Geo   Tweets  from  graphical   loca?ons  (e.g.,  US)  
  • 28. Seman2c-­‐level  filtered  tweets   Types   Tweet  samples   Influenza  confirma?on   I got flu n coughed a lot. Now my voice is like monster’s voice. Rrr Influenza  symptoms   My day: flu-like symptoms (headache, body aches, cough, chills, 100.9 fever). Swine flu not ruled out. #H1N1 Flu  shots   I’m still getting flu shots, nothing is worth flu turning into bronchitis into pneumonia Self  protec?on   Cover your mouth if coughing, use a tissue, wash your hands often & get a flu shot - protect and defend your community from #H1N1 Medica?on   Wondering why I didn’t take the flu shot, laying in bed with cough drops, medicine, and the remote