Slides from "Using Clojure for Sentiment Analysis of the Twittersphere" - EuroClojure 2014.
I had to convert from Keynote to pdf for upload, so apologies for loss of animations or notes.
7. Leiningen Versus the Ants
Clojure Versus Java
Functional Versus Object Oriented
Apache Ant
8. Blood aches pain
sick shivers spasm
vomit dizzy fainting
colic and in a
coma!
1 FAVORITE
9. Blood aches pain
sick shivers spasm
vomit dizzy fainting
colic and in a
coma!
1 FAVORITE
Scores HIGH for flu symptoms
10. Blood aches pain
sick shivers spasm
vomit dizzy fainting
colic and in a
coma!
1 FAVORITE
Who is Catalina Rubottom?
11. Blood aches pain
sick shivers spasm
vomit dizzy fainting
colic and in a
coma!
1 FAVORITE
HDFS/Hadoop
Mongo/Aggregation
Mongo/MapReduce
Postgres
30 million geo-tagged
tweets sent from UK
12. How can we do fast, real time analytics of
social media?
36. (wcar*
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
(car/expire “ENGLAND&JOVIALITY" 10)
(car/bitcount “ENGLAND&JOVIALITY"))
How many people in England are Happy?
37. (wcar*
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
(car/expire “ENGLAND&JOVIALITY" 10)
(car/bitcount “ENGLAND&JOVIALITY"))
How many people in England are Happy?
38. (wcar*
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
(car/expire “ENGLAND&JOVIALITY" 10)
(car/bitcount “ENGLAND&JOVIALITY"))
How many people in England are Happy?
39. (wcar*
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
(car/expire “ENGLAND&JOVIALITY" 10)
(car/bitcount “ENGLAND&JOVIALITY"))
How many people in England are Happy?
41. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
How many people in England are Happy?
42. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
How many people in England are Happy?
43. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
1
AND
How many people in England are Happy?
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
44. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
0 1
AND
How many people in England are Happy?
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
45. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
0 0 1
AND
How many people in England are Happy?
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
46. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1
AND
How many people in England are Happy?
(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")
47. ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1
JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1
0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1
How many people in England are Happy?
(car/bitcount “ENGLAND&JOVIALITY")
4
48. (wcar*
(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")
(car/expire "FATIGUE|HOSTILITY" 10)
(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"
"SCOTLAND" "FATIGUE|HOSTILITY")
(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)
(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
How many people in Scotland are Tired
and Grumpy?
49. (wcar*
(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")
(car/expire "FATIGUE|HOSTILITY" 10)
(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"
"SCOTLAND" "FATIGUE|HOSTILITY")
(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)
(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
How many people in Scotland are Tired
and Grumpy?
50. (wcar*
(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")
(car/expire "FATIGUE|HOSTILITY" 10)
(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"
"SCOTLAND" "FATIGUE|HOSTILITY")
(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)
(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
How many people in Scotland are Tired
and Grumpy?
51. (wcar*
(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")
(car/expire "FATIGUE|HOSTILITY" 10)
(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"
"SCOTLAND" "FATIGUE|HOSTILITY")
(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)
(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))
How many people in Scotland are Tired
and Grumpy?
75. Sanitisation
• Exclude media/advertising/spam
• Account for text speak
• Account for smileys & emoj’s
• Word stemming (or lemmatisation)
• Part of Speech Tagging (POS)
LMAO
GetRichQuick.com
dakrone/clojure-opennlp
79. Reverse Geocoding Issues
• Don’t want external services
• Don’t want heavy IO
• Don’t want round trips to the database
• Accuracy not too much of a concern