Roll-your-own personalized twitter search engine with elasticsearch, kibana, neo4j, linkurious, docker, and streams.

1. Twitter Power-User
HOWTO
@steveblackmon

2. ✦Visualize content and connections
✦Ad-hoc queries
✦Exploration
✦Focus on specific profiles and posts
Goals
2

3. ✦Assemble data about your twitter network
✦Analyze and Assess
✦…
✦Profit?
3
Plan

4. Data Visualization
4

5. Data Repositories
5

6. Why ?
✦ Handles de-duplication and merging
✦ Speaks JSON and REST
✦ Scales vertically and horizontally
✦ Because
6

7. Why ?
✦ Handles de-duplication and merging
✦ Relationships central to data model
✦ Built and optimized for graph traversal
queries
✦ Because
7

8. Supporting
Technologies
8

9. Why ?
✦Micro-service friendly
✦Works the same way across OS,
distribution, cloud provider
✦Build, ship, run anywhere with 3-5 shell
commands (build, [tag,] push, [pull,] run)
9

10. Why ?
✦Assemble pipelines of modular data
providers and database writers
✦Normalize your social databases to
community standards
✦Why waste time hunting for and
wrangling new SDKs?
10

11. 11

12. I exist. Who
am I?
Part 1: ID
12

13. Capture own history
Who am I?
13

14. Capture userstream
Who am I?
14

15. Other people do
too.
Who are they?
15
Part 2: EGO

16. Collect friend/follower
network
Who are they?
16

17. I am but one
piece of a vast
society.
What is my place?
Part 3: SUPER-EGO
17

18. What is my place?
Research interesting accounts

19. DEMO
19

20. 20

21. 21

22. 22

23. 23

24. 24

25. 25

26. HOWTO
26

27. ✦ Using linux?
• apt-get / yum install docker.io
✦ Using a mac?
✦ Install Boot2Docker
✦ http://boot2docker.io/
• boot2docker ssh
• ifconfig
• note the bridged IP (eth1)
27
Docker

28. ✦docker run -d --name elasticsearch -p 9200:9200 -p
9300:9300 -e PLUGINS=mobz/elasticsearch-
head,elasticsearch/marvel/latest
quay.io/steveblackmon/elasticsearch:1.4.4
✦docker run -d --name kibana -p 5601:5601 --link
elasticsearch:elasticsearch
quay.io/steveblackmon/kibana:4.0.0
✦docker run -d --name neo4j -p 7474:7474
quay.io/steveblackmon/neo4j:2.2.0.M4
28
Installation

29. ✦ docker run -d --name linkurious -p 3000:3000
—link neo4j:neo4j quay.io/steveblackmon/linkurious:latest
** LINKURIOUS IS NOT OPEN-SOURCE **
** PURCHASE A LICENSE AT http://linkurio.us **
** 20% OFF IF YOU MENTION THIS TALK **
** 30-DAY MONEY-BACK GUARANTEE **
** ACCESS TO THIS DOCKER REPO IF YOU BUY**
29
Installation (Cont.)

30. ✦developer.twitter.com
✦mytwitterid.com
✦mkdir streams
✦cd streams
Configuration
(Part 0)
30

31. Configuration (Part 1)
twitter.userstream.json
1. Add twitter oauth credentials
2. Add own userid to twitter.follow
3. Set elasticsearch.hosts
4. Set graph.hostname
twitter.history.json
1.Add twitter oauth credentials
2.Add own userid to twitter.info
3.Set elasticsearch.hosts

32. ✦docker run -d --name twitter.userstream -v
`pwd`/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.userstream.json -
cp /twitter-0.2-SNAPSHOT.jar
twitter.UserstreamPipeline
✦docker run -d --name twitter.history -v
`pwd`/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.history.json -cp
/twitter-0.2-SNAPSHOT.jar
twitter.HistoryPipeline
Execution (Part 1)
32

33. twitter.friends.json
1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
twitter.followers.json
1. Add twitter oauth credentials
2. Add own userid to twitter.info
3. Set graph.hostname
Configuration (Part 2)
33

34. ✦docker run -v `pwd`/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-
SNAPSHOT.jar twitter.FollowingPipeline
✦docker run -v `pwd`/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-
SNAPSHOT.jar twitter.FollowingPipeline
Execution (Part 2)
34

35. Configuration (Part 3)Export IDs of your connections with Cypher
•MATCH (v:page {handle:'handle'})-[:follow]-
(x:page) RETURN DISTINCT
SUBSTRING(x.id, 11);
twitter.connections.history.json
1.Add twitter oauth credentials
2.Add connection ids to twitter.info
3.Set elasticsearch.hosts
twitter.connections.friends.json
4.Add twitter oauth credentials
5.Add connection ids to twitter.info
6.Set graph.hostname
35

36. ✦docker run -v
/home/ubuntu/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.connections.histor
y.json -cp /twitter-0.2-SNAPSHOT.jar
twitter.HistoryPipeline
✦docker run -v
/home/ubuntu/streams:/streams
quay.io/steveblackmon/twitter:latest java -
Dconfig.file=/streams/twitter.connections.friend
s.json -cp /twitter-0.2-SNAPSHOT.jar
twitter.FollowingPipeline
Execution (Part 3)
36

37. @steveblackmon
37
FIN