Process the Twitter stream using Storm & Redstorm with Ruby & JRuby. Full working demo, code on github https://github.com/colinsurprenant/tweitgeist and live demo http://tweitgeist.needium.com/

1. Twitter
stream processing
Colin Surprenant
Big Data Montreal @colinsurprenant
April 2012 Lead ninja

2. Twitter - Spring 2012
● 350 million tweets/day
● 140 million active users
● >1 million applications using API

3. Daily Twitter @ Needium
50 000 000 processed tweets
5 000 opportunities
500 messages sent
100GB data

4. Anatomy of a tweet
What's in a tweet?
timestamp
user
message
avatar
Anything else?

6. How to get the tweets?
● Streaming API
Subscribe to realtime feeds moving forward
● REST Search API
Search request on past data (1 week)

7. Streaming API
public statuses from all users
● status/filter
track/location/follow
○ 5000 follow user ids
○ 400 track keywords
○ 25 location boxes
○ rate limited
● status/sample
○ 1% of all public statuses (message id mod 100)
○ two status/sample streams will result in same data

8. Streaming API
per user streams
● User Streams
○ all data required to update a user's display
○ requires user's OAuth token
○ statuses from followings, direct messages, mentions
○ cannot open large number of user streams from same host
● Site Streams
○ multiplexing of multiple User Streams

9. Streaming API
Firehose
need more/full data? only through partners
● gnip.com
● datasift.com
● filtering/tracking
●
$$$
partial to full Firehose
What's the catch?
lots of it

10. Search API
● REST API (http request/response)
○ search query
○ geocode (lat, long, radius)
○ result type (mixed/recent/popular)
○ since id
● max 100 rpp and 1500 results
● rate limited (~1 request/sec)

11. Storm
Distributed and fault-tolerant realtime computation
https://github.com/nathanmarz/storm

12. Storm
The promise
● Guaranteed data processing
● Horizontal scalability
● Fault-tolerance
● No intermediate message brokers
● Higher level abstraction than message passing
● Just work

13. RedStorm
Storm: Java + Closure
RedStorm: JRuby integration & DSL for Storm
+
Ruby + Storm on JVM
https://github.com/colinsurprenant/redstorm

14. Storm
Typical use cases
Stream Continuous Distributed
processing computation RPC

15. Storm
Concepts
Streams
Tuple Tuple Tuple Tuple Tuple
Unbounded sequence of tuples

16. Storm
Concepts
Spouts
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Source of streams

17. Storm
Concepts
Bolts
Tuple
Tuple
Tuple
Tuple
Tuple
Tuple Tuple Tuple Tuple Tuple
Tuple
Tuple
Tuple
Tuple
Tuple
Processes input streams and produce new streams

18. Storm
Concepts
Topology
Network of spouts and bolts

19. Storm
Concepts
bolt A Shuffle bolt B All
bolt A bolt B
random+fair distribution replication to all tasks
across tasks
Grouping
bolt A Fields bolt B bolt A Global bolt B
field X
field Y
partition on specified field entire stream to single task

20. Storm
What Storm does
● Distributes code
● Robust process management
● Monitors topologies and reassigns failed tasks
● Provides reliability by tracking tuple trees
● Routing and partitioning of stream

21. Tweitgeist
Live top 10 trending hashtags on Twitter
DEMO
https://github.com/colinsurprenant/tweitgeist
Live demo: http://tweitgeist.needium.com

22. Tweitgeist
Architecture
Partitioned Partitioned
counting ranking
Parallel
extraction Merging
Partitioned Partitioned
counting ranking
Queue
Queue
N partitions N partitions
Twitter
stream Frontend

23. Tweitgeist
Architecture
● Parallel computation across partitions of the stream
● Scalable architecture
● Work for full Twitter firehose with very little mods

24. Tweitgeist
Storm Topology
TwitterStream ExtractMessage ExtractHashtag RollingCount Rank Merge
Spout Bolt Bolt Bolt Bolt Bolt
Twitter
field hashtag
field hashtag
UI
global
shuffle
shuffle
Redis Redis
queue queue
stream message hashtag rolling ranking merging
reader extract extract counter
Shuffle grouping: Tuples are randomly distributed across the bolt's tasks
Fields grouping: The stream is partitioned by the fields specified in the grouping
Global grouping: The entire stream goes to a single one of the bolt's tasks

25. Tweitgeist
Topology definition

26. Tweitgeist
Where
Code
https://github.com/colinsurprenant/tweitgeist
Live demo
http://tweitgeist.needium.com