1. real-time large data
@raffi
deview - 17 september 2012

5. there are over
400 million tweets
a day

6. 4600 tweets a twe
a second ≈ 0.2 m

7. Pull Push
Targeted twitter.com User / Site Streams
home_timeline API Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams

8. Redis
Ingester
Search Cache
Redis
Fanout
Write API
Timeline Cache
HTTP Push
Push Compute
Hadoop
Batch Compute

9. Write API
Social
Ingester Fanout Graph
Service
Batch Compute
Timeline Cache
Push Compute
Search Cache
HTTP Push
Redis Redis
Redis Redis Hadoop
Earlybird Redis
Mobile
Push
Timeline
Blender
Service

10. Write API
Social
Ingester Fanout Graph
Service
Batch Compute
Timeline Cache
Push Compute
Search Cache
HTTP Push
Redis Redis
Redis Redis Hadoop
Earlybird Redis
Mobile
Push
Timeline
Blender
Service

11. Write API
Ingester Fanout
Batch Compute
Timeline Cache
Push Compute
Search Cache
HTTP Push
Redis Redis
Redis Redis Hadoop
Earlybird Redis
Mobile
Push
Timeline
Blender
Service

12. Write API
Ingester Fanout
Batch Compute
Timeline Cache
Push Compute
Search Index
HTTP Push
Redis Redis
Earlybird Redis Hadoop
Earlybird Redis
Mobile
Push
Timeline
Blender
Service

13. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
T-Bird
T-Flock +
Haplo
Monorail
Darkwing
Flock(s)

14. ROUTING PRESENTATION LOGIC STORAGE &
RETRIEVAL
Tweetypie
Monorail T-Bird
Gizmoduck
T-Flock +
Woodstar
Haplo
TFE TLS
Macaw
Darkwing
+Swift
Social
Graph
Service
Macaw
Flock(s)
+Disco
Story
Service

16. timeline delivery statistics
⇢ 30b deliveries / day (~20m / min)
⇢ 3.5 seconds @ p50 to deliver to 1m
⇢ ~350k deliveries / sec

23. #JoinTheFlock