Storm@Twitter, SIGMOD 2014
•
2 gefällt mir•1,599 views
Storm is a realtime fault-tolerant and distributed stream data processing system. Storm is used to run various critical computations in Twitter at scale, and in real-time. This talk was presented at the SIGMOD 2014 conference that accepted our paper Storm@Twitter. It describes the architecture of Storm and its methods for distributed scale-out and fault-tolerance. This presentation also describes how queries (aka. topologies) are executed in Storm, and presents some operational stories based on running Storm at Twitter. We also present results from an empirical evaluation demonstrating the resilience of Storm in dealing with machine failures.
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (19)
Ähnlich wie Storm@Twitter, SIGMOD 2014
Ähnlich wie Storm@Twitter, SIGMOD 2014 (20)
Mehr von Karthik Ramasamy
Mehr von Karthik Ramasamy (11)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Storm@Twitter, SIGMOD 2014
- 1. Storm @Twitter KARTHIK RAMASAMY @KARTHIKZ #TwitterAtSigmod #TwitterDataStorm Ankit Toshniwal, Siddarth Taneja, Amit Shukla, Jignesh Patel, Sanjeev Kulkarni Jason Jackson, Krishna Gade, Maosong Fu, Jake Donham, Nikunj Bhagat Sailesh Mittal and Dmitriy Ryaboy
- 4. GUARANTEED MESSAGE PROCESSING HORIZONTAL SCALABILITY ROBUST FAULT TOLERANCE CONCISE CODE- FOCUS ON LOGIC /b Ñ WHAT IS STORM? Streaming platform for analyzing realtime data as they arrive, so you can react to data as it happens.
- 5. STORM DATA MODEL TOPOLOGY Directed acyclic graph Vertices=computation, and edges=streams of data tuples SPOUTS Sources of data tuples for the topology Examples - Kafka/Kestrel/MySQL/Postgres BOLTS Process incoming tuples and emit outgoing tuples Examples - filtering/aggregation/join/arbitrary function , %
- 6. STORM TOPOLOGY % % % % % SPOUT 1 SPOUT 2 BOLT 1 BOLT 2 BOLT 3 BOLT 4 BOLT 5
- 7. WORD COUNT TOPOLOGY % % TWEET SPOUT PARSE TWEET BOLT WORD COUNT BOLT Live stream of Tweets #worldcup : 1M soccer: 400K ….
- 8. WORD COUNT TOPOLOGY % % TWEET SPOUT TASKS PARSE TWEET BOLT TASKS WORD COUNT BOLT TASKS %%%% %%%% When a parse tweet bolt task emits a tuple which word count bolt task should it send to?
- 9. STREAM GROUPINGS Random distribution of tuples Group tuples by a field or multiple fields Replicates tuples to all tasks SHUFFLE GROUPING FIELDS GROUPING ALL GROUPING Sends the entire stream to one task GLOBAL GROUPING / - ,.
- 11. STORM ARCHITECTURE Nimbus ZK CLUSTER SUPERVISOR W1 W2 W3 W4 SUPERVISOR W1 W2 W3 W4 TOPOLOGY SUBMISSION ASSIGNMENT MAPS SYNC CODE SLAVE NODE SLAVE NODE MASTER NODE
- 13. DATA FLOW IN STORM WORKERS In QueueIn QueueIn QueueIn QueueIn Queue TCP Receive Buffer In QueueIn QueueIn QueueIn QueueOut Queue Outgoing Message Buffer User Logic Thread User Logic Thread User Logic Thread User Logic Thread User Logic Thread User Logic Thread User Logic Thread User Logic Thread User Logic ThreadSend Thread Global Send Thread TCP Send Buffer Global Receive Thread Kernel Disruptor Queues 0mq Queues
- 15. 1 STORM METRICS SUPPORT AND TROUBLE SHOOTING 2 CONTINUOUS PERFORMANCE 3 CLUSTER AVAILABILITY
- 16. COLLECTING TOPOLOGY METRICS % % TWEET SPOUT PARSE TWEET BOLT WORD COUNT BOLT % METRICS BOLT SCRIBE
- 19. OVERLOADED ZOOKEEPER zk S1 S2 S3 Shared configuration W W W STORM Quickly exceeded number of clients Impacted uptime of other systems
- 20. OVERLOADED ZOOKEEPER zk S1 S2 S3 Detached configuration W W W STORM zk Increased to 300 workers per cluster > 300 - workers get killed and relaunched Worker heart beats written to zknode every 15 secs
- 21. OVERLOADED ZOOKEEPER zk S1 S2 S3 Scale up W W W STORM zk Increased to 1200 workers per cluster
- 22. 67% 33% OVERLOADED ZOOKEEPER KAFKA SPOUT Offset/partition is written every 2 secs ! ! STORM RUNTIME Workers write heart beats every 3 secs Analyzing zookeeper traffic
- 23. OVERLOADED ZOOKEEPER zk S1 S2 S3 Heart beat daemons W W W STORM zk 5000 workers per cluster and still growing! HH H KVKVKV
- 24. EXPT 1 STORM OVERHEADS JAVA PROGRAM Read from Kafka cluster and deserialize in a “for loop” Sustain input rate of 300K msgs/sec from Kafka topic EXPT 2 1-STAGE TOPOLOGY No acks to achieve at most once semantics Storm processes were co-located using isolation scheduler EXPT 3 1-STAGE TOPOLOGY WITH ACKS Enable acks for at least once semantics
- 25. STORM OVERHEADS AverageCPUUtilization 0% 20% 40% 60% 80% MachinesUsed 0 1 2 3 JAVA 1-STAGE 1-STAGE-ACK Machines Avg. CPU 77% 58.2%58.3% 3 11
- 27. STORM EXPERIMENTS Examine resiliency and efficiency during machine failures % % CLIENT EVENT SPOUT DISTRIBUTOR BOLT USER COUNT BOLT % AGGREGATOR BOLT SHUFFLE GROUPING FIELDS GROUPING FIELDS GROUPING COMPONENTS # TASKS client event spout 200 distributor bolt 200 user count bolt 300 aggregator bolt 20
- 28. STORM THROUGHPUT
- 29. STORM LATENCY
- 31. QUESTIONS and ANSWERS R $ Go ahead. Ask away.