This presentation gives you more detailed overview of Apache Storm (distributed real time computing system)

1. STORM ANATOMY
Cloud Computing Course
Prof Hanku Lee
Social Media Cloud Computing lab
MSAkhmedov Khumoyun

2. What is Stream processing
 Stream processing is a technical paradigm to process big volume of
unbound sequence of tuples in realtime
= stream
Source Stream Processor
• Continuous analytics
• Online machine learning
• Sensor data monitoring
• Financial trading …

3. Storm at Twitter
Twitter Web Analytics

4. What is Storm?
Storm is
• Fast & scalable
• Fault-tolerant
• Guarantees messages will be processed
• Easy to setup & operate
• Free & open source
distributed realtime computation system
- Originally developed by Nathan Marz at
BackType (acquired by Twitter)
- Written in Java and Clojure

5. Conceptual View

6. Physical View

7. Concepts
 Streams
 Spouts
 Bolts
 Topologies

8. Streams
Unbounded sequence of tuples

9. Spouts
Source of streams
• Read from Kafka queue
• Read from Twitter Streaming API

10. Bolts
Processes input streams and produces new streams

11. Bolts
• Functions
• Filters
• Aggregation
• Joins
• Talk to databases

12. Topology
Network of spouts and bolts

13. Tasks
Spouts and bolts execute as
many tasks across the cluster

14. Stream grouping
When a tuple is emitted, which task does it go to?

15. Stream grouping
• Shuffle grouping: pick a random task
• Fields grouping: consistent hashing on a
subset of tuple fields
• All grouping: send to all tasks
• Global grouping: pick task with lowest id

16. Starting topology

17. Starting topology

18. Storm : Fault-tolerance

19. Storm : Fault-tolerance

20. Storm : Fault-tolerance

21. Storm : Fault-tolerance

22. Storm : Fault-tolerance

23. Guarantees messages
will be processed

24. Message Passing (ZeroMQ)

25. Easy to setup & operate
• Setup ZooKeeper cluster
• Install dependencies on Nimbus and worker
machines
- ZeroMQ 2.1.7 and JZMQ
- Java 6 and Python 2.6.6
- unzip
• Download and extract a Storm release to Nimbus
and worker machines
• Fill in mandatory configuration into storm.yaml
• Launch daemons under supervision using “storm”
script

26. Cluster Summary

27. Topology Summary

28. Component Summary

29. Advanced Topics
• Distributed RPC
• Transactional topologies
• Trident
• Using non-JVM languages with Storm
• Unit testing
• Patterns

30. Real-time TwitterAnalytics
Trending Topics and SentimentAnalysis
Twitter
MySQL
Kafka
Storm Cluster
Hadoop (HDFS and HBase )
Twitter Crawler

32. THANK YOU FOR ATTENTION
Any Questions AreWelcome…