Apache Storm Internals
Die SlideShare-Präsentation wird heruntergeladen.
×
Hier ansehen
Weitere Verwandte Inhalte
Diashows für Sie (20)
Andere mochten auch (18)
Ähnlich wie Learning Stream Processing with Apache Storm (20)
Aktuellste (20)
Learning Stream Processing with Apache Storm
- 1. a [ b K Z
- 2. • • • CONTACT ME @edvorkin
- 3. • • • • • • • • • •
- 4. [
- 5. real-time medical news from curated Twitter feed
- 6. Every second, on average, around 6,000 tweets are tweeted on Twitter, which corresponds to over 350,000 tweets sent per minute, 500 million tweets per day 350,000 ^ 1 % = 3500 ^
- 7. •How to scale •How to deal with failures •What to do with failed messages •A lot of infrastructure concerns •Complexity •Tedious coding DB t *Image credit:Nathanmarz: slideshare: storm
- 8. Inherently BATCH-Oriented System
- 9. •Exponential rise in real- time data •New business opportunity •Economics of OSS and commodity hardware Stream processing has emerged as a key use case* *Source: Discover HDP2.1: Apache Storm for Stream Data Processing. Hortonworks. 2014
- 10. •Detecting fraud while someone swiping credit card •Place ad on website while someone is reading a specific article •Alerts on application and machine failures •Use stream-processing in batch oriented fashion
- 11. 4
- 12. % å å
- 13. Created by Nathan Martz Acquired by Twitter Apache Incubator Project Open sourced Part of Hortonworks HDP2 platform U a x Top Level Apache Project
- 14. Most mature, widely adopted framework Source: http://storm.incubator.apache.org/
- 15. Process endless stream of data. 1M+ messages / sec on a 10- 15 node cluster / 4
- 16. Guaranteed message processing Û
- 17. Tuples, Streams, Spouts, Bolts and Topologies Z å å å
- 18. TUPLE Storm data type: Immutable List of Key/Value pair of any data type word: “Hello” Count: 25 Frequency: 0.25
- 19. Unbounded Sequence of Tuples between nodes STREAM
- 20. SPOUT The Source of the Stream
- 21. Read from stream of data – queues, web logs, API calls, databases Spout responsibilities
- 22. BOLT ⚡
- 23. •Process tuples and perform actions: calculations, API calls, DB calls •Produce new output stream based on computations Bolt ⚡ F(x)
- 24. •A topology is a network of spouts and bolts •Defines data flow 4
- 25. •May have multiple spouts 4
- 26. •Each spout and bolt may have many instances that perform all the processing in parallel 4 • • • • •
- 27. How tuples are send between instances of spouts and bolts Random Distribution. Routes tuples to bolt based on the value of the field. Same values always route to the same bolt Replicates the tuple stream across all the bolt tasks. Each task receive a copy of tuple. Routes all tuple in the stream to single task. Should be used with caution. 4
- 28. å å å å
- 29. compile 'org.apache.storm:storm-core:0.9.2’ <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>0.9.2</version> </dependency>
- 30. Two 1 Households 1 Both 1 Alike 1 In 1 Dignity 1 sentence word Word ⚡ ⚡ ⚡ 3 final count: Two 20 Households 24 Both 22 Alike 1 In 1 Dignity 10 "Two households, both alike in dignity" Two Households Both alike in dignity
- 31. Data Source
- 32. SplitSentenceBolt Resource initialization
- 33. WordCountBolt
- 34. PrinterBolt
- 35. Linking it all together
- 36. How to scale stream processing q å å å å å
- 37. storm main components Machines in a storm cluster JVM processes running on a node. One or more per node. Java thread running within worker JVM process. Instances of spouts and bolts.
- 38. q
- 39. q
- 40. How tuples are send between instances of spouts and bolts
- 41. a å å å å å å
- 42. Tuple tree Reliable vs unreliable topologies
- 43. Methods from ISpout interface
- 44. Reliability in Bolts Anchoring Ack Fail
- 45. Unit testing Storm components a
- 46. BDD style of testing
- 47. Extending OutputCollector
- 48. Extending OutputCollector
- 49. Z å å å å å å å
- 50. Physical View 4
- 51. deploying topology to a cluster storm jar wordcount-1.0.jar com.demo.storm.WordCountTopology word- count-topology
- 52. Monitoring and performance tuning
- 53. x å å å å å å å å
- 54. Run under supervision: Monit, supervisord
- 55. Nimbus move work to another node
- 56. Supervisor will restart worker
- 57. Micro-Batch Stream Processing K å å å å å å å å å
- 58. Functions, Filters, aggregations, joins, grouping Ordered batches of tuples. Batches can be partitioned. Similar to Pig or Cascading Transactional spouts Trident has first class abstraction for reading and writing to stateful sources Ü 4
- 59. Stream processed in small batches •Each batch has a unique ID which is always the same on each replay •If one tuple failed, the whole batch is reprocessed •Higher throutput than storm but higher latency as well
- 60. How trident provides exactly –one semantics?
- 61. Store the count along with BatchID COUNT 100 BATCHID 1 COUNT 110 BATCHID 2 10 more tuples with batchId 2 Failure: Batch 2 replayed The same batchId (2) •Spout should replay a batch exactly as it was played before •Trident API hide dealing with batchID complexity
- 62. Word count with trident
- 63. Word count with Trident
- 64. Word count with Trident
- 65. Style of computation 4
- 66. By styles of computation 4
- 67. å å å å å å å å å å
- 68. Enhancing Twitter feed with lead Image and Title •Readability enhancements •Image Scaling •Remove duplicates •Custom Business Logic
- 69. Writing twitter spout
- 70. Status
- 71. use Twitter4J java library
- 72. use existing Spout from Storm contrib project on GitHub Spouts exists for: Twitter, Kafka, JMS, RabbitMQ, Amazon SQS, Kinesis, MongoDB….
- 73. •Storm takes care of scalability and fault-tolerance •What happens if there is burst in traffic?
- 74. Introducing Queuing Layer with Kafka Ñ
- 75. 4
- 76. Solr Indexing
- 77. Processing Groovy Rules (DSL) on a scale in real-time
- 78. å å å å å å å å å å å
- 79. Statsd and Storm Metrics API http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
- 80. •Use cache if you can: for example Google Guava caching utilities •In memory DB •Tick tuples (for batch updates)
- 81. •Linear classification (Perceptron, Passive-Aggresive, Winnow, AROW) •Linear regression (Perceptron, Passive-Aggresive) •Clustering (KMeans) •Feature scaling (standardization, normalization) •Text feature extraction •Stream statistics (mean, variance) •Pre-Trained Twitter sentiment classifier Trident-ML
- 82. http://www.michael-noll.com http://www.bigdata-cookbook.com/post/72320512609/storm-metrics- how-to http://svendvanderveken.wordpress.com/
- 83. edvorkin/Storm_Demo_Spring2GX
- 84. Go ahead. Ask away.
