Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Realtime Computation with Storm
Ähnlich wie Realtime Computation with Storm (20)
Mehr von boorad
Realtime Computation with Storm
- 1. Realtime Computation with Storm Brad Anderson banderson@maprtech.com @boorad
- 3. Definition & Overview Interoperability Use Cases
- 4. Stream Processing CEP Distributed RPC
- 5. Before Storm Queues Workers
- 7. Storm Guaranteed data processing Horizontal scalability Fault-tolerance No intermediate message brokers! Higher level abstraction than message passing
- 8. Concepts
- 9. streams Tuple Tuple Tuple Tuple Tuple Tuple Tuple Unbounded sequence of tuples
- 11. spouts public interface ISpout extends Serializable { void open(Map conf, TopologyContext context, SpoutOutputCollector collector); void close(); void nextTuple(); void ack(Object msgId); void fail(Object msgId); }
- 12. bolts Processes input streams and produces new streams
- 13. bolts public class DoubleAndTripleBolt extends BaseRichBolt { private OutputCollectorBase _collector; public void prepare(Map conf, TopologyContext context, OutputCollectorBase collector) { _collector = collector; } public void execute(Tuple input) { int val = input.getInteger(0); _collector.emit(input, new Values(val*2, val*3)); _collector.ack(input); } public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("double", "triple")); } }
- 14. topologies Network of spouts and bolts
- 16. TridentTopology topology = new TridentTopology(); TridentState wordCounts = topology.newStream("spout1", spout) .each(new Fields("sentence"), new Split(), new Fields("word")) .groupBy(new Fields("word")) .persistentAggregate(new MemoryMapState.Factory(), new Count(), new Fields("count")) .parallelismHint(6);
- 17. Interoperability
- 18. spouts •Kafka (with transactions) • Kestrel • JMS • AMQP • Beanstalkd
- 19. bolts • Functions • Filters • Aggregation • Joins • Talk to databases, Hadoop write-behind
- 20. Storm realtime processes Queue Apps Raw Data Business Value Hadoop batch processes
- 21. Storm realtime processes Queue Apps Raw Data Business Value Hadoop Parallel Cluster Ingest batch processes
- 22. Storm realtime processes Queue Apps Raw Data Business Value Hadoop batch processes
- 23. Storm realtime processes Apps Raw Data Business Value Hadoop batch processes
- 24. Use Cases
- 25. Twitter Follower Distinct Tweeter Follower follower Follower Distinct URL Tweeter follower Reach Follower Follower Distinct Tweeter follower Follower
- 26. Heartbyte
- 27. Fleet Logistics
- 28. http://github.com/{tdunning | boorad}/mapr-spout Brad Anderson banderson@maprtech.com @boorad
- 29. Thank you. http://github.com/{tdunning | boorad}/mapr-spout Brad Anderson banderson@maprtech.com @boorad
Hinweis der Redaktion
- \n
- C - Best accessible distributed realtime computation system going\nA - Learn about and start using Storm\nB - You will get a great new tool in your technology stack - interesting uses\n
- CEP - continuous\n\nNot HFT-grade\n\n
- \n
- scaling is painful\npoor fault tolerance\ncoding is hard\n
- \n
- \n
- tweets stock ticks manufacturing machine data sensor messages\n
- \n
- \n
- \n
- \n
- DAG\n\nruns continuously\n
- abstractions like Cascading, Hive, Pig make MR approachable\n\ncode size reduction\n
- \n
- \n
- kestrel - via thrift\nkafka - transactional topologies, idempotentcy, process only once\nactivemq\n
- \n
- current architecture\n\ndata ingest tool for hadoop (avoid Flume madness)\n
- new architecture\n
- \n
- Trending Topics (stream processing of the firehose)\ncomputing the ‘reach’ of a URL (Dist RPC)\n
- \n
- Android devices, sampling geo every 5 seconds\nroute optimization\nroad tax reduction\nidle alerts\n
- C - Exciting times, much like Hadoop/NoSQL beginning\nA - Start tinkering with Storm, integrate into your workflows\nB - be more responsive in turning data into information\n