Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, and is a lot of fun to use! We will talk about how Storm is architected, how to interoperate with Hadoop, and a few real-world use-cases.

1. Realtime Computation
with Storm
Brad Anderson
banderson@maprtech.com
@boorad

3. Definition & Overview
Interoperability
Use Cases

4. Stream Processing
CEP
Distributed RPC

5. Source Data
• Social Media • Weather Data
Feeds
• Auctions of Ad
• Network Sensors Impressions
• App/Web Logs • Payment
• Stock Tick Data Transactions

6. Before Storm
Queues Workers

7. Example
(simplified)

8. Storm
Guaranteed data processing
Horizontal scalability
Fault-tolerance
No intermediate message brokers!
Higher level abstraction than message passing
“Just works”

9. Concepts

10. streams
Tuple Tuple Tuple Tuple Tuple Tuple Tuple
Unbounded sequence of tuples

11. spouts
Source of streams

12. spouts
public
interface
ISpout
extends
Serializable
{
void
open(Map
conf,
TopologyContext
context,
SpoutOutputCollector
collector);
void
close();
void
nextTuple();
void
ack(Object
msgId);
void
fail(Object
msgId);
}

13. bolts
Processes input streams and produces new streams

14. bolts
public
class
DoubleAndTripleBolt
extends
BaseRichBolt
{
private
OutputCollectorBase
_collector;
public
void
prepare(Map
conf,
TopologyContext
context,
OutputCollectorBase
collector)
{
_collector
=
collector;
}
public
void
execute(Tuple
input)
{
int
val
=
input.getInteger(0);
_collector.emit(input,
new
Values(val*2,
val*3));
_collector.ack(input);
}
public
void
declareOutputFields(OutputFieldsDeclarer
declarer)
{
declarer.declare(new
Fields("double",
"triple"));
}
}

15. topologies
Network of spouts and bolts

16. topologies
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"));

17. Trident
Cascading for Storm

18. Trident Facilities
• Joins
• Aggregations
• Grouping
• Functions
• Filters
• Consistent, Exactly-Once Semantics

19. TridentTopology
topology
=
new
TridentTopology();
TridentState
wordCounts
=
topology.newStream("spout1",
spout)
.each(new
Fields("sentence"),
new
Split(),
new
Fields("word"))
.groupBy(new
Fields("word"))
.persistentAggregate(new
MemoryMapState.Factory(),
new
Count(),
new
Fields("count"))
.parallelismHint(6);

20. Interoperability

21. spouts
•Kafka (with transactions)
• Kestrel
• JMS
• AMQP
• Beanstalkd

22. bolts
• Functions
• Filters
• Aggregation
• Joins
• Talk to databases, Hadoop write-behind

23. Storm
realtime
processes
Queue Apps
Raw
Data Business
Value
Hadoop
batch
processes

24. Storm
realtime
processes
Queue Apps
Raw
Data Business
Value
Hadoop
Parallel Cluster Ingest
batch
processes

25. Storm
realtime
processes
Apps
Queue TailSpout
Raw
Data Business
Franz Value
Hadoop
batch
processes

26. Storm
realtime
processes
Apps
TailSpout
Raw
Data Business
Franz Value
Hadoop
batch
processes

27. Use Cases

28. Twitter
Follower
Distinct
Tweeter Follower follower
Follower
Distinct
URL Tweeter follower Reach
Follower
Follower
Distinct
Tweeter follower
Follower

29. Heartbyte

30. Fleet Logistics

31. http://github.com/{tdunning | boorad}/mapr-spout
Brad Anderson
banderson@maprtech.com
@boorad

32. Thank you.
http://github.com/{tdunning | boorad}/mapr-spout
Brad Anderson
banderson@maprtech.com
@boorad