This document provides an overview of developing Java streaming applications with Apache Storm. It discusses what Storm is, its conceptual model including tuples, streams, spouts, bolts and topologies. It demonstrates developing a word count topology with code examples. It also covers Storm's runtime architecture, additional features like reliability, and integrating Storm with technologies like Kafka and HBase.

1. Page1
Developing Java Streaming Applications
with Apache Storm
Lester Martin www.ajug.org - Nov 2017

2. Page2
Connection before Content
Lester Martin – Hadoop/Spark/Storm Trainer & Consultant
lester.martin@gmail.com
http://lester.website (links to blog, twitter,
github, LI, FB, etc)

3. Page3
Agenda – Needs Updating!!!!
• What is Storm?
• Conceptual Model
• Compile Time
• DEMO: Develop Word Count Topology
• Runtime
• DEMO: Submit Word Count Topology
• Additional Features
• DEMO: Kafka > Storm > HBase Topology in Local Cluster

4. Page4
What is Storm?

5. Page5
Storm is …
à Streaming
– Key enabler of the Lambda Architecture
à Fast
– Clocked at 1M+ messages per second per node
à Scalable
– Thousands of workers per cluster
à Fault Tolerant
– Failure is expected, and embraced
à Reliable
– Guaranteed message delivery
– Exactly-once semantics

6. Page6
Storm in the Lambda Architecture
persists data
Hadoop
batch processing
batch feeds
Update event models
Pattern templates, key-
performance indicators, and
alerts
Dashboards and Applications
Stormreal-time data
feeds

7. Page7
Conceptual Model

8. Page8
TUPLE
{…}

9. Page9
Tuple
à Unit of work to be processes
à Immutable ordered set of serializable values
à Fields must have assigned name
{…}

10. Page10
Stream
à Core abstraction of Storm
à Unbounded sequence of Tuples
{…} {…} {…} {…} {…} {…} {…}

11. Page11
SPOUT

12. Page12
Spout
à Source of Streams
à Wrap an event source and emit Tuples

13. Page13
Message Queues
Message queues are often the source of the data processed by Storm
Storm Spouts integrate with many types of message queues
real-time data
source
operating
systems,
services and
applications,
sensors
Kestrel,
RabbitMQ,
AMQP, Kafka,
JMS, others…
message
queue
log entries,
events, errors,
status
messages, etc.
Storm
data from queue
is read by Storm

14. Page14
BOLT

15. Page15
Bolt
à Core unit of computation
à Receive Tuples and do stuff
à Optionally, emit additional Tuples

16. Page16
Bolt
à Write to a data store

17. Page17
Bolt
à Read from a data store

18. Page18
Bolt
à Perform arbitrary computation

19. Page19
Bolt
à (Optionally) Emit additional Stream(s)

20. Page20
TOPOLOGY

21. Page21
Topology
à DAG of Spouts and Bolts
à Data Flow Representation
à Streaming Computation

22. Page22
Topology
à Storm executes Spouts and Bolts as Tasks that run in parallel on
multiple machines

23. Page23
Parallel Execution of Topology Components
a logical
topology
spout A
bolt A bolt B
bolt C
a physical
implementation
machine A
machine B
machine E
machine C
machine D
machine F
machine G
spout A
two tasks
bolt A
two tasks
bolt B two
tasks
bolt C
one task

24. Page24
Stream Groupings
Stream Groupings determine how Storm routes Tuples between Tasks
Grouping Type Routing Behavior
Shuffle Randomized round-robin (evenly distribute
load to downstream Bolts)
Fields Ensures all Tuples with the same Field
value(s) are always routed to the same Task
All Replicates Stream across all the Bolt’s
Tasks (use with care)
Other options Including custom RYO grouping logic

25. Page25
Compile Time
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields(”sentence"));
}

26. Page26
Example Spout Code (1 of 2)
public class RandomSentenceSpout extends BaseRichSpout {
SpoutOutputCollector _collector;
Random _rand;
@Override
public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
_collector = collector;
_rand = new Random();
}
@Override
public void nextTuple() {
Utils.sleep(100);
String[] sentences = new String[]{ "the cow jumped over the moon", "an apple a day keeps
the doctor away", "four score and seven years ago", "snow white and the seven dwarfs",
"i am at two with nature" };
String sentence = sentences[_rand.nextInt(sentences.length)];
_collector.emit(new Values(sentence));
}
Continued next page…
Storm uses open to open the spout and provide it with its configuration,
a context object providing information about components in the
topology, and an output collector used to emit tuples.
Storm uses nextTuple to request
the spout emit the next tuple.
The spout uses emit to send a
tuple to one or more bolts.
Name of the spout class. Storm spout class used as a “template”.

27. Page27
Example Spout Code (2 of 2)
@Override
public void ack(Object id) {
}
@Override
public void fail(Object id) {
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields(”sentence"));
}
}
Storm calls the spout’s ack method to signal that
a tuple has been fully processed.
Storm calls the spout’s fail method to signal
that a tuple has not been fully processed.
The declareOutputFields
method names the fields in a tuple.
Continued…

28. Page28
Example Bolt Code
public static class ExclamationBolt extends BaseRichBolt {
OutputCollector _collector;
public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
_collector = collector;
}
public void execute(Tuple tuple) {
_collector.emit(tuple, new Values(tuple.getString(0) + "!!!"));
_collector.ack(tuple);
}
public void cleanup(); {
}
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("word"));
}
}
The prepare method
provides the bolt with
its configuration and
an
OutputCollector
used to emit tuples.
The execute method
receives a tuple from a
stream and emits a
new tuple. It also
provides an ack
method that can be
used after successful
delivery.
The cleanup method
releases system
resources when bolt is
shut down.
Names the fields in the output
tuples. More detail later.
Name of the bolt class. Bolt class used as a “template.”

29. Page29
Example Topology Code
public static main(String[] args) throws exception {
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(“words”, new TestWordSpout());
builder.setBolt(“exclaim1”, new NewExclamationBolt()).shuffleGrouping(“words”);
builder.setBolt(“exclaim2”, new NewExclamationBolt()).shuffleGrouping(“exclaim1”);
Config conf = new Config();
StormSubmitter.submitTopology(”add-exclamation", conf, builder.createTopology());
}
This code…
words exclaim1 exclaim2
shuffleGrouping shuffleGrouping
…builds this
Topology.
runs code in
TestWordSpout()
runs code in
NewExclamationBolt()
runs code in
NewExclamationBolt()

30. Page30
DEMO
Develop Word Count Topology

31. Page31
Runtime
Nimbus
Supervisor
Supervisor
Supervisor
Supervisor

32. Page32
Physical View

33. Page33
Topology Submitter uploads topology:
• topology.jar
• topology.ser
• conf.ser
Topology Deployment

34. Page34
Topology Deployment
Nimbus calculates assignments and sends to Zookeeper

35. Page35
Topology Deployment
Supervisor nodes receive assignment information
via Zookeeper watches

36. Page36
Topology Deployment
Supervisor nodes download topology from Nimbus:
• topology.jar
• topology.ser
• conf.ser

37. Page37
Topology Deployment
Supervisors spawn workers (JVM processes)

38. Page38
DEMO
Submit Topology to Storm Topology

39. Page39
Additional Features
FAIL

40. Page40
Local Versus Distributed Storm Clusters
The topology program code submitted to Storm using storm jar is
different when submitting to local mode versus a distributed cluster.
The submitTopology method is used in both cases.
• The difference is the class that contains the submitTopology method.
Config conf = new Config();
LocalCluster cluster = new LocalCluster();
LocalCluster.submitTopology("mytopology", conf, topology);
Config conf = new Config();
StormSubmitter.submitTopology("mytopology", conf, topology);
Instantiate a local
cluster object.
Submit a topology
to a local cluster.
Submit a topology to a
distributed cluster.
Same method
name, different
classes
Same method
name, different
classes.

41. Page41
Reliable Processing
Bolts may emit Tuples Anchored to one received.
Tuple “B” is a descendant of Tuple “A”

42. Page42
Reliable Processing
Multiple Anchorings form a Tuple tree
(bolts not shown)

43. Page43
Reliable Processing
Bolts can Acknowledge that a tuple
has been processed successfully.
ACK

44. Page44
Reliable Processing
Bolts can also Fail a tuple to trigger a spout to
replay the original.
FAIL

45. Page45
Reliable Processing
Any failure in the Tuple tree will trigger a
replay of the original tuple

46. Page46
More Stuff
à Topology description/deployment options
– Flux
– Storm SQL
à Polyglot development
à Micro-batching with Trident
à Fault tolerance & deployment isolation
à Integrations
– Messaging; Kafka, Redis, Kestrel, Kinesis, MQTT, JMS
– Databases; HBase, Hive, Druid, Cassandra, MongoDB, JDBC
– Search Engines; Solr, Elasticsearch
– HDFS
– And more!

47. Page47
DEMO
Kafka > Storm > HBase Topology in a Local Cluster

48. Page48
Kafka > Storm > HBase Example
Requirements:
• Land simulated server logs into Kafka
• Configure a Kafka Bolt to consume the server log messages
• Ignore all messages that are not either WARN or ERROR
• Persist WARN and ERROR messages into HBase
– Keep 10 most recent messages for each server
– Maintain a running total of these concerning messages
• Publish these messages back to Kafka
Kafka
Kafka
HBase
HBaseParse FilterKafka
Kafka

49. Page49
Questions?
Lester Martin – Hadoop/Spark/Storm Trainer & Consultant
lester.martin@gmail.com
http://lester.website (links to blog, twitter, github, LI, FB, etc)
THANKS FOR YOUR TIME!!