Twitter Storm

Twitter Storm
Realtime distributed computations

Sergey Lukjanov <slukjanov@mirantis.com>
Dmitry Mescheryakov <dmescheryakov@mirantis.com>

Wednesday, October 3, 12

Real-time data processing

2

before Twitter Storm:

network of queues and workers

2



MESSAGES QUEUE

2



Message
routing can be
complex!

MESSAGES QUEUE

2


MESSAGES
QUEUE

MESSAGES QUEUE

MESSAGES QUEUE

3

Queues replication is
needed for reliability MESSAGES
QUEUE

MESSAGES QUEUE

MESSAGES QUEUE

3

QUEUE

MESSAGES QUEUE

Hard to maintain
queues

MESSAGES QUEUE

3

QUEUE

Each new
computation branch
requires routing
MESSAGES QUEUE reconﬁguration

Hard to maintain
queues

MESSAGES QUEUE

3

Twitter Storm

4

Twitter Storm
distributed;

4

Twitter Storm
distributed;

fault-tolerant;

4

Twitter Storm
distributed;

fault-tolerant;

real-time;

4

Twitter Storm
distributed;

fault-tolerant;

real-time;

computation;

4

Twitter Storm
distributed;

fault-tolerant;

real-time;

computation;

fail-fast components.

4

(Very) basic info

5

(Very) basic info
created by Nathan Marz from Backtype/Twitter;

5

(Very) basic info

Eclipse Public License 1.0;

5

(Very) basic info


open sourced at September 19th, 2011;

5

(Very) basic info



about 16k Java and 7k Clojure LoC;

5

(Very) basic info




most watched Java repo at Github (> 4k watchers);

5

(Very) basic info




most watched Java repo at Github (> 4k watchers);

active UG.

5

Current status

6

Current status
current stable release: 0.8.1;

6

Current status

0.8.2 with small bug ﬁxes is already on the way;

6

Current status


0.9.0 with major core improvements is planned;

6

Current status



not very active contributions, we can try to get into;

6

Current status



not very active contributions, we can try to get into;

used by over 30 companies (such as Twitter, Groupon,
Alibaba, GumGum, etc).

6

Key properties

7

Key properties
extremely broad set of use cases:

streams processing;

database updating;

distributed rpc;

7

Key properties

streams processing;

database updating;

distributed rpc;

scalable and extremely robust;

7

Key properties

streams processing;

database updating;

distributed rpc;


guarantees no data loss;

7

Key properties

streams processing;

database updating;

distributed rpc;



fault-tolerant;

7

Key properties

streams processing;

database updating;

distributed rpc;



fault-tolerant;

programming language agnostic.
7

Key concepts
Tuples (ordered list of elements)

8

Key concepts
Tuples (ordered list of elements)

( “Saratov”, “slukjanov”, “event1”, “10/3/12 16:20”)

8

Key concepts
Streams (unbounded sequence of tuples)

9

Key concepts
Streams (unbounded sequence of tuples)

TUPLE TUPLE TUPLE TUPLE TUPLE

9

Key concepts
Spouts (source of streams)

10

Key concepts


10

Key concepts


Spouts can talk with:
queues;

logs;

API calls;

event data.
10

Key concepts
Bolts (process tuples and create new streams)

11

Key concepts
Bolts (process tuples and create new streams)
LE
TUP
LE
TUP
LE
TUP
LE
TUP
LE
TUP


TUP
LE
TUP
LE
TUP
LE
TUP
LE
TUP
LE

11

Key concepts
You can do the following things in Bolts:
apply functions / transformations;

ﬁlter;

aggregation;

streaming joins;

access DBs, APIs, etc...

12

Key concepts
Topologies (a directed graph of Spouts and Bolts)

13

Key concepts

LE
TUP
LE
TUP
LE
TUP
LE
TUP
LE
TUP


TUP
LE
TUP
LE
TUP
LE
TUP
LE
TUP
L

13

Key concepts

TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE

LE LE
TUP TUP
LE LE
TUP TUP
LE LE
TUP TUP
LE LE
TUP TUP
LE LE
TUP TUP

TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE TUPLE

14

Key concepts
Tasks (instances of spouts and bolts)

15

Key concepts
Tasks (instances of spouts and bolts)

Task 1

Task 2

Task 3

Task 4

15

Key concepts
Supervisor
Cluster

Zookeeper Supervisor

Nimbus Zookeeper Supervisor


Supervisor

16

Key concepts
Supervisor
Cluster

UI Zookeeper Supervisor



Supervisor

16

Key concepts
Supervisor
Cluster



Hadoop’s Job Zookeeper Supervisor
tracker

Supervisor

16

Key concepts
Supervisor
Cluster



Hadoop’s Job Zookeeper Supervisor
tracker

Hadoop’s
Supervisor
Task tracker
16

Based on

17

Based on
Apache Zookeeper (maintaining conﬁgs);

17

Based on

MQ (transport layer);

17

Based on

Apache Thrift (cross-language bridge, rpc);

17

Based on


LMAX Disruptor (bounded prod-cons queue);

17

Based on


LMAX Disruptor (bounded prod-cons queue);

Kryo (serialization framework).

17

Grouping

18

Grouping
shuﬄe (randomly and evenly distributed);

18

Grouping

local or shuﬄe (local workers are preferred);

18

Grouping


fields (the stream is partitioned by specified fields);

18

Grouping



all (the stream is replicated across all the bolt’s tasks);

18

Grouping




global (the entire stream goes to a single bolt’s task);

18

Grouping





direct (producers could directly emit tuples);

18

Grouping





direct (producers could directly emit tuples);

custom (implement interface CustomStreamGrouping).

18

WordCount sample

19

WordCount sample
random sentence generator;

19

WordCount sample

sentence splitter;

19

WordCount sample

sentence splitter;

word counter;

19

WordCount sample

sentence splitter;

word counter;

ping spout (metronome).

19

WordCount sample

20

WordCount sample
SENTENCE
GENERATOR

SENTENCE SENTENCE

20

WordCount sample
SENTENCE SENTENCE
GENERATOR SPLITTER

SENTENCE SENTENCE WORD WORD

20

WordCount sample
SENTENCE SENTENCE WORD
GENERATOR SPLITTER COUNTER


20

WordCount sample
SENTENCE SENTENCE GROUP WORD
GENERATOR SPLITTER BY WORD COUNTER


20

WordCount sample


PING

PING
PING
GENERATOR

20

WordCount sample


PING

PING
PING
GENERATOR

SOUT

20

WordCount sample


PING

PING
PING
GENERATOR

DB

20

Sentence generator
public class RandSentenceGenerator extends BaseRichSpout {
private SpoutOutputCollector collector;
private Random random;

private String[] sentences;

@Override
public void open(Map map, TopologyContext ctx, SpoutOutputCollector collector) {
this.collector = collector;
this.random = new Random();
this.sentences = <sentences array>;
}

@Override
public void nextTuple() {
Utils.sleep(10);
String sentence = sentences[random.nextInt(sentences.length)];
collector.emit(new Values(sentence));
}

@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
declarer.declare(new Fields("sentence"));
}
}

21

Sentence splitter
public class SplitSentence extends BaseBasicBolt {
@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String sentence = tuple.getString(0);
for (String word : sentence.split("s")) {
collector.emit(new Values(word));
}
}

@Override
declarer.declare(new Fields("word"));
}
}

22

Word count
public class WordCount extends BaseBasicBolt {
private HashMultiset<String> words = HashMultiset.create();

@Override
public void prepare(Map conf, TopologyContext ctx) {
super.prepare(conf, ctx);
this.logger = Logger.getLogger(this.getClass());
this.name = ctx.getThisComponentId();
this.task = ctx.getThisTaskIndex();
}

@Override
public void execute(Tuple tuple, BasicOutputCollector collector) {
String source = tuple.getSourceComponent();
if ("split".equals(source)) {
words.add(tuple.getString(0));
} else if ("ping".equals(source)) {
logger.warn("RESULT " + name + ":" + task + " :: " + words);
}
}

@Override
declarer.declare(new Fields("word", "count"));
}
}
23

Topology builder
public class WordCounter {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();

builder.setSpout("source", new RandSentenceGenerator(), 3);
builder.setSpout("ping", new PingSpout());

builder.setBolt("split", new SplitSentence(), 8)
.shuffleGrouping("source");
builder.setBolt("count", new WordCount(), 12)
.fieldsGrouping("split", new Fields("word"))
.allGrouping("ping");

<topology submitting>

}
}

24

Topology submitter
public class WordCounter {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();

<building topology>

Config conf = new Config();
conf.setDebug(true);
conf.setNumWorkers(3);

StormSubmitter.submitTopology("tplg-name", conf, builder.createTopology());
}
}

25

Multilang support

26

Multilang support
DSLs for Scala, JRuby and Clojure;

26

Multilang support

ShellSpout, ShellBolt;

26

Multilang support

ShellSpout, ShellBolt;

json-based protocol:

receive/emit tuples;

ack/fail tuples;

write to logs.

26

Online logs processing

27

Application RABBITMQ

Application QUEUE

Application

Twitter Storm Cluster

STORAGE CASSANDRA

27


28


RABBITMQ

28

QUEUE
CONSUMER

LOG
MESSAGES

RABBITMQ

28

QUEUE MESSAGE
CONSUMER PARSER
SHUFFLE GROUPING
MESSAGE MESSAGE

LOG
MESSAGES

RABBITMQ

28

QUEUE MESSAGE EVENT
CONSUMER PARSER AGGREGATOR
SHUFFLE GROUPING FIELDS GROUPING
MESSAGE MESSAGE EVENT EVENT

LOG
MESSAGES

RABBITMQ

28

QUEUE MESSAGE EVENT
CONSUMER PARSER AGGREGATOR
SHUFFLE GROUPING FIELDS GROUPING
MESSAGE MESSAGE EVENT EVENT

LOG REALTIME
MESSAGES INFO & STATS

RABBITMQ CASSANDRA

28

Storm fault-tolerance

29

Storm fault-tolerance
Parts of Storm cluster:

Zookeeper nodes;

Nimbus (master) node;

Supervisor nodes.

29

Nimbus as a point of failure

30

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;

30

when Nimbus is down:

topologies continue to work;

tasks from failing nodes aren’t respawned;

can’t upload a new topology or rebalance an old one;

impossible to run Nimbus at another node:

either ﬁx the failed node;

or create new and resubmit all topologies.

30

Tuple types
spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent
tuple(s) (child or spout ones).

31

Tuple types
spout tuple - emitted from Spouts;

child tuple - emitted from Bolts, based on parent
tuple(s) (child or spout ones).
[“the”] [“the”, 1]

[“cow”] [“cow”, 1]

[“the cow jumped [“jumped”] [“jumped”, 1]
over the moon”]

[“over”] [“over”, 1]

[“the”] [“the”, 2]

[“moon”] [“moon”, 1]

31

Reliability API Guaranties
public class QueueConsumer extends BaseRichSpout {

...

@Override
public void nextTuple() {
Message msg = queueClient.popMessage();
collector.emit(msg.getPayload(), msg.getId());
}

@Override
public void ack(Object msgId) {
queueClient.ack(msgId);
}

@Override
public void fail(Object msgId) {
queueClient.fail(msgId);
}

...

}

32

Tuple tree tracking

33

Tuple tree tracking
spout tuple creation:

collector.emit(values, msgId);

33

Tuple tree tracking


child tuple creation:

collector.emit(parentTuples, values);

33

Tuple tree tracking




tuple end of processing:

collector.ack(tuple);

33

Tuple tree tracking




tuple end of processing:

collector.ack(tuple);

tuple failed to process:

collector.fail(tuple);
33

Disabling reliability API

34

globally:

Config.TOPOLOGY_ACKER_EXECUTORS = 0;

34

globally:


on topology level:


34

globally:


on topology level:


for a single tuple:


34

Acker system impl
every tuple is assigned a random 64-bit ID

35

Acker system impl

[1] [2] [3]
Spout Bolt A Bolt B Bolt C

35

Acker system impl

[1] [2] [3]

[1] emit

35

Acker system impl

[1] [2] [3]

[1] emit
[2] emit

35

Acker system impl

[1] [2] [3]

[1] emit
[2] emit
[1] ack

35

Acker system impl

[1] [2] [3]

[1] emit
[2] emit
[1] ack
[3] emit

35

Acker system impl

[1] [2] [3]

[1] emit
[2] emit
[1] ack
[3] emit
[2] ack

35

Acker system impl

[1] [2] [3]

[1] emit
[2] emit
[1] ack
[3] emit
[2] ack
[3] ack

35

Acker - simpliﬁed algo

36

tuple tree:

{ Spout tuple ID, set };

36

tuple tree:

{ Spout tuple ID, set };

message processing:

[tuple_id] emit: set.add(tuple_id);

[tuple_id] ack: set.remove(tuple_id);

if (set.size == 0): send ack to parent spout.

36

Acker - real algo
tuple tree:

{ Spout tuple ID, ackVal: int64 };

message processing:

[tuple_id] emit: ackVal ^= tuple_id;

[tuple_id] ack: ackVal ^= tuple_id;

if (ackVal == 0): send ack to parent spout.

37

Correctness of the tracking

38

bolt fails before sending ack for a tuple:

no ack arrive before timeout, spout tuple fails;

38



acker fails before acking tuple tree processing:

-- the same as above --;

38



acker fails before acking tuple tree processing:

-- the same as above --;

spout fails before acking message:

the message source should handle client’s death.

38

Reliability API - Conclusion

39

easy to dismiss:

on message - at most one processing;

39

easy to dismiss:


if using, little overhead and high durability:

one message - at least one processing;

39

easy to dismiss:


if using, little overhead and high durability:

one message - at least one processing;

with some further work (transactions, Trident API):

one message - exactly one processing.

39

Transactional approach: design #1

40


MESSAGE TUPLE COMMIT

Spout Bolt A

40



Spout Bolt A

input provides messages in strong order;

40



Spout Bolt A


each message is assigned Transaction ID;

40



Spout Bolt A


each message is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).

40


BATCH OF BATCH
MESSAGES OF TUPLES COMMIT

Spout Bolt A


each batch of messages is assigned Transaction ID;

if (curr_tx_id > prev_tx_id) commit(result, curr_tx_id).

41

the same as #2, but each transaction is split:

processing phase;

commit phase;

42


processing phase;

commit phase;

process phases might intersect for diﬀerence
transactions;

42


processing phase;

commit phase;

process phases might intersect for diﬀerence
transactions;

commit phases go in strong order.

42

Trident API: Intro

43

Trident API: Intro
high-level abstraction for doing realtime computations;

43

Trident API: Intro

high throughput (millions of messages per second);

43

Trident API: Intro


stateful stream processing;

43

Trident API: Intro



low latency distributed querying;

43

Trident API: Intro




diﬀerent semantics (including exactly-once one);

43

Trident API: Intro




diﬀerent semantics (including exactly-once one);

smth. like Pig or Cascading.

43

Trident API: Operations

44

partition-local operations (w/o network transfer):

function, ﬁlter, partitionAggregate, stateQuery, etc;

44



repartitioning operations (grouping);

44




aggregations operations:

aggregate, persistentAggregate;

44






operations on grouped streams;

44






operations on grouped streams;

merges and joins.

44

Trident API: Demo
TridentTopology topology = new TridentTopology();
TridentState wordCounts =
topology.newStream("spout1", new FixedBatchSpout())
.each(new Fields("sentence"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.persistentAggregate(new MemoryMapState.Factory(), new Count(),
new Fields("count"))
.parallelismHint(6);

topology.newDRPCStream("words")
.each(new Fields("args"), new Split(), new Fields("word"))
.groupBy(new Fields("word"))
.stateQuery(wordCounts, new Fields("word"), new MapGet(), new Fields("count"))
.each(new Fields("count"), new FilterNull())
.aggregate(new Fields("count"), new Sum(), new Fields("sum"));

Config config = new Config();
config.setMaxSpoutPending(100);
cluster.submitTopology("word-count-tplg", config, topology.build());

DRPCClient client = new DRPCClient("drpc.server.host", 3772);

System.out.println(client.execute("words", "cat dog the man"));
System.out.println(client.execute("words", "cat"));
// prints the JSON-encoded result, e.g.: "[[5078]]"

45

Q&A

46

Twitter Storm

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Mehr von Sergey Lukjanov

Mehr von Sergey Lukjanov (8)

Twitter Storm