TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
No data loss pipeline with apache kafka
1. No Data Loss Pipeline
with Apache Kafka
Jiangjie (Becket) Qin @ LinkedIn
2. ● Data loss
o producer.send(record) is called but record
did not end up in consumer as expected
● Message reordering
o send(record1) is called before send(record2)
o record2 shows in broker before record1 does
o matters in cases like DB replication
Data loss and message reordering
3. Kafka based data pipeline
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
Today’s Agenda:
● No data loss
● No message reordering
● Mirror maker enhancement
○ Customized consumer rebalance listener
○ Message handler
4. Synchronous send is safe but slow...
producer.send(record).get()
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
5. Using asynchronous send with callback can
be tricky
producer.send(record,callback)
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
6. Producer can cause data loss when
● block.on.buffer.full=false
● retries are exhausted
● sending message without using acks=all
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
7. Is this good enough?
producer.send(record,callback)
● block.on.buffer.full=TRUE
● retries=Long.MAX_VALUE
● acks=all
● resend in callback when message send failed
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
9. Message reordering might also happen if:
● producer is closed carelessly
o close producer in user thread, or
o close without using close(0)
Producer
Record
Accumulator
Sender Thread
Kafka Broker
Timeline
1.msg 0
2.callback(msg 0) ack expt.
User
Thread
close prod.
3.msg 1
notify
10. ● close producer in the callback on error
● close producer with close(0) to prevent further
sending after previous message send failed
Producer
Record
Accumulator
Sender Thread
Kafka Broker
Timeline
1.msg 0
2.callback(msg 0) ack expt.
User
Thread
close(0)
notify
11. To prevent data loss:
● block.on.buffer.full=TRUE
● retries=Long.MAX_VALUE (for some use cases)
● acks=all
To prevent reordering:
● max.in.flight.requests.per.connection=1
● close producer in callback with close(0) on send failure
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
12. Not a perfect solution:
● Producer needs to be closed to guarantee message
order. E.g. In mirror maker, one message send failure
to a topic should not affect the whole pipeline.
● When producer is down, message in buffer will still be
lost
Producer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
13. Correct producer setting is not enough
● acks=all still can lose data when unclean
leader election happens.
● Two replicas are needed at any time to
guarantee data persistence.
Kafka Brokers
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
14. ● replication factor >= 3
● min.isr = 2
● Replication factor > min.isr
o If replication factor = min.isr, partition will
be offline when one replica is down
Kafka Brokers
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
16. ● Consumer might lose message when offsets are
committed carelessly. E.g. commit offsets before
processing messages completely
o Disable auto.offset.commit
o Commit offsets only after the messages are
processed
Consumer
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
17. Kafka based data pipeline
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
Today’s Agenda:
● No data loss
● No message reordering
● Mirror maker enhancement
○ Customized consumer rebalance listener
○ Message handler
18. ● Consume-then-produce pattern
● Only commit consumer offsets of
messages acked by target cluster
● Default to no-data-loss and no-
reordering settings
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
19. ● Customized Consumer Rebalance
Listener
o Can be used to propagate topic change from
source cluster to target cluster. E.g.
partition number change, new topic
creation.
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
20. ● Customized Message Handler, useful for
o partition-to-partition mirror
o filtering out messages
o message format conversion
o other simple message processing
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker
21. ● startup/shutdown acceleration
o parallelized startup and shutdown
o 26 nodes cluster with 4 consumer each
takes about 1 min to startup and shutdown
Mirror Maker Enhancement
Kafka Cluster
(Colo 1)
Producer
Kafka Cluster
(Colo 2)
ConsumerMirror Maker