Confluent and Apache Kafka. Patterns / Anti Patterns

1. Confluent & Apache Kafka
Patterns / Anti-patterns
Marcelo Manta and Jean Louis Boudart

2. Problem ?

3. With

4. How

5. Store &ETL Process
Publish &
Subscribe
In short

6. From a simple idea

7. From a simple idea

8. with great properties !
• Scalability
• Retention
• Durability
• Replication
• Security
• Resiliency
• Throughput
• Ordering
• Exactly Once Semantic
• Transaction
• Idempotency
• Immutability
• …
8

9. So goooooood

10. What could potentially go wrong
?

12. …which is true for any data
systems

13. Not thinking about Durability

14. Data durability
14
If you didn’t think
about it… it’s not
durable!

18. And you might have lost data!

19. Data durability
1
9
Kafka is notwaiting
for a disk flush by
default.
Durability is achieved
through replication.

24. Data durability
24
It depends on your
configuration...

29. Data durability
29
acks=1 (default
value) good for
latency
acks=all
good for durability

31. acks=all
3
1
The leader will wait
for the full setof
in-sync replicas to
acknowledge the
record.

35. min.insync.replicas
35
minimum number of
replicas that must
acknowledge.
Default is 1.

37. Data Durability while Producing ?
Tune it with the parameters acks
and min.insync.replicas

38. defaults
38
The default values
are optimized for
availability & latency.
If durability is more
important, tune it!

39. Deploying on multi datacenters
?

40. r ·-::i- _• ;.!!.c_:,;--... . , L.L:.-_::;;.
.--. •c - ?:
. '-:"
?"%
-:-
':..
¥..
c.
"
.;
'<
-..
".:-,:,'<',,,,"'=
,...,,../-.-,<.a!.J V(fl..?---..,..'.. '?,
8i''{..
·g
----,-,,.>- *cf" ,., '--. z _ _,.,....,-,;,,:_).
:; ?li-·-:-... _J
C re, .Dubli- ,',.7_ I(-<'---_
i {..'t
..J
· •--·/4·
.• "' /4,
_; . ,.c.,,;
-"->.
'·r
,
.& ( r ('..
.-, ..._- r,.I, •:;-,
.c
..:;, •.:
.. r (
...
. ) "'IJ J_J
,,v ,/
>
;o
i.;
,
..
•'
.-..
-·

41. Multi-dc
41
It’s quite
complicated...
It’s easy to makeit
wrong on many
levels.
It could be a 3htalk.

42. Multi-dc Disaster recovery for
multi datacenter
42

43. What about the consumers
?

44. consumers
44
Consumer can read
only committed data.

46. Think about data durability and
decide of the best trade-off for you

47. Throughput,
latency, durability,
availability
Optimizing your Apache
Kafka deployment
47

48. Focusing only on the happy path

52. retries
52
It will cause theclient
to resend any record
whose send fails with
a potentially transient
error.
Default value : 0

54. retries
54
Use built in retries !
Bump it from0 to
infinity!

55. retries
55
But you are exposed
to a different kind of
issue…

57. enable.idempotence
57
When set to 'true',
the producer will
ensure that exactly
one copy of each
message is written.
Default value: false

60. Use built in idempotency!

61. But it does not save you from
- Managing exception and failure
- Developing Idempotent consumer

62. No Idempotent consumer

64. At least once (default)
At most once
Exactly Once

71. commit
7
1
Manually committing
aggressively...
Add a huge workload
on Apache Kafka

73. commit
73
Manually committing
aggressively...
Does not provide
exactly once
semantic

74. Embrace at least once

75. Rely on Kafka Streams
with Exactly Once !

76. No exception handling

78. Future<RecordMetadata> send(ProducerRecord<K, V> record);

79. Future<RecordMetadata> send(ProducerRecord<K, V> record,
Callback callback);

80. producer.send(record, (metadata, exception) -> {
});

81. error handling
81
We don’t expect the
unexpected until the
unexpected is
expected.

83. error handling
83
A message can not
be processed

84. error handling
84
A message can not
be processed
A message doesn’t
have the expected
schema

85. Retry

87. Infinite retry

89. Write to a dead letter queue and
continue

91. Ignore and continue

93. No silver bullet

94. Handle the exceptions
!
https://eng.uber.com/reliable-reprocessing/

95. No data governance

99. governance
99
Changes in producers
might impact
consumers

100. governance
100
Schema registry

103. Share Schemas

104. Let bad citizens wander around

106. Leverage Security, ACL and Quota
Security
Authorization and ACLs
Enforcing Client Quotas

107. Installing prod on Sunday night

109. configuration
109
If you use thedefault
configuration…
You will have issues!

110. Please read the doc
Running Kafka in Production
Running ZooKeeper in Production

111. Not configuring your OS

113. os
11
3
Tune at least your
open file descriptors
and mmap count.

114. Configure your os
Running Kafka in Production

115. Disregarding Apache Zookeeper

116. Not understanding Ordering

117. No monitoring

118. Too much partitions

119. Not enough partitions

120. Partition key choice

121. Topics vs Partitions

122. Call external services in Kafka
Streams

123. Questions