Speaker: Damien Gasparina, Engineer, Confluent Here's how to fail at Apache Kafka brilliantly! https://www.meetup.com/Paris-Data-Engineers/events/260694777/

1. 1Confidential
Apache Kafka® – Pattern and Anti-Pattern
How to fail at Apache Kafka brilliantly!

2. This is me
- Engineer at Confluent
- Ex-MongoDB, ex-Oracle
- Cute smooth curly hair

5. Distributed Logs
Scalable, Fault Tolerant, Concurrent, Strongly Ordered, Retentive

6. Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

7. Producer Consumer
Partitions
Kafka Cluster

9. … which is true for
any data systems

10. And today we will discuss
about the most common one

11. Problem #1
Not thinking about Durability
* How to fail *

13. Durability is achieved
throught Replication

14. When will the cluster acknowlege ?
L
Producer

15. As soon as it is on the page cache of
the leader...
L
Producer

16. Acks
Acks=1 (default value) : good for latency
Acks=all : good for durability

17. Replication before acknolowdging
L
Producer
Wait for Data
replication
Acks=all

18. Acks = All
This means the leader will wait for the
full set of in-sync replicas
to acknowledge the record.

19. But only to the In-Sync Replicas...
L
Producer
Acks=all

20. … which could be only one server
L
Producer
Acks=all

22. min.insync.replicas
minimum number of replicas that must
Acknowledge (default 1)

23. Data Durability while Producing
Tune it with the parameters
acks and min.isr

24. The default values are optimized
For availability & latency
If durability is more important, tune it !

25. For consumer, it’s a lot
easier
Consumer can read only replicated data

26. If you are using transaction
isolation.level=read_commited
(default read_uncommited)

27. Think about data durability and decide
of the best trade-off for you
* How to succeed *

28. Problem #2
Focusing only on
the happy path
* How to fail *

30. Some parameters to know

31. Builtin Retries
Retries : will cause the client to resend any record
whose send fails with a potentially transient error.
Default value : MAX_INT (before AK 2.1 : 0...)

32. What’s happening in case of issue with
retry ?
Producer No longer leader
L
The leader moved to a
different broker

33. Use builtin retries !
* How to succeed *

34. But you are exposed to a
different kind of issue

35. Message duplication
L
Producer
Timeout

36. Builtin Idempotency
enable.idempotence: When set to 'true',
the producer will ensure that exactly
one copy of each message is written.
Default value: false

37. With Retries and Idempotency
L
Producer
Timeout
UUID
UUID

38. With Retries and Idempotency
L
Producer
Timeout
UUID
UUID
UUID
UUIDUUID

39. Use builtin idempotency!
* How to succeed *

40. Problem #4
No Idempotent consumer
* How to fail *

42. At least once (default)
At most once
Exactly Once ?

43. Embrace at least once !
* How to succeed *

44. Rely on Kafka Streams with
Exactly Once !
* How to succeed *

46. Problem #4
No exception handling
* How to fail *

48. Producer Consumer
What to do in case of an error ?

49. Plenty of potential issues

50. 1) A message can not be processed
2) A message doesn’t have the expected
schema

51. Multiple strategy

52. 1) Retry
while (this.getRunning()) {
try {
var consumerRecords = consumer.poll(1000) ;
} catch (Exception e) {
Logger.error(e);
continue ;
}
for (var record : consumerRecords) {
try {
/* Processing messages */
} catch (Exception e) { … }
}
}

53. 2) Write to a dead letter queue and
continue
while (this.getRunning()) {
var consumerRecords = consumer.poll(1000) ;
for (var record : consumerRecords) {
try {
/* Processing messages */
} catch (Exception e) {
producer.send(« dead-my-topic », new ProducerRecord(…)) ;
Logger.error(e);
}
}
}

54. 3) Ignore & continue
kafkaProducer.send(record, (
(metadata, exception)→ {
if (exception != null) {
/* Something bad happened */
/* But those are ephemaral data anyway */
Logger.error(exception) ;
}
})
);

55. There is no silver bullet
solution

56. Handle the exceptions !
* How to succeed *
https://eng.uber.com/reliable-reprocessing/

57. Problem #5
No data governance
* How to fail *

59. Changes in producers might
impact consumers

60. Sharing dataset!
As soon as producer and consumer are
Part of different teams.
Consider using a Schema Registry !
* How to succeed *

61. Consumer
Producer
Application 1
Consider using quotas
as soon as there is multi-tenancy
Producer
Application 2
Application 3
Consumer
Application 4

62. Problem #6
Installing prod Sunday night
* How to fail *

63. A typical devops team

64. Misconfiguration !
If you use the default configuration…
You will have surprises!
* How to fail *

65. The best things to do is to .…
Read the goddam manual !
Running Apache Kafka in Production
Running Apache Zookeeper in Production
* How to succeed *

66. Not using security
* How to fail *

67. Ordering?
* How to fail *

68. Too many partitions
* How to fail *

69. Too few partitions
* How to fail *

70. Calling external service in
Kafka Streams
* How to fail *

71. No Monitoring
* How to fail *

72. Problems
Data durability
Error Handling & transient issues
Data governance
Misconfiguration
Monitoring

73. DO
Understand Apache Kafka
Configure your clients & cluster properly
Follow engineering best-practices
https://www.confluent.io/white-paper/optimizing-your-apache-kafka-deployment/

74. And if you can not do all
of this
Consider using a Cloud Service

75. And if you can
We are hiring !

76. References
• Running Apache Kafka in production : https://docs.confluent.io/current/kafka/deployment.html
• Running Apache Zookeeper in production : https://docs.confluent.io/current/zookeeper/deployment.html
• Confluent Reference Architecutre :
https://www.confluent.io/resources/apache-kafka-confluent-enterprise-reference-architecture/
• The goddam documentation : https://docs.confluent.io/current/getting-started.html
• Kafka Summit : https://kafka-summit.org/

77. I hope it has been useful :=)

78. Questions ?