Interacting with Apache Kafka seems straightforward at first, you “just” push and pull messages. Yet it can quickly become a source of frustration as the user encounters timeouts, vague error descriptions and disappearing messages. Experience helps a lot and I’m here to share what I know. In this talk you will learn the tips & tricks I wish I had known at the beginning of my Apache Kafka journey. We’ll discuss topics like producer acknowledgments, server and consumer parameters (auto_offset_reset anyone?) that are commonly overlooked causing lots of developer’s pain. I’ll share with you how to generate code that works as expected on the first run, making your first integration painless. These tips will kickstart your Apache Kafka experience in Python and save you hours of debugging.

1. Practical
Tips & Tricks
for Apache Kafka
Messages Integration
Francesco Tisiot - Developer Advocate @ftisiot

2. @ftisiot

3. @ftisiot
Libraries

4. @ftisiot
kafka-python pip install kafka-python
Consume
Admin
Produce

5. @ftisiot
confluent-kafka-python
pip install confluent-kafka
Schema
Registry
Avro
Karapace

6. @ftisiot
Connect

7. @ftisiot
KafkaProducer(
bootstrap_servers='kafka:1234',
)
How many brokers?

8. @ftisiot
KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
)

9. @ftisiot
KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
security_protocol='SSL',
ssl_cafile='ca.pem',
ssl_certfile='service.cert',
ssl_keyfile='service.key'
)
SSL

10. @ftisiot
Produce

11. @ftisiot
producer = KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
)
Minimal Setup

12. @ftisiot
producer = KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_serializer=
lambda v: json.dumps(v).encode(‘ascii'),
)
Data Conversion
https://aiven.io/blog/tips-for-designing-payloads

13. @ftisiot
producer = KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_serializer=
lambda v: json.dumps(v).encode('ascii'),
acks=1,
)
Confirmation
[0,1,'all']

14. @ftisiot
producer = KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_serializer=
lambda v: json.dumps(v).encode('ascii'),
acks=1,
retries=5,
) What do we do in case of
Errors?

15. @ftisiot

16. @ftisiot
producer = KafkaProducer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_serializer=
lambda v: json.dumps(v).encode('ascii'),
acks=1,
retries=5,
max_in_flight_requests_per_connection=1
)
Avoids out of order messages

17. @ftisiot
producer.send(
'my_shop_topic',
key=b'Milan',
value=
{
'order_id': 1,
'product_name': 'Nutella Jar',
'quantity': 12
}
)
batch_size

18. @ftisiot
producer.flush(timeout=10)

19. @ftisiot
KafkaTimeoutError: Failed to update
metadata after 60.0 secs.
Pushing to
Non Existing
Topic
auto.create.topics.enable=true

20. @ftisiot
Admin

21. @ftisiot
from kafka.admin import KafkaAdminClient, NewTopic
admin_client = KafkaAdminClient(
bootstrap_servers='kafka:1234,kafka2:5678'
)
topic_list = [
NewTopic(
name='short-retention-topic',
num_partitions=5,
replication_factor=3,
topic_configs={'retention.ms': 10000}
)]
admin_client.create_topics(
new_topics=topic_list, validate_only=False
)

22. @ftisiot
Consume

23. @ftisiot
consumer = KafkaConsumer(
bootstrap_servers=
'kafka:1234,kafka2:5678'
)
consumer.subscribe(
topics=['testA']
)
for message in consumer:
print(message.value)

24. @ftisiot
auto_offset_reset='earliest'

25. @ftisiot
consumer.subscribe(
topics=['testA','testB']
)

26. @ftisiot
consumer.subscribe(
pattern='test*'
)
consumer.subscribe(
topics=['testA','testB']
)
metadata_max_age_ms

27. @ftisiot
consumer = KafkaConsumer(
bootstrap_servers='kafka:1234,kafka2:5678',
)

28. @ftisiot
consumer = KafkaConsumer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_deserializer =
lambda v: json.loads(v.decode('ascii')),
)

29. @ftisiot
consumer = KafkaConsumer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_deserializer =
lambda v: json.loads(v.decode('ascii')),
auto_offset_reset='earliest',
metadata_max_age_ms=10000,
)

30. @ftisiot
consumer = KafkaConsumer(
bootstrap_servers='kafka:1234,kafka2:5678',
value_deserializer =
lambda v: json.loads(v.decode('ascii')),
auto_offset_reset='earliest',
metadata_max_age_ms=10000,
group_id='snake-consumer-group',
client_id='franz-snake'
)

31. @ftisiot
subscribe() assign()

32. @ftisiot
consumer.subscribe(topics=['test'])
group_id='snakes'

33. @ftisiot
consumer.assign(
[TopicPartition('test', 0)]
)

34. @ftisiot
Connect
Produce
Admin
Consume

35. @ftisiot
Resources
https://aiven.io
https://kafka-python.readthedocs.io/
https://aiven.io/blog/create-your-own-data-stream-for-
kafka-with-python-and-faker
kafka-summit-2021
500$