Kafka 101 and Developer Best Practices

Kafka 101 &
Developer Best Practices

Agenda
● Kafka Overview
● Kafka 101
● Best Practices for Writing to Kafka: A tour of the
Producer
● Best Practices for Reading from Kafka: The
Consumer
● General Considerations

3
ETL/Data Integration Messaging
Batch
Expensive
Time Consuming
Diﬃcult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)

4
Batch
Expensive
Time Consuming
Diﬃcult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)

5
ETL/Data Integration
Batch
Expensive
Time Consuming
Messaging
Diﬃcult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Transient Messages
Stored records

6
ETL/Data Integration
Batch
Expensive
Time Consuming
Messaging
Diﬃcult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Transient Messages
Stored records
Both of these are a complete mismatch
to how your business works.

7
Transient Messages
Stored records
Messaging
Batch
Expensive
Time Consuming
Diﬃcult to Scale
No Persistence
Data Loss
No Replay
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)
Event Streaming Paradigm
High Throughput
Durable
Persistent
Maintains Order
Fast (Low Latency)

8
Fast (Low Latency)
To rethink data as not stored records
or transient messages, but instead as
a continually updating stream of events

9
Fast (Low Latency)

Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Confluent, Inc.
...
Device
Logs ... ...
...
Data Stores Logs 3rd Party Apps Custom Apps / Microservices
Real-time
Customer 360
Financial Fraud
Detection
Real-time
Risk Analytics
Real-time
Payments
Machine
Learning
Models
...
Event-Streaming Applications
Universal Event Pipeline
Amazon
S3
SaaS
apps
Confluent: Central Nervous System For Enterprise

Confluent uniquely
enables Event Streaming
success
Hall of Innovation
CTO Innovation
Award Winner
2019
Enterprise Technology
Innovation
AWARDS
Confluent founders are
original creators of Kafka
Confluent team wrote 80%
of Kafka commits and has
over 1M hours technical
experience with Kafka
Confluent helps enterprises
successfully deploy event
streaming at scale and
accelerate time to market
Confluent Platform extends
Apache Kafka to be a
secure, enterprise-ready
platform

Scalability of a
Filesystem
Guarantees of a
Database
Distributed By
Design
Rewind and Replay

15
Kafka Broker
Kafka Broker
Kafka Broker
Kafka Broker
Kafka
Producer
Kafka
Producer
Kafka
Producer
Kafka
Consumer
Kafka
Consumer
(grouped)
Kafka
Consumer
/Producer
Kafka Broker
Writers Readers
Kafka Cluster

KAFKA
A MODERN, DISTRIBUTED
PLATFORM FOR DATA
STREAMS

Kafka Topics
my-topic
my-topic-partition-0
broker-1
broker-2
broker-3

Creating a topic
$ kafka-topics --zookeeper zk:2181
--create
--topic my-topic
--replication-factor 3
--partitions 3
Or use the AdminClient API!

Producing to Kafka
Time

Producing to Kafka
Time
C C
C

Kafka’s distributed nature
Broker 1
Topic-1
partition-1
Broker 2 Broker 3 Broker 4
Topic-1
partition-1
Topic-1
partition-1
Leader Follower
Topic-1
partition-2
Topic-1
partition-2
Topic-1
partition-2
Topic-1
partition-3
Topic-1
partition-4
Topic-1
partition-3
Topic-1
partition-3
Topic-1
partition-4
Topic-1
partition-4

Producer
Clients - Producer Design
Producer Record
Topic
[Partition]
[Key]
Value
Serializer Partitioner
Topic A
Partition 0
Batch 0
Batch 1
Batch 2
Topic B
Partition 1
Batch 0
Batch 1
Batch 2
Kafka
Broker
Send()
Retry
?
Fail
?
Yes
No
Can’t retry, throw
exception
Success: return
metadata
Yes

The Serializer
Kafka doesn’t care about what you send to it as long as
it’s been converted to a byte stream beforehand.
JSON
CSV
Avro
Protobufs
XML
SERIALIZERS
01001010 01010011 01001111 01001110
01000011 01010011 01010110
01001010 01010011 01001111 01001110
01010000 01110010 01101111 01110100 ...
01011000 01001101 01001100
(if you must)
Reference
https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html

The Serializer
private Properties kafkaProps = new Properties();
kafkaProps.put(“bootstrap.servers”, “broker1:9092,broker2:9092”);
kafkaProps.put(“key.serializer”, “org.apache.kafka.common.serialization.StringSerializer”);
kafkaProps.put("value.serializer", "io.confluent.kafka.serializers.KafkaAvroSerializer");
kafkaProps.put("schema.registry.url", "https://schema-registry:8083");
producer = new KafkaProducer<String, SpecificRecord>(kafkaProps);
Reference
https://kafka.apache.org/10/documentation/streams/developer-guide/datatypes.html

Producer Record
Topic
[Partition]
[Key]
Value
Record keys determine the partition with the default kafka
partitioner
If a key isn’t provided, messages will be
produced in a round robin fashion
partitioner
Record Keys and why they’re important -
Ordering

Keys are used in the default partitioning algorithm:
partition = hash(key) % numPartitions
Ordering
Producer Record
Topic
[Partition]
AAA
Value
partitioner
partitioner

Ordering
Producer Record
Topic
[Partition]
BBB
Value
partitioner
partitioner

Ordering
Producer Record
Topic
[Partition]
CCC
Value
partitioner
partitioner

Ordering
Producer Record
Topic
[Partition]
DDD
Value
partitioner
partitioner

Record Keys and why they’re important - Key
Cardinality
Consumers
Key cardinality affects the amount
of work done by the individual
consumers in a group. Poor key
choice can lead to uneven
workloads.
Keys in Kafka don’t have to be
primitives, like strings or ints. Like
values, they can be be anything:
JSON, Avro, etc… So create a key
that will evenly distribute groups of
records around the partitions.
Car·di·nal·i·ty
/ˌkärdəˈnalədē/
Noun
the number of elements in a set or other grouping, as a property of that grouping.

{
“Name”: “John Smith”,
“Address”: “123 Apple St.”,
“Zip”: “19101”
}
You don’t have to but... use a Schema!
Data
Producer
Service
Data
Consumer
Service
{
“City”: “Philadelphia”,
“State”: “PA”,
“Zip”: “19101”
}
send JSON
“Where’s record.City?”
Reference
https://www.conﬂuent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you
-really-need-one/

Schema Registry: Make Data Backwards Compatible and Future-Proof
● Define the expected fields for each Kafka topic
● Automatically handle schema changes (e.g. new
fields)
● Prevent backwards incompatible
changes
● Support multi-data center environments
Elastic
Cassandra
HDFS
Example Consumers
Serializer
App 1
Serializer
App 2
!
Kafka Topic
!
Schema
Registry
Open Source Feature

{
“Zip”: “19101”,
“City”: “NA”,
“State”: “NA”
}
Avro allows for evolution of schemas
Data
Producer
Service
Data
Consumer
Service
{
“City”: “Philadelphia”,
“State”: “PA”,
“Zip”: “19101”
}
send AvroRecord
Schema
Registry
Version 1
Version 2
Reference
https://www.conﬂuent.io/blog/schema-registry-kafka-stream-processing-yes-virginia-you
-really-need-one/

Developing with Confluent Schema Registry
We provide several Maven plugins for developing with
the Confluent Schema Registry
● download - download a subject’s schema to
your project
● register - register a new schema to the
schema registry from your development env
● test-compatibility - test changes made to
a schema against compatibility rules set by the
schema registry
Reference
https://docs.confluent.io/current/schema-registry/docs/maven-plugin.html
<plugin>
<groupId>io.confluent</groupId>
<artifactId>kafka-schema-registry-maven-plug
<version>5.0.0</version>
<configuration>
<schemaRegistryUrls>
<param>http://192.168.99.100:808
</schemaRegistryUrls>
<outputDirectory>src/main/avro</outp
<subjectPatterns>
<param>^TestSubject000-(key|valu
</subjectPatterns>
</configuration>
</plugin>

Use Kafka’s Headers
Reference
https://cwiki.apache.org/conﬂuence/display/KAFKA/KIP-82+-+Add+Record+Headers
Producer Record
Topic
[Partition]
[Timestamp]
Value
[Headers]
[Key]
Kafka Headers are simply an interface that requires a key of type
String, and a value of type byte[], the headers are stored in an
iterator in the ProducerRecord .
Example Use Cases
● Data lineage: reference previous topic partition/offsets
● Producing host/application/owner
● Message routing
● Encryption metadata (which key pair was this message
payload encrypted with?)

Producer Guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
acks=0
Reference
https://www.conﬂuent.io/blog/exactly-once-semantics-are-p
ossible-heres-how-apache-kafka-does-it/

Producer Guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
ack
Producer Properties
acks=1
Reference

Producer Guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
acks=all
min.insync.replica=2
ack

Producer Guarantees - without exactly once
guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
acks=all
{key: 1234 data: abcd} - offset 3345
Failed ack
Successful write
Reference

Producer Guarantees - without exactly once
guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
acks=all
{key: 1234, data: abcd} - offset 3345
{key: 1234, data: abcd} - offset 3346
retry
ack
dupe!
Reference

Producer Guarantees - with exactly once
guarantees
P
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
enable.idempotence=true
max.inflight.requests.per.connection=5
acks = “all”
retries > 0 (preferably MAX_INT)
(pid, seq) [payload]
(100, 1) {key: 1234, data: abcd} - offset 3345
(100, 1) {key: 1234, data: abcd} - rejected, ack re-sent
(100, 2) {key: 5678, data: efgh} - offset 3346
retry
ack
no dupe!
Reference

Transactional Producer
Producer
T1 T1 T1 T1 T1
KafkaProducer producer = createKafkaProducer(
“bootstrap.servers”, “broker:9092”,
“transactional.id”, “my-transactional-id”);
producer.initTransactions();
-- send some records --
producer.commitTransaction();
Consumer
KafkaConsumer consumer = createKafkaConsumer(
“bootstrap.servers”, “broker:9092”,
“group.id”, “my-group-id”,
"isolation.level", "read_committed");
Reference
https://www.conﬂuent.io/blog/transactions-apache-kafka/

A basic Java Consumer
final Consumer<String, String> consumer = new KafkaConsumer<String,
String>(props);
consumer.subscribe(Arrays.asList(topic));
try {
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
-- Do Some Work --
}
}
} finally {
consumer.close();
}
}

Consuming From Kafka - Single Consumer
C

Consuming From Kafka - Grouped Consumers
C
C
C1
C
C
C2

C C
C C

0 1
2 3

0, 3 1
2 3

Kafka’s Interceptors
ProducerInterceptor
onSend(ProducerRecord<K, V> record)
Returns ProducerRecord<K, V> . Called from
send() before key and value get serialized and
partition is assigned. This method is allowed to
modify the record.
onAcknowledgement(RecordMetadata metadata,
java.lang.Exception exception)
This method is called when the record sent to the
server has been acknowledged, or when sending
the record fails before it gets sent to the server.
Used for observability and reporting.
ConsumerInterceptor
onConsume(ConsumerRecords<K,V> records)
Called just before the records are returned by
KafkaConsumer.poll()
This method is allowed to modify consumer
records, in which case the new records will be
returned.
onCommit(Map<TopicPartition,OffsetAndMetada
ta> offsets)
This is called when offsets get committed.
Used for observability and reporting
Reference
https://kafka.apache.org/20/javadoc/org/apache/kafka/clien
ts/producer/ProducerInterceptor.html
Reference
https://kafka.apache.org/20/javadoc/org/apache/kafka/clien
ts/consumer/ConsumerInterceptor.html

Should I pool connections?
NO!
Since Kafka connections are long-lived, there is no reason to
pool connections. It’s common to keep one connection per
thread.

Use a good client!
Clients
● Java/Scala - default clients, comes with Kafka
● C/C++ - https://github.com/edenhill/librdkafka
● C#/.Net - https://github.com/confluentinc/confluent-kafka-dotnet
● Python - https://github.com/confluentinc/confluent-kafka-python
● Golang - https://github.com/confluentinc/confluent-kafka-go
● Node/JavaScript - https://github.com/Blizzard/node-rdkafka (not supported by Confluent!)
New Kafka features will only be available to modern, updated clients!

Resources
Free E-Books from Confluent!
I Heart Logs:
https://www.confluent.io/ebook/i-heart-logs-event-data-stream-processing-and-data-integration/
Kafka: The Definitive Guide: https://www.confluent.io/resources/kafka-the-definitive-guide/
Designing Event Driven Systems:
https://www.confluent.io/designing-event-driven-systems/
Confluent Blog: https://www.confluent.io/blog
Thank You!

Thank you!
pascal@conﬂuent.io

Kafka 101 and Developer Best Practices

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Kafka 101 and Developer Best Practices

Ähnlich wie Kafka 101 and Developer Best Practices (20)

Mehr von confluent

Mehr von confluent (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Kafka 101 and Developer Best Practices