(Gwen Shapira + Matthias J. Sax, Confluent) Kafka Summit SF 2018 Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. Kafka Streams is not only scalable, but fully elastic allowing for dynamic scale-in and scale-out as the library handles state migration transparently in the background. By running Kafka Streams applications on Kubernetes, you will be able to use Kubernetes powerful control plane to standardize and simplify the application management—from deployment to dynamic scaling. In this technical deep dive, we’ll explain the internals of dynamic scaling and state migration in Kafka Streams. We’ll then show, with a live demo, how a Kafka Streams application can run in a Docker container on Kubernetes and the dynamic scaling of an application running in Kubernetes.

1. 1
1
Deploying Kafka Streams
Applications with Docker
and Kubernetes
Gwen Shapira | Principal Data Architect
Matthias J. Sax | Software Engineer

2. Agenda
• Kafka Streams 101
• How do Kafka Streams applications scale?
• Kubernetes 101
• Recommendations for Kafka Streams
• Demo:
https://github.com/gwenshap/kafka-streams-stockstats

3. 3
3
Your App
Kafka Streams
KafkaConnect
KafkaConnect
OtherSystems
OtherSystems
Kafka Streams – 101

4. 4
4
Stock Trade Stats Example
StreamsBuilder builder = new StreamsBuilder();
KStream<String, Trade> source = builder.stream("stocks-topic");
KTable<Windowed<String>, TradeStats> stats = source
.groupByKey().windowedBy(TimeWindows.of(5_000L))
.advanceBy(1_000L))
.aggregate(() -> new TradeStats(), // init
(k, v, tradestats) -> tradestats.add(v)); // agg
stats.toStream() // get changelog stream from table
.mapValues((trade) -> trade.computeAvgPrice());
.to("stockstats-output");
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();

5. 5
5
Topologies
Source Node
Processor Node
Processor Node
Sink Node
streams
state stores
Processor
Topology
builder.stream
source.groupByKey()
.windowedBy(…)
.aggregate()
mapValues()
to(…)

6. 6
6
How Do
Kafka Streams Application
Scale?

7. 7
7
Partitions, Tasks, and Consumer Groups
input topic
result topic
4 input topic
partitions
=> 4 tasks
Task
executes
processor
topology
One consumer
group:
can be
executed with
1 to 4 thread on
1 to 4 machines

8. 9
9
Scaling with State
Instance 1
Trade Stats App
“no state”

9. 10
10
Instance 1
Trade Stats App
Instance 2
Trade Stats App
“no state”
Scaling with State

10. 11
11
Instance 1
Trade Stats App
Instance 2 Instance 3
Trade Stats App Trade Stats App
“no state”
Scaling with State

11. 12
12
Scaling and Fault-Tolerance
Two Sides of the Same Coin

12. 13
13
Fault-Tolerance
Instance 1
Trade Stats App
Instance 2 Instance 3
Trade Stats App Trade Stats App

13. 14
14
Fault-Tolerant State
Input Topic
Result Topic
Changelog Topic
State Updates

14. 15
15
Trade Stats App
Migrate State
Instance 2
Trade Stats App
Changelog Topic
Instance 1
Trade Stats App
restore

15. • Changelog topics are log compacted
• Size of changelog topic linear in size of state
Large state implies high recovery times
Recovery Time

16. 18
18
Recovery Overhead
Changelog topic
Segments
(default size 1GB)
Min Topic Size: 21 GB (per shard)
Recovery overhead about 5%
After compaction
Segments
(default size 1GB)
State size: 20 GB (per shard)
Topic size can grow larger
if not compacted
Active Segment
Active Segment

17. 19
19
Recovery Overhead
Changelog topic
Segments
(default size 1GB)
Active Segment
Compaction
Segment
(only 100 MB)
State size: 100 MB (per shard)
Min Topic Size: 1.1 GB (per shard)
Recovery overhead about 1000%
But recovery is about x20 faster
Active Segment

18. • Recovery overhead is proportional to
segment-size / state-size
• Segment-size is smaller than state-size => reduced
overhead
• Update changelog topic segment size accordingly
• topic config: log.segments.bytes
• log cleaner interval important, too
Recovery Overhead

19. 21
21
Kubernetes 101

20. 22
22
Container:
Stock Trade Stats App

21. 23
23
Pod:
Container:
Stock Trade Stats App
Container:
Prometheus JMX Exporter Sidecar
Storage
IP

22. 24
24
Pod:
Container
Container
ReplicaSet = pod selection + # of replicas
Deployment = ReplicaSet + update policy
Pod:
Container
Container
Pod:
Container
Container
Kubernetes Pod
replica,
Not Kafka partition
replica…

23. 25
25
Node 2
Node 1
Pod:
Streams
Pod:
Streams
Affinity = place pods on same host
Node 1
Pod:
Streams
Pod:
StreamsAnti-Affinity = place pods on different hosts

24. 26
26
Frontend
Pod
Frontend
Pod
Frontend
Pod
Service
Backend
Pod
Backend
Pod
Backend
Pod
Static IP /
Name
Ephemeral IP

25. 27
27
Pod 1:
Container
Container
StatefulSet = deployment + identity per pod
Pod 2:
Container
Container
Pod 3:
Container
Container
Pod1
Storage
Pod2
Storage
Pod3
Storage
Headless Service:
servicename.namespace.svc.cluster.local
Pod-1 Pod-2 Pod-3

26. 28
28
Recommendations for Kafka Streams

27. 29
29
Stock Stats App
Kafka Streams
Instance 1
Stock Stats App
Kafka Streams
Instance 2
Stock Stats App
Kafka Streams
Instance 3

28. 30
30
This sure looks
stateful.
I see state right
there!

29. 31
31
WordCount App
Kafka Streams
Instance 1
WordCount App
Kafka Streams
Instance 2
WordCount App
Kafka Streams
Instance 3

30. 32
32
StatefulSets are
new and
complicated.
We don’t need
them.

31. 33
33
Recovering state
takes time.
Statelful is faster.

32. 34
34
But I’ll want to scale-
out and back
anyway.
Besides, I don’t really
trust my storage
admin.

33. 35
Recommendations:
● Keep change-log shards small
● If you trust your storage:
Use StatefulSets
● Use anti-affinity when possible
● Use “parallel” pod management

34. 37
37
Goats are cool.
But does it work?

35. 38
38
kind: "Deployment"
metadata:
name: "streams-stock-stats"
labels:
app: "streams-stock-stats"
spec:
replicas: 1
selector:
matchLabels:
app: "streams-stock-stats"
template:
metadata:
labels:
app: "streams-stock-stats"
spec:
affinity:
podAntiAffinity:
…
containers:
- name: "kafka-streams-stockstat"
image: "gcr.io/gwen-test-202722/kafka-streams-stockstat:latest"

36. 39
39
kind: Service
spec:
clusterIP: None
---
kind: StatefulSet
spec:
replicas: 2
podManagementPolicy: "Parallel"
spec:
containers:
- name: kafka-streams-stockstat
image: gcr.io/…kafka-streams-stockstat:latest
volumeMounts:
- name: rocksdb
mountPath: /var/lib/kafka-streams
volumeClaimTemplates:
- metadata:
name: rocksdb
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi

37. 40
40
Automate Deployment and Management of Apache Kafka®
Confluent Operator enables you to:
Automate provisioning of
Kafka pods in minutes
Monitor SLAs through
Confluent Control Center or
Prometheus
Scale your Kafkas clusters
elastically
Operate at scale with
enterprise support from
Confluent
Want to learn more about running Kafka on Kubernetes?
confluent.io/kubernetes

38. 41
41
Demo Placeholder

39. Summary
• Kafka Streams has recoverable state, that gives
streams apps easy elasticity and high availability
• Kubernetes makes it easy to scale applications
• It also has StatefulSets for applications with state.
• Now you know how to deploy Kafka Streams on
Kubernetes and take advantage on all the scalability and
high-availability capabilities

40. 43
43