"Have you ever had your stateful Kafka Streams app killed by Kubernetes with the termination reason ""OOMKilled""? Even if you did set up JVM heap limit, the pod still got killed? This is likely due to your RocksDB off-heap memory usage. This talk will explore ways of diagnosing the problem, including: - analysis of heap and off-heap memory; - diving into RocksDB metrics; - looking at k8s java pod measurements. It will also show a possible solution to the problem with RocksDB and pod memory tuning. Attendants will get hands-on experience of live debugging Kafka Streams app together with useful tips and tricks on its production usage in Kubernetes. As a Streaming Platform Owner at Raiffeisen Bank and a former Data Engineer with 5+ years of experience, I will share some insights on building scalable and reliable Kafka Streams stateful applications."

1. What to Do if Your
Kafka Streams App Gets OOMKilled?
Andrey Serebryanskiy

2. {
Andrey Serebryanskiy
Streaming Platform Owner at Raiffeisen Bank
What to Do if Your Kafka
Streams App Gets
OOMKilled?

3. The problem
3
Kafka Kafka Streams App
kubernetes
RocksDB

4. What is the problem with
this app?

5. Launch with resource limits
5
helm/templates/deployment.yaml
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -jar
- app.jar
...

6. How to check if it is OOMKilled?
6
kubectl describe pod your-pod-name -n your-namespace
Name: your-pod-name
...
Containers:
app:
...
Last State: Terminated
Reason: OOMKilled
Exit Code: 137

7. Example app

8. Example app
8
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> ...);
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}

9. Example app
9
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> ...);
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}

10. Example app
10
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> logMessageCount());
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}

11. Example app
11
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> logMessageCount());
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}

12. Example app
12
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
...
runApp(kafkaStreams);
}
Please find full application code here:
https://github.com/a-serebryanskiy/kafka-streams-oom-killed

13. So you have your app
OOMKilled

14. Launch with resource limits
14
helm/templates/deployment.yaml
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -jar
- app.jar
...

15. Let’s add heap limits
15
helm/templates/deployment.yaml
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -XshowSettings:VM
- -XX:MinRAMPercentage=50.0
- -jar
- app.jar
...

16. Let’s add heap limits
16
helm/templates/deployment.yaml
...
resources:
limits:
memory: 256Mi
requests:
memory: 128Mi
command:
- java
args:
- -XshowSettings:VM
- -XX:MinRAMPercentage=50.0
- -jar
- app.jar
...
https://www.baeldung.com/java-jvm-parameters-rampercentage
VM settings:
Max. Heap Size (Estimated): 121.81M
Property settings:
java.version = 11.0.12
Turns out 50.0 is already a default value

17. App memory performance
17
Taken from grafana
container memory limit
container memory usage
jvm memory
100 mb

18. Kafka Streams
memory usage

19. Kafka Streams app memory
19
JVM Heap + RocksDB
Confluent article about Kafka Streams memory tunning:
https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html
State Store
Kafka Streams
App
put(key, val)

20. Kafka Streams app memory
20
JVM Heap + RocksDB
Kafka Streams
App
CachingKeyValueStore
- ThreadCache context.cache()
Indexes
bloom
filters
block
cache
OS page
cache
memtable
State Store - RocksDB
! These items size
depends on number of
unique keys
JVM Heap
Not controlled by JVM
RocksDB memory details: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB
put(key, val)
native put(key, val)

21. How to fix unbounded RocksDB memory usage?
21
Application.java
properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
”your.package.BoundedMemoryRocksDBConfig");
your.package.BoundedMemoryRocksDBConfig.java
@Override
public void setConfig(..., Options options, ...) {
BlockBasedTableConfig tableConfig = options.tableFormatConfig();
Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false);
tableConfig.setBlockCache(cache);
tableConfig.setCacheIndexAndFilterBlocks(true);
options.setWriteBufferManager(writeBufferManager);
...
}

22. How to fix unbounded RocksDB memory usage?
22
Application.java
properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
”your.package.BoundedMemoryRocksDBConfig");
your.package.BoundedMemoryRocksDBConfig.java
@Override
public void setConfig(..., Options options, ...) {
BlockBasedTableConfig tableConfig = options.tableFormatConfig();
Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false);
tableConfig.setBlockCache(cache);
tableConfig.setCacheIndexAndFilterBlocks(true);
options.setWriteBufferManager(writeBufferManager);
...
}

23. How to fix unbounded RocksDB memory usage?
23
Application.java
properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG,
”your.package.BoundedMemoryRocksDBConfig");
your.package.BoundedMemoryRocksDBConfig.java
@Override
public void setConfig(..., Options options, ...) {
BlockBasedTableConfig tableConfig = options.tableFormatConfig();
Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false);
tableConfig.setBlockCache(cache);
tableConfig.setCacheIndexAndFilterBlocks(true);
}
Pay attention to the number of stores and partitions:
https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb

24. How to compute my
RocksDB memory?

25. Dynamic memory allocation
25
your.package.BoundedMemoryRocksDBConfig.java
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

26. Dynamic memory allocation
26
your.package.BoundedMemoryRocksDBConfig.java
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

27. Dynamic memory allocation
27
About using container props as env vars:
https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
helm/templates/deployment.yaml
...
env:
- name: CONTAINER_MEMORY_LIMIT
valueFrom:
resourceFieldRef:
containerName: app
resource: limits.memory
...

28. Dynamic memory allocation
28
your.package.BoundedMemoryRocksDBConfig.java
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

29. Dynamic memory allocation
29
helm/templates/deployment.yaml
env:
- name: OS_MEMORY_PERCENTAGE
value: "0.1"
- name: OFF_HEAP_SIZE_MB
value: "128"
- name: MAX_HEAP_SIZE_MB
value: "{{ .Values.heapSizeMb }}"

30. Dynamic memory allocation
30
your.package.BoundedMemoryRocksDBConfig.java
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

31. Dynamic memory allocation
31
helm/templates/deployment.yaml
env:
- name: OS_MEMORY_PERCENTAGE
value: "0.1"
# computed it based on the jcmd output
- name: OFF_HEAP_SIZE_MB
value: "128"
- name: MAX_HEAP_SIZE_MB
value: "{{ .Values.heapSizeMb }}"

32. If you would like to analyze non-heap JVM mem
32
1. Make sure you use JDK (not JRE) Docker image
2. Add to JVM args -XX:NativeMemoryTracking=summary
3. Execute command in container shell:
helm/templates/deployment.yaml
command:
- java
args:
- -XX:NativeMemoryTracking=summary
- -jar
- app.jar
bash
kubectl exec pod/your-pod-name -n your-namespace –it -- /bin/bash -c “jcmd 1 VM.native_memory”

33. Dynamic memory allocation
33
your.package.BoundedMemoryRocksDBConfig.java
private static long computeTotalRocksDbMem() {
long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT"));
double osPercentage = getEnv("OS_MEMORY_PERCENTAGE"));
long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB"));
long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB"));
long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024;
return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes;
}

34. Dynamic memory allocation
34
helm/templates/deployment.yaml
env:
- name: OFF_HEAP_SIZE_MB
value: "128"
- name: MAX_HEAP_SIZE_MB
value: "{{ .Values.heapSizeMb }}”
args:
- -Xmx{{ .Values.heapSizeMb }}m
- -jar
- app.jar
helm/values.yaml
heapSizeMb: 64

35. If you would like to profile your app
35
1) helm/templates/deployment.yaml
command:
- java
args:
- -Dcom.sun.management.jmxremote
- -Dcom.sun.management.jmxremote.port=13089
- -Dcom.sun.management.jmxremote.ssl=false
- -Dcom.sun.management.jmxremote.local.only=false
- -Dcom.sun.management.jmxremote.authenticate=false
- -Dcom.sun.management.jmxremote.rmi.port=13089
- -Djava.rmi.server.hostname=localhost
- -jar
- app.jar
ports:
- containerPort: 13089
name: jmx
protocol: TCP
2) kubectl port-forward pod/your-pod-name -n your-namespace 13089:13089

36. App memory performance (after fix)
36
Taken from grafana

37. Limitations are not the only way!
37

38. Links and materials
38
• How JVM analyze memory inside Docker container–
https://merikan.com/2019/04/jvm-in-a-container/#java-10
• How to use jcmd to analyze non-heap memory–
https://www.baeldung.com/native-memory-tracking-in-jvm
• Why use container_memory_working_set_bytes instead of container_memory_usage_bytes
https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e6
• How RocksDB store works with Kafka Streams:
https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/
• Linux memory controller for cgroups
https://lwn.net/Articles/432224/

39. /
Thank you!
39
https://t.me/aserebryanskiy
a.serebrianskiy@gmail.com