"You might already use all known strategies to choose the right number of partitions for your newly created Apache Kafka topic. You apply the best recommendations to evenly distribute data across partitions in the topic. You even have metrics to observe and inform you on that. You do everything right. But then reality happens.Despite best efforts, data is published unevenly, making it slow, expensive, and difficult to consume data from a topic. The future is full of unexpected impossible to predict events, and it doesn't care about rules or normal distributions. This doesn't mean that we can simply disregard good practices. However, we need a plan for when things don't go according to anyone's calculations. Come to this talk to learn what to do when the data distribution across topic partitions is badly broken and as a result significantly hurt consuming applications performances, increasing lag and slowing data processing. We'll talk of existing strategies, including how you can replace an existing struggling topic with a new one and rebalance the data across new partitions using new rules. What dangers can happen and what to do when the state of keys is no longer guaranteed? Why is partition scaling considered to be a dangerous operation? We'll also look at this problem from the point of view of consumers, how to scale them to more partitions and what to keep in mind when using stateful systems. This talk is for those who have sufficient expertise with Apache Kafka and want to bring their knowledge to the next level. However, we'll use simple language and accessible explanations, so even if you're a Kafka beginner, join this session to understand the challenges of uneven data replication and strategies to fix it."

1. The dark and dirty side of
fixing uneven partitions
Olena Kutsenko
Sr. Developer Advocate
Aiven
Olena Babenko
Staff Software Engineer
Aiven

2. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
It all started well…

3. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Recommended strategies for partitioning
➔ Select number of partitions based on how data is consumed

4. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Recommended strategies for partitioning
➔ Select number of partitions based on how data is consumed
➔ Select number of partitioning neither too low nor to high

5. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Recommended strategies for partitioning
➔ Select number of partitions based on how data is consumed
➔ Select number of partitioning neither too low nor to high
➔ Use keys with the highest cardinality

6. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Recommended strategies for partitioning
➔ Select number of partitions based on how data is consumed
➔ Select number of partitioning neither too low nor to high
➔ Use keys with the highest cardinality
➔ Be mindful of data distribution over time

7. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Recommended strategies for partitioning
➔ Select number of partitions based on how data is consumed
➔ Select number of partitioning neither too low nor to high
➔ Use keys with the highest cardinality
➔ Be mindful of data distribution over time
➔ Consider potential edge cases

8. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You were pretty happy
about the results

9. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Nothing predicted the storm

10. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Or so you thought

11. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Partition 1 47%
Partition 2 34%
Partition 3 7%
Partition 4 4%
Partition 5 4%
Partition 6 4%
Data balancing gone wild

12. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system

13. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:

14. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers

15. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers
➔ Consumers:

16. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers
➔ Consumers:
◆ Increased consumer lag

17. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers
➔ Consumers:
◆ Increased consumer lag
◆ Consumers that are assigned to a hot partition require bigger resources

18. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers
➔ Consumers:
◆ Increased consumer lag
◆ Consumers that are assigned to a hot partition require bigger resources
◆ Underutilisation of resources when vertical scaling with k8s

19. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
How uneven partitions affect the system
➔ Brokers:
◆ Heavy load on the file system -> slower brokers
➔ Consumers:
◆ Increased consumer lag
◆ Consumers that are assigned to a hot partition require bigger resources
◆ Underutilisation of resources when vertical scaling with k8s
◆ Out-of-memory exception cycle

20. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
What to do now?

21. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
“Premature optimization
is the root of all evil”
Donald Knuth

22. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You can’t avoid the change.
Embrace the inevitable.

23. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Today you’ll learn
●

24. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Today you’ll learn
● Different recipes to deal with uneven partitioning

25. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Today you’ll learn
● Different recipes to deal with uneven partitioning
● From easiest 🌶

26. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Today you’ll learn
● Different recipes to deal with uneven partitioning
● From easiest 🌶 to more difficult 🌶🌶🌶

27. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
🌶🌶🌶 The advanced techniques will help you
● Rebalance records across partitions
● Scale your topic up or down
● Be effective at disaster recovery

28. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Partition 1 47%
Partition 2 34%
Partition 3 7%
Partition 4 4%
Partition 5 4%
Partition 6 4%

29. 15%
12%
13%
11%
13%
11%
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5
Partition 6
14%
12%
Partition 7
Partition 8

30. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 1. Easy🌶

31. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 1. Easy🌶
If you don’t use keys..

32. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions

33. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!

34. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!
- Pay attention to

35. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!
- Pay attention to
- Data retention period

36. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!
- Pay attention to
- Data retention period
- Number of consumers

37. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!
- Pay attention to
- Data retention period
- Number of consumers
- Data distribution over time

38. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
No keys - increase the number of partitions
- This way you can’t scale down, but you can scale up!
- Pay attention to
- Data retention period
- Number of consumers
- Data distribution over time
- Linger_ms and batch_size for sticky partitioning

39. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 2. Moderate🌶🌶

40. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 2. Moderate🌶🌶
One or two keys are hot

41. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You still can add new partitions

42. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You still can add new partitions…. kinda

43. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The key challenge:
github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/
producer/internals/BuiltInPartitioner.java

44. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The key challenge:
github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/
producer/internals/BuiltInPartitioner.java

45. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You will need
● Calculate which key is hot 🔥

46. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You will need
● Calculate which key is hot 🔥
● Keep the state

47. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You will need
● Calculate which key is hot 🔥
● Keep the state
● Not mess up old keys

48. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
You will need
● Calculate which key is hot 🔥
● Keep the state
● Not mess up old keys
● Use custom partitioner

49. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}

50. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}

51. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}

52. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 3. Getting hot🌶🌶🌶

53. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Level 3. Getting hot🌶🌶🌶
Time to migrate to a new topic

54. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The time will come…
when you need to re-create the topic

55. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The time will come…
when you need to re-create the topic
➔ Rebalance records across partitions

56. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The time will come…
when you need to re-create the topic
➔ Rebalance records across partitions
➔ Scale your topic up or down

57. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
The time will come…
when you need to re-create the topic
➔ Rebalance records across partitions
➔ Scale your topic up or down
➔ To do disaster recovery

58. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Let’s re-create the topic
and MIGRATE!

59. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Three main steps

60. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13

61. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13
P1
P2
P0
P3
0 12
🗝
1 13
🗝
2 14
3 15
🗝
🗝
New topic

62. …
…
Producers Consumers
P1
P0
P1
P2
P0
P3
Old topic
New topic
0 6 12
🗝
🗝
0 12
🗝
1 13
🗝
1 7 13
2 14
3 15
🗝
🗝

63. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Our goals when migrating
1. Keep downtime to bare minimum
2. No duplicates

64. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Partition 1 47%
Partition 2 34%
Partition 3 7%
Partition 4 4%
Partition 5 4%
Partition 6 4%

65. 15%
12%
13%
11%
13%
11%
Partition 1
Partition 2
Partition 3
Partition 4
Partition 5
Partition 6
14%
12%
Partition 7
Partition 8

66. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Two options

67. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Two options
(in reality way more, but similar in
essence)

68. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Option 1: Stop the world producers

69. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Option 1: Stop the world producers
Sharp cut

70. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13

71. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13

72. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13

73. …
Producers Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13
P1
P2
P0
P3
New topic
0 12
🗝
1 13
🗝
2 14
3 15
🗝
🗝

74. …
Producers
Consumers
P1
P0
Old topic
0 6 12
🗝
🗝 1 7 13
P1
P2
P0
P3
New topic
0 12
🗝
1 13
🗝
2 14
3 15
🗝
🗝

75. …
Producers
Consumers
P1
P0 0 6 12
🗝
🗝 1 7 13
P1
P2
P0
P3
New topic
0 12
🗝
1 13
🗝
2 14
3 15
🗝
🗝
Old topic

76. …
Consumers
P1
P0 0 6 12
🗝
🗝 1 7 13
P1
P2
P0
P3
New topic
0 12
🗝
1 13
🗝
2 14
3 15
🗝
🗝
Producers
Old topic

77. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Advantages
➔ No skipped messages
➔ Prevention of duplicates
➔ No need for extra compute to replicate data from old to new topic

78. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Limitations
➔ Downtime
➔ Difficult to test new setup and challenging to roll back
➔ Limited time window for migration
➔ Need for seamless collaboration among teams
➔ All-or-nothing migration

79. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Our goals when migrating
1. Keep downtime to bare minimum
2. No duplicates

80. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Option 2: Gradual switch relying on
replicated data

81. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
ABOVE - Olena K
BELOW - Olena B

82. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Time for plan B

83. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Time for plan B
with Olena B

84. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Strategy

85. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 1

86. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 2

87. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

88. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

89. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

90. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

91. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 4

92. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 4

93. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 5

94. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 1
New topic creation

95. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 1

96. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Partition 1 47%
Partition 2 34%
Partition 3 7%
Partition 4 4%
Partition 5 4%
Partition 6 4%

97. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Partition 1 44%
Partition 2 30%
Partition 3 6%
Partition 4 3%
Partition 5 3%
Partition 6 3%
Partition 7 3%
Partition 8 3%

98. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
Had to redo whole process
because of too few/many partitions

99. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 2
Fast and reliable data pump

100. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 2

101. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Data pump application requirements
- Simple
- Fast.
- Reliable
Kafka
Streams Java

102. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Require too much resources if not simple enough
- Cannot keep up if it is too complicated
- Data losses if application is not reliable
- Data loss or duplicates because records from from different
partitions get shuffled
WARNING. Records/keys almost certainly will be mixed.

103. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

104. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

105. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

106. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

107. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

108. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

109. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

110. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

111. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

112. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

113. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

114. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

115. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

116. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

117. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

118. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
New partitions have mix of data from old partitions

119. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

120. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

121. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

122. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

123. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

124. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

125. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

126. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

127. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

128. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events
If consumers had stopped when order is not correct.
- Read some records one more time
OR
- Skip some records

129. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events

130. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events
Be careful
during data pump catch up and
if you use big batches to read data

131. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Out of order events
Old topic timestamps from metadata
could be used to preserve chronological order

132. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3
Gradual consumer switch.

133. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

134. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Spikes
- Too long downtime for consumers
- Data loss or duplicates

135. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 3

136. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Consumer groups translations

137. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Simple Consumer Group Translation

138. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Simple Consumer Group Translation

139. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Simple Consumer Group Translation
Old Consumer Group:
Partition 0: offset 13
Partition 2: offset 23
Last consumed event
Partition 0: offset 12
Partition 2: offset 22

140. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Simple Consumer Group Translation
Timestamps:
Partition 0: 07:01:04
Partition 2: 07:01:03
Earliest timestamp:
07:01:03

141. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Simple Consumer Group Translation

142. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Spikes
- Too long downtime for consumers
- Data loss or duplicates

143. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group
Translations

144. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group Translation

145. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group Translation

146. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group Translation

147. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group Translation

148. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Streaming Consumer Group Translation

149. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Spikes
- Too long downtime for consumers
- Data loss or duplicates

150. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translations

151. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translation

152. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translation

153. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translation
Consumer Group 1:
Partition 0: offset 13
Partition 2: offset 23

154. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translation
For r in records:
P = r.metadata.old_partition
If offsets[P] <= r.metadata.offset:
return
Consumer Group 1:
Partition 0: offset 13
Partition 2: offset 23

155. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Mirror Maker offset translation
Data pump -> MirrorSourceTask
Old + New records metadata -> Records in Offset Sync topics
Offset translation -> MirrorCheckpointTask
Problems:
- Main usecase data transfer between 2 clusters, not a same
- Till version 3.3 offset translation by measuring the 'distance' between
the MM2 offset sync and the upstream consumer group, and then
assuming that the same distance applies in the downstream topic.

156. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Spikes
- Too long downtime for consumers
- Data loss or duplicates
- Poor offsets estimations
- Bad timing for offsets translation

157. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Bad timing for offset translations

158. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Bad timing for Offset Translation

159. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Bad timing for Offset Translation
Either
Start from 32:
duplicate B1, B2
OR
Start from 35:
A2 is lost

160. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
“Bounded” stream

161. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Gradual consumer switch
Earliest offset Duplicates guaranteed
Consumer’s earliest timestamp High probability of
duplicates
Offset translation A few duplicates
Offset translation + “late events”
tracking
Almost no duplicates
Offset translation + “late events”
tracking + “Bounded” stream
approach
Rare/no duplicates

162. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Gradual consumer switch

163. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
More topics to talk about
- Apache Mirror Maker implementation details
- Stateless vs Stateful consumers
- Idempotence
- Changing schemas
- New key selection strategy

164. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 4
Gradual producers switch

165. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Step 4

166. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Risks
- Data loss or duplicates if data pump is not fast enough

167. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
To summarize it all

168. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Key learnings
● No keys - add partitions 🌶
● A few hot keys - you still can add partitions 🌶🌶
● Workarounds are not sufficient? - Migrate the topic 🌶🌶🌶

169. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Migrate the topic 🌶🌶🌶
● Sharp cut - stop the producers first
○ Exactly once delivery
○ Expect the downtime
○ All-or-nothing migration
● Generic gradual switch
○ Minimal downtime
○ Possibility to test before switching
○ Switch consumer groups gradually
○ Minimize chance of duplicates

170. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:

171. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:

172. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Olena Kutsenko
Sr. Developer Advocate
Olena Babenko
Staff Software Engineer

173. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Olena Kutsenko
Sr. Developer Advocate
Olena Babenko
Staff Software Engineer

174. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Olena Kutsenko
Sr. Developer Advocate
Olena Babenko
Staff Software Engineer
The trusted open source
data platform for everyone

175. olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
#G8
The trusted open source
data platform for everyone