This talk is aimed to give developers who are interested to scale their streaming application with Exactly-Once (EOS) guarantees. Since the original release, EOS processing has received wide adoption as a much needed feature inside the community, and has also exposed various scalability and usability issues when applied in production systems. To address those issues, we improved on the existing EOS model by integrating static Producer transaction semantics with dynamic Consumer group semantics. We will have a deep-dive into the newly added features (KIP-447), from which the audience will have more insight into the scalability v.s. semantics guarantees tradeoffs and how Kafka Streams specifically leveraged them to help scale EOS streaming applications written in this library. We would also present how the EOS code can be simplified with plain Producer and Consumer. Come to learn more if you wish to adopt this improved EOS feature and get started on building your own EOS application today!

1. 1
Exactly-Once Made Fast
Boyang Chen: Engineer@Confluent
Guozhang Wang: Engineer@Confluent

2. 2
- Recap: exactly-once semantics (EOS) for Kafka
- What cost are we paying for EOS today
- Closing the gaps: usability and scalability
Overview

3. 3
Back in 2017..

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9
Commit

10. 10
Commit

11. 11
Commit

12. 12
Commit
Atomic

13. 13
The Kafka Approach for Exactly Once
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions
[KIP-98, KIP-129]

14. 14
The Kafka Approach for Exactly Once
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions
[KIP-98, KIP-129]

15. 15
Kafka Transactions

16. 16

17. 17
txn-status: non-exist, partitions: {}

18. 18
txn-status: non-exist, partitions: {}
producer.initTxn();

19. 19
txn-status: empty, partitions: {}
producer.initTxn();
non-exist

20. 20
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
}
} catch (KafkaException e) {
}txn-status: empty, partitions: {}
producer.initTxn();

21. 21
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: empty, partitions: {}
producer.initTxn();

22. 22
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();
non-exist {}

23. 23
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

24. 24
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

25. 25
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

26. 26
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0}
producer.initTxn();

27. 27
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();
{output-0}

28. 28
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

29. 29
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();
on-going

30. 30
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();

31. 31
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: prep-commit, partitions: {output-0, offset-0}
producer.initTxn();

32. 32
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: committed, partitions: {output-0, offset-0}
producer.initTxn();
prep-commit

33. 33
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

34. 34
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: on-going, partitions: {output-0, offset-0}
producer.initTxn();

35. 35
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();
on-going

36. 36
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();

37. 37
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: prep-abort, partitions: {output-0, offset-0}
producer.initTxn();

38. 38
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(..);
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}txn-status: aborted, partitions: {output-0, offset-0}
producer.initTxn();
prep-abort

39. 39
The Kafka Approach for Exactly Once:
1) Idempotent writes in order within a single topic partition
2) Transactional writes across multiple output topic partitions
3) Guarantee single writer for any input topic partitions

40. 40

41. 41

42. 42
At a given time, an input partition should only be processed by a single client

43. 43
At a given time, an input partition should only be processed by a single client
Consumer Group

44. 44
At a given time, an input partition should only be processed by a single client
Consumer Group

45. 45
At a given time, an input partition should only be processed by a single client
Consumer Group

46. 46
The “Single Writer” Fencing Problem

47. 47
When Taking Over the Partition:
1) The previous txn must have completed commit or abort so there are no
concurrent transactions.
2) Other clients will be fenced write processing results for those input
partitions, a.k.a we have a “single writer”.

48. 48
Transactional ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

49. 49
Transaction ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

50. 50
Transaction ID: defines single writer scope
1) Configured by the unique producer `transactional.id` property.
2) Enforced fencing by a monotonic epoch for each id.
3) Producer initialization await pending transaction completion.

51. 51
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 1

52. 52
Consumer Group
txn.Id = A, epoch = 1

53. 53
Consumer Group
txn.Id = A, epoch = 1

54. 54
Consumer Group
txn.Id = A, epoch = 1

55. 55
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, initializing...

56. 56
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 2

57. 57
Consumer Group
txn.Id = A, epoch = 1
txn.Id = B, epoch = 2
Num. producer transaction IDs ~= num. input partitions
Producers need to be dynamically created when rebalance

58. 58
EOS Scalability

59. 59
Number of Input
Partitions
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00

60. 60
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00

61. 61
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once

62. 62
KIP-447

63. 63
What problems
are KIP-447
solving ?

64. 64
What problems
are KIP-447
solving ?
● Make one producer per process model work

65. 65
What problems
are KIP-447
solving ?
● Make one producer per process model work
● Unblock technical challenges
○ Commit fencing
○ Concurrent transaction

66. 66
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

67. 67
● We are fencing on the transactional producer side,
which assumes a static partition assignment
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

68. 68
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

69. 69

70. 70

71. 71

72. 72

73. 73

74. 74

75. 75
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

76. 76
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

77. 77

78. 78

79. 79

80. 80

81. 81

82. 82

83. 83

84. 84
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
● Add new APIs
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

85. 85
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(offsets);
producer.commitTxn();
} catch (KafkaException e) {
}txn-status: on-going, partitions: {output, offset}

86. 86
try {
producer.beginTxn();
records = consumer.poll();
for (Record rec <- records) {
// process ..
producer.send(“output”, ..);
}
producer.sendOffsets(offsets,
consumer.groupMetadata());
producer.commitTxn();
} catch (KafkaException e) {
}
txn-status: on-going, partitions: {output, offset}

87. 87
● We are fencing on the transactional producer side,
which assumes a static partition assignment
● Consumer group partition assignments are dynamic
in practice
● Action: fence zombie producer commit
○ Different from epoch fencing
○ Utilize consumer group generation ~= epoch
● Add new APIs
○ Expose group generation through
consumer#groupMetadata()
○ Commit transaction with consumer metadata through
producer#sendOffsetsToTransaction(offsets,
groupMetadata)
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

88. 88
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction

89. 89
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction
● Only one open transaction allowed for each input
partition

90. 90
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section

91. 91
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Pending offsets could be completed later,
causing duplicate processing

92. 92
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Pending offsets could be completed later,
causing duplicate processing
○ Observed: consumer always needs to fetch offset after
rebalance

93. 93
What problems
are KIP-447
solving ?
● Commit fencing
● Concurrent Transaction
● Only one open transaction allowed for each input
partition
● Offset commit is the only critical section
○ Observed: Pending offsets could be completed later,
causing duplicate processing
○ Observed: consumer always needs to fetch offset after
rebalance
○ Action: OffsetFetchRequest will back-off until pending
commits are cleared, either by previous transaction
complete or timeout

94. 94

95. 95

96. 96

97. 97

98. 98

99. 99

100. 100
447 Summary

101. 101
447 Summary
● Resolve the semantic mismatch between producer
and consumer
○ Commit Fencing
○ Concurrent Transaction

102. 102
447 Summary
● Resolve the semantic mismatch between producer
and consumer
○ Commit Fencing
○ Concurrent Transaction
● Make the one producer per processing unit possible

103. 103
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once

104. 104
Number of Input
Partitions
At Least Once
Growth of Producers
Number of
Applications
5 10 15 20 25 30
600
500
400
300
200
1
00
Exactly Once
Exactly Once After 447

105. 105
447 Summary
● Resolve the semantic mismatch between producer
and consumer
○ Commit Fencing
○ Concurrent Transaction
● Make the one producer per processing unit possible
● Next we will talk about our scale testing result, and
how to turn on 447 for Kafka Streams, a major EOS
adopter

106. 106
Scale Testing

107. 107
Prove the 447 scalability improvement to
break the limit
- At_least_once
- Exactly_once
- Exactly_once_beta (post KIP-447 EOS)
Scale Testing

108. 108
Scale Testing

109. 109
Scale Testing
● Num.brokers = 3
● Num.input.partitions = 200
● Num.output.partitions = 100
● Test.interval.ms = 4 min
● Num.threads = 3
● Num.records.second = 1000
● Commit.interval.ms = 1 second
● Num.instances = 10, 20, 30...

110. 110
Scale Testing
● Num.brokers = 3
● Num.input.partitions = 200
● Num.output.partitions = 100
● Test.interval.ms = 4 min
● Num.threads = 3
● Num.records.second = 1000
● Commit.interval.ms = 1 second
● Num.instances = 10, 20, 30...

111. 111
Scale Testing
● At_least_once and
exactly_once_beta perform
steadily
● Exactly_once (pre KIP-447)
throughput degrades
significantly around 20-25
applications

112. 112
Opt-in on Kafka Streams

113. 113
Upgrade Procedure

114. 114
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5

115. 115
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5
● Upgrade the stream application binary and keep the
PROCESSING_GUARATNEE setting at "exactly_once". Do the first rolling
bounce, and make sure the group is stable with every instance on 2.6 binary.

116. 116
Upgrade Procedure
● Rolling bounce brokers to >= Apache Kafka 2.5
● Upgrade the stream application binary and keep the
PROCESSING_GUARATNEE setting at "exactly_once". Do the first rolling
bounce, and make sure the group is stable with every instance on 2.6 binary.
● Upgrade the PROCESSING_GUARANTEE setting to "exactly_once_beta" and do
a second rolling bounce to starts using new thread producer for EOS.

117. 117
Conclusion

118. 118
Kafka has an elegant transaction model which has held up well.

119. 119
Kafka has an elegant transaction model which has held up well.
● Address hardness with usability by fixing the
producer/consumer semantic mismatch

120. 120
Kafka has an elegant transaction model which has held up well.
● Address hardness with usability by fixing the
producer/consumer semantic mismatch
● Breakthrough on the scalability

121. 121
Kafka has an elegant transaction model which has held up well.
● Address hardness with usability by fixing the
producer/consumer semantic mismatch
● Breakthrough on the scalability
● How to opt-in new EOS model on Kafka Streams

122. 122
Thank you!

123. 123
- KIP-98 - Exactly Once Delivery and Transactional
Messaging - Apache Kafka
- KIP-447: Producer scalability for exactly once
semantics - Apache Kafka
Resources