Event Driven Services come in many shapes and sizes from tiny functions that dip into an event stream, right through to heavy, stateful services. This talk makes the case for building such services, be they large or small, using a streaming platform. We will walk through a number of patterns for putting these together using a distributed log, the Kafka Streams API and it’s transactional guarantees.

1. 1
Building Event Driven Services
with Apache Kafka and Kafka
Streams
Ben Stopford
@benstopford

2. 2
Build
Features
Build for
the Future

3. 3
Evolution!

4. 4
KAFKA
Serving
Layer
(Cassandra etc.)
Kafka Streams /
KSQL
Streaming Platforms
Data is embedded in
each engine
High Throughput
Messaging
Clustered
Java App

5. 5
authorization_attempts possible_fraud
Streaming Example

6. 6
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

7. 7
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

8. 8
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

9. 9
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

10. 10
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

11. 11
CREATE STREAM possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
authorization_attempts possible_fraud

12. 12
Streaming == Manipulating Data in Flight

13. 13
Business
Applications

14. 14
EcosystemsApp
Increasingly we build ecosystems

15. 15
SOA / Microservices / EDA
Customer
Service
Shipping
Service

16. 16
The Problem is DATA

17. 17
Most services share the same core facts.
Catalog
Most services live
in here

18. 18
So how do we share data between
services?
Orders
Service
Shipping
Service
Customer
Service
Webserver

19. 19
Buying an iPad (with REST)
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver

20. 20
Buying an iPad with Events
Message Broker (Kafka)
Notification Data is
replicated
(incrementally)
Submit
Order
Order
Created
Customer
Updated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

21. 21
Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

22. 22
Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA

23. 23
Events have two hats
Notification Data
replication

24. 24
Events are the key to scalable service
ecosystems

25. 25
Streaming is the toolset for dealing with
events as they move!

26. 26
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

27. 27
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

28. 28
What is a Distributed Log?

29. 29
Shard on the way in
Producing
Services
Kafka
Consuming
Services

30. 30
Each shard is a queue
Producing
Services
Kafka
Consuming
Services

31. 31
Consumers share load
Producing
Services
Kafka
Consuming
Services

32. 32
Reduces to a globally ordered queue

33. 33
Load Balanced Services

34. 34
Fault Tolerant Services

35. 35
Build ‘Always On’ Services
Rely on Fault
Tolerant Broker

36. 36
Services can “Rewind & Replay” the log
Rewind & Replay

37. 37
Compacted Log
(retains only latest version)
Version 3
Version 2
Version 1
Version 2
Version 1
Version 5
Version 4
Version 3
Version 2
Version 1

38. 38
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

39. 39
Kafka Connect
Kafka
Connect
Kafka
Connect
Kafka

40. 40
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

41. 41
A database engine for
data-in-flight

42. 42
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW (SIZE 5 MINUTE)
GROUP BY card_number
HAVING count(*) > 3;
Continuously Running Queries

43. 43
Features: similar to database
query engine
JoinFilter
Aggr-
egate
View
Window

44. 44
KAFKA
Buffer 5 mins
Windows

45. 45
Lookup tables
KAFKA

46. 46
Stateful Stream Processing
stream
Compacted
stream
Join
Stream
Table
KafkaKafka Streams

47. 47
Scales Out

48. 48
Embeddable
Orders Service
Kafka
Orders Service
Orders Service

49. 49
Streaming is about
1. Processing data
incrementally
2. Moving data to where
it needs to be
processed (quickly
and efficiently)
On Notification
Data Replication

50. 50
Kafka: a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

51. 51
Steps to Streaming Services

52. 52
1. Take Responsibility for the past and
evolve

53. 53
Stay Simple. Take Responsibility for the past
Browser
Webserver

54. 54
Evolve Forwards
Browser
Webserver
Orders
Service

55. 55
2. Raise events. Don’t talk to services.

56. 56
Events have two hats
Notification Data
replication

57. 57
Raise events. Don’t talk to services
Browser
Webserver
Orders
Service

58. 58
KAFKA
Order
Requested
Order
Received
Browser
Webserver
Orders
Service
Raise events. Don’t talk to services

59. 59
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Raise events. Don’t talk to services

60. 60
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Use Kafka as a Backbone for Events

61. 61
3. Use Connect (& CDC) to evolve away
from legacy

62. 62
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Evolve away from Legacy

63. 63KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Use the Database as a ‘Seam’
Connect
Products

64. 64
4. Make use of Schemas

65. 65KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Use Schemas to validate Backwards Compatibility
Connect
Products
Schema Registry

66. 66
5. Use the Single Writer Principal

67. 67KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Apply the single writer principal
Connect
Products
Schema Registry
Order
Completed

68. 68
Orders
Service
Email
Service
T1 T2
T3
T4
REST
Service
T5
Single Writer Principal

69. 69
Single Writer Principal
- Creates local consistency points in
the absence of Global Consistency
- Makes schema upgrades easier to
manage.

70. 70
6. Store Datasets in the Log

71. 71
Messaging that Remembers
Orders Customers
Payments
Stock

72. 72
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
New Service, No Problem!
Connect
Products
Schema Registry
Order
Completed Repricing

73. 73
Orders Customers
Payments
Stock
Single, Shared Source of Truth

74. 74
But how do you query a log?

75. 75
7. Move Data to Code

76. 76

77. 77
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service Stock
Stock
Materialize Stock ‘View’ Inside Service
KAFKA

78. 78
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service Stock
Stock
Take only the data we need
KAFKA

79. 79
Data Movement
Be realistic:
• Network is no longer the bottleneck
• Indexing is:
• In memory indexes help
• Keep datasets focused

80. 80
8. Use the log instead as a ‘database’

81. 81
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Apply Event Sourcing
KAFKA
Table

82. 82
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Order Service Loads Reserved Stocks on Startup
KAFKA

83. 83
Kafka has several features for reducing
the need to move data on startup
- Standby Replicas
- Disk Checkpoints
- Compacted topics

84. 84
9. Use Transactions to tie All
Interactions Together

85. 85
OrderRequested
(IPad)
2a. Order Validated
2c. Offset Commit
2b. IPad Reserved
Internal State:
Stock = 17
Reservations = 2
Tie Events & State with Transactions

86. 86
Connect
TRANSACTION
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Transactions
KAFKA

87. 87
10. Bridge the Sync/Async Divide with a
Streaming Ecosystem

88. 88
POST
GET
Load
Balancer
ORDERSORDERS
OVTOPIC
Order
Validations
KAFKA
INVENTORY
Orders
Inventory
Fraud
Service
Order
Details
Service
Inventory
Service
(see previous figure)
Order
Created
Order
Validated
Orders View
Q in CQRS
Orders
Service
C is CQRS
Services in the Micro: Orders Service
Find the code online!

89. 89
Orders Customers
Payments
Stock
Query Engine (Kstreams/KSQL)
Larger Ecosystems
HISTORICAL
EVENT STREAMS

90. 90
Kafka
KAFKA
New York
Tokyo
London
Global / Disconnected Ecosystems

91. 91
So…

92. 92
Good architectures have little to do
with this:

93. 93
It’s about how systems evolves over time

94. 94
Request driven isn’t enough
• High coupling
• Hard to handle
async flows
• Hard to move and
join datasets.

95. 95
Leverage the Duality of Events
Notification Data
replication

96. 96
With a toolset built for data in flight

97. 97
• Broadcast events
• Retain them in the log
• Evolve the event-stream with
streaming functions
• Recasting the event stream into
views when you need to query.
Event Driven Services

98. 98
Confluent
Streaming
Platform

99. 99
Thank You
@benstopford
Blog Series: https://www.confluent.io/blog/tag/microservices/