Event Sourcing and CQRS are two popular patterns for implementing a Microservices architectures. With Event Sourcing we do not store the state of an object, but instead store all the events impacting its state. Then to retrieve an object state, we have to read the different events related to a certain object and apply them one by one. CQRS (Command Query Responsibility Segregation) on the other hand is a way to dissociate writes (Command) and reads (Query). Event Sourcing and CQRS are frequently grouped and used together to form something bigger. While it is possible to implement CQRS without Event Sourcing, the opposite is not necessarily correct. In order to implement Event Sourcing, an efficient Event Store is needed. But is that also true when combining Event Sourcing and CQRS? And what is an event store in the first place and what features should it implement? This presentation will first discuss what functionalities an event store should offer and then present how Apache Kafka can be used to implement an event store. But is Kafka good enough or do specific event store solutions such as AxonDB or Event Store provide a better solution?

1. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
http://guidoschmutz.wordpress.comgschmutz
Kafka as an Event Store – is it Good Enough?
Guido Schmutz
W-JAX Munich – 6.11.2019

2. gschmutz
Agenda
1. How do we build applications traditionally?
2. CQRS & Event Sourcing
3. What exactly is an Event Store?
4. Implementing Event Store
5. Summary

3. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF
HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH
Guido
Working at Trivadis for more than 22 years
Consultant, Trainer, Platform Architect for Java,
Oracle, SOA and Big Data / Fast Data
Oracle Groundbreaker Ambassador & Oracle ACE
Director
@gschmutz guidoschmutz.wordpress.com
173rd
edition

4. gschmutz
How do we build applications
traditionally?

5. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model

6. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
Object/Relational
Impedance Mismatch
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity

7. gschmutz
Data Access Layer
Monolithic System - using Layered Architecture
User Interface
Account UI
Service Layer
{ }
Account API
Database
Customer API
REST
Customer UI
Domain
Model
Account DAO
{ }REST
Customer DAO
DB Model
• Traditional approach
to persistence
• Store current state
• CRUD operations
• Coupling between
read & write
• Increased Complexity
SELECT acc.id, acc.account_number, acc.account_type.id,
acct.name, cus.first_name, cus.last_name,
addr.street, addr.city
FROM account_t acc
, account_type_t acct
, customer_t cus
, cust_adr_t cusa
, address_t addr
WHERE acc.account_type_id = acct.account_type_id
AND adc.customer_id = cus.customer_id
AND cusa.customer_id = cus.customer_id
AND cusa.address_id = addr.address_id
AND cusa.type = 'MAIN'

8. gschmutz
Microservices - using Layered Architecture
Microservices …
• are responsible for their data
• might use NoSQL instead of
RDBMS
• often still use traditional
approach to persistence
• “Data silos” do no longer support
database join
• keep synchronous
communication to a minimum
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Account API
Account
Account Logic
Product Microservice
{ }
Product API
Product
Product Logic
Finance App
Finance UI UI Logic
GUI
REST
REST
REST
sync request/response
async, event pub/sub

9. gschmutz
Events
Distribute to all handlers
strong ordering req’s
No results
Queries
Route with load balancing
Sometimes scatter-gather
Provide result
Three mechanisms through which services can
interact
Commands
Route to single handler
Use consistent hashing
Provide Result
Adapted from Axon IQ

10. gschmutz
Domain Driven Design (DDD) – Concepts
• Domain Objects – hold the state of the
application
• Entity – Domain Objects with an identity
• Value Object – an immutable type that is
distinguishable only by the state of its
properties and has no identity
• Aggregate - A cluster of domain objects
that can be treated as a single unit
• Aggregate Root – one object of aggregate
is root object. Any reference from outside
goes through aggregate root
Aggregate Root
Account Aggregate
Customer
Aggregate
Aggregate Root

11. gschmutz
Microservices with Event-driven communication
This is Event Streaming and
not really Event Sourcing
Customer Microservice
{ }
Customer API
Customer
Customer Logic
Account Microservice
{ }
Order API
Order
Order Logic
Product Microservice
{ }
Product API
Product
Product Logic
REST
REST
REST
Event Hub
(pub/sub)
Customer
Mat View
sync request/response
async, event pub/sub
Finance App
Finance UI UI Logic
GUI

12. gschmutz
CQRS & Event Sourcing

13. gschmutz
Command Query Responsibility Segregation
(CQRS)
Optimize for write and read differently
API is split between
• commands - trigger changes in state
• queries - provide read access to the state
Still using CRUD pattern, but separates
”R” from CRUD
Might involve eventual consistency
between write and read model
Data Storage
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection
Handler
UI Logic
CDC
command
query
project
read
insert
update
delete
1
2
3
4

14. gschmutz
Local CQRS vs. System Wide CQRS
Local CQRS System-Wide CQRS
Write Model
Read Model
(read-only)
Service
Command
API
Query
API
App
UI
Projection
Handler
UI Logic
CDC
command
query
project
read
insert
update
delete
Write Model
Read Model
(read-only)
Service
Command
API
App
UI
Projection
Handler
UI Logic
CDC
command
project
insert
update
delete
Service
Write Model
command
Command
API
insert
update
delete

15. gschmutz
Event Sourcing – Persist state-changing events
and not state
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000

16. gschmutz
Event Sourcing
persists the state of an aggregate as a
sequence of state-changing events
Each event describes a state change
that occurred to the aggregate in the
past
new event is appended to the list of
events
an aggregate’s current state is
reconstructed by replaying the events
=> a.k.a ”rehydration”
Rehydration also needed for queries
Event Store
ServiceApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
Service
Subscribe
publish
publish
apply (append)
REST
Data Storage
trigger replycommand
command
1 2
3
4
5
5

17. gschmutz
Event Sourcing - ”Rehydrate” State
1. Create an empty Aggregate
object
2. Read all events stored for
that Aggregate from event
store
3. Apply each event to the
Aggregate object in the
correct order
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
Account
<empty>
Account
id: 123
accountType: Savings
balance: 0
Account
id: 123
accountType: Savings
balance: 3000
transactions: [+1000, +2000]
Account
id: 123
accountType: Savings
balance: 2500
transactions:[+1000, +2000, -500]
Account
id: 123
accountType: Savings
balance: 1000
transactions: [+1000]
applyTo
applyTo
applyTo
applyTo

18. gschmutz
Event Sourcing – Write Path CreateAccount
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
AccountCreated
id: 123
accountType: Savings
10
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
Account Aggregate
<empty>

19. gschmutz
Event Sourcing – Write Path DepositMoney
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
10 20
Account Aggregate
id: 123
accountType: Savings
balance: 0

20. gschmutz
Event Sourcing – Write Path DepositMoney
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
10 20 100
Account Aggregate
id: 123
accountType: Savings
balance: 1000

21. gschmutz
Event Sourcing – Write Path WithdrawMoney
command
Create an event for every state change of Aggregate
Persist the stream to event store (preserving event order)
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
AccountCreated
id: 123
accountType: Savings
MoneyDeposited
id: 123
amount: 1000
MoneyDeposited
id: 123
amount: 2000
MoneyWithdrawn
id: 123
amount: 500
10 20 100 2000
Account Aggregate
id: 123
accountType: Savings
balance: 3000

22. gschmutz
Event Sourcing - Potential Benefits
1. Subscribe to changes from other
Aggregates
2. Examine a historical record of every
change that has ever been applied
on the model
3. Use the event store data for trend,
forcast and other business analytics
4. Consider “what if” questions by
replaying events to Aggregates which
have experimental enhancements
5. Patch errors by adding ”correction”
events (if it is legally allowed)
6. Perform “undo” and “redo”
operations by replying varying sets
of Events

23. gschmutz
Event Sourcing & CQRS
Event sourcing is commonly
combined with the CQRS pattern
Combines best of Event Sourcing and
CQRS
Project events published by Event
Store into Read Model (Materialized
Views)
Write Model and Read Model might
only support eventual consistency
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection
Handler
command
query read
1
Event Store
publish
apply (append)
trigger reply
2
3
4
5
publish
project
5
6

24. gschmutz
Snapshot Optimization in Event Sequence
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
# Timestamp Aggregate ID Event Event Payload
1 10 A32B3DE AccountCreated { id: 123, accountType: Savings}
2 20 A32B3DE MoneyDeposited { id: 123, amount: 1000}
3 100 A32B3DE MoneyDeposited { id: 123, amount: 2000}
4 2000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
5 2000 A32B3DE AccountSnapshot { id: 123, accountType: Savings, amount: 2500}
6 3000 A32B3DE MoneyWithdrawn { id: 123, amount: 500}
Snapshots for optimizing rehydration

25. gschmutz
What is an Event Store?

26. gschmutz
Event Store Capabilities
1. Append Events efficiently
2. Read aggregate’s events in order
3. Full Sequential Read (over all
aggregates)
4. Consistent writes
5. Event versioning
6. Subscribable event stream
7. Correction events (O)
8. Ingestion & event time, bi-
temporal (O)
9. Adhoc-Query on event store (O)
10.Snapshot Optimization (O)
11.High-Availability and Reliability (O)

27. gschmutz
How many Event Stores do we need ?
{ }
API
State
Logic
REST
Event Store
{ }
API
State
Logic
REST
Event Store
Microservice
{ }
API
State
Logic
REST
Event Store
Microservice
{ }
API
State
Logic
REST
Microservice
{ }
API
State
Logic
REST
Event Store
Event
Microservice
{ }
API
State
Logic
REST
OR
Microservice
Microservice

28. gschmutz
Implementing an Event Store

29. gschmutz
Event Store Implementations
• Event Store (https://eventstore.org/) – by Greg Young
• Axon Framework & Relational DB (https://axoniq.io/) - by Axon IQ
• Axon DB (https://axoniq.io/) - by Axon IQ
• Eventuate (https://eventuate.io/) – by Eventuate.io
• Serialized (https://serialized.io/) – by Serialized.io
• Build your own ….
• Apache Kafka ???

30. gschmutz
Implementing an Event Store:
using Kafka Broker

31. gschmutz
Apache Kafka – A Streaming Platform
Kafka Cluster
Consumer 1 Consume 2r
Broker 1 Broker 2 Broker 3
Zookeeper
Ensemble
ZK 1 ZK 2ZK 3
Schema
Registry
Service 1
Management
Control Center
Kafka Manager
KAdmin
Producer 1 Producer 2
kafkacat
Data Retention:
• Never
• Time (TTL) or Size-based
• Log-Compacted based
Producer3Producer3
ConsumerConsumer 3

32. gschmutz
No SPoF, highly available
Consumer polls for new messages based on
offset
Apache Kafka – A Streaming Platform
horizontally scalable, guaranteed order

33. gschmutz
Kafka as an Event Store
1. One, single-partitioned Kafka topic per Aggregate
2. One, partitioned Kafka topic per Aggregate Type
3. One single, highly partitioned Kafka topic for all Aggregate Types
Should you put several Event Types in the same Kafka topic?:
https://www.confluent.io/blog/put-several-event-types-kafka-topic/

34. gschmutz
1) One, single-partitioned Kafka topic per
Aggregate Instance
This will guarantee that the events are stored
in order
Reading state of an aggregate is as simple as
reading a topic from offset 0
Not really feasible as there will be just too
many topics needed
Kafka
Customer Aggregate
Account Aggregate

35. gschmutz
2) One, partitioned Kafka topic per Aggregate
Type
• Required number of partitions is dependent
on number of aggregate instances
• Events are produced with aggregate-id as the
key
• guarantees that events are stored in order
• For reading state of an aggregate, all data of
all aggregate instances have to be scanned =>
slow
• Possible optimization: only read the partition
where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate

36. gschmutz
3) One single, highly partitioned Kafka topic for
all Aggregate Types
• Required number of partitions is dependent
on number of aggregate types * instances
• Events are produced with aggregate-id as the
key
• guarantees that events are stored in order
• For reading state of an aggregate, all data of
all aggregate types & instances have to be
scanned => really slow
• Possible optimization: only read the partition
where aggregate instance is stored
Kafka
Customer Aggregate
Account Aggregate

37. gschmutz
Kafka as an Event Store
# Capability Kafka Broker
1 Append events efficiently yes
2 Read aggregate’s events in order not efficiently
3 Full sequential Read yes
4 Consistent Writes no
5 Event Versioning yes (if Avro is used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no, but extra time can be passed in header
9 Snapshot Optimization (O) no
10 Ad-Hoc Query on Events (O) no
11 High-Availability and Reliability (O) yes

38. gschmutz
Event Store
Kafka is not a Database … a Database is not
Kafka
We can use Kafka to run part of our own Event
Store implementation
add a database to get missing capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes to
happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Other App
Consumer

39. gschmutz
Event Store
Kafka is not a Database … a Database is not
Kafka
We can use Kafka to run our own Event Store
implementation
adding a database to get missing capabilities
But be careful with Dual Write!
• Would need distributed transactions
• Otherwise no guarantee for both writes to
happen
Application
{ }
API DatabaseBiz Logic
REST
Event Hub
Other App
Consumer

40. gschmutz
Event StoreEvent Store
Two solutions for avoiding «dual write»
Write Event first then consume it to write it to
database
Write through database (CDC, outbox design
pattern)
Application
{ }
API
Database
Biz Logic
REST
Event Hub
Other App
Biz Logic
Application
{ }
API
Database
REST
Biz Logic
CDC
Event Hub
CDC
Connector
Other App
Biz Logic
Publish

41. gschmutz
Implementing an Event Store:
using Axon Framework

42. gschmutz
Axon
• Spring Boot with Axon Framework
for Application
• MongoDB for Event Store
• Kafka Broker for Event Bus
• Kafka Streams or KSQL for Projection
Handler
• Kafka Connect / Spring Boot to
persist in read model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply

43. gschmutz
Event Sourcing with Axon
Account
Events
Account
Command
Account Aggregate
Account Command
Response
Account App
Event Store
Account
Customer
Projection
Command Handler
Event Handler
Account Query
Projection Handler
Query Handler
Account Query
Account Query
Response
Customer
Event
https://github.com/gschmutz/various-demos/tree/master/event-sourcing

44. gschmutz
Event Sourcing with Axon - Aggregate
@Aggregate
public class AccountAggregate{
@AggregateIdentifier
private String id;
private BigDecimal balance;
private String forCustomerId;
private String accountType;
@CommandHandler
...
@EventSourcingHandler
...

45. gschmutz
Event Sourcing with Axon - Command Handler
@CommandHandler
public AccountAggregate(AccountCreateCommand command)
{
Assert.hasLength(command.getForCustomerId(),
"CustomerId must have a value");
Assert.hasLength(command.getAccountType(),
"AccountType must have a value");
...
apply(new AccountCreatedEvent(command.getId(),
command.getForCustomerId(),
command.getAccountType(),
new BigDecimal("0")));
}

46. gschmutz
Event Sourcing with Axon – Command Handler
@CommandHandler
public void on(WithdrawMoneyCommand command) {
Assert.isTrue(command.getAmount() > 0,
"Amount should be a positive number");
if(command.getAmount().compareTo(this.balance) > 0 ) {
throw new InsufficientBalanceException(
"Insufficient balance. Trying to withdraw:" +
command.getAmount() +
", but current balance is: " + this.balance);
}
apply(new MoneyWithdrawnEvent(command.getId(),
command.getAmount()));
}

47. gschmutz
Event Sourcing with Axon – Event Handler
@EventSourcingHandler
public void handle(AccountCreatedEvent event) {
id = event.getId();
forCustomerId = event.getForCustomerId();
accountType = event.getAccountType();
balance = event.getBalance();
}
@EventSourcingHandler
public void handle(MoneyWithdrawnEvent event) {
balance = balance.subtract(event.getAmount());
}

48. gschmutz
Event Sourcing with Axon – Projection Handler
public class AccountQueryController {
@Autowired
private AccountRepository accRepo;
@EventHandler
public void on(AccountCreatedEvent event,@Timestamp Instant instant) {
Account account = new Account(event.getId(),event.getBalance(),
event.getAccHolder(),event.getAccHolderName(),
instant.toString());
accRepo.insert(account);
}
@EventHandler
public void on(MoneyDepositedEvent event,@Timestamp Instant instant) {
Account account = accRepo.findByAccountNo(event.getId());
account.setBalance(account.getBalance().add(event.getAmount()));
account.setLastUpdated(instant.toString());
accRepo.save(account);
}

49. gschmutz
Axon with Axon DB
• Spring Boot with Axon Framework
for Application
• Axon DB for Event Store and Event
Bus
• Spring Boot for Projection Handler
• Spring Boot to persist in read
model
• NoSQL and/or RDBMS for read
model
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply

50. gschmutz
Axon as an Event Store
# Capability Axon Framework Axon Framework & Axon DB
1 Append events efficiently yes yes
2 Read aggregate’s events in order yes yes
3 Full sequential Read yes yes
4 Consistent Writes yes yes
5 Event Versioning yes yes
6 Subscribeable Event Stream yes yes
7 Correction events (O) no no
8 Event time & ingestion time, aka. Bi-temporal (O) no no
9 Snapshot Optimization (O) yes yes
10 Ad-Hoc Query on Events (O) yes yes
11 High-Availability and Reliability (O) possible yes

51. gschmutz
Implementing an Event Store:
using Kafka and Kafka
Streams

52. gschmutz
Apache Kafka – A Streaming Platform
Source
Connector
Sink
Connector
trucking_
driver
KSQL Engine
Kafka Streams
Kafka Broker

53. gschmutz
Kafka & Kafka Streams
• Kafka Streams with State for Event
Store
• Kafka Broker for Event Bus
• Kafka Streams or KSQL for
Projection Handler
• No reply of events, current
snapshot is held in state store
AggregateApp
UI
UI Logic
Command API &
Handler
Event Handler(s)
REST
Data Storage
Query API Read Model
(read-only)
{ }
REST
Projection Handler
publish
command
query read
project
Event Store
publish
apply (append)
trigger reply

54. gschmutz
Account
Event Handler
Event Sourcing with Kafka Streams
Account
Created
Money
Deposited
Money
Withdrawn
Command
Account Command
Handler
Command
Response
Account API
Account
Snapshot
Account
Snapshot
Customer
Snapshot
Account
Customer API
Account
Customer
Projection
Account
Customer
Projector
Account
Customer Rec
persist
https://github.com/gschmutz/various-demos/tree/master/event-sourcing

55. gschmutz
Kafka & Kafka Streams as an Event Store
# Capability Kafka & Kafka Streams
1 Append events efficiently yes
2 Read aggregate’s events in order no (snapshot state only holds current snapshot)
3 Full sequential Read no
4 Consistent Writes yes (only one event per aggregate in flight)
5 Event Versioning yes (if Avro used)
6 Subscribeable Event Stream yes
7 Correction events (O) no
8 Event time & ingestion time, aka. Bi-temporal (O) no
9 Snapshot Optimization (O) yes (snapshot state only)
10 Ad-Hoc Query on events (O) limited (KSQL, Presto on Kafka, Drill on Kafka, …)
11 High-Availability and Reliability (O) yes

56. gschmutz
Summary

57. gschmutz
Summary
• Event Sourcing and CQRS might be more natural to business people than
IT => we are used to work with “CRUD based persistence”
• Event Sourcing provides history and logging for free
• Kafka Broker alone is really “just” Event Streaming, not Event Sourcing
• Axon Framework supports the implementation of Event Sourcing
applications with Pluggable Event Store and Event Bus implementations
• Axon DB implements an Event Store and an Event Bus
• Kafka and Kafka Streams with State Store supports event sourcing in a
”streaming fashion” with current snapshot