Yet another presentation about Event Sourcing? Yes and no. Event Sourcing is a really great concept. Some could say it’s a Holy Grail of the software architecture. I might agree with that, while remembering that everything comes with a price. This session is a summary of my experience with ES gathered while working on 3 different commercial products. Instead of theoretical aspects, I will focus on possible challenges with ES implementation. What could explode (very often with delayed ignition)? How and where to store events effectively? What are possible schema evolution solutions? How to achieve the highest level of scalability and live with eventual consistency? And many other interesting topics that you might face when experimenting with ES.

1. Event Sourcing
what could possibly go wrong?
Andrzej Ludwikowski

2. About me
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski
➔ academy.softwaremill.com

3. SoftwareMill

4. SoftwareMill

5. SoftwareMill

6. SoftwareMill
JVM
BLOGGERS

7. SoftwareMill
● Hibernate Envers
● Alpakka Kafka
● Sttp
● Scala Clippy
● Quicklens
● MacWire

8. What is Event Sourcing?
DB
Order {
items=[itemA, itemB]
}

9. What is Event Sourcing?
DB
DB
Order {
items=[itemA, itemB]
}
ItemAdded(itemA)
ItemAdded(itemC)
ItemRemoved(itemC)
ItemAdded(itemB)

10. What is Event Sourcing?
DB
DB
Order {
items=[itemA, itemB]
}
ItemAdded(itemA)
ItemAdded(itemC)
ItemRemoved(itemC)
ItemAdded(itemB)

11. History
● 9000 BC, Mesopotamian Clay Tablets,
e.g. for market transactions

12. History
● 2005, Event Sourcing
“Enterprise applications that use Event Sourcing
are rarer, but I have seen a few applications (or
parts of applications) that use it.”

13. Why Event Sourcing?
● complete log of every state change
● debugging
● performance
● scalability
● microservices integration pattern

14. ES and CQRS
Client
Query Service
Data access
Queries
Read
model
Read
model
Read
models
Command Service
Domain
Events
Commands

15. ES and CQRS level 1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Transaction
Command Service
Domain
Events
Commands

16. ES and CQRS level 1
● Entry-level, synchronous & transactional event sourcing
● Implementing event sourcing using a relational database
● slick-eventsourcing

17. ES and CQRS level 1
+ easy to implement
+ easy to reason about
+ 0 eventual consistency
- performance
- scalability

18. ES and CQRS level 2
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
Transaction
Command Service
Domain
Events
Commands

19. ES and CQRS level 2
+/- performance
+/- scalability
- eventual consistency
- increased events DB load ?
- lags

20. ES and CQRS level 3
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
Transaction
event
bus
Command Service
Domain
Events
Commands

21. ES and CQRS level 3.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction
Command Service
Domain
Events
Commands

22. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction
Command Service
Domain
Events
Commands

23. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction
At-least-once delivery
Command Service
Domain
Events
Commands

24. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

25. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

26. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

27. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

28. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

29. ES and CQRS level 3.1.1
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction
?

30. ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
model
Read
model
Read
models
Projector
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
`
Sharded
Cluster

31. ES and CQRS level 3.x
+ performance
+ scalability
- eventual consistency
- complex implementation
- locking vs Single Writer

32. ES and CQRS alternatives
● Change Capture Data (CDC) logging instead of event bus?
● event bus instead of DB?

33. Not covered but worth to check
● Command Sourcing
● Event Collaboration

34. ES implementation?
● custom
● library
● framework

35. ES from domain perspective
● commands, events, state
● 2 methods on state
○ process(command: Command): List[Event]
○ apply(event: Event): State

36. ES from application perspective
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

37. import javax.persistence.*;
import java.util.List;
@Entity
public class Issue {
@EmbeddedId
private IssueId id;
private String name;
private IssueStatus status;
@OneToMany(cascade = CascadeType.MERGE)
private List<IssueComment> comments;
...
public void changeStatusTo(IssueStatus newStatus) {
if (this.status == IssueStatus.DONE
&& newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW
&& newStatus == IssueStatus.DONE) {
throw new RuntimeException(String.format("Cannot change issue status from %s to %s",
this.status, newStatus));
}
this.status = newStatus;
}
...
}

38. import javax.persistence.*;
import java.util.List;
@Entity
public class Issue {
@EmbeddedId
private IssueId id;
private String name;
private IssueStatus status;
@OneToMany(cascade = CascadeType.MERGE)
private List<IssueComment> comments;
...
public void changeStatusTo(IssueStatus newStatus) {
if (this.status == IssueStatus.DONE
&& newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW
&& newStatus == IssueStatus.DONE) {
throw new RuntimeException(String.format("Cannot change issue status from %s to %s",
this.status, newStatus));
}
this.status = newStatus;
}
...
}

39. import org.axonframework.commandhandling.*
import org.axonframework.eventsourcing.*
@Aggregate(repository = "userAggregateRepository")
public class User {
@AggregateIdentifier
private UserId userId;
private String passwordHash;
@CommandHandler
public boolean handle(AuthenticateUserCommand cmd) {
boolean success = this.passwordHash.equals(hashOf(cmd.getPassword()));
if (success) {
apply(new UserAuthenticatedEvent(userId));
}
return success;
}
@EventSourcingHandler
public void on(UserCreatedEvent event) {
this.userId = event.getUserId();
this.passwordHash = event.getPassword();
}
private String hashOf(char[] password) {
return DigestUtils.sha1(String.valueOf(password));
}
}

40. import akka.Done
import com.lightbend.lagom.scaladsl.*
import play.api.libs.json.{Format, Json}
import com.example.auction.utils.JsonFormats._
class UserEntity extends PersistentEntity {
override def initialState = None
override def behavior: Behavior = {
case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onReadOnlyCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists")
}
case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] {
case (GetUser, ctx, state) => ctx.reply(state)
}.onCommand[CreateUser, Done] {
case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done))
}.onEvent {
case (UserCreated(name), state) => Some(User(name))
}
}
}

41. ES packaging
● github.com/aludwiko/event-sourcing-akka-persistence

42. import java.time.Instant
import info.ludwikowski.es.user.domain.UserCommand.*
import info.ludwikowski.es.user.domain.UserEvent.*
import scala.util.{Failure, Success, Try}
case class User (userId: UserId, name: String, email: Email, funds: Funds) {
def process(command: UserCommand): Either[List[UserEvent]] = command match {
case c: UpdateEmail => updateEmail(c)
case c: DepositFunds => deposit(c)
case c: WithdrawFunds => withdraw(c)
...
}
def apply(event: UserEvent): User = ??? //pattern matching
}

43. ES packaging
● snapshotting
● fail-over
● recover
● debugging
● sharding
● serialization & schema evolution
● concurrency access
● etc.

44. ES packaging
● domain logic
● domain validation
● 0 ES framework imports

45. library vs. framework
● Akka Persistence vs. Lagom
● Akka Persistence Typed

46. Event storage
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

47. Event storage for Akka Persistence
● file
● RDBMS
● Event Store
● MongoDB
● Kafka
● Cassandra

48. akka-persistence-jdbc trap
val theTag = s"%$tag%"
sql"""
SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber",
"#$message", "#$tags"
FROM (
SELECT * FROM #$theTableName
WHERE "#$tags" LIKE $theTag
AND "#$ordering" > $theOffset
AND "#$ordering" <= $maxOffset
ORDER BY "#$ordering"
)
WHERE rownum <= $max"""

49. akka-persistence-jdbc trap
SELECT * FROM events_journal
WHERE tags LIKE ‘%some_tag%’;

50. Cassandra perfect for ES?
● partitioning by design
● replication by design
● leaderless (no single point of failure)
● optimised for writes (2 nodes = 100 000 tx/s)
● near-linear horizontal scaling

51. ScyllaDB ?
● Cassandra without JVM
○ same protocol, SSTable compatibility
● C++ and Seastar lib
● up to 1,000,000 IOPS/node
● not fully supported by Akka Persistence

52. Event serialization
● plain text
○ JSON
○ XML
○ YAML
● binary
○ java serialization
○ Avro
○ Protocol Buffers (Protobuf)
○ Thrift
○ Kryo

53. Plain text Binary
human-readable deserialization required

54. Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -

55. Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress

56. Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress
slow fast

57. Plain text Binary
human-readable deserialization required
precision issues (JSON IEEE 754, DoS) -
storage consumption compress
slow fast
poor schema evolution support full schema evolution support

58. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

59. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

60. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

61. Binary
● java serialization
● Avro
● Protocol Buffers (Protobuf)
● Thrift
● Kryo

62. Multi-language support
● Avro
○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala
● Protocol Buffers (Protobuf)
○ even more than Avro

63. Speed
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

64. Size
https://code.google.com/archive/p/thrift-protobuf-compare/wikis/Benchmarking.wiki

65. ● backward - V2 can read V1
Full compatibility
Application
Events
V1
V2

66. ● forward - V2 can read V3
Full compatibility
Application
Events
V1, V2
V2
Application
Application
V2
V3

67. ● forward - V2 can read V3
Full compatibility
Events
Read
model
Read
model
Read
models
Projector
V2
V3

68. Schema evolution - full compatibility
Protocol Buffers Avro
Add field + (optional) + (default value)
Remove field + + (default value)
Rename field + + (aliases)
https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

69. Protobuf schema management
//user-events.proto
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

70. Protobuf schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
package user.application
UserCreatedEvent(
userId: String,
operationId: String,
createdAt: Long,
name: String,
email: String
)

71. Protobuf schema management
● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated
● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent

72. Protobuf schema management
+ clean domain
- a lot of boilerplate code

73. Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}

74. Avro deserialization
Bytes Deserialization Object
Reader Schema
Writer Schema

75. Avro writer schema source
● add schema to the payload
● custom solution
○ schema in /resources
○ schema in external storage (must be fault-tolerant)
○ Darwin project
● Schema Registry

76. Avro schema management
package user.domain
UserCreated(
userId: UserId,
operationId: OperationId,
createdAt: Instant,
name: String,
email: Email
) extends UserEvent
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}

77. Protocol Buffers vs. Avro
{
"type" : "record",
"name" : "UserCreated",
"namespace" :
"info.ludwikowski.es.user.domain",
"fields" : [ {
"name" : "userId",
"type" : "string" }, {
"name" : "operationId",
"type" : "string" }, {
"name" : "createdAt",
"type" : "long" }, {
"name" : "name",
"type" : "string" }, {
"name" : "email",
"type" : "string"
} ]
}
message UserCreatedEvent {
string user_id = 1;
string operation_id = 2;
int64 created_at = 3;
string name = 4;
string email = 5;
}

78. Avro schema management
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution

79. Avro
+ less boilerplate code
+/- clean domain
- reader & writer schema distribution
Protobuf
+ clean domain
- a lot of boilerplate code

80. Avro vs. Protocol Buffers
● The best serialization strategy for Event Sourcing

81. Event payload
● delta event
● rich event (event enrichment)
● + metadata
○ seq_num
○ created_at
○ event_id
○ command_id
○ correlation_id

84. State replay time
● snapshotting
● write-through cache

85. Memory consumption

87. Immutable vs. mutable state?
● add/remove ImmutableList 17.496 ops/s
● add/remove TreeMap 2201.731 ops/s

88. Fixing state
● healing command

89. Updating all aggregates
User(id)
Command(user_id) Event(user_id)
Event(user_id)
Event(user_id)

90. Handling duplicates
Events
Read
model
Read
model
Read
models
Projector
event
bus
At-least-once delivery

91. https://www.seriouspuzzles.com/unicorns-in-fairy-land-500pc-jigsaw-puzzle-by-eurographics/

92. Handling duplicates
Events
Read
model
Read
model
Read
models
Projector
event
bus
At-least-once delivery
Events

93. Handling duplicates
Events
Read
model
Read
model
Read
models
Projector
event
bus
idempotent updates
Events

94. Event + seq_no
Event + seq_no
Handling duplicates
Events
Read
model
Read
model
Read
models
Projector
event
bus
Event + seq_no
read model update +
seq_no
Events

95. Broken read model
Events
ad model
ead model
Read
models
Projector
event
bus
Events

96. Broken read model
Events
ad model
ead model
Read
models
Projector
event
bus
read model update + offset
(manual offset management)
Events

97. Multi aggregate transactional update
● rethink aggregates boundaries
● compensating action
○ optimistic
○ pessimistic

98. Pessimistic compensation action
User account
Cinema show

99. Pessimistic compensation action
User account
Cinema show
charged

100. Pessimistic compensation action
User account
Cinema show
charged
booked

101. Pessimistic compensation action
User account
Cinema show
charged
booked sold out

102. Pessimistic compensation action
User account
Cinema show
charged
booked
booked sold out

103. Pessimistic compensation action
User account
Cinema show
charged
booked
booked sold out

104. Pessimistic compensation action
User account
Cinema show
charged
booked
booked sold out
refund

105. Optimistic compensation action
User account
Cinema show
charged
booked sold out

106. Optimistic compensation action
User account
Cinema show booked
booked sold out
overbooked

107. Optimistic compensation action
User account
Cinema show
charged
booked
booked sold out
overbooked
?

108. Saga
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Updater
event
bus
Transaction

109. Saga - choreography
Command Service
Domain
Events
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction

110. Saga - orchestration
Command Service
Domain
Events
Client
Query Service
Data access
Commands Queries
Read
model
Read
model
Read
models
Projector
event
bus
Transaction
Command Service
Domain
Events
Commands

111. Saga
● should be persistable
● events order should be irrelevant
● time window limitation
● compensating action must be commutative

112. Saga
● Sagas with ES
● DDD, Saga & Event-sourcing
● Applying Saga Pattern
● Microservice Patterns

113. ES with RODO/GDPR
● “right to forget” with:
○ data shredding (and/or deleting)
■ events, state, views, read models
○ retention policy
■ message brokers, backups, logs
○ data before RODO migration

114. ES and CQRS level 3.2
Events
Client
Query Service
Data access
Commands
Queries
Read
model
Read
model
Read
models
Projector
event
bus
Command Service
Domain
Command Service
Domain
Command Service
Domain
Transaction
Sharding
Clustering

115. Cluster = split brain
1
5 4
3
Load balancer
2

116. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)

117. Cluster = split brain
1
5 4
3
Load balancer
2

118. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)

119. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)

120. Cluster = split brain
1
5 4
3
Load balancer
2
User(1)
Command(1)
User(1)
Command(1)

121. Cluster best practises
● remember about the split brain
● very good monitoring & alerting
● a lot of failover tests
● cluster also on dev/staging
● keep it as small as possible (code base, number of nodes, etc.)

122. Summary
● carefully choose ES lib/framework
● there is no perfect database for event sourcing
● understand event/command/state schema evolution
● eventual consistency is your friend
● scaling is complex
● database inside-out
● log-based processing mindset

123. Want to learn more?
● Reactive Event Sourcing with Akka (Java/Scala)
● Akka Typed Fundamentals (Java/Scala)
● Akka Cluster
● Performance tests with Gatling

125. Rate me please :)

126. Thank you and Q&A
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski

127. WE
WANT
YOU

128. Thank you and Q&A
➔
➔ ludwikowski.info
➔ github.com/aludwiko
➔ @aludwikowski