Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019
•
1 gefällt mir•434 views
Yet another presentation about Event Sourcing? Yes and no. Event Sourcing is a really great concept. Some could say it’s a Holy Grail of the software architecture. True, but everything comes with a price. This session is a summary of my experience with ES gathered while working on 3 different commercial products. Instead of theoretical aspects, I will focus on possible challenges with ES implementation. What could explode? How and where to store events effectively? What are possible schema evolution solutions? How to achieve the highest level of scalability and live with eventual consistency?
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Ähnlich wie Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019
Ähnlich wie Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019 (20)
Mehr von Codemotion
Mehr von Codemotion (20)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Andrzej Ludwikowski - Event Sourcing - what could possibly go wrong? - Codemotion Rome 2019
- 1. Event Sourcing - what could go wrong? Andrzej Ludwikowski
- 2. Event Sourcing - what could go wrong? Andrzej Ludwikowski Rome | March 22 - 23, 2019
- 3. About me ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
- 4. SoftwareMill
- 5. SoftwareMill
- 6. SoftwareMill
- 7. SoftwareMill ● Hibernate Envers ● Alpakka Kafka ● Sttp ● Scala Clippy ● Quicklens ● MacWire
- 8. What is Event Sourcing? DB Order { items=[itemA, itemB] }
- 9. What is Event Sourcing? DB DB Order { items=[itemA, itemB] } ItemAdded(itemA) ItemAdded(itemC) ItemRemoved(itemC) ItemAdded(itemB)
- 10. History ● 9000 BC, Mesopotamian Clay Tablets, e.g. for market transactions
- 11. History ● 2005, Event Sourcing “Enterprise applications that use Event Sourcing are rarer, but I have seen a few applications (or parts of applications) that use it.”
- 12. Why Event Sourcing? ● complete log of every state change ● debugging ● performance ● scalability
- 13. ES and CQRS Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models
- 14. ES and CQRS level 1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Transaction
- 15. ES and CQRS level 1 ● Entry-level, synchronous & transactional event sourcing ● slick-eventsourcing
- 16. ES and CQRS level 1 + easy to implement + easy to reason about + 0 eventual consistency - performance - scalability
- 17. ES and CQRS level 2 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction
- 18. ES and CQRS level 2 +/- performance +/- scalability - eventual consistency - increased events DB load - lags
- 19. ES and CQRS level 3 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater Transaction event bus
- 20. ES and CQRS level 3.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 21. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 22. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction At-least-once delivery
- 23. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 24. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 25. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 26. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 27. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 28. ES and CQRS level 3.1.1 Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction ?
- 29. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction ` Sharded Cluster
- 30. ES and CQRS level 3.x + performance + scalability - eventual consistency - complex implementation
- 31. ES and CQRS alternatives ● Change Capture Data (CDC) logging instead of event bus? ● event bus instead of DB?
- 32. Not covered but worth to check ● Command Sourcing ● Event Collaboration
- 33. ES implementation? ● custom ● library ● framework
- 34. ES from domain perspective ● commands, events, state ● 2 methods on state ○ process(command: Command): List[Event] ○ apply(event: Event): State
- 35. ES from application perspective ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
- 36. import javax.persistence.*; import java.util.List; @Entity public class Issue { @EmbeddedId private IssueId id; private String name; private IssueStatus status; @OneToMany(cascade = CascadeType.MERGE) private List<IssueComment> comments; ... public void changeStatusTo(IssueStatus newStatus) { if (this.status == IssueStatus.DONE && newStatus == IssueStatus.NEW || this.status == IssueStatus.NEW && newStatus == IssueStatus.DONE) { throw new RuntimeException(String.format("Cannot change issue status from %s to %s", this.status, newStatus)); } this.status = newStatus; } ... }
- 37. import org.axonframework.commandhandling.* import org.axonframework.eventsourcing.* @Aggregate(repository = "userAggregateRepository") public class User { @AggregateIdentifier private UserId userId; private String passwordHash; @CommandHandler public boolean handle(AuthenticateUserCommand cmd) { boolean success = this.passwordHash.equals(hashOf(cmd.getPassword())); if (success) { apply(new UserAuthenticatedEvent(userId)); } return success; } @EventSourcingHandler public void on(UserCreatedEvent event) { this.userId = event.getUserId(); this.passwordHash = event.getPassword(); } private String hashOf(char[] password) { return DigestUtils.sha1(String.valueOf(password)); } }
- 38. import akka.Done import com.lightbend.lagom.scaladsl.* import play.api.libs.json.{Format, Json} import com.example.auction.utils.JsonFormats._ class UserEntity extends PersistentEntity { override def initialState = None override def behavior: Behavior = { case Some(user) => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onReadOnlyCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.invalidCommand("User already exists") } case None => Actions().onReadOnlyCommand[GetUser.type, Option[User]] { case (GetUser, ctx, state) => ctx.reply(state) }.onCommand[CreateUser, Done] { case (CreateUser(name), ctx, state) => ctx.thenPersist(UserCreated(name))(_ => ctx.reply(Done)) }.onEvent { case (UserCreated(name), state) => Some(User(name)) } } }
- 40. import java.time.Instant import info.ludwikowski.es.user.domain.UserCommand.* import info.ludwikowski.es.user.domain.UserEvent.* import scala.util.{Failure, Success, Try} final case class User (userId: UserId, name: String, email: Email) { def applyEvent(userEvent: UserEvent): Try[User] = ??? //pattern matching def process(userCommand: UserCommand): Try[List[UserEvent]] = ??? //pattern matching } object User { def from(u: UserCreated): User = User(u.userId, u.name, u.email) }
- 41. ES packaging ● snapshotting ● fail-over ● recover ● debugging ● sharding ● serialization & schema evolution ● concurrency access ● etc.
- 42. ES packaging ● domain logic ● domain validation ● 0 ES framework imports
- 43. library vs. framework ● Akka Persistence vs. Lagom
- 44. Event storage ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
- 45. Event storage for Akka Persistence ● file ● RDBMS ● Event Store ● MongoDB ● Kafka ● Cassandra
- 46. akka-persistence-jdbc trap val theTag = s"%$tag%" sql""" SELECT "#$ordering", "#$deleted", "#$persistenceId", "#$sequenceNumber", "#$message", "#$tags" FROM ( SELECT * FROM #$theTableName WHERE "#$tags" LIKE $theTag AND "#$ordering" > $theOffset AND "#$ordering" <= $maxOffset ORDER BY "#$ordering" ) WHERE rownum <= $max"""
- 47. akka-persistence-jdbc trap SELECT * FROM events_journal WHERE tags LIKE ‘%some_tag%’;
- 48. Cassandra perfect for ES? ● partitioning by design ● replication by design ● leaderless (no single point of failure) ● optimised for writes (2 nodes = 100 000 tx/s) ● near-linear horizontal scaling
- 49. ScyllaDB ? ● Cassandra without JVM ○ same protocol, SSTable compatibility ● C++ and Seastar lib ● up to 1,000,000 IOPS/node ● not fully supported by Akka Persistence
- 50. Event serialization ● plain text ○ JSON ○ XML ○ YAML ● binary ○ java serialization ○ Avro ○ Protocol Buffers (Protobuf) ○ Thrift ○ Kryo
- 51. Plain text Binary human-readable deserialization required
- 52. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) -
- 53. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress
- 54. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast
- 55. Plain text Binary human-readable deserialization required precision issues (JSON IEEE 754, DoS) - storage consumption compress slow fast poor schema evolution support full schema evolution support
- 56. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 57. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 58. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 59. Binary ● java serialization ● Avro ● Protocol Buffers (Protobuf) ● Thrift ● Kryo
- 60. Multi-language support ● Avro ○ C, C++, C#, Go, Haskell, Java, Perl, PHP, Python, Ruby, Scala ● Protocol Buffers (Protobuf) ○ even more than Avro
- 63. Full compatibility Application Events ● backward - V2 can read V1 V1 V2
- 64. ● forward - V2 can read V3 Full compatibility Application Events V1, V2 V2 Application Application V2 V3
- 65. Schema evolution - full compatibility Protocol Buffers Avro Add field + (optional) + (default value) Remove field + + (default value) Rename field + + (aliases) https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html
- 66. Protobuf schema management //user-events.proto message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; } package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
- 67. Protobuf schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent package user.application UserCreatedEvent( userId: String, operationId: String, createdAt: Long, name: String, email: String )
- 68. Protobuf schema management ● def toDomain(event: UserCreatedEvent): UserEvent.UserCreated ● def toSerializable(event: UserEvent.UserCreated): UserCreatedEvent
- 69. Protobuf schema management + clean domain - a lot of boilerplate code
- 70. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
- 71. Avro deserialization Bytes Deserialization Object Reader SchemaWriter Schema
- 72. Avro writer schema source ● add schema to the payload ● custom solution ○ schema in /resources ○ schema in external storage (must be fault-tolerant) ○ Darwin project ● Schema Registry
- 73. Avro schema management package user.domain UserCreated( userId: UserId, operationId: OperationId, createdAt: Instant, name: String, email: Email ) extends UserEvent { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] }
- 74. Protocol Buffers vs. Avro { "type" : "record", "name" : "UserCreated", "namespace" : "info.ludwikowski.es.user.domain", "fields" : [ { "name" : "userId", "type" : "string" }, { "name" : "operationId", "type" : "string" }, { "name" : "createdAt", "type" : "long" }, { "name" : "name", "type" : "string" }, { "name" : "email", "type" : "string" } ] } message UserCreatedEvent { string user_id = 1; string operation_id = 2; int64 created_at = 3; string name = 4; string email = 5; }
- 75. Avro schema management + less boilerplate code +/- clean domain - reader & writer schema distribution
- 76. Avro + less boilerplate code +/- clean domain - reader & writer schema distribution Protobuf + clean domain - a lot of boilerplate code
- 81. Immutable vs. mutable state? ● add/remove ImmutableList 17.496 ops/s ● add/remove TreeMap 2201.731 ops/s
- 83. Fixing state ● healing command
- 84. Updating all aggregates User(id)Command(user_id) Event(user_id)Event(user_id)Event(user_id)
- 85. Handling duplicates Events Read modelRead modelRead models Updater event bus At-least-once delivery
- 87. Handling duplicates Events Read modelRead modelRead models Updater event bus At-least-once delivery
- 89. Event + seq_noEvent + seq_no Handling duplicates Events Read modelRead modelRead models Updater event bus Event + seq_no read model update + seq_no
- 90. Broken read model Events ad model ead model Read models Updater event bus
- 91. Broken read model Events ad model ead model Read models Updater event bus read model update + offset (manual offset management)
- 92. Multi aggregate transactional update ● rethink aggregates boundaries ● compensating action ○ optimistic ○ pessimistic
- 93. Pessimistic compensation action User account Cinema hall
- 94. Pessimistic compensation action User account Cinema hall charged
- 95. Pessimistic compensation action User account Cinema hall charged booked
- 96. Pessimistic compensation action User account Cinema hall charged booked sold out
- 97. Pessimistic compensation action User account Cinema hall charged booked booked sold out
- 98. Pessimistic compensation action User account Cinema hall charged booked booked sold out
- 99. Pessimistic compensation action User account Cinema hall charged booked booked sold out refund
- 100. Optimistic compensation action User account Cinema hall charged booked sold out
- 101. Optimistic compensation action User account Cinema hall booked booked sold out overbooked
- 102. Optimistic compensation action User account Cinema hall charged booked booked sold out overbooked ?
- 103. Saga, post-processing Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 104. Saga, post-processing Command Service Domain Events Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 105. Saga, post-processing Command Service Domain Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Transaction
- 106. Saga, post-processing ● should be persistable ● events order should be irrelevant ● time window limitation ● compensating action must be commutative
- 107. Saga, post-processing ● Sagas with ES ● DDD, Saga & Event-sourcing ● Applying Saga Pattern
- 108. ES with RODO/GDPR ● “right to forget” with: ○ data shredding (and/or deleting) ■ events, state, views, read models ○ retention policy ■ message brokers, backups, logs ○ data before RODO migration
- 109. ES and CQRS level 3.2 Events Client Query Service Data access Commands Queries Read modelRead modelRead models Updater event bus Command Service Domain Command Service Domain Command Service Domain Transaction Sharding Clustering
- 110. Cluster = split brain 1 5 4 3 Load balancer 2
- 111. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1)
- 112. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
- 113. Cluster = split brain 1 5 4 3 Load balancer 2 User(1)
- 114. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1)
- 115. Cluster = split brain 1 5 4 3 Load balancer 2 User(1) Command(1) User(1) Command(1)
- 116. Cluster best practises ● remember about the split brain ● very good monitoring & alerting ● a lot of failover tests ● cluster also on dev/staging ● keep it as small as possible (code base, number of nodes, etc.)
- 117. Summary ● carefully choose ES lib/framework ● understand event/command/state schema evolution ● eventual consistency is your friend ● scaling is complex ● database inside-out ● log-based processing mindset
- 119. Rate me please :)
- 120. Thank you and Q&A ➔ ➔ ludwikowski.info ➔ github.com/aludwiko ➔ @aludwikowski
- 121. WE WANT YOU