What if the pitfalls identified in Peter Deutsch’s “eight fallacies of distributed computing” were not inconveniences but opportunities? We present our view of the emerging patterns within distributed systems architecture and argue for a modern semantics of distributed systems based on sympathy with the network. In our approach, event sourcing and stream processing provide the processing model, while microservices and bounded contexts provide the domain model. We discuss the implementation of Concursus, an open source framework for bringing event sourcing patterns to distributed applications, which represents the evolution of our thinking about event sourcing, based on our practical experience of implementing distributed systems in production.

1. Concursus
Event Sourcing Evolved
GOTO London 2016

2. Introductions
Dominic Fox
Twitter: @dynamic_proxy
Email: dominic.fox@opencredo.com
Tareq Abedrabbo
Twitter: @tareq_abedrabbo
Email: tareq.abedrabbo@opencredo.com
Concursus
Page: https://opencredo.com/publications/concursus/
Github: http://github.com/opencredo/concursus

3. Agenda
• History
• Concepts
• Example
• Domain model
• Processing model
• Programming model
• Future directions

4. What is Concursus?
A toolkit for processing and organising messy data in an distributed context.

5. The Concursus Timeline
Observations
Conception and design
Prototype
Open source implementation
Technical report and blogs

6. Event Sourcing
“Event Sourcing ensures that all changes to application state are stored as a sequence of
events. Not just can we query these events, we can also use the event log to reconstruct
past states, and as a foundation to automatically adjust the state to cope with retroactive
changes.”
http://martinfowler.com/eaaDev/EventSourcing.html

7. What is Concursus?
Problems Concursus addresses:
ü Processing events in a scalable and reliable way
ü Processing guarantees and ordering: exactly once, out of order, repeated or missed
delivery, etc..
ü Building meaningful domain models to reason about and build business logic around
ü Flexibility: building additional views as needed

8. Tendencies:
• From internet of users to internet of things
• From “presence” to “presents”
• From monoliths to microservices
Why Concursus?

9. From Internet of Users to Internet of Things

10. From Presence to Presents

11. From Monoliths to Microservices

12. “Write First, Reason Later”
2016-­‐10-­‐12
09:06:31.432
Received at Depot
2016-­‐10-­‐12
09:06:32.106
Received at Depot
2016-­‐10-­‐12
09:06:34.740
Received at Depot
2016-­‐10-­‐12
11:35:02.163
Loaded onto Truck
2016-­‐10-­‐12
11:40:21.032
Loaded onto Truck
2016-­‐10-­‐12
11:38:51.204
Loaded onto Truck
2016-­‐10-­‐12
14:12:44.021
Delivery Failed
2016-­‐10-­‐12
15:00:31.322
Delivered
2016-­‐10-­‐12
15:11:05.038
Delivered

13. “Write First, Reason Later”

14. Handling Events
ü Delivery constraints
out of order, repeated, delayed or missed delivery
ü Processing guarantees
at least once or exactly once processing, idempotency
ü Ordering
partial ordering across aggregates (with reasonable assumptions)

15. Data Processing Layers
ü Durable
sufficiently durable buffer for async processing (what’s happening)
ü Persistent
a permanent record of everything that has happened (what happened)
ü Transient
fast and consistent, but also disposable state (what happens)

16. Building Blocks
• Java 8 and Kotlin: APIs
• Cassandra: Persistent state (Event store)
• Kafka: Durable state (Message broker)
• Hazelcast: Transient state (cache, idempotency filters)
• Also, RabbitMQ and Redis

17. Sources of Inspiration
Stream processing frameworks such as Apache Storm and Spark
Google papers: Cloud dataflow, MillWheel
Apache Spark papers
The Axon CQRS framework
Domain Driven Design
Functional programming

18. Summary
Concursus
=
Event sourcing
+
Stream processing
+
Bounded contexts (DDD)
+
Distributed computing

19. Received at
Depot
Loaded onto
Truck
Delivered
Delivery Failed

20. Domain Model: Events
Received
at
Depot
Loaded
onto
Truck
Delivery
Failed
Received
at
Depot
Loaded
onto
Truck
Delivered

21. aggregateType: parcel
aggregateId: 69016fb5-1d69-4a34-
910b-f8ff5c702ad9
eventTimestamp: 2016-03-31 10:31:17.981
parameters: { “depotId”: “Lewisham” }
Domain Model: Events
Received
at
Depot
Loaded
onto
Truck
Delivery
Failed
Received
at
Depot
Loaded
onto
Truck
Delivered

22. aggregateType: parcel
aggregateId: 69016fb5-1d69-4a34-
910b-f8ff5c702ad9
eventTimestamp: 2016-03-38 08:15:23.104
parameters: { “truckId”: “J98 257” }
Domain Model: Events
Received
at
Depot
Loaded
onto
Truck
Delivery
Failed
Received
at
Depot
Loaded
onto
Truck
Delivered

23. eventTimestamp: 2016-03-31T10:36:42.171Z
processingTimestamp: 2016-03-31T10:36:48.3904Z
parameters: { “deliveryAddress”: “123 Sudbury
Avenue, Droitwich DR4 8PQ”}
Domain Model: Events
aggregateType: parcel
aggregateId: 69016fb5-1d69-4a34-
910b-f8ff5c702ad9
Received at Depot
Loaded onto Truck
Delivery Failed
Received at Depot
Loaded onto Truck
Delivered

24. Domain Model: Summary
Every Event occurs to an Aggregate, identified by its type and id.
Every Event has an eventTimestamp, generated by the source of
the event.
An Event History is a log of Events, ordered by eventTimestamp,
with an additional processingTimestamp which records when the
Event was captured.

25. Network
Event sources Event processors
Events arrive:
• Partitioned
• Interleaved
• Out-of-order
Processing Model: Ordering

26. Log is:
• Partitioned by aggregate id
• Ordered by event timestamp
Processing Model: Ordering

27. CREATE TABLE IF NOT EXISTS concursus.Event (
aggregateType text,
aggregateId text,
eventTimestamp timestamp,
streamId text,
processingId timeuuid,
name text,
version text,
parameters map<text, text>,
characteristics int,
PRIMARY KEY((aggregateType, aggregateId), eventTimestamp,
streamId)
) WITH CLUSTERING ORDER BY (eventTimestamp DESC);
Cassandra Schema

28. Cassandra
Event Store
RabbitMQ Topic
Downstream
processingLog
events
Publish
events
Cassandra & AMQP

29. Cassandra
Event Store
RabbitMQ Topic
Downstream
processing
out-of-order events
ordered query results
Cassandra & AMQP

30. Cassandra
Event Store
Kafka Topic
Downstream
processing
Event store
listener
Publish
events
Log
events
Cassandra & Kafka

31. Processing Model: Summary
Events arrive partitioned, interleaved and out-of-order.
Events are sorted into event histories by aggregate type and id.
Events are sorted within event histories by event timestamp, not
processing timestamp.
Event consumers need to take into account the possibility that an
event history may be incomplete at the time it is read – consider
using a watermark to give incoming events time to “settle”.

32. Programming Model: Core Metaphor
Received
at
Depot
Loaded
onto
Truck
Delivery
Failed
Received
at
Depot
Loaded
onto
Truck
Delivered

33. Received
at
Depot
Loaded
onto
Truck
Delivery
Failed
Received
at
Depot
Loaded
onto
Truck
Delivered
Consumer<Event>
Programming Model: Core Metaphor

34. You give me a Consumer<Event>, and I send Events to it one at a time:
Emitting Events

35. I implement Consumer<Event>, and handle Events that are sent to me.
Handling Events

36. Java 8 Mapping

37. Java 8 Mapping

38. Java 8 Mapping

39. Kotlin Mapping
sealed class ParcelEvent {
class ReceivedAtDepot(val depotId: String): ParcelEvent()
class LoadedOntoTruck(val truckId: String): ParcelEvent()
class Delivered(val destinationId: String): ParcelEvent()
class DeliveryFailed(): ParcelEvent()
}

40. Kotlin Mapping
eventBus.dispatchTo(parcelId,
ReceivedAtDepot(depotId = "Lewisham Depot") at start,
LoadedOntoLorry(lorryId = "Truck CU50 ZCV") at
start.plus(2, DAYS)
)

41. Kotlin Mapping
fun describeEvent(event: ParcelEvent): Unit = when (event) {
is ReceivedAtDepot -­‐> println("Received at depot:
${event.depotId}")
is LoadedOntoTruck -­‐> println("Loaded onto truck:
${event.truckId}")
is Delivered -­‐> println("Delivered to:
${event.destinationId}")
is DeliveryFailed -­‐> println("Delivery failed")
}

42. Event-handling middleware is a chain of Consumer<Event>s that transforms, routes, persists
and dispatches events. A single event submitted to this chain may be:
■ Checked against an idempotency filter (e.g. a Hazelcast distributed cache)
■ Serialised to JSON
■ Written to a message queue topic
■ Retrieved from the topic and deserialised
■ Persisted to an event store (e.g. Cassandra)
■ Published to an event handler which maintains a query-optimised view of part of the system
■ Published to an event handler which maintains an index of aggregates by event property
values (e.g. lightbulbs by wattage)
Event-Handling Middleware

43. • Kafka Streams
• Narrative threads across event histories
• Generic Attribute indexing
• State management and caching
• Improved cloud tooling
Future Directions

44. Thank you for listening
Any questions?