4. @jeppec
Accidental complexity from distributed service integration
Warehouse
Service
Order
Service
Billing
Service
UI
Send-
Invoice
Save-Order
Reserve-
Items
Local transaction between 2
local “Services”
DUAL WRITE
using Remote call
7. @jeppec
The 11 Fallacies of Distributed Computing
These fallacies are assumptions architects, designers and developers
of distributed systems are likely to make. The fallacies will be proven
wrong in the long run - resulting in all sorts of troubles and pains for
the solution and architects who made the assumptions.
1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.
9. The System is atomic/monolithic
10. The System is Finished
11. Business logic can and should be centralized
See https://arnon.me/wp-content/uploads/Files/fallacies.pdf
8. @jeppec
Order:Save-Order()è
call Warehouse:Reserve-Items()
call Billing:Send-Invoice()
if (Billing:Call-Failed:Too-Busy?)
Wait-A-While()
call Billing:Send-Invoice()
if (Billing:Call-Failed:Too-Busy?)
Wait-A-Little-While-Longer()
call Billing:Send-Invoice()
if (Billing:Call-Failed:IO-Error?)
Save-We-Need-Check-If-Call-Billing-Succeded-After-All
AND We-Need-To-Retry call Order:Save-Order and call Warehouse:Reserve-Items
AND Tell-Customer-That-This-Operation-Perhaps-Went-Well
if (Billing:Call-Went-Well?)
commit()
Accidental complexity from distributed service integration
Warehouse
Service
Order
Service
Billing
Service
UI
Send-
Invoice
Save-Order
Reserve-
Items
Local transaction between 2
Components
11. @jeppec
Synchronous Request/Response
lowers our tolerance for faults
• When you get an IO error
• When servers crash or restarts
• When databases are down
• When deadlocks occurs in our databases
• Do you retry?
With synchronous Request/Response we can loose business data if there’s no automatic retry mechanism.
Also if the operation we retry isn’t idempotent* we risk having the side effect multiple times!
Client Server
Processing
The same message can be
processed more than once
*Idempotence describes the quality of an
operation in which result and state does
not change if the operation is performed
more than 1 time
Request
Processing
Duplicated Request
Duplicated Response
Response
14. @jeppec
Sales system
Sales
Update customer status (e.g. Gold customer)
Bookkeeping
Deliver goods
Delivery
system
Deliveries
Customer/
CRM system
Customers
SAP
Bookkeeping
Complete
Purchase
Transaction
Coordinator
Transactional
Resource
Request-to-Prepare
Commit
Prepared
Done
Prepare
Phase
Commit
Phase
2 Phase Commit
15. @jeppec
What’s wrong with distributed transactions?
• Transactions lock resources while active
• Services are autonomous
• Can’t be expected to finish within a certain time interval
• Locking keeps other transactions from completing their
job
• Locking doesn’t scale
• X Phase Commit is fragile by design
17. @jeppec
Dual write problem
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Mutating
Remote call
Expected
transaction
boundary No simple solution (except a classic monolith)
• Local-commit-then-publish
• If the app crashes after local-
commit then the publish
operation isn’t performed
• Publish-then-local-commit
• If the local-commit fails then
we’ve already formed the publish
operation
• Billing Service may receive
published message BEFORE the
Order Service has committed
its transaction
19. @jeppec
Dual write problem
Orchestration / Process Manager / Saga*
Order
Service
Billing
Service
Client
Send-
Invoice
Transaction
boundary
Transaction
boundary
Save-Order
• Requires only local transactions
• Single coordinator
• Increases eventual consistency
• Requires idempotent operations
due to retries
• Requires compensations /
rollback
• Tighter coupling as the
orchestrator instructs other
services about what to do
* This isn’t the real Saga pattern
24. @jeppec
Dual write problem
Choreography with Outbox pattern
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Transaction
boundary
Transaction
boundary
• You can either use a Change
Data Capture (CDC) approach
e.g. using a database Write
Ahead Log (WAL)
• Risk of CRUD events!
• Or a manual OutBox pattern
where Save-Order writes data
and the event/command into
the OutBox within the same
database. Typically requires
polling and coordination +
recovery options in a multi cluster
setup
Events/Commands
Change
Data
Capture
/
Polling
Async Push or Poll
26. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book
Car Rental Hotel Rental
WorkQueue:
- Queue: “Rent Car”
- Queue: “Book Hotel”
- Queue: “Reserve Flight”
Routing slip
Flight Booking
27. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book
Car Rental Hotel Rental
WorkQueue:
- Queue: “Book Hotel”
- Queue : “Reserve Flight”
WorkStack:
- ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip Flight Booking
28. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book
Car Rental Hotel Rental
WorkQueue:
- Queue : “Reserve Flight”
WorkStack:
- Hotel ReservationId: XYZ
Cancellation: “Cancel Hotel”
- Car ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip
Book
Flight Booking
29. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book Book
Car Rental Hotel Rental
WorkStack:
- Hotel ReservationId: XYZ
Cancellation: “Cancel Hotel”
- Car ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip
Flight Booking
30. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book Book
Car Rental Hotel Rental
Cancel
WorkStack:
- Hotel ReservationId: XYZ
Cancellation: “Cancel Hotel”
- Car ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip
Flight Booking
31. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book Book
Car Rental Hotel Rental
Cancel
Cancel
WorkStack:
- Car ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip
Flight Booking
32. @jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book Book Book
Car Rental Hotel Rental
Cancel
Cancel
Cancelled
Flight Booking
34. @jeppec
Dual write problem
Choreography with Event Sourcing
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Transaction
boundary
Transaction
boundary
Event Store
Async
Push
or Pull
• Requires only local transactions
• Natively event driven: The Service
consumes and produces events
• Proven Audit Trail
• Flexible and adaptable
• Captures business intentions as
events
• Requires idempotent operations
35. @jeppec
Event Sourcing
Business-Objects/Aggregates track their own Domain Events
and derive their state from said Events
OrderCreated ProductAdded ProductAdded ProductRemoved ProductAdded OrderAccepted
Time
07:39
Time
07:40
Time
07:41
Time
07:45
Time
07:46
Time
07:50
36. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
37. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
Orderline
38. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
Orderline
Orderline
39. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
Orderline
40. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
Orderline
Orderline
41. @jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 102 2 2014-01-06 7:41 {Guid-3} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 104 4 2014-01-06 7:46 {Guid-5} ProductAdded <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order
Accepted: true
Orderline
Orderline
42. @jeppec
Client handled subscriptions
RSocketEvent
StreamSubscription
Local storage
EventStore
RSocketEvent
StreamSubscription
Local storage
EventStreamSubscription
Message
EventStreamSubscription
Message
EventStoreEventStreamPublisher
EventStoreEventStreamPublisher
Event
Event
Event Event
Supports
Single Instance
Subscriber,
which ensures
that only one
instance of
Subscriber B
has an active
subscription.
Other
instances of
the same
subscriber are
hot-standby
<<Topic Subscriber>>
Customer_Service:Some_Ac:OrderEvents
<<Topic Publisher>>
Sales_Service:OrderEvents
RSocketServer
tcp://subscribe-event-stream
A
B Subscriber B
RSocket Request/Stream
Event-Stream
Subscriber A
RSocket Request/Stream
Event-Stream
Flux<PersistedEvent> eventStream(long fromInclusiveGlobalOrder,
Option<String> subscriptionId)