This document summarizes Chris Richardson's presentation on using sagas to maintain data consistency in a microservices architecture. Richardson explains that ACID transactions cannot be used across services due to limitations of distributed transactions. Instead, sagas can be used, where each service performs a local transaction and messages are passed between services. There are two main approaches to coordinating sagas: choreography-based using domain events, and orchestration-based where a central saga manages the process. Countermeasures like semantic locking are also needed to prevent data anomalies from the lack of isolation in sagas.
7. @crichardson
Loose coupling =
encapsulated data
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
Change schema without coordinating with other teams
8. @crichardson
ACID transactions within
services are fine
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
ACID transaction ACID transaction
9. @crichardson
But how to maintain data
consistency across services?
Order Service Customer Service
Order Database Customer Database
Order table
Customer
table
orderTotal creditLimit
availableCredit
?
10. @crichardson
Enforcing the customer’s credit limit
createOrder(customerId, orderTotal)
Pre-conditions:
• customerId is valid
Post-conditions
• Order was created
• Customer.availableCredit -= orderTotal
Customer class
Invariant:
availableCredit >= 0
availableCredit <= creditLimit
Spans
services
11. @crichardson
Cannot use ACID transactions
that span services
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
SELECT CREDIT_LIMIT
FROM CUSTOMERS WHERE CUSTOMER_ID = ?
…
… CHECK FOR SUFFICIENT CREDIT….
…
INSERT INTO ORDERS …
…
COMMIT TRANSACTION
Private to the
Order Service
Private to the
Customer Service
Distributed transactions
12. @crichardson
2PC is not an option
Guarantees consistency
BUT
2PC coordinator is a single point of failure
Chatty: at least O(4n) messages, with retries O(n^2)
Reduced throughput due to locks
Not supported by many NoSQL databases (or message brokers)
CAP theorem 2PC impacts availability
….
16. @crichardson
Saga
Use Sagas instead of 2PC
Distributed transaction
Service A Service B
Service A
Local
transaction
Service B
Local
transaction
Service C
Local
transaction
X Service C
17. @crichardson
Order Service
Create Order Saga
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder() Initiates saga
20. @crichardson
Rolling back sagas
Use compensating transactions
Developer must write application logic to “rollback” eventually
consistent transactions
Careful design required!
22. @crichardson
Order Service
Create Order Saga - rollback
Local transaction
Order
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
reject
order()
createOrder()
FAIL
Insufficient credit
23. @crichardson
Writing compensating
transactions isn’t always easy
Write them so they will always succeed
If a compensating transaction fails no clear way to recover
Challenge: Undoing changes when data has already been
changed by a different transaction/saga
More on this later
Actions that can’t be undone, e.g. sending email
Reorder saga and move to end
24. Structure of Create Order
Saga
T1:
Create Order
C1:
Reject Order
T2:
Reserve Credit
T3:
Approve Order
Needs to be compensatable
because Reserve Credit can
fail
Last step that can fail: Does not
need compensation
Can’t fail
25. Saga structure: 3 phases
T1 C1
…
T2 C2
Tn+1
Tn+2
….
Compensatable transactions
Pivot transaction = GO/NO GO
Retriable transactions that can’t fail
and don’t need compensation
Great for actions that cannot be undone
26. Ordering the steps of a saga
Dependencies between
steps:
Result of one step is
used as input to next
e.g. Create Order before
Reserving Credit
But often you have a choice
Simplify by moving updates
to after pivot transaction
Update
Customer
Undo Update
Customer
…
Pivot Transaction
Update
Customer
…
Pivot Transaction
No compensation required
Reorder
27. @crichardson
Challenge: Sagas are ACD
Atomicity
Saga implementation ensures that all transactions are
executed OR all are compensated
Consistency
Referential integrity within a service handled by local databases
Referential integrity across services handled by application
Durability
Durability handled by local databases
29. @crichardson
Anomaly: Lost update
Ti: Create
Order
Tj: Approve
Order
Ti: Cancel
Order
Saga 1
Create Order
Saga 2
Cancel order
Time
Overwrites
cancelled order
Also: A compensating transaction that simply restores
previous value can cause a lost update
30. @crichardson
Anomaly: Dirty reads
Ti: Reserve
Credit
Ci: Unreserve
credit
Ti: Reserve
Credit
Reads
uncommitted
changes
Order rejected unnecessarily OR Credit limit exceeded
Time
Saga 1
Create Order
Saga 2
Create Order
Ti: Reserve
InventoryX
Temporarily changed
31. @crichardson
Use countermeasures to
prevent anomalies
Use semantic lock to prevent lost updates
e.g. a PENDING order cannot be cancelled
Reorder saga to eliminate dirty reads caused by compensating transactions
Ci undoes changes made by Ti => possibility of dirty read
Eliminate Ci by moving update to (after) pivot transaction
Use commutative updates to eliminate lost updates caused by compensating
transactions
Customer.reserveCredit() and releaseCredit()
Account.debit() and credit()
…
https://www.slideshare.net/chris.e.richardson/saturn-2018-managing-data-consistency-in-a-microservice-architecture-using-sagas
33. @crichardson
How to sequence the saga
transactions?
After the completion of transaction Ti “something” must
decide what step to execute next
Success: which T(i+1) - branching
Failure: C(i - 1)
34. @crichardson
Use asynchronous, broker-
based messaging
Order Service
Customer
Service
….
Message broker
Guaranteed atleast once delivery ensures a saga
complete when its participants are temporarily unavailable
35. @crichardson
Saga step = a transaction local
to a service (a.k.a. participant)
Service
Database Message Broker
update Send message/event
How to
make atomic
without 2PC?*
*Transactional Outbox pattern or Event Sourcing
37. @crichardson
Option #1: Choreography-based
coordination using events
Order
Service
Customer
Service
Order created
Credit Reserved
Credit Limit Exceeded
Create Order
OR
Customer
creditLimit
creditReservations
Order
state
total
create()
reserveCredit()
approve()/
reject()
Order events channel
Customer events channel
41. Benefits and drawbacks of
choreography
Benefits
Simple, especially when using event
sourcing
Participants are loosely coupled
Drawbacks
Decentralized implementation -
potentially difficult to understand
Cyclic dependencies - services listen
to each other’s events, e.g.
Customer Service must know about
all Order events that affect credit
Overloads domain objects, e.g.
Order and Customer know too
much
Events = indirect way to make
something happen
https://github.com/eventuate-examples/eventuate-examples-java-customers-and-orders
42. @crichardson
Order Service
Option #2: Orchestration-based saga
coordination
Local transaction
Order
state=PENDING
createOrder()
Customer Service
Local transaction
Customer
reserveCredit()
Order Service
Local transaction
Order
state=APPROVED
approve
order()
createOrder()
CreateOrderSaga
InvokesInvokesInvokes
44. Saga orchestrator behavior
On create:
Invokes a saga participant
Persists state in database
Wait for a reply
On reply:
Load state from database
Determine which saga
participant to invoke next
Invokes saga participant
Updates its state
Persists updated state
Wait for a reply
…
45. @crichardson
Order Service
CreateOrderSaga orchestrator
Customer Service
Create Order
Customer
creditLimit
creditReservations
...
Order
state
total…
reserveCredit
CreateOrder
Saga
OrderService
create()
create()
approve()
Credit Reserved
Customer command channel
Saga reply channel
https://github.com/eventuate-tram/eventuate-tram-sagas-examples-customers-and-orders
48. @crichardson
Eventuate Tram Sagas
Open-source Saga orchestration framework
Currently for Java
https://github.com/eventuate-tram/eventuate-tram-sagas
https://github.com/eventuate-tram/eventuate-tram-sagas-
examples-customers-and-orders
49. Benefits and drawbacks of
orchestration
Benefits
Centralized coordination
logic is easier to understand
Current state is in database
Reduced coupling, e.g.
Customer knows less
Reduces cyclic
dependencies
Drawbacks
Risk of smart sagas
directing dumb services
50. @crichardson
Summary
Microservices tackle complexity and accelerate development
Database per service is essential for loose coupling
Use ACID transactions within services
Use orchestration-based or choreography-based sagas to
maintain data consistency across services
Use countermeasures to reduce impact of anomalies caused
by lack of isolation