Handwritten Text Recognition for manuscripts and early printed texts
How to build more reliable, robust and scalable distributed systems
1. How to build more reliable,
robust and scalable distributed
systems
Lars-Erik Kindblad
Senior Consultant
Twitter: @kindblad
E-mail: le@kindblad.com
2. The Book Shop
Place order should:
1. Save the order in a database
2. Charge the customer’s credit
card
3. Send a confirmation e-mail to
the customer
3. The typical way – request/response
Place order
Browser
1. Add order
OrderService
OrderController
2. Charge credit card
PaymentService
3. Send mail
NotificationService
Also called synchronous remote procedure call
4. Problem #1 – the order is lost
Failures:
Network
Webservice
Database++
Place order
1. Add order
Browser
OrderController
Error page
Error
OrderService
Exception
2. Charge credit card
PaymentService
3. Send mail
NotificationService
Order is lost
User will receive an error
User might leave – lost sale
User might retry
5. Problem #2 – no transactional management
Place order
1. Add order
Browser
OrderService
OrderController
Error page
Exception
2. Charge credit card
PaymentService
3. Send mail
Error
Order is stored
Payment is processed
E-mail is not sent
User receives an error and might retry:
Order might be stored twice in the database
Credit card might be charged multiple times
NotificationService
7. The message must also be processed
Queue
1. Connect
2. Receive message PlaceOrder
PlaceOrder
Worker
3. PlaceOrder
Message Handler
4. Add order
OrderService
5. Charge credit card
PaymentService
6. Send mail
NotificationService
8. Benefits
Very fast on the frontend – put the message on the queue
The order is never lost
Automatically retries during errors
4. Rollback. Put the message
back on the queue
and retry
Queue
1. Connect
2. Receive message
PlaceOrder
PlaceOrder
Worker
3. PlaceOrder
Error
Message Handler
9. We still have poor transactional management
1. Add order
Handle PlaceOrder
OrderService
2. Charge credit card
PaymentService
3. Send mail
Exception
Error
NotificationService
Order is created
Credit card is charged
E-mail sending fails
The message is put back on the queue and will be retried
Order is duplicated and credit card will be charged twice
10. Solution
Split into many messages - one message per transactional boundary
Place order
OrderController
Browser
1. Add PlaceOrder message
All the services must
be idempotent to be
100% reliable
PlaceOrder message
2. Handle PlaceOrder
Add PayOrder message
PayOrder message
Queue
OrderService
3. Handle PayOrder
Add SendMail
SendMail message
4. Handle SendMail
NotificationService
If this fails the
message is put back
on the queue and will
be retried
PaymentService
11. Scaling
Request/Response
Needs to process unpredictable many requests
Messaging
Needs to store unpredictable number of messages
The workers only process a predictable number of messages, even during peaks
Scale up
Concurrently process multiple messages
Scale out
Use multiple workers
12. Messaging challenges
1. ID generation
The order ID is first available in the worker
2. Eventual consistency
The PlaceOrder message might not have been picked up yet
13. NServiceBus
Lightweight messaging framework for .NET
Great choice for implementing one-way messaging + publish/subscribe
Open Source but not free
Available at http://particular.net/