2. Agenda
Explain scenario
Implement using a single database
Explain concurrency issues under load
Implement using a sharded database
Implement using WebSphere eXtreme Scale and
chained transactions
@billynewport
2
3. Scenario
Large eCommerce web site
Problem is order checkout
We track inventory levels for each SKU
Orders during checkout need to adjust available
inventory.
@billynewport
3
4. Shopping cart metrics
Millions of SKUs
Cart size of 5 items for electronics/big ticket items
Cart size of 20 items for clothing
Expect concurrent load of 2500 checkouts per second
@billynewport
4
5. Database
Begin
For each item in cart
Select for update where sku = item.sku
Decrement available sku level
If not available then rollback…
Update level where sku = item.sku
Commit
Cart items randomly distributed amongst all 2m items, lots
of concurrency.
Simple enough, right? All is good?
@billynewport
5
6. Problem: cabbage patch dolls
Cabbage patch dolls are popular this fall…
@billynewport
6
7. Database killers!
The dolls cause major
concurrency problems
Lots of row level locks
Contention on doll rows
Possible table lock escalation
App server thread issues
Connection pools empty
Then DEATH!
They aren’t sweet and cuddly any
more…
@billynewport
7
8. Database killers
We need a way to get locks to decrement inventory
But, we don’t want to hold the lock for very long
Bigger carts make the problem worse, all the locks held
for longer
Ideally, hold locks for constant time
Any contentious items make problem worse
@billynewport
8
9. Solution
Hold lock on inventory rows for as short a time as
possible
Decouple this from size of cart.
How?
@billynewport
9
10. Chained transactions
Programmers think of transactions in synchronous
terms.
Begin / Do Work / Hold locks / Commit
Easy to program, bad for concurrency.
@billynewport
10
11. Inspiration
Microsoft had COM objects with apartment model
threading.
Modern Actor support is similar. Some state with a
mailbox.
BPEL supports flows with compensation
Data meets actors is a good analogy
Send a message (cart) to a group of actors identified
using their keys with a compensator
@billynewport
11
12. Alternative
We need to think asynchronously in terms of flows with
compensation
Map of <SKU Key/SKU Amount>
Brick:
Do
{ code to reduce inventory level for SKU }
Undo
{ code to increase level inventory for SKU }
Provide Map with do/undo bricks
Easy to program, great concurrency.
@billynewport
12
13. Transactions and sharded
stores
Option 1: Write transaction to one shard then
spread out asynchronously
Option 2: 2 phase commit
Option 3: Chained transactions
@billynewport
13
14. Transactions and sharded
stores
Option 1: Write transaction to one shard then spread
out asynchronously
Option 2: 2 phase commit
Option 3: Chained transactions
@billynewport
14
15. Transactions and sharded
stores
Option 1: Write transaction to one shard then spread
out asynchronously
Option 2: 2 phase commit
Option 3: Chained transactions
@billynewport
15
16. Implementation
1PC only required
Data store supporting
Row locks
Row oriented data
Integrated FIFO messaging
IBM WebSphere eXtreme Scale provides these
capabilities.
@billynewport
16
17. Implementation
Application makes map and code bricks
Submits transaction as an asynchronous job.
Uses a Future to check on job outcome.
Do blocks can trigger flow reversal if a problem occurs.
Invoke undo block for each completed step
@billynewport
17
18. Mechanism
Loop
Receive message for actor key
Process it
Send modified cart to next ‘sku’ using local ‘transmit q’
Commit transaction
Background thread pushes messages in transmit q to
the destination shards using exactly once semantics.
@billynewport
18
19. Performance
A single logical transaction will be slower than a 1PC
DBMS implementation.
However, under concurrent load then it will deliver:
Higher throughput
Better response times
Thru better contention management
Each ‘SKU’ only locked for a very short period
@billynewport
19
20. Generalization
This could be thought of as a workflow engine.
But, a big difference here is that a workflow engine usually
talks with a remote store.
Here:
the flow state is the MESSAGE
It moves around to where the data is for the next step
Using a MESSAGE for flow state rather than a database
means it scales linearly.
The message ‘store’ is integrated and scales with the data
store.
@billynewport
20
21. Architecture Comparison
Conventional Message oriented
@billynewport
21
Flow
DB
Appl
DB
Msg
Store
BP
Engi
ne
BP
Engi
ne
BP
Engine
Flow
State
Flow
Edge
= Msg
Integrated
Msg/Data store
Appl
DB
Write
behind
Integrated
Msg/Data store
Integrated
Msg/Data store
22. Sample implementation
Coming soon.
Running in lab
Working with several eCommerce customers looking to
implement soon.
Soon to be published on github as sample code.
@billynewport
22