High performance
queues with Cassandra
Mikalai Alimenkou
http://xpinjection.com
@xpinjection

Kiev, Ukraine
#евромайдан

How to
process
data, master
?
Asynchronously!

Queues usage scenario

More
realistic
scenario

More
realistic
scenario

Are you crazy?
Cassandra for
queues?

So many cool MQ providers

Initial expected loading

Some specific requirements
1
Message
Queue
External Service
Provider

Some specific requirements
Message
Queue
1
External Service
Provider

Some specific requirements
2
Message
Queue
External Service
Provider

Some specific requirements
Message
Queue
2
1
Redeliver
after 1 hour
External Service
Provider

Some specific requirements
2
Message
Queue
External Service
Provider
Redeliver
after 1 hour
Redelivery business logic and external service
hourly based usage limits

Idea came from railways

“Body flow” in regular life

Message batches “station”
QUEUE
NOW
ALMOST READY
+1 HOUR
WAITING
+6 HOURS
WAITING
+12 HOURS
WAITING

System components: Message
MESSAGE = REAL REQUEST DATA WITH UNIQUE ID
1ST FIELD
3rd FIELD
{ field data}
MESSAGE ID
2nd FIELD
{ field data}
{ field data}
ID FORMAT ALLOWS 4096 MESSAGES
PER MILLISECOND FROM ONE NODE
Timestamp
44 bits
Counter
12 bits
Cluster node ID
8 bits

System components: Batch
• Open for at least 1 second
• Closing if opened for > 10 seconds
• Closing if has > 100 messages
Ascending columns ordering
1ST MESSSAGE ID
2nd MESSAGE ID
3rd MESSAGE ID
BATCH ID { opt message data} { opt message data} { opt message data}
ID FORMAT REQUIRE BATCH TO BE OPENED FOR > 1 SECOND
Timestamp
Rounded to seconds
Cluster node ID + Batch Type
Last 3 digits

System components: Queue
• Similar to batch
• Unlimited
• May have batches with past time
Ascending columns ordering
1ST BATCH ID
QUEUE NAME
2nd BATCH ID
3rd BATCH ID
{ processed at }
{ processed at }
{ processed at }

System components: Broker
batches polling
BROKER
check batch time
process batch
PROCESSOR
QUEUE
lock batch for
processing
ZOOKEEPER
•
•
•
•
•
Natural pre-fetch thanks to batches
Easy to control messages processing
Simple concurrency model
Easy scalable between nodes
No high loading on Cassandra

System components: Processor
PROCESSOR

System components: Processor
PROCESSOR
OK

System components: Processor
PROCESSOR

System components: Processor
PROCESSOR
redeliver
on failure
ANOTHER BATCH
•
•
•
•
Tries to process messages as quickly as possible
On error just redeliver message
Messages are processed concurrently
Any redelivery business logic is easy to implement

Warnings and benefits
• Message and batch must be
checked before processing
• Hard to explain “queue” size
• Separate columns for status
tracking of message
• Perform correct compaction
from time to time
• Expected loading is handled
with single node
• Everything works on
commodity hardware
• Single storage for all data
• System is easily scalable and
reliable (no message was
lost)

Show me the code!

@xpinjection
http://xpinjection.com
mikalai.alimenkou@xpinjection.com