Leveraging MongoDB as a queue infrastructure. This talk describes a low-overhead approach for building a queue as a service built on top of MongoDB, covering concerns of durability, long running processes and distributed processing.

1. QUEUE IN THE
CLOUD WITH
MONGODB
MONGODB LA 2013
NURI HALPERIN

2. QUEUE

3. USAGE
Ordered execution
Buffering consumer/producer
Work distribution

4. GOALS OF PROJECT
Leverage Mongo
• Reduce ops overhead by reusing infrastructure
• Map queue semantics to Mongo’s strengths
Reliable
• Durable - support long running process
• Resilient to machine failure
• Narrow down window of failure/ data loss.
Centralized, distributed:
• Multiple producers
• Multiple consumers

5. ITERATION 0
Capped collection – not the perfect choice
• Tailing queue seems attractive, but…
• Need external sync to avoid double-consume
• Secondary indexes and updating are anti-pattern
Relaxing FIFO is OK
• No guarantee that first-popped is first done
• Multi-client is negated if they have to sync on execution order
• Race condition for queue insertion has same effect
Conclusion: Project doesn’t use capped collection and
relaxes FIFO.

6. PARANOID BY DESIGN
Network dies
Process dies
DB dies
Machine dies Poison letter Dead letter

7. ITERATION 1
db.q4foo.save({v:{f:1}})
db.q4foo.findAndModify({query: {}, sort: {_id:1}, remove: true})
Hot: quick and simple
Not: dead client, dead in transit, no trace

8. ARE WE THERE YET?
Network dies
Process dies
DB dies
Machine dies Poison letter Dead letter

9. QUEUE SEMANTICS
Local / Memory Distributed
Push Put
Pop Get << visibility >>
<< exception >> Release << retry >>
Delete
<< exception >>

10. ITERATION 2
db.q4foo.save({v:{f:1}, dq: null})
db.q4foo.findAndModify( {
query: { dq: null},
sort: {_id:1},
update:{ $set: { dq: later(60)}}})
… If processing was success => delete..
Hot: If client dies, item remains in queue. Data not lost.
Not: index on _id less useful in high volume.

11. ARE WE THERE YET?
Network dies
Process dies
DB dies
Machine dies Poison letter Dead letter

12. ITERATION 3
db.q4foo.save({v:{f:1}, dq: null, pc: 0})
db.q4foo.findAndModify({
query: { dq: null, pc:{$lt:3}},
sort: {_id:1},
update:{$set:{dq:later(60)},$inc:{pc:1}}}) // consume
db.q4foo.findAndModify({
query: {_id:"..."},
update:{$set:{dq: null}}}) // release
Hot: An item can be retried automatically (pc) after released.
Exhausted item remains in queue.
Not: Not strict FIFO.

13. ARE WE THERE? YES.
Network dies
Process dies
DB dies
Machine dies Poison letter Dead letter

14. ITERATION 4
Ensure your queue writes use applicable durability
• db.q4foo.save() + getLastError(…)
• db.q4foo.findAndModify () + getLastError(…)
Replica sets for durability only. No capacity or speed gain.

15. OTHER THOUGHTS
Create admin jobs to monitor queues:
• Growth
• Retries exhausted
Consider TTL risks (ex: client failure before calling Release())
Consider idempotent operations when possible
Design clients to back off polling
Separate queue vs. extra “topic” field
Consider dedicated DB for write-lock scope
Capped vs. regular collection – capped now can have _id, in-place update.

16. Q&A
Thank you!
Nuri Halperin
nuri@plusnconsulting.com