ItalianCoders 19 June 2013. Ho to create a scalable and distributed job queue using Redis and Ruby

1. When crontab must die
Francesco Laurita
ItalianCoders 19 June 2013, Milan - Italy
Building Scalable,
Distributed Job
Queues with Redis
(and Ruby)
Wednesday, June 19, 13

2. Playing with your backend
A Front End request is supposed to return a response really fast
There are certain actions that may take longer and can’t be executed “live”
* Processing an order (fraud detection, external service call)
* Converting a video
* Resizing an Image
* In general, Not customer-facing operations that might take longer than few
milliseconds (even delivering an email)
Wednesday, June 19, 13

3. Processing Jobs: The wrong way (crontab)
• It loads your database
• Almost impossible to scale
• Painful errors recovery
• Waste of time between each schedule
Wednesday, June 19, 13

4. Processing Jobs: The wrong way (crontab)
• It loads your database
• Almost impossible to scale
• Painful errors recovery
• Waste of time between each schedule
Wednesday, June 19, 13

5. Processing Jobs like a pro: Message Queue
• A producer pushes a message into a jobs queue
• One ore more consumer consume messages from a job queue
LPUSH RPOP
RPUSHLPOP
Wednesday, June 19, 13

6. Processing Jobs like a pro: Message Queue
Wednesday, June 19, 13

7. Our choice: Redis
What is it?
* Is a NoSQL DB
* Is an advanced Key/Value Store
* Is a caching server
* Is a lot of things...
Wednesday, June 19, 13

8. Our choice: Redis
What is it?
* It is a way to share Memory over TCP/IP
Can share memory (data structure) between different processes and
languages
* List (LinkedList) --> queue.pop, queue.push
* Hash --> {}
* Set --> Set
* SortedSet --> SortedSet.new
* ....
Wednesday, June 19, 13

9. A real use case
Q1
Q2
Q4
Q5
S1
S2
Q3 Q6
Qi = queue i | Sj = service j
S3
Wednesday, June 19, 13

10. Simple Queue vs Reliable Queue
Simple queue:
1) pushing values into a queue
2) a consumer waits for these values using polling or a blocking operation
* Is not reliable: a message can be lost
- Network problems
- Consumer crashes while consuming message
Wednesday, June 19, 13

11. Simple Queue vs Reliable Queue
Reliable queue:
1) pushing values into a queue
2) a consumer fetches a message and at the same time pushes it into a
processing list
3) remove the message from the processing list once message has been
processed
Wednesday, June 19, 13

12. Simple Queue vs Reliable Queue
Messages queueProducer Consumer
Processing list
/dev/null
push
push
pop
lrem
Wednesday, June 19, 13

13. Reliable Queue in Ruby
https://github.com/taganaka/redis-queue
gem install redis-queue
Wednesday, June 19, 13

14. Reliable Queue in Ruby
S1 S2 S3 S4 Sx
Wednesday, June 19, 13

15. Everything seems perfect....but
•Keep in mind that
• Redis is an in-memory database
• All of the operations are done atomically in memory
•But
• RAM cannot be expanded indefinitely
• RAM is expensive
• A queue of 1M of messages is about 370MB occupied on Redis (about 400
chars for each entry)
Wednesday, June 19, 13

16. That’s all Folks...Questions?
twitter.com/flaurita
github.com/taganaka
francesco@gild.com
facebook.com/francesco.laurita
www.gild.com
We are hiring!
Wednesday, June 19, 13