A talk I gave at the Python Ireland meetup in June 2015 about rq (python-rq.org), a queuing library for the Python programming language that is backed by Redis.

1. Task queuing with
Redis and rq
Nadja Deininger

2. Who am I?
Python developer
Software engineer at Beautifeye
@machine_lady

3. Why use queues?
- Execute long tasks asynchronously
- Parallelise work
- Distribute work

4. Queues in Python
- Celery
- RabbitMQ
- Surely, many more…
- rq

5. Redis
redis.io
Key-value store
Supports lists (and other data structures)
Can be used to host queues

6. rq
python-rq.org
Needs a running Redis server - and that’s it!

7. Getting started
import rq
from redis import Redis
from my_module import my_function
with rq.Connection(Redis()):
q = rq.Queue("my_queue")
job = q.enqueue(my_function,
args=("arg1", "arg2"))

8. Performing work
rq comes with a default worker script:
$ rqworker queue1 queue2
Picks up jobs from the first nonempty queue.

9. Customise worker behaviour
Write your own worker class to e.g.
- manage a database connection
- avoid using os.fork
and use it by specifying
$ rqworker -w ‘path.to.MyWorker’

10. Get job results
Return value of enqueued function (if any) can
be accessed through job.result
Results are stored under job’s Redis hash
- default TTL: 500s
- job.result will be None until task is done

11. Dealing with failure
By default: put a job onto the failed queue;
can be accessed through rq.FailedQueue
Failed jobs don’t expire.
They can be retried (usually manually).

12. Dealing with failure differently
Supply your own exception handler to a Worker
instance:
worker = rq.Worker(queues,
exc_handler=my_exception_handler)

13. Monitoring rq - command line

14. rq-dashboard - web monitoring tool

15. rq-dashboard - web monitoring tool

16. Lessons learned
- Module imports can impact performance
- Don’t let queues get too large
- Functions cannot depend on context

17. Questions?
Nadja Deininger
nadja@ef.gy
@machine_lady