As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great start, however as your load increases, you very quickly outgrew that simplistic queuing system. We will cover where getting started using Resque and Redis, how to test your jobs, when it makes sense to use Resque, implementations of Resque in other languages, and look how I've used Resque.

1. Background Jobs with
Resque
Jon Homan
Apprentice @ Obtiva
http://jonhoman.com
@jonhoman

3. Agenda
• Resque
• Examples
• Real world use
• Plugins
• Alternate Implementations
• When to use Resque

4. Background Jobs
• Remote API calls
• Uploading images
• Number crunching
• Updating RSS feeds

5. Resque
https://github.com/defunkt/resque
• Open-source library for creating background
jobs
• Redis backend
• Support for multiple queues
• Queues are processed by workers

6. More Resque Features
• Monitor with god or monit
• Distributed processing
• Priorities
• Persistent queues
• Memory leak resistant
• Web interface

7. resque-web

8. Redis
http://redis.io/
• Key-value datastore
• data structure server
• constant time pushes and pops
• atomic operations

9. All “Foreground”
class FeedsController
def create
feed = Feed.create!
feed.fetch
redirect_to feed_path(feed)
end
end
class Feed
def fetch
rss = RSS::Parse.parse(url, false)
feed.title = rss.title
end
end

10. Queue up creation
class FeedsController
def create
feed = Feed.create!
Queue.push(feed) # won’t block UI
redirect_to feed_path(feed)
end
end
class Queue
def push(feed)
queue << feed
end
end

11. Now with Resque
class FeedsController
def create
feed = Feed.create!
Resque.enqueue(FeedCreator, feed.id)
redirect_to feed_path(feed)
end
end

12. How to process jobs
class FeedCreator
@queue = :feed_creation
def perform(feed_id)
feed = Feed.find(feed_id)
rss = RSS::Parser.parse(feed.url,false)
feed.title = rss.title
# create all items in the rss
end
end

13. Workers?
• Long-running Rake tasks
• Look for jobs every n seconds

14. How to start workers
$ rake resque:work
$ QUEUE=feed rake resque:work
$ QUEUE=feed, twitter rake resque:work
$ QUEUE=* rake resque:work
$ COUNT=3 QUEUE=* rake resque:work

15. Testing w/ RSpec
it "stores the name of the feed" do
FeedCreator.perform(feed.id)
feed.reload.title.should == "Feed
end

16. Testing w/ RSpec
it "creates Items for every item in the feed" do
FeedCreator.perform(feed.id)
feed.reload.items.count.should == 5
end

17. Testing w/ RSpec
it “parses the feed” do
RSS::Parser.should_receive(:parse)
FeedCreator.perform(feed.id)
end

18. Real world uses
• GitHub
• Groupon
• lifeStreams

19. GitHub
35 different types of jobs:
• Warming caches
• Counting disk usage
• Building tarballs
• Building Rubygems
• Building graphs
• Updating search index

20. Groupon
• Resque for background jobs
• Notifications
• Lockable worker
• Redis for semaphores

21. lifeStreams
• Process updates to blog feeds
• Publish updates to Facebook, Twitter, Email

22. Plugins
• resque_spec
• Resque Heroku Autoscaler
• resque multi step
• ResqueMailer
• resque-retry
• and at least 25 others

23. ResqueSpec
https://github.com/leshill/resque_spec
• Resque plugin that adds resque-specific
matchers
• have_queued
• have_scheduled
• have_queue_size_of

24. Examples
it “adds a feed to the creation queue” do
post :create, :feed
FeedCreator.should have_queued(feed.id)
end
it “schedules a feed update” do
feed.update
FeedUpdater.should have_scheduled(feed.id)
end

25. Resque Heroku
Autoscaler
• Scales Heroku workers
• Start worker if none running
• Stops worker after jobs are processed

26. RHA
require 'resque/plugins/resque_heroku_autoscaler'
class TestJob
extend Resque::Plugins::HerokuAutoscaler
@queue = :test
def perform
...awesome stuff...
end
end

27. Resque Mailer
• Asynchronously deliver email in Rails

28. Resque Mailer
class MyMailer < ActionMailer::Base
include Resque::Mailer
end
MyMailer.subject_email(params).deliver
QUEUE=mailer rake environment resque:work

29. resque-retry
• Retry, delay and exponential backoff support

30. resque-retry
require 'resque-retry'
class ExampleRetryJob
extend Resque::Plugins::Retry
@queue = :example_queue
@retry_limit = 3
@retry_delay = 60
def self.perform(*args)
# your magic/heavy lifting goes here.
end
end

31. resque-retry
no delay, 1m, 10m, 1h, 3h, 6h
@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]

32. Alternate
Implementations
• C
• Java
• Scala
• .NET
• Python
• PHP
• node.js

33. jesque
// Add a job to the queue
final Job job = new Job("TestAction",
new Object[]{ 1, 2, true, "test", Arrays.asList("inner",
4ß)});
final Client client = new ClientImpl(config);
client.enqueue("foo", job);
client.end();
// Start a worker to run jobs from the queue
final Worker worker = new WorkerImpl(config,
Arrays.asList("foo"), Arrays.asList(TestAction.class));
final Thread workerThread = new Thread(worker);
workerThread.start();

34. Resque?
• Multiple queues
• Potentially HUGE queues
• Admin UI
• Expecting chaos/failures

35. DelayedJob?
• Numeric priorities
• Queue is small
• Add whole objects to queue
• No need to setup Redis

36. Do your homework

37. I didn’t
(kinda)

38. Resources
github.com/defunkt/resque
redis.io
redistogo.com

39. Questions?