Sidekiq, Delayed_job, or Resque? Sidekiq is a free and open-source Ruby job scheduler that was created by Mike Perham. When developing a Ruby on Rails application, you might find yourself overwhelmed with myriads of tasks that must be executed at the same time. For example: - Sending emails, - Charging credit cards, - Interacting with external APIs… Sidekiq allows you to scale your application by performing work in the background. It came out 10 years ago, in 2012! Mastering Sidekiq is not something you’ll need a triple digits IQ for. Nevertheless, stable and reliable software is a must for processing thousands of workers. This tool uses exclusively Redis as its database, which might not suit all. Memory leaks are also a thing to be aware of… In our latest article, we decided to compare Sidekiq to its alternatives — Delayed_job and Resque — to see if this omniscient tool still stands the test of time.

1. Sidekiq Tutorial & Overview:
Is Sidekiq For Ruby Still Worth
Working With After 10 Years?
Sherlock Holmes needs Dr. Watson. Ruby programming needs
Sidekiq. Or Delayed_job, or Resque?
Sometimes our clients are interested in what technologies we are
using to build their web application projects. We are happy to share,
although it’s not easy to lecture people on technologies and not to
be a total bore. So, prepare yourselves, that time of year has come
again.
All because Sidekiq, probably the most common background job
system for Ruby, just turned 10 years old! This overview will be
useful for those who are starting out with Ruby on Rails
programming, and those who are picky about their technology
stack. We provide tutorials on how to start Sidekiq, schedule and
test Sidekiq workers, send emails, and more.

2. What is Sidekiq?
Bruce Lee against Chuck Norris via Giphy
Sidekiq is a free and open-source Ruby job scheduler that was
created by Mike Perham. Ruby programming language is one of our
most used and well-mastered tools we respect and love. If you’re
interested in why we recommend it for clients’ projects, check out
our separate article on Ruby. If you don’t have that spare 10
minutes of your life to get to know our inner development processes
(we totally understand), then let’s round it up in 4 sentences and
introduce you to the major topic of this article:
1. Ruby is a programming language.
2. When developing a Ruby on Rails application, you might find
yourself overwhelmed with myriads of tasks that must be executed
at the same time.

3. 3. For example, sending emails, charging credit cards, interacting
with external APIs, and other data processing.
4. Developers use tools like Sidekiq that allow them to run
background jobs, i.e., automated processes outside of the
request-response cycle so users don’t have to wait in a queue.
So, Sidekiq is a default tool for a Ruby application that improves its
performance and response wait time.
Enterprise version
While Sidekiq is free by default, it doesn’t schedule jobs and only
executes them. The paid version of the tool — Enterprise version —
comes with scheduling. Mark Perham, its creator, didn’t aim to
divide the tool into paid and free versions. Rather, this idea came to
him accidentally.
Redis
Sidekiq works closely with Redis 4.0+ to organize job queues an d
store statistics. Redis (stands for Remote Dictionary Server) is a
fast open source in-memory key-value data store. It provides
sub-millisecond response times and allows real-time applications to
run millions of requests per second.

4. Workers and queues
The scheduler is run as the copy of the application in a separate
process and can be configured to use several worker threads. Each
worker is assigned to one or many queues. Queues represent the
list of tasks in the order they are added and can be named by
purpose (default, mail, reports) or by priority (low, normal, high) —
it’s completely up to the developer.
Sidekick 6.3, the latest version of the tool, came out in November
2021. Introducing the new version, Mark Perham elaborates on how
to use the term “worker” correctly. What exactly is a worker?
“Are you talking about a process? A thread? A type of
job? I encourage developers to stop using the term
worker and use include Sidekiq::Job in your job classes”
— M.P.
class SomeJob
include Sidekiq::Job
sidekiq_options ...
def perform(*args)
end
end

5. Plugins
Sidekiq has a myriad of prominent features: retries with
configurable periods, flexible error handling, a web UI and a ton of
plugins extending its functionality further.
One of such plugins, for example, introduces repeated tasks on
schedules, which were historically solved by tools based on cron.
Cron is harder to configure and takes time to start the whole app for
execution of a single task.
Another plugin monitors various metrics and checks for Sidekiq to
avoid errors. You can find many more on GitHub!

6. How did Sidekiq get so popular?
Single, married, multi-threading? Image by @dwfries on Medium
In 2012, Mike Perham was the first to introduce a multi-threading
background job system. Resque is multi-process and single-thread.
Delayed_job can work ‌
for single thread processing only. Sidekiq
was the first multi-threading tool in the Ruby community. With time,
it became the main driver of avoiding thread safety bugs in the Rails
ecosystem. Because Sidekiq is omnipresent, it has stabilized the
thread safety of Rails.

7. Who uses Sidekiq?
Sidekiq has become the de facto standard for web applications
these days. According to Stackshare, Sidekiq is used by such
well-known companies as:
● Adidas,
● GitLab,
● Product Hunt,
● Send Grid,
● New Relic,
● 500px,
● and many more.
A pick into companies using Sidekiq via Stackshare

8. How does Sidekiq compare to other
similar tools?
There are alternative projects with more or less the same set of
features. In this article, we compare Sidekiq to its most popular
competitors: Resque and Delayed_job.
Resque
Resque is a Ruby library for creating background jobs. You can
place them on multiple queues and process them later. Background
jobs can be any Ruby class or module. Existing classes can be
converted to background jobs. Resque also offers an option to
create new classes.
Resque has a slightly different concurrency model from Sidekiq,
which may or may not be what you are looking for. While Sidekiq
uses threads for its workers (and can’t scale onto many CPU
cores), Resque uses processes, which makes it slightly more
heavy-weight on one side, but lets it use cores of CPUs. This is
important for computationally heavy tasks when the single-CPU
threading model doesn’t apply well.

9. Sidekiq vs Resque statistic on stackshare
Unlike Sidekiq, Resque doesn’t require thread safety and works
with any gem (Ruby programs and libraries). According to
developers though, Sidekiq runs faster and uses much less
memory.
Delayed_job

10. Delayed_job, a.k.a. DJ is a priority queue, database-oriented tool
that allows to execute background tasks asynchronously. It was
originally extracted from Shopify and has become very popular
through the years.
According to Doug Breaker, who researched DJ in comparison to
Sidekiq, there are several features the latter doesn’t have:
● Easy integration with Rails that doesn’t depend on a
relational database/non-relational database.
● Custom data store. You can use the main data store to
process tasks offline and customize a data store to fit
your needs.
● No need to run multiple dependencies at once, e.g.,
instead of requiring multiple services and dependencies
to run a program, you can limit the dependencies that you
need.
Delayed::job, sadly, doesn’t support multi-threading, unlike Sidekiq.
The latter also runs faster, is scalable, and has the ability to run
Redis automatically.

11. Sidekiq vs Delayed_job statistic on stackshare
Using Sidekiq in plain Ruby
Now that you know of the alternatives, let’s learn how to use
Sidekiq in practice.
Sidekiq is framework-agnostic and can be used in any Ruby
application. The official Getting Started guide provides detailed
steps on how to get up and running.

12. The gist is that you:
1. Add Sidekiq to your Gemfile:
# Gemfile
gem 'sidekiq'
2. Define worker classes:
# app/workers/generate_report_worker.rb
class GenerateReportWorker
include Sidekiq::Worker
def perform(user_id)
# do your reporting here
end
end
3. Start Sidekiq as a separate process:
$ bundle exec sidekiq
4. Start scheduling your workers:
# Task is to be executed as soon as possible

13. GenerateReportWorker.perform_async(user.id)
# Task is to be executed in 5 minutes
GenerateReportWorker.perform_in(5 * 60, user.id)
# Task is to be executed at a certain moment (3 hours from
now)
GenerateReportWorker.perform_at(Time.now + 10800,
user.id
5. To execute a worker immediately:
MySimpleWorker.new.perform("I was performed!")
Using Sidekiq in Ruby on Rails
There is not much difference between using Sidekiq in plain old
Ruby or in Ruby on Rails programming. You still get a nice
integration with Rails Active Job framework and you can use
Date/Time helpers when scheduling future tasks:
Date/Time helpers examples
# Generate report next Monday
GenerateReportWorker.perform_at(Time.current.next_week,
user.id)

14. E.g., generate a report in 5 minutes:
GenerateReportWorker.perform_in(5.minutes, user.id)
Active Job integration
Active Job is a standard interface for interacting with job runners.
According to the official guide, it is a framework with a variety of
queuing back-ends. The jobs could be:
● Regularly scheduled clean-ups,
● Billing charges,
● Emails,
● Anything you can imagine running in parallel.
If you want your workers to not be Sidekiq-specific, you need to
take several steps:
1. Define your workers as ActiveJob jobs:
# app/jobs/generate_report_job.rb
class GenerateReportJob < ActiveJob::Base
# The name of the queue to put this job into
queue_as :default

15. def perform(user_id)
# do your reporting here
end
end
2. Configure Rails application to use the correct adapter:
# config/application.rb
class Application < Rails::Application
# ...
config.active_job.queue_adapter = :sidekiq
end
3. Use ActiveJob syntax for scheduling:
# Generate report as soon as possible
GenerateReportJob.perform_later(user.id)
There are plenty of configuration options.

16. Sending emails with ActionMailer and
Sidekiq
ActionMailer allows you to send emails (surprise!) from your Ruby
app. Here is what you need to send emails asynchronously with
Sidekiq:
1. For creating a mailer:
# app/mailers/users_mailer.rb
class UsersMailer < ActionMailer::Base
def welcome_email(user_id)
@user = User.find(user_id)
mail(to: @user.email, subject: "Welcome") do |format|
format.text
format.html
end
end
end
2. See views on emails:
app/views/users_mailer/welcome_email.html.erb - HTML version
app/views/users_mailer/welcome_email.text.erb - TEXT version
3. Finally, sending emails:
user = User.find(1)
mail = UsersMailer.welcome_email(user.id)
# mail.deliver_now
mail.deliver_later

17. Testing Sidekiq jobs (with RSpec
framework)
Sidekiq provides tools for testing the various aspects of your
workers at any stage of their lifecycle. To test workers directly, use:
worker = MyWorker.new
worker.perform(:my_arg)
Or you can use an inline mode:
# implementation
class DeleteFromRemoteApiWorker
include Sidekiq::Worker
def perform(item_ids)
ApiWrapper.delete_items(item_ids) # dependency
end
end
# test
describe DeleteFromRemoteApiWorker do
let(:items) do
# ...
end
it "delegates the work to the API wrapper as expected" do
allow(ApiWrapper).to receive(:delete_items)
item_ids = items.map(&:id)
described_class.perform_async(item_ids)
expect(ApiWrapper).to(
have_received(:delete_items).with(item_ids)

18. )
end
end
Why do we (still) use Sidekiq in
outsourced projects?
Mastering Sidekiq is not something you’ll need a triple digits IQ for.
Nevertheless, ‌
stable and reliable software is a must for processing
thousands of workers. This tool uses exclusively Redis as its
database, which might not suit all. Memory leaks are also a thing to
be aware of…
Nevertheless, Sidekiq architecture is perfect when working with
complex Ruby applications. For example, we created a virtual
learning environment complete with live chats, video streaming,
billing charges, and more, with Sidekiq in our tech stack.
Sidekiq is ideal for those who require fast speed, executing 7100
jobs per second. Multi-threading abilities mean that multiple jobs
can be queued in the background without affecting the synchronous
workflow of the application, improving overall performance.
So, is Sidekiq still our best buddy? Yes! Happy birthday, Sidekiq! 🎂

19. Image via Giphy