Sidekiq

Messaging & StreamingMessaging/QueueRuby

What it is

Sidekiq is a high-performance background job processor for Ruby, using threads and Redis to run many jobs per process.

Include Sidekiq::Job in a class and define perform. Jobs are serialised to Redis and executed by a worker process with configurable concurrency and retries.

Installation

gem 'sidekiq'

Getting started

The smallest useful thing you can do with it, and what each part means.

A job with retry configuration

ruby

1234567891011121314151617class ImportBooksJob
  include Sidekiq::Job

  sidekiq_options queue: :imports, retry: 5, backtrace: true

  def perform(import_id)
    import = Import.find(import_id)

    # Jobs must be idempotent — a retry may run this again.
    return if import.completed?

    ImportService.new(import).run
  end
end

ImportBooksJob.perform_async(import.id)
ImportBooksJob.perform_in(5.minutes, import.id)

Arguments must be simple JSON-serialisable values. Passing an ActiveRecord object is both large and stale by the time the worker runs — pass the id.

Advanced usage

Where the library earns its place over a simpler alternative.

Concurrency, uniqueness and threading hazards

ruby

12345678910111213141516171819202122232425# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 4]   # weighted — polled more often
  - [default, 2]
  - [low, 1]

# The database pool must be at least the concurrency, or threads block
# waiting for a connection.
#   RAILS_MAX_THREADS >= :concurrency

class ReportJob
  include Sidekiq::Job

  # Prevent overlapping runs across every worker process.
  def perform(report_id)
    lock = Redis.new.set("report:#{report_id}", 1, nx: true, ex: 300)
    return unless lock
    begin
      generate(report_id)
    ensure
      Redis.new.del("report:#{report_id}")
    end
  end
end

The connection-pool mismatch is the single most common Sidekiq problem: with concurrency 10 and a pool of 5, half the threads sit blocked and throughput collapses.

Errors and fixes

The failures you are most likely to hit, and what actually resolves them.

could not obtain a connection from the pool: The database pool is smaller than Sidekiq's concurrency. Raise RAILS_MAX_THREADS to match.
Jobs land in the dead set: Retries were exhausted. Inspect the error in the web UI, fix the cause, and retry from there.

Best practices

Write idempotent jobs — retries mean a job can run more than once.
Pass ids, never ActiveRecord objects or large payloads.
Set the database pool size at least equal to Sidekiq concurrency.
Secure the web UI; it can inspect arguments and re-run jobs.

Background

Why it exists, and what it was reacting to.

Sidekiq's threaded model made it far more memory-efficient than the fork-per-worker alternatives of the time, and it became the default for background work in Rails applications.

← All Ruby libraries Ruby guide