Skip to content

Celery

Distributed task queue for work that should not happen inside a web request.

Developer UtilitiesCLI/UtilsPython
Official docs ↗GitHub ↗

What it is

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation but supports scheduling as well.

Celery allows you to define tasks in Python functions, which can be executed asynchronously or scheduled periodically. It supports multiple brokers like RabbitMQ, Redis, and Amazon SQS, and provides tools for monitoring and managing task execution.

Requires
A message broker — Redis or RabbitMQ
Licence
BSD 3-clause

When to use it

The question documentation cannot answer for you — because it cannot recommend something else.

Reach for it when

  • Sending email, generating reports, processing uploads — anything slow or failure-prone
  • Scheduled and periodic jobs with retries and result tracking
  • You need to distribute work across multiple worker machines

Look elsewhere when

  • A single small application only needs a couple of background jobs — a simpler queue may suffice
  • You cannot run a broker such as Redis or RabbitMQ

Installation

pip install celery

Getting started

The smallest useful thing you can do with it, and what each part means.

Defining a simple task
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y
Defines a Celery app with Redis as the broker and a simple `add` task that can be executed asynchronously.
Calling a task asynchronously
result = add.delay(4, 6)
print(result.get(timeout=10))
Calls the `add` task asynchronously using `delay()` and retrieves the result with a timeout.

Advanced usage

Where the library earns its place over a simpler alternative.

Periodic tasks with Celery Beat
from celery.schedules import crontab
app.conf.beat_schedule = {
    'add-every-minute': {
        'task': 'tasks.add',
        'schedule': crontab(minute='*'),
        'args': (2, 3),
    },
}
Schedules the `add` task to run every minute using Celery Beat.
Chaining tasks
from celery import chain
result = chain(add.s(2,3), add.s(4))().get()
Chains tasks together so that the output of one task is used as the input to the next.
Using multiple brokers
app = Celery('tasks', broker=['redis://localhost:6379/0', 'amqp://guest@localhost//'])
Configures Celery to use multiple brokers for redundancy or load balancing.

Errors and fixes

The failures you are most likely to hit, and what actually resolves them.

TimeoutError
Set appropriate `timeout` when retrieving results and handle task failures with retries.
BrokerConnectionError
Check that your message broker is running and accessible. Ensure network connectivity.
TaskRevokedError
Occurs when a task is revoked before execution. Handle with try/except and consider retries.

Best practices

  • Use Redis or RabbitMQ as a reliable broker for production environments.
  • Keep tasks idempotent to allow retries safely.
  • Use separate queues for different priorities or types of tasks.
  • Monitor task execution using Flower or Celery events.
  • Avoid long-running tasks in synchronous workflows; delegate them to Celery workers.

Alternatives

Comparable options, and the reason you would pick one over the other.

Background

Why it exists, and what it was reacting to.

Celery was created by Ask Solem and released in 2009. It was designed to provide a simple and reliable framework to run background tasks in Python applications, supporting distributed processing across multiple workers, queues, and brokers.

Related libraries

Others in Developer Utilities for Python.

← All Python librariesPython guide