Distributed Task Queue

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.

The execution units, called tasks, are executed concurrently on a single or more worker servers. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Celery is already used in production to process millions of tasks a day.

Celery is written in Python, but the protocol can be implemented in any language. It can also operate with other languages using webhooks.

The recommended message broker is RabbitMQ, but support for Redis and databases is also available.

Celery is easy to integrate with Django, Pylons and Flask, using the django-celery, celery-pylons and flask-celeryadd-on packages.

Example

This is a simple task adding two numbers:

from celery.decorators import task

@task
def add(x, y):
    return x + y

You can execute the task in the background, or wait for it to finish:

>>> result = add.delay(8, 8)
>>> result.wait() # wait for and return the result
16

Getting Started

  1. Install celery by download or pip install -U celery
  2. Set up RabbitMQ or one of the ghetto queue solutions.
  3. Select one of the following guides:

Community

There is a mailing-list available for general discussion.

For those craving real, human interaction, there is also an IRC channel (#celery on irc.freenode.net).

Finally, if you find a bug or would like to request a feature, please submit an issue.

Celery 2.0 released!

By @asksol on 2010-07-02.

We're proud to announce the release of Celery 2.0. This version replaces the Django ORM with SQLAlchemy as the database result store. With this, Celery no longer depends on Django. Django integration is now available as a separate package called django-celery.

In addition there are a lot of new features: a curses monitor, time limits, complex crontab expressions, callbacks, simplified routing, and more. Everything is detailed in the changelog, so be sure to read it before upgrading.

Celery 1.0.6 released!

By @asksol on 2010-06-30.

RabbitMQ 1.8.0 came with stricter equivalence checks that broke the AMQP result backend, this release resolves this. If you've already used the AMQP backend you need to delete the previous declarations. For instructions please read the full changelog. Download from PyPI, or simply install the upgrade using pip install -U celery==1.0.6.


Celery 1.0.5 released!

By @asksol on 2010-06-01.

This release contains some important bugfixes related to shutdown and broker connection loss, as well as some other minor fixes. Also AbortableTask has been added to contrib. Please read the full changelog before you upgrade. Download from PyPI, or simply install the upgrade using pip install -U celery.


Celery 1.0.3 released!

By @asksol on 2010-05-15.

This release contains a drastic improvement in reliability and performance. Please read the full changelog before you upgrade. Download from PyPI, or simply install the upgrade using pip install -U celery.


Celery 1.0.1 released!

By @asksol on 2010-03-20.

This is a bugfix release and has some important changes to the shutdown procedure. Also improved compatibility with Windows and Python 2.4. Read the full Changelog for more information. Download from PyPI, or simply install the upgrade using pip install -U celery.


Celery 1.0 released!

By @asksol on 2010-02-10

Celery 1.0 has finally been released! It is available on PyPI for downloading. You can also install it via pip install celery. You can read the announcement here.


1.0 is in beta.

By @asksol on 2010-02-08

1.0 is scheduled to be released this week! Please help us test the latest release candiate to make this happen. To upgrade from an earlier version, please read the changelog.


New website.

By @asksol on 2010-02-08

We finally got a home page. Big thanks to Jan Henrik Helmers