celery - Distributed Task Queue for Django.¶

Downloading and installing from source¶

Download the latest version of celery from http://pypi.python.org/pypi/celery/

You can install it by doing the following,:

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install # as root

Using the development version¶

You can clone the repository by doing the following:

$ git clone git://github.com/ask/celery.git celery

Installing RabbitMQ¶

See Installing RabbitMQ over at RabbitMQ’s website. For Mac OS X see Installing RabbitMQ on OS X.

Setting up RabbitMQ¶

To use celery we need to create a RabbitMQ user, a virtual host and allow that user access to that virtual host:

$ rabbitmqctl add_user myuser mypassword

$ rabbitmqctl add_vhost myvhost

From RabbitMQ version 1.6.0 and onward you have to use the new ACL features to allow access:

$ rabbitmqctl set_permissions -p myvhost myuser "" ".*" ".*"

See the RabbitMQ Admin Guide for more information about access control.

If you are still using version 1.5.0 or below, please use map_user_vhost:

$ rabbitmqctl map_user_vhost myuser myvhost

Configuring your Django project to use Celery¶

You only need three simple steps to use celery with your Django project.

1. Add celery to INSTALLED_APPS.

2. Create the celery database tables:

   $ python manage.py syncdb

3. Configure celery to use the AMQP user and virtual host we created

   before, by adding the following to your settings.py:

   AMQP_SERVER = "localhost"
   AMQP_PORT = 5672
   AMQP_USER = "myuser"
   AMQP_PASSWORD = "mypassword"
   AMQP_VHOST = "myvhost"
   

That’s it.

There are more options available, like how many processes you want to process work in parallel (the CELERY_CONCURRENCY setting), and the backend used for storing task statuses. But for now, this should do. For all of the options available, please consult the API Reference

Note: If you’re using SQLite as the Django database back-end, celeryd will only be able to process one task at a time, this is because SQLite doesn’t allow concurrent writes.

Running the celery worker server¶

To test this we’ll be running the worker server in the foreground, so we can see what’s going on without consulting the logfile:

$ python manage.py celeryd

However, in production you probably want to run the worker in the background, as a daemon:

$ python manage.py celeryd --detach

For a complete listing of the command line arguments available, with a short description, you can use the help command:

$ python manage.py help celeryd

Defining and executing tasks¶

Please note All of these tasks has to be stored in a real module, they can’t be defined in the python shell or ipython/bpython. This is because the celery worker server needs access to the task function to be able to run it. So while it looks like we use the python shell to define the tasks in these examples, you can’t do it this way. Put them in the tasks module of your Django application. The worker server will automatically load any tasks.py file for all of the applications listed in settings.INSTALLED_APPS. Executing tasks using delay and apply_async can be done from the python shell, but keep in mind that since arguments are pickled, you can’t use custom classes defined in the shell session.

While you can use regular functions, the recommended way is to define a task class. This way you can cleanly upgrade the task to use the more advanced features of celery later.

This is a task that basically does nothing but take some arguments, and return a value:

>>> from celery.task import Task
>>> from celery.registry import tasks
>>> class MyTask(Task):
...     name = "myapp.mytask"
...     def run(self, some_arg, **kwargs):
...         logger = self.get_logger(**kwargs)
...         logger.info("Did something: %s" % some_arg)
...         return 42
>>> tasks.register(MyTask)

Now if we want to execute this task, we can use the delay method of the task class (this is a handy shortcut to the apply_async method which gives you greater control of the task execution).

>>> from myapp.tasks import MyTask
>>> MyTask.delay(some_arg="foo")

At this point, the task has been sent to the message broker. The message broker will hold on to the task until a celery worker server has successfully picked it up.

Right now we have to check the celery worker logfiles to know what happened with the task. This is because we didn’t keep the AsyncResult object returned by delay.

The AsyncResult lets us find the state of the task, wait for the task to finish and get its return value (or exception if the task failed).

So, let’s execute the task again, but this time we’ll keep track of the task:

>>> result = MyTask.delay("do_something", some_arg="foo bar baz")
>>> result.ready() # returns True if the task has finished processing.
False
>>> result.result # task is not ready, so no return value yet.
None
>>> result.get()   # Waits until the task is done and return the retval.
42
>>> result.result
42
>>> result.successful() # returns True if the task didn't end in failure.
True

If the task raises an exception, the result.success() will be False, and result.result will contain the exception instance raised.

Auto-discovery of tasks¶

celery has an auto-discovery feature like the Django Admin, that automatically loads any tasks.py module in the applications listed in settings.INSTALLED_APPS. This autodiscovery is used by the celery worker to find registered tasks for your Django project.

Periodic Tasks¶

Periodic tasks are tasks that are run every n seconds. Here’s an example of a periodic task:

>>> from celery.task import PeriodicTask
>>> from celery.registry import tasks
>>> from datetime import timedelta
>>> class MyPeriodicTask(PeriodicTask):
...     name = "foo.my-periodic-task"
...     run_every = timedelta(seconds=30)
...
...     def run(self, **kwargs):
...         logger = self.get_logger(**kwargs)
...         logger.info("Running periodic task!")
...
>>> tasks.register(MyPeriodicTask)

Note: Periodic tasks does not support arguments, as this doesn’t really make sense.

Mailing list¶

For discussions about the usage, development, and future of celery, please join the celery-users mailing list.

IRC¶

Come chat with us on IRC. The #celery channel is located at the Freenode network.