README 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371
  1. ===================================================
  2. celery - Distributed Task Queue for Django/Python
  3. ===================================================
  4. :Version: 0.6.0
  5. Introduction
  6. ============
  7. **NOTE:** See the FAQ for information about using celery outside of Django.
  8. ``celery`` is a distributed task queue framework for Django/Python.
  9. It is used for executing tasks *asynchronously*, routed to one or more
  10. worker servers, running concurrently using multiprocessing.
  11. It is designed to solve certain problems related to running websites
  12. demanding high-availability and performance.
  13. It is perfect for filling caches, posting updates to twitter, mass
  14. downloading data like syndication feeds or web scraping. Use-cases are
  15. plentiful. Implementing these features asynchronously using ``celery`` is
  16. easy and fun, and the performance improvements can make it more than
  17. worthwhile.
  18. Overview
  19. ========
  20. This is a high level overview of the architecture.
  21. .. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg
  22. The broker is an AMQP server pushing tasks to the worker servers.
  23. A worker server is a networked machine running ``celeryd``. This can be one or
  24. more machines, depending on the workload. See `A look inside the worker`_ to
  25. see how the worker server works.
  26. The result of the task can be stored for later retrieval (called its
  27. "tombstone").
  28. Features
  29. ========
  30. * Uses AMQP messaging (RabbitMQ, ZeroMQ) to route tasks to the
  31. worker servers.
  32. * You can run as many worker servers as you want, and still
  33. be *guaranteed that the task is only executed once.*
  34. * Tasks are executed *concurrently* using the Python 2.6
  35. ``multiprocessing`` module (also available as a back-port
  36. to older python versions)
  37. * Supports *periodic tasks*, which makes it a (better) replacement
  38. for cronjobs.
  39. * When a task has been executed, the return value can be stored using
  40. either a MySQL/Oracle/PostgreSQL/SQLite database, Memcached,
  41. or Tokyo Tyrant back-end.
  42. * If the task raises an exception, the exception instance is stored,
  43. instead of the return value.
  44. * All tasks has a Universally Unique Identifier (UUID), which is the
  45. task id, used for querying task status and return values.
  46. * Supports *task-sets*, which is a task consisting of several sub-tasks.
  47. You can find out how many, or if all of the sub-tasks has been executed.
  48. Excellent for progress-bar like functionality.
  49. * Has a ``map`` like function that uses tasks, called ``dmap``.
  50. * However, you rarely want to wait for these results in a web-environment.
  51. You'd rather want to use Ajax to poll the task status, which is
  52. available from a URL like ``celery/<task_id>/status/``. This view
  53. returns a JSON-serialized data structure containing the task status,
  54. and the return value if completed, or exception on failure.
  55. * The worker can collect statistics, like, how many tasks has been
  56. executed by type, and the time it took to process them. Very useful
  57. for monitoring and profiling.
  58. * Pool workers are supervised, so if for some reason a worker crashes
  59. it is automatically replaced by a new worker.
  60. API Reference Documentation
  61. ===========================
  62. The `API Reference`_ is hosted at Github
  63. (http://ask.github.com/celery)
  64. .. _`API Reference`: http://ask.github.com/celery/
  65. Installation
  66. =============
  67. You can install ``celery`` either via the Python Package Index (PyPI)
  68. or from source.
  69. To install using ``pip``,::
  70. $ pip install celery
  71. To install using ``easy_install``,::
  72. $ easy_install celery
  73. Downloading and installing from source
  74. --------------------------------------
  75. Download the latest version of ``celery`` from
  76. http://pypi.python.org/pypi/celery/
  77. You can install it by doing the following,::
  78. $ tar xvfz celery-0.0.0.tar.gz
  79. $ cd celery-0.0.0
  80. $ python setup.py build
  81. # python setup.py install # as root
  82. Using the development version
  83. ------------------------------
  84. You can clone the repository by doing the following::
  85. $ git clone git://github.com/ask/celery.git celery
  86. Usage
  87. =====
  88. Installing RabbitMQ
  89. -------------------
  90. See `Installing RabbitMQ`_ over at RabbitMQ's website. For Mac OS X
  91. see `Installing RabbitMQ on OS X`_.
  92. .. _`Installing RabbitMQ`: http://www.rabbitmq.com/install.html
  93. .. _`Installing RabbitMQ on OS X`:
  94. http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/
  95. Setting up RabbitMQ
  96. -------------------
  97. To use celery we need to create a RabbitMQ user, a virtual host and
  98. allow that user access to that virtual host::
  99. $ rabbitmqctl add_user myuser mypassword
  100. $ rabbitmqctl add_vhost myvhost
  101. From RabbitMQ version 1.6.0 and onward you have to use the new ACL features
  102. to allow access::
  103. $ rabbitmqctl set_permissions -p myvhost myuser "" ".*" ".*"
  104. See the RabbitMQ `Admin Guide`_ for more information about `access control`_.
  105. .. _`Admin Guide`: http://www.rabbitmq.com/admin-guide.html
  106. .. _`access control`: http://www.rabbitmq.com/admin-guide.html#access-control
  107. If you are still using version 1.5.0 or below, please use ``map_user_vhost``::
  108. $ rabbitmqctl map_user_vhost myuser myvhost
  109. Configuring your Django project to use Celery
  110. ---------------------------------------------
  111. You only need three simple steps to use celery with your Django project.
  112. 1. Add ``celery`` to ``INSTALLED_APPS``.
  113. 2. Create the celery database tables::
  114. $ python manage.py syncdb
  115. 3. Configure celery to use the AMQP user and virtual host we created
  116. before, by adding the following to your ``settings.py``::
  117. AMQP_SERVER = "localhost"
  118. AMQP_PORT = 5672
  119. AMQP_USER = "myuser"
  120. AMQP_PASSWORD = "mypassword"
  121. AMQP_VHOST = "myvhost"
  122. That's it.
  123. There are more options available, like how many processes you want to process
  124. work in parallel (the ``CELERY_CONCURRENCY`` setting), and the backend used
  125. for storing task statuses. But for now, this should do. For all of the options
  126. available, please consult the `API Reference`_
  127. **Note**: If you're using SQLite as the Django database back-end,
  128. ``celeryd`` will only be able to process one task at a time, this is
  129. because SQLite doesn't allow concurrent writes.
  130. Running the celery worker server
  131. --------------------------------
  132. To test this we'll be running the worker server in the foreground, so we can
  133. see what's going on without consulting the logfile::
  134. $ python manage.py celeryd
  135. However, in production you probably want to run the worker in the
  136. background, as a daemon::
  137. $ python manage.py celeryd --detach
  138. For a complete listing of the command line arguments available, with a short
  139. description, you can use the help command::
  140. $ python manage.py help celeryd
  141. Defining and executing tasks
  142. ----------------------------
  143. **Please note** All of these tasks has to be stored in a real module, they can't
  144. be defined in the python shell or ipython/bpython. This is because the celery
  145. worker server needs access to the task function to be able to run it.
  146. So while it looks like we use the python shell to define the tasks in these
  147. examples, you can't do it this way. Put them in the ``tasks`` module of your
  148. Django application. The worker server will automatically load any ``tasks.py``
  149. file for all of the applications listed in ``settings.INSTALLED_APPS``.
  150. Executing tasks using ``delay`` and ``apply_async`` can be done from the
  151. python shell, but keep in mind that since arguments are pickled, you can't
  152. use custom classes defined in the shell session.
  153. While you can use regular functions, the recommended way is to define
  154. a task class. This way you can cleanly upgrade the task to use the more
  155. advanced features of celery later.
  156. This is a task that basically does nothing but take some arguments,
  157. and return a value:
  158. >>> from celery.task import Task
  159. >>> from celery.registry import tasks
  160. >>> class MyTask(Task):
  161. ... def run(self, some_arg, **kwargs):
  162. ... logger = self.get_logger(**kwargs)
  163. ... logger.info("Did something: %s" % some_arg)
  164. ... return 42
  165. >>> tasks.register(MyTask)
  166. Now if we want to execute this task, we can use the ``delay`` method of the
  167. task class (this is a handy shortcut to the ``apply_async`` method which gives
  168. you greater control of the task execution).
  169. >>> from myapp.tasks import MyTask
  170. >>> MyTask.delay(some_arg="foo")
  171. At this point, the task has been sent to the message broker. The message
  172. broker will hold on to the task until a celery worker server has successfully
  173. picked it up.
  174. *Note* If everything is just hanging when you execute ``delay``, please check
  175. that RabbitMQ is running, and that the user/password has access to the virtual
  176. host you configured earlier.
  177. Right now we have to check the celery worker logfiles to know what happened with
  178. the task. This is because we didn't keep the ``AsyncResult`` object returned
  179. by ``delay``.
  180. The ``AsyncResult`` lets us find the state of the task, wait for the task to
  181. finish and get its return value (or exception if the task failed).
  182. So, let's execute the task again, but this time we'll keep track of the task:
  183. >>> result = MyTask.delay("do_something", some_arg="foo bar baz")
  184. >>> result.ready() # returns True if the task has finished processing.
  185. False
  186. >>> result.result # task is not ready, so no return value yet.
  187. None
  188. >>> result.get() # Waits until the task is done and return the retval.
  189. 42
  190. >>> result.result
  191. 42
  192. >>> result.successful() # returns True if the task didn't end in failure.
  193. True
  194. If the task raises an exception, the ``result.success()`` will be ``False``,
  195. and ``result.result`` will contain the exception instance raised.
  196. Auto-discovery of tasks
  197. -----------------------
  198. ``celery`` has an auto-discovery feature like the Django Admin, that
  199. automatically loads any ``tasks.py`` module in the applications listed
  200. in ``settings.INSTALLED_APPS``. This autodiscovery is used by the celery
  201. worker to find registered tasks for your Django project.
  202. Periodic Tasks
  203. ---------------
  204. Periodic tasks are tasks that are run every ``n`` seconds.
  205. Here's an example of a periodic task:
  206. >>> from celery.task import PeriodicTask
  207. >>> from celery.registry import tasks
  208. >>> from datetime import timedelta
  209. >>> class MyPeriodicTask(PeriodicTask):
  210. ... run_every = timedelta(seconds=30)
  211. ...
  212. ... def run(self, **kwargs):
  213. ... logger = self.get_logger(**kwargs)
  214. ... logger.info("Running periodic task!")
  215. ...
  216. >>> tasks.register(MyPeriodicTask)
  217. **Note:** Periodic tasks does not support arguments, as this doesn't
  218. really make sense.
  219. A look inside the worker
  220. ========================
  221. .. image:: http://cloud.github.com/downloads/ask/celery/InsideTheWorker-v2.jpg
  222. Getting Help
  223. ============
  224. Mailing list
  225. ------------
  226. For discussions about the usage, development, and future of celery,
  227. please join the `celery-users`_ mailing list.
  228. .. _`celery-users`: http://groups.google.com/group/celery-users/
  229. IRC
  230. ---
  231. Come chat with us on IRC. The `#celery`_ channel is located at the `Freenode`_
  232. network.
  233. .. _`#celery`: irc://irc.freenode.net/celery
  234. .. _`Freenode`: http://freenode.net
  235. Bug tracker
  236. ===========
  237. If you have any suggestions, bug reports or annoyances please report them
  238. to our issue tracker at http://github.com/ask/celery/issues/
  239. Contributing
  240. ============
  241. Development of ``celery`` happens at Github: http://github.com/ask/celery
  242. You are highly encouraged to participate in the development
  243. of ``celery``. If you don't like Github (for some reason) you're welcome
  244. to send regular patches.
  245. License
  246. =======
  247. This software is licensed under the ``New BSD License``. See the ``LICENSE``
  248. file in the top distribution directory for the full license text.
  249. .. # vim: syntax=rst expandtab tabstop=4 shiftwidth=4 shiftround