introduction.txt 13 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301
  1. :Version: 2.6.0rc3
  2. :Web: http://celeryproject.org/
  3. :Download: http://pypi.python.org/pypi/celery/
  4. :Source: http://github.com/ask/celery/
  5. :Keywords: task queue, job queue, asynchronous, rabbitmq, amqp, redis,
  6. python, webhooks, queue, distributed
  7. --
  8. .. contents::
  9. :local:
  10. .. _celery-synopsis:
  11. Synopsis
  12. ========
  13. Celery is an open source asynchronous task queue/job queue based on
  14. distributed message passing. It is focused on real-time operation,
  15. but supports scheduling as well.
  16. The execution units, called tasks, are executed concurrently on one or
  17. more worker nodes using multiprocessing, `Eventlet`_ or `gevent`_. Tasks can
  18. execute asynchronously (in the background) or synchronously
  19. (wait until ready).
  20. Celery is used in production systems to process millions of tasks a day.
  21. Celery is written in Python, but the protocol can be implemented in any
  22. language. It can also `operate with other languages using webhooks`_.
  23. There's also `RCelery` for the Ruby programming language, and a `PHP client`.
  24. The recommended message broker is `RabbitMQ`_, but support for
  25. `Redis`_, `MongoDB`_, `Beanstalk`_, `Amazon SQS`_, `CouchDB`_ and
  26. databases (using `SQLAlchemy`_ or the `Django ORM`_) is also available.
  27. Celery is easy to integrate with web frameworks, some of which even have
  28. integration packages:
  29. +--------------------+------------------------+
  30. | `Django`_ | `django-celery`_ |
  31. +--------------------+------------------------+
  32. | `Pyramid`_ | `pyramid_celery`_ |
  33. +--------------------+------------------------+
  34. | `Pylons`_ | `celery-pylons`_ |
  35. +--------------------+------------------------+
  36. | `Flask`_ | `flask-celery`_ |
  37. +--------------------+------------------------+
  38. | `web2py`_ | `web2py-celery`_ |
  39. +--------------------+------------------------+
  40. | `Tornado`_ | `tornado-celery`_ |
  41. +--------------------+------------------------+
  42. .. _`RCelery`: http://leapfrogdevelopment.github.com/rcelery/
  43. .. _`PHP client`: https://github.com/gjedeer/celery-php
  44. .. _`RabbitMQ`: http://www.rabbitmq.com/
  45. .. _`Redis`: http://code.google.com/p/redis/
  46. .. _`SQLAlchemy`: http://www.sqlalchemy.org/
  47. .. _`Django`: http://djangoproject.com/
  48. .. _`Django ORM`: http://djangoproject.com/
  49. .. _`Eventlet`: http://eventlet.net/
  50. .. _`gevent`: http://gevent.org/
  51. .. _`Beanstalk`: http://kr.github.com/beanstalkd/
  52. .. _`MongoDB`: http://mongodb.org/
  53. .. _`CouchDB`: http://couchdb.apache.org/
  54. .. _`Amazon SQS`: http://aws.amazon.com/sqs/
  55. .. _`Pylons`: http://pylonshq.com/
  56. .. _`Flask`: http://flask.pocoo.org/
  57. .. _`web2py`: http://web2py.com/
  58. .. _`Bottle`: http://bottlepy.org/
  59. .. _`Pyramid`: http://docs.pylonsproject.org/en/latest/docs/pyramid.html
  60. .. _`pyramid_celery`: http://pypi.python.org/pypi/pyramid_celery/
  61. .. _`django-celery`: http://pypi.python.org/pypi/django-celery
  62. .. _`celery-pylons`: http://pypi.python.org/pypi/celery-pylons
  63. .. _`flask-celery`: http://github.com/ask/flask-celery/
  64. .. _`web2py-celery`: http://code.google.com/p/web2py-celery/
  65. .. _`Tornado`: http://www.tornadoweb.org/
  66. .. _`tornado-celery`: http://github.com/mher/tornado-celery/
  67. .. _`operate with other languages using webhooks`:
  68. http://ask.github.com/celery/userguide/remote-tasks.html
  69. .. _`limited support`:
  70. http://kombu.readthedocs.org/en/latest/introduction.html#transport-comparison
  71. .. _celery-overview:
  72. Overview
  73. ========
  74. This is a high level overview of the architecture.
  75. .. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg
  76. The broker delivers tasks to the worker nodes.
  77. A worker node is a networked machine running `celeryd`. This can be one or
  78. more machines depending on the workload.
  79. The result of the task can be stored for later retrieval (called its
  80. "tombstone").
  81. .. _celery-example:
  82. Example
  83. =======
  84. You probably want to see some code by now, so here's an example task
  85. adding two numbers:
  86. .. code-block:: python
  87. from celery import task
  88. @task
  89. def add(x, y):
  90. return x + y
  91. You can execute the task in the background, or wait for it to finish::
  92. >>> result = add.delay(4, 4)
  93. >>> result.wait() # wait for and return the result
  94. 8
  95. Simple!
  96. .. _celery-features:
  97. Features
  98. ========
  99. +-----------------+----------------------------------------------------+
  100. | Messaging | Supported brokers include `RabbitMQ`_, `Redis`_, |
  101. | | `Beanstalk`_, `MongoDB`_, `CouchDB`_, and popular |
  102. | | SQL databases. |
  103. +-----------------+----------------------------------------------------+
  104. | Fault-tolerant | Excellent configurable error recovery when using |
  105. | | `RabbitMQ`, ensures your tasks are never lost. |
  106. +-----------------+----------------------------------------------------+
  107. | Distributed | Runs on one or more machines. Supports |
  108. | | broker `clustering`_ and `HA`_ when used in |
  109. | | combination with `RabbitMQ`_. You can set up new |
  110. | | workers without central configuration (e.g. use |
  111. | | your grandma's laptop to help if the queue is |
  112. | | temporarily congested). |
  113. +-----------------+----------------------------------------------------+
  114. | Concurrency | Concurrency is achieved by using multiprocessing, |
  115. | | `Eventlet`_, `gevent` or a mix of these. |
  116. +-----------------+----------------------------------------------------+
  117. | Scheduling | Supports recurring tasks like cron, or specifying |
  118. | | an exact date or countdown for when after the task |
  119. | | should be executed. |
  120. +-----------------+----------------------------------------------------+
  121. | Latency | Low latency means you are able to execute tasks |
  122. | | *while the user is waiting*. |
  123. +-----------------+----------------------------------------------------+
  124. | Return Values | Task return values can be saved to the selected |
  125. | | result store backend. You can wait for the result, |
  126. | | retrieve it later, or ignore it. |
  127. +-----------------+----------------------------------------------------+
  128. | Result Stores | Database, `MongoDB`_, `Redis`_, `Tokyo Tyrant`, |
  129. | | `Cassandra`, or `AMQP`_ (message notification). |
  130. +-----------------+----------------------------------------------------+
  131. | Webhooks | Your tasks can also be HTTP callbacks, enabling |
  132. | | cross-language communication. |
  133. +-----------------+----------------------------------------------------+
  134. | Rate limiting | Supports rate limiting by using the token bucket |
  135. | | algorithm, which accounts for bursts of traffic. |
  136. | | Rate limits can be set for each task type, or |
  137. | | globally for all. |
  138. +-----------------+----------------------------------------------------+
  139. | Routing | Using AMQP's flexible routing model you can route |
  140. | | tasks to different workers, or select different |
  141. | | message topologies, by configuration or even at |
  142. | | runtime. |
  143. +-----------------+----------------------------------------------------+
  144. | Remote-control | Worker nodes can be controlled from remote by |
  145. | | using broadcast messaging. A range of built-in |
  146. | | commands exist in addition to the ability to |
  147. | | easily define your own. (AMQP/Redis only) |
  148. +-----------------+----------------------------------------------------+
  149. | Monitoring | You can capture everything happening with the |
  150. | | workers in real-time by subscribing to events. |
  151. | | A real-time web monitor is in development. |
  152. +-----------------+----------------------------------------------------+
  153. | Serialization | Supports Pickle, JSON, YAML, or easily defined |
  154. | | custom schemes. One task invocation can have a |
  155. | | different scheme than another. |
  156. +-----------------+----------------------------------------------------+
  157. | Tracebacks | Errors and tracebacks are stored and can be |
  158. | | investigated after the fact. |
  159. +-----------------+----------------------------------------------------+
  160. | UUID | Every task has an UUID (Universally Unique |
  161. | | Identifier), which is the task id used to query |
  162. | | task status and return value. |
  163. +-----------------+----------------------------------------------------+
  164. | Retries | Tasks can be retried if they fail, with |
  165. | | configurable maximum number of retries, and delays |
  166. | | between each retry. |
  167. +-----------------+----------------------------------------------------+
  168. | Task Sets | A Task set is a task consisting of several |
  169. | | sub-tasks. You can find out how many, or if all |
  170. | | of the sub-tasks has been executed, and even |
  171. | | retrieve the results in order. Progress bars, |
  172. | | anyone? |
  173. +-----------------+----------------------------------------------------+
  174. | Made for Web | You can query status and results via URLs, |
  175. | | enabling the ability to poll task status using |
  176. | | Ajax. |
  177. +-----------------+----------------------------------------------------+
  178. | Error Emails | Can be configured to send emails to the |
  179. | | administrators when tasks fails. |
  180. +-----------------+----------------------------------------------------+
  181. .. _`clustering`: http://www.rabbitmq.com/clustering.html
  182. .. _`HA`: http://www.rabbitmq.com/pacemaker.html
  183. .. _`AMQP`: http://www.amqp.org/
  184. .. _`Stomp`: http://stomp.codehaus.org/
  185. .. _`Tokyo Tyrant`: http://tokyocabinet.sourceforge.net/
  186. .. _celery-documentation:
  187. Documentation
  188. =============
  189. The `latest documentation`_ with user guides, tutorials and API reference
  190. is hosted at Github.
  191. .. _`latest documentation`: http://ask.github.com/celery/
  192. .. _celery-installation:
  193. Installation
  194. ============
  195. You can install Celery either via the Python Package Index (PyPI)
  196. or from source.
  197. To install using `pip`,::
  198. $ pip install -U Celery
  199. To install using `easy_install`,::
  200. $ easy_install -U Celery
  201. Bundles
  202. -------
  203. Celery also defines a group of bundles that can be used
  204. to install Celery and the dependencies for a given feature.
  205. The following bundles are available:
  206. :`celery-with-redis`_:
  207. for using Redis as a broker.
  208. :`celery-with-mongodb`_:
  209. for using MongoDB as a broker.
  210. :`django-celery-with-redis`_:
  211. for Django, and using Redis as a broker.
  212. :`django-celery-with-mongodb`_:
  213. for Django, and using MongoDB as a broker.
  214. :`bundle-celery`_:
  215. convenience bundle installing *Celery* and related packages.
  216. .. _`celery-with-redis`:
  217. http://pypi.python.org/pypi/celery-with-redis/
  218. .. _`celery-with-mongodb`:
  219. http://pypi.python.org/pypi/celery-with-mongdb/
  220. .. _`django-celery-with-redis`:
  221. http://pypi.python.org/pypi/django-celery-with-redis/
  222. .. _`django-celery-with-mongodb`:
  223. http://pypi.python.org/pypi/django-celery-with-mongdb/
  224. .. _`bundle-celery`:
  225. http://pypi.python.org/pypi/bundle-celery/
  226. .. _celery-installing-from-source:
  227. Downloading and installing from source
  228. --------------------------------------
  229. Download the latest version of Celery from
  230. http://pypi.python.org/pypi/celery/
  231. You can install it by doing the following,::
  232. $ tar xvfz celery-0.0.0.tar.gz
  233. $ cd celery-0.0.0
  234. $ python setup.py build
  235. # python setup.py install # as root
  236. .. _celery-installing-from-git:
  237. Using the development version
  238. -----------------------------
  239. You can clone the repository by doing the following::
  240. $ git clone git://github.com/ask/celery.git