README.rst 11 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426
  1. =================================
  2. celery - Distributed Task Queue
  3. =================================
  4. .. image:: http://cloud.github.com/downloads/celery/celery/celery_128.png
  5. :Version: 3.1.26 (Cipater)
  6. :Web: http://celeryproject.org/
  7. :Download: http://pypi.python.org/pypi/celery/
  8. :Source: http://github.com/celery/celery/
  9. :Keywords: task queue, job queue, asynchronous, async, rabbitmq, amqp, redis,
  10. python, webhooks, queue, distributed
  11. --
  12. What is a Task Queue?
  13. =====================
  14. Task queues are used as a mechanism to distribute work across threads or
  15. machines.
  16. A task queue's input is a unit of work, called a task, dedicated worker
  17. processes then constantly monitor the queue for new work to perform.
  18. Celery communicates via messages, usually using a broker
  19. to mediate between clients and workers. To initiate a task a client puts a
  20. message on the queue, the broker then delivers the message to a worker.
  21. A Celery system can consist of multiple workers and brokers, giving way
  22. to high availability and horizontal scaling.
  23. Celery is a library written in Python, but the protocol can be implemented in
  24. any language. So far there's RCelery_ for the Ruby programming language, and a
  25. `PHP client`, but language interoperability can also be achieved
  26. by using webhooks.
  27. .. _RCelery: http://leapfrogdevelopment.github.com/rcelery/
  28. .. _`PHP client`: https://github.com/gjedeer/celery-php
  29. .. _`using webhooks`:
  30. http://docs.celeryproject.org/en/latest/userguide/remote-tasks.html
  31. What do I need?
  32. ===============
  33. Celery version 3.0 runs on,
  34. - Python (2.5, 2.6, 2.7, 3.2, 3.3)
  35. - PyPy (1.8, 1.9)
  36. - Jython (2.5, 2.7).
  37. This is the last version to support Python 2.5,
  38. and from Celery 3.1, Python 2.6 or later is required.
  39. The last version to support Python 2.4 was Celery series 2.2.
  40. *Celery* is usually used with a message broker to send and receive messages.
  41. The RabbitMQ, Redis transports are feature complete,
  42. but there's also experimental support for a myriad of other solutions, including
  43. using SQLite for local development.
  44. *Celery* can run on a single machine, on multiple machines, or even
  45. across datacenters.
  46. Get Started
  47. ===========
  48. If this is the first time you're trying to use Celery, or you are
  49. new to Celery 3.0 coming from previous versions then you should read our
  50. getting started tutorials:
  51. - `First steps with Celery`_
  52. Tutorial teaching you the bare minimum needed to get started with Celery.
  53. - `Next steps`_
  54. A more complete overview, showing more features.
  55. .. _`First steps with Celery`:
  56. http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html
  57. .. _`Next steps`:
  58. http://docs.celeryproject.org/en/latest/getting-started/next-steps.html
  59. Celery is...
  60. ============
  61. - **Simple**
  62. Celery is easy to use and maintain, and does *not need configuration files*.
  63. It has an active, friendly community you can talk to for support,
  64. including a `mailing-list`_ and and an IRC channel.
  65. Here's one of the simplest applications you can make::
  66. from celery import Celery
  67. app = Celery('hello', broker='amqp://guest@localhost//')
  68. @app.task
  69. def hello():
  70. return 'hello world'
  71. - **Highly Available**
  72. Workers and clients will automatically retry in the event
  73. of connection loss or failure, and some brokers support
  74. HA in way of *Master/Master* or *Master/Slave* replication.
  75. - **Fast**
  76. A single Celery process can process millions of tasks a minute,
  77. with sub-millisecond round-trip latency (using RabbitMQ,
  78. py-librabbitmq, and optimized settings).
  79. - **Flexible**
  80. Almost every part of *Celery* can be extended or used on its own,
  81. Custom pool implementations, serializers, compression schemes, logging,
  82. schedulers, consumers, producers, autoscalers, broker transports and much more.
  83. It supports...
  84. ==============
  85. - **Message Transports**
  86. - RabbitMQ_, Redis_,
  87. - MongoDB_ (experimental), Amazon SQS (experimental),
  88. - CouchDB_ (experimental), SQLAlchemy_ (experimental),
  89. - Django ORM (experimental), `IronMQ`_
  90. - and more...
  91. - **Concurrency**
  92. - Prefork, Eventlet_, gevent_, threads/single threaded
  93. - **Result Stores**
  94. - AMQP, Redis
  95. - memcached, MongoDB
  96. - SQLAlchemy, Django ORM
  97. - Apache Cassandra, IronCache
  98. - **Serialization**
  99. - *pickle*, *json*, *yaml*, *msgpack*.
  100. - *zlib*, *bzip2* compression.
  101. - Cryptographic message signing.
  102. .. _`Eventlet`: http://eventlet.net/
  103. .. _`gevent`: http://gevent.org/
  104. .. _RabbitMQ: http://rabbitmq.com
  105. .. _Redis: http://redis.io
  106. .. _MongoDB: http://mongodb.org
  107. .. _Beanstalk: http://kr.github.com/beanstalkd
  108. .. _CouchDB: http://couchdb.apache.org
  109. .. _SQLAlchemy: http://sqlalchemy.org
  110. .. _`IronMQ`: http://iron.io
  111. Framework Integration
  112. =====================
  113. Celery is easy to integrate with web frameworks, some of which even have
  114. integration packages:
  115. +--------------------+------------------------+
  116. | `Django`_ | not needed |
  117. +--------------------+------------------------+
  118. | `Pyramid`_ | `pyramid_celery`_ |
  119. +--------------------+------------------------+
  120. | `Pylons`_ | `celery-pylons`_ |
  121. +--------------------+------------------------+
  122. | `Flask`_ | not needed |
  123. +--------------------+------------------------+
  124. | `web2py`_ | `web2py-celery`_ |
  125. +--------------------+------------------------+
  126. | `Tornado`_ | `tornado-celery`_ |
  127. +--------------------+------------------------+
  128. The integration packages are not strictly necessary, but they can make
  129. development easier, and sometimes they add important hooks like closing
  130. database connections at ``fork``.
  131. .. _`Django`: http://djangoproject.com/
  132. .. _`Pylons`: http://pylonsproject.org/
  133. .. _`Flask`: http://flask.pocoo.org/
  134. .. _`web2py`: http://web2py.com/
  135. .. _`Bottle`: http://bottlepy.org/
  136. .. _`Pyramid`: http://docs.pylonsproject.org/en/latest/docs/pyramid.html
  137. .. _`pyramid_celery`: http://pypi.python.org/pypi/pyramid_celery/
  138. .. _`django-celery`: http://pypi.python.org/pypi/django-celery
  139. .. _`celery-pylons`: http://pypi.python.org/pypi/celery-pylons
  140. .. _`web2py-celery`: http://code.google.com/p/web2py-celery/
  141. .. _`Tornado`: http://www.tornadoweb.org/
  142. .. _`tornado-celery`: http://github.com/mher/tornado-celery/
  143. .. _celery-documentation:
  144. Documentation
  145. =============
  146. The `latest documentation`_ with user guides, tutorials and API reference
  147. is hosted at Read The Docs.
  148. .. _`latest documentation`: http://docs.celeryproject.org/en/latest/
  149. .. _celery-installation:
  150. Installation
  151. ============
  152. You can install Celery either via the Python Package Index (PyPI)
  153. or from source.
  154. To install using `pip`,::
  155. $ pip install -U Celery
  156. To install using `easy_install`,::
  157. $ easy_install -U Celery
  158. .. _bundles:
  159. Bundles
  160. -------
  161. Celery also defines a group of bundles that can be used
  162. to install Celery and the dependencies for a given feature.
  163. You can specify these in your requirements or on the ``pip`` comand-line
  164. by using brackets. Multiple bundles can be specified by separating them by
  165. commas.
  166. ::
  167. $ pip install "celery[librabbitmq]"
  168. $ pip install "celery[librabbitmq,redis,auth,msgpack]"
  169. The following bundles are available:
  170. Serializers
  171. ~~~~~~~~~~~
  172. :celery[auth]:
  173. for using the auth serializer.
  174. :celery[msgpack]:
  175. for using the msgpack serializer.
  176. :celery[yaml]:
  177. for using the yaml serializer.
  178. Concurrency
  179. ~~~~~~~~~~~
  180. :celery[eventlet]:
  181. for using the eventlet pool.
  182. :celery[gevent]:
  183. for using the gevent pool.
  184. :celery[threads]:
  185. for using the thread pool.
  186. Transports and Backends
  187. ~~~~~~~~~~~~~~~~~~~~~~~
  188. :celery[librabbitmq]:
  189. for using the librabbitmq C library.
  190. :celery[redis]:
  191. for using Redis as a message transport or as a result backend.
  192. :celery[mongodb]:
  193. for using MongoDB as a message transport (*experimental*),
  194. or as a result backend (*supported*).
  195. :celery[sqs]:
  196. for using Amazon SQS as a message transport (*experimental*).
  197. :celery[memcache]:
  198. for using memcached as a result backend.
  199. :celery[cassandra]:
  200. for using Apache Cassandra as a result backend.
  201. :celery[couchdb]:
  202. for using CouchDB as a message transport (*experimental*).
  203. :celery[couchbase]:
  204. for using CouchBase as a result backend.
  205. :celery[beanstalk]:
  206. for using Beanstalk as a message transport (*experimental*).
  207. :celery[zookeeper]:
  208. for using Zookeeper as a message transport.
  209. :celery[zeromq]:
  210. for using ZeroMQ as a message transport (*experimental*).
  211. :celery[sqlalchemy]:
  212. for using SQLAlchemy as a message transport (*experimental*),
  213. or as a result backend (*supported*).
  214. :celery[pyro]:
  215. for using the Pyro4 message transport (*experimental*).
  216. :celery[slmq]:
  217. for using the SoftLayer Message Queue transport (*experimental*).
  218. .. _celery-installing-from-source:
  219. Downloading and installing from source
  220. --------------------------------------
  221. Download the latest version of Celery from
  222. http://pypi.python.org/pypi/celery/
  223. You can install it by doing the following,::
  224. $ tar xvfz celery-0.0.0.tar.gz
  225. $ cd celery-0.0.0
  226. $ python setup.py build
  227. # python setup.py install
  228. The last command must be executed as a privileged user if
  229. you are not currently using a virtualenv.
  230. .. _celery-installing-from-git:
  231. Using the development version
  232. -----------------------------
  233. With pip
  234. ~~~~~~~~
  235. The Celery development version also requires the development
  236. versions of ``kombu``, ``amqp`` and ``billiard``.
  237. You can install the latest snapshot of these using the following
  238. pip commands::
  239. $ pip install https://github.com/celery/celery/zipball/master#egg=celery
  240. $ pip install https://github.com/celery/billiard/zipball/master#egg=billiard
  241. $ pip install https://github.com/celery/py-amqp/zipball/master#egg=amqp
  242. $ pip install https://github.com/celery/kombu/zipball/master#egg=kombu
  243. With git
  244. ~~~~~~~~
  245. Please the Contributing section.
  246. .. _getting-help:
  247. Getting Help
  248. ============
  249. .. _mailing-list:
  250. Mailing list
  251. ------------
  252. For discussions about the usage, development, and future of celery,
  253. please join the `celery-users`_ mailing list.
  254. .. _`celery-users`: http://groups.google.com/group/celery-users/
  255. .. _irc-channel:
  256. IRC
  257. ---
  258. Come chat with us on IRC. The **#celery** channel is located at the `Freenode`_
  259. network.
  260. .. _`Freenode`: http://freenode.net
  261. .. _bug-tracker:
  262. Bug tracker
  263. ===========
  264. If you have any suggestions, bug reports or annoyances please report them
  265. to our issue tracker at http://github.com/celery/celery/issues/
  266. .. _wiki:
  267. Wiki
  268. ====
  269. http://wiki.github.com/celery/celery/
  270. .. _contributing-short:
  271. Contributing
  272. ============
  273. Development of `celery` happens at Github: http://github.com/celery/celery
  274. You are highly encouraged to participate in the development
  275. of `celery`. If you don't like Github (for some reason) you're welcome
  276. to send regular patches.
  277. Be sure to also read the `Contributing to Celery`_ section in the
  278. documentation.
  279. .. _`Contributing to Celery`:
  280. http://docs.celeryproject.org/en/master/contributing.html
  281. .. _license:
  282. License
  283. =======
  284. This software is licensed under the `New BSD License`. See the ``LICENSE``
  285. file in the top distribution directory for the full license text.
  286. .. # vim: syntax=rst expandtab tabstop=4 shiftwidth=4 shiftround
  287. .. image:: https://d2weczhvl823v0.cloudfront.net/celery/celery/trend.png
  288. :alt: Bitdeli badge
  289. :target: https://bitdeli.com/free