introduction.txt 5.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165
  1. :Version: 1.0.0-pre1
  2. :Keywords: task queue, job queue, asynchronous, rabbitmq, amqp, redis.
  3. --
  4. Celery is a task queue/job queue based on distributed message passing.
  5. It is focused on real-time operation, but has support for scheduling as well.
  6. The execution units, called tasks, are executed concurrently on one or more
  7. worker servers, asynchronously (in the background) or synchronously
  8. (wait until ready).
  9. Celery is already used in production to process millions of tasks a day.
  10. It was first created for Django, but is now usable from Python as well.
  11. It can also `operate with other languages via HTTP+JSON`_.
  12. .. _`operate with other languages via HTTP+JSON`: http://bit.ly/CgXSc
  13. Overview
  14. ========
  15. This is a high level overview of the architecture.
  16. .. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg
  17. The broker pushes tasks to the worker servers.
  18. A worker server is a networked machine running ``celeryd``. This can be one or
  19. more machines, depending on the workload.
  20. The result of the task can be stored for later retrieval (called its
  21. "tombstone").
  22. Example
  23. =======
  24. You probably want to see some code by now, so I'll give you an example task
  25. adding two numbers:
  26. .. code-block:: python
  27. from celery.decorators import task
  28. @task
  29. def add(x, y):
  30. return x + y
  31. You can execute the task in the background, or wait for it to finish::
  32. >>> result = add.delay(4, 4)
  33. >>> result.wait() # wait for and return the result
  34. 8
  35. Simple!
  36. Features
  37. ========
  38. * Uses messaging (AMQP: RabbitMQ, ZeroMQ, Qpid) to route tasks to the
  39. worker servers. Experimental support for STOMP (ActiveMQ) is also
  40. available. For simple setups it's also possible to use Redis or an
  41. SQL database as the message queue.
  42. * You can run as many worker servers as you want, and still
  43. be *guaranteed that the task is only executed once.*
  44. * Tasks are executed *concurrently* using the Python 2.6
  45. :mod:`multiprocessing` module (also available as a back-port
  46. to older python versions)
  47. * Supports *periodic tasks*, which makes it a (better) replacement
  48. for cronjobs.
  49. * When a task has been executed, the return value can be stored using
  50. either a MySQL/Oracle/PostgreSQL/SQLite database, Memcached,
  51. `MongoDB`_, `Redis`_ or `Tokyo Tyrant`_ back-end. For high-performance
  52. you can also use AMQP messages to publish results.
  53. * Supports calling tasks over HTTP to support multiple programming
  54. languages and systems.
  55. * Supports several serialization schemes, like pickle, json, yaml and
  56. supports registering custom encodings .
  57. * If the task raises an exception, the exception instance is stored,
  58. instead of the return value, and it's possible to inspect the traceback
  59. after the fact.
  60. * All tasks has a Universally Unique Identifier (UUID), which is the
  61. task id, used for querying task status and return values.
  62. * Tasks can be retried if they fail, with a configurable maximum number
  63. of retries.
  64. * Tasks can be configured to run at a specific time and date in the
  65. future (ETA) or you can set a countdown in seconds for when the
  66. task should be executed.
  67. * Supports *task-sets*, which is a task consisting of several sub-tasks.
  68. You can find out how many, or if all of the sub-tasks has been executed.
  69. Excellent for progress-bar like functionality.
  70. * However, you rarely want to wait for these results in a web-environment.
  71. You'd rather want to use Ajax to poll the task status, which is
  72. available from a URL like ``celery/<task_id>/status/``. This view
  73. returns a JSON-serialized data structure containing the task status,
  74. and the return value if completed, or exception on failure.
  75. * Pool workers are supervised, so if for some reason a worker crashes
  76. it is automatically replaced by a new worker.
  77. * Can be configured to send e-mails to the administrators when a task
  78. fails.
  79. .. _`MongoDB`: http://www.mongodb.org/
  80. .. _`Redis`: http://code.google.com/p/redis/
  81. .. _`Tokyo Tyrant`: http://tokyocabinet.sourceforge.net/
  82. Documentation
  83. =============
  84. The `latest documentation`_ with user guides, tutorials and API reference
  85. is hosted at Github.
  86. .. _`latest documentation`: http://ask.github.com/celery/
  87. Installation
  88. =============
  89. You can install ``celery`` either via the Python Package Index (PyPI)
  90. or from source.
  91. To install using ``pip``,::
  92. $ pip install celery
  93. To install using ``easy_install``,::
  94. $ easy_install celery
  95. Downloading and installing from source
  96. --------------------------------------
  97. Download the latest version of ``celery`` from
  98. http://pypi.python.org/pypi/celery/
  99. You can install it by doing the following,::
  100. $ tar xvfz celery-0.0.0.tar.gz
  101. $ cd celery-0.0.0
  102. $ python setup.py build
  103. # python setup.py install # as root
  104. Using the development version
  105. ------------------------------
  106. You can clone the repository by doing the following::
  107. $ git clone git://github.com/ask/celery.git
  108. A look inside the components
  109. ============================
  110. .. image:: http://cloud.github.com/downloads/ask/celery/Celery1.0-inside-worker.jpg