configuration.rst 10 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378
  1. ============================
  2. Configuration and defaults
  3. ============================
  4. This document describes the configuration options available.
  5. If you're using celery in a Django project these settings should be defined
  6. in your projects ``settings.py`` file.
  7. In a regular Python environment using the default loader you must create
  8. the ``celeryconfig.py`` module and make sure it is available on the
  9. Python path.
  10. Example configuration file
  11. ==========================
  12. This is an example configuration file to get you started,
  13. it should contain all you need to run a basic celery set-up.
  14. .. code-block:: python
  15. CELERY_BACKEND = "database"
  16. DATABASE_ENGINE = "sqlite3"
  17. DATABASE_NAME = "mydatabase.db"
  18. BROKER_HOST = "localhost"
  19. BROKER_PORT = 5672
  20. BROKER_VHOST = "/"
  21. BROKER_USER = "guest"
  22. BROKER_PASSWORD = "guest"
  23. ## If you're doing mostly I/O you can have higher concurrency,
  24. ## if mostly spending time in the CPU, try to keep it close to the
  25. ## number of CPUs on your machine.
  26. # CELERYD_CONCURRENCY = 8
  27. CELERYD_LOG_FILE = "celeryd.log"
  28. CELERYD_PID_FILE = "celeryd.pid"
  29. CELERYD_DAEMON_LOG_LEVEL = "INFO"
  30. Concurrency settings
  31. ====================
  32. * CELERYD_CONCURRENCY
  33. The number of concurrent worker processes, executing tasks simultaneously.
  34. Defaults to the number of CPUs in the system.
  35. Task result backend settings
  36. ============================
  37. * CELERY_BACKEND
  38. The backend used to store task results (tombstones).
  39. Can be one of the following:
  40. * database (default)
  41. Use a relational database supported by the Django ORM.
  42. * cache
  43. Use `memcached`_ to store the results.
  44. * mongodb
  45. Use `MongoDB`_ to store the results.
  46. * pyredis
  47. Use `Redis`_ to store the results.
  48. * tyrant
  49. Use `Tokyo Tyrant`_ to store the results.
  50. * amqp
  51. Send results back as AMQP messages
  52. (**WARNING** While very fast, you must make sure you only
  53. try to receive the result once).
  54. .. _`memcached`: http://memcached.org
  55. .. _`MongoDB`: http://mongodb.org
  56. .. _`Redis`: http://code.google.com/p/redis/
  57. .. _`Tokyo Tyrant`: http://1978th.net/tokyotyrant/
  58. Database backend settings
  59. =========================
  60. Please see the Django ORM database settings documentation:
  61. http://docs.djangoproject.com/en/dev/ref/settings/#database-engine
  62. If you use this backend make sure to initialize the database tables
  63. after configuration. When using celery with a Django project this
  64. means executing::
  65. $ python manage.py syncdb
  66. When using celery in a regular Python environment you have to execute::
  67. $ celeryinit
  68. Example configuration
  69. ---------------------
  70. .. code-block:: python
  71. CELERY_BACKEND = "database"
  72. DATABASE_ENGINE = "mysql"
  73. DATABASE_USER = "myusername"
  74. DATABASE_PASSWORD = "mypassword"
  75. DATABASE_NAME = "mydatabase"
  76. DATABASE_HOST = "localhost"
  77. Cache backend settings
  78. ======================
  79. Please see the documentation for the Django cache framework settings:
  80. http://docs.djangoproject.com/en/dev/topics/cache/#memcached
  81. To use a custom cache backend for Celery, while using another for Django,
  82. you should use the ``CELERY_CACHE_BACKEND`` setting instead of the regular
  83. django ``CACHE_BACKEND`` setting.
  84. Example configuration
  85. ---------------------
  86. Using a single memcached server:
  87. .. code-block:: python
  88. CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
  89. Using multiple memcached servers:
  90. .. code-block:: python
  91. CELERY_BACKEND = "cache"
  92. CACHE_BACKEND = 'memcached://172.19.26.240:11211;172.19.26.242:11211/'
  93. Tokyo Tyrant backend settings
  94. =============================
  95. **NOTE** The Tokyo Tyrant backend requires the :mod:`pytyrant` library:
  96. http://pypi.python.org/pypi/pytyrant/
  97. This backend requires the following configuration directives to be set:
  98. * TT_HOST
  99. Hostname of the Tokyo Tyrant server.
  100. * TT_PORT
  101. The port the Tokyo Tyrant server is listening to.
  102. Example configuration
  103. ---------------------
  104. .. code-block:: python
  105. CELERY_BACKEND = "tyrant"
  106. TT_HOST = "localhost"
  107. TT_PORT = 1978
  108. Redis backend settings
  109. ======================
  110. **NOTE** The Redis backend requires the :mod:`redis` library:
  111. http://pypi.python.org/pypi/redis/0.5.5
  112. To install the redis package use ``pip`` or ``easy_install``::
  113. $ pip install redis
  114. This backend requires the following configuration directives to be set:
  115. * REDIS_HOST
  116. Hostname of the Redis database server. e.g. ``"localhost"``.
  117. * REDIS_PORT
  118. Port to the Redis database server. e.g. ``6379``.
  119. Also, the following optional configuration directives are available:
  120. * REDIS_DB
  121. Name of the database to use. Default is ``celery_results``.
  122. * REDIS_TIMEOUT
  123. Timeout in seconds before we give up establishing a connection
  124. to the Redis server.
  125. * REDIS_CONNECT_RETRY
  126. Retry connecting if an connection could not be established. Default is
  127. false.
  128. Example configuration
  129. ---------------------
  130. .. code-block:: python
  131. CELERY_BACKEND = "pyredis"
  132. REDIS_HOST = "localhost"
  133. REDIS_PORT = 6739
  134. REDIS_DATABASE = "celery_results"
  135. REDIS_CONNECT_RETRY=True
  136. MongoDB backend settings
  137. ========================
  138. **NOTE** The MongoDB backend requires the :mod:`pymongo` library:
  139. http://github.com/mongodb/mongo-python-driver/tree/master
  140. * CELERY_MONGODB_BACKEND_SETTINGS
  141. This is a dict supporting the following keys:
  142. * host
  143. Hostname of the MongoDB server. Defaults to "localhost".
  144. * port
  145. The port the MongoDB server is listening to. Defaults to 27017.
  146. * user
  147. Username to authenticate to the MongoDB server as (optional).
  148. * password
  149. Password to authenticate to the MongoDB server (optional).
  150. * database
  151. The database name to connect to. Defaults to "celery".
  152. * taskmeta_collection
  153. The collection name to store task metadata.
  154. Defaults to "celery_taskmeta".
  155. Example configuration
  156. ---------------------
  157. .. code-block:: python
  158. CELERY_BACKEND = "mongodb"
  159. CELERY_MONGODB_BACKEND_SETTINGS = {
  160. "host": "192.168.1.100",
  161. "port": 30000,
  162. "database": "mydb",
  163. "taskmeta_collection": "my_taskmeta_collection",
  164. }
  165. Broker settings
  166. ===============
  167. * CELERY_AMQP_EXCHANGE
  168. Name of the AMQP exchange.
  169. * CELERY_AMQP_EXCHANGE_TYPE
  170. The type of exchange. If the exchange type is ``direct``, all messages
  171. receives all tasks. However, if the exchange type is ``topic``, you can
  172. route e.g. some tasks to one server, and others to the rest.
  173. See `Exchange types and the effect of bindings`_.
  174. .. _`Exchange types and the effect of bindings`:
  175. http://bit.ly/wpamqpexchanges
  176. * CELERY_AMQP_PUBLISHER_ROUTING_KEY
  177. The default AMQP routing key used when publishing tasks.
  178. * CELERY_AMQP_CONSUMER_ROUTING_KEY
  179. The AMQP routing key used when consuming tasks.
  180. * CELERY_AMQP_CONSUMER_QUEUE
  181. The name of the AMQP queue.
  182. * CELERY_AMQP_CONSUMER_QUEUES
  183. Dictionary defining multiple AMQP queues.
  184. * CELERY_AMQP_CONNECTION_TIMEOUT
  185. The timeout in seconds before we give up establishing a connection
  186. to the AMQP server. Default is 4 seconds.
  187. * CELERY_AMQP_CONNECTION_RETRY
  188. Automatically try to re-establish the connection to the AMQP broker if
  189. it's lost.
  190. The time between retries is increased for each retry, and is
  191. not exhausted before ``CELERY_AMQP_CONNECTION_MAX_RETRIES`` is exceeded.
  192. This behaviour is on by default.
  193. * CELERY_AMQP_CONNECTION_MAX_RETRIES
  194. Maximum number of retries before we give up re-establishing a connection
  195. to the AMQP broker.
  196. If this is set to ``0`` or ``None``, we will retry forever.
  197. Default is 100 retries.
  198. Task execution settings
  199. =======================
  200. * SEND_CELERY_TASK_ERROR_EMAILS
  201. If set to ``True``, errors in tasks will be sent to admins by e-mail.
  202. If unset, it will send the e-mails if ``settings.DEBUG`` is False.
  203. * CELERY_ALWAYS_EAGER
  204. If this is ``True``, all tasks will be executed locally by blocking
  205. until it is finished. ``apply_async`` and ``Task.delay`` will return
  206. a :class:`celery.result.EagerResult` which emulates the behaviour of
  207. :class:`celery.result.AsyncResult`, except the result has already
  208. been evaluated.
  209. Tasks will never be sent to the queue, but executed locally
  210. instead.
  211. * CELERY_TASK_RESULT_EXPIRES
  212. Time (in seconds, or a :class:`datetime.timedelta` object) for when after
  213. stored task tombstones are deleted.
  214. **NOTE**: For the moment this only works for the database and MongoDB
  215. backends., except the result has already
  216. been evaluated.
  217. * CELERY_TASK_SERIALIZER
  218. A string identifying the default serialization
  219. method to use. Can be ``pickle`` (default),
  220. ``json``, ``yaml``, or any custom serialization methods that have
  221. been registered with :mod:`carrot.serialization.registry`.
  222. Default is ``pickle``.
  223. * CELERY_STORE_ERRORS_EVEN_IF_IGNORED
  224. If set, the worker stores all task errors in the result store even if
  225. ``Task.ignore_result`` is on.
  226. * CELERY_IMPORTS
  227. A sequence of modules to import when the celery daemon starts. This is
  228. useful to add tasks if you are not using django or cannot use task
  229. autodiscovery.
  230. Logging settings
  231. ================
  232. * CELERYD_LOG_FILE
  233. The default filename the worker daemon logs messages to, can be
  234. overridden using the `--logfile`` option to ``celeryd``.
  235. The default is to log using ``stderr`` if running in the foreground,
  236. when running in the background, detached as a daemon, the default
  237. logfile is ``celeryd.log``.
  238. * CELERYD_DAEMON_LOG_LEVEL
  239. Worker log level, can be any of ``DEBUG``, ``INFO``, ``WARNING``,
  240. ``ERROR``, ``CRITICAL``, or ``FATAL``.
  241. See the :mod:`logging` module for more information.
  242. * CELERYD_DAEMON_LOG_FORMAT
  243. The format to use for log messages. Can be overridden using
  244. the ``--loglevel`` option to ``celeryd``.
  245. Default is ``[%(asctime)s: %(levelname)s/%(processName)s] %(message)s``
  246. See the Python :mod:`logging` module for more information about log
  247. formats.
  248. Process settings
  249. ================
  250. * CELERYD_PID_FILE
  251. Full path to the daemon pid file. Default is ``celeryd.pid``.
  252. Can be overridden using the ``--pidfile`` option to ``celeryd``.