configuration.rst 32 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349
  1. .. _configuration:
  2. ============================
  3. Configuration and defaults
  4. ============================
  5. This document describes the configuration options available.
  6. If you're using the default loader, you must create the :file:`celeryconfig.py`
  7. module and make sure it is available on the Python path.
  8. .. contents::
  9. :local:
  10. :depth: 2
  11. .. _conf-example:
  12. Example configuration file
  13. ==========================
  14. This is an example configuration file to get you started.
  15. It should contain all you need to run a basic Celery set-up.
  16. .. code-block:: python
  17. # List of modules to import when celery starts.
  18. CELERY_IMPORTS = ("myapp.tasks", )
  19. ## Result store settings.
  20. CELERY_RESULT_BACKEND = "database"
  21. CELERY_RESULT_DBURI = "sqlite:///mydatabase.db"
  22. ## Broker settings.
  23. BROKER_HOST = "localhost"
  24. BROKER_PORT = 5672
  25. BROKER_VHOST = "/"
  26. BROKER_USER = "guest"
  27. BROKER_PASSWORD = "guest"
  28. ## Worker settings
  29. ## If you're doing mostly I/O you can have more processes,
  30. ## but if mostly spending CPU, try to keep it close to the
  31. ## number of CPUs on your machine. If not set, the number of CPUs/cores
  32. ## available will be used.
  33. CELERYD_CONCURRENCY = 10
  34. # CELERYD_LOG_FILE = "celeryd.log"
  35. # CELERYD_LOG_LEVEL = "INFO"
  36. Configuration Directives
  37. ========================
  38. .. _conf-concurrency:
  39. Concurrency settings
  40. --------------------
  41. .. setting:: CELERYD_CONCURRENCY
  42. CELERYD_CONCURRENCY
  43. ~~~~~~~~~~~~~~~~~~~
  44. The number of concurrent worker processes/threads/green threads, executing
  45. tasks.
  46. Defaults to the number of available CPUs.
  47. .. setting:: CELERYD_PREFETCH_MULTIPLIER
  48. CELERYD_PREFETCH_MULTIPLIER
  49. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  50. How many messages to prefetch at a time multiplied by the number of
  51. concurrent processes. The default is 4 (four messages for each
  52. process). The default setting is usually a good choice, however -- if you
  53. have very long running tasks waiting in the queue and you have to start the
  54. workers, note that the first worker to start will receive four times the
  55. number of messages initially. Thus the tasks may not be fairly distributed
  56. to the workers.
  57. .. _conf-result-backend:
  58. Task result backend settings
  59. ----------------------------
  60. .. setting:: CELERY_RESULT_BACKEND
  61. CELERY_RESULT_BACKEND
  62. ~~~~~~~~~~~~~~~~~~~~~
  63. :Deprecated aliases: ``CELERY_BACKEND``
  64. The backend used to store task results (tombstones).
  65. Disabled by default.
  66. Can be one of the following:
  67. * database
  68. Use a relational database supported by `SQLAlchemy`_.
  69. See :ref:`conf-database-result-backend`.
  70. * cache
  71. Use `memcached`_ to store the results.
  72. See :ref:`conf-cache-result-backend`.
  73. * mongodb
  74. Use `MongoDB`_ to store the results.
  75. See :ref:`conf-mongodb-result-backend`.
  76. * redis
  77. Use `Redis`_ to store the results.
  78. See :ref:`conf-redis-result-backend`.
  79. * tyrant
  80. Use `Tokyo Tyrant`_ to store the results.
  81. See :ref:`conf-tyrant-result-backend`.
  82. * amqp
  83. Send results back as AMQP messages
  84. See :ref:`conf-amqp-result-backend`.
  85. .. warning:
  86. While the AMQP result backend is very efficient, you must make sure
  87. you only receive the same result once. See :doc:`userguide/executing`).
  88. .. _`SQLAlchemy`: http://sqlalchemy.org
  89. .. _`memcached`: http://memcached.org
  90. .. _`MongoDB`: http://mongodb.org
  91. .. _`Redis`: http://code.google.com/p/redis/
  92. .. _`Tokyo Tyrant`: http://1978th.net/tokyotyrant/
  93. .. _conf-database-result-backend:
  94. Database backend settings
  95. -------------------------
  96. .. setting:: CELERY_RESULT_DBURI
  97. CELERY_RESULT_DBURI
  98. ~~~~~~~~~~~~~~~~~~~
  99. Please see `Supported Databases`_ for a table of supported databases.
  100. To use this backend you need to configure it with an
  101. `Connection String`_, some examples include:
  102. .. code-block:: python
  103. # sqlite (filename)
  104. CELERY_RESULT_DBURI = "sqlite:///celerydb.sqlite"
  105. # mysql
  106. CELERY_RESULT_DBURI = "mysql://scott:tiger@localhost/foo"
  107. # postgresql
  108. CELERY_RESULT_DBURI = "postgresql://scott:tiger@localhost/mydatabase"
  109. # oracle
  110. CELERY_RESULT_DBURI = "oracle://scott:tiger@127.0.0.1:1521/sidname"
  111. See `Connection String`_ for more information about connection
  112. strings.
  113. .. setting:: CELERY_RESULT_ENGINE_OPTIONS
  114. CELERY_RESULT_ENGINE_OPTIONS
  115. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  116. To specify additional SQLAlchemy database engine options you can use
  117. the :setting:`CELERY_RESULT_ENGINE_OPTIONS` setting::
  118. # echo enables verbose logging from SQLAlchemy.
  119. CELERY_RESULT_ENGINE_OPTIONS = {"echo": True}
  120. .. _`Supported Databases`:
  121. http://www.sqlalchemy.org/docs/core/engines.html#supported-databases
  122. .. _`Connection String`:
  123. http://www.sqlalchemy.org/docs/core/engines.html#database-urls
  124. Example configuration
  125. ~~~~~~~~~~~~~~~~~~~~~
  126. .. code-block:: python
  127. CELERY_RESULT_BACKEND = "database"
  128. CELERY_RESULT_DBURI = "mysql://user:password@host/dbname"
  129. .. _conf-amqp-result-backend:
  130. AMQP backend settings
  131. ---------------------
  132. .. setting:: CELERY_AMQP_TASK_RESULT_EXPIRES
  133. CELERY_AMQP_TASK_RESULT_EXPIRES
  134. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  135. The time in seconds of which the task result queues should expire.
  136. .. note::
  137. AMQP result expiration requires RabbitMQ versions 2.1.0 and higher.
  138. .. setting:: CELERY_AMQP_TASK_RESULT_CONNECTION_MAX
  139. CELERY_AMQP_TASK_RESULT_CONNECTION_MAX
  140. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  141. Maximum number of connections used by the AMQP result backend simultaneously.
  142. Default is 1 (a single connection per process).
  143. .. setting:: CELERY_RESULT_EXCHANGE
  144. CELERY_RESULT_EXCHANGE
  145. ~~~~~~~~~~~~~~~~~~~~~~
  146. Name of the exchange to publish results in. Default is `"celeryresults"`.
  147. .. setting:: CELERY_RESULT_EXCHANGE_TYPE
  148. CELERY_RESULT_EXCHANGE_TYPE
  149. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  150. The exchange type of the result exchange. Default is to use a `direct`
  151. exchange.
  152. .. setting:: CELERY_RESULT_SERIALIZER
  153. CELERY_RESULT_SERIALIZER
  154. ~~~~~~~~~~~~~~~~~~~~~~~~
  155. Result message serialization format. Default is `"pickle"`. See
  156. :ref:`executing-serializers`.
  157. .. setting:: CELERY_RESULT_PERSISTENT
  158. CELERY_RESULT_PERSISTENT
  159. ~~~~~~~~~~~~~~~~~~~~~~~~
  160. If set to :const:`True`, result messages will be persistent. This means the
  161. messages will not be lost after a broker restart. The default is for the
  162. results to be transient.
  163. Example configuration
  164. ~~~~~~~~~~~~~~~~~~~~~
  165. .. code-block:: python
  166. CELERY_RESULT_BACKEND = "amqp"
  167. CELERY_AMQP_TASK_RESULT_EXPIRES = 18000 # 5 hours.
  168. .. _conf-cache-result-backend:
  169. Cache backend settings
  170. ----------------------
  171. .. note::
  172. The cache backend supports the `pylibmc`_ and `python-memcached`
  173. libraries. The latter is used only if `pylibmc`_ is not installed.
  174. .. setting:: CELERY_CACHE_BACKEND
  175. CELERY_CACHE_BACKEND
  176. ~~~~~~~~~~~~~~~~~~~~
  177. Using a single memcached server:
  178. .. code-block:: python
  179. CELERY_CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
  180. Using multiple memcached servers:
  181. .. code-block:: python
  182. CELERY_RESULT_BACKEND = "cache"
  183. CELERY_CACHE_BACKEND = 'memcached://172.19.26.240:11211;172.19.26.242:11211/'
  184. .. setting:: CELERY_CACHE_BACKEND_OPTIONS
  185. The "dummy" backend stores the cache in memory only:
  186. CELERY_CACHE_BACKEND = "dummy"
  187. CELERY_CACHE_BACKEND_OPTIONS
  188. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  189. You can set pylibmc options using the :setting:`CELERY_CACHE_BACKEND_OPTIONS`
  190. setting:
  191. .. code-block:: python
  192. CELERY_CACHE_BACKEND_OPTIONS = {"binary": True,
  193. "behaviors": {"tcp_nodelay": True}}
  194. .. _`pylibmc`: http://sendapatch.se/projects/pylibmc/
  195. .. _conf-tyrant-result-backend:
  196. Tokyo Tyrant backend settings
  197. -----------------------------
  198. .. note::
  199. The Tokyo Tyrant backend requires the :mod:`pytyrant` library:
  200. http://pypi.python.org/pypi/pytyrant/
  201. This backend requires the following configuration directives to be set:
  202. .. setting:: TT_HOST
  203. TT_HOST
  204. ~~~~~~~
  205. Host name of the Tokyo Tyrant server.
  206. .. setting:: TT_PORT
  207. TT_PORT
  208. ~~~~~~~
  209. The port the Tokyo Tyrant server is listening to.
  210. Example configuration
  211. ~~~~~~~~~~~~~~~~~~~~~
  212. .. code-block:: python
  213. CELERY_RESULT_BACKEND = "tyrant"
  214. TT_HOST = "localhost"
  215. TT_PORT = 1978
  216. .. _conf-redis-result-backend:
  217. Redis backend settings
  218. ----------------------
  219. .. note::
  220. The Redis backend requires the :mod:`redis` library:
  221. http://pypi.python.org/pypi/redis/
  222. To install the redis package use `pip` or `easy_install`::
  223. $ pip install redis
  224. This backend requires the following configuration directives to be set.
  225. .. setting:: CELERY_REDIS_HOST
  226. CELERY_REDIS_HOST
  227. ~~~~~~~~~~~~~~~~~
  228. Host name of the Redis database server. e.g. `"localhost"`.
  229. .. setting:: CELERY_REDIS_PORT
  230. CELERY_REDIS_PORT
  231. ~~~~~~~~~~~~~~~~~
  232. Port to the Redis database server. e.g. `6379`.
  233. .. setting:: CELERY_REDIS_DB
  234. CELERY_REDIS_DB
  235. ~~~~~~~~~~~~~~~
  236. Database number to use. Default is 0
  237. .. setting:: CELERY_REDIS_PASSWORD
  238. CELERY_REDIS_PASSWORD
  239. ~~~~~~~~~~~~~~~~~~~~~
  240. Password used to connect to the database.
  241. Example configuration
  242. ~~~~~~~~~~~~~~~~~~~~~
  243. .. code-block:: python
  244. CELERY_RESULT_BACKEND = "redis"
  245. CELERY_REDIS_HOST = "localhost"
  246. CELERY_REDIS_PORT = 6379
  247. CELERY_REDIS_DB = 0
  248. .. _conf-mongodb-result-backend:
  249. MongoDB backend settings
  250. ------------------------
  251. .. note::
  252. The MongoDB backend requires the :mod:`pymongo` library:
  253. http://github.com/mongodb/mongo-python-driver/tree/master
  254. .. setting:: CELERY_MONGODB_BACKEND_SETTINGS
  255. CELERY_MONGODB_BACKEND_SETTINGS
  256. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  257. This is a dict supporting the following keys:
  258. * host
  259. Host name of the MongoDB server. Defaults to "localhost".
  260. * port
  261. The port the MongoDB server is listening to. Defaults to 27017.
  262. * user
  263. User name to authenticate to the MongoDB server as (optional).
  264. * password
  265. Password to authenticate to the MongoDB server (optional).
  266. * database
  267. The database name to connect to. Defaults to "celery".
  268. * taskmeta_collection
  269. The collection name to store task meta data.
  270. Defaults to "celery_taskmeta".
  271. .. _example-mongodb-result-config:
  272. Example configuration
  273. ~~~~~~~~~~~~~~~~~~~~~
  274. .. code-block:: python
  275. CELERY_RESULT_BACKEND = "mongodb"
  276. CELERY_MONGODB_BACKEND_SETTINGS = {
  277. "host": "192.168.1.100",
  278. "port": 30000,
  279. "database": "mydb",
  280. "taskmeta_collection": "my_taskmeta_collection",
  281. }
  282. .. _conf-messaging:
  283. Message Routing
  284. ---------------
  285. .. _conf-messaging-routing:
  286. .. setting:: CELERY_QUEUES
  287. CELERY_QUEUES
  288. ~~~~~~~~~~~~~
  289. The mapping of queues the worker consumes from. This is a dictionary
  290. of queue name/options. See :ref:`guide-routing` for more information.
  291. The default is a queue/exchange/binding key of `"celery"`, with
  292. exchange type `direct`.
  293. You don't have to care about this unless you want custom routing facilities.
  294. .. setting:: CELERY_ROUTES
  295. CELERY_ROUTES
  296. ~~~~~~~~~~~~~
  297. A list of routers, or a single router used to route tasks to queues.
  298. When deciding the final destination of a task the routers are consulted
  299. in order. See :ref:`routers` for more information.
  300. .. setting:: CELERY_CREATE_MISSING_QUEUES
  301. CELERY_CREATE_MISSING_QUEUES
  302. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  303. If enabled (default), any queues specified that is not defined in
  304. :setting:`CELERY_QUEUES` will be automatically created. See
  305. :ref:`routing-automatic`.
  306. .. setting:: CELERY_DEFAULT_QUEUE
  307. CELERY_DEFAULT_QUEUE
  308. ~~~~~~~~~~~~~~~~~~~~
  309. The queue used by default, if no custom queue is specified. This queue must
  310. be listed in :setting:`CELERY_QUEUES`. The default is: `celery`.
  311. .. seealso::
  312. :ref:`routing-changing-default-queue`
  313. .. setting:: CELERY_DEFAULT_EXCHANGE
  314. CELERY_DEFAULT_EXCHANGE
  315. ~~~~~~~~~~~~~~~~~~~~~~~
  316. Name of the default exchange to use when no custom exchange is
  317. specified. The default is: `celery`.
  318. .. setting:: CELERY_DEFAULT_EXCHANGE_TYPE
  319. CELERY_DEFAULT_EXCHANGE_TYPE
  320. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  321. Default exchange type used when no custom exchange is specified.
  322. The default is: `direct`.
  323. .. setting:: CELERY_DEFAULT_ROUTING_KEY
  324. CELERY_DEFAULT_ROUTING_KEY
  325. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  326. The default routing key used when sending tasks.
  327. The default is: `celery`.
  328. .. setting:: CELERY_DEFAULT_DELIVERY_MODE
  329. CELERY_DEFAULT_DELIVERY_MODE
  330. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  331. Can be `transient` or `persistent`. The default is to send
  332. persistent messages.
  333. .. _conf-broker-connection:
  334. Broker Settings
  335. ---------------
  336. .. setting:: BROKER_TRANSPORT
  337. BROKER_TRANSPORT
  338. ~~~~~~~~~~~~~~~~
  339. :Aliases: ``BROKER_BACKEND``
  340. :Deprecated aliases: ``CARROT_BACKEND``
  341. The Kombu transport to use. Default is ``amqplib``.
  342. You can use a custom transport class name, or select one of the
  343. built-in transports: ``amqplib``, ``pika``, ``redis``, ``beanstalk``,
  344. ``sqlalchemy``, ``django``, ``mongodb``, ``couchdb``.
  345. .. setting:: BROKER_HOST
  346. BROKER_HOST
  347. ~~~~~~~~~~~
  348. Hostname of the broker.
  349. .. setting:: BROKER_PORT
  350. BROKER_PORT
  351. ~~~~~~~~~~~
  352. Custom port of the broker. Default is to use the default port for the
  353. selected backend.
  354. .. setting:: BROKER_USER
  355. BROKER_USER
  356. ~~~~~~~~~~~
  357. Username to connect as.
  358. .. setting:: BROKER_PASSWORD
  359. BROKER_PASSWORD
  360. ~~~~~~~~~~~~~~~
  361. Password to connect with.
  362. .. setting:: BROKER_VHOST
  363. BROKER_VHOST
  364. ~~~~~~~~~~~~
  365. Virtual host. Default is `"/"`.
  366. .. setting:: BROKER_USE_SSL
  367. BROKER_USE_SSL
  368. ~~~~~~~~~~~~~~
  369. Use SSL to connect to the broker. Off by default. This may not be supported
  370. by all transports.
  371. .. setting:: BROKER_POOL_LIMIT
  372. BROKER_POOL_LIMIT
  373. ~~~~~~~~~~~~~~~~~
  374. The maximum number of connections that can be open in the connection pool.
  375. A good default value could be 10, or more if you're using eventlet/gevent
  376. or lots of threads.
  377. If set to :const:`None` or 0 the connection pool will be disabled and
  378. connections will be established and closed for every use.
  379. **Disabled by default.**
  380. .. setting:: BROKER_CONNECTION_TIMEOUT
  381. BROKER_CONNECTION_TIMEOUT
  382. ~~~~~~~~~~~~~~~~~~~~~~~~~
  383. The default timeout in seconds before we give up establishing a connection
  384. to the AMQP server. Default is 4 seconds.
  385. .. setting:: BROKER_CONNECTION_RETRY
  386. BROKER_CONNECTION_RETRY
  387. ~~~~~~~~~~~~~~~~~~~~~~~
  388. Automatically try to re-establish the connection to the AMQP broker if lost.
  389. The time between retries is increased for each retry, and is
  390. not exhausted before :setting:`BROKER_CONNECTION_MAX_RETRIES` is
  391. exceeded.
  392. This behavior is on by default.
  393. .. setting:: BROKER_CONNECTION_MAX_RETRIES
  394. BROKER_CONNECTION_MAX_RETRIES
  395. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  396. Maximum number of retries before we give up re-establishing a connection
  397. to the AMQP broker.
  398. If this is set to :const:`0` or :const:`None`, we will retry forever.
  399. Default is 100 retries.
  400. .. setting:: BROKER_TRANSPORT_OPTIONS
  401. BROKER_TRANSPORT_OPTIONS
  402. ~~~~~~~~~~~~~~~~~~~~~~~~
  403. A dict of additional options passed to the underlying transport.
  404. See your transport user manual for supported options (if any).
  405. .. _conf-task-execution:
  406. Task execution settings
  407. -----------------------
  408. .. setting:: CELERY_ALWAYS_EAGER
  409. CELERY_ALWAYS_EAGER
  410. ~~~~~~~~~~~~~~~~~~~
  411. If this is :const:`True`, all tasks will be executed locally by blocking until
  412. the task returns. ``apply_async()`` and ``Task.delay()`` will return
  413. an :class:`~celery.result.EagerResult` instance, which emulates the API
  414. and behavior of :class:`~celery.result.AsyncResult`, except the result
  415. is already evaluated.
  416. That is, tasks will be executed locally instead of being sent to
  417. the queue.
  418. .. setting:: CELERY_EAGER_PROPAGATES_EXCEPTIONS
  419. CELERY_EAGER_PROPAGATES_EXCEPTIONS
  420. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  421. If this is :const:`True`, eagerly executed tasks (applied by `task.apply()`,
  422. or when the :setting:`CELERY_ALWAYS_EAGER` setting is enabled), will
  423. propagate exceptions.
  424. It's the same as always running ``apply()`` with ``throw=True``.
  425. .. setting:: CELERY_IGNORE_RESULT
  426. CELERY_IGNORE_RESULT
  427. ~~~~~~~~~~~~~~~~~~~~
  428. Whether to store the task return values or not (tombstones).
  429. If you still want to store errors, just not successful return values,
  430. you can set :setting:`CELERY_STORE_ERRORS_EVEN_IF_IGNORED`.
  431. .. setting:: CELERY_MESSAGE_COMPRESSION
  432. CELERY_MESSAGE_COMPRESSION
  433. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  434. Default compression used for task messages.
  435. Can be ``"gzip"``, ``"bzip2"`` (if available), or any custom
  436. compression schemes registered in the Kombu compression registry.
  437. The default is to send uncompressed messages.
  438. .. setting:: CELERY_TASK_RESULT_EXPIRES
  439. CELERY_TASK_RESULT_EXPIRES
  440. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  441. Time (in seconds, or a :class:`~datetime.timedelta` object) for when after
  442. stored task tombstones will be deleted.
  443. A built-in periodic task will delete the results after this time
  444. (:class:`celery.task.backend_cleanup`).
  445. .. note::
  446. For the moment this only works with the database, cache, redis and MongoDB
  447. backends. For the AMQP backend see
  448. :setting:`CELERY_AMQP_TASK_RESULT_EXPIRES`.
  449. When using the database or MongoDB backends, `celerybeat` must be
  450. running for the results to be expired.
  451. .. setting:: CELERY_MAX_CACHED_RESULTS
  452. CELERY_MAX_CACHED_RESULTS
  453. ~~~~~~~~~~~~~~~~~~~~~~~~~
  454. Result backends caches ready results used by the client.
  455. This is the total number of results to cache before older results are evicted.
  456. The default is 5000.
  457. .. setting:: CELERY_TRACK_STARTED
  458. CELERY_TRACK_STARTED
  459. ~~~~~~~~~~~~~~~~~~~~
  460. If :const:`True` the task will report its status as "started" when the
  461. task is executed by a worker. The default value is :const:`False` as
  462. the normal behaviour is to not report that level of granularity. Tasks
  463. are either pending, finished, or waiting to be retried. Having a "started"
  464. state can be useful for when there are long running tasks and there is a
  465. need to report which task is currently running.
  466. .. setting:: CELERY_TASK_SERIALIZER
  467. CELERY_TASK_SERIALIZER
  468. ~~~~~~~~~~~~~~~~~~~~~~
  469. A string identifying the default serialization method to use. Can be
  470. `pickle` (default), `json`, `yaml`, `msgpack` or any custom serialization
  471. methods that have been registered with :mod:`kombu.serialization.registry`.
  472. .. seealso::
  473. :ref:`executing-serializers`.
  474. .. setting:: CELERY_TASK_PUBLISH_RETRY
  475. CELERY_TASK_PUBLISH_RETRY
  476. ~~~~~~~~~~~~~~~~~~~~~~~~~
  477. Decides if publishing task messages will be retried in the case
  478. of connection loss or other connection errors.
  479. See also :setting:`CELERY_TASK_PUBLISH_RETRY_POLICY`.
  480. Disabled by default.
  481. .. setting:: CELERY_TASK_PUBLISH_RETRY_POLICY
  482. Defines the default policy when retrying publishing a task message in
  483. the case of connection loss or other connection errors.
  484. This is a mapping that must contain the following keys:
  485. * `max_retries`
  486. Maximum number of retries before giving up, in this case the
  487. exception that caused the retry to fail will be raised.
  488. A value of 0 or :const:`None` means it will retry forever.
  489. The default is to retry 3 times.
  490. * `interval_start`
  491. Defines the number of seconds (float or integer) to wait between
  492. retries. Default is 0, which means the first retry will be
  493. instantaneous.
  494. * `interval_step`
  495. On each consecutive retry this number will be added to the retry
  496. delay (float or integer). Default is 0.2.
  497. * `interval_max`
  498. Maximum number of seconds (float or integer) to wait between
  499. retries. Default is 0.2.
  500. With the default policy of::
  501. {"max_retries": 3,
  502. "interval_start": 0,
  503. "interval_step": 0.2,
  504. "interval_max": 0.2}
  505. the maximum time spent retrying will be 0.4 seconds. It is set relatively
  506. short by default because a connection failure could lead to a retry pile effect
  507. if the broker connection is down: e.g. many web server processes waiting
  508. to retry blocking other incoming requests.
  509. CELERY_TASK_PUBLISH_RETRY_POLICY
  510. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  511. .. setting:: CELERY_DEFAULT_RATE_LIMIT
  512. CELERY_DEFAULT_RATE_LIMIT
  513. ~~~~~~~~~~~~~~~~~~~~~~~~~
  514. The global default rate limit for tasks.
  515. This value is used for tasks that does not have a custom rate limit
  516. The default is no rate limit.
  517. .. setting:: CELERY_DISABLE_RATE_LIMITS
  518. CELERY_DISABLE_RATE_LIMITS
  519. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  520. Disable all rate limits, even if tasks has explicit rate limits set.
  521. .. setting:: CELERY_ACKS_LATE
  522. CELERY_ACKS_LATE
  523. ~~~~~~~~~~~~~~~~
  524. Late ack means the task messages will be acknowledged **after** the task
  525. has been executed, not *just before*, which is the default behavior.
  526. .. seealso::
  527. FAQ: :ref:`faq-acks_late-vs-retry`.
  528. .. _conf-celeryd:
  529. Worker: celeryd
  530. ---------------
  531. .. setting:: CELERY_IMPORTS
  532. CELERY_IMPORTS
  533. ~~~~~~~~~~~~~~
  534. A sequence of modules to import when the celery daemon starts.
  535. This is used to specify the task modules to import, but also
  536. to import signal handlers and additional remote control commands, etc.
  537. .. setting:: CELERYD_MAX_TASKS_PER_CHILD
  538. CELERYD_MAX_TASKS_PER_CHILD
  539. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  540. Maximum number of tasks a pool worker process can execute before
  541. it's replaced with a new one. Default is no limit.
  542. .. setting:: CELERYD_TASK_TIME_LIMIT
  543. CELERYD_TASK_TIME_LIMIT
  544. ~~~~~~~~~~~~~~~~~~~~~~~
  545. Task hard time limit in seconds. The worker processing the task will
  546. be killed and replaced with a new one when this is exceeded.
  547. .. setting:: CELERYD_TASK_SOFT_TIME_LIMIT
  548. CELERYD_TASK_SOFT_TIME_LIMIT
  549. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  550. Task soft time limit in seconds.
  551. The :exc:`~celery.exceptions.SoftTimeLimitExceeded` exception will be
  552. raised when this is exceeded. The task can catch this to
  553. e.g. clean up before the hard time limit comes.
  554. Example:
  555. .. code-block:: python
  556. from celery.task import task
  557. from celery.exceptions import SoftTimeLimitExceeded
  558. @task()
  559. def mytask():
  560. try:
  561. return do_work()
  562. except SoftTimeLimitExceeded:
  563. cleanup_in_a_hurry()
  564. .. setting:: CELERY_STORE_ERRORS_EVEN_IF_IGNORED
  565. CELERY_STORE_ERRORS_EVEN_IF_IGNORED
  566. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  567. If set, the worker stores all task errors in the result store even if
  568. :attr:`Task.ignore_result <celery.task.base.Task.ignore_result>` is on.
  569. .. setting:: CELERYD_STATE_DB
  570. CELERYD_STATE_DB
  571. ~~~~~~~~~~~~~~~~
  572. Name of the file used to stores persistent worker state (like revoked tasks).
  573. Can be a relative or absolute path, but be aware that the suffix `.db`
  574. may be appended to the file name (depending on Python version).
  575. Can also be set via the :option:`--statedb` argument to
  576. :mod:`~celery.bin.celeryd`.
  577. Not enabled by default.
  578. .. setting:: CELERYD_ETA_SCHEDULER_PRECISION
  579. CELERYD_ETA_SCHEDULER_PRECISION
  580. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  581. Set the maximum time in seconds that the ETA scheduler can sleep between
  582. rechecking the schedule. Default is 1 second.
  583. Setting this value to 1 second means the schedulers precision will
  584. be 1 second. If you need near millisecond precision you can set this to 0.1.
  585. .. _conf-error-mails:
  586. Error E-Mails
  587. -------------
  588. .. setting:: CELERY_SEND_TASK_ERROR_EMAILS
  589. CELERY_SEND_TASK_ERROR_EMAILS
  590. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  591. The default value for the `Task.send_error_emails` attribute, which if
  592. set to :const:`True` means errors occurring during task execution will be
  593. sent to :setting:`ADMINS` by email.
  594. .. setting:: CELERY_TASK_ERROR_WHITELIST
  595. CELERY_TASK_ERROR_WHITELIST
  596. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  597. A white list of exceptions to send error emails for.
  598. .. setting:: ADMINS
  599. ADMINS
  600. ~~~~~~
  601. List of `(name, email_address)` tuples for the administrators that should
  602. receive error emails.
  603. .. setting:: SERVER_EMAIL
  604. SERVER_EMAIL
  605. ~~~~~~~~~~~~
  606. The email address this worker sends emails from.
  607. Default is celery@localhost.
  608. .. setting:: EMAIL_HOST
  609. EMAIL_HOST
  610. ~~~~~~~~~~
  611. The mail server to use. Default is `"localhost"`.
  612. .. setting:: EMAIL_HOST_USER
  613. EMAIL_HOST_USER
  614. ~~~~~~~~~~~~~~~
  615. User name (if required) to log on to the mail server with.
  616. .. setting:: EMAIL_HOST_PASSWORD
  617. EMAIL_HOST_PASSWORD
  618. ~~~~~~~~~~~~~~~~~~~
  619. Password (if required) to log on to the mail server with.
  620. .. setting:: EMAIL_PORT
  621. EMAIL_PORT
  622. ~~~~~~~~~~
  623. The port the mail server is listening on. Default is `25`.
  624. .. setting:: EMAIL_TIMEOUT
  625. EMAIL_TIMEOUT
  626. ~~~~~~~~~~~~~
  627. Timeout in seconds for when we give up trying to connect
  628. to the SMTP server when sending emails.
  629. The default is 2 seconds.
  630. .. _conf-example-error-mail-config:
  631. Example E-Mail configuration
  632. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  633. This configuration enables the sending of error emails to
  634. george@vandelay.com and kramer@vandelay.com:
  635. .. code-block:: python
  636. # Enables error emails.
  637. CELERY_SEND_TASK_ERROR_EMAILS = True
  638. # Name and email addresses of recipients
  639. ADMINS = (
  640. ("George Costanza", "george@vandelay.com"),
  641. ("Cosmo Kramer", "kosmo@vandelay.com"),
  642. )
  643. # Email address used as sender (From field).
  644. SERVER_EMAIL = "no-reply@vandelay.com"
  645. # Mailserver configuration
  646. EMAIL_HOST = "mail.vandelay.com"
  647. EMAIL_PORT = 25
  648. # EMAIL_HOST_USER = "servers"
  649. # EMAIL_HOST_PASSWORD = "s3cr3t"
  650. .. _conf-events:
  651. Events
  652. ------
  653. .. setting:: CELERY_SEND_EVENTS
  654. CELERY_SEND_EVENTS
  655. ~~~~~~~~~~~~~~~~~~
  656. Send events so the worker can be monitored by tools like `celerymon`.
  657. .. setting:: CELERY_SEND_TASK_SENT_EVENT
  658. CELERY_SEND_TASK_SENT_EVENT
  659. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  660. If enabled, a `task-sent` event will be sent for every task so tasks can be
  661. tracked before they are consumed by a worker.
  662. Disabled by default.
  663. .. setting:: CELERY_EVENT_SERIALIZER
  664. CELERY_EVENT_SERIALIZER
  665. ~~~~~~~~~~~~~~~~~~~~~~~
  666. Message serialization format used when sending event messages.
  667. Default is `"json"`. See :ref:`executing-serializers`.
  668. .. _conf-broadcast:
  669. Broadcast Commands
  670. ------------------
  671. .. setting:: CELERY_BROADCAST_QUEUE
  672. CELERY_BROADCAST_QUEUE
  673. ~~~~~~~~~~~~~~~~~~~~~~
  674. Name prefix for the queue used when listening for broadcast messages.
  675. The workers host name will be appended to the prefix to create the final
  676. queue name.
  677. Default is `"celeryctl"`.
  678. .. setting:: CELERY_BROADCAST_EXCHANGE
  679. CELERY_BROADCAST_EXCHANGE
  680. ~~~~~~~~~~~~~~~~~~~~~~~~~
  681. Name of the exchange used for broadcast messages.
  682. Default is `"celeryctl"`.
  683. .. setting:: CELERY_BROADCAST_EXCHANGE_TYPE
  684. CELERY_BROADCAST_EXCHANGE_TYPE
  685. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  686. Exchange type used for broadcast messages. Default is `"fanout"`.
  687. .. _conf-logging:
  688. Logging
  689. -------
  690. .. setting:: CELERYD_HIJACK_ROOT_LOGGER
  691. CELERYD_HIJACK_ROOT_LOGGER
  692. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  693. By default any previously configured logging options will be reset,
  694. because the Celery apps "hijacks" the root logger.
  695. If you want to customize your own logging then you can disable
  696. this behavior.
  697. .. note::
  698. Logging can also be customized by connecting to the
  699. :signal:`celery.signals.setup_logging` signal.
  700. .. setting:: CELERYD_LOG_FILE
  701. CELERYD_LOG_FILE
  702. ~~~~~~~~~~~~~~~~
  703. The default file name the worker daemon logs messages to. Can be overridden
  704. using the :option:`--logfile` option to :mod:`~celery.bin.celeryd`.
  705. The default is :const:`None` (`stderr`)
  706. .. setting:: CELERYD_LOG_LEVEL
  707. CELERYD_LOG_LEVEL
  708. ~~~~~~~~~~~~~~~~~
  709. Worker log level, can be one of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  710. :const:`ERROR` or :const:`CRITICAL`.
  711. Can also be set via the :option:`--loglevel` argument to
  712. :mod:`~celery.bin.celeryd`.
  713. See the :mod:`logging` module for more information.
  714. .. setting:: CELERYD_LOG_COLOR
  715. CELERYD_LOG_COLOR
  716. ~~~~~~~~~~~~~~~~~
  717. Enables/disables colors in logging output by the Celery apps.
  718. By default colors are enabled if
  719. 1) the app is logging to a real terminal, and not a file.
  720. 2) the app is not running on Windows.
  721. .. setting:: CELERYD_LOG_FORMAT
  722. CELERYD_LOG_FORMAT
  723. ~~~~~~~~~~~~~~~~~~
  724. The format to use for log messages.
  725. Default is `[%(asctime)s: %(levelname)s/%(processName)s] %(message)s`
  726. See the Python :mod:`logging` module for more information about log
  727. formats.
  728. .. setting:: CELERYD_TASK_LOG_FORMAT
  729. CELERYD_TASK_LOG_FORMAT
  730. ~~~~~~~~~~~~~~~~~~~~~~~
  731. The format to use for log messages logged in tasks. Can be overridden using
  732. the :option:`--loglevel` option to :mod:`~celery.bin.celeryd`.
  733. Default is::
  734. [%(asctime)s: %(levelname)s/%(processName)s]
  735. [%(task_name)s(%(task_id)s)] %(message)s
  736. See the Python :mod:`logging` module for more information about log
  737. formats.
  738. .. setting:: CELERY_REDIRECT_STDOUTS
  739. CELERY_REDIRECT_STDOUTS
  740. ~~~~~~~~~~~~~~~~~~~~~~~
  741. If enabled `stdout` and `stderr` will be redirected
  742. to the current logger.
  743. Enabled by default.
  744. Used by :program:`celeryd` and :program:`celerybeat`.
  745. .. setting:: CELERY_REDIRECT_STDOUTS_LEVEL
  746. CELERY_REDIRECT_STDOUTS_LEVEL
  747. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  748. The log level output to `stdout` and `stderr` is logged as.
  749. Can be one of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  750. :const:`ERROR` or :const:`CRITICAL`.
  751. Default is :const:`WARNING`.
  752. .. _conf-custom-components:
  753. Custom Component Classes (advanced)
  754. -----------------------------------
  755. .. setting:: CELERYD_POOL
  756. CELERYD_POOL
  757. ~~~~~~~~~~~~
  758. Name of the pool class used by the worker.
  759. You can use a custom pool class name, or select one of
  760. the built-in aliases: ``processes``, ``eventlet``, ``gevent``.
  761. Default is ``processes``.
  762. .. setting:: CELERYD_AUTOSCALER
  763. CELERYD_AUTOSCALER
  764. ~~~~~~~~~~~~~~~~~~
  765. Name of the autoscaler class to use.
  766. Default is ``"celery.worker.autoscale.Autoscaler"``.
  767. .. setting:: CELERYD_CONSUMER
  768. CELERYD_CONSUMER
  769. ~~~~~~~~~~~~~~~~
  770. Name of the consumer class used by the worker.
  771. Default is :class:`celery.worker.consumer.Consumer`
  772. .. setting:: CELERYD_MEDIATOR
  773. CELERYD_MEDIATOR
  774. ~~~~~~~~~~~~~~~~
  775. Name of the mediator class used by the worker.
  776. Default is :class:`celery.worker.controllers.Mediator`.
  777. .. setting:: CELERYD_ETA_SCHEDULER
  778. CELERYD_ETA_SCHEDULER
  779. ~~~~~~~~~~~~~~~~~~~~~
  780. Name of the ETA scheduler class used by the worker.
  781. Default is :class:`celery.utils.timer2.Timer`, or one overrided
  782. by the pool implementation.
  783. .. _conf-celerybeat:
  784. Periodic Task Server: celerybeat
  785. --------------------------------
  786. .. setting:: CELERYBEAT_SCHEDULE
  787. CELERYBEAT_SCHEDULE
  788. ~~~~~~~~~~~~~~~~~~~
  789. The periodic task schedule used by :mod:`~celery.bin.celerybeat`.
  790. See :ref:`beat-entries`.
  791. .. setting:: CELERYBEAT_SCHEDULER
  792. CELERYBEAT_SCHEDULER
  793. ~~~~~~~~~~~~~~~~~~~~
  794. The default scheduler class. Default is
  795. `"celery.beat.PersistentScheduler"`.
  796. Can also be set via the :option:`-S` argument to
  797. :mod:`~celery.bin.celerybeat`.
  798. .. setting:: CELERYBEAT_SCHEDULE_FILENAME
  799. CELERYBEAT_SCHEDULE_FILENAME
  800. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  801. Name of the file used by `PersistentScheduler` to store the last run times
  802. of periodic tasks. Can be a relative or absolute path, but be aware that the
  803. suffix `.db` may be appended to the file name (depending on Python version).
  804. Can also be set via the :option:`--schedule` argument to
  805. :mod:`~celery.bin.celerybeat`.
  806. .. setting:: CELERYBEAT_MAX_LOOP_INTERVAL
  807. CELERYBEAT_MAX_LOOP_INTERVAL
  808. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  809. The maximum number of seconds :mod:`~celery.bin.celerybeat` can sleep
  810. between checking the schedule. Default is 300 seconds (5 minutes).
  811. .. setting:: CELERYBEAT_LOG_FILE
  812. CELERYBEAT_LOG_FILE
  813. ~~~~~~~~~~~~~~~~~~~
  814. The default file name to log messages to. Can be overridden using
  815. the `--logfile` option to :mod:`~celery.bin.celerybeat`.
  816. The default is :const:`None` (`stderr`).
  817. .. setting:: CELERYBEAT_LOG_LEVEL
  818. CELERYBEAT_LOG_LEVEL
  819. ~~~~~~~~~~~~~~~~~~~~
  820. Logging level. Can be any of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  821. :const:`ERROR`, or :const:`CRITICAL`.
  822. Can also be set via the :option:`--loglevel` argument to
  823. :mod:`~celery.bin.celerybeat`.
  824. See the :mod:`logging` module for more information.
  825. .. _conf-celerymon:
  826. Monitor Server: celerymon
  827. -------------------------
  828. .. setting:: CELERYMON_LOG_FILE
  829. CELERYMON_LOG_FILE
  830. ~~~~~~~~~~~~~~~~~~
  831. The default file name to log messages to. Can be overridden using
  832. the :option:`--logfile` argument to `celerymon`.
  833. The default is :const:`None` (`stderr`)
  834. .. setting:: CELERYMON_LOG_LEVEL
  835. CELERYMON_LOG_LEVEL
  836. ~~~~~~~~~~~~~~~~~~~
  837. Logging level. Can be any of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  838. :const:`ERROR`, or :const:`CRITICAL`.
  839. See the :mod:`logging` module for more information.
  840. .. setting:: CELERYMON_LOG_FORMAT
  841. CELERYMON_LOG_FORMAT
  842. ~~~~~~~~~~~~~~~~~~~~
  843. The format to use for log messages.
  844. Default is `[%(asctime)s: %(levelname)s/%(processName)s] %(message)s`
  845. See the Python :mod:`logging` module for more information about log
  846. formats.