configuration.rst 62 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739174017411742174317441745174617471748174917501751175217531754175517561757175817591760176117621763176417651766176717681769177017711772177317741775177617771778177917801781178217831784178517861787178817891790179117921793179417951796179717981799180018011802180318041805180618071808180918101811181218131814181518161817181818191820182118221823182418251826182718281829183018311832183318341835183618371838183918401841184218431844184518461847184818491850185118521853185418551856185718581859186018611862186318641865186618671868186918701871187218731874187518761877187818791880188118821883188418851886188718881889189018911892189318941895189618971898189919001901190219031904190519061907190819091910191119121913191419151916191719181919192019211922192319241925192619271928192919301931193219331934193519361937193819391940194119421943194419451946194719481949195019511952195319541955195619571958195919601961196219631964196519661967196819691970197119721973197419751976197719781979198019811982198319841985198619871988198919901991199219931994199519961997199819992000200120022003200420052006200720082009201020112012201320142015201620172018201920202021202220232024202520262027202820292030203120322033203420352036203720382039204020412042204320442045204620472048204920502051205220532054205520562057205820592060206120622063206420652066206720682069207020712072207320742075207620772078207920802081208220832084208520862087208820892090209120922093209420952096209720982099210021012102210321042105210621072108210921102111211221132114211521162117211821192120212121222123212421252126212721282129213021312132213321342135213621372138213921402141214221432144214521462147214821492150215121522153215421552156215721582159216021612162216321642165216621672168216921702171217221732174217521762177217821792180218121822183218421852186218721882189219021912192219321942195219621972198219922002201220222032204220522062207220822092210221122122213221422152216221722182219222022212222222322242225222622272228222922302231223222332234223522362237223822392240224122422243224422452246224722482249225022512252225322542255225622572258225922602261226222632264226522662267226822692270227122722273227422752276227722782279228022812282228322842285228622872288228922902291229222932294229522962297229822992300230123022303230423052306230723082309231023112312231323142315231623172318
  1. .. _configuration:
  2. ============================
  3. Configuration and defaults
  4. ============================
  5. This document describes the configuration options available.
  6. If you're using the default loader, you must create the :file:`celeryconfig.py`
  7. module and make sure it is available on the Python path.
  8. .. contents::
  9. :local:
  10. :depth: 2
  11. .. _conf-example:
  12. Example configuration file
  13. ==========================
  14. This is an example configuration file to get you started.
  15. It should contain all you need to run a basic Celery set-up.
  16. .. code-block:: python
  17. ## Broker settings.
  18. broker_url = 'amqp://guest:guest@localhost:5672//'
  19. # List of modules to import when celery starts.
  20. imports = ('myapp.tasks',)
  21. ## Using the database to store task state and results.
  22. result_backend = 'db+sqlite:///results.db'
  23. task_annotations = {'tasks.add': {'rate_limit': '10/s'}}
  24. .. _conf-old-settings-map:
  25. New lowercase settings
  26. ======================
  27. Version 4.0 introduced new lower case settings and setting organization.
  28. The major difference between previous versions, apart from the lower case
  29. names, are the renaming of some prefixes, like ``celerybeat_`` to ``beat_``,
  30. ``celeryd_`` to ``worker_``, and most of the top level ``celery_`` settings
  31. have been moved into a new ``task_`` prefix.
  32. Celery will still be able to read old configuration files, so there is no
  33. rush in moving to the new settings format.
  34. ===================================== ==============================================
  35. **Setting name** **Replace with**
  36. ===================================== ==============================================
  37. ``CELERY_ACCEPT_CONTENT`` :setting:`accept_content`
  38. ``ADMINS`` :setting:`admins`
  39. ``CELERY_ENABLE_UTC`` :setting:`enable_utc`
  40. ``CELERY_IMPORTS`` :setting:`imports`
  41. ``CELERY_INCLUDE`` :setting:`include`
  42. ``SERVER_EMAIL`` :setting:`server_email`
  43. ``CELERY_TIMEZONE`` :setting:`timezone`
  44. ``CELERYBEAT_MAX_LOOP_INTERVAL`` :setting:`beat_max_loop_interval`
  45. ``CELERYBEAT_SCHEDULE`` :setting:`beat_schedule`
  46. ``CELERYBEAT_SCHEDULER`` :setting:`beat_scheduler`
  47. ``CELERYBEAT_SCHEDULE_FILENAME`` :setting:`beat_schedule_filename`
  48. ``CELERYBEAT_SYNC_EVERY`` :setting:`beat_sync_every`
  49. ``BROKER_URL`` :setting:`broker_url`
  50. ``BROKER_TRANSPORT`` :setting:`broker_transport`
  51. ``BROKER_TRANSPORT_OPTIONS`` :setting:`broker_transport_options`
  52. ``BROKER_CONNECTION_TIMEOUT`` :setting:`broker_connection_timeout`
  53. ``BROKER_CONNECTION_RETRY`` :setting:`broker_connection_retry`
  54. ``BROKER_CONNECTION_MAX_RETRIES`` :setting:`broker_connection_max_retries`
  55. ``BROKER_FAILOVER_STRATEGY`` :setting:`broker_failover_strategy`
  56. ``BROKER_HEARTBEAT`` :setting:`broker_heartbeat`
  57. ``BROKER_LOGIN_METHOD`` :setting:`broker_login_method`
  58. ``BROKER_POOL_LIMIT`` :setting:`broker_pool_limit`
  59. ``BROKER_USE_SSL`` :setting:`broker_use_ssl`
  60. ``CELERY_CACHE_BACKEND`` :setting:`cache_backend`
  61. ``CELERY_CACHE_BACKEND_OPTIONS`` :setting:`cache_backend_options`
  62. ``CASSANDRA_COLUMN_FAMILY`` :setting:`cassandra_table`
  63. ``CASSANDRA_ENTRY_TTL`` :setting:`cassandra_entry_ttl`
  64. ``CASSANDRA_KEYSPACE`` :setting:`cassandra_keyspace`
  65. ``CASSANDRA_PORT`` :setting:`cassandra_port`
  66. ``CASSANDRA_READ_CONSISTENCY`` :setting:`cassandra_read_consistency`
  67. ``CASSANDRA_SERVERS`` :setting:`cassandra_servers`
  68. ``CASSANDRA_WRITE_CONSISTENCY`` :setting:`cassandra_write_consistency`
  69. ``CELERY_COUCHBASE_BACKEND_SETTINGS`` :setting:`couchbase_backend_settings`
  70. ``EMAIL_HOST`` :setting:`email_host`
  71. ``EMAIL_HOST_USER`` :setting:`email_host_user`
  72. ``EMAIL_HOST_PASSWORD`` :setting:`email_host_password`
  73. ``EMAIL_PORT`` :setting:`email_port`
  74. ``EMAIL_TIMEOUT`` :setting:`email_timeout`
  75. ``EMAIL_USE_SSL`` :setting:`email_use_ssl`
  76. ``EMAIL_USE_TLS`` :setting:`email_use_tls`
  77. ``CELERY_MONGODB_BACKEND_SETTINGS`` :setting:`mongodb_backend_settings`
  78. ``CELERY_EVENT_QUEUE_EXPIRES`` :setting:`event_queue_expires`
  79. ``CELERY_EVENT_QUEUE_TTL`` :setting:`event_queue_ttl`
  80. ``CELERY_EVENT_SERIALIZER`` :setting:`event_serializer`
  81. ``CELERY_REDIS_DB`` :setting:`redis_db`
  82. ``CELERY_REDIS_HOST`` :setting:`redis_host`
  83. ``CELERY_REDIS_MAX_CONNECTIONS`` :setting:`redis_max_connections`
  84. ``CELERY_REDIS_PASSWORD`` :setting:`redis_password`
  85. ``CELERY_REDIS_PORT`` :setting:`redis_port`
  86. ``CELERY_RESULT_BACKEND`` :setting:`result_backend`
  87. ``CELERY_MAX_CACHED_RESULTS`` :setting:`result_cache_max`
  88. ``CELERY_MESSAGE_COMPRESSION`` :setting:`result_compression`
  89. ``CELERY_RESULT_EXCHANGE`` :setting:`result_exchange`
  90. ``CELERY_RESULT_EXCHANGE_TYPE`` :setting:`result_exchange_type`
  91. ``CELERY_TASK_RESULT_EXPIRES`` :setting:`result_expires`
  92. ``CELERY_RESULT_PERSISTENT`` :setting:`result_persistent`
  93. ``CELERY_RESULT_SERIALIZER`` :setting:`result_serializer`
  94. ``CELERY_RESULT_DBURI`` :setting:`sqlalchemy_dburi`
  95. ``CELERY_RESULT_ENGINE_OPTIONS`` :setting:`sqlalchemy_engine_options`
  96. ``-*-_DB_SHORT_LIVED_SESSIONS`` :setting:`sqlalchemy_short_lived_sessions`
  97. ``CELERY_RESULT_DB_TABLE_NAMES`` :setting:`sqlalchemy_db_names`
  98. ``CELERY_SECURITY_CERTIFICATE`` :setting:`security_certificate`
  99. ``CELERY_SECURITY_CERT_STORE`` :setting:`security_cert_store`
  100. ``CELERY_SECURITY_KEY`` :setting:`security_key`
  101. ``CELERY_ACKS_LATE`` :setting:`task_acks_late`
  102. ``CELERY_ALWAYS_EAGER`` :setting:`task_always_eager`
  103. ``CELERY_ANNOTATIONS`` :setting:`task_annotations`
  104. ``CELERY_MESSAGE_COMPRESSION`` :setting:`task_compression`
  105. ``CELERY_CREATE_MISSING_QUEUES`` :setting:`task_create_missing_queues`
  106. ``CELERY_DEFAULT_DELIVERY_MODE`` :setting:`task_default_delivery_mode`
  107. ``CELERY_DEFAULT_EXCHANGE`` :setting:`task_default_exchange`
  108. ``CELERY_DEFAULT_EXCHANGE_TYPE`` :setting:`task_default_exchange_type`
  109. ``CELERY_DEFAULT_QUEUE`` :setting:`task_default_queue`
  110. ``CELERY_DEFAULT_RATE_LIMIT`` :setting:`task_default_rate_limit`
  111. ``CELERY_DEFAULT_ROUTING_KEY`` :setting:`task_default_routing_key`
  112. ``-'-_EAGER_PROPAGATES_EXCEPTIONS`` :setting:`task_eager_propagates`
  113. ``CELERY_IGNORE_RESULT`` :setting:`task_ignore_result`
  114. ``CELERY_TASK_PUBLISH_RETRY`` :setting:`task_publish_retry`
  115. ``CELERY_TASK_PUBLISH_RETRY_POLICY`` :setting:`task_publish_retry_policy`
  116. ``CELERY_QUEUES`` :setting:`task_queues`
  117. ``CELERY_ROUTES`` :setting:`task_routes`
  118. ``CELERY_SEND_TASK_ERROR_EMAILS`` :setting:`task_send_error_emails`
  119. ``CELERY_SEND_TASK_SENT_EVENT`` :setting:`task_send_sent_event`
  120. ``CELERY_TASK_SERIALIZER`` :setting:`task_serializer`
  121. ``CELERYD_TASK_SOFT_TIME_LIMIT`` :setting:`task_soft_time_limit`
  122. ``CELERYD_TASK_TIME_LIMIT`` :setting:`task_time_limit`
  123. ``CELERY_TRACK_STARTED`` :setting:`task_track_started`
  124. ``CELERYD_AGENT`` :setting:`worker_agent`
  125. ``CELERYD_AUTOSCALER`` :setting:`worker_autoscaler`
  126. ``CELERYD_AUTORELAODER`` :setting:`worker_autoreloader`
  127. ``CELERYD_CONCURRENCY`` :setting:`worker_concurrency`
  128. ``CELERYD_CONSUMER`` :setting:`worker_consumer`
  129. ``CELERY_WORKER_DIRECT`` :setting:`worker_direct`
  130. ``CELERY_DISABLE_RATE_LIMITS`` :setting:`worker_disable_rate_limits`
  131. ``CELERY_ENABLE_REMOTE_CONTROL`` :setting:`worker_enable_remote_control`
  132. ``CELERYD_FORCE_EXECV`` :setting:`worker_force_execv`
  133. ``CELERYD_HIJACK_ROOT_LOGGER`` :setting:`worker_hijack_root_logger`
  134. ``CELERYD_LOG_COLOR`` :setting:`worker_log_color`
  135. ``CELERYD_LOG_FORMAT`` :setting:`worker_log_format`
  136. ``CELERYD_WORKER_LOST_WAIT`` :setting:`worker_lost_wait`
  137. ``CELERYD_MAX_TASKS_PER_CHILD`` :setting:`worker_max_tasks_per_child`
  138. ``CELERYD_POOL`` :setting:`worker_pool`
  139. ``CELERYD_POOL_PUTLOCKS`` :setting:`worker_pool_putlocks`
  140. ``CELERYD_POOL_RESTARTS`` :setting:`worker_pool_restarts`
  141. ``CELERYD_PREFETCH_MULTIPLIER`` :setting:`worker_prefetch_multiplier`
  142. ``CELERYD_REDIRECT_STDOUTS`` :setting:`worker_redirect_stdouts`
  143. ``CELERYD_REDIRECT_STDOUTS_LEVEL`` :setting:`worker_redirect_stdouts_level`
  144. ``CELERYD_SEND_EVENTS`` :setting:`worker_send_task_events`
  145. ``CELERYD_STATE_DB`` :setting:`worker_state_db`
  146. ``CELERYD_TASK_LOG_FORMAT`` :setting:`worker_task_log_format`
  147. ``CELERYD_TIMER`` :setting:`worker_timer`
  148. ``CELERYD_TIMER_PRECISION`` :setting:`worker_timer_precision`
  149. ===================================== ==============================================
  150. Configuration Directives
  151. ========================
  152. .. _conf-datetime:
  153. General settings
  154. ----------------
  155. .. setting:: accept_content
  156. accept_content
  157. ~~~~~~~~~~~~~~
  158. A whitelist of content-types/serializers to allow.
  159. If a message is received that is not in this list then
  160. the message will be discarded with an error.
  161. By default any content type is enabled (including pickle and yaml)
  162. so make sure untrusted parties do not have access to your broker.
  163. See :ref:`guide-security` for more.
  164. Example::
  165. # using serializer name
  166. accept_content = ['json']
  167. # or the actual content-type (MIME)
  168. accept_content = ['application/json']
  169. Time and date settings
  170. ----------------------
  171. .. setting:: enable_utc
  172. enable_utc
  173. ~~~~~~~~~~
  174. .. versionadded:: 2.5
  175. If enabled dates and times in messages will be converted to use
  176. the UTC timezone.
  177. Note that workers running Celery versions below 2.5 will assume a local
  178. timezone for all messages, so only enable if all workers have been
  179. upgraded.
  180. Enabled by default since version 3.0.
  181. .. setting:: timezone
  182. timezone
  183. ~~~~~~~~
  184. Configure Celery to use a custom time zone.
  185. The timezone value can be any time zone supported by the `pytz`_
  186. library.
  187. If not set the UTC timezone is used. For backwards compatibility
  188. there is also a :setting:`enable_utc` setting, and this is set
  189. to false the system local timezone is used instead.
  190. .. _`pytz`: http://pypi.python.org/pypi/pytz/
  191. .. _conf-tasks:
  192. Task settings
  193. -------------
  194. .. setting:: task_annotations
  195. task_annotations
  196. ~~~~~~~~~~~~~~~~
  197. This setting can be used to rewrite any task attribute from the
  198. configuration. The setting can be a dict, or a list of annotation
  199. objects that filter for tasks and return a map of attributes
  200. to change.
  201. This will change the ``rate_limit`` attribute for the ``tasks.add``
  202. task:
  203. .. code-block:: python
  204. task_annotations = {'tasks.add': {'rate_limit': '10/s'}}
  205. or change the same for all tasks:
  206. .. code-block:: python
  207. task_annotations = {'*': {'rate_limit': '10/s'}}
  208. You can change methods too, for example the ``on_failure`` handler:
  209. .. code-block:: python
  210. def my_on_failure(self, exc, task_id, args, kwargs, einfo):
  211. print('Oh no! Task failed: {0!r}'.format(exc))
  212. task_annotations = {'*': {'on_failure': my_on_failure}}
  213. If you need more flexibility then you can use objects
  214. instead of a dict to choose which tasks to annotate:
  215. .. code-block:: python
  216. class MyAnnotate(object):
  217. def annotate(self, task):
  218. if task.name.startswith('tasks.'):
  219. return {'rate_limit': '10/s'}
  220. task_annotations = (MyAnnotate(), {…})
  221. .. setting:: task_compression
  222. task_compression
  223. ~~~~~~~~~~~~~~~~
  224. Default compression used for task messages.
  225. Can be ``gzip``, ``bzip2`` (if available), or any custom
  226. compression schemes registered in the Kombu compression registry.
  227. The default is to send uncompressed messages.
  228. .. setting:: task_protocol
  229. task_protocol
  230. ~~~~~~~~~~~~~
  231. Default task message protocol version.
  232. Supports protocols: 1 and 2 (default is 1 for backwards compatibility).
  233. .. setting:: task_serializer
  234. task_serializer
  235. ~~~~~~~~~~~~~~~
  236. A string identifying the default serialization method to use. Can be
  237. `pickle` (default), `json`, `yaml`, `msgpack` or any custom serialization
  238. methods that have been registered with :mod:`kombu.serialization.registry`.
  239. .. seealso::
  240. :ref:`calling-serializers`.
  241. .. setting:: task_publish_retry
  242. task_publish_retry
  243. ~~~~~~~~~~~~~~~~~~
  244. .. versionadded:: 2.2
  245. Decides if publishing task messages will be retried in the case
  246. of connection loss or other connection errors.
  247. See also :setting:`task_publish_retry_policy`.
  248. Enabled by default.
  249. .. setting:: task_publish_retry_policy
  250. task_publish_retry_policy
  251. ~~~~~~~~~~~~~~~~~~~~~~~~~
  252. .. versionadded:: 2.2
  253. Defines the default policy when retrying publishing a task message in
  254. the case of connection loss or other connection errors.
  255. See :ref:`calling-retry` for more information.
  256. .. _conf-task-execution:
  257. Task execution settings
  258. -----------------------
  259. .. setting:: task_always_eager
  260. task_always_eager
  261. ~~~~~~~~~~~~~~~~~
  262. If this is :const:`True`, all tasks will be executed locally by blocking until
  263. the task returns. ``apply_async()`` and ``Task.delay()`` will return
  264. an :class:`~celery.result.EagerResult` instance, which emulates the API
  265. and behavior of :class:`~celery.result.AsyncResult`, except the result
  266. is already evaluated.
  267. That is, tasks will be executed locally instead of being sent to
  268. the queue.
  269. .. setting:: task_eager_propagates
  270. task_eager_propagates
  271. ~~~~~~~~~~~~~~~~~~~~~
  272. If this is :const:`True`, eagerly executed tasks (applied by `task.apply()`,
  273. or when the :setting:`task_always_eager` setting is enabled), will
  274. propagate exceptions.
  275. It's the same as always running ``apply()`` with ``throw=True``.
  276. .. setting:: task_ignore_result
  277. task_ignore_result
  278. ~~~~~~~~~~~~~~~~~~
  279. Whether to store the task return values or not (tombstones).
  280. If you still want to store errors, just not successful return values,
  281. you can set :setting:`task_store_errors_even_if_ignored`.
  282. .. setting:: task_store_errors_even_if_ignored
  283. task_store_errors_even_if_ignored
  284. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  285. If set, the worker stores all task errors in the result store even if
  286. :attr:`Task.ignore_result <celery.task.base.Task.ignore_result>` is on.
  287. .. setting:: task_track_started
  288. task_track_started
  289. ~~~~~~~~~~~~~~~~~~
  290. If :const:`True` the task will report its status as 'started' when the
  291. task is executed by a worker. The default value is :const:`False` as
  292. the normal behaviour is to not report that level of granularity. Tasks
  293. are either pending, finished, or waiting to be retried. Having a 'started'
  294. state can be useful for when there are long running tasks and there is a
  295. need to report which task is currently running.
  296. .. setting:: task_time_limit
  297. task_time_limit
  298. ~~~~~~~~~~~~~~~
  299. Task hard time limit in seconds. The worker processing the task will
  300. be killed and replaced with a new one when this is exceeded.
  301. .. setting:: task_soft_time_limit
  302. task_soft_time_limit
  303. ~~~~~~~~~~~~~~~~~~~~
  304. Task soft time limit in seconds.
  305. The :exc:`~@SoftTimeLimitExceeded` exception will be
  306. raised when this is exceeded. The task can catch this to
  307. e.g. clean up before the hard time limit comes.
  308. Example:
  309. .. code-block:: python
  310. from celery.exceptions import SoftTimeLimitExceeded
  311. @app.task
  312. def mytask():
  313. try:
  314. return do_work()
  315. except SoftTimeLimitExceeded:
  316. cleanup_in_a_hurry()
  317. .. setting:: task_acks_late
  318. task_acks_late
  319. ~~~~~~~~~~~~~~
  320. Late ack means the task messages will be acknowledged **after** the task
  321. has been executed, not *just before*, which is the default behavior.
  322. .. seealso::
  323. FAQ: :ref:`faq-acks_late-vs-retry`.
  324. .. setting:: task_reject_on_worker_lost
  325. task_reject_on_worker_lost
  326. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  327. Even if :setting:`task_acks_late` is enabled, the worker will
  328. acknowledge tasks when the worker process executing them abrubtly
  329. exits or is signaled (e.g. :sig:`KILL`/:sig:`INT`, etc).
  330. Setting this to true allows the message to be requeued instead,
  331. so that the task will execute again by the same worker, or another
  332. worker.
  333. .. warning::
  334. Enabling this can cause message loops; make sure you know
  335. what you're doing.
  336. .. setting:: task_default_rate_limit
  337. task_default_rate_limit
  338. ~~~~~~~~~~~~~~~~~~~~~~~
  339. The global default rate limit for tasks.
  340. This value is used for tasks that does not have a custom rate limit
  341. The default is no rate limit.
  342. .. seealso::
  343. The setting:`worker_disable_rate_limits` setting can
  344. disable all rate limits.
  345. .. _conf-result-backend:
  346. Task result backend settings
  347. ----------------------------
  348. .. setting:: result_backend
  349. result_backend
  350. ~~~~~~~~~~~~~~
  351. The backend used to store task results (tombstones).
  352. Disabled by default.
  353. Can be one of the following:
  354. * rpc
  355. Send results back as AMQP messages
  356. See :ref:`conf-rpc-result-backend`.
  357. * database
  358. Use a relational database supported by `SQLAlchemy`_.
  359. See :ref:`conf-database-result-backend`.
  360. * redis
  361. Use `Redis`_ to store the results.
  362. See :ref:`conf-redis-result-backend`.
  363. * cache
  364. Use `memcached`_ to store the results.
  365. See :ref:`conf-cache-result-backend`.
  366. * mongodb
  367. Use `MongoDB`_ to store the results.
  368. See :ref:`conf-mongodb-result-backend`.
  369. * cassandra
  370. Use `Cassandra`_ to store the results.
  371. See :ref:`conf-cassandra-result-backend`.
  372. * elasticsearch
  373. Use `Elasticsearch`_ to store the results.
  374. See :ref:`conf-elasticsearch-result-backend`.
  375. * ironcache
  376. Use `IronCache`_ to store the results.
  377. See :ref:`conf-ironcache-result-backend`.
  378. * couchbase
  379. Use `Couchbase`_ to store the results.
  380. See :ref:`conf-couchbase-result-backend`.
  381. * couchdb
  382. Use `CouchDB`_ to store the results.
  383. See :ref:`conf-couchdb-result-backend`.
  384. * amqp
  385. Older AMQP backend (badly) emulating a database-based backend.
  386. See :ref:`conf-amqp-result-backend`.
  387. * filesystem
  388. Use a shared directory to store the results.
  389. See :ref:`conf-filesystem-result-backend`.
  390. .. warning:
  391. While the AMQP result backend is very efficient, you must make sure
  392. you only receive the same result once. See :doc:`userguide/calling`).
  393. .. _`SQLAlchemy`: http://sqlalchemy.org
  394. .. _`memcached`: http://memcached.org
  395. .. _`MongoDB`: http://mongodb.org
  396. .. _`Redis`: http://redis.io
  397. .. _`Cassandra`: http://cassandra.apache.org/
  398. .. _`Elasticsearch`: https://aws.amazon.com/elasticsearch-service/
  399. .. _`IronCache`: http://www.iron.io/cache
  400. .. _`CouchDB`: http://www.couchdb.com/
  401. .. _`Couchbase`: http://www.couchbase.com/
  402. .. setting:: result_serializer
  403. result_serializer
  404. ~~~~~~~~~~~~~~~~~
  405. Result serialization format. Default is ``pickle``. See
  406. :ref:`calling-serializers` for information about supported
  407. serialization formats.
  408. .. setting:: result_compression
  409. result_compression
  410. ~~~~~~~~~~~~~~~~~~
  411. Optional compression method used for task results.
  412. Supports the same options as the :setting:`task_serializer` setting.
  413. Default is no compression.
  414. .. setting:: result_expires
  415. result_expires
  416. ~~~~~~~~~~~~~~
  417. Time (in seconds, or a :class:`~datetime.timedelta` object) for when after
  418. stored task tombstones will be deleted.
  419. A built-in periodic task will delete the results after this time
  420. (``celery.backend_cleanup``), assuming that ``celery beat`` is
  421. enabled. The task runs daily at 4am.
  422. A value of :const:`None` or 0 means results will never expire (depending
  423. on backend specifications).
  424. Default is to expire after 1 day.
  425. .. note::
  426. For the moment this only works with the amqp, database, cache, redis and MongoDB
  427. backends.
  428. When using the database or MongoDB backends, `celery beat` must be
  429. running for the results to be expired.
  430. .. setting:: result_cache_max
  431. result_cache_max
  432. ~~~~~~~~~~~~~~~~
  433. Enables client caching of results, which can be useful for the old 'amqp'
  434. backend where the result is unavailable as soon as one result instance
  435. consumes it.
  436. This is the total number of results to cache before older results are evicted.
  437. A value of 0 or None means no limit, and a value of :const:`-1`
  438. will disable the cache.
  439. Disabled by default.
  440. .. _conf-database-result-backend:
  441. Database backend settings
  442. -------------------------
  443. Database URL Examples
  444. ~~~~~~~~~~~~~~~~~~~~~
  445. To use the database backend you have to configure the
  446. :setting:`result_backend` setting with a connection URL and the ``db+``
  447. prefix:
  448. .. code-block:: python
  449. result_backend = 'db+scheme://user:password@host:port/dbname'
  450. Examples::
  451. # sqlite (filename)
  452. result_backend = 'db+sqlite:///results.sqlite'
  453. # mysql
  454. result_backend = 'db+mysql://scott:tiger@localhost/foo'
  455. # postgresql
  456. result_backend = 'db+postgresql://scott:tiger@localhost/mydatabase'
  457. # oracle
  458. result_backend = 'db+oracle://scott:tiger@127.0.0.1:1521/sidname'
  459. .. code-block:: python
  460. Please see `Supported Databases`_ for a table of supported databases,
  461. and `Connection String`_ for more information about connection
  462. strings (which is the part of the URI that comes after the ``db+`` prefix).
  463. .. _`Supported Databases`:
  464. http://www.sqlalchemy.org/docs/core/engines.html#supported-databases
  465. .. _`Connection String`:
  466. http://www.sqlalchemy.org/docs/core/engines.html#database-urls
  467. .. setting:: sqlalchemy_dburi
  468. sqlalchemy_dburi
  469. ~~~~~~~~~~~~~~~~
  470. This setting is no longer used as it's now possible to specify
  471. the database URL directly in the :setting:`result_backend` setting.
  472. .. setting:: sqlalchemy_engine_options
  473. sqlalchemy_engine_options
  474. ~~~~~~~~~~~~~~~~~~~~~~~~~
  475. To specify additional SQLAlchemy database engine options you can use
  476. the :setting:`sqlalchmey_engine_options` setting::
  477. # echo enables verbose logging from SQLAlchemy.
  478. sqlalchemy_engine_options = {'echo': True}
  479. .. setting:: sqlalchemy_short_lived_sessions
  480. sqlalchemy_short_lived_sessions
  481. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  482. sqlalchemy_short_lived_sessions = True
  483. Short lived sessions are disabled by default. If enabled they can drastically reduce
  484. performance, especially on systems processing lots of tasks. This option is useful
  485. on low-traffic workers that experience errors as a result of cached database connections
  486. going stale through inactivity. For example, intermittent errors like
  487. `(OperationalError) (2006, 'MySQL server has gone away')` can be fixed by enabling
  488. short lived sessions. This option only affects the database backend.
  489. .. setting:: sqlalchemy_table_names
  490. sqlalchemy_table_names
  491. ~~~~~~~~~~~~~~~~~~~~~~
  492. When SQLAlchemy is configured as the result backend, Celery automatically
  493. creates two tables to store result metadata for tasks. This setting allows
  494. you to customize the table names:
  495. .. code-block:: python
  496. # use custom table names for the database result backend.
  497. sqlalchemy_table_names = {
  498. 'task': 'myapp_taskmeta',
  499. 'group': 'myapp_groupmeta',
  500. }
  501. .. _conf-rpc-result-backend:
  502. RPC backend settings
  503. --------------------
  504. .. setting:: result_persistent
  505. result_persistent
  506. ~~~~~~~~~~~~~~~~~
  507. If set to :const:`True`, result messages will be persistent. This means the
  508. messages will not be lost after a broker restart. The default is for the
  509. results to be transient.
  510. Example configuration
  511. ~~~~~~~~~~~~~~~~~~~~~
  512. .. code-block:: python
  513. result_backend = 'rpc://'
  514. result_persistent = False
  515. .. _conf-cache-result-backend:
  516. Cache backend settings
  517. ----------------------
  518. .. note::
  519. The cache backend supports the `pylibmc`_ and `python-memcached`
  520. libraries. The latter is used only if `pylibmc`_ is not installed.
  521. Using a single memcached server:
  522. .. code-block:: python
  523. result_backend = 'cache+memcached://127.0.0.1:11211/'
  524. Using multiple memcached servers:
  525. .. code-block:: python
  526. result_backend = """
  527. cache+memcached://172.19.26.240:11211;172.19.26.242:11211/
  528. """.strip()
  529. The "memory" backend stores the cache in memory only:
  530. .. code-block:: python
  531. result_backend = 'cache'
  532. cache_backend = 'memory'
  533. .. setting:: cache_backend_options
  534. cache_backend_options
  535. ~~~~~~~~~~~~~~~~~~~~~
  536. You can set pylibmc options using the :setting:`cache_backend_options`
  537. setting:
  538. .. code-block:: python
  539. cache_backend_options = {
  540. 'binary': True,
  541. 'behaviors': {'tcp_nodelay': True},
  542. }
  543. .. _`pylibmc`: http://sendapatch.se/projects/pylibmc/
  544. .. setting:: cache_backend
  545. cache_backend
  546. ~~~~~~~~~~~~~
  547. This setting is no longer used as it's now possible to specify
  548. the cache backend directly in the :setting:`result_backend` setting.
  549. .. _conf-redis-result-backend:
  550. Redis backend settings
  551. ----------------------
  552. Configuring the backend URL
  553. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  554. .. note::
  555. The Redis backend requires the :mod:`redis` library:
  556. http://pypi.python.org/pypi/redis/
  557. To install the redis package use `pip` or `easy_install`:
  558. .. code-block:: console
  559. $ pip install redis
  560. This backend requires the :setting:`result_backend`
  561. setting to be set to a Redis URL::
  562. result_backend = 'redis://:password@host:port/db'
  563. For example::
  564. result_backend = 'redis://localhost/0'
  565. which is the same as::
  566. result_backend = 'redis://'
  567. The fields of the URL are defined as follows:
  568. - *host*
  569. Host name or IP address of the Redis server. e.g. `localhost`.
  570. - *port*
  571. Port to the Redis server. Default is 6379.
  572. - *db*
  573. Database number to use. Default is 0.
  574. The db can include an optional leading slash.
  575. - *password*
  576. Password used to connect to the database.
  577. .. setting:: redis_max_connections
  578. redis_max_connections
  579. ~~~~~~~~~~~~~~~~~~~~~
  580. Maximum number of connections available in the Redis connection
  581. pool used for sending and retrieving results.
  582. .. setting:: redis_socket_timeout
  583. redis_socket_timeout
  584. ~~~~~~~~~~~~~~~~~~~~
  585. Socket timeout for connections to Redis from the result backend
  586. in seconds (int/float)
  587. Default is 5 seconds.
  588. .. _conf-mongodb-result-backend:
  589. MongoDB backend settings
  590. ------------------------
  591. .. note::
  592. The MongoDB backend requires the :mod:`pymongo` library:
  593. http://github.com/mongodb/mongo-python-driver/tree/master
  594. .. setting:: mongodb_backend_settings
  595. mongodb_backend_settings
  596. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  597. This is a dict supporting the following keys:
  598. * database
  599. The database name to connect to. Defaults to ``celery``.
  600. * taskmeta_collection
  601. The collection name to store task meta data.
  602. Defaults to ``celery_taskmeta``.
  603. * max_pool_size
  604. Passed as max_pool_size to PyMongo's Connection or MongoClient
  605. constructor. It is the maximum number of TCP connections to keep
  606. open to MongoDB at a given time. If there are more open connections
  607. than max_pool_size, sockets will be closed when they are released.
  608. Defaults to 10.
  609. * options
  610. Additional keyword arguments to pass to the mongodb connection
  611. constructor. See the :mod:`pymongo` docs to see a list of arguments
  612. supported.
  613. .. _example-mongodb-result-config:
  614. Example configuration
  615. ~~~~~~~~~~~~~~~~~~~~~
  616. .. code-block:: python
  617. result_backend = 'mongodb://192.168.1.100:30000/'
  618. mongodb_backend_settings = {
  619. 'database': 'mydb',
  620. 'taskmeta_collection': 'my_taskmeta_collection',
  621. }
  622. .. _conf-cassandra-result-backend:
  623. cassandra backend settings
  624. --------------------------
  625. .. note::
  626. This Cassandra backend driver requires :mod:`cassandra-driver`.
  627. https://pypi.python.org/pypi/cassandra-driver
  628. To install, use `pip` or `easy_install`:
  629. .. code-block:: console
  630. $ pip install cassandra-driver
  631. This backend requires the following configuration directives to be set.
  632. .. setting:: cassandra_servers
  633. cassandra_servers
  634. ~~~~~~~~~~~~~~~~~
  635. List of ``host`` Cassandra servers. e.g.::
  636. cassandra_servers = ['localhost']
  637. .. setting:: cassandra_port
  638. cassandra_port
  639. ~~~~~~~~~~~~~~
  640. Port to contact the Cassandra servers on. Default is 9042.
  641. .. setting:: cassandra_keyspace
  642. cassandra_keyspace
  643. ~~~~~~~~~~~~~~~~~~
  644. The keyspace in which to store the results. e.g.::
  645. cassandra_keyspace = 'tasks_keyspace'
  646. .. setting:: cassandra_table
  647. cassandra_table
  648. ~~~~~~~~~~~~~~~
  649. The table (column family) in which to store the results. e.g.::
  650. cassandra_table = 'tasks'
  651. .. setting:: cassandra_read_consistency
  652. cassandra_read_consistency
  653. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  654. The read consistency used. Values can be ``ONE``, ``TWO``, ``THREE``, ``QUORUM``, ``ALL``,
  655. ``LOCAL_QUORUM``, ``EACH_QUORUM``, ``LOCAL_ONE``.
  656. .. setting:: cassandra_write_consistency
  657. cassandra_write_consistency
  658. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  659. The write consistency used. Values can be ``ONE``, ``TWO``, ``THREE``, ``QUORUM``, ``ALL``,
  660. ``LOCAL_QUORUM``, ``EACH_QUORUM``, ``LOCAL_ONE``.
  661. .. setting:: cassandra_entry_ttl
  662. cassandra_entry_ttl
  663. ~~~~~~~~~~~~~~~~~~~
  664. Time-to-live for status entries. They will expire and be removed after that many seconds
  665. after adding. Default (None) means they will never expire.
  666. .. setting:: cassandra_auth_provider
  667. cassandra_auth_provider
  668. ~~~~~~~~~~~~~~~~~~~~~~~
  669. AuthProvider class within ``cassandra.auth`` module to use. Values can be
  670. ``PlainTextAuthProvider`` or ``SaslAuthProvider``.
  671. .. setting:: cassandra_auth_kwargs
  672. cassandra_auth_kwargs
  673. ~~~~~~~~~~~~~~~~~~~~~
  674. Named arguments to pass into the auth provider. e.g.::
  675. cassandra_auth_kwargs = {
  676. username: 'cassandra',
  677. password: 'cassandra'
  678. }
  679. Example configuration
  680. ~~~~~~~~~~~~~~~~~~~~~
  681. .. code-block:: python
  682. cassandra_servers = ['localhost']
  683. cassandra_keyspace = 'celery'
  684. cassandra_table = 'tasks'
  685. cassandra_read_consistency = 'ONE'
  686. cassandra_write_consistency = 'ONE'
  687. cassandra_entry_ttl = 86400
  688. .. _conf-elasticsearch-result-backend:
  689. Elasticsearch backend settings
  690. ------------------------------
  691. To use `Elasticsearch`_ as the result backend you simply need to
  692. configure the :setting:`result_backend` setting with the correct URL.
  693. Example configuration
  694. ~~~~~~~~~~~~~~~~~~~~~
  695. .. code-block:: python
  696. result_backend = 'elasticsearch://example.com:9200/index_name/doc_type'
  697. .. _conf-riak-result-backend:
  698. Riak backend settings
  699. ---------------------
  700. .. note::
  701. The Riak backend requires the :mod:`riak` library:
  702. http://pypi.python.org/pypi/riak/
  703. To install the riak package use `pip` or `easy_install`:
  704. .. code-block:: console
  705. $ pip install riak
  706. This backend requires the :setting:`result_backend`
  707. setting to be set to a Riak URL::
  708. result_backend = 'riak://host:port/bucket'
  709. For example::
  710. result_backend = 'riak://localhost/celery
  711. which is the same as::
  712. result_backend = 'riak://'
  713. The fields of the URL are defined as follows:
  714. - *host*
  715. Host name or IP address of the Riak server. e.g. `'localhost'`.
  716. - *port*
  717. Port to the Riak server using the protobuf protocol. Default is 8087.
  718. - *bucket*
  719. Bucket name to use. Default is `celery`.
  720. The bucket needs to be a string with ascii characters only.
  721. Altenatively, this backend can be configured with the following configuration directives.
  722. .. setting:: riak_backend_settings
  723. riak_backend_settings
  724. ~~~~~~~~~~~~~~~~~~~~~
  725. This is a dict supporting the following keys:
  726. * host
  727. The host name of the Riak server. Defaults to "localhost".
  728. * port
  729. The port the Riak server is listening to. Defaults to 8087.
  730. * bucket
  731. The bucket name to connect to. Defaults to "celery".
  732. * protocol
  733. The protocol to use to connect to the Riak server. This is not configurable
  734. via :setting:`result_backend`
  735. .. _conf-ironcache-result-backend:
  736. IronCache backend settings
  737. --------------------------
  738. .. note::
  739. The IronCache backend requires the :mod:`iron_celery` library:
  740. http://pypi.python.org/pypi/iron_celery
  741. To install the iron_celery package use `pip` or `easy_install`:
  742. .. code-block:: console
  743. $ pip install iron_celery
  744. IronCache is configured via the URL provided in :setting:`result_backend`, for example::
  745. result_backend = 'ironcache://project_id:token@'
  746. Or to change the cache name::
  747. ironcache:://project_id:token@/awesomecache
  748. For more information, see: https://github.com/iron-io/iron_celery
  749. .. _conf-couchbase-result-backend:
  750. Couchbase backend settings
  751. --------------------------
  752. .. note::
  753. The Couchbase backend requires the :mod:`couchbase` library:
  754. https://pypi.python.org/pypi/couchbase
  755. To install the couchbase package use `pip` or `easy_install`:
  756. .. code-block:: console
  757. $ pip install couchbase
  758. This backend can be configured via the :setting:`result_backend`
  759. set to a couchbase URL::
  760. result_backend = 'couchbase://username:password@host:port/bucket'
  761. .. setting:: couchbase_backend_settings
  762. couchbase_backend_settings
  763. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  764. This is a dict supporting the following keys:
  765. * host
  766. Host name of the Couchbase server. Defaults to ``localhost``.
  767. * port
  768. The port the Couchbase server is listening to. Defaults to ``8091``.
  769. * bucket
  770. The default bucket the Couchbase server is writing to.
  771. Defaults to ``default``.
  772. * username
  773. User name to authenticate to the Couchbase server as (optional).
  774. * password
  775. Password to authenticate to the Couchbase server (optional).
  776. .. _conf-couchdb-result-backend:
  777. CouchDB backend settings
  778. ------------------------
  779. .. note::
  780. The CouchDB backend requires the :mod:`pycouchdb` library:
  781. https://pypi.python.org/pypi/pycouchdb
  782. To install the couchbase package use `pip` or `easy_install`:
  783. .. code-block:: console
  784. $ pip install pycouchdb
  785. This backend can be configured via the :setting:`result_backend`
  786. set to a couchdb URL::
  787. result_backend = 'couchdb://username:password@host:port/container'
  788. The URL is formed out of the following parts:
  789. * username
  790. User name to authenticate to the CouchDB server as (optional).
  791. * password
  792. Password to authenticate to the CouchDB server (optional).
  793. * host
  794. Host name of the CouchDB server. Defaults to ``localhost``.
  795. * port
  796. The port the CouchDB server is listening to. Defaults to ``8091``.
  797. * container
  798. The default container the CouchDB server is writing to.
  799. Defaults to ``default``.
  800. .. _conf-amqp-result-backend:
  801. AMQP backend settings
  802. ---------------------
  803. .. admonition:: Do not use in production.
  804. This is the old AMQP result backend that creates one queue per task,
  805. if you want to send results back as message please consider using the
  806. RPC backend instead, or if you need the results to be persistent
  807. use a result backend designed for that purpose (e.g. Redis, or a database).
  808. .. note::
  809. The AMQP backend requires RabbitMQ 1.1.0 or higher to automatically
  810. expire results. If you are running an older version of RabbitMQ
  811. you should disable result expiration like this:
  812. result_expires = None
  813. .. setting:: result_exchange
  814. result_exchange
  815. ~~~~~~~~~~~~~~~
  816. Name of the exchange to publish results in. Default is `celeryresults`.
  817. .. setting:: result_exchange_type
  818. result_exchange_type
  819. ~~~~~~~~~~~~~~~~~~~~
  820. The exchange type of the result exchange. Default is to use a `direct`
  821. exchange.
  822. result_persistent
  823. ~~~~~~~~~~~~~~~~~
  824. If set to :const:`True`, result messages will be persistent. This means the
  825. messages will not be lost after a broker restart. The default is for the
  826. results to be transient.
  827. Example configuration
  828. ~~~~~~~~~~~~~~~~~~~~~
  829. .. code-block:: python
  830. result_backend = 'amqp'
  831. result_expires = 18000 # 5 hours.
  832. .. _conf-filesystem-result-backend:
  833. Filesystem backend settings
  834. ---------------------------
  835. This backend can be configured using a file URL, for example::
  836. CELERY_RESULT_BACKEND = 'file:///var/celery/results'
  837. The configured directory needs to be shared and writeable by all servers using
  838. the backend.
  839. If you are trying Celery on a single system you can simply use the backend
  840. without any further configuration. For larger clusters you could use NFS,
  841. `GlusterFS`_, CIFS, `HDFS`_ (using FUSE) or any other filesystem.
  842. .. _`GlusterFS`: http://www.gluster.org/
  843. .. _`HDFS`: http://hadoop.apache.org/
  844. .. _conf-messaging:
  845. Message Routing
  846. ---------------
  847. .. _conf-messaging-routing:
  848. .. setting:: task_queues
  849. task_queues
  850. ~~~~~~~~~~~
  851. Most users will not want to specify this setting and should rather use
  852. the :ref:`automatic routing facilities <routing-automatic>`.
  853. If you really want to configure advanced routing, this setting should
  854. be a list of :class:`kombu.Queue` objects the worker will consume from.
  855. Note that workers can be overriden this setting via the `-Q` option,
  856. or individual queues from this list (by name) can be excluded using
  857. the `-X` option.
  858. Also see :ref:`routing-basics` for more information.
  859. The default is a queue/exchange/binding key of ``celery``, with
  860. exchange type ``direct``.
  861. See also :setting:`task_routes`
  862. .. setting:: task_routes
  863. task_routes
  864. ~~~~~~~~~~~~~
  865. A list of routers, or a single router used to route tasks to queues.
  866. When deciding the final destination of a task the routers are consulted
  867. in order.
  868. A router can be specified as either:
  869. * A router class instance.
  870. * A string which provides the path to a router class
  871. * A dict containing router specification:
  872. Will be converted to a :class:`celery.routes.MapRoute` instance.
  873. * A list of ``(pattern, route)`` tuples:
  874. Will be converted to a :class:`celery.routes.MapRoute` instance.
  875. Examples:
  876. .. code-block:: python
  877. task_routes = {
  878. 'celery.ping': 'default',
  879. 'mytasks.add': 'cpu-bound',
  880. 'feed.tasks.*': 'feeds', # <-- glob pattern
  881. re.compile(r'(image|video)\.tasks\..*'): 'media', # <-- regex
  882. 'video.encode': {
  883. 'queue': 'video',
  884. 'exchange': 'media'
  885. 'routing_key': 'media.video.encode',
  886. },
  887. }
  888. task_routes = ('myapp.tasks.Router', {'celery.ping': 'default})
  889. Where ``myapp.tasks.Router`` could be:
  890. .. code-block:: python
  891. class Router(object):
  892. def route_for_task(self, task, args=None, kwargs=None):
  893. if task == 'celery.ping':
  894. return {'queue': 'default'}
  895. ``route_for_task`` may return a string or a dict. A string then means
  896. it's a queue name in :setting:`task_queues`, a dict means it's a custom route.
  897. When sending tasks, the routers are consulted in order. The first
  898. router that doesn't return ``None`` is the route to use. The message options
  899. is then merged with the found route settings, where the routers settings
  900. have priority.
  901. Example if :func:`~celery.execute.apply_async` has these arguments:
  902. .. code-block:: python
  903. Task.apply_async(immediate=False, exchange='video',
  904. routing_key='video.compress')
  905. and a router returns:
  906. .. code-block:: python
  907. {'immediate': True, 'exchange': 'urgent'}
  908. the final message options will be:
  909. .. code-block:: python
  910. immediate=True, exchange='urgent', routing_key='video.compress'
  911. (and any default message options defined in the
  912. :class:`~celery.task.base.Task` class)
  913. Values defined in :setting:`task_routes` have precedence over values defined in
  914. :setting:`task_queues` when merging the two.
  915. With the follow settings:
  916. .. code-block:: python
  917. task_queues = {
  918. 'cpubound': {
  919. 'exchange': 'cpubound',
  920. 'routing_key': 'cpubound',
  921. },
  922. }
  923. task_routes = {
  924. 'tasks.add': {
  925. 'queue': 'cpubound',
  926. 'routing_key': 'tasks.add',
  927. 'serializer': 'json',
  928. },
  929. }
  930. The final routing options for ``tasks.add`` will become:
  931. .. code-block:: javascript
  932. {'exchange': 'cpubound',
  933. 'routing_key': 'tasks.add',
  934. 'serializer': 'json'}
  935. See :ref:`routers` for more examples.
  936. .. setting:: task_queue_ha_policy
  937. task_queue_ha_policy
  938. ~~~~~~~~~~~~~~~~~~~~
  939. :brokers: RabbitMQ
  940. This will set the default HA policy for a queue, and the value
  941. can either be a string (usually ``all``):
  942. .. code-block:: python
  943. task_queue_ha_policy = 'all'
  944. Using 'all' will replicate the queue to all current nodes,
  945. Or you can give it a list of nodes to replicate to:
  946. .. code-block:: python
  947. task_queue_ha_policy = ['rabbit@host1', 'rabbit@host2']
  948. Using a list will implicitly set ``x-ha-policy`` to 'nodes' and
  949. ``x-ha-policy-params`` to the given list of nodes.
  950. See http://www.rabbitmq.com/ha.html for more information.
  951. .. setting:: worker_direct
  952. worker_direct
  953. ~~~~~~~~~~~~~
  954. This option enables so that every worker has a dedicated queue,
  955. so that tasks can be routed to specific workers.
  956. The queue name for each worker is automatically generated based on
  957. the worker hostname and a ``.dq`` suffix, using the ``C.dq`` exchange.
  958. For example the queue name for the worker with node name ``w1@example.com``
  959. becomes::
  960. w1@example.com.dq
  961. Then you can route the task to the task by specifying the hostname
  962. as the routing key and the ``C.dq`` exchange::
  963. task_routes = {
  964. 'tasks.add': {'exchange': 'C.dq', 'routing_key': 'w1@example.com'}
  965. }
  966. .. setting:: task_create_missing_queues
  967. task_create_missing_queues
  968. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  969. If enabled (default), any queues specified that are not defined in
  970. :setting:`task_queues` will be automatically created. See
  971. :ref:`routing-automatic`.
  972. .. setting:: task_default_queue
  973. task_default_queue
  974. ~~~~~~~~~~~~~~~~~~
  975. The name of the default queue used by `.apply_async` if the message has
  976. no route or no custom queue has been specified.
  977. This queue must be listed in :setting:`task_queues`.
  978. If :setting:`task_queues` is not specified then it is automatically
  979. created containing one queue entry, where this name is used as the name of
  980. that queue.
  981. The default is: `celery`.
  982. .. seealso::
  983. :ref:`routing-changing-default-queue`
  984. .. setting:: task_default_exchange
  985. task_default_exchange
  986. ~~~~~~~~~~~~~~~~~~~~~
  987. Name of the default exchange to use when no custom exchange is
  988. specified for a key in the :setting:`task_queues` setting.
  989. The default is: `celery`.
  990. .. setting:: task_default_exchange_type
  991. task_default_exchange_type
  992. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  993. Default exchange type used when no custom exchange type is specified
  994. for a key in the :setting:`task_queues` setting.
  995. The default is: `direct`.
  996. .. setting:: task_default_routing_key
  997. task_default_routing_key
  998. ~~~~~~~~~~~~~~~~~~~~~~~~
  999. The default routing key used when no custom routing key
  1000. is specified for a key in the :setting:`task_queues` setting.
  1001. The default is: `celery`.
  1002. .. setting:: task_default_delivery_mode
  1003. task_default_delivery_mode
  1004. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1005. Can be `transient` or `persistent`. The default is to send
  1006. persistent messages.
  1007. .. _conf-broker-settings:
  1008. Broker Settings
  1009. ---------------
  1010. .. setting:: broker_url
  1011. broker_url
  1012. ~~~~~~~~~~
  1013. Default broker URL. This must be an URL in the form of::
  1014. transport://userid:password@hostname:port/virtual_host
  1015. Only the scheme part (``transport://``) is required, the rest
  1016. is optional, and defaults to the specific transports default values.
  1017. The transport part is the broker implementation to use, and the
  1018. default is ``amqp``, which uses ``librabbitmq`` by default or falls back to
  1019. ``pyamqp`` if that is not installed. Also there are many other choices including
  1020. ``redis``, ``beanstalk``, ``sqlalchemy``, ``django``, ``mongodb``,
  1021. ``couchdb``.
  1022. It can also be a fully qualified path to your own transport implementation.
  1023. More than broker URL, of the same transport, can also be specified.
  1024. The broker URLs can be passed in as a single string that is semicolon delimited::
  1025. broker_url = 'transport://userid:password@hostname:port//;transport://userid:password@hostname:port//'
  1026. Or as a list::
  1027. broker_url = [
  1028. 'transport://userid:password@localhost:port//',
  1029. 'transport://userid:password@hostname:port//'
  1030. ]
  1031. The brokers will then be used in the :setting:`broker_failover_strategy`.
  1032. See :ref:`kombu:connection-urls` in the Kombu documentation for more
  1033. information.
  1034. .. setting:: broker_read_url
  1035. .. setting:: broker_write_url
  1036. broker_read_url / broker_write_url
  1037. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1038. These settings can be configured, instead of :setting:`broker_url` to specify
  1039. different connection parameters for broker connections used for consuming and
  1040. producing.
  1041. Example::
  1042. broker_read_url = 'amqp://user:pass@broker.example.com:56721'
  1043. broker_write_url = 'amqp://user:pass@broker.example.com:56722'
  1044. Both options can also be specified as a list for failover alternates, see
  1045. :setting:`broker_url` for more information.
  1046. .. setting:: broker_failover_strategy
  1047. broker_failover_strategy
  1048. ~~~~~~~~~~~~~~~~~~~~~~~~
  1049. Default failover strategy for the broker Connection object. If supplied,
  1050. may map to a key in 'kombu.connection.failover_strategies', or be a reference
  1051. to any method that yields a single item from a supplied list.
  1052. Example::
  1053. # Random failover strategy
  1054. def random_failover_strategy(servers):
  1055. it = list(it) # don't modify callers list
  1056. shuffle = random.shuffle
  1057. for _ in repeat(None):
  1058. shuffle(it)
  1059. yield it[0]
  1060. broker_failover_strategy = random_failover_strategy
  1061. .. setting:: broker_heartbeat
  1062. broker_heartbeat
  1063. ~~~~~~~~~~~~~~~~
  1064. :transports supported: ``pyamqp``
  1065. It's not always possible to detect connection loss in a timely
  1066. manner using TCP/IP alone, so AMQP defines something called heartbeats
  1067. that's is used both by the client and the broker to detect if
  1068. a connection was closed.
  1069. Heartbeats are disabled by default.
  1070. If the heartbeat value is 10 seconds, then
  1071. the heartbeat will be monitored at the interval specified
  1072. by the :setting:`broker_heartbeat_checkrate` setting, which by default is
  1073. double the rate of the heartbeat value
  1074. (so for the default 10 seconds, the heartbeat is checked every 5 seconds).
  1075. .. setting:: broker_heartbeat_checkrate
  1076. broker_heartbeat_checkrate
  1077. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1078. :transports supported: ``pyamqp``
  1079. At intervals the worker will monitor that the broker has not missed
  1080. too many heartbeats. The rate at which this is checked is calculated
  1081. by dividing the :setting:`broker_heartbeat` value with this value,
  1082. so if the heartbeat is 10.0 and the rate is the default 2.0, the check
  1083. will be performed every 5 seconds (twice the heartbeat sending rate).
  1084. .. setting:: broker_use_ssl
  1085. broker_use_ssl
  1086. ~~~~~~~~~~~~~~
  1087. :transports supported: ``pyamqp``, ``redis``
  1088. Toggles SSL usage on broker connection and SSL settings.
  1089. If ``True`` the connection will use SSL with default SSL settings.
  1090. If set to a dict, will configure SSL connection according to the specified
  1091. policy. The format used is python `ssl.wrap_socket()
  1092. options <https://docs.python.org/3/library/ssl.html#ssl.wrap_socket>`_.
  1093. Default is ``False`` (no SSL).
  1094. Note that SSL socket is generally served on a separate port by the broker.
  1095. Example providing a client cert and validating the server cert against a custom
  1096. certificate authority:
  1097. .. code-block:: python
  1098. import ssl
  1099. broker_use_ssl = {
  1100. 'keyfile': '/var/ssl/private/worker-key.pem',
  1101. 'certfile': '/var/ssl/amqp-server-cert.pem',
  1102. 'ca_certs': '/var/ssl/myca.pem',
  1103. 'cert_reqs': ssl.CERT_REQUIRED
  1104. }
  1105. .. warning::
  1106. Be careful using ``broker_use_ssl=True``. It is possible that your default
  1107. configuration will not validate the server cert at all. Please read Python
  1108. `ssl module security
  1109. considerations <https://docs.python.org/3/library/ssl.html#ssl-security>`_.
  1110. .. setting:: broker_pool_limit
  1111. broker_pool_limit
  1112. ~~~~~~~~~~~~~~~~~
  1113. .. versionadded:: 2.3
  1114. The maximum number of connections that can be open in the connection pool.
  1115. The pool is enabled by default since version 2.5, with a default limit of ten
  1116. connections. This number can be tweaked depending on the number of
  1117. threads/greenthreads (eventlet/gevent) using a connection. For example
  1118. running eventlet with 1000 greenlets that use a connection to the broker,
  1119. contention can arise and you should consider increasing the limit.
  1120. If set to :const:`None` or 0 the connection pool will be disabled and
  1121. connections will be established and closed for every use.
  1122. Default (since 2.5) is to use a pool of 10 connections.
  1123. .. setting:: broker_connection_timeout
  1124. broker_connection_timeout
  1125. ~~~~~~~~~~~~~~~~~~~~~~~~~
  1126. The default timeout in seconds before we give up establishing a connection
  1127. to the AMQP server. Default is 4 seconds.
  1128. .. setting:: broker_connection_retry
  1129. broker_connection_retry
  1130. ~~~~~~~~~~~~~~~~~~~~~~~
  1131. Automatically try to re-establish the connection to the AMQP broker if lost.
  1132. The time between retries is increased for each retry, and is
  1133. not exhausted before :setting:`broker_connection_max_retries` is
  1134. exceeded.
  1135. This behavior is on by default.
  1136. .. setting:: broker_connection_max_retries
  1137. broker_connection_max_retries
  1138. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1139. Maximum number of retries before we give up re-establishing a connection
  1140. to the AMQP broker.
  1141. If this is set to :const:`0` or :const:`None`, we will retry forever.
  1142. Default is 100 retries.
  1143. .. setting:: broker_login_method
  1144. broker_login_method
  1145. ~~~~~~~~~~~~~~~~~~~
  1146. Set custom amqp login method, default is ``AMQPLAIN``.
  1147. .. setting:: broker_transport_options
  1148. broker_transport_options
  1149. ~~~~~~~~~~~~~~~~~~~~~~~~
  1150. .. versionadded:: 2.2
  1151. A dict of additional options passed to the underlying transport.
  1152. See your transport user manual for supported options (if any).
  1153. Example setting the visibility timeout (supported by Redis and SQS
  1154. transports):
  1155. .. code-block:: python
  1156. broker_transport_options = {'visibility_timeout': 18000} # 5 hours
  1157. .. _conf-worker:
  1158. Worker
  1159. ------
  1160. .. setting:: imports
  1161. imports
  1162. ~~~~~~~
  1163. A sequence of modules to import when the worker starts.
  1164. This is used to specify the task modules to import, but also
  1165. to import signal handlers and additional remote control commands, etc.
  1166. The modules will be imported in the original order.
  1167. .. setting:: include
  1168. include
  1169. ~~~~~~~
  1170. Exact same semantics as :setting:`imports`, but can be used as a means
  1171. to have different import categories.
  1172. The modules in this setting are imported after the modules in
  1173. :setting:`imports`.
  1174. .. _conf-concurrency:
  1175. .. setting:: worker_concurrency
  1176. worker_concurrency
  1177. ~~~~~~~~~~~~~~~~~~
  1178. The number of concurrent worker processes/threads/green threads executing
  1179. tasks.
  1180. If you're doing mostly I/O you can have more processes,
  1181. but if mostly CPU-bound, try to keep it close to the
  1182. number of CPUs on your machine. If not set, the number of CPUs/cores
  1183. on the host will be used.
  1184. Defaults to the number of available CPUs.
  1185. .. setting:: worker_prefetch_multiplier
  1186. worker_prefetch_multiplier
  1187. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1188. How many messages to prefetch at a time multiplied by the number of
  1189. concurrent processes. The default is 4 (four messages for each
  1190. process). The default setting is usually a good choice, however -- if you
  1191. have very long running tasks waiting in the queue and you have to start the
  1192. workers, note that the first worker to start will receive four times the
  1193. number of messages initially. Thus the tasks may not be fairly distributed
  1194. to the workers.
  1195. To disable prefetching, set :setting:`worker_prefetch_multiplier` to 1.
  1196. Changing that setting to 0 will allow the worker to keep consuming
  1197. as many messages as it wants.
  1198. For more on prefetching, read :ref:`optimizing-prefetch-limit`
  1199. .. note::
  1200. Tasks with ETA/countdown are not affected by prefetch limits.
  1201. .. setting:: worker_lost_wait
  1202. worker_lost_wait
  1203. ~~~~~~~~~~~~~~~~
  1204. In some cases a worker may be killed without proper cleanup,
  1205. and the worker may have published a result before terminating.
  1206. This value specifies how long we wait for any missing results before
  1207. raising a :exc:`@WorkerLostError` exception.
  1208. Default is 10.0
  1209. .. setting:: worker_max_tasks_per_child
  1210. worker_max_tasks_per_child
  1211. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1212. Maximum number of tasks a pool worker process can execute before
  1213. it's replaced with a new one. Default is no limit.
  1214. .. setting:: worker_max_memory_per_child
  1215. worker_max_memory_per_child
  1216. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1217. Maximum amount of resident memory that may be consumed by a
  1218. worker before it will be replaced by a new worker. If a single
  1219. task causes a worker to exceed this limit, the task will be
  1220. completed, and the worker will be replaced afterwards. Default:
  1221. no limit.
  1222. .. setting:: worker_disable_rate_limits
  1223. worker_disable_rate_limits
  1224. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1225. Disable all rate limits, even if tasks has explicit rate limits set.
  1226. .. setting:: worker_state_db
  1227. worker_state_db
  1228. ~~~~~~~~~~~~~~~
  1229. Name of the file used to stores persistent worker state (like revoked tasks).
  1230. Can be a relative or absolute path, but be aware that the suffix `.db`
  1231. may be appended to the file name (depending on Python version).
  1232. Can also be set via the :option:`--statedb` argument to
  1233. :mod:`~celery.bin.worker`.
  1234. Not enabled by default.
  1235. .. setting:: worker_timer_precision
  1236. worker_timer_precision
  1237. ~~~~~~~~~~~~~~~~~~~~~~
  1238. Set the maximum time in seconds that the ETA scheduler can sleep between
  1239. rechecking the schedule. Default is 1 second.
  1240. Setting this value to 1 second means the schedulers precision will
  1241. be 1 second. If you need near millisecond precision you can set this to 0.1.
  1242. .. setting:: worker_enable_remote_control
  1243. worker_enable_remote_control
  1244. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1245. Specify if remote control of the workers is enabled.
  1246. Default is :const:`True`.
  1247. .. _conf-error-mails:
  1248. Error E-Mails
  1249. -------------
  1250. .. setting:: task_send_error_emails
  1251. task_send_error_emails
  1252. ~~~~~~~~~~~~~~~~~~~~~~
  1253. The default value for the `Task.send_error_emails` attribute, which if
  1254. set to :const:`True` means errors occurring during task execution will be
  1255. sent to :setting:`admins` by email.
  1256. Disabled by default.
  1257. .. setting:: admins
  1258. admins
  1259. ~~~~~~
  1260. List of `(name, email_address)` tuples for the administrators that should
  1261. receive error emails.
  1262. .. setting:: server_email
  1263. server_email
  1264. ~~~~~~~~~~~~
  1265. The email address this worker sends emails from.
  1266. Default is celery@localhost.
  1267. .. setting:: email_host
  1268. email_host
  1269. ~~~~~~~~~~
  1270. The mail server to use. Default is ``localhost``.
  1271. .. setting:: email_host_user
  1272. email_host_user
  1273. ~~~~~~~~~~~~~~~
  1274. User name (if required) to log on to the mail server with.
  1275. .. setting:: email_host_password
  1276. email_host_password
  1277. ~~~~~~~~~~~~~~~~~~~
  1278. Password (if required) to log on to the mail server with.
  1279. .. setting:: email_port
  1280. email_port
  1281. ~~~~~~~~~~
  1282. The port the mail server is listening on. Default is `25`.
  1283. .. setting:: email_use_ssl
  1284. email_use_ssl
  1285. ~~~~~~~~~~~~~
  1286. Use SSL when connecting to the SMTP server. Disabled by default.
  1287. .. setting:: email_use_tls
  1288. email_use_tls
  1289. ~~~~~~~~~~~~~
  1290. Use TLS when connecting to the SMTP server. Disabled by default.
  1291. .. setting:: email_timeout
  1292. email_timeout
  1293. ~~~~~~~~~~~~~
  1294. Timeout in seconds for when we give up trying to connect
  1295. to the SMTP server when sending emails.
  1296. The default is 2 seconds.
  1297. .. setting:: email_charset
  1298. email_charset
  1299. ~~~~~~~~~~~~~
  1300. .. versionadded:: 4.0
  1301. Charset for outgoing emails. Default is 'utf-8'.
  1302. .. _conf-example-error-mail-config:
  1303. Example E-Mail configuration
  1304. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1305. This configuration enables the sending of error emails to
  1306. george@vandelay.com and kramer@vandelay.com:
  1307. .. code-block:: python
  1308. # Enables error emails.
  1309. task_send_error_emails = True
  1310. # Name and email addresses of recipients
  1311. admins = (
  1312. ('George Costanza', 'george@vandelay.com'),
  1313. ('Cosmo Kramer', 'kosmo@vandelay.com'),
  1314. )
  1315. # Email address used as sender (From field).
  1316. server_email = 'no-reply@vandelay.com'
  1317. # Mailserver configuration
  1318. email_host = 'mail.vandelay.com'
  1319. email_port = 25
  1320. # email_host_user = 'servers'
  1321. # email_host_password = 's3cr3t'
  1322. .. _conf-events:
  1323. Events
  1324. ------
  1325. .. setting:: worker_send_task_events
  1326. worker_send_task_events
  1327. ~~~~~~~~~~~~~~~~~~~~~~~
  1328. Send task-related events so that tasks can be monitored using tools like
  1329. `flower`. Sets the default value for the workers :option:`-E` argument.
  1330. .. setting:: task_send_sent_event
  1331. task_send_sent_event
  1332. ~~~~~~~~~~~~~~~~~~~~
  1333. .. versionadded:: 2.2
  1334. If enabled, a :event:`task-sent` event will be sent for every task so tasks can be
  1335. tracked before they are consumed by a worker.
  1336. Disabled by default.
  1337. .. setting:: event_queue_ttl
  1338. event_queue_ttl
  1339. ~~~~~~~~~~~~~~~~~~~~~~
  1340. :transports supported: ``amqp``
  1341. Message expiry time in seconds (int/float) for when messages sent to a monitor clients
  1342. event queue is deleted (``x-message-ttl``)
  1343. For example, if this value is set to 10 then a message delivered to this queue
  1344. will be deleted after 10 seconds.
  1345. Disabled by default.
  1346. .. setting:: event_queue_expires
  1347. event_queue_expires
  1348. ~~~~~~~~~~~~~~~~~~~
  1349. :transports supported: ``amqp``
  1350. Expiry time in seconds (int/float) for when after a monitor clients
  1351. event queue will be deleted (``x-expires``).
  1352. Default is never, relying on the queue autodelete setting.
  1353. .. setting:: event_serializer
  1354. event_serializer
  1355. ~~~~~~~~~~~~~~~~
  1356. Message serialization format used when sending event messages.
  1357. Default is ``json``. See :ref:`calling-serializers`.
  1358. .. _conf-logging:
  1359. Logging
  1360. -------
  1361. .. setting:: worker_hijack_root_logger
  1362. worker_hijack_root_logger
  1363. ~~~~~~~~~~~~~~~~~~~~~~~~~
  1364. .. versionadded:: 2.2
  1365. By default any previously configured handlers on the root logger will be
  1366. removed. If you want to customize your own logging handlers, then you
  1367. can disable this behavior by setting
  1368. `worker_hijack_root_logger = False`.
  1369. .. note::
  1370. Logging can also be customized by connecting to the
  1371. :signal:`celery.signals.setup_logging` signal.
  1372. .. setting:: worker_log_color
  1373. worker_log_color
  1374. ~~~~~~~~~~~~~~~~~
  1375. Enables/disables colors in logging output by the Celery apps.
  1376. By default colors are enabled if
  1377. 1) the app is logging to a real terminal, and not a file.
  1378. 2) the app is not running on Windows.
  1379. .. setting:: worker_log_format
  1380. worker_log_format
  1381. ~~~~~~~~~~~~~~~~~
  1382. The format to use for log messages.
  1383. Default is::
  1384. [%(asctime)s: %(levelname)s/%(processName)s] %(message)s
  1385. See the Python :mod:`logging` module for more information about log
  1386. formats.
  1387. .. setting:: worker_task_log_format
  1388. worker_task_log_format
  1389. ~~~~~~~~~~~~~~~~~~~~~~
  1390. The format to use for log messages logged in tasks.
  1391. Default is::
  1392. [%(asctime)s: %(levelname)s/%(processName)s]
  1393. [%(task_name)s(%(task_id)s)] %(message)s
  1394. See the Python :mod:`logging` module for more information about log
  1395. formats.
  1396. .. setting:: worker_redirect_stdouts
  1397. worker_redirect_stdouts
  1398. ~~~~~~~~~~~~~~~~~~~~~~~
  1399. If enabled `stdout` and `stderr` will be redirected
  1400. to the current logger.
  1401. Enabled by default.
  1402. Used by :program:`celery worker` and :program:`celery beat`.
  1403. .. setting:: worker_redirect_stdouts_level
  1404. worker_redirect_stdouts_level
  1405. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1406. The log level output to `stdout` and `stderr` is logged as.
  1407. Can be one of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  1408. :const:`ERROR` or :const:`CRITICAL`.
  1409. Default is :const:`WARNING`.
  1410. .. _conf-security:
  1411. Security
  1412. --------
  1413. .. setting:: security_key
  1414. security_key
  1415. ~~~~~~~~~~~~
  1416. .. versionadded:: 2.5
  1417. The relative or absolute path to a file containing the private key
  1418. used to sign messages when :ref:`message-signing` is used.
  1419. .. setting:: security_certificate
  1420. security_certificate
  1421. ~~~~~~~~~~~~~~~~~~~~
  1422. .. versionadded:: 2.5
  1423. The relative or absolute path to an X.509 certificate file
  1424. used to sign messages when :ref:`message-signing` is used.
  1425. .. setting:: security_cert_store
  1426. security_cert_store
  1427. ~~~~~~~~~~~~~~~~~~~
  1428. .. versionadded:: 2.5
  1429. The directory containing X.509 certificates used for
  1430. :ref:`message-signing`. Can be a glob with wildcards,
  1431. (for example :file:`/etc/certs/*.pem`).
  1432. .. _conf-custom-components:
  1433. Custom Component Classes (advanced)
  1434. -----------------------------------
  1435. .. setting:: worker_pool
  1436. worker_pool
  1437. ~~~~~~~~~~~
  1438. Name of the pool class used by the worker.
  1439. .. admonition:: Eventlet/Gevent
  1440. Never use this option to select the eventlet or gevent pool.
  1441. You must use the `-P` option to :program:`celery worker` instead, to
  1442. ensure the monkey patches are not applied too late, causing things
  1443. to break in strange ways.
  1444. Default is ``celery.concurrency.prefork:TaskPool``.
  1445. .. setting:: worker_pool_restarts
  1446. worker_pool_restarts
  1447. ~~~~~~~~~~~~~~~~~~~~
  1448. If enabled the worker pool can be restarted using the
  1449. :control:`pool_restart` remote control command.
  1450. Disabled by default.
  1451. .. setting:: worker_autoscaler
  1452. worker_autoscaler
  1453. ~~~~~~~~~~~~~~~~~
  1454. .. versionadded:: 2.2
  1455. Name of the autoscaler class to use.
  1456. Default is ``celery.worker.autoscale:Autoscaler``.
  1457. .. setting:: worker_autoreloader
  1458. worker_autoreloader
  1459. ~~~~~~~~~~~~~~~~~~~
  1460. Name of the autoreloader class used by the worker to reload
  1461. Python modules and files that have changed.
  1462. Default is: ``celery.worker.autoreload:Autoreloader``.
  1463. .. setting:: worker_consumer
  1464. worker_consumer
  1465. ~~~~~~~~~~~~~~~
  1466. Name of the consumer class used by the worker.
  1467. Default is :class:`celery.worker.consumer.Consumer`
  1468. .. setting:: worker_timer
  1469. worker_timer
  1470. ~~~~~~~~~~~~
  1471. Name of the ETA scheduler class used by the worker.
  1472. Default is :class:`kombu.async.hub.timer.Timer`, or one overrided
  1473. by the pool implementation.
  1474. .. _conf-celerybeat:
  1475. Beat Settings (:program:`celery beat`)
  1476. --------------------------------------
  1477. .. setting:: beat_schedule
  1478. beat_schedule
  1479. ~~~~~~~~~~~~~
  1480. The periodic task schedule used by :mod:`~celery.bin.beat`.
  1481. See :ref:`beat-entries`.
  1482. .. setting:: beat_scheduler
  1483. beat_scheduler
  1484. ~~~~~~~~~~~~~~
  1485. The default scheduler class. Default is ``celery.beat:PersistentScheduler``.
  1486. Can also be set via the :option:`-S` argument to
  1487. :mod:`~celery.bin.beat`.
  1488. .. setting:: beat_schedule_filename
  1489. beat_schedule_filename
  1490. ~~~~~~~~~~~~~~~~~~~~~~
  1491. Name of the file used by `PersistentScheduler` to store the last run times
  1492. of periodic tasks. Can be a relative or absolute path, but be aware that the
  1493. suffix `.db` may be appended to the file name (depending on Python version).
  1494. Can also be set via the :option:`--schedule` argument to
  1495. :mod:`~celery.bin.beat`.
  1496. .. setting:: beat_sync_every
  1497. beat_sync_every
  1498. ~~~~~~~~~~~~~~~
  1499. The number of periodic tasks that can be called before another database sync
  1500. is issued.
  1501. Defaults to 0 (sync based on timing - default of 3 minutes as determined by
  1502. scheduler.sync_every). If set to 1, beat will call sync after every task
  1503. message sent.
  1504. .. setting:: beat_max_loop_interval
  1505. beat_max_loop_interval
  1506. ~~~~~~~~~~~~~~~~~~~~~~
  1507. The maximum number of seconds :mod:`~celery.bin.beat` can sleep
  1508. between checking the schedule.
  1509. The default for this value is scheduler specific.
  1510. For the default celery beat scheduler the value is 300 (5 minutes),
  1511. but for e.g. the django-celery database scheduler it is 5 seconds
  1512. because the schedule may be changed externally, and so it must take
  1513. changes to the schedule into account.
  1514. Also when running celery beat embedded (:option:`-B`) on Jython as a thread
  1515. the max interval is overridden and set to 1 so that it's possible
  1516. to shut down in a timely manner.