configuration.rst 61 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237123812391240124112421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285128612871288128912901291129212931294129512961297129812991300130113021303130413051306130713081309131013111312131313141315131613171318131913201321132213231324132513261327132813291330133113321333133413351336133713381339134013411342134313441345134613471348134913501351135213531354135513561357135813591360136113621363136413651366136713681369137013711372137313741375137613771378137913801381138213831384138513861387138813891390139113921393139413951396139713981399140014011402140314041405140614071408140914101411141214131414141514161417141814191420142114221423142414251426142714281429143014311432143314341435143614371438143914401441144214431444144514461447144814491450145114521453145414551456145714581459146014611462146314641465146614671468146914701471147214731474147514761477147814791480148114821483148414851486148714881489149014911492149314941495149614971498149915001501150215031504150515061507150815091510151115121513151415151516151715181519152015211522152315241525152615271528152915301531153215331534153515361537153815391540154115421543154415451546154715481549155015511552155315541555155615571558155915601561156215631564156515661567156815691570157115721573157415751576157715781579158015811582158315841585158615871588158915901591159215931594159515961597159815991600160116021603160416051606160716081609161016111612161316141615161616171618161916201621162216231624162516261627162816291630163116321633163416351636163716381639164016411642164316441645164616471648164916501651165216531654165516561657165816591660166116621663166416651666166716681669167016711672167316741675167616771678167916801681168216831684168516861687168816891690169116921693169416951696169716981699170017011702170317041705170617071708170917101711171217131714171517161717171817191720172117221723172417251726172717281729173017311732173317341735173617371738173917401741174217431744174517461747174817491750175117521753175417551756175717581759176017611762176317641765176617671768176917701771177217731774177517761777177817791780178117821783178417851786178717881789179017911792179317941795179617971798179918001801180218031804180518061807180818091810181118121813181418151816181718181819182018211822182318241825182618271828182918301831183218331834183518361837183818391840184118421843184418451846184718481849185018511852185318541855185618571858185918601861186218631864186518661867186818691870187118721873187418751876187718781879188018811882188318841885188618871888188918901891189218931894189518961897189818991900190119021903190419051906190719081909191019111912191319141915191619171918191919201921192219231924192519261927192819291930193119321933193419351936193719381939194019411942194319441945194619471948194919501951195219531954195519561957195819591960196119621963196419651966196719681969197019711972197319741975197619771978197919801981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050205120522053205420552056205720582059206020612062206320642065206620672068206920702071207220732074207520762077207820792080208120822083208420852086208720882089209020912092209320942095209620972098209921002101210221032104210521062107210821092110211121122113211421152116211721182119212021212122212321242125212621272128212921302131213221332134213521362137213821392140214121422143214421452146214721482149215021512152215321542155215621572158215921602161216221632164216521662167216821692170217121722173217421752176217721782179218021812182218321842185218621872188218921902191219221932194219521962197219821992200220122022203220422052206220722082209221022112212221322142215221622172218221922202221222222232224222522262227222822292230223122322233223422352236223722382239224022412242224322442245224622472248224922502251225222532254225522562257225822592260226122622263226422652266226722682269227022712272227322742275227622772278227922802281228222832284228522862287228822892290229122922293229422952296229722982299230023012302230323042305230623072308230923102311231223132314231523162317231823192320232123222323232423252326232723282329233023312332233323342335233623372338233923402341234223432344234523462347
  1. .. _configuration:
  2. ============================
  3. Configuration and defaults
  4. ============================
  5. This document describes the configuration options available.
  6. If you're using the default loader, you must create the :file:`celeryconfig.py`
  7. module and make sure it's available on the Python path.
  8. .. contents::
  9. :local:
  10. :depth: 2
  11. .. _conf-example:
  12. Example configuration file
  13. ==========================
  14. This is an example configuration file to get you started.
  15. It should contain all you need to run a basic Celery set-up.
  16. .. code-block:: python
  17. ## Broker settings.
  18. broker_url = 'amqp://guest:guest@localhost:5672//'
  19. # List of modules to import when the Celery worker starts.
  20. imports = ('myapp.tasks',)
  21. ## Using the database to store task state and results.
  22. result_backend = 'db+sqlite:///results.db'
  23. task_annotations = {'tasks.add': {'rate_limit': '10/s'}}
  24. .. _conf-old-settings-map:
  25. New lowercase settings
  26. ======================
  27. Version 4.0 introduced new lower case settings and setting organization.
  28. The major difference between previous versions, apart from the lower case
  29. names, are the renaming of some prefixes, like ``celerybeat_`` to ``beat_``,
  30. ``celeryd_`` to ``worker_``, and most of the top level ``celery_`` settings
  31. have been moved into a new ``task_`` prefix.
  32. Celery will still be able to read old configuration files, so there's no
  33. rush in moving to the new settings format.
  34. ===================================== ==============================================
  35. **Setting name** **Replace with**
  36. ===================================== ==============================================
  37. ``CELERY_ACCEPT_CONTENT`` :setting:`accept_content`
  38. ``CELERY_ENABLE_UTC`` :setting:`enable_utc`
  39. ``CELERY_IMPORTS`` :setting:`imports`
  40. ``CELERY_INCLUDE`` :setting:`include`
  41. ``CELERY_TIMEZONE`` :setting:`timezone`
  42. ``CELERYBEAT_MAX_LOOP_INTERVAL`` :setting:`beat_max_loop_interval`
  43. ``CELERYBEAT_SCHEDULE`` :setting:`beat_schedule`
  44. ``CELERYBEAT_SCHEDULER`` :setting:`beat_scheduler`
  45. ``CELERYBEAT_SCHEDULE_FILENAME`` :setting:`beat_schedule_filename`
  46. ``CELERYBEAT_SYNC_EVERY`` :setting:`beat_sync_every`
  47. ``BROKER_URL`` :setting:`broker_url`
  48. ``BROKER_TRANSPORT`` :setting:`broker_transport`
  49. ``BROKER_TRANSPORT_OPTIONS`` :setting:`broker_transport_options`
  50. ``BROKER_CONNECTION_TIMEOUT`` :setting:`broker_connection_timeout`
  51. ``BROKER_CONNECTION_RETRY`` :setting:`broker_connection_retry`
  52. ``BROKER_CONNECTION_MAX_RETRIES`` :setting:`broker_connection_max_retries`
  53. ``BROKER_FAILOVER_STRATEGY`` :setting:`broker_failover_strategy`
  54. ``BROKER_HEARTBEAT`` :setting:`broker_heartbeat`
  55. ``BROKER_LOGIN_METHOD`` :setting:`broker_login_method`
  56. ``BROKER_POOL_LIMIT`` :setting:`broker_pool_limit`
  57. ``BROKER_USE_SSL`` :setting:`broker_use_ssl`
  58. ``CELERY_CACHE_BACKEND`` :setting:`cache_backend`
  59. ``CELERY_CACHE_BACKEND_OPTIONS`` :setting:`cache_backend_options`
  60. ``CASSANDRA_COLUMN_FAMILY`` :setting:`cassandra_table`
  61. ``CASSANDRA_ENTRY_TTL`` :setting:`cassandra_entry_ttl`
  62. ``CASSANDRA_KEYSPACE`` :setting:`cassandra_keyspace`
  63. ``CASSANDRA_PORT`` :setting:`cassandra_port`
  64. ``CASSANDRA_READ_CONSISTENCY`` :setting:`cassandra_read_consistency`
  65. ``CASSANDRA_SERVERS`` :setting:`cassandra_servers`
  66. ``CASSANDRA_WRITE_CONSISTENCY`` :setting:`cassandra_write_consistency`
  67. ``CELERY_COUCHBASE_BACKEND_SETTINGS`` :setting:`couchbase_backend_settings`
  68. ``CELERY_MONGODB_BACKEND_SETTINGS`` :setting:`mongodb_backend_settings`
  69. ``CELERY_EVENT_QUEUE_EXPIRES`` :setting:`event_queue_expires`
  70. ``CELERY_EVENT_QUEUE_TTL`` :setting:`event_queue_ttl`
  71. ``CELERY_EVENT_QUEUE_PREFIX`` :setting:`event_queue_prefix`
  72. ``CELERY_EVENT_SERIALIZER`` :setting:`event_serializer`
  73. ``CELERY_REDIS_DB`` :setting:`redis_db`
  74. ``CELERY_REDIS_HOST`` :setting:`redis_host`
  75. ``CELERY_REDIS_MAX_CONNECTIONS`` :setting:`redis_max_connections`
  76. ``CELERY_REDIS_PASSWORD`` :setting:`redis_password`
  77. ``CELERY_REDIS_PORT`` :setting:`redis_port`
  78. ``CELERY_RESULT_BACKEND`` :setting:`result_backend`
  79. ``CELERY_MAX_CACHED_RESULTS`` :setting:`result_cache_max`
  80. ``CELERY_MESSAGE_COMPRESSION`` :setting:`result_compression`
  81. ``CELERY_RESULT_EXCHANGE`` :setting:`result_exchange`
  82. ``CELERY_RESULT_EXCHANGE_TYPE`` :setting:`result_exchange_type`
  83. ``CELERY_TASK_RESULT_EXPIRES`` :setting:`result_expires`
  84. ``CELERY_RESULT_PERSISTENT`` :setting:`result_persistent`
  85. ``CELERY_RESULT_SERIALIZER`` :setting:`result_serializer`
  86. ``CELERY_RESULT_DBURI`` Use :setting:`result_backend` instead.
  87. ``CELERY_RESULT_ENGINE_OPTIONS`` :setting:`database_engine_options`
  88. ``[...]_DB_SHORT_LIVED_SESSIONS`` :setting:`database_short_lived_sessions`
  89. ``CELERY_RESULT_DB_TABLE_NAMES`` :setting:`database_db_names`
  90. ``CELERY_SECURITY_CERTIFICATE`` :setting:`security_certificate`
  91. ``CELERY_SECURITY_CERT_STORE`` :setting:`security_cert_store`
  92. ``CELERY_SECURITY_KEY`` :setting:`security_key`
  93. ``CELERY_ACKS_LATE`` :setting:`task_acks_late`
  94. ``CELERY_ALWAYS_EAGER`` :setting:`task_always_eager`
  95. ``CELERY_ANNOTATIONS`` :setting:`task_annotations`
  96. ``CELERY_MESSAGE_COMPRESSION`` :setting:`task_compression`
  97. ``CELERY_CREATE_MISSING_QUEUES`` :setting:`task_create_missing_queues`
  98. ``CELERY_DEFAULT_DELIVERY_MODE`` :setting:`task_default_delivery_mode`
  99. ``CELERY_DEFAULT_EXCHANGE`` :setting:`task_default_exchange`
  100. ``CELERY_DEFAULT_EXCHANGE_TYPE`` :setting:`task_default_exchange_type`
  101. ``CELERY_DEFAULT_QUEUE`` :setting:`task_default_queue`
  102. ``CELERY_DEFAULT_RATE_LIMIT`` :setting:`task_default_rate_limit`
  103. ``CELERY_DEFAULT_ROUTING_KEY`` :setting:`task_default_routing_key`
  104. ``[...]_EAGER_PROPAGATES_EXCEPTIONS`` :setting:`task_eager_propagates`
  105. ``CELERY_IGNORE_RESULT`` :setting:`task_ignore_result`
  106. ``CELERY_TASK_PUBLISH_RETRY`` :setting:`task_publish_retry`
  107. ``CELERY_TASK_PUBLISH_RETRY_POLICY`` :setting:`task_publish_retry_policy`
  108. ``CELERY_QUEUES`` :setting:`task_queues`
  109. ``CELERY_ROUTES`` :setting:`task_routes`
  110. ``CELERY_SEND_TASK_SENT_EVENT`` :setting:`task_send_sent_event`
  111. ``CELERY_TASK_SERIALIZER`` :setting:`task_serializer`
  112. ``CELERYD_TASK_SOFT_TIME_LIMIT`` :setting:`task_soft_time_limit`
  113. ``CELERYD_TASK_TIME_LIMIT`` :setting:`task_time_limit`
  114. ``CELERY_TRACK_STARTED`` :setting:`task_track_started`
  115. ``CELERYD_AGENT`` :setting:`worker_agent`
  116. ``CELERYD_AUTOSCALER`` :setting:`worker_autoscaler`
  117. ``CELERYD_CONCURRENCY`` :setting:`worker_concurrency`
  118. ``CELERYD_CONSUMER`` :setting:`worker_consumer`
  119. ``CELERY_WORKER_DIRECT`` :setting:`worker_direct`
  120. ``CELERY_DISABLE_RATE_LIMITS`` :setting:`worker_disable_rate_limits`
  121. ``CELERY_ENABLE_REMOTE_CONTROL`` :setting:`worker_enable_remote_control`
  122. ``CELERYD_HIJACK_ROOT_LOGGER`` :setting:`worker_hijack_root_logger`
  123. ``CELERYD_LOG_COLOR`` :setting:`worker_log_color`
  124. ``CELERYD_LOG_FORMAT`` :setting:`worker_log_format`
  125. ``CELERYD_WORKER_LOST_WAIT`` :setting:`worker_lost_wait`
  126. ``CELERYD_MAX_TASKS_PER_CHILD`` :setting:`worker_max_tasks_per_child`
  127. ``CELERYD_POOL`` :setting:`worker_pool`
  128. ``CELERYD_POOL_PUTLOCKS`` :setting:`worker_pool_putlocks`
  129. ``CELERYD_POOL_RESTARTS`` :setting:`worker_pool_restarts`
  130. ``CELERYD_PREFETCH_MULTIPLIER`` :setting:`worker_prefetch_multiplier`
  131. ``CELERYD_REDIRECT_STDOUTS`` :setting:`worker_redirect_stdouts`
  132. ``CELERYD_REDIRECT_STDOUTS_LEVEL`` :setting:`worker_redirect_stdouts_level`
  133. ``CELERYD_SEND_EVENTS`` :setting:`worker_send_task_events`
  134. ``CELERYD_STATE_DB`` :setting:`worker_state_db`
  135. ``CELERYD_TASK_LOG_FORMAT`` :setting:`worker_task_log_format`
  136. ``CELERYD_TIMER`` :setting:`worker_timer`
  137. ``CELERYD_TIMER_PRECISION`` :setting:`worker_timer_precision`
  138. ===================================== ==============================================
  139. Configuration Directives
  140. ========================
  141. .. _conf-datetime:
  142. General settings
  143. ----------------
  144. .. setting:: accept_content
  145. ``accept_content``
  146. ~~~~~~~~~~~~~~~~~~
  147. Default: ``{'json'}`` (set, list, or tuple).
  148. A white-list of content-types/serializers to allow.
  149. If a message is received that's not in this list then
  150. the message will be discarded with an error.
  151. By default any content type is enabled, including pickle and yaml,
  152. so make sure untrusted parties don't have access to your broker.
  153. See :ref:`guide-security` for more.
  154. Example::
  155. # using serializer name
  156. accept_content = ['json']
  157. # or the actual content-type (MIME)
  158. accept_content = ['application/json']
  159. Time and date settings
  160. ----------------------
  161. .. setting:: enable_utc
  162. ``enable_utc``
  163. ~~~~~~~~~~~~~~
  164. .. versionadded:: 2.5
  165. Default: Enabled by default since version 3.0.
  166. If enabled dates and times in messages will be converted to use
  167. the UTC timezone.
  168. Note that workers running Celery versions below 2.5 will assume a local
  169. timezone for all messages, so only enable if all workers have been
  170. upgraded.
  171. .. setting:: timezone
  172. ``timezone``
  173. ~~~~~~~~~~~~
  174. .. versionadded:: 2.5
  175. Default: ``"UTC"``.
  176. Configure Celery to use a custom time zone.
  177. The timezone value can be any time zone supported by the :pypi:`pytz`
  178. library.
  179. If not set the UTC timezone is used. For backwards compatibility
  180. there's also a :setting:`enable_utc` setting, and this is set
  181. to false the system local timezone is used instead.
  182. .. _conf-tasks:
  183. Task settings
  184. -------------
  185. .. setting:: task_annotations
  186. ``task_annotations``
  187. ~~~~~~~~~~~~~~~~~~~~
  188. .. versionadded:: 2.5
  189. Default: :const:`None`.
  190. This setting can be used to rewrite any task attribute from the
  191. configuration. The setting can be a dict, or a list of annotation
  192. objects that filter for tasks and return a map of attributes
  193. to change.
  194. This will change the ``rate_limit`` attribute for the ``tasks.add``
  195. task:
  196. .. code-block:: python
  197. task_annotations = {'tasks.add': {'rate_limit': '10/s'}}
  198. or change the same for all tasks:
  199. .. code-block:: python
  200. task_annotations = {'*': {'rate_limit': '10/s'}}
  201. You can change methods too, for example the ``on_failure`` handler:
  202. .. code-block:: python
  203. def my_on_failure(self, exc, task_id, args, kwargs, einfo):
  204. print('Oh no! Task failed: {0!r}'.format(exc))
  205. task_annotations = {'*': {'on_failure': my_on_failure}}
  206. If you need more flexibility then you can use objects
  207. instead of a dict to choose the tasks to annotate:
  208. .. code-block:: python
  209. class MyAnnotate:
  210. def annotate(self, task):
  211. if task.name.startswith('tasks.'):
  212. return {'rate_limit': '10/s'}
  213. task_annotations = (MyAnnotate(), {other,})
  214. .. setting:: task_compression
  215. ``task_compression``
  216. ~~~~~~~~~~~~~~~~~~~~
  217. Default: :const:`None`
  218. Default compression used for task messages.
  219. Can be ``gzip``, ``bzip2`` (if available), or any custom
  220. compression schemes registered in the Kombu compression registry.
  221. The default is to send uncompressed messages.
  222. .. setting:: task_protocol
  223. ``task_protocol``
  224. ~~~~~~~~~~~~~~~~~
  225. .. versionadded: 4.0
  226. Default: 2 (since 4.0).
  227. Set the default task message protocol version used to send tasks.
  228. Supports protocols: 1 and 2.
  229. Protocol 2 is supported by 3.1.24 and 4.x+.
  230. .. setting:: task_serializer
  231. ``task_serializer``
  232. ~~~~~~~~~~~~~~~~~~~
  233. Default: ``"json"`` (since 4.0, earlier: pickle).
  234. A string identifying the default serialization method to use. Can be
  235. `json` (default), `pickle`, `yaml`, `msgpack`, or any custom serialization
  236. methods that have been registered with :mod:`kombu.serialization.registry`.
  237. .. seealso::
  238. :ref:`calling-serializers`.
  239. .. setting:: task_publish_retry
  240. ``task_publish_retry``
  241. ~~~~~~~~~~~~~~~~~~~~~~
  242. .. versionadded:: 2.2
  243. Default: Enabled.
  244. Decides if publishing task messages will be retried in the case
  245. of connection loss or other connection errors.
  246. See also :setting:`task_publish_retry_policy`.
  247. .. setting:: task_publish_retry_policy
  248. ``task_publish_retry_policy``
  249. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  250. .. versionadded:: 2.2
  251. Default: See :ref:`calling-retry`.
  252. Defines the default policy when retrying publishing a task message in
  253. the case of connection loss or other connection errors.
  254. .. _conf-task-execution:
  255. Task execution settings
  256. -----------------------
  257. .. setting:: task_always_eager
  258. ``task_always_eager``
  259. ~~~~~~~~~~~~~~~~~~~~~
  260. Default: Disabled.
  261. If this is :const:`True`, all tasks will be executed locally by blocking until
  262. the task returns. ``apply_async()`` and ``Task.delay()`` will return
  263. an :class:`~celery.result.EagerResult` instance, that emulates the API
  264. and behavior of :class:`~celery.result.AsyncResult`, except the result
  265. is already evaluated.
  266. That is, tasks will be executed locally instead of being sent to
  267. the queue.
  268. .. setting:: task_eager_propagates
  269. ``task_eager_propagates``
  270. ~~~~~~~~~~~~~~~~~~~~~~~~~
  271. Default: Disabled.
  272. If this is :const:`True`, eagerly executed tasks (applied by `task.apply()`,
  273. or when the :setting:`task_always_eager` setting is enabled), will
  274. propagate exceptions.
  275. It's the same as always running ``apply()`` with ``throw=True``.
  276. .. setting:: task_remote_tracebacks
  277. ``task_remote_tracebacks``
  278. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  279. Default: Disabled.
  280. If enabled task results will include the workers stack when re-raising
  281. task errors.
  282. This requires the :pypi:`tblib` library, that can be installed using
  283. :command:`pip`:
  284. .. code-block:: console
  285. $ pip install celery[tblib]
  286. See :ref:`bundles` for information on combining multiple extension
  287. requirements.
  288. .. setting:: task_ignore_result
  289. ``task_ignore_result``
  290. ~~~~~~~~~~~~~~~~~~~~~~
  291. Default: Disabled.
  292. Whether to store the task return values or not (tombstones).
  293. If you still want to store errors, just not successful return values,
  294. you can set :setting:`task_store_errors_even_if_ignored`.
  295. .. setting:: task_store_errors_even_if_ignored
  296. ``task_store_errors_even_if_ignored``
  297. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  298. Default: Disabled.
  299. If set, the worker stores all task errors in the result store even if
  300. :attr:`Task.ignore_result <celery.task.base.Task.ignore_result>` is on.
  301. .. setting:: task_track_started
  302. ``task_track_started``
  303. ~~~~~~~~~~~~~~~~~~~~~~
  304. Default: Disabled.
  305. If :const:`True` the task will report its status as 'started' when the
  306. task is executed by a worker. The default value is :const:`False` as
  307. the normal behavior is to not report that level of granularity. Tasks
  308. are either pending, finished, or waiting to be retried. Having a 'started'
  309. state can be useful for when there are long running tasks and there's a
  310. need to report what task is currently running.
  311. .. setting:: task_time_limit
  312. ``task_time_limit``
  313. ~~~~~~~~~~~~~~~~~~~
  314. Default: No time limit.
  315. Task hard time limit in seconds. The worker processing the task will
  316. be killed and replaced with a new one when this is exceeded.
  317. .. setting:: task_soft_time_limit
  318. ``task_soft_time_limit``
  319. ~~~~~~~~~~~~~~~~~~~~~~~~
  320. Default: No soft time limit.
  321. Task soft time limit in seconds.
  322. The :exc:`~@SoftTimeLimitExceeded` exception will be
  323. raised when this is exceeded. For example, the task can catch this to
  324. clean up before the hard time limit comes:
  325. .. code-block:: python
  326. from celery.exceptions import SoftTimeLimitExceeded
  327. @app.task
  328. def mytask():
  329. try:
  330. return do_work()
  331. except SoftTimeLimitExceeded:
  332. cleanup_in_a_hurry()
  333. .. setting:: task_acks_late
  334. ``task_acks_late``
  335. ~~~~~~~~~~~~~~~~~~
  336. Default: Disabled.
  337. Late ack means the task messages will be acknowledged **after** the task
  338. has been executed, not *just before* (the default behavior).
  339. .. seealso::
  340. FAQ: :ref:`faq-acks_late-vs-retry`.
  341. .. setting:: task_reject_on_worker_lost
  342. ``task_reject_on_worker_lost``
  343. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  344. Default: Disabled.
  345. Even if :setting:`task_acks_late` is enabled, the worker will
  346. acknowledge tasks when the worker process executing them abruptly
  347. exits or is signaled (e.g., :sig:`KILL`/:sig:`INT`, etc).
  348. Setting this to true allows the message to be re-queued instead,
  349. so that the task will execute again by the same worker, or another
  350. worker.
  351. .. warning::
  352. Enabling this can cause message loops; make sure you know
  353. what you're doing.
  354. .. setting:: task_default_rate_limit
  355. ``task_default_rate_limit``
  356. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  357. Default: No rate limit.
  358. The global default rate limit for tasks.
  359. This value is used for tasks that doesn't have a custom rate limit
  360. .. seealso::
  361. The setting:`worker_disable_rate_limits` setting can
  362. disable all rate limits.
  363. .. _conf-result-backend:
  364. Task result backend settings
  365. ----------------------------
  366. .. setting:: result_backend
  367. ``result_backend``
  368. ~~~~~~~~~~~~~~~~~~
  369. Default: No result backend enabled by default.
  370. The backend used to store task results (tombstones).
  371. Can be one of the following:
  372. * ``rpc``
  373. Send results back as AMQP messages
  374. See :ref:`conf-rpc-result-backend`.
  375. * ``database``
  376. Use a relational database supported by `SQLAlchemy`_.
  377. See :ref:`conf-database-result-backend`.
  378. * ``redis``
  379. Use `Redis`_ to store the results.
  380. See :ref:`conf-redis-result-backend`.
  381. * ``cache``
  382. Use `Memcached`_ to store the results.
  383. See :ref:`conf-cache-result-backend`.
  384. * ``cassandra``
  385. Use `Cassandra`_ to store the results.
  386. See :ref:`conf-cassandra-result-backend`.
  387. * ``elasticsearch``
  388. Use `Elasticsearch`_ to store the results.
  389. See :ref:`conf-elasticsearch-result-backend`.
  390. * ``ironcache``
  391. Use `IronCache`_ to store the results.
  392. See :ref:`conf-ironcache-result-backend`.
  393. * ``couchbase``
  394. Use `Couchbase`_ to store the results.
  395. See :ref:`conf-couchbase-result-backend`.
  396. * ``couchdb``
  397. Use `CouchDB`_ to store the results.
  398. See :ref:`conf-couchdb-result-backend`.
  399. * ``filesystem``
  400. Use a shared directory to store the results.
  401. See :ref:`conf-filesystem-result-backend`.
  402. * ``consul``
  403. Use the `Consul`_ K/V store to store the results
  404. See :ref:`conf-consul-result-backend`.
  405. .. warning:
  406. While the AMQP result backend is very efficient, you must make sure
  407. you only receive the same result once. See :doc:`userguide/calling`).
  408. .. _`SQLAlchemy`: http://sqlalchemy.org
  409. .. _`Memcached`: http://memcached.org
  410. .. _`Redis`: http://redis.io
  411. .. _`Cassandra`: http://cassandra.apache.org/
  412. .. _`Elasticsearch`: https://aws.amazon.com/elasticsearch-service/
  413. .. _`IronCache`: http://www.iron.io/cache
  414. .. _`CouchDB`: http://www.couchdb.com/
  415. .. _`Couchbase`: http://www.couchbase.com/
  416. .. _`Consul`: http://consul.io/
  417. .. setting:: result_serializer
  418. ``result_serializer``
  419. ~~~~~~~~~~~~~~~~~~~~~
  420. Default: ``json`` since 4.0 (earlier: pickle).
  421. Result serialization format.
  422. See :ref:`calling-serializers` for information about supported
  423. serialization formats.
  424. .. setting:: result_compression
  425. ``result_compression``
  426. ~~~~~~~~~~~~~~~~~~~~~~
  427. Default: No compression.
  428. Optional compression method used for task results.
  429. Supports the same options as the :setting:`task_serializer` setting.
  430. .. setting:: result_expires
  431. ``result_expires``
  432. ~~~~~~~~~~~~~~~~~~
  433. Default: Expire after 1 day.
  434. Time (in seconds, or a :class:`~datetime.timedelta` object) for when after
  435. stored task tombstones will be deleted.
  436. A built-in periodic task will delete the results after this time
  437. (``celery.backend_cleanup``), assuming that ``celery beat`` is
  438. enabled. The task runs daily at 4am.
  439. A value of :const:`None` or 0 means results will never expire (depending
  440. on backend specifications).
  441. .. note::
  442. For the moment this only works with the AMQP, database, cache,
  443. and Redis backends.
  444. When using the database backend, ``celery beat`` must be
  445. running for the results to be expired.
  446. .. setting:: result_cache_max
  447. ``result_cache_max``
  448. ~~~~~~~~~~~~~~~~~~~~
  449. Default: Disabled by default.
  450. Enables client caching of results.
  451. This can be useful for the old deprecated
  452. 'amqp' backend where the result is unavailable as soon as one result instance
  453. consumes it.
  454. This is the total number of results to cache before older results are evicted.
  455. A value of 0 or None means no limit, and a value of :const:`-1`
  456. will disable the cache.
  457. Disabled by default.
  458. .. _conf-database-result-backend:
  459. Database backend settings
  460. -------------------------
  461. Database URL Examples
  462. ~~~~~~~~~~~~~~~~~~~~~
  463. To use the database backend you have to configure the
  464. :setting:`result_backend` setting with a connection URL and the ``db+``
  465. prefix:
  466. .. code-block:: python
  467. result_backend = 'db+scheme://user:password@host:port/dbname'
  468. Examples::
  469. # sqlite (filename)
  470. result_backend = 'db+sqlite:///results.sqlite'
  471. # mysql
  472. result_backend = 'db+mysql://scott:tiger@localhost/foo'
  473. # postgresql
  474. result_backend = 'db+postgresql://scott:tiger@localhost/mydatabase'
  475. # oracle
  476. result_backend = 'db+oracle://scott:tiger@127.0.0.1:1521/sidname'
  477. .. code-block:: python
  478. Please see `Supported Databases`_ for a table of supported databases,
  479. and `Connection String`_ for more information about connection
  480. strings (this is the part of the URI that comes after the ``db+`` prefix).
  481. .. _`Supported Databases`:
  482. http://www.sqlalchemy.org/docs/core/engines.html#supported-databases
  483. .. _`Connection String`:
  484. http://www.sqlalchemy.org/docs/core/engines.html#database-urls
  485. .. setting:: database_engine_options
  486. ``database_engine_options``
  487. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  488. Default: ``{}`` (empty mapping).
  489. To specify additional SQLAlchemy database engine options you can use
  490. the :setting:`sqlalchmey_engine_options` setting::
  491. # echo enables verbose logging from SQLAlchemy.
  492. app.conf.database_engine_options = {'echo': True}
  493. .. setting:: database_short_lived_sessions
  494. ``database_short_lived_sessions``
  495. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  496. Default: Disabled by default.
  497. Short lived sessions are disabled by default. If enabled they can drastically reduce
  498. performance, especially on systems processing lots of tasks. This option is useful
  499. on low-traffic workers that experience errors as a result of cached database connections
  500. going stale through inactivity. For example, intermittent errors like
  501. `(OperationalError) (2006, 'MySQL server has gone away')` can be fixed by enabling
  502. short lived sessions. This option only affects the database backend.
  503. .. setting:: database_table_names
  504. ``database_table_names``
  505. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  506. Default: ``{}`` (empty mapping).
  507. When SQLAlchemy is configured as the result backend, Celery automatically
  508. creates two tables to store result meta-data for tasks. This setting allows
  509. you to customize the table names:
  510. .. code-block:: python
  511. # use custom table names for the database result backend.
  512. database_table_names = {
  513. 'task': 'myapp_taskmeta',
  514. 'group': 'myapp_groupmeta',
  515. }
  516. .. _conf-rpc-result-backend:
  517. RPC backend settings
  518. --------------------
  519. .. setting:: result_persistent
  520. ``result_persistent``
  521. ~~~~~~~~~~~~~~~~~~~~~
  522. Default: Disabled by default (transient messages).
  523. If set to :const:`True`, result messages will be persistent. This means the
  524. messages won't be lost after a broker restart.
  525. Example configuration
  526. ~~~~~~~~~~~~~~~~~~~~~
  527. .. code-block:: python
  528. result_backend = 'rpc://'
  529. result_persistent = False
  530. .. _conf-cache-result-backend:
  531. Cache backend settings
  532. ----------------------
  533. .. note::
  534. The cache backend supports the :pypi:`pylibmc` and :pypi:`python-memcached`
  535. libraries. The latter is used only if :pypi:`pylibmc` isn't installed.
  536. Using a single Memcached server:
  537. .. code-block:: python
  538. result_backend = 'cache+memcached://127.0.0.1:11211/'
  539. Using multiple Memcached servers:
  540. .. code-block:: python
  541. result_backend = """
  542. cache+memcached://172.19.26.240:11211;172.19.26.242:11211/
  543. """.strip()
  544. The "memory" backend stores the cache in memory only:
  545. .. code-block:: python
  546. result_backend = 'cache'
  547. cache_backend = 'memory'
  548. .. setting:: cache_backend_options
  549. ``cache_backend_options``
  550. ~~~~~~~~~~~~~~~~~~~~~~~~~
  551. Default: ``{}`` (empty mapping).
  552. You can set :pypi:`pylibmc` options using the :setting:`cache_backend_options`
  553. setting:
  554. .. code-block:: python
  555. cache_backend_options = {
  556. 'binary': True,
  557. 'behaviors': {'tcp_nodelay': True},
  558. }
  559. .. setting:: cache_backend
  560. ``cache_backend``
  561. ~~~~~~~~~~~~~~~~~
  562. This setting is no longer used as it's now possible to specify
  563. the cache backend directly in the :setting:`result_backend` setting.
  564. .. _conf-redis-result-backend:
  565. Redis backend settings
  566. ----------------------
  567. Configuring the backend URL
  568. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  569. .. note::
  570. The Redis backend requires the :pypi:`redis` library.
  571. To install this package use :command:`pip`:
  572. .. code-block:: console
  573. $ pip install celery[redis]
  574. See :ref:`bundles` for information on combining multiple extension
  575. requirements.
  576. This backend requires the :setting:`result_backend`
  577. setting to be set to a Redis URL::
  578. result_backend = 'redis://:password@host:port/db'
  579. For example::
  580. result_backend = 'redis://localhost/0'
  581. is the same as::
  582. result_backend = 'redis://'
  583. The fields of the URL are defined as follows:
  584. #. ``password``
  585. Password used to connect to the database.
  586. #. ``host``
  587. Host name or IP address of the Redis server (e.g., `localhost`).
  588. #. ``port``
  589. Port to the Redis server. Default is 6379.
  590. #. ``db``
  591. Database number to use. Default is 0.
  592. The db can include an optional leading slash.
  593. .. setting:: redis_max_connections
  594. ``redis_max_connections``
  595. ~~~~~~~~~~~~~~~~~~~~~~~~~
  596. Default: No limit.
  597. Maximum number of connections available in the Redis connection
  598. pool used for sending and retrieving results.
  599. .. setting:: redis_socket_connect_timeout
  600. ``redis_socket_connect_timeout``
  601. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  602. .. versionadded:: 5.0.1
  603. Default: :const:`None`
  604. Socket timeout for connections to Redis from the result backend
  605. in seconds (int/float)
  606. .. setting:: redis_socket_timeout
  607. ``redis_socket_timeout``
  608. ~~~~~~~~~~~~~~~~~~~~~~~~
  609. Default: 5.0 seconds.
  610. Socket timeout for reading/writing operations to the Redis server
  611. in seconds (int/float), used by the redis result backend.
  612. .. _conf-cassandra-result-backend:
  613. Cassandra backend settings
  614. --------------------------
  615. .. note::
  616. This Cassandra backend driver requires :pypi:`cassandra-driver`.
  617. To install, use :command:`pip`:
  618. .. code-block:: console
  619. $ pip install celery[cassandra]
  620. See :ref:`bundles` for information on combining multiple extension
  621. requirements.
  622. This backend requires the following configuration directives to be set.
  623. .. setting:: cassandra_servers
  624. ``cassandra_servers``
  625. ~~~~~~~~~~~~~~~~~~~~~
  626. Default: ``[]`` (empty list).
  627. List of ``host`` Cassandra servers. For example::
  628. cassandra_servers = ['localhost']
  629. .. setting:: cassandra_port
  630. ``cassandra_port``
  631. ~~~~~~~~~~~~~~~~~~
  632. Default: 9042.
  633. Port to contact the Cassandra servers on.
  634. .. setting:: cassandra_keyspace
  635. ``cassandra_keyspace``
  636. ~~~~~~~~~~~~~~~~~~~~~~
  637. Default: None.
  638. The key-space in which to store the results. For example::
  639. cassandra_keyspace = 'tasks_keyspace'
  640. .. setting:: cassandra_table
  641. ``cassandra_table``
  642. ~~~~~~~~~~~~~~~~~~~
  643. Default: None.
  644. The table (column family) in which to store the results. For example::
  645. cassandra_table = 'tasks'
  646. .. setting:: cassandra_read_consistency
  647. ``cassandra_read_consistency``
  648. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  649. Default: None.
  650. The read consistency used. Values can be ``ONE``, ``TWO``, ``THREE``, ``QUORUM``, ``ALL``,
  651. ``LOCAL_QUORUM``, ``EACH_QUORUM``, ``LOCAL_ONE``.
  652. .. setting:: cassandra_write_consistency
  653. ``cassandra_write_consistency``
  654. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  655. Default: None.
  656. The write consistency used. Values can be ``ONE``, ``TWO``, ``THREE``, ``QUORUM``, ``ALL``,
  657. ``LOCAL_QUORUM``, ``EACH_QUORUM``, ``LOCAL_ONE``.
  658. .. setting:: cassandra_entry_ttl
  659. ``cassandra_entry_ttl``
  660. ~~~~~~~~~~~~~~~~~~~~~~~
  661. Default: None.
  662. Time-to-live for status entries. They will expire and be removed after that many seconds
  663. after adding. A value of :const:`None` (default) means they will never expire.
  664. .. setting:: cassandra_auth_provider
  665. ``cassandra_auth_provider``
  666. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  667. Default: :const:`None`.
  668. AuthProvider class within ``cassandra.auth`` module to use. Values can be
  669. ``PlainTextAuthProvider`` or ``SaslAuthProvider``.
  670. .. setting:: cassandra_auth_kwargs
  671. ``cassandra_auth_kwargs``
  672. ~~~~~~~~~~~~~~~~~~~~~~~~~
  673. Default: ``{}`` (empty mapping).
  674. Named arguments to pass into the authentication provider. For example:
  675. .. code-block:: python
  676. cassandra_auth_kwargs = {
  677. username: 'cassandra',
  678. password: 'cassandra'
  679. }
  680. Example configuration
  681. ~~~~~~~~~~~~~~~~~~~~~
  682. .. code-block:: python
  683. cassandra_servers = ['localhost']
  684. cassandra_keyspace = 'celery'
  685. cassandra_table = 'tasks'
  686. cassandra_read_consistency = 'ONE'
  687. cassandra_write_consistency = 'ONE'
  688. cassandra_entry_ttl = 86400
  689. .. _conf-elasticsearch-result-backend:
  690. Elasticsearch backend settings
  691. ------------------------------
  692. To use `Elasticsearch`_ as the result backend you simply need to
  693. configure the :setting:`result_backend` setting with the correct URL.
  694. Example configuration
  695. ~~~~~~~~~~~~~~~~~~~~~
  696. .. code-block:: python
  697. result_backend = 'elasticsearch://example.com:9200/index_name/doc_type'
  698. .. _conf-riak-result-backend:
  699. Riak backend settings
  700. ---------------------
  701. .. note::
  702. The Riak backend requires the :pypi:`riak` library.
  703. To install the this package use :command:`pip`:
  704. .. code-block:: console
  705. $ pip install celery[riak]
  706. See :ref:`bundles` for information on combining multiple extension
  707. requirements.
  708. This backend requires the :setting:`result_backend`
  709. setting to be set to a Riak URL::
  710. result_backend = 'riak://host:port/bucket'
  711. For example::
  712. result_backend = 'riak://localhost/celery
  713. is the same as::
  714. result_backend = 'riak://'
  715. The fields of the URL are defined as follows:
  716. #. ``host``
  717. Host name or IP address of the Riak server (e.g., `'localhost'`).
  718. #. ``port``
  719. Port to the Riak server using the protobuf protocol. Default is 8087.
  720. #. ``bucket``
  721. Bucket name to use. Default is `celery`.
  722. The bucket needs to be a string with ASCII characters only.
  723. Alternatively, this backend can be configured with the following configuration directives.
  724. .. setting:: riak_backend_settings
  725. ``riak_backend_settings``
  726. ~~~~~~~~~~~~~~~~~~~~~~~~~
  727. Default: ``{}`` (empty mapping).
  728. This is a dict supporting the following keys:
  729. * ``host``
  730. The host name of the Riak server. Defaults to ``"localhost"``.
  731. * ``port``
  732. The port the Riak server is listening to. Defaults to 8087.
  733. * ``bucket``
  734. The bucket name to connect to. Defaults to "celery".
  735. * ``protocol``
  736. The protocol to use to connect to the Riak server. This isn't configurable
  737. via :setting:`result_backend`
  738. .. _conf-ironcache-result-backend:
  739. IronCache backend settings
  740. --------------------------
  741. .. note::
  742. The IronCache backend requires the :pypi:`iron_celery` library:
  743. To install this package use :command:`pip`:
  744. .. code-block:: console
  745. $ pip install iron_celery
  746. IronCache is configured via the URL provided in :setting:`result_backend`, for example::
  747. result_backend = 'ironcache://project_id:token@'
  748. Or to change the cache name::
  749. ironcache:://project_id:token@/awesomecache
  750. For more information, see: https://github.com/iron-io/iron_celery
  751. .. _conf-couchbase-result-backend:
  752. Couchbase backend settings
  753. --------------------------
  754. .. note::
  755. The Couchbase backend requires the :pypi:`couchbase` library.
  756. To install this package use :command:`pip`:
  757. .. code-block:: console
  758. $ pip install celery[couchbase]
  759. See :ref:`bundles` for instructions how to combine multiple extension
  760. requirements.
  761. This backend can be configured via the :setting:`result_backend`
  762. set to a Couchbase URL:
  763. .. code-block:: python
  764. result_backend = 'couchbase://username:password@host:port/bucket'
  765. .. setting:: couchbase_backend_settings
  766. ``couchbase_backend_settings``
  767. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  768. Default: ``{}`` (empty mapping).
  769. This is a dict supporting the following keys:
  770. * ``host``
  771. Host name of the Couchbase server. Defaults to ``localhost``.
  772. * ``port``
  773. The port the Couchbase server is listening to. Defaults to ``8091``.
  774. * ``bucket``
  775. The default bucket the Couchbase server is writing to.
  776. Defaults to ``default``.
  777. * ``username``
  778. User name to authenticate to the Couchbase server as (optional).
  779. * ``password``
  780. Password to authenticate to the Couchbase server (optional).
  781. .. _conf-couchdb-result-backend:
  782. CouchDB backend settings
  783. ------------------------
  784. .. note::
  785. The CouchDB backend requires the :pypi:`pycouchdb` library:
  786. To install this Couchbase package use :command:`pip`:
  787. .. code-block:: console
  788. $ pip install celery[couchdb]
  789. See :ref:`bundles` for information on combining multiple extension
  790. requirements.
  791. This backend can be configured via the :setting:`result_backend`
  792. set to a CouchDB URL::
  793. result_backend = 'couchdb://username:password@host:port/container'
  794. The URL is formed out of the following parts:
  795. * ``username``
  796. User name to authenticate to the CouchDB server as (optional).
  797. * ``password``
  798. Password to authenticate to the CouchDB server (optional).
  799. * ``host``
  800. Host name of the CouchDB server. Defaults to ``localhost``.
  801. * ``port``
  802. The port the CouchDB server is listening to. Defaults to ``8091``.
  803. * ``container``
  804. The default container the CouchDB server is writing to.
  805. Defaults to ``default``.
  806. .. _conf-filesystem-result-backend:
  807. File-system backend settings
  808. ----------------------------
  809. This backend can be configured using a file URL, for example::
  810. CELERY_RESULT_BACKEND = 'file:///var/celery/results'
  811. The configured directory needs to be shared and writable by all servers using
  812. the backend.
  813. If you're trying Celery on a single system you can simply use the backend
  814. without any further configuration. For larger clusters you could use NFS,
  815. `GlusterFS`_, CIFS, `HDFS`_ (using FUSE), or any other file-system.
  816. .. _`GlusterFS`: http://www.gluster.org/
  817. .. _`HDFS`: http://hadoop.apache.org/
  818. .. _conf-consul-result-backend:
  819. Consul K/V store backend settings
  820. ---------------------------------
  821. The Consul backend can be configured using a URL, for example:
  822. CELERY_RESULT_BACKEND = 'consul://localhost:8500/'
  823. The backend will storage results in the K/V store of Consul
  824. as individual keys.
  825. The backend supports auto expire of results using TTLs in Consul.
  826. .. _conf-messaging:
  827. Message Routing
  828. ---------------
  829. .. _conf-messaging-routing:
  830. .. setting:: task_queues
  831. ``task_queues``
  832. ~~~~~~~~~~~~~~~
  833. Default: :const:`None` (queue taken from default queue settings).
  834. Most users will not want to specify this setting and should rather use
  835. the :ref:`automatic routing facilities <routing-automatic>`.
  836. If you really want to configure advanced routing, this setting should
  837. be a list of :class:`kombu.Queue` objects the worker will consume from.
  838. Note that workers can be overridden this setting via the
  839. :option:`-Q <celery worker -Q>` option, or individual queues from this
  840. list (by name) can be excluded using the :option:`-X <celery worker -X>`
  841. option.
  842. Also see :ref:`routing-basics` for more information.
  843. The default is a queue/exchange/binding key of ``celery``, with
  844. exchange type ``direct``.
  845. See also :setting:`task_routes`
  846. .. setting:: task_routes
  847. ``task_routes``
  848. ~~~~~~~~~~~~~~~
  849. Default: :const:`None`.
  850. A list of routers, or a single router used to route tasks to queues.
  851. When deciding the final destination of a task the routers are consulted
  852. in order.
  853. A router can be specified as either:
  854. * A function with the signature ``(name, args, kwargs,
  855. options, task=None, **kwargs)``
  856. * A string providing the path to a router function.
  857. * A dict containing router specification:
  858. Will be converted to a :class:`celery.routes.MapRoute` instance.
  859. * A list of ``(pattern, route)`` tuples:
  860. Will be converted to a :class:`celery.routes.MapRoute` instance.
  861. Examples:
  862. .. code-block:: python
  863. task_routes = {
  864. 'celery.ping': 'default',
  865. 'mytasks.add': 'cpu-bound',
  866. 'feed.tasks.*': 'feeds', # <-- glob pattern
  867. re.compile(r'(image|video)\.tasks\..*'): 'media', # <-- regex
  868. 'video.encode': {
  869. 'queue': 'video',
  870. 'exchange': 'media'
  871. 'routing_key': 'media.video.encode',
  872. },
  873. }
  874. task_routes = ('myapp.tasks.route_task', {'celery.ping': 'default})
  875. Where ``myapp.tasks.route_task`` could be:
  876. .. code-block:: python
  877. def route_task(self, name, args, kwargs, options, task=None, **kw):
  878. if task == 'celery.ping':
  879. return {'queue': 'default'}
  880. ``route_task`` may return a string or a dict. A string then means
  881. it's a queue name in :setting:`task_queues`, a dict means it's a custom route.
  882. When sending tasks, the routers are consulted in order. The first
  883. router that doesn't return ``None`` is the route to use. The message options
  884. is then merged with the found route settings, where the routers settings
  885. have priority.
  886. Example if :func:`~celery.execute.apply_async` has these arguments:
  887. .. code-block:: python
  888. Task.apply_async(immediate=False, exchange='video',
  889. routing_key='video.compress')
  890. and a router returns:
  891. .. code-block:: python
  892. {'immediate': True, 'exchange': 'urgent'}
  893. the final message options will be:
  894. .. code-block:: python
  895. immediate=True, exchange='urgent', routing_key='video.compress'
  896. (and any default message options defined in the
  897. :class:`~celery.task.base.Task` class)
  898. Values defined in :setting:`task_routes` have precedence over values defined in
  899. :setting:`task_queues` when merging the two.
  900. With the follow settings:
  901. .. code-block:: python
  902. task_queues = {
  903. 'cpubound': {
  904. 'exchange': 'cpubound',
  905. 'routing_key': 'cpubound',
  906. },
  907. }
  908. task_routes = {
  909. 'tasks.add': {
  910. 'queue': 'cpubound',
  911. 'routing_key': 'tasks.add',
  912. 'serializer': 'json',
  913. },
  914. }
  915. The final routing options for ``tasks.add`` will become:
  916. .. code-block:: javascript
  917. {'exchange': 'cpubound',
  918. 'routing_key': 'tasks.add',
  919. 'serializer': 'json'}
  920. See :ref:`routers` for more examples.
  921. .. setting:: task_queue_ha_policy
  922. ``task_queue_ha_policy``
  923. ~~~~~~~~~~~~~~~~~~~~~~~~
  924. :brokers: RabbitMQ
  925. Default: :const:`None`.
  926. This will set the default HA policy for a queue, and the value
  927. can either be a string (usually ``all``):
  928. .. code-block:: python
  929. task_queue_ha_policy = 'all'
  930. Using 'all' will replicate the queue to all current nodes,
  931. Or you can give it a list of nodes to replicate to:
  932. .. code-block:: python
  933. task_queue_ha_policy = ['rabbit@host1', 'rabbit@host2']
  934. Using a list will implicitly set ``x-ha-policy`` to 'nodes' and
  935. ``x-ha-policy-params`` to the given list of nodes.
  936. See http://www.rabbitmq.com/ha.html for more information.
  937. .. setting:: task_queue_max_priority
  938. ``task_queue_max_priority``
  939. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  940. :brokers: RabbitMQ
  941. Default: :const:`None`.
  942. See :ref:`routing-options-rabbitmq-priorities`.
  943. .. setting:: worker_direct
  944. ``worker_direct``
  945. ~~~~~~~~~~~~~~~~~
  946. Default: Disabled.
  947. This option enables so that every worker has a dedicated queue,
  948. so that tasks can be routed to specific workers.
  949. The queue name for each worker is automatically generated based on
  950. the worker hostname and a ``.dq`` suffix, using the ``C.dq`` exchange.
  951. For example the queue name for the worker with node name ``w1@example.com``
  952. becomes::
  953. w1@example.com.dq
  954. Then you can route the task to the task by specifying the hostname
  955. as the routing key and the ``C.dq`` exchange::
  956. task_routes = {
  957. 'tasks.add': {'exchange': 'C.dq', 'routing_key': 'w1@example.com'}
  958. }
  959. .. setting:: task_create_missing_queues
  960. ``task_create_missing_queues``
  961. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  962. Default: Enabled.
  963. If enabled (default), any queues specified that aren't defined in
  964. :setting:`task_queues` will be automatically created. See
  965. :ref:`routing-automatic`.
  966. .. setting:: task_default_queue
  967. ``task_default_queue``
  968. ~~~~~~~~~~~~~~~~~~~~~~
  969. Default: ``"celery"``.
  970. The name of the default queue used by `.apply_async` if the message has
  971. no route or no custom queue has been specified.
  972. This queue must be listed in :setting:`task_queues`.
  973. If :setting:`task_queues` isn't specified then it's automatically
  974. created containing one queue entry, where this name is used as the name of
  975. that queue.
  976. .. seealso::
  977. :ref:`routing-changing-default-queue`
  978. .. setting:: task_default_exchange
  979. ``task_default_exchange``
  980. ~~~~~~~~~~~~~~~~~~~~~~~~~
  981. Default: ``"celery"``.
  982. Name of the default exchange to use when no custom exchange is
  983. specified for a key in the :setting:`task_queues` setting.
  984. .. setting:: task_default_exchange_type
  985. ``task_default_exchange_type``
  986. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  987. Default: ``"direct"``.
  988. Default exchange type used when no custom exchange type is specified
  989. for a key in the :setting:`task_queues` setting.
  990. .. setting:: task_default_routing_key
  991. ``task_default_routing_key``
  992. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  993. Default: ``"celery"``.
  994. The default routing key used when no custom routing key
  995. is specified for a key in the :setting:`task_queues` setting.
  996. .. setting:: task_default_delivery_mode
  997. ``task_default_delivery_mode``
  998. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  999. Default: ``"persistent"``.
  1000. Can be `transient` (messages not written to disk) or `persistent` (written to
  1001. disk).
  1002. .. _conf-broker-settings:
  1003. Broker Settings
  1004. ---------------
  1005. .. setting:: broker_url
  1006. ``broker_url``
  1007. ~~~~~~~~~~~~~~
  1008. Default: ``"amqp://"``
  1009. Default broker URL. This must be a URL in the form of::
  1010. transport://userid:password@hostname:port/virtual_host
  1011. Only the scheme part (``transport://``) is required, the rest
  1012. is optional, and defaults to the specific transports default values.
  1013. The transport part is the broker implementation to use, and the
  1014. default is ``amqp``, (uses ``librabbitmq`` if installed or falls back to
  1015. ``pyamqp``). There are also other choices available, including;
  1016. ``redis://``, ``sqs://``, and ``qpid://``.
  1017. The scheme can also be a fully qualified path to your own transport
  1018. implementation::
  1019. broker_url = 'proj.transports.MyTransport://localhost'
  1020. More than one broker URL, of the same transport, can also be specified.
  1021. The broker URLs can be passed in as a single string that's semicolon delimited::
  1022. broker_url = 'transport://userid:password@hostname:port//;transport://userid:password@hostname:port//'
  1023. Or as a list::
  1024. broker_url = [
  1025. 'transport://userid:password@localhost:port//',
  1026. 'transport://userid:password@hostname:port//'
  1027. ]
  1028. The brokers will then be used in the :setting:`broker_failover_strategy`.
  1029. See :ref:`kombu:connection-urls` in the Kombu documentation for more
  1030. information.
  1031. .. setting:: broker_read_url
  1032. .. setting:: broker_write_url
  1033. ``broker_read_url`` / ``broker_write_url``
  1034. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1035. Default: Taken from :setting:`broker_url`.
  1036. These settings can be configured, instead of :setting:`broker_url` to specify
  1037. different connection parameters for broker connections used for consuming and
  1038. producing.
  1039. Example::
  1040. broker_read_url = 'amqp://user:pass@broker.example.com:56721'
  1041. broker_write_url = 'amqp://user:pass@broker.example.com:56722'
  1042. Both options can also be specified as a list for failover alternates, see
  1043. :setting:`broker_url` for more information.
  1044. .. setting:: broker_failover_strategy
  1045. ``broker_failover_strategy``
  1046. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1047. Default: ``"round-robin"``.
  1048. Default failover strategy for the broker Connection object. If supplied,
  1049. may map to a key in 'kombu.connection.failover_strategies', or be a reference
  1050. to any method that yields a single item from a supplied list.
  1051. Example::
  1052. # Random failover strategy
  1053. def random_failover_strategy(servers):
  1054. it = list(it) # don't modify callers list
  1055. shuffle = random.shuffle
  1056. for _ in repeat(None):
  1057. shuffle(it)
  1058. yield it[0]
  1059. broker_failover_strategy = random_failover_strategy
  1060. .. setting:: broker_heartbeat
  1061. ``broker_heartbeat``
  1062. ~~~~~~~~~~~~~~~~~~~~
  1063. :transports supported: ``pyamqp``
  1064. Default: ``120.0`` (negotiated by server).
  1065. Note: This value is only used by the worker, clients do not use
  1066. a heartbeat at the moment.
  1067. It's not always possible to detect connection loss in a timely
  1068. manner using TCP/IP alone, so AMQP defines something called heartbeats
  1069. that's is used both by the client and the broker to detect if
  1070. a connection was closed.
  1071. If the heartbeat value is 10 seconds, then
  1072. the heartbeat will be monitored at the interval specified
  1073. by the :setting:`broker_heartbeat_checkrate` setting (by default
  1074. this is set to double the rate of the heartbeat value,
  1075. so for the 10 seconds, the heartbeat is checked every 5 seconds).
  1076. .. setting:: broker_heartbeat_checkrate
  1077. ``broker_heartbeat_checkrate``
  1078. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1079. :transports supported: ``pyamqp``
  1080. Default: 2.0.
  1081. At intervals the worker will monitor that the broker hasn't missed
  1082. too many heartbeats. The rate at which this is checked is calculated
  1083. by dividing the :setting:`broker_heartbeat` value with this value,
  1084. so if the heartbeat is 10.0 and the rate is the default 2.0, the check
  1085. will be performed every 5 seconds (twice the heartbeat sending rate).
  1086. .. setting:: broker_use_ssl
  1087. ``broker_use_ssl``
  1088. ~~~~~~~~~~~~~~~~~~
  1089. :transports supported: ``pyamqp``, ``redis``
  1090. Default: Disabled.
  1091. Toggles SSL usage on broker connection and SSL settings.
  1092. If ``True`` the connection will use SSL with default SSL settings.
  1093. If set to a dict, will configure SSL connection according to the specified
  1094. policy. The format used is Python's :func:`ssl.wrap_socket` options.
  1095. Note that SSL socket is generally served on a separate port by the broker.
  1096. Example providing a client cert and validating the server cert against a custom
  1097. certificate authority:
  1098. .. code-block:: python
  1099. import ssl
  1100. broker_use_ssl = {
  1101. 'keyfile': '/var/ssl/private/worker-key.pem',
  1102. 'certfile': '/var/ssl/amqp-server-cert.pem',
  1103. 'ca_certs': '/var/ssl/myca.pem',
  1104. 'cert_reqs': ssl.CERT_REQUIRED
  1105. }
  1106. .. warning::
  1107. Be careful using ``broker_use_ssl=True``. It's possible that your default
  1108. configuration won't validate the server cert at all. Please read Python
  1109. `ssl module security
  1110. considerations <https://docs.python.org/3/library/ssl.html#ssl-security>`_.
  1111. .. setting:: broker_pool_limit
  1112. ``broker_pool_limit``
  1113. ~~~~~~~~~~~~~~~~~~~~~
  1114. .. versionadded:: 2.3
  1115. Default: 10.
  1116. The maximum number of connections that can be open in the connection pool.
  1117. The pool is enabled by default since version 2.5, with a default limit of ten
  1118. connections. This number can be tweaked depending on the number of
  1119. threads/green-threads (eventlet/gevent) using a connection. For example
  1120. running eventlet with 1000 greenlets that use a connection to the broker,
  1121. contention can arise and you should consider increasing the limit.
  1122. If set to :const:`None` or 0 the connection pool will be disabled and
  1123. connections will be established and closed for every use.
  1124. .. setting:: broker_connection_timeout
  1125. ``broker_connection_timeout``
  1126. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1127. Default: 4.0.
  1128. The default timeout in seconds before we give up establishing a connection
  1129. to the AMQP server. This setting is disabled when using
  1130. gevent.
  1131. .. setting:: broker_connection_retry
  1132. ``broker_connection_retry``
  1133. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1134. Default: Enabled.
  1135. Automatically try to re-establish the connection to the AMQP broker if lost.
  1136. The time between retries is increased for each retry, and is
  1137. not exhausted before :setting:`broker_connection_max_retries` is
  1138. exceeded.
  1139. .. setting:: broker_connection_max_retries
  1140. ``broker_connection_max_retries``
  1141. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1142. Default: 100.
  1143. Maximum number of retries before we give up re-establishing a connection
  1144. to the AMQP broker.
  1145. If this is set to :const:`0` or :const:`None`, we'll retry forever.
  1146. .. setting:: broker_login_method
  1147. ``broker_login_method``
  1148. ~~~~~~~~~~~~~~~~~~~~~~~
  1149. Default: ``"AMQPLAIN"``.
  1150. Set custom amqp login method.
  1151. .. setting:: broker_transport_options
  1152. ``broker_transport_options``
  1153. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1154. .. versionadded:: 2.2
  1155. Default: ``{}`` (empty mapping).
  1156. A dict of additional options passed to the underlying transport.
  1157. See your transport user manual for supported options (if any).
  1158. Example setting the visibility timeout (supported by Redis and SQS
  1159. transports):
  1160. .. code-block:: python
  1161. broker_transport_options = {'visibility_timeout': 18000} # 5 hours
  1162. .. _conf-worker:
  1163. Worker
  1164. ------
  1165. .. setting:: imports
  1166. ``imports``
  1167. ~~~~~~~~~~~
  1168. Default: ``[]`` (empty list).
  1169. A sequence of modules to import when the worker starts.
  1170. This is used to specify the task modules to import, but also
  1171. to import signal handlers and additional remote control commands, etc.
  1172. The modules will be imported in the original order.
  1173. .. setting:: include
  1174. ``include``
  1175. ~~~~~~~~~~~
  1176. Default: ``[]`` (empty list).
  1177. Exact same semantics as :setting:`imports`, but can be used as a means
  1178. to have different import categories.
  1179. The modules in this setting are imported after the modules in
  1180. :setting:`imports`.
  1181. .. _conf-concurrency:
  1182. .. setting:: worker_concurrency
  1183. ``worker_concurrency``
  1184. ~~~~~~~~~~~~~~~~~~~~~~
  1185. Default: Number of CPU cores.
  1186. The number of concurrent worker processes/threads/green threads executing
  1187. tasks.
  1188. If you're doing mostly I/O you can have more processes,
  1189. but if mostly CPU-bound, try to keep it close to the
  1190. number of CPUs on your machine. If not set, the number of CPUs/cores
  1191. on the host will be used.
  1192. .. setting:: worker_prefetch_multiplier
  1193. ``worker_prefetch_multiplier``
  1194. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1195. Default: 4.
  1196. How many messages to prefetch at a time multiplied by the number of
  1197. concurrent processes. The default is 4 (four messages for each
  1198. process). The default setting is usually a good choice, however -- if you
  1199. have very long running tasks waiting in the queue and you have to start the
  1200. workers, note that the first worker to start will receive four times the
  1201. number of messages initially. Thus the tasks may not be fairly distributed
  1202. to the workers.
  1203. To disable prefetching, set :setting:`worker_prefetch_multiplier` to 1.
  1204. Changing that setting to 0 will allow the worker to keep consuming
  1205. as many messages as it wants.
  1206. For more on prefetching, read :ref:`optimizing-prefetch-limit`
  1207. .. note::
  1208. Tasks with ETA/countdown aren't affected by prefetch limits.
  1209. .. setting:: worker_lost_wait
  1210. ``worker_lost_wait``
  1211. ~~~~~~~~~~~~~~~~~~~~
  1212. Default: 10.0 seconds.
  1213. In some cases a worker may be killed without proper cleanup,
  1214. and the worker may have published a result before terminating.
  1215. This value specifies how long we wait for any missing results before
  1216. raising a :exc:`@WorkerLostError` exception.
  1217. .. setting:: worker_max_tasks_per_child
  1218. ``worker_max_tasks_per_child``
  1219. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1220. Maximum number of tasks a pool worker process can execute before
  1221. it's replaced with a new one. Default is no limit.
  1222. .. setting:: worker_max_memory_per_child
  1223. ``worker_max_memory_per_child``
  1224. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1225. Default: No limit.
  1226. Type: int (kilobytes)
  1227. Maximum amount of resident memory, in kilobytes, that may be consumed by a
  1228. worker before it will be replaced by a new worker. If a single
  1229. task causes a worker to exceed this limit, the task will be
  1230. completed, and the worker will be replaced afterwards.
  1231. Example:
  1232. .. code-block:: python
  1233. worker_max_memory_per_child = 12000 # 12MB
  1234. .. setting:: worker_disable_rate_limits
  1235. ``worker_disable_rate_limits``
  1236. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1237. Default: Disabled (rate limits enabled).
  1238. Disable all rate limits, even if tasks has explicit rate limits set.
  1239. .. setting:: worker_state_db
  1240. ``worker_state_db``
  1241. ~~~~~~~~~~~~~~~~~~~
  1242. Default: :const:`None`.
  1243. Name of the file used to stores persistent worker state (like revoked tasks).
  1244. Can be a relative or absolute path, but be aware that the suffix `.db`
  1245. may be appended to the file name (depending on Python version).
  1246. Can also be set via the :option:`celery worker --statedb` argument.
  1247. .. setting:: worker_timer_precision
  1248. ``worker_timer_precision``
  1249. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1250. Default: 1.0 seconds.
  1251. Set the maximum time in seconds that the ETA scheduler can sleep between
  1252. rechecking the schedule.
  1253. Setting this value to 1 second means the schedulers precision will
  1254. be 1 second. If you need near millisecond precision you can set this to 0.1.
  1255. .. setting:: worker_enable_remote_control
  1256. ``worker_enable_remote_control``
  1257. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1258. Default: Enabled by default.
  1259. Specify if remote control of the workers is enabled.
  1260. .. _conf-events:
  1261. Events
  1262. ------
  1263. .. setting:: worker_send_task_events
  1264. ``worker_send_task_events``
  1265. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1266. Default: Disabled by default.
  1267. Send task-related events so that tasks can be monitored using tools like
  1268. `flower`. Sets the default value for the workers
  1269. :option:`-E <celery worker -E>` argument.
  1270. .. setting:: task_send_sent_event
  1271. ``task_send_sent_event``
  1272. ~~~~~~~~~~~~~~~~~~~~~~~~
  1273. .. versionadded:: 2.2
  1274. Default: Disabled by default.
  1275. If enabled, a :event:`task-sent` event will be sent for every task so tasks can be
  1276. tracked before they're consumed by a worker.
  1277. .. setting:: event_queue_ttl
  1278. ``event_queue_ttl``
  1279. ~~~~~~~~~~~~~~~~~~~
  1280. :transports supported: ``amqp``
  1281. Default: 5.0 seconds.
  1282. Message expiry time in seconds (int/float) for when messages sent to a monitor clients
  1283. event queue is deleted (``x-message-ttl``)
  1284. For example, if this value is set to 10 then a message delivered to this queue
  1285. will be deleted after 10 seconds.
  1286. .. setting:: event_queue_expires
  1287. ``event_queue_expires``
  1288. ~~~~~~~~~~~~~~~~~~~~~~~
  1289. :transports supported: ``amqp``
  1290. Default: 60.0 seconds.
  1291. Expiry time in seconds (int/float) for when after a monitor clients
  1292. event queue will be deleted (``x-expires``).
  1293. .. setting:: event_queue_prefix
  1294. ``event_queue_prefix``
  1295. ~~~~~~~~~~~~~~~~~~~~~~
  1296. Default: ``"celeryev"``.
  1297. The prefix to use for event receiver queue names.
  1298. .. setting:: event_serializer
  1299. ``event_serializer``
  1300. ~~~~~~~~~~~~~~~~~~~~
  1301. Default: ``"json"``.
  1302. Message serialization format used when sending event messages.
  1303. .. seealso::
  1304. :ref:`calling-serializers`.
  1305. .. _conf-control:
  1306. Remote Control Commands
  1307. -----------------------
  1308. .. note::
  1309. To disable remote control commands see
  1310. the :setting:`worker_enable_remote_control` setting.
  1311. .. setting:: control_queue_ttl
  1312. ``control_queue_ttl``
  1313. ~~~~~~~~~~~~~~~~~~~~~
  1314. Default: 300.0
  1315. Time in seconds, before a message in a remote control command queue
  1316. will expire.
  1317. If using the default of 300 seconds, this means that if a remote control
  1318. command is sent and no worker picks it up within 300 seconds, the command
  1319. is discarded.
  1320. This setting also applies to remote control reply queues.
  1321. .. setting:: control_queue_expires
  1322. ``control_queue_expires``
  1323. ~~~~~~~~~~~~~~~~~~~~~~~~~
  1324. Default: 10.0
  1325. Time in seconds, before an unused remote control command queue is deleted
  1326. from the broker.
  1327. This setting also applies to remote control reply queues.
  1328. .. _conf-logging:
  1329. Logging
  1330. -------
  1331. .. setting:: worker_hijack_root_logger
  1332. ``worker_hijack_root_logger``
  1333. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1334. .. versionadded:: 2.2
  1335. Default: Enabled by default (hijack root logger).
  1336. By default any previously configured handlers on the root logger will be
  1337. removed. If you want to customize your own logging handlers, then you
  1338. can disable this behavior by setting
  1339. `worker_hijack_root_logger = False`.
  1340. .. note::
  1341. Logging can also be customized by connecting to the
  1342. :signal:`celery.signals.setup_logging` signal.
  1343. .. setting:: worker_log_color
  1344. ``worker_log_color``
  1345. ~~~~~~~~~~~~~~~~~~~~
  1346. Default: Enabled if app is logging to a terminal.
  1347. Enables/disables colors in logging output by the Celery apps.
  1348. .. setting:: worker_log_format
  1349. ``worker_log_format``
  1350. ~~~~~~~~~~~~~~~~~~~~~
  1351. Default:
  1352. .. code-block:: text
  1353. "[%(asctime)s: %(levelname)s/%(processName)s] %(message)s"
  1354. The format to use for log messages.
  1355. See the Python :mod:`logging` module for more information about log
  1356. formats.
  1357. .. setting:: worker_task_log_format
  1358. ``worker_task_log_format``
  1359. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1360. Default:
  1361. .. code-block:: text
  1362. "[%(asctime)s: %(levelname)s/%(processName)s]
  1363. [%(task_name)s(%(task_id)s)] %(message)s"
  1364. The format to use for log messages logged in tasks.
  1365. See the Python :mod:`logging` module for more information about log
  1366. formats.
  1367. .. setting:: worker_redirect_stdouts
  1368. ``worker_redirect_stdouts``
  1369. ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1370. Default: Enabled by default.
  1371. If enabled `stdout` and `stderr` will be redirected
  1372. to the current logger.
  1373. Used by :program:`celery worker` and :program:`celery beat`.
  1374. .. setting:: worker_redirect_stdouts_level
  1375. ``worker_redirect_stdouts_level``
  1376. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1377. Default: :const:`WARNING`.
  1378. The log level output to `stdout` and `stderr` is logged as.
  1379. Can be one of :const:`DEBUG`, :const:`INFO`, :const:`WARNING`,
  1380. :const:`ERROR`, or :const:`CRITICAL`.
  1381. .. _conf-security:
  1382. Security
  1383. --------
  1384. .. setting:: security_key
  1385. ``security_key``
  1386. ~~~~~~~~~~~~~~~~
  1387. Default: :const:`None`.
  1388. .. versionadded:: 2.5
  1389. The relative or absolute path to a file containing the private key
  1390. used to sign messages when :ref:`message-signing` is used.
  1391. .. setting:: security_certificate
  1392. ``security_certificate``
  1393. ~~~~~~~~~~~~~~~~~~~~~~~~
  1394. Default: :const:`None`.
  1395. .. versionadded:: 2.5
  1396. The relative or absolute path to an X.509 certificate file
  1397. used to sign messages when :ref:`message-signing` is used.
  1398. .. setting:: security_cert_store
  1399. ``security_cert_store``
  1400. ~~~~~~~~~~~~~~~~~~~~~~~
  1401. Default: :const:`None`.
  1402. .. versionadded:: 2.5
  1403. The directory containing X.509 certificates used for
  1404. :ref:`message-signing`. Can be a glob with wild-cards,
  1405. (for example :file:`/etc/certs/*.pem`).
  1406. .. _conf-custom-components:
  1407. Custom Component Classes (advanced)
  1408. -----------------------------------
  1409. .. setting:: worker_pool
  1410. ``worker_pool``
  1411. ~~~~~~~~~~~~~~~
  1412. Default: ``"prefork"`` (``celery.concurrency.prefork:TaskPool``).
  1413. Name of the pool class used by the worker.
  1414. .. admonition:: Eventlet/Gevent
  1415. Never use this option to select the eventlet or gevent pool.
  1416. You must use the :option:`-P <celery worker -P>` option to
  1417. :program:`celery worker` instead, to ensure the monkey patches
  1418. aren't applied too late, causing things to break in strange ways.
  1419. .. setting:: worker_pool_restarts
  1420. ``worker_pool_restarts``
  1421. ~~~~~~~~~~~~~~~~~~~~~~~~
  1422. Default: Disabled by default.
  1423. If enabled the worker pool can be restarted using the
  1424. :control:`pool_restart` remote control command.
  1425. .. setting:: worker_autoscaler
  1426. ``worker_autoscaler``
  1427. ~~~~~~~~~~~~~~~~~~~~~
  1428. .. versionadded:: 2.2
  1429. Default: ``"celery.worker.autoscale:Autoscaler"``.
  1430. Name of the autoscaler class to use.
  1431. .. setting:: worker_consumer
  1432. ``worker_consumer``
  1433. ~~~~~~~~~~~~~~~~~~~
  1434. Default: ``"celery.worker.consumer:Consumer"``.
  1435. Name of the consumer class used by the worker.
  1436. .. setting:: worker_timer
  1437. ``worker_timer``
  1438. ~~~~~~~~~~~~~~~~
  1439. Default: ``"kombu.async.hub.timer:Timer"``.
  1440. Name of the ETA scheduler class used by the worker.
  1441. Default is or set by the pool implementation.
  1442. .. _conf-celerybeat:
  1443. Beat Settings (:program:`celery beat`)
  1444. --------------------------------------
  1445. .. setting:: beat_schedule
  1446. ``beat_schedule``
  1447. ~~~~~~~~~~~~~~~~~
  1448. Default: ``{}`` (empty mapping).
  1449. The periodic task schedule used by :mod:`~celery.bin.beat`.
  1450. See :ref:`beat-entries`.
  1451. .. setting:: beat_scheduler
  1452. ``beat_scheduler``
  1453. ~~~~~~~~~~~~~~~~~~
  1454. Default: ``"celery.beat:PersistentScheduler"``.
  1455. The default scheduler class. May be set to
  1456. ``"django_celery_beat.schedulers:DatabaseScheduler"`` for instance,
  1457. if used alongside `django-celery-beat` extension.
  1458. Can also be set via the :option:`celery beat -S` argument.
  1459. .. setting:: beat_schedule_filename
  1460. ``beat_schedule_filename``
  1461. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1462. Default: ``"celerybeat-schedule"``.
  1463. Name of the file used by `PersistentScheduler` to store the last run times
  1464. of periodic tasks. Can be a relative or absolute path, but be aware that the
  1465. suffix `.db` may be appended to the file name (depending on Python version).
  1466. Can also be set via the :option:`celery beat --schedule` argument.
  1467. .. setting:: beat_sync_every
  1468. ``beat_sync_every``
  1469. ~~~~~~~~~~~~~~~~~~~
  1470. Default: 0.
  1471. The number of periodic tasks that can be called before another database sync
  1472. is issued.
  1473. A value of 0 (default) means sync based on timing - default of 3 minutes as determined by
  1474. scheduler.sync_every. If set to 1, beat will call sync after every task
  1475. message sent.
  1476. .. setting:: beat_max_loop_interval
  1477. ``beat_max_loop_interval``
  1478. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1479. Default: 0.
  1480. The maximum number of seconds :mod:`~celery.bin.beat` can sleep
  1481. between checking the schedule.
  1482. The default for this value is scheduler specific.
  1483. For the default Celery beat scheduler the value is 300 (5 minutes),
  1484. but for the :pypi:`django-celery-beat` database scheduler it's 5 seconds
  1485. because the schedule may be changed externally, and so it must take
  1486. changes to the schedule into account.
  1487. Also when running Celery beat embedded (:option:`-B <celery worker -B>`)
  1488. on Jython as a thread the max interval is overridden and set to 1 so
  1489. that it's possible to shut down in a timely manner.