application.rst 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558
  1. .. _guide-app:
  2. =============
  3. Application
  4. =============
  5. .. contents::
  6. :local:
  7. :depth: 1
  8. The Celery library must be instantiated before use, this instance
  9. is called an application (or *app* for short).
  10. The application is thread-safe so that multiple Celery applications
  11. with different configurations, components, and tasks can co-exist in the
  12. same process space.
  13. Let's create one now:
  14. .. code-block:: pycon
  15. >>> from celery import Celery
  16. >>> app = Celery()
  17. >>> app
  18. <Celery __main__:0x100469fd0>
  19. The last line shows the textual representation of the application:
  20. including the name of the app class (``Celery``), the name of the
  21. current main module (``__main__``), and the memory address of the object
  22. (``0x100469fd0``).
  23. Main Name
  24. =========
  25. Only one of these is important, and that's the main module name.
  26. Let's look at why that is.
  27. When you send a task message in Celery, that message won't contain
  28. any source code, but only the name of the task you want to execute.
  29. This works similarly to how host names work on the internet: every worker
  30. maintains a mapping of task names to their actual functions, called the *task
  31. registry*.
  32. Whenever you define a task, that task will also be added to the local registry:
  33. .. code-block:: pycon
  34. >>> @app.task
  35. ... def add(x, y):
  36. ... return x + y
  37. >>> add
  38. <@task: __main__.add>
  39. >>> add.name
  40. __main__.add
  41. >>> app.tasks['__main__.add']
  42. <@task: __main__.add>
  43. and there you see that ``__main__`` again; whenever Celery isn't able
  44. to detect what module the function belongs to, it uses the main module
  45. name to generate the beginning of the task name.
  46. This is only a problem in a limited set of use cases:
  47. #. If the module that the task is defined in is run as a program.
  48. #. If the application is created in the Python shell (REPL).
  49. For example here, where the tasks module is also used to start a worker
  50. with :meth:`@worker_main`:
  51. :file:`tasks.py`:
  52. .. code-block:: python
  53. from celery import Celery
  54. app = Celery()
  55. @app.task
  56. def add(x, y): return x + y
  57. if __name__ == '__main__':
  58. app.worker_main()
  59. When this module is executed the tasks will be named starting with "``__main__``",
  60. but when the module is imported by another process, say to call a task,
  61. the tasks will be named starting with "``tasks``" (the real name of the module):
  62. .. code-block:: pycon
  63. >>> from tasks import add
  64. >>> add.name
  65. tasks.add
  66. You can specify another name for the main module:
  67. .. code-block:: pycon
  68. >>> app = Celery('tasks')
  69. >>> app.main
  70. 'tasks'
  71. >>> @app.task
  72. ... def add(x, y):
  73. ... return x + y
  74. >>> add.name
  75. tasks.add
  76. .. seealso:: :ref:`task-names`
  77. Configuration
  78. =============
  79. There are several options you can set that'll change how
  80. Celery works. These options can be set directly on the app instance,
  81. or you can use a dedicated configuration module.
  82. The configuration is available as :attr:`@conf`:
  83. .. code-block:: pycon
  84. >>> app.conf.timezone
  85. 'Europe/London'
  86. where you can also set configuration values directly:
  87. .. code-block:: pycon
  88. >>> app.conf.enable_utc = True
  89. or update several keys at once by using the ``update`` method:
  90. .. code-block:: python
  91. >>> app.conf.update(
  92. ... enable_utc=True,
  93. ... timezone='Europe/London',
  94. ...)
  95. The configuration object consists of multiple dictionaries
  96. that are consulted in order:
  97. #. Changes made at run-time.
  98. #. The configuration module (if any)
  99. #. The default configuration (:mod:`celery.app.defaults`).
  100. You can even add new default sources by using the :meth:`@add_defaults`
  101. method.
  102. .. seealso::
  103. Go to the :ref:`Configuration reference <configuration>` for a complete
  104. listing of all the available settings, and their default values.
  105. ``config_from_object``
  106. ----------------------
  107. The :meth:`@config_from_object` method loads configuration
  108. from a configuration object.
  109. This can be a configuration module, or any object with configuration attributes.
  110. Note that any configuration that was previously set will be reset when
  111. :meth:`~@config_from_object` is called. If you want to set additional
  112. configuration you should do so after.
  113. Example 1: Using the name of a module
  114. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  115. The :meth:`@config_from_object` method can take the fully qualified
  116. name of a Python module, or even the name of a Python attribute,
  117. for example: ``"celeryconfig"``, ``"myproj.config.celery"``, or
  118. ``"myproj.config:CeleryConfig"``:
  119. .. code-block:: python
  120. from celery import Celery
  121. app = Celery()
  122. app.config_from_object('celeryconfig')
  123. The ``celeryconfig`` module may then look like this:
  124. :file:`celeryconfig.py`:
  125. .. code-block:: python
  126. enable_utc = True
  127. timezone = 'Europe/London'
  128. and the app will be able to use it as long as ``import celeryconfig`` is
  129. possible.
  130. Example 2: Passing an actual module object
  131. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  132. You can also pass an already imported module object, but this
  133. isn't always recommended.
  134. .. tip::
  135. Using the name of a module is recommended as this means the module does
  136. not need to be serialized when the prefork pool is used. If you're
  137. experiencing configuration problems or pickle errors then please
  138. try using the name of a module instead.
  139. .. code-block:: python
  140. import celeryconfig
  141. from celery import Celery
  142. app = Celery()
  143. app.config_from_object(celeryconfig)
  144. Example 3: Using a configuration class/object
  145. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  146. .. code-block:: python
  147. from celery import Celery
  148. app = Celery()
  149. class Config:
  150. enable_utc = True
  151. timezone = 'Europe/London'
  152. app.config_from_object(Config)
  153. # or using the fully qualified name of the object:
  154. # app.config_from_object('module:Config')
  155. ``config_from_envvar``
  156. ----------------------
  157. The :meth:`@config_from_envvar` takes the configuration module name
  158. from an environment variable
  159. For example -- to load configuration from a module specified in the
  160. environment variable named :envvar:`CELERY_CONFIG_MODULE`:
  161. .. code-block:: python
  162. import os
  163. from celery import Celery
  164. #: Set default configuration module name
  165. os.environ.setdefault('CELERY_CONFIG_MODULE', 'celeryconfig')
  166. app = Celery()
  167. app.config_from_envvar('CELERY_CONFIG_MODULE')
  168. You can then specify the configuration module to use via the environment:
  169. .. code-block:: console
  170. $ CELERY_CONFIG_MODULE="celeryconfig.prod" celery worker -l info
  171. .. _app-censored-config:
  172. Censored configuration
  173. ----------------------
  174. If you ever want to print out the configuration, as debugging information
  175. or similar, you may also want to filter out sensitive information like
  176. passwords and API keys.
  177. Celery comes with several utilities useful for presenting the configuration,
  178. one is :meth:`~celery.app.utils.Settings.humanize`:
  179. .. code-block:: pycon
  180. >>> app.conf.humanize(with_defaults=False, censored=True)
  181. This method returns the configuration as a tabulated string. This will
  182. only contain changes to the configuration by default, but you can include the
  183. built-in default keys and values by enabling the ``with_defaults`` argument.
  184. If you instead want to work with the configuration as a dictionary, you
  185. can use the :meth:`~celery.app.utils.Settings.table` method:
  186. .. code-block:: pycon
  187. >>> app.conf.table(with_defaults=False, censored=True)
  188. Please note that Celery won't be able to remove all sensitive information,
  189. as it merely uses a regular expression to search for commonly named keys.
  190. If you add custom settings containing sensitive information you should name
  191. the keys using a name that Celery identifies as secret.
  192. A configuration setting will be censored if the name contains any of
  193. these sub-strings:
  194. ``API``, ``TOKEN``, ``KEY``, ``SECRET``, ``PASS``, ``SIGNATURE``, ``DATABASE``
  195. Laziness
  196. ========
  197. The application instance is lazy, meaning it won't be evaluated
  198. until it's actually needed.
  199. Creating a :class:`@Celery` instance will only do the following:
  200. #. Create a logical clock instance, used for events.
  201. #. Create the task registry.
  202. #. Set itself as the current app (but not if the ``set_as_current``
  203. argument was disabled)
  204. #. Call the :meth:`@on_init` callback (does nothing by default).
  205. The :meth:`@task` decorators don't create the tasks at the point when
  206. the task is defined, instead it'll defer the creation
  207. of the task to happen either when the task is used, or after the
  208. application has been *finalized*,
  209. This example shows how the task isn't created until
  210. you use the task, or access an attribute (in this case :meth:`repr`):
  211. .. code-block:: pycon
  212. >>> @app.task
  213. >>> def add(x, y):
  214. ... return x + y
  215. >>> type(add)
  216. <class 'celery.local.PromiseProxy'>
  217. >>> add.__evaluated__()
  218. False
  219. >>> add # <-- causes repr(add) to happen
  220. <@task: __main__.add>
  221. >>> add.__evaluated__()
  222. True
  223. *Finalization* of the app happens either explicitly by calling
  224. :meth:`@finalize` -- or implicitly by accessing the :attr:`@tasks`
  225. attribute.
  226. Finalizing the object will:
  227. #. Copy tasks that must be shared between apps
  228. Tasks are shared by default, but if the
  229. ``shared`` argument to the task decorator is disabled,
  230. then the task will be private to the app it's bound to.
  231. #. Evaluate all pending task decorators.
  232. #. Make sure all tasks are bound to the current app.
  233. Tasks are bound to an app so that they can read default
  234. values from the configuration.
  235. .. _default-app:
  236. .. topic:: The "default app"
  237. Celery didn't always have applications, it used to be that
  238. there was only a module-based API, and for backwards compatibility
  239. the old API is still there until the release of Celery 5.0.
  240. Celery always creates a special app - the "default app",
  241. and this is used if no custom application has been instantiated.
  242. The :mod:`celery.task` module is there to accommodate the old API,
  243. and shouldn't be used if you use a custom app. You should
  244. always use the methods on the app instance, not the module based API.
  245. For example, the old Task base class enables many compatibility
  246. features where some may be incompatible with newer features, such
  247. as task methods:
  248. .. code-block:: python
  249. from celery.task import Task # << OLD Task base class.
  250. from celery import Task # << NEW base class.
  251. The new base class is recommended even if you use the old
  252. module-based API.
  253. Breaking the chain
  254. ==================
  255. While it's possible to depend on the current app
  256. being set, the best practice is to always pass the app instance
  257. around to anything that needs it.
  258. I call this the "app chain", since it creates a chain
  259. of instances depending on the app being passed.
  260. The following example is considered bad practice:
  261. .. code-block:: python
  262. from celery import current_app
  263. class Scheduler(object):
  264. def run(self):
  265. app = current_app
  266. Instead it should take the ``app`` as an argument:
  267. .. code-block:: python
  268. class Scheduler(object):
  269. def __init__(self, app):
  270. self.app = app
  271. Internally Celery uses the :func:`celery.app.app_or_default` function
  272. so that everything also works in the module-based compatibility API
  273. .. code-block:: python
  274. from celery.app import app_or_default
  275. class Scheduler(object):
  276. def __init__(self, app=None):
  277. self.app = app_or_default(app)
  278. In development you can set the :envvar:`CELERY_TRACE_APP`
  279. environment variable to raise an exception if the app
  280. chain breaks:
  281. .. code-block:: console
  282. $ CELERY_TRACE_APP=1 celery worker -l info
  283. .. topic:: Evolving the API
  284. Celery has changed a lot in the 7 years since it was initially
  285. created.
  286. For example, in the beginning it was possible to use any callable as
  287. a task:
  288. .. code-block:: pycon
  289. def hello(to):
  290. return 'hello {0}'.format(to)
  291. >>> from celery.execute import apply_async
  292. >>> apply_async(hello, ('world!',))
  293. or you could also create a ``Task`` class to set
  294. certain options, or override other behavior
  295. .. code-block:: python
  296. from celery.task import Task
  297. from celery.registry import tasks
  298. class Hello(Task):
  299. queue = 'hipri'
  300. def run(self, to):
  301. return 'hello {0}'.format(to)
  302. tasks.register(Hello)
  303. >>> Hello.delay('world!')
  304. Later, it was decided that passing arbitrary call-able's
  305. was an anti-pattern, since it makes it very hard to use
  306. serializers other than pickle, and the feature was removed
  307. in 2.0, replaced by task decorators:
  308. .. code-block:: python
  309. from celery.task import task
  310. @task(queue='hipri')
  311. def hello(to):
  312. return 'hello {0}'.format(to)
  313. Abstract Tasks
  314. ==============
  315. All tasks created using the :meth:`~@task` decorator
  316. will inherit from the application's base :attr:`~@Task` class.
  317. You can specify a different base class using the ``base`` argument:
  318. .. code-block:: python
  319. @app.task(base=OtherTask):
  320. def add(x, y):
  321. return x + y
  322. To create a custom task class you should inherit from the neutral base
  323. class: :class:`celery.Task`.
  324. .. code-block:: python
  325. from celery import Task
  326. class DebugTask(Task):
  327. def __call__(self, *args, **kwargs):
  328. print('TASK STARTING: {0.name}[{0.request.id}]'.format(self))
  329. return super(DebugTask, self).__call__(*args, **kwargs)
  330. .. tip::
  331. If you override the tasks ``__call__`` method, then it's very important
  332. that you also call super so that the base call method can set up the
  333. default request used when a task is called directly.
  334. The neutral base class is special because it's not bound to any specific app
  335. yet. Once a task is bound to an app it'll read configuration to set default
  336. values, and so on.
  337. To realize a base class you need to create a task using the :meth:`@task`
  338. decorator:
  339. .. code-block:: python
  340. @app.task(base=DebugTask)
  341. def add(x, y):
  342. return x + y
  343. It's even possible to change the default base class for an application
  344. by changing its :meth:`@Task` attribute:
  345. .. code-block:: pycon
  346. >>> from celery import Celery, Task
  347. >>> app = Celery()
  348. >>> class MyBaseTask(Task):
  349. ... queue = 'hipri'
  350. >>> app.Task = MyBaseTask
  351. >>> app.Task
  352. <unbound MyBaseTask>
  353. >>> @app.task
  354. ... def add(x, y):
  355. ... return x + y
  356. >>> add
  357. <@task: __main__.add>
  358. >>> add.__class__.mro()
  359. [<class add of <Celery __main__:0x1012b4410>>,
  360. <unbound MyBaseTask>,
  361. <unbound Task>,
  362. <type 'object'>]