celery.rst 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510
  1. ===========================================
  2. :mod:`celery` --- Distributed processing
  3. ===========================================
  4. .. currentmodule:: celery
  5. .. module:: celery
  6. :synopsis: Distributed processing
  7. .. moduleauthor:: Ask Solem <ask@celeryproject.org>
  8. .. sectionauthor:: Ask Solem <ask@celeryproject.org>
  9. --------------
  10. This module is the main entry-point for the Celery API.
  11. It includes commonly needed things for calling tasks,
  12. and creating Celery applications.
  13. ===================== ===================================================
  14. :class:`Celery` celery application instance
  15. :class:`group` group tasks together
  16. :class:`chain` chain tasks together
  17. :class:`chord` chords enable callbacks for groups
  18. :class:`subtask` task signatures
  19. :data:`current_app` proxy to the current application instance
  20. :data:`current_task` proxy to the currently executing task
  21. ===================== ===================================================
  22. :class:`Celery` application objects
  23. -----------------------------------
  24. .. versionadded:: 2.5
  25. .. class:: Celery(main='__main__', broker='amqp://localhost//', ...)
  26. :param main: Name of the main module if running as `__main__`.
  27. :keyword broker: URL of the default broker used.
  28. :keyword loader: The loader class, or the name of the loader class to use.
  29. Default is :class:`celery.loaders.app.AppLoader`.
  30. :keyword backend: The result store backend class, or the name of the
  31. backend class to use. Default is the value of the
  32. :setting:`CELERY_RESULT_BACKEND` setting.
  33. :keyword amqp: AMQP object or class name.
  34. :keyword events: Events object or class name.
  35. :keyword log: Log object or class name.
  36. :keyword control: Control object or class name.
  37. :keyword set_as_current: Make this the global current app.
  38. :keyword tasks: A task registry or the name of a registry class.
  39. .. attribute:: Celery.main
  40. Name of the `__main__` module. Required for standalone scripts.
  41. If set this will be used instead of `__main__` when automatically
  42. generating task names.
  43. .. attribute:: Celery.conf
  44. Current configuration.
  45. .. attribute:: user_options
  46. Custom options for command-line programs.
  47. See :ref:`extending-commandoptions`
  48. .. attribute:: steps
  49. Custom bootsteps to extend and modify the worker.
  50. See :ref:`extending-bootsteps`.
  51. .. attribute:: Celery.current_task
  52. The instance of the task that is being executed, or :const:`None`.
  53. .. attribute:: Celery.amqp
  54. AMQP related functionality: :class:`~@amqp`.
  55. .. attribute:: Celery.backend
  56. Current backend instance.
  57. .. attribute:: Celery.loader
  58. Current loader instance.
  59. .. attribute:: Celery.control
  60. Remote control: :class:`~@control`.
  61. .. attribute:: Celery.events
  62. Consuming and sending events: :class:`~@events`.
  63. .. attribute:: Celery.log
  64. Logging: :class:`~@log`.
  65. .. attribute:: Celery.tasks
  66. Task registry.
  67. Accessing this attribute will also finalize the app.
  68. .. attribute:: Celery.pool
  69. Broker connection pool: :class:`~@pool`.
  70. This attribute is not related to the workers concurrency pool.
  71. .. attribute:: Celery.Task
  72. Base task class for this app.
  73. .. method:: Celery.close
  74. Cleans-up after application, like closing any pool connections.
  75. Only necessary for dynamically created apps for which you can
  76. use the with statement::
  77. with Celery(set_as_current=False) as app:
  78. with app.connection() as conn:
  79. pass
  80. .. method:: Celery.bugreport
  81. Returns a string with information useful for the Celery core
  82. developers when reporting a bug.
  83. .. method:: Celery.config_from_object(obj, silent=False)
  84. Reads configuration from object, where object is either
  85. an object or the name of a module to import.
  86. :keyword silent: If true then import errors will be ignored.
  87. .. code-block:: python
  88. >>> celery.config_from_object("myapp.celeryconfig")
  89. >>> from myapp import celeryconfig
  90. >>> celery.config_from_object(celeryconfig)
  91. .. method:: Celery.config_from_envvar(variable_name, silent=False)
  92. Read configuration from environment variable.
  93. The value of the environment variable must be the name
  94. of a module to import.
  95. .. code-block:: python
  96. >>> os.environ["CELERY_CONFIG_MODULE"] = "myapp.celeryconfig"
  97. >>> celery.config_from_envvar("CELERY_CONFIG_MODULE")
  98. .. method:: Celery.autodiscover_tasks(packages, related_name="tasks")
  99. With a list of packages, try to import modules of a specific name (by
  100. default 'tasks').
  101. For example if you have an (imagined) directory tree like this::
  102. foo/__init__.py
  103. tasks.py
  104. models.py
  105. bar/__init__.py
  106. tasks.py
  107. models.py
  108. baz/__init__.py
  109. models.py
  110. Then calling ``app.autodiscover_tasks(['foo', bar', 'baz'])`` will
  111. result in the modules ``foo.tasks`` and ``bar.tasks`` being imported.
  112. .. method:: Celery.add_defaults(d)
  113. Add default configuration from dict ``d``.
  114. If the argument is a callable function then it will be regarded
  115. as a promise, and it won't be loaded until the configuration is
  116. actually needed.
  117. This method can be compared to::
  118. >>> celery.conf.update(d)
  119. with a difference that 1) no copy will be made and 2) the dict will
  120. not be transferred when the worker spawns child processes, so
  121. it's important that the same configuration happens at import time
  122. when pickle restores the object on the other side.
  123. .. method:: Celery.setup_security(...)
  124. Setup the message-signing serializer.
  125. This will affect all application instances (a global operation).
  126. Disables untrusted serializers and if configured to use the ``auth``
  127. serializer will register the auth serializer with the provided settings
  128. into the Kombu serializer registry.
  129. :keyword allowed_serializers: List of serializer names, or content_types
  130. that should be exempt from being disabled.
  131. :keyword key: Name of private key file to use.
  132. Defaults to the :setting:`CELERY_SECURITY_KEY` setting.
  133. :keyword cert: Name of certificate file to use.
  134. Defaults to the :setting:`CELERY_SECURITY_CERTIFICATE` setting.
  135. :keyword store: Directory containing certificates.
  136. Defaults to the :setting:`CELERY_SECURITY_CERT_STORE` setting.
  137. :keyword digest: Digest algorithm used when signing messages.
  138. Default is ``sha1``.
  139. :keyword serializer: Serializer used to encode messages after
  140. they have been signed. See :setting:`CELERY_TASK_SERIALIZER` for
  141. the serializers supported.
  142. Default is ``json``.
  143. .. method:: Celery.start(argv=None)
  144. Run :program:`celery` using `argv`.
  145. Uses :data:`sys.argv` if `argv` is not specified.
  146. .. method:: Celery.task(fun, ...)
  147. Decorator to create a task class out of any callable.
  148. Examples:
  149. .. code-block:: python
  150. @celery.task
  151. def refresh_feed(url):
  152. return ...
  153. with setting extra options:
  154. .. code-block:: python
  155. @celery.task(exchange="feeds")
  156. def refresh_feed(url):
  157. return ...
  158. .. admonition:: App Binding
  159. For custom apps the task decorator returns proxy
  160. objects, so that the act of creating the task is not performed
  161. until the task is used or the task registry is accessed.
  162. If you are depending on binding to be deferred, then you must
  163. not access any attributes on the returned object until the
  164. application is fully set up (finalized).
  165. .. method:: Celery.send_task(name[, args[, kwargs[, ...]]])
  166. Send task by name.
  167. :param name: Name of task to call (e.g. `"tasks.add"`).
  168. :keyword result_cls: Specify custom result class. Default is
  169. using :meth:`AsyncResult`.
  170. Otherwise supports the same arguments as :meth:`@-Task.apply_async`.
  171. .. attribute:: Celery.AsyncResult
  172. Create new result instance. See :class:`~celery.result.AsyncResult`.
  173. .. attribute:: Celery.GroupResult
  174. Create new group result instance.
  175. See :class:`~celery.result.GroupResult`.
  176. .. method:: Celery.worker_main(argv=None)
  177. Run :program:`celery worker` using `argv`.
  178. Uses :data:`sys.argv` if `argv` is not specified.
  179. .. attribute:: Celery.Worker
  180. Worker application. See :class:`~@Worker`.
  181. .. attribute:: Celery.WorkController
  182. Embeddable worker. See :class:`~@WorkController`.
  183. .. attribute:: Celery.Beat
  184. Celerybeat scheduler application.
  185. See :class:`~@Beat`.
  186. .. method:: Celery.connection(url=default, [ssl, [transport_options={}]])
  187. Establish a connection to the message broker.
  188. :param url: Either the URL or the hostname of the broker to use.
  189. :keyword hostname: URL, Hostname/IP-address of the broker.
  190. If an URL is used, then the other argument below will
  191. be taken from the URL instead.
  192. :keyword userid: Username to authenticate as.
  193. :keyword password: Password to authenticate with
  194. :keyword virtual_host: Virtual host to use (domain).
  195. :keyword port: Port to connect to.
  196. :keyword ssl: Defaults to the :setting:`BROKER_USE_SSL` setting.
  197. :keyword transport: defaults to the :setting:`BROKER_TRANSPORT`
  198. setting.
  199. :returns :class:`kombu.Connection`:
  200. .. method:: Celery.connection_or_acquire(connection=None)
  201. For use within a with-statement to get a connection from the pool
  202. if one is not already provided.
  203. :keyword connection: If not provided, then a connection will be
  204. acquired from the connection pool.
  205. .. method:: Celery.producer_or_acquire(producer=None)
  206. For use within a with-statement to get a producer from the pool
  207. if one is not already provided
  208. :keyword producer: If not provided, then a producer will be
  209. acquired from the producer pool.
  210. .. method:: Celery.mail_admins(subject, body, fail_silently=False)
  211. Sends an email to the admins in the :setting:`ADMINS` setting.
  212. .. method:: Celery.select_queues(queues=[])
  213. Select a subset of queues, where queues must be a list of queue
  214. names to keep.
  215. .. method:: Celery.now()
  216. Returns the current time and date as a :class:`~datetime.datetime`
  217. object.
  218. .. method:: Celery.set_current()
  219. Makes this the current app for this thread.
  220. .. method:: Celery.finalize()
  221. Finalizes the app by loading built-in tasks,
  222. and evaluating pending task decorators
  223. .. attribute:: Celery.Pickler
  224. Helper class used to pickle this application.
  225. Canvas primitives
  226. -----------------
  227. See :ref:`guide-canvas` for more about creating task workflows.
  228. .. class:: group(task1[, task2[, task3[,... taskN]]])
  229. Creates a group of tasks to be executed in parallel.
  230. Example::
  231. >>> res = group([add.s(2, 2), add.s(4, 4)])()
  232. >>> res.get()
  233. [4, 8]
  234. A group is lazy so you must call it to take action and evaluate
  235. the group.
  236. Calling the group returns :class:`~@GroupResult`.
  237. .. class:: chain(task1[, task2[, task3[,... taskN]]])
  238. Chains tasks together, so that each tasks follows each other
  239. by being applied as a callback of the previous task.
  240. If called with only one argument, then that argument must
  241. be an iterable of tasks to chain.
  242. Example::
  243. >>> res = chain(add.s(2, 2), add.s(4))()
  244. is effectively :math:`(2 + 2) + 4)`::
  245. >>> res.get()
  246. 8
  247. Calling a chain will return the result of the last task in the chain.
  248. You can get to the other tasks by following the ``result.parent``'s::
  249. >>> res.parent.get()
  250. 4
  251. .. class:: chord(header[, body])
  252. A chord consists of a header and a body.
  253. The header is a group of tasks that must complete before the callback is
  254. called. A chord is essentially a callback for a group of tasks.
  255. Example::
  256. >>> res = chord([add.s(2, 2), add.s(4, 4)])(sum_task.s())
  257. is effectively :math:`\Sigma ((2 + 2) + (4 + 4))`::
  258. >>> res.get()
  259. 12
  260. The body is applied with the return values of all the header
  261. tasks as a list.
  262. .. class:: subtask(task=None, args=(), kwargs={}, options={})
  263. Describes the arguments and execution options for a single task invocation.
  264. Used as the parts in a :class:`group` or to safely pass
  265. tasks around as callbacks.
  266. Subtasks can also be created from tasks::
  267. >>> add.subtask(args=(), kwargs={}, options={})
  268. or the ``.s()`` shortcut::
  269. >>> add.s(*args, **kwargs)
  270. :param task: Either a task class/instance, or the name of a task.
  271. :keyword args: Positional arguments to apply.
  272. :keyword kwargs: Keyword arguments to apply.
  273. :keyword options: Additional options to :meth:`Task.apply_async`.
  274. Note that if the first argument is a :class:`dict`, the other
  275. arguments will be ignored and the values in the dict will be used
  276. instead.
  277. >>> s = subtask("tasks.add", args=(2, 2))
  278. >>> subtask(s)
  279. {"task": "tasks.add", args=(2, 2), kwargs={}, options={}}
  280. .. method:: subtask.delay(*args, \*\*kwargs)
  281. Shortcut to :meth:`apply_async`.
  282. .. method:: subtask.apply_async(args=(), kwargs={}, ...)
  283. Apply this task asynchronously.
  284. :keyword args: Partial args to be prepended to the existing args.
  285. :keyword kwargs: Partial kwargs to be merged with the existing kwargs.
  286. :keyword options: Partial options to be merged with the existing
  287. options.
  288. See :meth:`~@Task.apply_async`.
  289. .. method:: subtask.apply(args=(), kwargs={}, ...)
  290. Same as :meth:`apply_async` but executed the task inline instead
  291. of sending a task message.
  292. .. method:: subtask.clone(args=(), kwargs={}, ...)
  293. Returns a copy of this subtask.
  294. :keyword args: Partial args to be prepended to the existing args.
  295. :keyword kwargs: Partial kwargs to be merged with the existing kwargs.
  296. :keyword options: Partial options to be merged with the existing
  297. options.
  298. .. method:: subtask.replace(args=None, kwargs=None, options=None)
  299. Replace the args, kwargs or options set for this subtask.
  300. These are only replaced if the selected is not :const:`None`.
  301. .. method:: subtask.link(other_subtask)
  302. Add a callback task to be applied if this task
  303. executes successfully.
  304. :returns: ``other_subtask`` (to work with :func:`~functools.reduce`).
  305. .. method:: subtask.link_error(other_subtask)
  306. Add a callback task to be applied if an error occurs
  307. while executing this task.
  308. :returns: ``other_subtask`` (to work with :func:`~functools.reduce`)
  309. .. method:: subtask.set(...)
  310. Set arbitrary options (same as ``.options.update(...)``).
  311. This is a chaining method call (i.e. it returns itself).
  312. .. method:: subtask.flatten_links()
  313. Gives a recursive list of dependencies (unchain if you will,
  314. but with links intact).
  315. Proxies
  316. -------
  317. .. data:: current_app
  318. The currently set app for this thread.
  319. .. data:: current_task
  320. The task currently being executed
  321. (only set in the worker, or when eager/apply is used).