app-overview.rst 6.5 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233
  1. =============================
  2. "The Big Instance" Refactor
  3. =============================
  4. The `app` branch is a work-in-progress to remove
  5. the use of a global configuration in Celery.
  6. Celery can now be instantiated and several
  7. instances of Celery may exist in the same process space.
  8. Also, large parts can be customized without resorting to monkey
  9. patching.
  10. Examples
  11. ========
  12. Creating a Celery instance::
  13. >>> from celery import Celery
  14. >>> app = Celery()
  15. >>> app.config_from_object('celeryconfig')
  16. >>> #app.config_from_envvar('CELERY_CONFIG_MODULE')
  17. Creating tasks:
  18. .. code-block:: python
  19. @app.task
  20. def add(x, y):
  21. return x + y
  22. Creating custom Task subclasses:
  23. .. code-block:: python
  24. Task = celery.create_task_cls()
  25. class DebugTask(Task):
  26. def on_failure(self, *args, **kwargs):
  27. import pdb
  28. pdb.set_trace()
  29. @app.task(base=DebugTask)
  30. def add(x, y):
  31. return x + y
  32. Starting a worker:
  33. .. code-block:: python
  34. worker = celery.Worker(loglevel='INFO')
  35. Getting access to the configuration:
  36. .. code-block:: python
  37. celery.conf.task_always_eager = True
  38. celery.conf['task_always_eager'] = True
  39. Controlling workers::
  40. >>> celery.control.inspect().active()
  41. >>> celery.control.rate_limit(add.name, '100/m')
  42. >>> celery.control.broadcast('shutdown')
  43. >>> celery.control.discard_all()
  44. Other interesting attributes::
  45. # Establish broker connection.
  46. >>> celery.broker_connection()
  47. # AMQP Specific features.
  48. >>> celery.amqp
  49. >>> celery.amqp.Router
  50. >>> celery.amqp.get_queues()
  51. >>> celery.amqp.get_task_consumer()
  52. # Loader
  53. >>> celery.loader
  54. # Default backend
  55. >>> celery.backend
  56. As you can probably see, this really opens up another
  57. dimension of customization abilities.
  58. Deprecated
  59. ==========
  60. * ``celery.task.ping``
  61. ``celery.task.PingTask``
  62. Inferior to the ping remote control command.
  63. Will be removed in Celery 2.3.
  64. Aliases (Pending deprecation)
  65. =============================
  66. * ``celery.task.base``
  67. * ``.Task`` -> {``app.Task`` / :class:`celery.app.task.Task`}
  68. * ``celery.task.sets``
  69. * ``.TaskSet`` -> {``app.TaskSet``}
  70. * ``celery.decorators`` / ``celery.task``
  71. * ``.task`` -> {``app.task``}
  72. * ``celery.execute``
  73. * ``.apply_async`` -> {``task.apply_async``}
  74. * ``.apply`` -> {``task.apply``}
  75. * ``.send_task`` -> {``app.send_task``}
  76. * ``.delay_task`` -> *no alternative*
  77. * ``celery.log``
  78. * ``.get_default_logger`` -> {``app.log.get_default_logger``}
  79. * ``.setup_logger`` -> {``app.log.setup_logger``}
  80. * ``.get_task_logger`` -> {``app.log.get_task_logger``}
  81. * ``.setup_task_logger`` -> {``app.log.setup_task_logger``}
  82. * ``.setup_logging_subsystem`` -> {``app.log.setup_logging_subsystem``}
  83. * ``.redirect_stdouts_to_logger`` -> {``app.log.redirect_stdouts_to_logger``}
  84. * ``celery.messaging``
  85. * ``.establish_connection`` -> {``app.broker_connection``}
  86. * ``.with_connection`` -> {``app.with_connection``}
  87. * ``.get_consumer_set`` -> {``app.amqp.get_task_consumer``}
  88. * ``.TaskPublisher`` -> {``app.amqp.TaskPublisher``}
  89. * ``.TaskConsumer`` -> {``app.amqp.TaskConsumer``}
  90. * ``.ConsumerSet`` -> {``app.amqp.ConsumerSet``}
  91. * ``celery.conf.*`` -> {``app.conf``}
  92. **NOTE**: All configuration keys are now named the same
  93. as in the configuration. So the key ``task_always_eager``
  94. is accessed as::
  95. >>> app.conf.task_always_eager
  96. instead of::
  97. >>> from celery import conf
  98. >>> conf.always_eager
  99. * ``.get_queues`` -> {``app.amqp.get_queues``}
  100. * ``celery.task.control``
  101. * ``.broadcast`` -> {``app.control.broadcast``}
  102. * ``.rate_limit`` -> {``app.control.rate_limit``}
  103. * ``.ping`` -> {``app.control.ping``}
  104. * ``.revoke`` -> {``app.control.revoke``}
  105. * ``.discard_all`` -> {``app.control.discard_all``}
  106. * ``.inspect`` -> {``app.control.inspect``}
  107. * ``celery.utils.info``
  108. * ``.humanize_seconds`` -> ``celery.utils.time.humanize_seconds``
  109. * ``.textindent`` -> ``celery.utils.textindent``
  110. * ``.get_broker_info`` -> {``app.amqp.get_broker_info``}
  111. * ``.format_broker_info`` -> {``app.amqp.format_broker_info``}
  112. * ``.format_queues`` -> {``app.amqp.format_queues``}
  113. Default App Usage
  114. =================
  115. To be backward compatible, it must be possible
  116. to use all the classes/functions without passing
  117. an explicit app instance.
  118. This is achieved by having all app-dependent objects
  119. use :data:`~celery.app.default_app` if the app instance
  120. is missing.
  121. .. code-block:: python
  122. from celery.app import app_or_default
  123. class SomeClass(object):
  124. def __init__(self, app=None):
  125. self.app = app_or_default(app)
  126. The problem with this approach is that there's a chance
  127. that the app instance is lost along the way, and everything
  128. seems to be working normally. Testing app instance leaks
  129. is hard. The environment variable :envvar:`CELERY_TRACE_APP`
  130. can be used, when this is enabled :func:`celery.app.app_or_default`
  131. will raise an exception whenever it has to go back to the default app
  132. instance.
  133. App Dependency Tree
  134. -------------------
  135. * {``app``}
  136. * ``celery.loaders.base.BaseLoader``
  137. * ``celery.backends.base.BaseBackend``
  138. * {``app.TaskSet``}
  139. * ``celery.task.sets.TaskSet`` (``app.TaskSet``)
  140. * [``app.TaskSetResult``]
  141. * ``celery.result.TaskSetResult`` (``app.TaskSetResult``)
  142. * {``app.AsyncResult``}
  143. * ``celery.result.BaseAsyncResult`` / ``celery.result.AsyncResult``
  144. * ``celery.bin.worker.WorkerCommand``
  145. * ``celery.apps.worker.Worker``
  146. * ``celery.worker.WorkerController``
  147. * ``celery.worker.consumer.Consumer``
  148. * ``celery.worker.request.Request``
  149. * ``celery.events.EventDispatcher``
  150. * ``celery.worker.control.ControlDispatch``
  151. * ``celery.worker.control.registry.Panel``
  152. * ``celery.pidbox.BroadcastPublisher``
  153. * ``celery.pidbox.BroadcastConsumer``
  154. * ``celery.beat.EmbeddedService``
  155. * ``celery.bin.events.EvCommand``
  156. * ``celery.events.snapshot.evcam``
  157. * ``celery.events.snapshot.Polaroid``
  158. * ``celery.events.EventReceiver``
  159. * ``celery.events.cursesmon.evtop``
  160. * ``celery.events.EventReceiver``
  161. * ``celery.events.cursesmon.CursesMonitor``
  162. * ``celery.events.dumper``
  163. * ``celery.events.EventReceiver``
  164. * ``celery.bin.amqp.AMQPAdmin``
  165. * ``celery.bin.beat.BeatCommand``
  166. * ``celery.apps.beat.Beat``
  167. * ``celery.beat.Service``
  168. * ``celery.beat.Scheduler``