guide.rst 8.4 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335
  1. .. _internals-guide:
  2. ================================
  3. Contributors Guide to the Code
  4. ================================
  5. .. contents::
  6. :local:
  7. Philosophy
  8. ==========
  9. The API>RCP Precedence Rule
  10. ---------------------------
  11. - The API is more important than Readability
  12. - Readability is more important than Convention
  13. - Convention is more important than Performance
  14. - …unless the code is a proven hot-spot.
  15. More important than anything else is the end-user API.
  16. Conventions must step aside, and any suffering is always alleviated
  17. if the end result is a better API.
  18. Conventions and Idioms Used
  19. ===========================
  20. Classes
  21. -------
  22. Naming
  23. ~~~~~~
  24. - Follows :pep:`8`.
  25. - Class names must be `CamelCase`.
  26. - but not if they are verbs, verbs shall be `lower_case`:
  27. .. code-block:: python
  28. # - test case for a class
  29. class TestMyClass(Case): # BAD
  30. pass
  31. class test_MyClass(Case): # GOOD
  32. pass
  33. # - test case for a function
  34. class TestMyFunction(Case): # BAD
  35. pass
  36. class test_my_function(Case): # GOOD
  37. pass
  38. # - "action" class (verb)
  39. class UpdateTwitterStatus(object): # BAD
  40. pass
  41. class update_twitter_status(object): # GOOD
  42. pass
  43. .. note::
  44. Sometimes it makes sense to have a class mask as a function,
  45. and there is precedence for this in the Python standard library (e.g.
  46. :class:`~contextlib.contextmanager`). Celery examples include
  47. :class:`~celery.signature`, :class:`~celery.chord`,
  48. ``inspect``, :class:`~kombu.utils.functional.promise` and more..
  49. - Factory functions and methods must be `CamelCase` (excluding verbs):
  50. .. code-block:: python
  51. class Celery(object):
  52. def consumer_factory(self): # BAD
  53. ...
  54. def Consumer(self): # GOOD
  55. ...
  56. Default values
  57. ~~~~~~~~~~~~~~
  58. Class attributes serve as default values for the instance,
  59. as this means that they can be set by either instantiation or inheritance.
  60. **Example:**
  61. .. code-block:: python
  62. class Producer(object):
  63. active = True
  64. serializer = 'json'
  65. def __init__(self, serializer=None):
  66. self.serializer = serializer or self.serializer
  67. # must check for None when value can be false-y
  68. self.active = active if active is not None else self.active
  69. A subclass can change the default value:
  70. .. code-block:: python
  71. TaskProducer(Producer):
  72. serializer = 'pickle'
  73. and the value can be set at instantiation:
  74. .. code-block:: pycon
  75. >>> producer = TaskProducer(serializer='msgpack')
  76. Exceptions
  77. ~~~~~~~~~~
  78. Custom exceptions raised by an objects methods and properties
  79. should be available as an attribute and documented in the
  80. method/property that throw.
  81. This way a user doesn't have to find out where to import the
  82. exception from, but rather use ``help(obj)`` and access
  83. the exception class from the instance directly.
  84. **Example**:
  85. .. code-block:: python
  86. class Empty(Exception):
  87. pass
  88. class Queue(object):
  89. Empty = Empty
  90. def get(self):
  91. """Get the next item from the queue.
  92. :raises Queue.Empty: if there are no more items left.
  93. """
  94. try:
  95. return self.queue.popleft()
  96. except IndexError:
  97. raise self.Empty()
  98. Composites
  99. ~~~~~~~~~~
  100. Similarly to exceptions, composite classes should be override-able by
  101. inheritance and/or instantiation. Common sense can be used when
  102. selecting what classes to include, but often it's better to add one
  103. too many: predicting what users need to override is hard (this has
  104. saved us from many a monkey patch).
  105. **Example**:
  106. .. code-block:: python
  107. class Worker(object):
  108. Consumer = Consumer
  109. def __init__(self, connection, consumer_cls=None):
  110. self.Consumer = consumer_cls or self.Consumer
  111. def do_work(self):
  112. with self.Consumer(self.connection) as consumer:
  113. self.connection.drain_events()
  114. Applications vs. "single mode"
  115. ==============================
  116. In the beginning Celery was developed for Django, simply because
  117. this enabled us get the project started quickly, while also having
  118. a large potential user base.
  119. In Django there is a global settings object, so multiple Django projects
  120. can't co-exist in the same process space, this later posed a problem
  121. for using Celery with frameworks that doesn't have this limitation.
  122. Therefore the app concept was introduced. When using apps you use 'celery'
  123. objects instead of importing things from celery sub-modules, this
  124. (unfortunately) also means that Celery essentially has two API's.
  125. Here's an example using Celery in single-mode:
  126. .. code-block:: python
  127. from celery import task
  128. from celery.task.control import inspect
  129. from .models import CeleryStats
  130. @task
  131. def write_stats_to_db():
  132. stats = inspect().stats(timeout=1)
  133. for node_name, reply in stats:
  134. CeleryStats.objects.update_stat(node_name, stats)
  135. and here's the same using Celery app objects:
  136. .. code-block:: python
  137. from .celery import celery
  138. from .models import CeleryStats
  139. @app.task
  140. def write_stats_to_db():
  141. stats = celery.control.inspect().stats(timeout=1)
  142. for node_name, reply in stats:
  143. CeleryStats.objects.update_stat(node_name, stats)
  144. In the example above the actual application instance is imported
  145. from a module in the project, this module could look something like this:
  146. .. code-block:: python
  147. from celery import Celery
  148. app = Celery(broker='amqp://')
  149. Module Overview
  150. ===============
  151. - celery.app
  152. This is the core of Celery: the entry-point for all functionality.
  153. - celery.loaders
  154. Every app must have a loader. The loader decides how configuration
  155. is read, what happens when the worker starts, when a task starts and ends,
  156. and so on.
  157. The loaders included are:
  158. - app
  159. Custom celery app instances uses this loader by default.
  160. - default
  161. "single-mode" uses this loader by default.
  162. Extension loaders also exist, like :pypi:`django-celery`,
  163. :pypi:`celery-pylons` and so on.
  164. - celery.worker
  165. This is the worker implementation.
  166. - celery.backends
  167. Task result backends live here.
  168. - celery.apps
  169. Major user applications: worker and beat.
  170. The command-line wrappers for these are in celery.bin (see below)
  171. - celery.bin
  172. Command-line applications.
  173. :file:`setup.py` creates setuptools entry-points for these.
  174. - celery.concurrency
  175. Execution pool implementations (prefork, eventlet, gevent, solo).
  176. - celery.db
  177. Database models for the SQLAlchemy database result backend.
  178. (should be moved into :mod:`celery.backends.database`)
  179. - celery.events
  180. Sending and consuming monitoring events, also includes curses monitor,
  181. event dumper and utilities to work with in-memory cluster state.
  182. - celery.execute.trace
  183. How tasks are executed and traced by the worker, and in eager mode.
  184. - celery.security
  185. Security related functionality, currently a serializer using
  186. cryptographic digests.
  187. - celery.task
  188. single-mode interface to creating tasks, and controlling workers.
  189. - celery.tests
  190. The unittest suite.
  191. - celery.utils
  192. Utility functions used by the celery code base.
  193. Much of it is there to be compatible across Python versions.
  194. - celery.contrib
  195. Additional public code that doesn't fit into any other name-space.
  196. Worker overview
  197. ===============
  198. * `celery.bin.worker:Worker`
  199. This is the command-line interface to the worker.
  200. Responsibilities:
  201. * Daemonization when :option:`--detach <celery worker --detach>` set,
  202. * dropping privileges when using :option:`--uid <celery worker --uid>`/
  203. :option:`--gid <celery worker --gid>` arguments
  204. * Installs "concurrency patches" (eventlet/gevent monkey patches).
  205. ``app.worker_main(argv)`` calls
  206. ``instantiate('celery.bin.worker:Worker')(app).execute_from_commandline(argv)``
  207. * `app.Worker` -> `celery.apps.worker:Worker`
  208. Responsibilities:
  209. * sets up logging and redirects standard outs
  210. * installs signal handlers (`TERM`/`HUP`/`STOP`/`USR1` (cry)/`USR2` (rdb))
  211. * prints banner and warnings (e.g. pickle warning)
  212. * handles the :option:`celery worker --purge` argument
  213. * `app.WorkController` -> `celery.worker.WorkController`
  214. This is the real worker, built up around bootsteps.