CONTRIBUTING.rst 34 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014101510161017101810191020102110221023102410251026102710281029103010311032103310341035103610371038103910401041104210431044104510461047104810491050105110521053105410551056105710581059106010611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119112011211122112311241125112611271128112911301131113211331134113511361137113811391140114111421143114411451146114711481149115011511152115311541155115611571158115911601161116211631164116511661167116811691170117111721173117411751176117711781179118011811182118311841185118611871188118911901191119211931194119511961197119811991200120112021203120412051206120712081209121012111212121312141215121612171218121912201221122212231224122512261227
  1. .. _contributing:
  2. ==============
  3. Contributing
  4. ==============
  5. Welcome!
  6. This document is fairly extensive and you aren't really expected
  7. to study this in detail for small contributions;
  8. The most important rule is that contributing must be easy
  9. and that the community is friendly and not nitpicking on details,
  10. such as coding style.
  11. If you're reporting a bug you should read the Reporting bugs section
  12. below to ensure that your bug report contains enough information
  13. to successfully diagnose the issue, and if you're contributing code
  14. you should try to mimic the conventions you see surrounding the code
  15. you're working on, but in the end all patches will be cleaned up by
  16. the person merging the changes so don't worry too much.
  17. .. contents::
  18. :local:
  19. .. _community-code-of-conduct:
  20. Community Code of Conduct
  21. =========================
  22. The goal is to maintain a diverse community that's pleasant for everyone.
  23. That's why we would greatly appreciate it if everyone contributing to and
  24. interacting with the community also followed this Code of Conduct.
  25. The Code of Conduct covers our behavior as members of the community,
  26. in any forum, mailing list, wiki, website, Internet relay chat (IRC), public
  27. meeting or private correspondence.
  28. The Code of Conduct is heavily based on the `Ubuntu Code of Conduct`_, and
  29. the `Pylons Code of Conduct`_.
  30. .. _`Ubuntu Code of Conduct`: https://www.ubuntu.com/community/conduct
  31. .. _`Pylons Code of Conduct`: http://docs.pylonshq.com/community/conduct.html
  32. Be considerate
  33. --------------
  34. Your work will be used by other people, and you in turn will depend on the
  35. work of others. Any decision you take will affect users and colleagues, and
  36. we expect you to take those consequences into account when making decisions.
  37. Even if it's not obvious at the time, our contributions to Celery will impact
  38. the work of others. For example, changes to code, infrastructure, policy,
  39. documentation and translations during a release may negatively impact
  40. others work.
  41. Be respectful
  42. -------------
  43. The Celery community and its members treat one another with respect. Everyone
  44. can make a valuable contribution to Celery. We may not always agree, but
  45. disagreement is no excuse for poor behavior and poor manners. We might all
  46. experience some frustration now and then, but we cannot allow that frustration
  47. to turn into a personal attack. It's important to remember that a community
  48. where people feel uncomfortable or threatened isn't a productive one. We
  49. expect members of the Celery community to be respectful when dealing with
  50. other contributors as well as with people outside the Celery project and with
  51. users of Celery.
  52. Be collaborative
  53. ----------------
  54. Collaboration is central to Celery and to the larger free software community.
  55. We should always be open to collaboration. Your work should be done
  56. transparently and patches from Celery should be given back to the community
  57. when they're made, not just when the distribution releases. If you wish
  58. to work on new code for existing upstream projects, at least keep those
  59. projects informed of your ideas and progress. It many not be possible to
  60. get consensus from upstream, or even from your colleagues about the correct
  61. implementation for an idea, so don't feel obliged to have that agreement
  62. before you begin, but at least keep the outside world informed of your work,
  63. and publish your work in a way that allows outsiders to test, discuss, and
  64. contribute to your efforts.
  65. When you disagree, consult others
  66. ---------------------------------
  67. Disagreements, both political and technical, happen all the time and
  68. the Celery community is no exception. It's important that we resolve
  69. disagreements and differing views constructively and with the help of the
  70. community and community process. If you really want to go a different
  71. way, then we encourage you to make a derivative distribution or alternate
  72. set of packages that still build on the work we've done to utilize as common
  73. of a core as possible.
  74. When you're unsure, ask for help
  75. --------------------------------
  76. Nobody knows everything, and nobody is expected to be perfect. Asking
  77. questions avoids many problems down the road, and so questions are
  78. encouraged. Those who are asked questions should be responsive and helpful.
  79. However, when asking a question, care must be taken to do so in an appropriate
  80. forum.
  81. Step down considerately
  82. -----------------------
  83. Developers on every project come and go and Celery is no different. When you
  84. leave or disengage from the project, in whole or in part, we ask that you do
  85. so in a way that minimizes disruption to the project. This means you should
  86. tell people you're leaving and take the proper steps to ensure that others
  87. can pick up where you leave off.
  88. .. _reporting-bugs:
  89. Reporting Bugs
  90. ==============
  91. .. _vulnsec:
  92. Security
  93. --------
  94. You must never report security related issues, vulnerabilities or bugs
  95. including sensitive information to the bug tracker, or elsewhere in public.
  96. Instead sensitive bugs must be sent by email to ``security@celeryproject.org``.
  97. If you'd like to submit the information encrypted our PGP key is::
  98. -----BEGIN PGP PUBLIC KEY BLOCK-----
  99. Version: GnuPG v1.4.15 (Darwin)
  100. mQENBFJpWDkBCADFIc9/Fpgse4owLNvsTC7GYfnJL19XO0hnL99sPx+DPbfr+cSE
  101. 9wiU+Wp2TfUX7pCLEGrODiEP6ZCZbgtiPgId+JYvMxpP6GXbjiIlHRw1EQNH8RlX
  102. cVxy3rQfVv8PGGiJuyBBjxzvETHW25htVAZ5TI1+CkxmuyyEYqgZN2fNd0wEU19D
  103. +c10G1gSECbCQTCbacLSzdpngAt1Gkrc96r7wGHBBSvDaGDD2pFSkVuTLMbIRrVp
  104. lnKOPMsUijiip2EMr2DvfuXiUIUvaqInTPNWkDynLoh69ib5xC19CSVLONjkKBsr
  105. Pe+qAY29liBatatpXsydY7GIUzyBT3MzgMJlABEBAAG0MUNlbGVyeSBTZWN1cml0
  106. eSBUZWFtIDxzZWN1cml0eUBjZWxlcnlwcm9qZWN0Lm9yZz6JATgEEwECACIFAlJp
  107. WDkCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEOArFOUDCicIw1IH/26f
  108. CViDC7/P13jr+srRdjAsWvQztia9HmTlY8cUnbmkR9w6b6j3F2ayw8VhkyFWgYEJ
  109. wtPBv8mHKADiVSFARS+0yGsfCkia5wDSQuIv6XqRlIrXUyqJbmF4NUFTyCZYoh+C
  110. ZiQpN9xGhFPr5QDlMx2izWg1rvWlG1jY2Es1v/xED3AeCOB1eUGvRe/uJHKjGv7J
  111. rj0pFcptZX+WDF22AN235WYwgJM6TrNfSu8sv8vNAQOVnsKcgsqhuwomSGsOfMQj
  112. LFzIn95MKBBU1G5wOs7JtwiV9jefGqJGBO2FAvOVbvPdK/saSnB+7K36dQcIHqms
  113. 5hU4Xj0RIJiod5idlRC5AQ0EUmlYOQEIAJs8OwHMkrdcvy9kk2HBVbdqhgAREMKy
  114. gmphDp7prRL9FqSY/dKpCbG0u82zyJypdb7QiaQ5pfPzPpQcd2dIcohkkh7G3E+e
  115. hS2L9AXHpwR26/PzMBXyr2iNnNc4vTksHvGVDxzFnRpka6vbI/hrrZmYNYh9EAiv
  116. uhE54b3/XhXwFgHjZXb9i8hgJ3nsO0pRwvUAM1bRGMbvf8e9F+kqgV0yWYNnh6QL
  117. 4Vpl1+epqp2RKPHyNQftbQyrAHXT9kQF9pPlx013MKYaFTADscuAp4T3dy7xmiwS
  118. crqMbZLzfrxfFOsNxTUGE5vmJCcm+mybAtRo4aV6ACohAO9NevMx8pUAEQEAAYkB
  119. HwQYAQIACQUCUmlYOQIbDAAKCRDgKxTlAwonCNFbB/9esir/f7TufE+isNqErzR/
  120. aZKZo2WzZR9c75kbqo6J6DYuUHe6xI0OZ2qZ60iABDEZAiNXGulysFLCiPdatQ8x
  121. 8zt3DF9BMkEck54ZvAjpNSern6zfZb1jPYWZq3TKxlTs/GuCgBAuV4i5vDTZ7xK/
  122. aF+OFY5zN7ciZHkqLgMiTZ+RhqRcK6FhVBP/Y7d9NlBOcDBTxxE1ZO1ute6n7guJ
  123. ciw4hfoRk8qNN19szZuq3UU64zpkM2sBsIFM9tGF2FADRxiOaOWZHmIyVZriPFqW
  124. RUwjSjs7jBVNq0Vy4fCu/5+e+XLOUBOoqtM5W7ELt0t1w9tXebtPEetV86in8fU2
  125. =0chn
  126. -----END PGP PUBLIC KEY BLOCK-----
  127. Other bugs
  128. ----------
  129. Bugs can always be described to the :ref:`mailing-list`, but the best
  130. way to report an issue and to ensure a timely response is to use the
  131. issue tracker.
  132. 1) **Create a GitHub account**.
  133. You need to `create a GitHub account`_ to be able to create new issues
  134. and participate in the discussion.
  135. .. _`create a GitHub account`: https://github.com/signup/free
  136. 2) **Determine if your bug is really a bug**.
  137. You shouldn't file a bug if you're requesting support. For that you can use
  138. the :ref:`mailing-list`, or :ref:`irc-channel`.
  139. 3) **Make sure your bug hasn't already been reported**.
  140. Search through the appropriate Issue tracker. If a bug like yours was found,
  141. check if you have new information that could be reported to help
  142. the developers fix the bug.
  143. 4) **Check if you're using the latest version**.
  144. A bug could be fixed by some other improvements and fixes - it might not have an
  145. existing report in the bug tracker. Make sure you're using the latest releases of
  146. celery, billiard, kombu, amqp, and vine.
  147. 5) **Collect information about the bug**.
  148. To have the best chance of having a bug fixed, we need to be able to easily
  149. reproduce the conditions that caused it. Most of the time this information
  150. will be from a Python traceback message, though some bugs might be in design,
  151. spelling or other errors on the website/docs/code.
  152. A) If the error is from a Python traceback, include it in the bug report.
  153. B) We also need to know what platform you're running (Windows, macOS, Linux,
  154. etc.), the version of your Python interpreter, and the version of Celery,
  155. and related packages that you were running when the bug occurred.
  156. C) If you're reporting a race condition or a deadlock, tracebacks can be
  157. hard to get or might not be that useful. Try to inspect the process to
  158. get more diagnostic data. Some ideas:
  159. * Enable Celery's :ref:`breakpoint signal <breakpoint_signal>` and use it
  160. to inspect the process's state. This will allow you to open a
  161. :mod:`pdb` session.
  162. * Collect tracing data using `strace`_(Linux),
  163. :command:`dtruss` (macOS), and :command:`ktrace` (BSD),
  164. `ltrace`_, and `lsof`_.
  165. D) Include the output from the :command:`celery report` command:
  166. .. code-block:: console
  167. $ celery -A proj report
  168. This will also include your configuration settings and it try to
  169. remove values for keys known to be sensitive, but make sure you also
  170. verify the information before submitting so that it doesn't contain
  171. confidential information like API tokens and authentication
  172. credentials.
  173. 6) **Submit the bug**.
  174. By default `GitHub`_ will email you to let you know when new comments have
  175. been made on your bug. In the event you've turned this feature off, you
  176. should check back on occasion to ensure you don't miss any questions a
  177. developer trying to fix the bug might ask.
  178. .. _`GitHub`: https://github.com
  179. .. _`strace`: https://en.wikipedia.org/wiki/Strace
  180. .. _`ltrace`: https://en.wikipedia.org/wiki/Ltrace
  181. .. _`lsof`: https://en.wikipedia.org/wiki/Lsof
  182. .. _issue-trackers:
  183. Issue Trackers
  184. --------------
  185. Bugs for a package in the Celery ecosystem should be reported to the relevant
  186. issue tracker.
  187. * :pypi:`celery`: https://github.com/celery/celery/issues/
  188. * :pypi:`kombu`: https://github.com/celery/kombu/issues
  189. * :pypi:`amqp`: https://github.com/celery/py-amqp/issues
  190. * :pypi:`vine`: https://github.com/celery/vine/issues
  191. * :pypi:`librabbitmq`: https://github.com/celery/librabbitmq/issues
  192. * :pypi:`django-celery-beat`: https://github.com/celery/django-celery-beat/issues
  193. * :pypi:`django-celery-results`: https://github.com/celery/django-celery-results/issues
  194. If you're unsure of the origin of the bug you can ask the
  195. :ref:`mailing-list`, or just use the Celery issue tracker.
  196. Contributors guide to the code base
  197. ===================================
  198. There's a separate section for internal details,
  199. including details about the code base and a style guide.
  200. Read :ref:`internals-guide` for more!
  201. .. _versions:
  202. Versions
  203. ========
  204. Version numbers consists of a major version, minor version and a release number.
  205. Since version 2.1.0 we use the versioning semantics described by
  206. SemVer: http://semver.org.
  207. Stable releases are published at PyPI
  208. while development releases are only available in the GitHub git repository as tags.
  209. All version tags starts with “v”, so version 0.8.0 is the tag v0.8.0.
  210. .. _git-branches:
  211. Branches
  212. ========
  213. Current active version branches:
  214. * dev (which git calls "master") (https://github.com/celery/celery/tree/master)
  215. * 4.0 (https://github.com/celery/celery/tree/4.0)
  216. * 3.1 (https://github.com/celery/celery/tree/3.1)
  217. * 3.0 (https://github.com/celery/celery/tree/3.0)
  218. You can see the state of any branch by looking at the Changelog:
  219. https://github.com/celery/celery/blob/master/Changelog
  220. If the branch is in active development the topmost version info should
  221. contain meta-data like:
  222. .. code-block:: restructuredtext
  223. 2.4.0
  224. ======
  225. :release-date: TBA
  226. :status: DEVELOPMENT
  227. :branch: dev (git calls this master)
  228. The ``status`` field can be one of:
  229. * ``PLANNING``
  230. The branch is currently experimental and in the planning stage.
  231. * ``DEVELOPMENT``
  232. The branch is in active development, but the test suite should
  233. be passing and the product should be working and possible for users to test.
  234. * ``FROZEN``
  235. The branch is frozen, and no more features will be accepted.
  236. When a branch is frozen the focus is on testing the version as much
  237. as possible before it is released.
  238. dev branch
  239. ----------
  240. The dev branch (called "master" by git), is where development of the next
  241. version happens.
  242. Maintenance branches
  243. --------------------
  244. Maintenance branches are named after the version -- for example,
  245. the maintenance branch for the 2.2.x series is named ``2.2``.
  246. Previously these were named ``releaseXX-maint``.
  247. The versions we currently maintain is:
  248. * 3.1
  249. This is the current series.
  250. * 3.0
  251. This is the previous series, and the last version to support Python 2.5.
  252. Archived branches
  253. -----------------
  254. Archived branches are kept for preserving history only,
  255. and theoretically someone could provide patches for these if they depend
  256. on a series that's no longer officially supported.
  257. An archived version is named ``X.Y-archived``.
  258. Our currently archived branches are:
  259. * :github_branch:`2.5-archived`
  260. * :github_branch:`2.4-archived`
  261. * :github_branch:`2.3-archived`
  262. * :github_branch:`2.1-archived`
  263. * :github_branch:`2.0-archived`
  264. * :github_branch:`1.0-archived`
  265. Feature branches
  266. ----------------
  267. Major new features are worked on in dedicated branches.
  268. There's no strict naming requirement for these branches.
  269. Feature branches are removed once they've been merged into a release branch.
  270. Tags
  271. ====
  272. - Tags are used exclusively for tagging releases. A release tag is
  273. named with the format ``vX.Y.Z`` -- for example ``v2.3.1``.
  274. - Experimental releases contain an additional identifier ``vX.Y.Z-id`` --
  275. for example ``v3.0.0-rc1``.
  276. - Experimental tags may be removed after the official release.
  277. .. _contributing-changes:
  278. Working on Features & Patches
  279. =============================
  280. .. note::
  281. Contributing to Celery should be as simple as possible,
  282. so none of these steps should be considered mandatory.
  283. You can even send in patches by email if that's your preferred
  284. work method. We won't like you any less, any contribution you make
  285. is always appreciated!
  286. However following these steps may make maintainers life easier,
  287. and may mean that your changes will be accepted sooner.
  288. Forking and setting up the repository
  289. -------------------------------------
  290. First you need to fork the Celery repository, a good introduction to this
  291. is in the GitHub Guide: `Fork a Repo`_.
  292. After you have cloned the repository you should checkout your copy
  293. to a directory on your machine:
  294. .. code-block:: console
  295. $ git clone git@github.com:username/celery.git
  296. When the repository is cloned enter the directory to set up easy access
  297. to upstream changes:
  298. .. code-block:: console
  299. $ cd celery
  300. $ git remote add upstream git://github.com/celery/celery.git
  301. $ git fetch upstream
  302. If you need to pull in new changes from upstream you should
  303. always use the ``--rebase`` option to ``git pull``:
  304. .. code-block:: console
  305. git pull --rebase upstream master
  306. With this option you don't clutter the history with merging
  307. commit notes. See `Rebasing merge commits in git`_.
  308. If you want to learn more about rebasing see the `Rebase`_
  309. section in the GitHub guides.
  310. If you need to work on a different branch than the one git calls ``master``, you can
  311. fetch and checkout a remote branch like this::
  312. git checkout --track -b 3.0-devel origin/3.0-devel
  313. .. _`Fork a Repo`: https://help.github.com/fork-a-repo/
  314. .. _`Rebasing merge commits in git`:
  315. https://notes.envato.com/developers/rebasing-merge-commits-in-git/
  316. .. _`Rebase`: https://help.github.com/rebase/
  317. .. _contributing-docker-development:
  318. Developing and Testing with Docker
  319. ----------------------------------
  320. Because of the many components of Celery, such as a broker and backend,
  321. `Docker`_ and `docker-compose`_ can be utilized to greatly simplify the
  322. development and testing cycle. The Docker configuration here requires a
  323. Docker version of at least 17.09.
  324. The Docker components can be found within the :file:`docker/` folder and the
  325. Docker image can be built via:
  326. .. code-block:: console
  327. $ docker-compose build celery
  328. and run via:
  329. .. code-block:: console
  330. $ docker-compose run --rm celery <command>
  331. where <command> is a command to execute in a Docker container. The `--rm` flag
  332. indicates that the container should be removed after it is exited and is useful
  333. to prevent accumulation of unwanted containers.
  334. Some useful commands to run:
  335. * ``bash``
  336. To enter the Docker container like a normal shell
  337. * ``make test``
  338. To run the test suite
  339. * ``tox``
  340. To run tox and test against a variety of configurations
  341. By default, docker-compose will mount the Celery and test folders in the Docker
  342. container, allowing code changes and testing to be immediately visible inside
  343. the Docker container. Environment variables, such as the broker and backend to
  344. use are also defined in the :file:`docker/docker-compose.yml` file.
  345. .. _`Docker`: https://www.docker.com/
  346. .. _`docker-compose`: https://docs.docker.com/compose/
  347. .. _contributing-testing:
  348. Running the unit test suite
  349. ---------------------------
  350. To run the Celery test suite you need to install a few dependencies.
  351. A complete list of the dependencies needed are located in
  352. :file:`requirements/test.txt`.
  353. If you're working on the development version, then you need to
  354. install the development requirements first:
  355. .. code-block:: console
  356. $ pip install -U -r requirements/dev.txt
  357. THIS REQUIREMENT FILE MAY NOT BE PRESENT, SKIP IF NOT FOUND.
  358. Both the stable and the development version have testing related
  359. dependencies, so install these next:
  360. .. code-block:: console
  361. $ pip install -U -r requirements/test.txt
  362. $ pip install -U -r requirements/default.txt
  363. After installing the dependencies required, you can now execute
  364. the test suite by calling :pypi:`py.test <pytest>`:
  365. .. code-block:: console
  366. $ py.test
  367. Some useful options to :command:`py.test` are:
  368. * ``-x``
  369. Stop running the tests at the first test that fails.
  370. * ``-s``
  371. Don't capture output
  372. * ``-v``
  373. Run with verbose output.
  374. If you want to run the tests for a single test file only
  375. you can do so like this:
  376. .. code-block:: console
  377. $ py.test t/unit/worker/test_worker_job.py
  378. .. _contributing-pull-requests:
  379. Creating pull requests
  380. ----------------------
  381. When your feature/bugfix is complete you may want to submit
  382. a pull requests so that it can be reviewed by the maintainers.
  383. Creating pull requests is easy, and also let you track the progress
  384. of your contribution. Read the `Pull Requests`_ section in the GitHub
  385. Guide to learn how this is done.
  386. You can also attach pull requests to existing issues by following
  387. the steps outlined here: https://bit.ly/koJoso
  388. .. _`Pull Requests`: http://help.github.com/send-pull-requests/
  389. .. _contributing-coverage:
  390. Calculating test coverage
  391. ~~~~~~~~~~~~~~~~~~~~~~~~~
  392. To calculate test coverage you must first install the :pypi:`pytest-cov` module.
  393. Installing the :pypi:`pytest-cov` module:
  394. .. code-block:: console
  395. $ pip install -U pytest-cov
  396. Code coverage in HTML format
  397. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  398. #. Run :command:`py.test` with the ``--cov-report=html`` argument enabled:
  399. .. code-block:: console
  400. $ py.test --cov=celery --cov-report=html
  401. #. The coverage output will then be located in the :file:`htmlcov/` directory:
  402. .. code-block:: console
  403. $ open htmlcov/index.html
  404. Code coverage in XML (Cobertura-style)
  405. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  406. #. Run :command:`py.test` with the ``--cov-report=xml`` argument enabled:
  407. .. code-block:: console
  408. $ py.test --cov=celery --cov-report=xml
  409. #. The coverage XML output will then be located in the :file:`coverage.xml` file.
  410. .. _contributing-tox:
  411. Running the tests on all supported Python versions
  412. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  413. There's a :pypi:`tox` configuration file in the top directory of the
  414. distribution.
  415. To run the tests for all supported Python versions simply execute:
  416. .. code-block:: console
  417. $ tox
  418. Use the ``tox -e`` option if you only want to test specific Python versions:
  419. .. code-block:: console
  420. $ tox -e 2.7
  421. Building the documentation
  422. --------------------------
  423. To build the documentation you need to install the dependencies
  424. listed in :file:`requirements/docs.txt` and :file:`requirements/default.txt`:
  425. .. code-block:: console
  426. $ pip install -U -r requirements/docs.txt
  427. $ pip install -U -r requirements/default.txt
  428. After these dependencies are installed you should be able to
  429. build the docs by running:
  430. .. code-block:: console
  431. $ cd docs
  432. $ rm -rf _build
  433. $ make html
  434. Make sure there are no errors or warnings in the build output.
  435. After building succeeds the documentation is available at :file:`_build/html`.
  436. .. _contributing-verify:
  437. Verifying your contribution
  438. ---------------------------
  439. To use these tools you need to install a few dependencies. These dependencies
  440. can be found in :file:`requirements/pkgutils.txt`.
  441. Installing the dependencies:
  442. .. code-block:: console
  443. $ pip install -U -r requirements/pkgutils.txt
  444. pyflakes & PEP-8
  445. ~~~~~~~~~~~~~~~~
  446. To ensure that your changes conform to :pep:`8` and to run pyflakes
  447. execute:
  448. .. code-block:: console
  449. $ make flakecheck
  450. To not return a negative exit code when this command fails use
  451. the ``flakes`` target instead:
  452. .. code-block:: console
  453. $ make flakes
  454. API reference
  455. ~~~~~~~~~~~~~
  456. To make sure that all modules have a corresponding section in the API
  457. reference please execute:
  458. .. code-block:: console
  459. $ make apicheck
  460. $ make indexcheck
  461. If files are missing you can add them by copying an existing reference file.
  462. If the module is internal it should be part of the internal reference
  463. located in :file:`docs/internals/reference/`. If the module is public
  464. it should be located in :file:`docs/reference/`.
  465. For example if reference is missing for the module ``celery.worker.awesome``
  466. and this module is considered part of the public API, use the following steps:
  467. Use an existing file as a template:
  468. .. code-block:: console
  469. $ cd docs/reference/
  470. $ cp celery.schedules.rst celery.worker.awesome.rst
  471. Edit the file using your favorite editor:
  472. .. code-block:: console
  473. $ vim celery.worker.awesome.rst
  474. # change every occurrence of ``celery.schedules`` to
  475. # ``celery.worker.awesome``
  476. Edit the index using your favorite editor:
  477. .. code-block:: console
  478. $ vim index.rst
  479. # Add ``celery.worker.awesome`` to the index.
  480. Commit your changes:
  481. .. code-block:: console
  482. # Add the file to git
  483. $ git add celery.worker.awesome.rst
  484. $ git add index.rst
  485. $ git commit celery.worker.awesome.rst index.rst \
  486. -m "Adds reference for celery.worker.awesome"
  487. .. _coding-style:
  488. Coding Style
  489. ============
  490. You should probably be able to pick up the coding style
  491. from surrounding code, but it is a good idea to be aware of the
  492. following conventions.
  493. * All Python code must follow the :pep:`8` guidelines.
  494. :pypi:`pep8` is a utility you can use to verify that your code
  495. is following the conventions.
  496. * Docstrings must follow the :pep:`257` conventions, and use the following
  497. style.
  498. Do this:
  499. .. code-block:: python
  500. def method(self, arg):
  501. """Short description.
  502. More details.
  503. """
  504. or:
  505. .. code-block:: python
  506. def method(self, arg):
  507. """Short description."""
  508. but not this:
  509. .. code-block:: python
  510. def method(self, arg):
  511. """
  512. Short description.
  513. """
  514. * Lines shouldn't exceed 78 columns.
  515. You can enforce this in :command:`vim` by setting the ``textwidth`` option:
  516. .. code-block:: vim
  517. set textwidth=78
  518. If adhering to this limit makes the code less readable, you have one more
  519. character to go on. This means 78 is a soft limit, and 79 is the hard
  520. limit :)
  521. * Import order
  522. * Python standard library (`import xxx`)
  523. * Python standard library (`from xxx import`)
  524. * Third-party packages.
  525. * Other modules from the current package.
  526. or in case of code using Django:
  527. * Python standard library (`import xxx`)
  528. * Python standard library (`from xxx import`)
  529. * Third-party packages.
  530. * Django packages.
  531. * Other modules from the current package.
  532. Within these sections the imports should be sorted by module name.
  533. Example:
  534. .. code-block:: python
  535. import threading
  536. import time
  537. from collections import deque
  538. from Queue import Queue, Empty
  539. from .platforms import Pidfile
  540. from .five import zip_longest, items, range
  541. from .utils.time import maybe_timedelta
  542. * Wild-card imports must not be used (`from xxx import *`).
  543. * For distributions where Python 2.5 is the oldest support version
  544. additional rules apply:
  545. * Absolute imports must be enabled at the top of every module::
  546. from __future__ import absolute_import
  547. * If the module uses the :keyword:`with` statement and must be compatible
  548. with Python 2.5 (celery isn't) then it must also enable that::
  549. from __future__ import with_statement
  550. * Every future import must be on its own line, as older Python 2.5
  551. releases didn't support importing multiple features on the
  552. same future import line::
  553. # Good
  554. from __future__ import absolute_import
  555. from __future__ import with_statement
  556. # Bad
  557. from __future__ import absolute_import, with_statement
  558. (Note that this rule doesn't apply if the package doesn't include
  559. support for Python 2.5)
  560. * Note that we use "new-style" relative imports when the distribution
  561. doesn't support Python versions below 2.5
  562. This requires Python 2.5 or later:
  563. .. code-block:: python
  564. from . import submodule
  565. .. _feature-with-extras:
  566. Contributing features requiring additional libraries
  567. ====================================================
  568. Some features like a new result backend may require additional libraries
  569. that the user must install.
  570. We use setuptools `extra_requires` for this, and all new optional features
  571. that require third-party libraries must be added.
  572. 1) Add a new requirements file in `requirements/extras`
  573. For the Cassandra backend this is
  574. :file:`requirements/extras/cassandra.txt`, and the file looks like this:
  575. .. code-block:: text
  576. pycassa
  577. These are pip requirement files so you can have version specifiers and
  578. multiple packages are separated by newline. A more complex example could
  579. be:
  580. .. code-block:: text
  581. # pycassa 2.0 breaks Foo
  582. pycassa>=1.0,<2.0
  583. thrift
  584. 2) Modify ``setup.py``
  585. After the requirements file is added you need to add it as an option
  586. to :file:`setup.py` in the ``extras_require`` section::
  587. extra['extras_require'] = {
  588. # ...
  589. 'cassandra': extras('cassandra.txt'),
  590. }
  591. 3) Document the new feature in :file:`docs/includes/installation.txt`
  592. You must add your feature to the list in the :ref:`bundles` section
  593. of :file:`docs/includes/installation.txt`.
  594. After you've made changes to this file you need to render
  595. the distro :file:`README` file:
  596. .. code-block:: console
  597. $ pip install -U requirements/pkgutils.txt
  598. $ make readme
  599. That's all that needs to be done, but remember that if your feature
  600. adds additional configuration options then these needs to be documented
  601. in :file:`docs/configuration.rst`. Also all settings need to be added to the
  602. :file:`celery/app/defaults.py` module.
  603. Result backends require a separate section in the :file:`docs/configuration.rst`
  604. file.
  605. .. _contact_information:
  606. Contacts
  607. ========
  608. This is a list of people that can be contacted for questions
  609. regarding the official git repositories, PyPI packages
  610. Read the Docs pages.
  611. If the issue isn't an emergency then it's better
  612. to :ref:`report an issue <reporting-bugs>`.
  613. Committers
  614. ----------
  615. Ask Solem
  616. ~~~~~~~~~
  617. :github: https://github.com/ask
  618. :twitter: https://twitter.com/#!/asksol
  619. Asif Saif Uddin
  620. ~~~~~~~~~~~~~~~
  621. :github: https://github.com/auvipy
  622. :twitter: https://twitter.com/#!/auvipy
  623. Dmitry Malinovsky
  624. ~~~~~~~~~~~~~~~~~
  625. :github: https://github.com/malinoff
  626. :twitter: https://twitter.com/__malinoff__
  627. Ionel Cristian Mărieș
  628. ~~~~~~~~~~~~~~~~~~~~~
  629. :github: https://github.com/ionelmc
  630. :twitter: https://twitter.com/ionelmc
  631. Mher Movsisyan
  632. ~~~~~~~~~~~~~~
  633. :github: https://github.com/mher
  634. :twitter: https://twitter.com/#!/movsm
  635. Omer Katz
  636. ~~~~~~~~~
  637. :github: https://github.com/thedrow
  638. :twitter: https://twitter.com/the_drow
  639. Steeve Morin
  640. ~~~~~~~~~~~~
  641. :github: https://github.com/steeve
  642. :twitter: https://twitter.com/#!/steeve
  643. Website
  644. -------
  645. The Celery Project website is run and maintained by
  646. Mauro Rocco
  647. ~~~~~~~~~~~
  648. :github: https://github.com/fireantology
  649. :twitter: https://twitter.com/#!/fireantology
  650. with design by:
  651. Jan Henrik Helmers
  652. ~~~~~~~~~~~~~~~~~~
  653. :web: http://www.helmersworks.com
  654. :twitter: https://twitter.com/#!/helmers
  655. .. _packages:
  656. Packages
  657. ========
  658. ``celery``
  659. ----------
  660. :git: https://github.com/celery/celery
  661. :CI: https://travis-ci.org/#!/celery/celery
  662. :Windows-CI: https://ci.appveyor.com/project/ask/celery
  663. :PyPI: :pypi:`celery`
  664. :docs: http://docs.celeryproject.org
  665. ``kombu``
  666. ---------
  667. Messaging library.
  668. :git: https://github.com/celery/kombu
  669. :CI: https://travis-ci.org/#!/celery/kombu
  670. :Windows-CI: https://ci.appveyor.com/project/ask/kombu
  671. :PyPI: :pypi:`kombu`
  672. :docs: https://kombu.readthedocs.io
  673. ``amqp``
  674. --------
  675. Python AMQP 0.9.1 client.
  676. :git: https://github.com/celery/py-amqp
  677. :CI: https://travis-ci.org/#!/celery/py-amqp
  678. :Windows-CI: https://ci.appveyor.com/project/ask/py-amqp
  679. :PyPI: :pypi:`amqp`
  680. :docs: https://amqp.readthedocs.io
  681. ``vine``
  682. --------
  683. Promise/deferred implementation.
  684. :git: https://github.com/celery/vine/
  685. :CI: https://travis-ci.org/#!/celery/vine/
  686. :Windows-CI: https://ci.appveyor.com/project/ask/vine
  687. :PyPI: :pypi:`vine`
  688. :docs: https://vine.readthedocs.io
  689. ``billiard``
  690. ------------
  691. Fork of multiprocessing containing improvements
  692. that'll eventually be merged into the Python stdlib.
  693. :git: https://github.com/celery/billiard
  694. :CI: https://travis-ci.org/#!/celery/billiard/
  695. :Windows-CI: https://ci.appveyor.com/project/ask/billiard
  696. :PyPI: :pypi:`billiard`
  697. ``django-celery-beat``
  698. ----------------------
  699. Database-backed Periodic Tasks with admin interface using the Django ORM.
  700. :git: https://github.com/celery/django-celery-beat
  701. :CI: https://travis-ci.org/#!/celery/django-celery-beat
  702. :Windows-CI: https://ci.appveyor.com/project/ask/django-celery-beat
  703. :PyPI: :pypi:`django-celery-beat`
  704. ``django-celery-results``
  705. -------------------------
  706. Store task results in the Django ORM, or using the Django Cache Framework.
  707. :git: https://github.com/celery/django-celery-results
  708. :CI: https://travis-ci.org/#!/celery/django-celery-results
  709. :Windows-CI: https://ci.appveyor.com/project/ask/django-celery-results
  710. :PyPI: :pypi:`django-celery-results`
  711. ``librabbitmq``
  712. ---------------
  713. Very fast Python AMQP client written in C.
  714. :git: https://github.com/celery/librabbitmq
  715. :PyPI: :pypi:`librabbitmq`
  716. ``cell``
  717. --------
  718. Actor library.
  719. :git: https://github.com/celery/cell
  720. :PyPI: :pypi:`cell`
  721. ``cyme``
  722. --------
  723. Distributed Celery Instance manager.
  724. :git: https://github.com/celery/cyme
  725. :PyPI: :pypi:`cyme`
  726. :docs: https://cyme.readthedocs.io/
  727. Deprecated
  728. ----------
  729. - ``django-celery``
  730. :git: https://github.com/celery/django-celery
  731. :PyPI: :pypi:`django-celery`
  732. :docs: http://docs.celeryproject.org/en/latest/django
  733. - ``Flask-Celery``
  734. :git: https://github.com/ask/Flask-Celery
  735. :PyPI: :pypi:`Flask-Celery`
  736. - ``celerymon``
  737. :git: https://github.com/celery/celerymon
  738. :PyPI: :pypi:`celerymon`
  739. - ``carrot``
  740. :git: https://github.com/ask/carrot
  741. :PyPI: :pypi:`carrot`
  742. - ``ghettoq``
  743. :git: https://github.com/ask/ghettoq
  744. :PyPI: :pypi:`ghettoq`
  745. - ``kombu-sqlalchemy``
  746. :git: https://github.com/ask/kombu-sqlalchemy
  747. :PyPI: :pypi:`kombu-sqlalchemy`
  748. - ``django-kombu``
  749. :git: https://github.com/ask/django-kombu
  750. :PyPI: :pypi:`django-kombu`
  751. - ``pylibrabbitmq``
  752. Old name for :pypi:`librabbitmq`.
  753. :git: :const:`None`
  754. :PyPI: :pypi:`pylibrabbitmq`
  755. .. _release-procedure:
  756. Release Procedure
  757. =================
  758. Updating the version number
  759. ---------------------------
  760. The version number must be updated two places:
  761. * :file:`celery/__init__.py`
  762. * :file:`docs/include/introduction.txt`
  763. After you have changed these files you must render
  764. the :file:`README` files. There's a script to convert sphinx syntax
  765. to generic reStructured Text syntax, and the make target `readme`
  766. does this for you:
  767. .. code-block:: console
  768. $ make readme
  769. Now commit the changes:
  770. .. code-block:: console
  771. $ git commit -a -m "Bumps version to X.Y.Z"
  772. and make a new version tag:
  773. .. code-block:: console
  774. $ git tag vX.Y.Z
  775. $ git push --tags
  776. Releasing
  777. ---------
  778. Commands to make a new public stable release:
  779. .. code-block:: console
  780. $ make distcheck # checks pep8, autodoc index, runs tests and more
  781. $ make dist # NOTE: Runs git clean -xdf and removes files not in the repo.
  782. $ python setup.py sdist upload --sign --identity='Celery Security Team'
  783. $ python setup.py bdist_wheel upload --sign --identity='Celery Security Team'
  784. If this is a new release series then you also need to do the
  785. following:
  786. * Go to the Read The Docs management interface at:
  787. https://readthedocs.org/projects/celery/?fromdocs=celery
  788. * Enter "Edit project"
  789. Change default branch to the branch of this series, for example, use
  790. the ``2.4`` branch for the 2.4 series.
  791. * Also add the previous version under the "versions" tab.
  792. .. _`mailing-list`: https://groups.google.com/group/celery-users
  793. .. _`irc-channel`: http://docs.celeryproject.org/en/latest/getting-started/resources.html#irc
  794. .. _`internals-guide`: http://docs.celeryproject.org/en/latest/internals/guide.html
  795. .. _`bundles`: http://docs.celeryproject.org/en/latest/getting-started/introduction.html#bundles
  796. .. _`report an issue`: http://docs.celeryproject.org/en/latest/contributing.html#reporting-bugs