whatsnew-4.0.rst 75 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211221231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631641651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042052062072082092102112122132142152162172182192202212222232242252262272282292302312322332342352362372382392402412422432442452462472482492502512522532542552562572582592602612622632642652662672682692702712722732742752762772782792802812822832842852862872882892902912922932942952962972982993003013023033043053063073083093103113123133143153163173183193203213223233243253263273283293303313323333343353363373383393403413423433443453463473483493503513523533543553563573583593603613623633643653663673683693703713723733743753763773783793803813823833843853863873883893903913923933943953963973983994004014024034044054064074084094104114124134144154164174184194204214224234244254264274284294304314324334344354364374384394404414424434444454464474484494504514524534544554564574584594604614624634644654664674684694704714724734744754764774784794804814824834844854864874884894904914924934944954964974984995005015025035045055065075085095105115125135145155165175185195205215225235245255265275285295305315325335345355365375385395405415425435445455465475485495505515525535545555565575585595605615625635645655665675685695705715725735745755765775785795805815825835845855865875885895905915925935945955965975985996006016026036046056066076086096106116126136146156166176186196206216226236246256266276286296306316326336346356366376386396406416426436446456466476486496506516526536546556566576586596606616626636646656666676686696706716726736746756766776786796806816826836846856866876886896906916926936946956966976986997007017027037047057067077087097107117127137147157167177187197207217227237247257267277287297307317327337347357367377387397407417427437447457467477487497507517527537547557567577587597607617627637647657667677687697707717727737747757767777787797807817827837847857867877887897907917927937947957967977987998008018028038048058068078088098108118128138148158168178188198208218228238248258268278288298308318328338348358368378388398408418428438448458468478488498508518528538548558568578588598608618628638648658668678688698708718728738748758768778788798808818828838848858868878888898908918928938948958968978988999009019029039049059069079089099109119129139149159169179189199209219229239249259269279289299309319329339349359369379389399409419429439449459469479489499509519529539549559569579589599609619629639649659669679689699709719729739749759769779789799809819829839849859869879889899909919929939949959969979989991000100110021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060106110621063106410651066106710681069107010711072107310741075107610771078107910801081108210831084108510861087108810891090109110921093109410951096109710981099110011011102110311041105110611071108110911101111111211131114111511161117111811191120112111221123112411251126112711281129113011311132113311341135113611371138113911401141114211431144114511461147114811491150115111521153115411551156115711581159116011611162116311641165116611671168116911701171117211731174117511761177117811791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237123812391240124112421243124412451246124712481249125012511252125312541255125612571258125912601261126212631264126512661267126812691270127112721273127412751276127712781279128012811282128312841285128612871288128912901291129212931294129512961297129812991300130113021303130413051306130713081309131013111312131313141315131613171318131913201321132213231324132513261327132813291330133113321333133413351336133713381339134013411342134313441345134613471348134913501351135213531354135513561357135813591360136113621363136413651366136713681369137013711372137313741375137613771378137913801381138213831384138513861387138813891390139113921393139413951396139713981399140014011402140314041405140614071408140914101411141214131414141514161417141814191420142114221423142414251426142714281429143014311432143314341435143614371438143914401441144214431444144514461447144814491450145114521453145414551456145714581459146014611462146314641465146614671468146914701471147214731474147514761477147814791480148114821483148414851486148714881489149014911492149314941495149614971498149915001501150215031504150515061507150815091510151115121513151415151516151715181519152015211522152315241525152615271528152915301531153215331534153515361537153815391540154115421543154415451546154715481549155015511552155315541555155615571558155915601561156215631564156515661567156815691570157115721573157415751576157715781579158015811582158315841585158615871588158915901591159215931594159515961597159815991600160116021603160416051606160716081609161016111612161316141615161616171618161916201621162216231624162516261627162816291630163116321633163416351636163716381639164016411642164316441645164616471648164916501651165216531654165516561657165816591660166116621663166416651666166716681669167016711672167316741675167616771678167916801681168216831684168516861687168816891690169116921693169416951696169716981699170017011702170317041705170617071708170917101711171217131714171517161717171817191720172117221723172417251726172717281729173017311732173317341735173617371738173917401741174217431744174517461747174817491750175117521753175417551756175717581759176017611762176317641765176617671768176917701771177217731774177517761777177817791780178117821783178417851786178717881789179017911792179317941795179617971798179918001801180218031804180518061807180818091810181118121813181418151816181718181819182018211822182318241825182618271828182918301831183218331834183518361837183818391840184118421843184418451846184718481849185018511852185318541855185618571858185918601861186218631864186518661867186818691870187118721873187418751876187718781879188018811882188318841885188618871888188918901891189218931894189518961897189818991900190119021903190419051906190719081909191019111912191319141915191619171918191919201921192219231924192519261927192819291930193119321933193419351936193719381939194019411942194319441945194619471948194919501951195219531954195519561957195819591960196119621963196419651966196719681969197019711972197319741975197619771978197919801981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201220132014201520162017201820192020202120222023202420252026202720282029203020312032203320342035203620372038203920402041204220432044204520462047204820492050205120522053205420552056205720582059206020612062206320642065206620672068206920702071207220732074207520762077207820792080208120822083208420852086208720882089209020912092209320942095209620972098209921002101210221032104210521062107210821092110211121122113211421152116211721182119212021212122212321242125212621272128212921302131213221332134213521362137213821392140214121422143214421452146214721482149215021512152215321542155215621572158215921602161216221632164216521662167216821692170217121722173217421752176217721782179218021812182218321842185218621872188218921902191219221932194219521962197219821992200220122022203220422052206220722082209221022112212221322142215221622172218221922202221222222232224222522262227222822292230223122322233223422352236223722382239224022412242224322442245224622472248224922502251225222532254225522562257225822592260226122622263226422652266226722682269227022712272227322742275227622772278227922802281228222832284228522862287228822892290229122922293229422952296229722982299230023012302230323042305230623072308230923102311231223132314231523162317231823192320232123222323232423252326232723282329233023312332233323342335233623372338233923402341234223432344234523462347234823492350
  1. .. _whatsnew-4.0:
  2. ===========================================
  3. What's new in Celery 4.0 (latentcall)
  4. ===========================================
  5. :Author: Ask Solem (``ask at celeryproject.org``)
  6. .. sidebar:: Change history
  7. What's new documents describe the changes in major versions,
  8. we also have a :ref:`changelog` that lists the changes in bugfix
  9. releases (0.0.x), while older series are archived under the :ref:`history`
  10. section.
  11. Celery is a simple, flexible, and reliable distributed system to
  12. process vast amounts of messages, while providing operations with
  13. the tools required to maintain such a system.
  14. It's a task queue with focus on real-time processing, while also
  15. supporting task scheduling.
  16. Celery has a large and diverse community of users and contributors,
  17. you should come join us :ref:`on IRC <irc-channel>`
  18. or :ref:`our mailing-list <mailing-list>`.
  19. To read more about Celery you should go read the :ref:`introduction <intro>`.
  20. While this version is backward compatible with previous versions
  21. it's important that you read the following section.
  22. This version is officially supported on CPython 2.7, 3.4, and 3.5.
  23. and also supported on PyPy.
  24. .. _`website`: http://celeryproject.org/
  25. .. topic:: Table of Contents
  26. Make sure you read the important notes before upgrading to this version.
  27. .. contents::
  28. :local:
  29. :depth: 3
  30. Preface
  31. =======
  32. Welcome to Celery 4!
  33. This is a massive release with over two years of changes.
  34. Not only does it come with many new features, but it also fixes
  35. a massive list of bugs, so in many ways you could call it
  36. our "Snow Leopard" release.
  37. The next major version of Celery will support Python 3.5 only, where
  38. we are planning to take advantage of the new asyncio library.
  39. This release would not have been possible without the support
  40. of my employer, `Robinhood`_ (we're hiring!).
  41. - Ask Solem
  42. Dedicated to Sebastian "Zeb" Bjørnerud (RIP),
  43. with special thanks to `Ty Wilkins`_, for designing our new logo,
  44. all the contributors who help make this happen, and my colleagues
  45. at `Robinhood`_.
  46. .. _`Ty Wilkins`: http://tywilkins.com
  47. .. _`Robinhood`: https://robinhood.com
  48. Wall of Contributors
  49. --------------------
  50. Aaron McMillin, Adam Chainz, Adam Renberg, Adriano Martins de Jesus,
  51. Adrien Guinet, Ahmet Demir, Aitor Gómez-Goiri, Alan Justino,
  52. Albert Wang, Alex Koshelev, Alex Rattray, Alex Williams, Alexander Koshelev,
  53. Alexander Lebedev, Alexander Oblovatniy, Alexey Kotlyarov, Ali Bozorgkhan,
  54. Alice Zoë Bevan–McGregor, Allard Hoeve, Alman One, Amir Rustamzadeh,
  55. Andrea Rabbaglietti, Andrea Rosa, Andrei Fokau, Andrew Rodionoff,
  56. Andrew Stewart, Andriy Yurchuk, Aneil Mallavarapu, Areski Belaid,
  57. Armenak Baburyan, Arthur Vuillard, Artyom Koval, Asif Saifuddin Auvi,
  58. Ask Solem, Balthazar Rouberol, Batiste Bieler, Berker Peksag,
  59. Bert Vanderbauwhede, Brendan Smithyman, Brian Bouterse, Bryce Groff,
  60. Cameron Will, ChangBo Guo, Chris Clark, Chris Duryee, Chris Erway,
  61. Chris Harris, Chris Martin, Chillar Anand, Colin McIntosh, Conrad Kramer,
  62. Corey Farwell, Craig Jellick, Cullen Rhodes, Dallas Marlow, Daniel Devine,
  63. Daniel Wallace, Danilo Bargen, Davanum Srinivas, Dave Smith, David Baumgold,
  64. David Harrigan, David Pravec, Dennis Brakhane, Derek Anderson,
  65. Dmitry Dygalo, Dmitry Malinovsky, Dongweiming, Dudás Ádám,
  66. Dustin J. Mitchell, Ed Morley, Edward Betts, Éloi Rivard, Emmanuel Cazenave,
  67. Fahad Siddiqui, Fatih Sucu, Feanil Patel, Federico Ficarelli, Felix Schwarz,
  68. Felix Yan, Fernando Rocha, Flavio Grossi, Frantisek Holop, Gao Jiangmiao,
  69. George Whewell, Gerald Manipon, Gilles Dartiguelongue, Gino Ledesma, Greg Wilbur,
  70. Guillaume Seguin, Hank John, Hogni Gylfason, Ilya Georgievsky,
  71. Ionel Cristian Mărieș, Ivan Larin, James Pulec, Jared Lewis, Jason Veatch,
  72. Jasper Bryant-Greene, Jeff Widman, Jeremy Tillman, Jeremy Zafran,
  73. Jocelyn Delalande, Joe Jevnik, Joe Sanford, John Anderson, John Barham,
  74. John Kirkham, John Whitlock, Jonathan Vanasco, Joshua Harlow, João Ricardo,
  75. Juan Carlos Ferrer, Juan Rossi, Justin Patrin, Kai Groner, Kevin Harvey,
  76. Kevin Richardson, Komu Wairagu, Konstantinos Koukopoulos, Kouhei Maeda,
  77. Kracekumar Ramaraju, Krzysztof Bujniewicz, Latitia M. Haskins, Len Buckens,
  78. Lev Berman, lidongming, Lorenzo Mancini, Lucas Wiman, Luke Pomfrey,
  79. Luyun Xie, Maciej Obuchowski, Manuel Kaufmann, Marat Sharafutdinov,
  80. Marc Sibson, Marcio Ribeiro, Marin Atanasov Nikolov, Mathieu Fenniak,
  81. Mark Parncutt, Mauro Rocco, Maxime Beauchemin, Maxime Vdb, Mher Movsisyan,
  82. Michael Aquilina, Michael Duane Mooring, Michael Permana, Mickaël Penhard,
  83. Mike Attwood, Mitchel Humpherys, Mohamed Abouelsaoud, Morris Tweed, Morton Fox,
  84. Môshe van der Sterre, Nat Williams, Nathan Van Gheem, Nicolas Unravel,
  85. Nik Nyby, Omer Katz, Omer Korner, Ori Hoch, Paul Pearce, Paulo Bu,
  86. Pavlo Kapyshin, Philip Garnero, Pierre Fersing, Piotr Kilczuk,
  87. Piotr Maślanka, Quentin Pradet, Radek Czajka, Raghuram Srinivasan,
  88. Randy Barlow, Raphael Michel, Rémy Léone, Robert Coup, Robert Kolba,
  89. Rockallite Wulf, Rodolfo Carvalho, Roger Hu, Romuald Brunet, Rongze Zhu,
  90. Ross Deane, Ryan Luckie, Rémy Greinhofer, Samuel Giffard, Samuel Jaillet,
  91. Sergey Azovskov, Sergey Tikhonov, Seungha Kim, Simon Peeters,
  92. Spencer E. Olson, Srinivas Garlapati, Stephen Milner, Steve Peak, Steven Sklar,
  93. Stuart Axon, Sukrit Khera, Tadej Janež, Taha Jahangir, Takeshi Kanemoto,
  94. Tayfun Sen, Tewfik Sadaoui, Thomas French, Thomas Grainger, Tomas Machalek,
  95. Tobias Schottdorf, Tocho Tochev, Valentyn Klindukh, Vic Kumar,
  96. Vladimir Bolshakov, Vladimir Gorbunov, Wayne Chang, Wieland Hoffmann,
  97. Wido den Hollander, Wil Langford, Will Thompson, William King, Yury Selivanov,
  98. Vytis Banaitis, Zoran Pavlovic, Xin Li, 許邱翔, :github_user:`allenling`,
  99. :github_user:`alzeih`, :github_user:`bastb`, :github_user:`bee-keeper`,
  100. :github_user:`ffeast`, :github_user:`firefly4268`,
  101. :github_user:`flyingfoxlee`, :github_user:`gdw2`, :github_user:`gitaarik`,
  102. :github_user:`hankjin`, :github_user:`lvh`, :github_user:`m-vdb`,
  103. :github_user:`kindule`, :github_user:`mdk`:, :github_user:`michael-k`,
  104. :github_user:`mozillazg`, :github_user:`nokrik`, :github_user:`ocean1`,
  105. :github_user:`orlo666`, :github_user:`raducc`, :github_user:`wanglei`,
  106. :github_user:`worldexception`, :github_user:`xBeAsTx`.
  107. .. note::
  108. This wall was automatically generated from git history,
  109. so sadly it doesn't not include the people who help with more important
  110. things like answering mailing-list questions.
  111. Upgrading from Celery 3.1
  112. =========================
  113. Step 1: Upgrade to Celery 3.1.25
  114. --------------------------------
  115. If you haven't already, the first step is to upgrade to Celery 3.1.25.
  116. This version adds forward compatibility to the new message protocol,
  117. so that you can incrementally upgrade from 3.1 to 4.0.
  118. Deploy the workers first by upgrading to 3.1.25, this means these
  119. workers can process messages sent by clients using both 3.1 and 4.0.
  120. After the workers are upgraded you can upgrade the clients (e.g. web servers).
  121. Step 2: Update your configuration with the new setting names
  122. ------------------------------------------------------------
  123. This version radically changes the configuration setting names,
  124. to be more consistent.
  125. The changes are fully backwards compatible, so you have the option to wait
  126. until the old setting names are deprecated, but to ease the transition
  127. we have included a command-line utility that rewrites your settings
  128. automatically.
  129. See :ref:`v400-upgrade-settings` for more information.
  130. Step 3: Read the important notes in this document
  131. -------------------------------------------------
  132. Make sure you are not affected by any of the important upgrade notes
  133. mentioned in the following section.
  134. An especially important note is that Celery now checks the arguments
  135. you send to a task by matching it to the signature (:ref:`v400-typing`).
  136. Step 4: Upgrade to Celery 4.0
  137. -----------------------------
  138. At this point you can upgrade your workers and clients with the new version.
  139. .. _v400-important:
  140. Important Notes
  141. ===============
  142. Dropped support for Python 2.6
  143. ------------------------------
  144. Celery now requires Python 2.7 or later,
  145. and also drops support for Python 3.3 so supported versions are:
  146. - CPython 2.7
  147. - CPython 3.4
  148. - CPython 3.5
  149. - PyPy 5.4 (``pypy2``)
  150. - PyPy 5.5-alpha (``pypy3``)
  151. Last major version to support Python 2
  152. --------------------------------------
  153. Starting from Celery 5.0 only Python 3.5+ will be supported.
  154. To make sure you're not affected by this change you should pin
  155. the Celery version in your requirements file, either to a specific
  156. version: ``celery==4.0.0``, or a range: ``celery>=4.0,<5.0``.
  157. Dropping support for Python 2 will enable us to remove massive
  158. amounts of compatibility code, and going with Python 3.5 allows
  159. us to take advantage of typing, async/await, asyncio, and similar
  160. concepts there's no alternative for in older versions.
  161. Celery 4.x will continue to work on Python 2.7, 3.4, 3.5; just as Celery 3.x
  162. still works on Python 2.6.
  163. Django support
  164. --------------
  165. Celery 4.x requires Django 1.8 or later, but we really recommend
  166. using at least Django 1.9 for the new ``transaction.on_commit`` feature.
  167. A common problem when calling tasks from Django is when the task is related
  168. to a model change, and you wish to cancel the task if the transaction is
  169. rolled back, or ensure the task is only executed after the changes have been
  170. written to the database.
  171. ``transaction.atomic`` enables you to solve this problem by adding
  172. the task as a callback to be called only when the transaction is committed.
  173. Example usage:
  174. .. code-block:: python
  175. from functools import partial
  176. from django.db import transaction
  177. from .models import Article, Log
  178. from .tasks import send_article_created_notification
  179. def create_article(request):
  180. with transaction.atomic():
  181. article = Article.objects.create(**request.POST)
  182. # send this task only if the rest of the transaction succeeds.
  183. transaction.on_commit(partial(
  184. send_article_created_notification.delay, article_id=article.pk))
  185. Log.objects.create(type=Log.ARTICLE_CREATED, object_pk=article.pk)
  186. Removed features
  187. ----------------
  188. - Microsoft Windows is no longer supported.
  189. The test suite is passing, and Celery seems to be working with Windows,
  190. but we make no guarantees as we are unable to diagnose issues on this
  191. platform. If you are a company requiring support on this platform,
  192. please get in touch.
  193. - Jython is no longer supported.
  194. Features removed for simplicity
  195. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  196. - Webhook task machinery (``celery.task.http``) has been removed.
  197. Nowadays it's easy to use the :pypi:`requests` module to write
  198. webhook tasks manually. We would love to use requests but we
  199. are simply unable to as there's a very vocal 'anti-dependency'
  200. mob in the Python community
  201. If you need backwards compatibility
  202. you can simply copy + paste the 3.1 version of the module and make sure
  203. it's imported by the worker:
  204. https://github.com/celery/celery/blob/3.1/celery/task/http.py
  205. - Tasks no longer sends error emails.
  206. This also removes support for ``app.mail_admins``, and any functionality
  207. related to sending emails.
  208. - ``celery.contrib.batches`` has been removed.
  209. This was an experimental feature, so not covered by our deprecation
  210. timeline guarantee.
  211. You can copy and pase the existing batches code for use within your projects:
  212. https://github.com/celery/celery/blob/3.1/celery/contrib/batches.py
  213. Features removed for lack of funding
  214. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  215. We announced with the 3.1 release that some transports were
  216. moved to experimental status, and that there'd be no official
  217. support for the transports.
  218. As this subtle hint for the need of funding failed
  219. we've removed them completely, breaking backwards compatibility.
  220. - Using the Django ORM as a broker is no longer supported.
  221. You can still use the Django ORM as a result backend:
  222. see :ref:`django-celery-results` section for more information.
  223. - Using SQLAlchemy as a broker is no longer supported.
  224. You can still use SQLAlchemy as a result backend.
  225. - Using CouchDB as a broker is no longer supported.
  226. You can still use CouchDB as a result backend.
  227. - Using IronMQ as a broker is no longer supported.
  228. - Using Beanstalk as a broker is no longer supported.
  229. In addition some features have been removed completely so that
  230. attempting to use them will raise an exception:
  231. - The ``--autoreload`` feature has been removed.
  232. This was an experimental feature, and not covered by our deprecation
  233. timeline guarantee. The flag is removed completely so the worker
  234. will crash at startup when present. Luckily this
  235. flag isn't used in production systems.
  236. - The experimental ``threads`` pool is no longer supported and has been removed.
  237. - The ``force_execv`` feature is no longer supported.
  238. The ``celery worker`` command now ignores the ``--no-execv``,
  239. ``--force-execv``, and the ``CELERYD_FORCE_EXECV`` setting.
  240. This flag will be removed completely in 5.0 and the worker
  241. will raise an error.
  242. - The old legacy "amqp" result backend has been deprecated, and will
  243. be removed in Celery 5.0.
  244. Please use the ``rpc`` result backend for RPC-style calls, and a
  245. persistent result backend for multi-consumer results.
  246. We think most of these can be fixed without considerable effort, so if you're
  247. interested in getting any of these features back, please get in touch.
  248. **Now to the good news**...
  249. New Task Message Protocol
  250. -------------------------
  251. .. :sha:`e71652d384b1b5df2a4e6145df9f0efb456bc71c`
  252. This version introduces a brand new task message protocol,
  253. the first major change to the protocol since the beginning of the project.
  254. The new protocol is enabled by default in this version and since the new
  255. version isn't backwards compatible you have to be careful when upgrading.
  256. The 3.1.25 version was released to add compatibility with the new protocol
  257. so the easiest way to upgrade is to upgrade to that version first, then
  258. upgrade to 4.0 in a second deployment.
  259. If you wish to keep using the old protocol you may also configure
  260. the protocol version number used:
  261. .. code-block:: python
  262. app = Celery()
  263. app.conf.task_protocol = 1
  264. Read more about the features available in the new protocol in the news
  265. section found later in this document.
  266. .. _v400-upgrade-settings:
  267. Lowercase setting names
  268. -----------------------
  269. In the pursuit of beauty all settings are now renamed to be in all
  270. lowercase and some setting names have been renamed for consistency.
  271. This change is fully backwards compatible so you can still use the uppercase
  272. setting names, but we would like you to upgrade as soon as possible and
  273. you can do this automatically using the :program:`celery upgrade settings`
  274. command:
  275. .. code-block:: console
  276. $ celery upgrade settings proj/settings.py
  277. This command will modify your module in-place to use the new lower-case
  278. names (if you want uppercase with a "``CELERY``" prefix see block below),
  279. and save a backup in :file:`proj/settings.py.orig`.
  280. .. _latentcall-django-admonition:
  281. .. admonition:: For Django users and others who want to keep uppercase names
  282. If you're loading Celery configuration from the Django settings module
  283. then you'll want to keep using the uppercase names.
  284. You also want to use a ``CELERY_`` prefix so that no Celery settings
  285. collide with Django settings used by other apps.
  286. To do this, you'll first need to convert your settings file
  287. to use the new consistent naming scheme, and add the prefix to all
  288. Celery related settings:
  289. .. code-block:: console
  290. $ celery upgrade settings proj/settings.py --django
  291. After upgrading the settings file, you need to set the prefix explicitly
  292. in your ``proj/celery.py`` module:
  293. .. code-block:: python
  294. app.config_from_object('django.conf:settings', namespace='CELERY')
  295. You can find the most up to date Django Celery integration example
  296. here: :ref:`django-first-steps`.
  297. .. note::
  298. This will also add a prefix to settings that didn't previously
  299. have one, for example ``BROKER_URL`` should be written
  300. ``CELERY_BROKER_URL`` with a namespace of ``CELERY``
  301. ``CELERY_BROKER_URL``.
  302. Luckily you don't have to manually change the files, as
  303. the :program:`celery upgrade settings --django` program should do the
  304. right thing.
  305. The loader will try to detect if your configuration is using the new format,
  306. and act accordingly, but this also means you're not allowed to mix and
  307. match new and old setting names, that's unless you provide a value for both
  308. alternatives.
  309. The major difference between previous versions, apart from the lower case
  310. names, are the renaming of some prefixes, like ``celerybeat_`` to ``beat_``,
  311. ``celeryd_`` to ``worker_``.
  312. The ``celery_`` prefix has also been removed, and task related settings
  313. from this name-space is now prefixed by ``task_``, worker related settings
  314. with ``worker_``.
  315. Apart from this most of the settings will be the same in lowercase, apart from
  316. a few special ones:
  317. ===================================== ==========================================================
  318. **Setting name** **Replace with**
  319. ===================================== ==========================================================
  320. ``CELERY_MAX_CACHED_RESULTS`` :setting:`result_cache_max`
  321. ``CELERY_MESSAGE_COMPRESSION`` :setting:`result_compression`/:setting:`task_compression`.
  322. ``CELERY_TASK_RESULT_EXPIRES`` :setting:`result_expires`
  323. ``CELERY_RESULT_DBURI`` :setting:`result_backend`
  324. ``CELERY_RESULT_ENGINE_OPTIONS`` :setting:`database_engine_options`
  325. ``-*-_DB_SHORT_LIVED_SESSIONS`` :setting:`database_short_lived_sessions`
  326. ``CELERY_RESULT_DB_TABLE_NAMES`` :setting:`database_db_names`
  327. ``CELERY_ACKS_LATE`` :setting:`task_acks_late`
  328. ``CELERY_ALWAYS_EAGER`` :setting:`task_always_eager`
  329. ``CELERY_ANNOTATIONS`` :setting:`task_annotations`
  330. ``CELERY_MESSAGE_COMPRESSION`` :setting:`task_compression`
  331. ``CELERY_CREATE_MISSING_QUEUES`` :setting:`task_create_missing_queues`
  332. ``CELERY_DEFAULT_DELIVERY_MODE`` :setting:`task_default_delivery_mode`
  333. ``CELERY_DEFAULT_EXCHANGE`` :setting:`task_default_exchange`
  334. ``CELERY_DEFAULT_EXCHANGE_TYPE`` :setting:`task_default_exchange_type`
  335. ``CELERY_DEFAULT_QUEUE`` :setting:`task_default_queue`
  336. ``CELERY_DEFAULT_RATE_LIMIT`` :setting:`task_default_rate_limit`
  337. ``CELERY_DEFAULT_ROUTING_KEY`` :setting:`task_default_routing_key`
  338. ``-"-_EAGER_PROPAGATES_EXCEPTIONS`` :setting:`task_eager_propagates`
  339. ``CELERY_IGNORE_RESULT`` :setting:`task_ignore_result`
  340. ``CELERY_TASK_PUBLISH_RETRY`` :setting:`task_publish_retry`
  341. ``CELERY_TASK_PUBLISH_RETRY_POLICY`` :setting:`task_publish_retry_policy`
  342. ``CELERY_QUEUES`` :setting:`task_queues`
  343. ``CELERY_ROUTES`` :setting:`task_routes`
  344. ``CELERY_SEND_TASK_SENT_EVENT`` :setting:`task_send_sent_event`
  345. ``CELERY_TASK_SERIALIZER`` :setting:`task_serializer`
  346. ``CELERYD_TASK_SOFT_TIME_LIMIT`` :setting:`task_soft_time_limit`
  347. ``CELERYD_TASK_TIME_LIMIT`` :setting:`task_time_limit`
  348. ``CELERY_TRACK_STARTED`` :setting:`task_track_started`
  349. ``CELERY_DISABLE_RATE_LIMITS`` :setting:`worker_disable_rate_limits`
  350. ``CELERY_ENABLE_REMOTE_CONTROL`` :setting:`worker_enable_remote_control`
  351. ``CELERYD_SEND_EVENTS`` :setting:`worker_send_task_events`
  352. ===================================== ==========================================================
  353. You can see a full table of the changes in :ref:`conf-old-settings-map`.
  354. Json is now the default serializer
  355. ----------------------------------
  356. The time has finally come to end the reign of :mod:`pickle` as the default
  357. serialization mechanism, and json is the default serializer starting from this
  358. version.
  359. This change was :ref:`announced with the release of Celery 3.1
  360. <last-version-to-enable-pickle>`.
  361. If you're still depending on :mod:`pickle` being the default serializer,
  362. then you have to configure your app before upgrading to 4.0:
  363. .. code-block:: python
  364. task_serializer = 'pickle'
  365. result_serializer = 'pickle'
  366. accept_content = {'pickle'}
  367. The Json serializer now also supports some additional types:
  368. - :class:`~datetime.datetime`, :class:`~datetime.time`, :class:`~datetime.date`
  369. Converted to json text, in ISO-8601 format.
  370. - :class:`~decimal.Decimal`
  371. Converted to json text.
  372. - :class:`django.utils.functional.Promise`
  373. Django only: Lazy strings used for translation etc., are evaluated
  374. and conversion to a json type is attempted.
  375. - :class:`uuid.UUID`
  376. Converted to json text.
  377. You can also define a ``__json__`` method on your custom classes to support
  378. JSON serialization (must return a json compatible type):
  379. .. code-block:: python
  380. class Person:
  381. first_name = None
  382. last_name = None
  383. address = None
  384. def __json__(self):
  385. return {
  386. 'first_name': self.first_name,
  387. 'last_name': self.last_name,
  388. 'address': self.address,
  389. }
  390. The Task base class no longer automatically register tasks
  391. ----------------------------------------------------------
  392. The :class:`~@Task` class is no longer using a special meta-class
  393. that automatically registers the task in the task registry.
  394. Instead this is now handled by the :class:`@task` decorators.
  395. If you're still using class based tasks, then you need to register
  396. these manually:
  397. .. code-block:: python
  398. class CustomTask(Task):
  399. def run(self):
  400. print('running')
  401. CustomTask = app.register_task(CustomTask())
  402. The best practice is to use custom task classes only for overriding
  403. general behavior, and then using the task decorator to realize the task:
  404. .. code-block:: python
  405. @app.task(bind=True, base=CustomTask)
  406. def custom(self):
  407. print('running')
  408. This change also means that the ``abstract`` attribute of the task
  409. no longer has any effect.
  410. .. _v400-typing:
  411. Task argument checking
  412. ----------------------
  413. The arguments of the task are now verified when calling the task,
  414. even asynchronously:
  415. .. code-block:: pycon
  416. >>> @app.task
  417. ... def add(x, y):
  418. ... return x + y
  419. >>> add.delay(8, 8)
  420. <AsyncResult: f59d71ca-1549-43e0-be41-4e8821a83c0c>
  421. >>> add.delay(8)
  422. Traceback (most recent call last):
  423. File "<stdin>", line 1, in <module>
  424. File "celery/app/task.py", line 376, in delay
  425. return self.apply_async(args, kwargs)
  426. File "celery/app/task.py", line 485, in apply_async
  427. check_arguments(*(args or ()), **(kwargs or {}))
  428. TypeError: add() takes exactly 2 arguments (1 given)
  429. You can disable the argument checking for any task by setting its
  430. :attr:`~@Task.typing` attribute to :const:`False`:
  431. .. code-block:: pycon
  432. >>> @app.task(typing=False)
  433. ... def add(x, y):
  434. ... return x + y
  435. Or if you would like to disable this completely for all tasks
  436. you can pass ``strict_typing=False`` when creating the app:
  437. .. code-block:: python
  438. app = Celery(..., strict_typing=False)
  439. Redis Events not backward compatible
  440. ------------------------------------
  441. The Redis ``fanout_patterns`` and ``fanout_prefix`` transport
  442. options are now enabled by default.
  443. Workers/monitors without these flags enabled won't be able to
  444. see workers with this flag disabled. They can still execute tasks,
  445. but they cannot receive each others monitoring messages.
  446. You can upgrade in a backward compatible manner by first configuring
  447. your 3.1 workers and monitors to enable the settings, before the final
  448. upgrade to 4.0:
  449. .. code-block:: python
  450. BROKER_TRANSPORT_OPTIONS = {
  451. 'fanout_patterns': True,
  452. 'fanout_prefix': True,
  453. }
  454. Redis Priorities Reversed
  455. -------------------------
  456. Priority 0 is now lowest, 9 is highest.
  457. This change was made to make priority support consistent with how
  458. it works in AMQP.
  459. Contributed by **Alex Koshelev**.
  460. Django: Auto-discover now supports Django app configurations
  461. ------------------------------------------------------------
  462. The ``autodiscover_tasks()`` function can now be called without arguments,
  463. and the Django handler will automatically find your installed apps:
  464. .. code-block:: python
  465. app.autodiscover_tasks()
  466. The Django integration :ref:`example in the documentation
  467. <django-first-steps>` has been updated to use the argument-less call.
  468. This also ensures compatibility with the new, ehm, ``AppConfig`` stuff
  469. introduced in recent Django versions.
  470. Worker direct queues no longer use auto-delete
  471. ----------------------------------------------
  472. Workers/clients running 4.0 will no longer be able to send
  473. worker direct messages to workers running older versions, and vice versa.
  474. If you're relying on worker direct messages you should upgrade
  475. your 3.x workers and clients to use the new routing settings first,
  476. by replacing :func:`celery.utils.worker_direct` with this implementation:
  477. .. code-block:: python
  478. from kombu import Exchange, Queue
  479. worker_direct_exchange = Exchange('C.dq2')
  480. def worker_direct(hostname):
  481. return Queue(
  482. '{hostname}.dq2'.format(hostname),
  483. exchange=worker_direct_exchange,
  484. routing_key=hostname,
  485. )
  486. This feature closed Issue #2492.
  487. Old command-line programs removed
  488. ---------------------------------
  489. Installing Celery will no longer install the ``celeryd``,
  490. ``celerybeat`` and ``celeryd-multi`` programs.
  491. This was announced with the release of Celery 3.1, but you may still
  492. have scripts pointing to the old names, so make sure you update these
  493. to use the new umbrella command:
  494. +-------------------+--------------+-------------------------------------+
  495. | Program | New Status | Replacement |
  496. +===================+==============+=====================================+
  497. | ``celeryd`` | **REMOVED** | :program:`celery worker` |
  498. +-------------------+--------------+-------------------------------------+
  499. | ``celerybeat`` | **REMOVED** | :program:`celery beat` |
  500. +-------------------+--------------+-------------------------------------+
  501. | ``celeryd-multi`` | **REMOVED** | :program:`celery multi` |
  502. +-------------------+--------------+-------------------------------------+
  503. .. _v400-news:
  504. News
  505. ====
  506. New protocol highlights
  507. -----------------------
  508. The new protocol fixes many problems with the old one, and enables
  509. some long-requested features:
  510. - Most of the data are now sent as message headers, instead of being
  511. serialized with the message body.
  512. In version 1 of the protocol the worker always had to deserialize
  513. the message to be able to read task meta-data like the task id,
  514. name, etc. This also meant that the worker was forced to double-decode
  515. the data, first deserializing the message on receipt, serializing
  516. the message again to send to child process, then finally the child process
  517. deserializes the message again.
  518. Keeping the meta-data fields in the message headers means the worker
  519. doesn't actually have to decode the payload before delivering
  520. the task to the child process, and also that it's now possible
  521. for the worker to reroute a task written in a language different
  522. from Python to a different worker.
  523. - A new ``lang`` message header can be used to specify the programming
  524. language the task is written in.
  525. - Worker stores results for internal errors like ``ContentDisallowed``,
  526. and other deserialization errors.
  527. - Worker stores results and sends monitoring events for unregistered
  528. task errors.
  529. - Worker calls callbacks/errbacks even when the result is sent by the
  530. parent process (e.g., :exc:`WorkerLostError` when a child process
  531. terminates, deserialization errors, unregistered tasks).
  532. - A new ``origin`` header contains information about the process sending
  533. the task (worker node-name, or PID and host-name information).
  534. - A new ``shadow`` header allows you to modify the task name used in logs.
  535. This is useful for dispatch like patterns, like a task that calls
  536. any function using pickle (don't do this at home):
  537. .. code-block:: python
  538. from celery import Task
  539. from celery.utils.imports import qualname
  540. class call_as_task(Task):
  541. def shadow_name(self, args, kwargs, options):
  542. return 'call_as_task:{0}'.format(qualname(args[0]))
  543. def run(self, fun, *args, **kwargs):
  544. return fun(*args, **kwargs)
  545. call_as_task = app.register_task(call_as_task())
  546. - New ``argsrepr`` and ``kwargsrepr`` fields contain textual representations
  547. of the task arguments (possibly truncated) for use in logs, monitors, etc.
  548. This means the worker doesn't have to deserialize the message payload
  549. to display the task arguments for informational purposes.
  550. - Chains now use a dedicated ``chain`` field enabling support for chains
  551. of thousands and more tasks.
  552. - New ``parent_id`` and ``root_id`` headers adds information about
  553. a tasks relationship with other tasks.
  554. - ``parent_id`` is the task id of the task that called this task
  555. - ``root_id`` is the first task in the work-flow.
  556. These fields can be used to improve monitors like flower to group
  557. related messages together (like chains, groups, chords, complete
  558. work-flows, etc).
  559. - ``app.TaskProducer`` replaced by :meth:`@amqp.create_task_message` and
  560. :meth:`@amqp.send_task_message`.
  561. Dividing the responsibilities into creating and sending means that
  562. people who want to send messages using a Python AMQP client directly,
  563. don't have to implement the protocol.
  564. The :meth:`@amqp.create_task_message` method calls either
  565. :meth:`@amqp.as_task_v2`, or :meth:`@amqp.as_task_v1` depending
  566. on the configured task protocol, and returns a special
  567. :class:`~celery.app.amqp.task_message` tuple containing the
  568. headers, properties and body of the task message.
  569. .. seealso::
  570. The new task protocol is documented in full here:
  571. :ref:`message-protocol-task-v2`.
  572. Prefork Pool Improvements
  573. -------------------------
  574. Tasks now log from the child process
  575. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  576. Logging of task success/failure now happens from the child process
  577. executing the task. As a result logging utilities,
  578. like Sentry can get full information about tasks, including
  579. variables in the traceback stack.
  580. ``-Ofair`` is now the default scheduling strategy
  581. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  582. To re-enable the default behavior in 3.1 use the ``-Ofast`` command-line
  583. option.
  584. There's been lots of confusion about what the ``-Ofair`` command-line option
  585. does, and using the term "prefetch" in explanations have probably not helped
  586. given how confusing this terminology is in AMQP.
  587. When a Celery worker using the prefork pool receives a task, it needs to
  588. delegate that task to a child process for execution.
  589. The prefork pool has a configurable number of child processes
  590. (``--concurrency``) that can be used to execute tasks, and each child process
  591. uses pipes/sockets to communicate with the parent process:
  592. - inqueue (pipe/socket): parent sends task to the child process
  593. - outqueue (pipe/socket): child sends result/return value to the parent.
  594. In Celery 3.1 the default scheduling mechanism was simply to send
  595. the task to the first ``inqueue`` that was writable, with some heuristics
  596. to make sure we round-robin between them to ensure each child process
  597. would receive the same amount of tasks.
  598. This means that in the default scheduling strategy, a worker may send
  599. tasks to the same child process that is already executing a task. If that
  600. task is long running, it may block the waiting task for a long time. Even
  601. worse, hundreds of short-running tasks may be stuck behind a long running task
  602. even when there are child processes free to do work.
  603. The ``-Ofair`` scheduling strategy was added to avoid this situation,
  604. and when enabled it adds the rule that no task should be sent to the a child
  605. process that is already executing a task.
  606. The fair scheduling strategy may perform slightly worse if you have only
  607. short running tasks.
  608. Limit child process resident memory size
  609. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  610. .. :sha:`5cae0e754128750a893524dcba4ae030c414de33`
  611. You can now limit the maximum amount of memory allocated per prefork
  612. pool child process by setting the worker
  613. :option:`--max-memory-per-child <celery worker --max-memory-per-child>` option,
  614. or the :setting:`worker_max_memory_per_child` setting.
  615. The limit is for RSS/resident memory size and is specified in kilobytes.
  616. A child process having exceeded the limit will be terminated and replaced
  617. with a new process after the currently executing task returns.
  618. See :ref:`worker-max-memory-per-child` for more information.
  619. Contributed by **Dave Smith**.
  620. One log-file per child process
  621. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  622. Init-scrips and :program:`celery multi` now uses the `%I` log file format
  623. option (e.g., :file:`/var/log/celery/%n%I.log`).
  624. This change was necessary to ensure each child
  625. process has a separate log file after moving task logging
  626. to the child process, as multiple processes writing to the same
  627. log file can cause corruption.
  628. You're encouraged to upgrade your init-scripts and
  629. :program:`celery multi` arguments to use this new option.
  630. Transports
  631. ----------
  632. RabbitMQ priority queue support
  633. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  634. See :ref:`routing-options-rabbitmq-priorities` for more information.
  635. Contributed by **Gerald Manipon**.
  636. Configure broker URL for read/write separately
  637. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  638. New :setting:`broker_read_url` and :setting:`broker_write_url` settings
  639. have been added so that separate broker URLs can be provided
  640. for connections used for consuming/publishing.
  641. In addition to the configuration options, two new methods have been
  642. added the app API:
  643. - ``app.connection_for_read()``
  644. - ``app.connection_for_write()``
  645. These should now be used in place of ``app.connection()`` to specify
  646. the intent of the required connection.
  647. .. note::
  648. Two connection pools are available: ``app.pool`` (read), and
  649. ``app.producer_pool`` (write). The latter doesn't actually give connections
  650. but full :class:`kombu.Producer` instances.
  651. .. code-block:: python
  652. def publish_some_message(app, producer=None):
  653. with app.producer_or_acquire(producer) as producer:
  654. ...
  655. def consume_messages(app, connection=None):
  656. with app.connection_or_acquire(connection) as connection:
  657. ...
  658. RabbitMQ queue extensions support
  659. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  660. Queue declarations can now set a message TTL and queue expiry time directly,
  661. by using the ``message_ttl`` and ``expires`` arguments
  662. New arguments have been added to :class:`~kombu.Queue` that lets
  663. you directly and conveniently configure RabbitMQ queue extensions
  664. in queue declarations:
  665. - ``Queue(expires=20.0)``
  666. Set queue expiry time in float seconds.
  667. See :attr:`kombu.Queue.expires`.
  668. - ``Queue(message_ttl=30.0)``
  669. Set queue message time-to-live float seconds.
  670. See :attr:`kombu.Queue.message_ttl`.
  671. - ``Queue(max_length=1000)``
  672. Set queue max length (number of messages) as int.
  673. See :attr:`kombu.Queue.max_length`.
  674. - ``Queue(max_length_bytes=1000)``
  675. Set queue max length (message size total in bytes) as int.
  676. See :attr:`kombu.Queue.max_length_bytes`.
  677. - ``Queue(max_priority=10)``
  678. Declare queue to be a priority queue that routes messages
  679. based on the ``priority`` field of the message.
  680. See :attr:`kombu.Queue.max_priority`.
  681. Amazon SQS transport now officially supported
  682. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  683. The SQS broker transport has been rewritten to use async I/O and as such
  684. joins RabbitMQ, Redis and QPid as officially supported transports.
  685. The new implementation also takes advantage of long polling,
  686. and closes several issues related to using SQS as a broker.
  687. This work was sponsored by Nextdoor.
  688. Apache QPid transport now officially supported
  689. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  690. Contributed by **Brian Bouterse**.
  691. Redis: Support for Sentinel
  692. ---------------------------
  693. You can point the connection to a list of sentinel URLs like:
  694. .. code-block:: text
  695. sentinel://0.0.0.0:26379;sentinel://0.0.0.0:26380/...
  696. where each sentinel is separated by a `;`. Multiple sentinels are handled
  697. by :class:`kombu.Connection` constructor, and placed in the alternative
  698. list of servers to connect to in case of connection failure.
  699. Contributed by **Sergey Azovskov**, and **Lorenzo Mancini**.
  700. Tasks
  701. -----
  702. Task Auto-retry Decorator
  703. ~~~~~~~~~~~~~~~~~~~~~~~~~
  704. Writing custom retry handling for exception events is so common
  705. that we now have built-in support for it.
  706. For this a new ``autoretry_for`` argument is now supported by
  707. the task decorators, where you can specify a tuple of exceptions
  708. to automatically retry for:
  709. .. code-block:: python
  710. from twitter.exceptions import FailWhaleError
  711. @app.task(autoretry_for=(FailWhaleError,))
  712. def refresh_timeline(user):
  713. return twitter.refresh_timeline(user)
  714. See :ref:`task-autoretry` for more information.
  715. Contributed by **Dmitry Malinovsky**.
  716. .. :sha:`75246714dd11e6c463b9dc67f4311690643bff24`
  717. ``Task.replace`` Improvements
  718. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  719. - ``self.replace(signature)`` can now replace any task, chord or group,
  720. and the signature to replace with can be a chord, group or any other
  721. type of signature.
  722. - No longer inherits the callbacks and errbacks of the existing task.
  723. If you replace a node in a tree, then you wouldn't expect the new node to
  724. inherit the children of the old node.
  725. - ``Task.replace_in_chord`` has been removed, use ``.replace`` instead.
  726. - If the replacement is a group, that group will be automatically converted
  727. to a chord, where the callback "accumulates" the results of the group tasks.
  728. A new built-in task (`celery.accumulate` was added for this purpose)
  729. Contributed by **Steeve Morin**, and **Ask Solem**.
  730. Remote Task Tracebacks
  731. ~~~~~~~~~~~~~~~~~~~~~~
  732. The new :setting:`task_remote_tracebacks` will make task tracebacks more
  733. useful by injecting the stack of the remote worker.
  734. This feature requires the additional :pypi:`tblib` library.
  735. Contributed by **Ionel Cristian Mărieș**.
  736. Handling task connection errors
  737. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  738. Connection related errors occurring while sending a task is now re-raised
  739. as a :exc:`kombu.exceptions.OperationalError` error:
  740. .. code-block:: pycon
  741. >>> try:
  742. ... add.delay(2, 2)
  743. ... except add.OperationalError as exc:
  744. ... print('Could not send task %r: %r' % (add, exc))
  745. See :ref:`calling-connection-errors` for more information.
  746. Gevent/Eventlet: Dedicated thread for consuming results
  747. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  748. When using :pypi:`gevent`, or :pypi:`eventlet` there is now a single
  749. thread responsible for consuming events.
  750. This means that if you have many calls retrieving results, there will be
  751. a dedicated thread for consuming them:
  752. .. code-block:: python
  753. result = add.delay(2, 2)
  754. # this call will delegate to the result consumer thread:
  755. # once the consumer thread has received the result this greenlet can
  756. # continue.
  757. value = result.get(timeout=3)
  758. This makes performing RPC calls when using gevent/eventlet perform much
  759. better.
  760. ``AsyncResult.then(on_success, on_error)``
  761. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  762. The AsyncResult API has been extended to support the :class:`~vine.promise` protocol.
  763. This currently only works with the RPC (amqp) and Redis result backends, but
  764. lets you attach callbacks to when tasks finish:
  765. .. code-block:: python
  766. import gevent.monkey
  767. monkey.patch_all()
  768. import time
  769. from celery import Celery
  770. app = Celery(broker='amqp://', backend='rpc')
  771. @app.task
  772. def add(x, y):
  773. return x + y
  774. def on_result_ready(result):
  775. print('Received result for id %r: %r' % (result.id, result.result,))
  776. add.delay(2, 2).then(on_result_ready)
  777. time.sleep(3) # run gevent event loop for a while.
  778. Demonstrated using :pypi:`gevent` here, but really this is an API that's more
  779. useful in callback-based event loops like :pypi:`twisted`, or :pypi:`tornado`.
  780. New Task Router API
  781. ~~~~~~~~~~~~~~~~~~~
  782. The :setting:`task_routes` setting can now hold functions, and map routes
  783. now support glob patterns and regexes.
  784. Instead of using router classes you can now simply define a function:
  785. .. code-block:: python
  786. def route_for_task(name, args, kwargs, options, task=None, **kwargs):
  787. from proj import tasks
  788. if name == tasks.add.name:
  789. return {'queue': 'hipri'}
  790. If you don't need the arguments you can use start arguments, just make
  791. sure you always also accept star arguments so that we have the ability
  792. to add more features in the future:
  793. .. code-block:: python
  794. def route_for_task(name, *args, **kwargs):
  795. from proj import tasks
  796. if name == tasks.add.name:
  797. return {'queue': 'hipri', 'priority': 9}
  798. Both the ``options`` argument and the new ``task`` keyword argument
  799. are new to the function-style routers, and will make it easier to write
  800. routers based on execution options, or properties of the task.
  801. The optional ``task`` keyword argument won't be set if a task is called
  802. by name using :meth:`@send_task`.
  803. For more examples, including using glob/regexes in routers please see
  804. :setting:`task_routes` and :ref:`routing-automatic`.
  805. Canvas Refactor
  806. ~~~~~~~~~~~~~~~
  807. The canvas/work-flow implementation have been heavily refactored
  808. to fix some long outstanding issues.
  809. .. :sha:`d79dcd8e82c5e41f39abd07ffed81ca58052bcd2`
  810. .. :sha:`1e9dd26592eb2b93f1cb16deb771cfc65ab79612`
  811. .. :sha:`e442df61b2ff1fe855881c1e2ff9acc970090f54`
  812. .. :sha:`0673da5c09ac22bdd49ba811c470b73a036ee776`
  813. - Error callbacks can now take real exception and traceback instances
  814. (Issue #2538).
  815. .. code-block:: pycon
  816. >>> add.s(2, 2).on_error(log_error.s()).delay()
  817. Where ``log_error`` could be defined as:
  818. .. code-block:: python
  819. @app.task
  820. def log_error(request, exc, traceback):
  821. with open(os.path.join('/var/errors', request.id), 'a') as fh:
  822. print('--\n\n{0} {1} {2}'.format(
  823. task_id, exc, traceback), file=fh)
  824. See :ref:`guide-canvas` for more examples.
  825. - ``chain(a, b, c)`` now works the same as ``a | b | c``.
  826. This means chain may no longer return an instance of ``chain``,
  827. instead it may optimize the workflow so that e.g. two groups
  828. chained together becomes one group.
  829. - Now unrolls groups within groups into a single group (Issue #1509).
  830. - chunks/map/starmap tasks now routes based on the target task
  831. - chords and chains can now be immutable.
  832. - Fixed bug where serialized signatures weren't converted back into
  833. signatures (Issue #2078)
  834. Fix contributed by **Ross Deane**.
  835. - Fixed problem where chains and groups didn't work when using JSON
  836. serialization (Issue #2076).
  837. Fix contributed by **Ross Deane**.
  838. - Creating a chord no longer results in multiple values for keyword
  839. argument 'task_id' (Issue #2225).
  840. Fix contributed by **Aneil Mallavarapu**.
  841. - Fixed issue where the wrong result is returned when a chain
  842. contains a chord as the penultimate task.
  843. Fix contributed by **Aneil Mallavarapu**.
  844. - Special case of ``group(A.s() | group(B.s() | C.s()))`` now works.
  845. - Chain: Fixed bug with incorrect id set when a subtask is also a chain.
  846. - ``group | group`` is now flattened into a single group (Issue #2573).
  847. - Fixed issue where ``group | task`` wasn't upgrading correctly
  848. to chord (Issue #2922).
  849. - Chords now properly sets ``result.parent`` links.
  850. - ``chunks``/``map``/``starmap`` are now routed based on the target task.
  851. - ``Signature.link`` now works when argument is scalar (not a list)
  852. (Issue #2019).
  853. - ``group()`` now properly forwards keyword arguments (Issue #3426).
  854. Fix contributed by **Samuel Giffard**.
  855. - A ``chord`` where the header group only consists of a single task
  856. is now turned into a simple chain.
  857. - Passing a ``link`` argument to ``group.apply_async()`` now raises an error
  858. (Issue #3508).
  859. - ``chord | sig`` now attaches to the chord callback (Issue #3356).
  860. Periodic Tasks
  861. --------------
  862. New API for configuring periodic tasks
  863. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  864. This new API enables you to use signatures when defining periodic tasks,
  865. removing the chance of mistyping task names.
  866. An example of the new API is :ref:`here <beat-entries>`.
  867. .. :sha:`bc18d0859c1570f5eb59f5a969d1d32c63af764b`
  868. .. :sha:`132d8d94d38f4050db876f56a841d5a5e487b25b`
  869. Optimized Beat implementation
  870. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  871. The :program:`celery beat` implementation has been optimized
  872. for millions of periodic tasks by using a heap to schedule entries.
  873. Contributed by **Ask Solem** and **Alexander Koshelev**.
  874. Schedule tasks based on sunrise, sunset, dawn and dusk
  875. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  876. See :ref:`beat-solar` for more information.
  877. Contributed by **Mark Parncutt**.
  878. Result Backends
  879. ---------------
  880. RPC Result Backend matured
  881. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  882. Lots of bugs in the previously experimental RPC result backend have been fixed
  883. and can now be considered to production use.
  884. Contributed by **Ask Solem**, **Morris Tweed**.
  885. Redis: Result backend optimizations
  886. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  887. ``result.get()`` is now using pub/sub for streaming task results
  888. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  889. Calling ``result.get()`` when using the Redis result backend
  890. used to be extremely expensive as it was using polling to wait
  891. for the result to become available. A default polling
  892. interval of 0.5 seconds didn't help performance, but was
  893. necessary to avoid a spin loop.
  894. The new implementation is using Redis Pub/Sub mechanisms to
  895. publish and retrieve results immediately, greatly improving
  896. task round-trip times.
  897. Contributed by **Yaroslav Zhavoronkov** and **Ask Solem**.
  898. New optimized chord join implementation
  899. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  900. This was an experimental feature introduced in Celery 3.1,
  901. that could only be enabled by adding ``?new_join=1`` to the
  902. result backend URL configuration.
  903. We feel that the implementation has been tested thoroughly enough
  904. to be considered stable and enabled by default.
  905. The new implementation greatly reduces the overhead of chords,
  906. and especially with larger chords the performance benefit can be massive.
  907. New Riak result backend introduced
  908. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  909. See :ref:`conf-riak-result-backend` for more information.
  910. Contributed by **Gilles Dartiguelongue**, **Alman One** and **NoKriK**.
  911. New CouchDB result backend introduced
  912. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  913. See :ref:`conf-couchdb-result-backend` for more information.
  914. Contributed by **Nathan Van Gheem**.
  915. New Consul result backend introduced
  916. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  917. Add support for Consul as a backend using the Key/Value store of Consul.
  918. Consul has an HTTP API where through you can store keys with their values.
  919. The backend extends KeyValueStoreBackend and implements most of the methods.
  920. Mainly to set, get and remove objects.
  921. This allows Celery to store Task results in the K/V store of Consul.
  922. Consul also allows to set a TTL on keys using the Sessions from Consul. This way
  923. the backend supports auto expiry of Task results.
  924. For more information on Consul visit https://consul.io/
  925. The backend uses :pypi:`python-consul` for talking to the HTTP API.
  926. This package is fully Python 3 compliant just as this backend is:
  927. .. code-block:: console
  928. $ pip install python-consul
  929. That installs the required package to talk to Consul's HTTP API from Python.
  930. You can also specify consul as an extension in your dependency on Celery:
  931. .. code-block:: console
  932. $ pip install celery[consul]
  933. See :ref:`bundles` for more information.
  934. Contributed by **Wido den Hollander**.
  935. Brand new Cassandra result backend
  936. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  937. A brand new Cassandra backend utilizing the new :pypi:`cassandra-driver`
  938. library is replacing the old result backend using the older
  939. :pypi:`pycassa` library.
  940. See :ref:`conf-cassandra-result-backend` for more information.
  941. To depend on Celery with Cassandra as the result backend use:
  942. .. code-block:: console
  943. $ pip install celery[cassandra]
  944. You can also combine multiple extension requirements,
  945. please see :ref:`bundles` for more information.
  946. .. # XXX What changed?
  947. New Elasticsearch result backend introduced
  948. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  949. See :ref:`conf-elasticsearch-result-backend` for more information.
  950. To depend on Celery with Elasticsearch as the result bakend use:
  951. .. code-block:: console
  952. $ pip install celery[elasticsearch]
  953. You can also combine multiple extension requirements,
  954. please see :ref:`bundles` for more information.
  955. Contributed by **Ahmet Demir**.
  956. New File-system result backend introduced
  957. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  958. See :ref:`conf-filesystem-result-backend` for more information.
  959. Contributed by **Môshe van der Sterre**.
  960. Event Batching
  961. --------------
  962. Events are now buffered in the worker and sent as a list, reducing
  963. the overhead required to send monitoring events.
  964. For authors of custom event monitors there will be no action
  965. required as long as you're using the Python Celery
  966. helpers (:class:`~@events.Receiver`) to implement your monitor.
  967. However, if you're parsing raw event messages you must now account
  968. for batched event messages, as they differ from normal event messages
  969. in the following way:
  970. - The routing key for a batch of event messages will be set to
  971. ``<event-group>.multi`` where the only batched event group
  972. is currently ``task`` (giving a routing key of ``task.multi``).
  973. - The message body will be a serialized list-of-dictionaries instead
  974. of a dictionary. Each item in the list can be regarded
  975. as a normal event message body.
  976. .. :sha:`03399b4d7c26fb593e61acf34f111b66b340ba4e`
  977. In Other News...
  978. ----------------
  979. Requirements
  980. ~~~~~~~~~~~~
  981. - Now depends on :ref:`Kombu 4.0 <kombu:version-4.0>`.
  982. - Now depends on :pypi:`billiard` version 3.5.
  983. - No longer depends on :pypi:`anyjson`. Good-bye old friend :(
  984. Tasks
  985. ~~~~~
  986. - The "anon-exchange" is now used for simple name-name direct routing.
  987. This increases performance as it completely bypasses the routing table,
  988. in addition it also improves reliability for the Redis broker transport.
  989. - An empty ResultSet now evaluates to True.
  990. Fix contributed by **Colin McIntosh**.
  991. - The default routing key and exchange name is now taken from the
  992. :setting:`task_default_queue` setting.
  993. This means that to change the name of the default queue, you now
  994. only have to set a single setting.
  995. - New :setting:`task_reject_on_worker_lost` setting, and
  996. :attr:`~@Task.reject_on_worker_lost` task attribute decides what happens
  997. when the child worker process executing a late ack task is terminated.
  998. Contributed by **Michael Permana**.
  999. - ``Task.subtask`` renamed to ``Task.signature`` with alias.
  1000. - ``Task.subtask_from_request`` renamed to
  1001. ``Task.signature_from_request`` with alias.
  1002. - The ``delivery_mode`` attribute for :class:`kombu.Queue` is now
  1003. respected (Issue #1953).
  1004. - Routes in :setting:`task-routes` can now specify a
  1005. :class:`~kombu.Queue` instance directly.
  1006. Example:
  1007. .. code-block:: python
  1008. task_routes = {'proj.tasks.add': {'queue': Queue('add')}}
  1009. - ``AsyncResult`` now raises :exc:`ValueError` if task_id is None.
  1010. (Issue #1996).
  1011. - Retried tasks didn't forward expires setting (Issue #3297).
  1012. - ``result.get()`` now supports an ``on_message`` argument to set a
  1013. callback to be called for every message received.
  1014. - New abstract classes added:
  1015. - :class:`~celery.utils.abstract.CallableTask`
  1016. Looks like a task.
  1017. - :class:`~celery.utils.abstract.CallableSignature`
  1018. Looks like a task signature.
  1019. - ``Task.replace`` now properly forwards callbacks (Issue #2722).
  1020. Fix contributed by **Nicolas Unravel**.
  1021. - ``Task.replace``: Append to chain/chord (Closes #3232)
  1022. Fixed issue #3232, adding the signature to the chain (if there's any).
  1023. Fixed the chord suppress if the given signature contains one.
  1024. Fix contributed by :github_user:`honux`.
  1025. - Task retry now also throws in eager mode.
  1026. Fix contributed by **Feanil Patel**.
  1027. Beat
  1028. ~~~~
  1029. - Fixed crontab infinite loop with invalid date.
  1030. When occurrence can never be reached (example, April, 31th), trying
  1031. to reach the next occurrence would trigger an infinite loop.
  1032. Try fixing that by raising a :exc:`RuntimeError` after 2,000 iterations
  1033. (Also added a test for crontab leap years in the process)
  1034. Fix contributed by **Romuald Brunet**.
  1035. - Now ensures the program exits with a non-zero exit code when an
  1036. exception terminates the service.
  1037. Fix contributed by **Simon Peeters**.
  1038. App
  1039. ~~~
  1040. - Dates are now always timezone aware even if
  1041. :setting:`enable_utc` is disabled (Issue #943).
  1042. Fix contributed by **Omer Katz**.
  1043. - **Config**: App preconfiguration is now also pickled with the configuration.
  1044. Fix contributed by **Jeremy Zafran**.
  1045. - The application can now change how task names are generated using
  1046. the :meth:`~@gen_task_name` method.
  1047. Contributed by **Dmitry Malinovsky**.
  1048. - App has new ``app.current_worker_task`` property that
  1049. returns the task that's currently being worked on (or :const:`None`).
  1050. (Issue #2100).
  1051. Logging
  1052. ~~~~~~~
  1053. - :func:`~celery.utils.log.get_task_logger` now raises an exception
  1054. if trying to use the name "celery" or "celery.task" (Issue #3475).
  1055. Execution Pools
  1056. ~~~~~~~~~~~~~~~
  1057. - **Eventlet/Gevent**: now enables AMQP heartbeat (Issue #3338).
  1058. - **Eventlet/Gevent**: Fixed race condition leading to "simultaneous read"
  1059. errors (Issue #2755).
  1060. - **Prefork**: Prefork pool now uses ``poll`` instead of ``select`` where
  1061. available (Issue #2373).
  1062. - **Prefork**: Fixed bug where the pool would refuse to shut down the
  1063. worker (Issue #2606).
  1064. - **Eventlet**: Now returns pool size in :program:`celery inspect stats`
  1065. command.
  1066. Contributed by **Alexander Oblovatniy**.
  1067. Testing
  1068. -------
  1069. - Celery is now a :pypi:`pytest` plugin, including fixtures
  1070. useful for unit and integration testing.
  1071. See the :ref:`testing user guide <testing>` for more information.
  1072. Transports
  1073. ~~~~~~~~~~
  1074. - ``amqps://`` can now be specified to require SSL.
  1075. - **Redis Transport**: The Redis transport now supports the
  1076. :setting:`broker_use_ssl` option.
  1077. Contributed by **Robert Kolba**.
  1078. - JSON serializer now calls ``obj.__json__`` for unsupported types.
  1079. This means you can now define a ``__json__`` method for custom
  1080. types that can be reduced down to a built-in json type.
  1081. Example:
  1082. .. code-block:: python
  1083. class Person:
  1084. first_name = None
  1085. last_name = None
  1086. address = None
  1087. def __json__(self):
  1088. return {
  1089. 'first_name': self.first_name,
  1090. 'last_name': self.last_name,
  1091. 'address': self.address,
  1092. }
  1093. - JSON serializer now handles datetime's, Django promise, UUID and Decimal.
  1094. - New ``Queue.consumer_arguments`` can be used for the ability to
  1095. set consumer priority via ``x-priority``.
  1096. See https://www.rabbitmq.com/consumer-priority.html
  1097. Example:
  1098. .. code-block:: python
  1099. consumer = Consumer(channel, consumer_arguments={'x-priority': 3})
  1100. - Queue/Exchange: ``no_declare`` option added (also enabled for
  1101. internal amq. exchanges).
  1102. Programs
  1103. ~~~~~~~~
  1104. - Celery is now using :mod:`argparse`, instead of :mod:`optparse`.
  1105. - All programs now disable colors if the controlling terminal is not a TTY.
  1106. - :program:`celery worker`: The ``-q`` argument now disables the startup
  1107. banner.
  1108. - :program:`celery worker`: The "worker ready" message is now logged
  1109. using severity info, instead of warn.
  1110. - :program:`celery multi`: ``%n`` format for is now synonym with
  1111. ``%N`` to be consistent with :program:`celery worker`.
  1112. - :program:`celery inspect`/:program:`celery control`: now supports a new
  1113. :option:`--json <celery inspect --json>` option to give output in json format.
  1114. - :program:`celery inspect registered`: now ignores built-in tasks.
  1115. - :program:`celery purge` now takes ``-Q`` and ``-X`` options
  1116. used to specify what queues to include and exclude from the purge.
  1117. - New :program:`celery logtool`: Utility for filtering and parsing
  1118. celery worker log-files
  1119. - :program:`celery multi`: now passes through `%i` and `%I` log
  1120. file formats.
  1121. - General: ``%p`` can now be used to expand to the full worker node-name
  1122. in log-file/pid-file arguments.
  1123. - A new command line option
  1124. :option:`--executable <celery worker --executable>` is now
  1125. available for daemonizing programs (:program:`celery worker` and
  1126. :program:`celery beat`).
  1127. Contributed by **Bert Vanderbauwhede**.
  1128. - :program:`celery worker`: supports new
  1129. :option:`--prefetch-multiplier <celery worker --prefetch-multiplier>` option.
  1130. Contributed by **Mickaël Penhard**.
  1131. - The ``--loader`` argument is now always effective even if an app argument is
  1132. set (Issue #3405).
  1133. - inspect/control now takes commands from registry
  1134. This means user remote-control commands can also be used from the
  1135. command-line.
  1136. Note that you need to specify the arguments/and type of arguments
  1137. for the arguments to be correctly passed on the command-line.
  1138. There are now two decorators, which use depends on the type of
  1139. command: `@inspect_command` + `@control_command`:
  1140. .. code-block:: python
  1141. from celery.worker.control import control_command
  1142. @control_command(
  1143. args=[('n', int)]
  1144. signature='[N=1]',
  1145. )
  1146. def something(state, n=1, **kwargs):
  1147. ...
  1148. Here ``args`` is a list of args supported by the command.
  1149. The list must contain tuples of ``(argument_name, type)``.
  1150. ``signature`` is just the command-line help used in e.g.
  1151. ``celery -A proj control --help``.
  1152. Commands also support `variadic` arguments, which means that any
  1153. arguments left over will be added to a single variable. Here demonstrated
  1154. by the ``terminate`` command which takes a signal argument and a variable
  1155. number of task_ids:
  1156. .. code-block:: python
  1157. from celery.worker.control import control_command
  1158. @control_command(
  1159. args=[('signal', str)],
  1160. signature='<signal> [id1, [id2, [..., [idN]]]]',
  1161. variadic='ids',
  1162. )
  1163. def terminate(state, signal, ids, **kwargs):
  1164. ...
  1165. This command can now be called using:
  1166. .. code-block:: console
  1167. $ celery -A proj control terminate SIGKILL id1 id2 id3`
  1168. See :ref:`worker-custom-control-commands` for more information.
  1169. Worker
  1170. ~~~~~~
  1171. - Improvements and fixes for :class:`~celery.utils.collections.LimitedSet`.
  1172. Getting rid of leaking memory + adding ``minlen`` size of the set:
  1173. the minimal residual size of the set after operating for some time.
  1174. ``minlen`` items are kept, even if they should've been expired.
  1175. Problems with older and even more old code:
  1176. #. Heap would tend to grow in some scenarios
  1177. (like adding an item multiple times).
  1178. #. Adding many items fast wouldn't clean them soon enough (if ever).
  1179. #. When talking to other workers, revoked._data was sent, but
  1180. it was processed on the other side as iterable.
  1181. That means giving those keys new (current)
  1182. time-stamp. By doing this workers could recycle
  1183. items forever. Combined with 1) and 2), this means that in
  1184. large set of workers, you're getting out of memory soon.
  1185. All those problems should be fixed now.
  1186. This should fix issues #3095, #3086.
  1187. Contributed by **David Pravec**.
  1188. - New settings to control remote control command queues.
  1189. - :setting:`control_queue_expires`
  1190. Set queue expiry time for both remote control command queues,
  1191. and remote control reply queues.
  1192. - :setting:`control_queue_ttl`
  1193. Set message time-to-live for both remote control command queues,
  1194. and remote control reply queues.
  1195. Contributed by **Alan Justino**.
  1196. - The :signal:`worker_shutdown` signal is now always called during shutdown.
  1197. Previously it would not be called if the worker instance was collected
  1198. by gc first.
  1199. - Worker now only starts the remote control command consumer if the
  1200. broker transport used actually supports them.
  1201. - Gossip now sets ``x-message-ttl`` for event queue to heartbeat_interval s.
  1202. (Issue #2005).
  1203. - Now preserves exit code (Issue #2024).
  1204. - Now rejects messages with an invalid ETA value (instead of ack, which means
  1205. they will be sent to the dead-letter exchange if one is configured).
  1206. - Fixed crash when the ``-purge`` argument was used.
  1207. - Log--level for unrecoverable errors changed from ``error`` to
  1208. ``critical``.
  1209. - Improved rate limiting accuracy.
  1210. - Account for missing timezone information in task expires field.
  1211. Fix contributed by **Albert Wang**.
  1212. - The worker no longer has a ``Queues`` bootsteps, as it is now
  1213. superfluous.
  1214. - Now emits the "Received task" line even for revoked tasks.
  1215. (Issue #3155).
  1216. - Now respects :setting:`broker_connection_retry` setting.
  1217. Fix contributed by **Nat Williams**.
  1218. - New :setting:`control_queue_ttl` and :setting:`control_queue_expires`
  1219. settings now enables you to configure remote control command
  1220. message TTLs, and queue expiry time.
  1221. Contributed by **Alan Justino**.
  1222. - New :data:`celery.worker.state.requests` enables O(1) loookup
  1223. of active/reserved tasks by id.
  1224. - Auto-scale didn't always update keep-alive when scaling down.
  1225. Fix contributed by **Philip Garnero**.
  1226. - Fixed typo ``options_list`` -> ``option_list``.
  1227. Fix contributed by **Greg Wilbur**.
  1228. - Some worker command-line arguments and ``Worker()`` class arguments have
  1229. been renamed for consistency.
  1230. All of these have aliases for backward compatibility.
  1231. - ``--send-events`` -> ``--task-events``
  1232. - ``--schedule`` -> ``--schedule-filename``
  1233. - ``--maxtasksperchild`` -> ``--max-tasks-per-child``
  1234. - ``Beat(scheduler_cls=)`` -> ``Beat(scheduler=)``
  1235. - ``Worker(send_events=True)`` -> ``Worker(task_events=True)``
  1236. - ``Worker(task_time_limit=)`` -> ``Worker(time_limit=``)
  1237. - ``Worker(task_soft_time_limit=)`` -> ``Worker(soft_time_limit=)``
  1238. - ``Worker(state_db=)`` -> ``Worker(statedb=)``
  1239. - ``Worker(working_directory=)`` -> ``Worker(workdir=)``
  1240. Debugging Utilities
  1241. ~~~~~~~~~~~~~~~~~~~
  1242. - :mod:`celery.contrib.rdb`: Changed remote debugger banner so that you can copy and paste
  1243. the address easily (no longer has a period in the address).
  1244. Contributed by **Jonathan Vanasco**.
  1245. - Fixed compatibility with recent :pypi:`psutil` versions (Issue #3262).
  1246. Signals
  1247. ~~~~~~~
  1248. - **App**: New signals for app configuration/finalization:
  1249. - :data:`app.on_configure <@on_configure>`
  1250. - :data:`app.on_after_configure <@on_after_configure>`
  1251. - :data:`app.on_after_finalize <@on_after_finalize>`
  1252. - **Task**: New task signals for rejected task messages:
  1253. - :data:`celery.signals.task_rejected`.
  1254. - :data:`celery.signals.task_unknown`.
  1255. - **Worker**: New signal for when a heartbeat event is sent.
  1256. - :data:`celery.signals.heartbeat_sent`
  1257. Contributed by **Kevin Richardson**.
  1258. Events
  1259. ~~~~~~
  1260. - Event messages now uses the RabbitMQ ``x-message-ttl`` option
  1261. to ensure older event messages are discarded.
  1262. The default is 5 seconds, but can be changed using the
  1263. :setting:`event_queue_ttl` setting.
  1264. - ``Task.send_event`` now automatically retries sending the event
  1265. on connection failure, according to the task publish retry settings.
  1266. - Event monitors now sets the :setting:`event_queue_expires`
  1267. setting by default.
  1268. The queues will now expire after 60 seconds after the monitor stops
  1269. consuming from it.
  1270. - Fixed a bug where a None value wasn't handled properly.
  1271. Fix contributed by **Dongweiming**.
  1272. - New :setting:`event_queue_prefix` setting can now be used
  1273. to change the default ``celeryev`` queue prefix for event receiver queues.
  1274. Contributed by **Takeshi Kanemoto**.
  1275. - ``State.tasks_by_type`` and ``State.tasks_by_worker`` can now be
  1276. used as a mapping for fast access to this information.
  1277. Deployment
  1278. ~~~~~~~~~~
  1279. - Generic init-scripts now support
  1280. :envvar:`CELERY_SU` and :envvar:`CELERYD_SU_ARGS` environment variables
  1281. to set the path and arguments for :command:`su` (:manpage:`su(1)`).
  1282. - Generic init-scripts now better support FreeBSD and other BSD
  1283. systems by searching :file:`/usr/local/etc/` for the configuration file.
  1284. Contributed by **Taha Jahangir**.
  1285. - Generic init-script: Fixed strange bug for ``celerybeat`` where
  1286. restart didn't always work (Issue #3018).
  1287. - The systemd init script now uses a shell when executing
  1288. services.
  1289. Contributed by **Tomas Machalek**.
  1290. Result Backends
  1291. ~~~~~~~~~~~~~~~
  1292. - Redis: Now has a default socket timeout of 120 seconds.
  1293. The default can be changed using the new :setting:`redis_socket_timeout`
  1294. setting.
  1295. Contributed by **Raghuram Srinivasan**.
  1296. - RPC Backend result queues are now auto delete by default (Issue #2001).
  1297. - RPC Backend: Fixed problem where exception
  1298. wasn't deserialized properly with the json serializer (Issue #2518).
  1299. Fix contributed by **Allard Hoeve**.
  1300. - CouchDB: The backend used to double-json encode results.
  1301. Fix contributed by **Andrew Stewart**.
  1302. - CouchDB: Fixed typo causing the backend to not be found
  1303. (Issue #3287).
  1304. Fix contributed by **Andrew Stewart**.
  1305. - MongoDB: Now supports setting the :setting:`result_serialzier` setting
  1306. to ``bson`` to use the MongoDB libraries own serializer.
  1307. Contributed by **Davide Quarta**.
  1308. - MongoDB: URI handling has been improved to use
  1309. database name, user and password from the URI if provided.
  1310. Contributed by **Samuel Jaillet**.
  1311. - SQLAlchemy result backend: Now ignores all result
  1312. engine options when using NullPool (Issue #1930).
  1313. - SQLAlchemy result backend: Now sets max char size to 155 to deal
  1314. with brain damaged MySQL Unicode implementation (Issue #1748).
  1315. - **General**: All Celery exceptions/warnings now inherit from common
  1316. :class:`~celery.exceptions.CeleryError`/:class:`~celery.exceptions.CeleryWarning`.
  1317. (Issue #2643).
  1318. Documentation Improvements
  1319. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  1320. Contributed by:
  1321. - Adam Chainz
  1322. - Amir Rustamzadeh
  1323. - Arthur Vuillard
  1324. - Batiste Bieler
  1325. - Berker Peksag
  1326. - Bryce Groff
  1327. - Daniel Devine
  1328. - Edward Betts
  1329. - Jason Veatch
  1330. - Jeff Widman
  1331. - Maciej Obuchowski
  1332. - Manuel Kaufmann
  1333. - Maxime Beauchemin
  1334. - Mitchel Humpherys
  1335. - Pavlo Kapyshin
  1336. - Pierre Fersing
  1337. - Rik
  1338. - Steven Sklar
  1339. - Tayfun Sen
  1340. - Wieland Hoffmann
  1341. Reorganization, Deprecations, and Removals
  1342. ==========================================
  1343. Incompatible changes
  1344. --------------------
  1345. - Prefork: Calling ``result.get()`` or joining any result from within a task
  1346. now raises :exc:`RuntimeError`.
  1347. In previous versions this would emit a warning.
  1348. - :mod:`celery.worker.consumer` is now a package, not a module.
  1349. - Module ``celery.worker.job`` renamed to :mod:`celery.worker.request`.
  1350. - Beat: ``Scheduler.Publisher``/``.publisher`` renamed to
  1351. ``.Producer``/``.producer``.
  1352. - Result: The task_name argument/attribute of :class:`@AsyncResult` was
  1353. removed.
  1354. This was historically a field used for :mod:`pickle` compatibility,
  1355. but is no longer needed.
  1356. - Backends: Arguments named ``status`` renamed to ``state``.
  1357. - Backends: ``backend.get_status()`` renamed to ``backend.get_state()``.
  1358. - Backends: ``backend.maybe_reraise()`` renamed to ``.maybe_throw()``
  1359. The promise API uses .throw(), so this change was made to make it more
  1360. consistent.
  1361. There's an alias available, so you can still use maybe_reraise until
  1362. Celery 5.0.
  1363. .. _v400-unscheduled-removals:
  1364. Unscheduled Removals
  1365. --------------------
  1366. - The experimental :mod:`celery.contrib.methods` feature has been removed,
  1367. as there were far many bugs in the implementation to be useful.
  1368. - The CentOS init-scripts have been removed.
  1369. These didn't really add any features over the generic init-scripts,
  1370. so you're encouraged to use them instead, or something like
  1371. :pypi:`supervisor`.
  1372. .. _v400-deprecations-reorg:
  1373. Reorganization Deprecations
  1374. ---------------------------
  1375. These symbols have been renamed, and while there's an alias available in this
  1376. version for backward compatibility, they will be removed in Celery 5.0, so
  1377. make sure you rename these ASAP to make sure it won't break for that release.
  1378. Chances are that you'll only use the first in this list, but you never
  1379. know:
  1380. - ``celery.utils.worker_direct`` ->
  1381. :meth:`celery.utils.nodenames.worker_direct`.
  1382. - ``celery.utils.nodename`` -> :meth:`celery.utils.nodenames.nodename`.
  1383. - ``celery.utils.anon_nodename`` ->
  1384. :meth:`celery.utils.nodenames.anon_nodename`.
  1385. - ``celery.utils.nodesplit`` -> :meth:`celery.utils.nodenames.nodesplit`.
  1386. - ``celery.utils.default_nodename`` ->
  1387. :meth:`celery.utils.nodenames.default_nodename`.
  1388. - ``celery.utils.node_format`` -> :meth:`celery.utils.nodenames.node_format`.
  1389. - ``celery.utils.host_format`` -> :meth:`celery.utils.nodenames.host_format`.
  1390. .. _v400-removals:
  1391. Scheduled Removals
  1392. ------------------
  1393. Modules
  1394. ~~~~~~~
  1395. - Module ``celery.worker.job`` has been renamed to :mod:`celery.worker.request`.
  1396. This was an internal module so shouldn't have any effect.
  1397. It's now part of the public API so must not change again.
  1398. - Module ``celery.task.trace`` has been renamed to ``celery.app.trace``
  1399. as the ``celery.task`` package is being phased out. The module
  1400. will be removed in version 5.0 so please change any import from::
  1401. from celery.task.trace import X
  1402. to::
  1403. from celery.app.trace import X
  1404. - Old compatibility aliases in the :mod:`celery.loaders` module
  1405. has been removed.
  1406. - Removed ``celery.loaders.current_loader()``, use: ``current_app.loader``
  1407. - Removed ``celery.loaders.load_settings()``, use: ``current_app.conf``
  1408. Result
  1409. ~~~~~~
  1410. - ``AsyncResult.serializable()`` and ``celery.result.from_serializable``
  1411. has been removed:
  1412. Use instead:
  1413. .. code-block:: pycon
  1414. >>> tup = result.as_tuple()
  1415. >>> from celery.result import result_from_tuple
  1416. >>> result = result_from_tuple(tup)
  1417. - Removed ``BaseAsyncResult``, use ``AsyncResult`` for instance checks
  1418. instead.
  1419. - Removed ``TaskSetResult``, use ``GroupResult`` instead.
  1420. - ``TaskSetResult.total`` -> ``len(GroupResult)``
  1421. - ``TaskSetResult.taskset_id`` -> ``GroupResult.id``
  1422. - Removed ``ResultSet.subtasks``, use ``ResultSet.results`` instead.
  1423. TaskSet
  1424. ~~~~~~~
  1425. TaskSet has been removed, as it was replaced by the ``group`` construct in
  1426. Celery 3.0.
  1427. If you have code like this:
  1428. .. code-block:: pycon
  1429. >>> from celery.task import TaskSet
  1430. >>> TaskSet(add.subtask((i, i)) for i in xrange(10)).apply_async()
  1431. You need to replace that with:
  1432. .. code-block:: pycon
  1433. >>> from celery import group
  1434. >>> group(add.s(i, i) for i in xrange(10))()
  1435. Events
  1436. ~~~~~~
  1437. - Removals for class :class:`celery.events.state.Worker`:
  1438. - ``Worker._defaults`` attribute.
  1439. Use ``{k: getattr(worker, k) for k in worker._fields}``.
  1440. - ``Worker.update_heartbeat``
  1441. Use ``Worker.event(None, timestamp, received)``
  1442. - ``Worker.on_online``
  1443. Use ``Worker.event('online', timestamp, received, fields)``
  1444. - ``Worker.on_offline``
  1445. Use ``Worker.event('offline', timestamp, received, fields)``
  1446. - ``Worker.on_heartbeat``
  1447. Use ``Worker.event('heartbeat', timestamp, received, fields)``
  1448. - Removals for class :class:`celery.events.state.Task`:
  1449. - ``Task._defaults`` attribute.
  1450. Use ``{k: getattr(task, k) for k in task._fields}``.
  1451. - ``Task.on_sent``
  1452. Use ``Worker.event('sent', timestamp, received, fields)``
  1453. - ``Task.on_received``
  1454. Use ``Task.event('received', timestamp, received, fields)``
  1455. - ``Task.on_started``
  1456. Use ``Task.event('started', timestamp, received, fields)``
  1457. - ``Task.on_failed``
  1458. Use ``Task.event('failed', timestamp, received, fields)``
  1459. - ``Task.on_retried``
  1460. Use ``Task.event('retried', timestamp, received, fields)``
  1461. - ``Task.on_succeeded``
  1462. Use ``Task.event('succeeded', timestamp, received, fields)``
  1463. - ``Task.on_revoked``
  1464. Use ``Task.event('revoked', timestamp, received, fields)``
  1465. - ``Task.on_unknown_event``
  1466. Use ``Task.event(short_type, timestamp, received, fields)``
  1467. - ``Task.update``
  1468. Use ``Task.event(short_type, timestamp, received, fields)``
  1469. - ``Task.merge``
  1470. Contact us if you need this.
  1471. Magic keyword arguments
  1472. ~~~~~~~~~~~~~~~~~~~~~~~
  1473. Support for the very old magic keyword arguments accepted by tasks is
  1474. finally removed in this version.
  1475. If you're still using these you have to rewrite any task still
  1476. using the old ``celery.decorators`` module and depending
  1477. on keyword arguments being passed to the task,
  1478. for example::
  1479. from celery.decorators import task
  1480. @task()
  1481. def add(x, y, task_id=None):
  1482. print('My task id is %r' % (task_id,))
  1483. should be rewritten into::
  1484. from celery import task
  1485. @task(bind=True)
  1486. def add(self, x, y):
  1487. print('My task id is {0.request.id}'.format(self))
  1488. Removed Settings
  1489. ----------------
  1490. The following settings have been removed, and is no longer supported:
  1491. Logging Settings
  1492. ~~~~~~~~~~~~~~~~
  1493. ===================================== =====================================
  1494. **Setting name** **Replace with**
  1495. ===================================== =====================================
  1496. ``CELERYD_LOG_LEVEL`` :option:`celery worker --loglevel`
  1497. ``CELERYD_LOG_FILE`` :option:`celery worker --logfile`
  1498. ``CELERYBEAT_LOG_LEVEL`` :option:`celery beat --loglevel`
  1499. ``CELERYBEAT_LOG_FILE`` :option:`celery beat --logfile`
  1500. ``CELERYMON_LOG_LEVEL`` celerymon is deprecated, use flower
  1501. ``CELERYMON_LOG_FILE`` celerymon is deprecated, use flower
  1502. ``CELERYMON_LOG_FORMAT`` celerymon is deprecated, use flower
  1503. ===================================== =====================================
  1504. Task Settings
  1505. ~~~~~~~~~~~~~~
  1506. ===================================== =====================================
  1507. **Setting name** **Replace with**
  1508. ===================================== =====================================
  1509. ``CELERY_CHORD_PROPAGATES`` N/A
  1510. ===================================== =====================================
  1511. Changes to internal API
  1512. -----------------------
  1513. - Module ``celery.datastructures`` renamed to :mod:`celery.utils.collections`.
  1514. - Module ``celery.utils.timeutils`` renamed to :mod:`celery.utils.time`.
  1515. - ``celery.utils.datastructures.DependencyGraph`` moved to
  1516. :mod:`celery.utils.graph`.
  1517. - ``celery.utils.jsonify`` is now :func:`celery.utils.serialization.jsonify`.
  1518. - ``celery.utils.strtobool`` is now
  1519. :func:`celery.utils.serialization.strtobool`.
  1520. - ``celery.utils.is_iterable`` has been removed.
  1521. Instead use:
  1522. .. code-block:: python
  1523. isinstance(x, collections.Iterable)
  1524. - ``celery.utils.lpmerge`` is now :func:`celery.utils.collections.lpmerge`.
  1525. - ``celery.utils.cry`` is now :func:`celery.utils.debug.cry`.
  1526. - ``celery.utils.isatty`` is now :func:`celery.platforms.isatty`.
  1527. - ``celery.utils.gen_task_name`` is now
  1528. :func:`celery.utils.imports.gen_task_name`.
  1529. - ``celery.utils.deprecated`` is now :func:`celery.utils.deprecated.Callable`
  1530. - ``celery.utils.deprecated_property`` is now
  1531. :func:`celery.utils.deprecated.Property`.
  1532. - ``celery.utils.warn_deprecated`` is now :func:`celery.utils.deprecated.warn`
  1533. .. _v400-deprecations:
  1534. Deprecation Time-line Changes
  1535. =============================
  1536. See the :ref:`deprecation-timeline`.