whatsnew-4.0.rst 55 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355135613571358135913601361136213631364136513661367136813691370137113721373137413751376137713781379138013811382138313841385138613871388138913901391139213931394139513961397139813991400140114021403140414051406140714081409141014111412141314141415141614171418141914201421142214231424142514261427142814291430143114321433143414351436143714381439144014411442144314441445144614471448144914501451145214531454145514561457145814591460146114621463146414651466146714681469147014711472147314741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532153315341535153615371538153915401541154215431544154515461547154815491550155115521553155415551556155715581559156015611562156315641565156615671568156915701571157215731574157515761577157815791580158115821583158415851586158715881589159015911592159315941595159615971598159916001601160216031604160516061607160816091610161116121613161416151616161716181619162016211622162316241625162616271628162916301631163216331634163516361637163816391640164116421643164416451646164716481649165016511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709171017111712171317141715171617171718171917201721172217231724172517261727172817291730173117321733173417351736173717381739
  1. .. _whatsnew-4.0:
  2. ===========================================
  3. What's new in Celery 4.0 (0Today8)
  4. ===========================================
  5. :Author: Ask Solem (``ask at celeryproject.org``)
  6. .. sidebar:: Change history
  7. What's new documents describe the changes in major versions,
  8. we also have a :ref:`changelog` that lists the changes in bugfix
  9. releases (0.0.x), while older series are archived under the :ref:`history`
  10. section.
  11. Celery is a simple, flexible, and reliable distributed system to
  12. process vast amounts of messages, while providing operations with
  13. the tools required to maintain such a system.
  14. It's a task queue with focus on real-time processing, while also
  15. supporting task scheduling.
  16. Celery has a large and diverse community of users and contributors,
  17. you should come join us :ref:`on IRC <irc-channel>`
  18. or :ref:`our mailing-list <mailing-list>`.
  19. To read more about Celery you should go read the :ref:`introduction <intro>`.
  20. While this version is backward compatible with previous versions
  21. it's important that you read the following section.
  22. This version is officially supported on CPython 2.7, 3.4, and 3.5.
  23. and also supported on PyPy.
  24. .. _`website`: http://celeryproject.org/
  25. .. topic:: Table of Contents
  26. Make sure you read the important notes before upgrading to this version.
  27. .. contents::
  28. :local:
  29. :depth: 2
  30. Preface
  31. =======
  32. XXX To be written
  33. Wall of Contributors
  34. --------------------
  35. XXX Needs update
  36. Aaron McMillin, Adam Renberg, Adrien Guinet, Ahmet Demir, Aitor Gómez-Goiri,
  37. Albert Wang, Alex Koshelev, Alex Rattray, Alex Williams, Alexander Koshelev,
  38. Alexander Lebedev, Alexander Oblovatniy, Alexey Kotlyarov, Ali Bozorgkhan,
  39. Alice Zoë Bevan–McGregor, Allard Hoeve, Alman One, Andrea Rabbaglietti,
  40. Andrea Rosa, Andrei Fokau, Andrew Rodionoff, Andriy Yurchuk,
  41. Aneil Mallavarapu, Areski Belaid, Artyom Koval, Ask Solem, Balthazar Rouberol,
  42. Berker Peksag, Bert Vanderbauwhede, Brian Bouterse, Chris Duryee, Chris Erway,
  43. Chris Harris, Chris Martin, Corey Farwell, Craig Jellick, Cullen Rhodes,
  44. Dallas Marlow, Daniel Wallace, Danilo Bargen, Davanum Srinivas, Dave Smith,
  45. David Baumgold, David Harrigan, David Pravec, Dennis Brakhane, Derek Anderson,
  46. Dmitry Malinovsky, Dudás Ádám, Dustin J. Mitchell, Ed Morley, Fatih Sucu,
  47. Feanil Patel, Felix Schwarz, Fernando Rocha, Flavio Grossi, Frantisek Holop,
  48. Gao Jiangmiao, Gerald Manipon, Gilles Dartiguelongue, Gino Ledesma,
  49. Hank John, Hogni Gylfason, Ilya Georgievsky, Ionel Cristian Mărieș,
  50. James Pulec, Jared Lewis, Jason Veatch, Jasper Bryant-Greene, Jeremy Tillman,
  51. Jocelyn Delalande, Joe Jevnik, John Anderson, John Kirkham, John Whitlock,
  52. Joshua Harlow, Juan Rossi, Justin Patrin, Kai Groner, Kevin Harvey,
  53. Konstantinos Koukopoulos, Kouhei Maeda, Kracekumar Ramaraju,
  54. Krzysztof Bujniewicz, Latitia M. Haskins, Len Buckens, Lorenzo Mancini,
  55. Lucas Wiman, Luke Pomfrey, Marcio Ribeiro, Marin Atanasov Nikolov,
  56. Mark Parncutt, Maxime Vdb, Mher Movsisyan, Michael (:github_user:`michael-k`),
  57. Michael Duane Mooring, Michael Permana, Mickaël Penhard, Mike Attwood,
  58. Morton Fox, Môshe van der Sterre, Nat Williams, Nathan Van Gheem, Nik Nyby,
  59. Omer Katz, Omer Korner, Ori Hoch, Paul Pearce, Paulo Bu, Philip Garnero,
  60. Piotr Maślanka, Radek Czajka, Raghuram Srinivasan, Randy Barlow,
  61. Rodolfo Carvalho, Roger Hu, Rongze Zhu, Ross Deane, Ryan Luckie,
  62. Rémy Greinhofer, Samuel Jaillet, Sergey Azovskov, Sergey Tikhonov,
  63. Seungha Kim, Steve Peak, Sukrit Khera, Tadej Janež, Tewfik Sadaoui,
  64. Thomas French, Thomas Grainger, Tobias Schottdorf, Tocho Tochev,
  65. Valentyn Klindukh, Vic Kumar, Vladimir Bolshakov, Vladimir Gorbunov,
  66. Wayne Chang, Wil Langford, Will Thompson, William King, Yury Selivanov,
  67. Zoran Pavlovic, 許邱翔, :github_user:`allenling`, :github_user:`bee-keeper`,
  68. :github_user:`ffeast`, :github_user:`flyingfoxlee`, :github_user:`gdw2`,
  69. :github_user:`gitaarik`, :github_user:`hankjin`, :github_user:`m-vdb`,
  70. :github_user:`mdk`, :github_user:`nokrik`, :github_user:`ocean1`,
  71. :github_user:`orlo666`, :github_user:`raducc`, :github_user:`wanglei`,
  72. :github_user:`worldexception`.
  73. .. _v400-important:
  74. Important Notes
  75. ===============
  76. Dropped support for Python 2.6
  77. ------------------------------
  78. Celery now requires Python 2.7 or later,
  79. and also drops support for Python 3.3 so supported versions are:
  80. - CPython 2.7
  81. - CPython 3.4
  82. - CPython 3.5
  83. - PyPy 5.3 (``pypy2``)
  84. - PyPy 2.4 (``pypy3``)
  85. Last major version to support Python 2
  86. --------------------------------------
  87. Starting from Celery 5.0 only Python 3.5+ will be supported.
  88. To make sure you're not affected by this change you should pin
  89. the Celery version in your requirements file, either to a specific
  90. version: ``celery==4.0.0``, or a range: ``celery>=4.0,<5.0``.
  91. Dropping support for Python 2 will enable us to remove massive
  92. amounts of compatibility code, and going with Python 3.5 allows
  93. us to take advantage of typing, async/await, asyncio, and similar
  94. concepts there's no alternative for in older versions.
  95. Celery 4.x will continue to work on Python 2.7, 3.4, 3.5; just as Celery 3.x
  96. still works on Python 2.6.
  97. Django support
  98. --------------
  99. Celery now supports Django 1.8 and newer versions.
  100. Removed features
  101. ----------------
  102. - Microsoft Windows is no longer supported.
  103. - Jython is no longer supported.
  104. Features removed for simplicity
  105. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  106. - Webhook task machinery (``celery.task.http``) has been removed.
  107. Nowadays it's easy to use the :pypi:`requests` module to write
  108. webhook tasks manually. We would love to use requests but we
  109. are simply unable to as there's a very vocal 'anti-dependency'
  110. mob in the Python community
  111. If you need backwards compatibility
  112. you can simply copy + paste the 3.1 version of the module and make sure
  113. it's imported by the worker:
  114. https://github.com/celery/celery/blob/3.1/celery/task/http.py
  115. - Task no longer sends error emails.
  116. This also removes support for ``app.mail_admins``, and any functionality
  117. related to sending emails.
  118. - ``celery.contrib.batches`` has been removed.
  119. This was an experimental feature, so not covered by our deprecation
  120. timeline guarantee.
  121. Features removed for lack of funding
  122. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  123. We announced with the 3.1 release that some transports were
  124. moved to experimental status, and that there'd be no official
  125. support for the transports, citing a lack of resources.
  126. As this subtle hint for the need of funding failed
  127. we've removed them completely, breaking backwards compatibility.
  128. - Using MongoDB as a broker is no longer supported.
  129. - Using the Django ORM as a broker is no longer supported.
  130. - Using SQLAlchemy as a broker is no longer supported.
  131. - Using CouchDB as a broker is no longer supported.
  132. - Using IronMQ as a broker is no longer supported.
  133. - Using Beanstalk as a broker is no longer supported.
  134. In addition some features have been removed completely so that
  135. attempting to use them will raise an exception:
  136. - The ``--autoreload`` feature has been removed.
  137. This was an experimental feature, and not covered by our deprecation
  138. timeline guarantee. The flag is removed completely so the worker
  139. will crash at startup when present. Luckily this
  140. flag isn't used in production systems.
  141. - The ``--autoscale`` feature has been removed.
  142. This flag is only used by companies to save money, but had
  143. bugs either nobody cared to work on, or sponsor a few hours of work to get it fixed.
  144. The flag has been removed completely, so you must remove this command-line
  145. argument or your workers will crash.
  146. - The experimental ``threads`` pool is no longer supported and has been removed.
  147. - The force_execv feature is no longer supported.
  148. The ``celery worker`` command now ignores the ``--no-execv``,
  149. ``--force-execv``, and the ``CELERYD_FORCE_EXECV`` setting.
  150. This flag will be removed completely in 5.0 and the worker
  151. will raise an error.
  152. - The old legacy "amqp" result backend has been deprecated, and will
  153. be removed in Celery 5.0.
  154. Please use the ``rpc`` result backend for RPC-style calls, and a
  155. persistent result backend for multi-consumer results.
  156. **Now to the good news**...
  157. New Task Message Protocol
  158. -------------------------
  159. .. :sha:`e71652d384b1b5df2a4e6145df9f0efb456bc71c`
  160. This version introduces a brand new task message protocol,
  161. the first major change to the protocol since the beginning of the project.
  162. The new protocol is enabled by default in this version and since the new
  163. version isn't backwards compatible you have to be careful when upgrading.
  164. The 3.1.24 version was released to add compatibility with the new protocol
  165. so the easiest way to upgrade is to upgrade to that version first, then
  166. upgrade to 4.0 in a second deployment.
  167. If you wish to keep using the old protocol you may also configure
  168. the protocol version number used:
  169. .. code-block:: python
  170. app = Celery()
  171. app.conf.task_protocol = 1
  172. Read more about the features available in the new protocol in the news
  173. section found later in this document.
  174. Lowercase setting names
  175. -----------------------
  176. In the pursuit of beauty all settings are now renamed to be in all
  177. lowercase and some setting names have been renamed for consistency.
  178. This change is fully backwards compatible so you can still use the uppercase
  179. setting names, but we would like you to upgrade as soon as possible and
  180. you can this automatically using the :program:`celery upgrade settings`
  181. command:
  182. .. code-block:: console
  183. $ celery upgrade settings proj/settings.py
  184. This command will modify your module in-place to use the new lower-case
  185. names (if you want uppercase with a "``CELERY``" prefix see block below),
  186. and save a backup in :file:`proj/settings.py.orig`.
  187. .. admonition:: For Django users and others who want to keep uppercase names
  188. If you're loading Celery configuration from the Django settings module
  189. then you'll want to keep using the uppercase names.
  190. You also want to use a ``CELERY_`` prefix so that no Celery settings
  191. collide with Django settings used by other apps.
  192. To do this, you'll first need to convert your settings file
  193. to use the new consistent naming scheme, and add the prefix to all
  194. Celery related settings:
  195. .. code-block:: console
  196. $ celery upgrade settings --django proj/settings.py
  197. After upgrading the settings file, you need to set the prefix explicitly
  198. in your ``proj/celery.py`` module:
  199. .. code-block:: python
  200. app.config_from_object('django.conf:settings', namespace='CELERY')
  201. You can find the most up to date Django Celery integration example
  202. here: :ref:`django-first-steps`.
  203. Note that this will also add a prefix to settings that didn't previously
  204. have one, like ``BROKER_URL``.
  205. Luckily you don't have to manually change the files, as
  206. the :program:`celery upgrade settings --django` program should do the
  207. right thing.
  208. The loader will try to detect if your configuration is using the new format,
  209. and act accordingly, but this also means you're not allowed to mix and
  210. match new and old setting names, that's unless you provide a value for both
  211. alternatives.
  212. The major difference between previous versions, apart from the lower case
  213. names, are the renaming of some prefixes, like ``celerybeat_`` to ``beat_``,
  214. ``celeryd_`` to ``worker_``.
  215. The ``celery_`` prefix has also been removed, and task related settings
  216. from this name-space is now prefixed by ``task_``, worker related settings
  217. with ``worker_``.
  218. Apart from this most of the settings will be the same in lowercase, apart from
  219. a few special ones:
  220. ===================================== ==========================================================
  221. **Setting name** **Replace with**
  222. ===================================== ==========================================================
  223. ``CELERY_MAX_CACHED_RESULTS`` :setting:`result_cache_max`
  224. ``CELERY_MESSAGE_COMPRESSION`` :setting:`result_compression`/:setting:`task_compression`.
  225. ``CELERY_TASK_RESULT_EXPIRES`` :setting:`result_expires`
  226. ``CELERY_RESULT_DBURI`` :setting:`sqlalchemy_dburi`
  227. ``CELERY_RESULT_ENGINE_OPTIONS`` :setting:`sqlalchemy_engine_options`
  228. ``-*-_DB_SHORT_LIVED_SESSIONS`` :setting:`sqlalchemy_short_lived_sessions`
  229. ``CELERY_RESULT_DB_TABLE_NAMES`` :setting:`sqlalchemy_db_names`
  230. ``CELERY_ACKS_LATE`` :setting:`task_acks_late`
  231. ``CELERY_ALWAYS_EAGER`` :setting:`task_always_eager`
  232. ``CELERY_ANNOTATIONS`` :setting:`task_annotations`
  233. ``CELERY_MESSAGE_COMPRESSION`` :setting:`task_compression`
  234. ``CELERY_CREATE_MISSING_QUEUES`` :setting:`task_create_missing_queues`
  235. ``CELERY_DEFAULT_DELIVERY_MODE`` :setting:`task_default_delivery_mode`
  236. ``CELERY_DEFAULT_EXCHANGE`` :setting:`task_default_exchange`
  237. ``CELERY_DEFAULT_EXCHANGE_TYPE`` :setting:`task_default_exchange_type`
  238. ``CELERY_DEFAULT_QUEUE`` :setting:`task_default_queue`
  239. ``CELERY_DEFAULT_RATE_LIMIT`` :setting:`task_default_rate_limit`
  240. ``CELERY_DEFAULT_ROUTING_KEY`` :setting:`task_default_routing_key`
  241. ``-"-_EAGER_PROPAGATES_EXCEPTIONS`` :setting:`task_eager_propagates`
  242. ``CELERY_IGNORE_RESULT`` :setting:`task_ignore_result`
  243. ``CELERY_TASK_PUBLISH_RETRY`` :setting:`task_publish_retry`
  244. ``CELERY_TASK_PUBLISH_RETRY_POLICY`` :setting:`task_publish_retry_policy`
  245. ``CELERY_QUEUES`` :setting:`task_queues`
  246. ``CELERY_ROUTES`` :setting:`task_routes`
  247. ``CELERY_SEND_TASK_SENT_EVENT`` :setting:`task_send_sent_event`
  248. ``CELERY_TASK_SERIALIZER`` :setting:`task_serializer`
  249. ``CELERYD_TASK_SOFT_TIME_LIMIT`` :setting:`task_soft_time_limit`
  250. ``CELERYD_TASK_TIME_LIMIT`` :setting:`task_time_limit`
  251. ``CELERY_TRACK_STARTED`` :setting:`task_track_started`
  252. ``CELERY_DISABLE_RATE_LIMITS`` :setting:`worker_disable_rate_limits`
  253. ``CELERY_ENABLE_REMOTE_CONTROL`` :setting:`worker_enable_remote_control`
  254. ``CELERYD_SEND_EVENTS`` :setting:`worker_send_task_events`
  255. ===================================== ==========================================================
  256. You can see a full table of the changes in :ref:`conf-old-settings-map`.
  257. JSON is now the default serializer
  258. ----------------------------------
  259. The time has finally come to end the reign of :mod:`pickle` as the default
  260. serialization mechanism, and json is the default serializer starting from this
  261. version.
  262. This change was :ref:`announced with the release of Celery 3.1
  263. <last-version-to-enable-pickle>`.
  264. If you're still depending on :mod:`pickle` being the default serializer,
  265. then you have to configure your app before upgrading to 4.0:
  266. .. code-block:: python
  267. task_serializer = 'pickle'
  268. result_serializer = 'pickle'
  269. accept_content = {'pickle'}
  270. The Task base class no longer automatically register tasks
  271. ----------------------------------------------------------
  272. The :class:`~@Task` class is no longer using a special meta-class
  273. that automatically registers the task in the task registry.
  274. Instead this is now handled by the :class:`@task` decorators.
  275. If you're still using class based tasks, then you need to register
  276. these manually:
  277. .. code-block:: python
  278. class CustomTask(Task):
  279. def run(self):
  280. print('running')
  281. app.tasks.register(CustomTask())
  282. The best practice is to use custom task classes only for overriding
  283. general behavior, and then using the task decorator to realize the task:
  284. .. code-block:: python
  285. @app.task(bind=True, base=CustomTask)
  286. def custom(self):
  287. print('running')
  288. This change also means the ``abstract`` attribute of the task
  289. no longer has any effect.
  290. Task argument checking
  291. ----------------------
  292. The arguments of the task are now verified when calling the task,
  293. even asynchronously:
  294. .. code-block:: pycon
  295. >>> @app.task
  296. ... def add(x, y):
  297. ... return x + y
  298. >>> add.delay(8, 8)
  299. <AsyncResult: f59d71ca-1549-43e0-be41-4e8821a83c0c>
  300. >>> add.delay(8)
  301. Traceback (most recent call last):
  302. File "<stdin>", line 1, in <module>
  303. File "celery/app/task.py", line 376, in delay
  304. return self.apply_async(args, kwargs)
  305. File "celery/app/task.py", line 485, in apply_async
  306. check_arguments(*(args or ()), **(kwargs or {}))
  307. TypeError: add() takes exactly 2 arguments (1 given)
  308. Redis Events not backward compatible
  309. ------------------------------------
  310. The Redis ``fanout_patterns`` and ``fanout_prefix`` transport
  311. options are now enabled by default.
  312. Workers/monitors without these flags enabled won't be able to
  313. see workers with this flag disabled. They can still execute tasks,
  314. but they cannot receive each others monitoring messages.
  315. You can upgrade in a backward compatible manner by first configuring
  316. your 3.1 workers and monitors to enable the settings, before the final
  317. upgrade to 4.0:
  318. .. code-block:: python
  319. BROKER_TRANSPORT_OPTIONS = {
  320. 'fanout_patterns': True,
  321. 'fanout_prefix': True,
  322. }
  323. Django: Auto-discover now supports Django app configurations
  324. ------------------------------------------------------------
  325. The ``autodiscover_tasks()`` function can now be called without arguments,
  326. and the Django handler will automatically find your installed apps:
  327. .. code-block:: python
  328. app.autodiscover_tasks()
  329. The Django integration :ref:`example in the documentation
  330. <django-first-steps>` has been updated to use the argument-less call.
  331. This also ensures compatibility with the new, ehm, ``AppConfig`` stuff
  332. introduced in recent Django versions.
  333. Worker direct queues no longer use auto-delete
  334. ----------------------------------------------
  335. Workers/clients running 4.0 will no longer be able to send
  336. worker direct messages to workers running older versions, and vice versa.
  337. If you're relying on worker direct messages you should upgrade
  338. your 3.x workers and clients to use the new routing settings first,
  339. by replacing :func:`celery.utils.worker_direct` with this implementation:
  340. .. code-block:: python
  341. from kombu import Exchange, Queue
  342. worker_direct_exchange = Exchange('C.dq2')
  343. def worker_direct(hostname):
  344. return Queue(
  345. '{hostname}.dq2'.format(hostname),
  346. exchange=worker_direct_exchange,
  347. routing_key=hostname,
  348. )
  349. This feature closed Issue #2492.
  350. Old command-line programs removed
  351. ---------------------------------
  352. Installing Celery will no longer install the ``celeryd``,
  353. ``celerybeat`` and ``celeryd-multi`` programs.
  354. This was announced with the release of Celery 3.1, but you may still
  355. have scripts pointing to the old names so make sure you update these
  356. to use the new umbrella command:
  357. +-------------------+--------------+-------------------------------------+
  358. | Program | New Status | Replacement |
  359. +===================+==============+=====================================+
  360. | ``celeryd`` | **REMOVED** | :program:`celery worker` |
  361. +-------------------+--------------+-------------------------------------+
  362. | ``celerybeat`` | **REMOVED** | :program:`celery beat` |
  363. +-------------------+--------------+-------------------------------------+
  364. | ``celeryd-multi`` | **REMOVED** | :program:`celery multi` |
  365. +-------------------+--------------+-------------------------------------+
  366. .. _v400-news:
  367. News
  368. ====
  369. New protocol highlights
  370. -----------------------
  371. The new protocol fixes many problems with the old one, and enables
  372. some long-requested features:
  373. - Most of the data are now sent as message headers, instead of being
  374. serialized with the message body.
  375. In version 1 of the protocol the worker always had to deserialize
  376. the message to be able to read task meta-data like the task id,
  377. name, etc. This also meant that the worker was forced to double-decode
  378. the data, first deserializing the message on receipt, serializing
  379. the message again to send to child process, then finally the child process
  380. deserializes the message again.
  381. Keeping the meta-data fields in the message headers means the worker
  382. doesn't actually have to decode the payload before delivering
  383. the task to the child process, and also that it's now possible
  384. for the worker to reroute a task written in a language different
  385. from Python to a different worker.
  386. - A new ``lang`` message header can be used to specify the programming
  387. language the task is written in.
  388. - Worker stores results for internal errors like ``ContentDisallowed``,
  389. and other deserialization errors.
  390. - Worker stores results and sends monitoring events for unregistered
  391. task errors.
  392. - Worker calls callbacks/errbacks even when the result is sent by the
  393. parent process (e.g., :exc:`WorkerLostError` when a child process
  394. terminates, deserialization errors, unregistered tasks).
  395. - A new ``origin`` header contains information about the process sending
  396. the task (worker node-name, or PID and host-name information).
  397. - A new ``shadow`` header allows you to modify the task name used in logs.
  398. This is useful for dispatch like patterns, like a task that calls
  399. any function using pickle (don't do this at home):
  400. .. code-block:: python
  401. from celery import Task
  402. from celery.utils.imports import qualname
  403. class call_as_task(Task):
  404. def shadow_name(self, args, kwargs, options):
  405. return 'call_as_task:{0}'.format(qualname(args[0]))
  406. def run(self, fun, *args, **kwargs):
  407. return fun(*args, **kwargs)
  408. call_as_task = app.tasks.register(call_as_task())
  409. - New ``argsrepr`` and ``kwargsrepr`` fields contain textual representations
  410. of the task arguments (possibly truncated) for use in logs, monitors, etc.
  411. This means the worker doesn't have to deserialize the message payload
  412. to display the task arguments for informational purposes.
  413. - Chains now use a dedicated ``chain`` field enabling support for chains
  414. of thousands and more tasks.
  415. - New ``parent_id`` and ``root_id`` headers adds information about
  416. a tasks relationship with other tasks.
  417. - ``parent_id`` is the task id of the task that called this task
  418. - ``root_id`` is the first task in the work-flow.
  419. These fields can be used to improve monitors like flower to group
  420. related messages together (like chains, groups, chords, complete
  421. work-flows, etc).
  422. - ``app.TaskProducer`` replaced by :meth:`@amqp.create_task_message` and
  423. :meth:`@amqp.send_task_message`.
  424. Dividing the responsibilities into creating and sending means that
  425. people who want to send messages using a Python AMQP client directly,
  426. doesn't have to implement the protocol.
  427. The :meth:`@amqp.create_task_message` method calls either
  428. :meth:`@amqp.as_task_v2`, or :meth:`@amqp.as_task_v1` depending
  429. on the configured task protocol, and returns a special
  430. :class:`~celery.app.amqp.task_message` tuple containing the
  431. headers, properties and body of the task message.
  432. .. seealso::
  433. The new task protocol is documented in full here:
  434. :ref:`message-protocol-task-v2`.
  435. Prefork: Tasks now log from the child process
  436. ---------------------------------------------
  437. Logging of task success/failure now happens from the child process
  438. executing the task. As a result logging utilities,
  439. like Sentry can get full information about tasks, including
  440. variables in the traceback stack.
  441. Prefork: One log-file per child process
  442. ---------------------------------------
  443. Init-scrips and :program:`celery multi` now uses the `%I` log file format
  444. option (e.g., :file:`/var/log/celery/%n%I.log`).
  445. This change was necessary to ensure each child
  446. process has a separate log file after moving task logging
  447. to the child process, as multiple processes writing to the same
  448. log file can cause corruption.
  449. You're encouraged to upgrade your init-scripts and
  450. :program:`celery multi` arguments to use this new option.
  451. Configure broker URL for read/write separately
  452. ----------------------------------------------
  453. New :setting:`broker_read_url` and :setting:`broker_write_url` settings
  454. have been added so that separate broker URLs can be provided
  455. for connections used for consuming/publishing.
  456. In addition to the configuration options, two new methods have been
  457. added the app API:
  458. - ``app.connection_for_read()``
  459. - ``app.connection_for_write()``
  460. These should now be used in place of ``app.connection()`` to specify
  461. the intent of the required connection.
  462. .. note::
  463. Two connection pools are available: ``app.pool`` (read), and
  464. ``app.producer_pool`` (write). The latter doesn't actually give connections
  465. but full :class:`kombu.Producer` instances.
  466. .. code-block:: python
  467. def publish_some_message(app, producer=None):
  468. with app.producer_or_acquire(producer) as producer:
  469. ...
  470. def consume_messages(app, connection=None):
  471. with app.connection_or_acquire(connection) as connection:
  472. ...
  473. Canvas Refactor
  474. ---------------
  475. The canvas/work-flow implementation have been heavily refactored
  476. to fix some long outstanding issues.
  477. .. :sha:`d79dcd8e82c5e41f39abd07ffed81ca58052bcd2`
  478. .. :sha:`1e9dd26592eb2b93f1cb16deb771cfc65ab79612`
  479. .. :sha:`e442df61b2ff1fe855881c1e2ff9acc970090f54`
  480. .. :sha:`0673da5c09ac22bdd49ba811c470b73a036ee776`
  481. - Error callbacks can now take real exception and traceback instances
  482. (Issue #2538).
  483. .. code-block:: pycon
  484. >>> add.s(2, 2).on_error(log_error.s()).delay()
  485. Where ``log_error`` could be defined as:
  486. .. code-block:: python
  487. @app.task
  488. def log_error(request, exc, traceback):
  489. with open(os.path.join('/var/errors', request.id), 'a') as fh:
  490. print('--\n\n{0} {1} {2}'.format(
  491. task_id, exc, traceback), file=fh)
  492. See :ref:`guide-canvas` for more examples.
  493. - Now unrolls groups within groups into a single group (Issue #1509).
  494. - chunks/map/starmap tasks now routes based on the target task
  495. - chords and chains can now be immutable.
  496. - Fixed bug where serialized signatures weren't converted back into
  497. signatures (Issue #2078)
  498. Fix contributed by **Ross Deane**.
  499. - Fixed problem where chains and groups didn't work when using JSON
  500. serialization (Issue #2076).
  501. Fix contributed by **Ross Deane**.
  502. - Creating a chord no longer results in multiple values for keyword
  503. argument 'task_id' (Issue #2225).
  504. Fix contributed by **Aneil Mallavarapu**.
  505. - Fixed issue where the wrong result is returned when a chain
  506. contains a chord as the penultimate task.
  507. Fix contributed by **Aneil Mallavarapu**.
  508. - Special case of ``group(A.s() | group(B.s() | C.s()))`` now works.
  509. - Chain: Fixed bug with incorrect id set when a subtask is also a chain.
  510. - ``group | group`` is now flattened into a single group (Issue #2573).
  511. - Fixed issue where ``group | task`` wasn't upgrading correctly
  512. to chord (Issue #2922).
  513. Amazon SQS transport now officially supported
  514. ---------------------------------------------
  515. The SQS broker transport has been rewritten to use async I/O and as such
  516. joins RabbitMQ and Redis as officially supported transports.
  517. The new implementation also takes advantage of long polling,
  518. and closes several issues related to using SQS as a broker.
  519. This work was sponsored by Nextdoor.
  520. Apache QPid transport now officially supported
  521. ----------------------------------------------
  522. Contributed by **Brian Bouterse**.
  523. Schedule tasks based on sunrise, sunset, dawn and dusk
  524. ------------------------------------------------------
  525. See :ref:`beat-solar` for more information.
  526. Contributed by **Mark Parncutt**.
  527. New API for configuring periodic tasks
  528. --------------------------------------
  529. This new API enables you to use signatures when defining periodic tasks,
  530. removing the chance of mistyping task names.
  531. An example of the new API is :ref:`here <beat-entries>`.
  532. .. :sha:`bc18d0859c1570f5eb59f5a969d1d32c63af764b`
  533. .. :sha:`132d8d94d38f4050db876f56a841d5a5e487b25b`
  534. Handling task connection errors
  535. -------------------------------
  536. Connection related errors occuring while sending a task is now re-raised
  537. as a :exc:`kombu.exceptions.OperationalError` error:
  538. .. code-block:: pycon
  539. >>> try:
  540. ... add.delay(2, 2)
  541. ... except add.OperationalError as exc:
  542. ... print('Could not send task %r: %r' % (add, exc))
  543. See :ref:`calling-connection-errors` for more information.
  544. RabbitMQ priority queue support
  545. -------------------------------
  546. See :ref:`routing-options-rabbitmq-priorities` for more information.
  547. Contributed by **Gerald Manipon**.
  548. Prefork: Limit child process resident memory size
  549. -------------------------------------------------
  550. .. :sha:`5cae0e754128750a893524dcba4ae030c414de33`
  551. You can now limit the maximum amount of memory allocated per prefork
  552. pool child process by setting the worker
  553. :option:`--max-memory-per-child <celery worker --max-memory-per-child>` option,
  554. or the :setting:`worker_max_memory_per_child` setting.
  555. The limit is for RSS/resident memory size and is specified in kilobytes.
  556. A child process having exceeded the limit will be terminated and replaced
  557. with a new process after the currently executing task returns.
  558. See :ref:`worker-max-memory-per-child` for more information.
  559. Contributed by **Dave Smith**.
  560. Redis: Result backend optimizations
  561. -----------------------------------
  562. RPC is now using pub/sub for streaming task results
  563. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  564. Calling ``result.get()`` when using the Redis result backend
  565. used to be extremely expensive as it was using polling to wait
  566. for the result to become available. A default polling
  567. interval of 0.5 seconds didn't help performance, but was
  568. necessary to avoid a spin loop.
  569. The new implementation is using Redis Pub/Sub mechanisms to
  570. publish and retrieve results immediately, greatly improving
  571. task round-trip times.
  572. Contributed by **Yaroslav Zhavoronkov** and **Ask Solem**.
  573. New optimized chord join implementation
  574. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  575. This was an experimental feature introduced in Celery 3.1,
  576. that could only be enabled by adding ``?new_join=1`` to the
  577. result backend URL configuration.
  578. We feel that the implementation has been tested thoroughly enough
  579. to be considered stable and enabled by default.
  580. The new implementation greatly reduces the overhead of chords,
  581. and especially with larger chords the performance benefit can be massive.
  582. New Riak result backend Introduced
  583. ----------------------------------
  584. See :ref:`conf-riak-result-backend` for more information.
  585. Contributed by **Gilles Dartiguelongue**, **Alman One** and **NoKriK**.
  586. New CouchDB result backend introduced
  587. -------------------------------------
  588. See :ref:`conf-couchdb-result-backend` for more information.
  589. Contributed by **Nathan Van Gheem**.
  590. New Consul result backend introduced
  591. ------------------------------------
  592. Add support for Consul as a backend using the Key/Value store of Consul.
  593. Consul has an HTTP API where through you can store keys with their values.
  594. The backend extends KeyValueStoreBackend and implements most of the methods.
  595. Mainly to set, get and remove objects.
  596. This allows Celery to store Task results in the K/V store of Consul.
  597. Consul also allows to set a TTL on keys using the Sessions from Consul. This way
  598. the backend supports auto expiry of Task results.
  599. For more information on Consul visit http://consul.io/
  600. The backend uses :pypi:`python-consul` for talking to the HTTP API.
  601. This package is fully Python 3 compliant just as this backend is:
  602. .. code-block:: console
  603. $ pip install python-consul
  604. That installs the required package to talk to Consul's HTTP API from Python.
  605. Contributed by **Wido den Hollander**.
  606. Brand new Cassandra result backend
  607. ----------------------------------
  608. A brand new Cassandra backend utilizing the new :pypi:`cassandra-driver`
  609. library is replacing the old result backend using the older
  610. :pypi:`pycassa` library.
  611. See :ref:`conf-cassandra-result-backend` for more information.
  612. .. # XXX What changed?
  613. New Elasticsearch result backend introduced
  614. -------------------------------------------
  615. See :ref:`conf-elasticsearch-result-backend` for more information.
  616. Contributed by **Ahmet Demir**.
  617. New File-system result backend introduced
  618. -----------------------------------------
  619. See :ref:`conf-filesystem-result-backend` for more information.
  620. Contributed by **Môshe van der Sterre**.
  621. Event Batching
  622. --------------
  623. Events are now buffered in the worker and sent as a list, reducing
  624. the overhead required to send monitoring events.
  625. For authors of custom event monitors there will be no action
  626. required as long as you're using the Python Celery
  627. helpers (:class:`~@events.Receiver`) to implement your monitor.
  628. However, if you're parsing raw event messages you must now account
  629. for batched event messages, as they differ from normal event messages
  630. in the following way:
  631. - The routing key for a batch of event messages will be set to
  632. ``<event-group>.multi`` where the only batched event group
  633. is currently ``task`` (giving a routing key of ``task.multi``).
  634. - The message body will be a serialized list-of-dictionaries instead
  635. of a dictionary. Each item in the list can be regarded
  636. as a normal event message body.
  637. .. :sha:`03399b4d7c26fb593e61acf34f111b66b340ba4e`
  638. ``Task.replace``
  639. ----------------
  640. Task.replace changed, removes Task.replace_in_chord.
  641. The two methods had almost the same functionality, but the old
  642. ``Task.replace`` would force the new task to inherit the
  643. callbacks/errbacks of the existing task.
  644. If you replace a node in a tree, then you wouldn't expect the new node to
  645. inherit the children of the old node, so this seems like unexpected
  646. behavior.
  647. So ``self.replace(sig)`` now works for any task, in addition ``sig`` can now
  648. be a group.
  649. Groups are automatically converted to a chord, where the callback
  650. will "accumulate" the results of the group tasks.
  651. A new built-in task (`celery.accumulate` was added for this purpose)
  652. Closes #817
  653. Optimized Beat implementation
  654. -----------------------------
  655. The :program:`celery beat` implementation has been optimized
  656. for millions of periodic tasks by using a heap to schedule entries.
  657. Contributed by **Ask Solem** and **Alexander Koshelev**.
  658. Task Auto-retry Decorator
  659. -------------------------
  660. Writing custom retry handling for exception events is so common
  661. that we now have built-in support for it.
  662. For this a new ``autoretry_for`` argument is now supported by
  663. the task decorators, where you can specify a tuple of exceptions
  664. to automatically retry for.
  665. See :ref:`task-autoretry` for more information.
  666. Contributed by **Dmitry Malinovsky**.
  667. .. :sha:`75246714dd11e6c463b9dc67f4311690643bff24`
  668. Remote Task Tracebacks
  669. ----------------------
  670. The new :setting:`task_remote_tracebacks` will make task tracebacks more
  671. useful by injecting the stack of the remote worker.
  672. This feature requires the additional :pypi:`tblib` library.
  673. Contributed by **Ionel Cristian Mărieș**.
  674. Async Result API
  675. ----------------
  676. eventlet/gevent drainers, promises, BLA BLA
  677. Closed issue #2529.
  678. RPC Result Backend matured
  679. --------------------------
  680. Lots of bugs in the previously experimental RPC result backend have been fixed
  681. and we now consider it production ready.
  682. Contributed by **Ask Solem**, **Morris Tweed**.
  683. New Task Router API
  684. -------------------
  685. The :setting:`task_routes` setting can now hold functions, and map routes
  686. now support glob patterns and regexes.
  687. Instead of using router classes you can now simply define a function:
  688. .. code-block:: python
  689. def route_for_task(name, args, kwargs, options, task=None, **kwargs):
  690. from proj import tasks
  691. if name == tasks.add.name:
  692. return {'queue': 'hipri'}
  693. If you don't need the arguments you can use start arguments, just make
  694. sure you always also accept star arguments so that we have the ability
  695. to add more features in the future:
  696. .. code-block:: python
  697. def route_for_task(name, *args, **kwargs):
  698. from proj import tasks
  699. if name == tasks.add.name:
  700. return {'queue': 'hipri', 'priority': 9}
  701. Both the ``options`` argument and the new ``task`` keyword argument
  702. are new to the function-style routers, and will make it easier to write
  703. routers based on execution options, or properties of the task.
  704. The optional ``task`` keyword argument won't be set if a task is called
  705. by name using :meth:`@send_task`.
  706. For more examples, including using glob/regexes in routers please see
  707. :setting:`task_routes` and :ref:`routing-automatic`.
  708. In Other News
  709. -------------
  710. Requirements
  711. ~~~~~~~~~~~~
  712. - Now depends on :ref:`Kombu 4.0 <kombu:version-4.0>`.
  713. - Now depends on :pypi:`billiard` version 3.5.
  714. - No longer depends on :pypi:`anyjson`. Good-bye old friend :(
  715. Tasks
  716. ~~~~~
  717. - The "anon-exchange" is now used for simple name-name direct routing.
  718. This increases performance as it completely bypasses the routing table,
  719. in addition it also improves reliability for the Redis broker transport.
  720. - An empty ResultSet now evaluates to True.
  721. Fix contributed by **Colin McIntosh**.
  722. - New :setting:`task_reject_on_worker_lost` setting, and
  723. :attr:`~@Task.reject_on_worker_lost` task attribute decides what happens
  724. when the child worker process executing a late ack task is terminated.
  725. Contributed by **Michael Permana**.
  726. - ``Task.subtask`` renamed to ``Task.signature`` with alias.
  727. - ``Task.subtask_from_request`` renamed to
  728. ``Task.signature_from_request`` with alias.
  729. - The ``delivery_mode`` attribute for :class:`kombu.Queue` is now
  730. respected (Issue #1953).
  731. - Routes in :setting:`task-routes` can now specify a
  732. :class:`~kombu.Queue` instance directly.
  733. Example:
  734. .. code-block:: python
  735. task_routes = {'proj.tasks.add': {'queue': Queue('add')}}
  736. - ``AsyncResult`` now raises :exc:`ValueError` if task_id is None.
  737. (Issue #1996).
  738. - Retried tasks didn't forward expires setting (Issue #3297).
  739. - ``result.get()`` now supports an ``on_message`` argument to set a
  740. callback to be called for every message received.
  741. - New abstract classes added:
  742. - :class:`~celery.utils.abstract.CallableTask`
  743. Looks like a task.
  744. - :class:`~celery.utils.abstract.CallableSignature`
  745. Looks like a task signature.
  746. - ``Task.replace`` now properly forwards callbacks (Issue #2722).
  747. Fix contributed by **Nicolas Unravel**.
  748. - ``Task.replace``: Append to chain/chord (Closes #3232)
  749. Fixed issue #3232, adding the signature to the chain (if there's any).
  750. Fixed the chord suppress if the given signature contains one.
  751. Fix contributed by :github_user:`honux`.
  752. - Task retry now also throws in eager mode.
  753. Fix contributed by **Feanil Patel**.
  754. Beat
  755. ~~~~
  756. - Fixed crontab infinite loop with invalid date.
  757. When occurrence can never be reached (example, April, 31th), trying
  758. to reach the next occurrence would trigger an infinite loop.
  759. Try fixing that by raising a RuntimeError after 2,000 iterations
  760. (Also added a test for crontab leap years in the process)
  761. Fix contributed by **Romuald Brunet**.
  762. - Now ensures the program exits with a non-zero exit code when an
  763. exception terminates the service.
  764. Fix contributed by **Simon Peeters**.
  765. App
  766. ~~~
  767. - Dates are now always timezone aware even if
  768. :setting:`enable_utc` is disabled (Issue #943).
  769. Fix contributed by **Omer Katz**.
  770. - **Config**: App preconfiguration is now also pickled with the configuration.
  771. Fix contributed by **Jeremy Zafran**.
  772. - The application can now change how task names are generated using
  773. the :meth:`~@gen_task_name` method.
  774. Contributed by **Dmitry Malinovsky**.
  775. - App has new ``app.current_worker_task`` property that
  776. returns the task that's currently being worked on (or :const:`None`).
  777. (Issue #2100).
  778. Execution Pools
  779. ~~~~~~~~~~~~~~~
  780. - **Eventlet/Gevent**: Fixed race condition leading to "simultaneous read"
  781. errors (Issue #2812).
  782. - **Prefork**: Prefork pool now uses ``poll`` instead of ``select`` where
  783. available (Issue #2373).
  784. - **Prefork**: Fixed bug where the pool would refuse to shut down the
  785. worker (Issue #2606).
  786. - **Eventlet**: Now returns pool size in :program:`celery inspect stats`
  787. command.
  788. Contributed by **Alexander Oblovatniy**.
  789. Transports
  790. ~~~~~~~~~~
  791. - **Redis Transport**: The Redis transport now supports the
  792. :setting:`broker_use_ssl` option.
  793. Programs
  794. ~~~~~~~~
  795. - :program:`celery multi`: ``%n`` format for is now synonym with
  796. ``%N`` to be consistent with :program:`celery worker`.
  797. - :program:`celery inspect`/:program:`celery control`: now supports a new
  798. :option:`--json <celery inspect --json>` option to give output in json format.
  799. - :program:`celery inspect registered`: now ignores built-in tasks.
  800. - :program:`celery purge` now takes ``-Q`` and ``-X`` options
  801. used to specify what queues to include and exclude from the purge.
  802. - New :program:`celery logtool`: Utility for filtering and parsing
  803. celery worker log-files
  804. - :program:`celery multi`: now passes through `%i` and `%I` log
  805. file formats.
  806. - General: ``%p`` can now be used to expand to the full worker node-name
  807. in log-file/pid-file arguments.
  808. - A new command line option
  809. :option:`--executable <celery worker --executable>` is now
  810. available for daemonizing programs (:program:`celery worker` and
  811. :program:`celery beat`).
  812. Contributed by **Bert Vanderbauwhede**.
  813. - :program:`celery worker`: supports new
  814. :option:`--prefetch-multiplier <celery worker --prefetch-multiplier>` option.
  815. Contributed by **Mickaël Penhard**.
  816. Worker
  817. ~~~~~~
  818. - Improvements and fixes for :class:`~celery.utils.collections.LimitedSet`.
  819. Getting rid of leaking memory + adding ``minlen`` size of the set:
  820. the minimal residual size of the set after operating for some time.
  821. ``minlen`` items are kept, even if they should've been expired.
  822. Problems with older and even more old code:
  823. #. Heap would tend to grow in some scenarios
  824. (like adding an item multiple times).
  825. #. Adding many items fast wouldn't clean them soon enough (if ever).
  826. #. When talking to other workers, revoked._data was sent, but
  827. it was processed on the other side as iterable.
  828. That means giving those keys new (current)
  829. time-stamp. By doing this workers could recycle
  830. items forever. Combined with 1) and 2), this means that in
  831. large set of workers, you're getting out of memory soon.
  832. All those problems should be fixed now.
  833. This should fix issues #3095, #3086.
  834. Contributed by **David Pravec**.
  835. - Worker now only starts the remote control command consumer if the
  836. broker transport used actually supports them.
  837. - Gossip now sets ``x-message-ttl`` for event queue to heartbeat_interval s.
  838. (Issue #2005).
  839. - Now preserves exit code (Issue #2024).
  840. - Fixed crash when the ``-purge`` argument was used.
  841. - Log--level for unrecoverable errors changed from ``error`` to
  842. ``critical``.
  843. - Improved rate limiting accuracy.
  844. - Account for missing timezone information in task expires field.
  845. Fix contributed by **Albert Wang**.
  846. - The worker no longer has a ``Queues`` bootsteps, as it is now
  847. superfluous.
  848. - Now emits the "Received task" line even for revoked tasks.
  849. (Issue #3155).
  850. - Now respects :setting:`broker_connection_retry` setting.
  851. Fix contributed by **Nat Williams**.
  852. - New :data:`celery.worker.state.requests` enables O(1) loookup
  853. of active/reserved tasks by id.
  854. - Auto-scale didn't always update keep-alive when scaling down.
  855. Fix contributed by **Philip Garnero**.
  856. - Fixed typo ``options_list`` -> ``option_list``.
  857. Fix contributed by **Greg Wilbur**.
  858. Debugging Utilities
  859. ~~~~~~~~~~~~~~~~~~~
  860. - :mod:`celery.contrib.rdb`: Changed remote debugger banner so that you can copy and paste
  861. the address easily (no longer has a period in the address).
  862. Contributed by **Jonathan Vanasco**.
  863. - Fixed compatibility with recent :pypi:`psutil` versions (Issue #3262).
  864. Signals
  865. ~~~~~~~
  866. - **App**: New signals for app configuration/finalization:
  867. - :data:`app.on_configure <@on_configure>`
  868. - :data:`app.on_after_configure <@on_after_configure>`
  869. - :data:`app.on_after_finalize <@on_after_finalize>`
  870. - **Task**: New task signals for rejected task messages:
  871. - :data:`celery.signals.task_rejected`.
  872. - :data:`celery.signals.task_unknown`.
  873. - **Worker**: New signal for when a heartbeat event is sent.
  874. - :data:`celery.signals.heartbeat_sent`
  875. Contributed by **Kevin Richardson**.
  876. Events
  877. ~~~~~~
  878. - Event messages now uses the RabbitMQ ``x-message-ttl`` option
  879. to ensure older event messages are discarded.
  880. The default is 5 seconds, but can be changed using the
  881. :setting:`event_queue_ttl` setting.
  882. - Event monitors now sets the :setting:`event_queue_expires`
  883. setting by default.
  884. The queues will now expire after 60 seconds after the monitor stops
  885. consuming from it.
  886. - Fixed a bug where a None value wasn't handled properly.
  887. Fix contributed by **Dongweiming**.
  888. - New :setting:`event_queue_prefix` setting can now be used
  889. to change the default ``celeryev`` queue prefix for event receiver queues.
  890. Contributed by **Takeshi Kanemoto**.
  891. - ``State.tasks_by_type`` and ``State.tasks_by_worker`` can now be
  892. used as a mapping for fast access to this information.
  893. Canvas
  894. ~~~~~~
  895. - ``chunks``/``map``/``starmap`` are now routed based on the target task.
  896. - ``Signature.link`` now works when argument is scalar (not a list)
  897. (Issue #2019).
  898. Deployment
  899. ~~~~~~~~~~
  900. - Generic init-scripts now support
  901. :envvar:`CELERY_SU` and :envvar:`CELERYD_SU_ARGS` environment variables
  902. to set the path and arguments for :command:`su` (:manpage:`su(1)`).
  903. - Generic init-scripts now better support FreBSD and other BSD
  904. systems by searching :file:`/usr/local/etc/` for the configuration file.
  905. Contributed by **Taha Jahangir**.
  906. - Generic init-script: Fixed strange bug for ``celerybeat`` where
  907. restart didn't always work (Issue #3018).
  908. - The systemd init script now uses a shell when executing
  909. services.
  910. Contributed by **Tomas Machalek**.
  911. Result Backends
  912. ~~~~~~~~~~~~~~~
  913. - Redis: Now has a default socket timeout of 5 seconds.
  914. The default can be changed using the new :setting:`redis_socket_timeout`
  915. setting.
  916. Contributed by **Raghuram Srinivasan**.
  917. - RPC Backend result queues are now auto delete by default (Issue #2001).
  918. - RPC Backend: Fixed problem where exception
  919. wasn't deserialized properly with the json serializer (Issue #2518).
  920. Fix contributed by **Allard Hoeve**.
  921. - CouchDB: Fixed typo causing the backend to not be found
  922. (Issue #3287).
  923. Fix contributed by **Andrew Stewart**.
  924. - MongoDB: Now supports setting the :setting:`result_serialzier` setting
  925. to ``bson`` to use the MongoDB libraries own serializer.
  926. Contributed by **Davide Quarta**.
  927. - MongoDB: URI handling has been improved to use
  928. database name, user and password from the URI if provided.
  929. Contributed by **Samuel Jaillet**.
  930. - SQLAlchemy result backend: Now ignores all result
  931. engine options when using NullPool (Issue #1930).
  932. - SQLAlchemy result backend: Now sets max char size to 155 to deal
  933. with brain damaged MySQL unicode implementation (Issue #1748).
  934. - **General**: All Celery exceptions/warnings now inherit from common
  935. :class:`~celery.exceptions.CeleryError`/:class:`~celery.exceptions.CeleryWarning`.
  936. (Issue #2643).
  937. Documentation Improvements
  938. ~~~~~~~~~~~~~~~~~~~~~~~~~~
  939. Contributed by:
  940. - Adam Chainz
  941. - Arthur Vuillard
  942. - Batiste Bieler
  943. - Daniel Devine
  944. - Edward Betts
  945. - Jason Veatch
  946. - Jeff Widman
  947. - Manuel Kaufmann
  948. - Maxime Beauchemin
  949. - Mitchel Humpherys
  950. - Rik
  951. - Tayfun Sen
  952. Reorganization, Deprecations, and Removals
  953. ==========================================
  954. Incompatible changes
  955. --------------------
  956. - Prefork: Calling ``result.get()`` or joining any result from within a task
  957. now raises :exc:`RuntimeError`.
  958. In previous versions this would emit a warning.
  959. - :mod:`celery.worker.consumer` is now a package, not a module.
  960. - Module ``celery.worker.job`` renamed to :mod:`celery.worker.request`.
  961. - Beat: ``Scheduler.Publisher``/``.publisher`` renamed to
  962. ``.Producer``/``.producer``.
  963. - Result: The task_name argument/attribute of :class:`@AsyncResult` was
  964. removed.
  965. This was historically a field used for :mod:`pickle` compatibility,
  966. but is no longer needed.
  967. - Backends: Arguments named ``status`` renamed to ``state``.
  968. - Backends: ``backend.get_status()`` renamed to ``backend.get_state()``.
  969. .. _v400-unscheduled-removals:
  970. Unscheduled Removals
  971. --------------------
  972. - The experimental :mod:`celery.contrib.methods` feature has been removed,
  973. as there were far many bugs in the implementation to be useful.
  974. - The CentOS init-scripts have been removed.
  975. These didn't really add any features over the generic init-scripts,
  976. so you're encouraged to use them instead, or something like
  977. :pypi:`supervisor`.
  978. .. _v400-deprecations-reorg:
  979. Reorganization Deprecations
  980. ---------------------------
  981. These symbols have been renamed, and while there's an alias available in this
  982. version for backward compatibility, they will be removed in Celery 5.0, so
  983. make sure you rename these ASAP to make sure it won't break for that release.
  984. Chances are that you'll only use the first in this list, but you never
  985. know:
  986. - ``celery.utils.worker_direct`` ->
  987. :meth:`celery.utils.nodenames.worker_direct`.
  988. - ``celery.utils.nodename`` -> :meth:`celery.utils.nodenames.nodename`.
  989. - ``celery.utils.anon_nodename`` ->
  990. :meth:`celery.utils.nodenames.anon_nodename`.
  991. - ``celery.utils.nodesplit`` -> :meth:`celery.utils.nodenames.nodesplit`.
  992. - ``celery.utils.default_nodename`` ->
  993. :meth:`celery.utils.nodenames.default_nodename`.
  994. - ``celery.utils.node_format`` -> :meth:`celery.utils.nodenames.node_format`.
  995. - ``celery.utils.host_format`` -> :meth:`celery.utils.nodenames.host_format`.
  996. .. _v400-removals:
  997. Scheduled Removals
  998. ------------------
  999. Modules
  1000. ~~~~~~~
  1001. - Module ``celery.worker.job`` has been renamed to :mod:`celery.worker.request`.
  1002. This was an internal module so shouldn't have any effect.
  1003. It's now part of the public API so must not change again.
  1004. - Module ``celery.task.trace`` has been renamed to ``celery.app.trace``
  1005. as the ``celery.task`` package is being phased out. The module
  1006. will be removed in version 5.0 so please change any import from::
  1007. from celery.task.trace import X
  1008. to::
  1009. from celery.app.trace import X
  1010. - Old compatibility aliases in the :mod:`celery.loaders` module
  1011. has been removed.
  1012. - Removed ``celery.loaders.current_loader()``, use: ``current_app.loader``
  1013. - Removed ``celery.loaders.load_settings()``, use: ``current_app.conf``
  1014. Result
  1015. ~~~~~~
  1016. - ``AsyncResult.serializable()`` and ``celery.result.from_serializable``
  1017. has been removed:
  1018. Use instead:
  1019. .. code-block:: pycon
  1020. >>> tup = result.as_tuple()
  1021. >>> from celery.result import result_from_tuple
  1022. >>> result = result_from_tuple(tup)
  1023. - Removed ``BaseAsyncResult``, use ``AsyncResult`` for instance checks
  1024. instead.
  1025. - Removed ``TaskSetResult``, use ``GroupResult`` instead.
  1026. - ``TaskSetResult.total`` -> ``len(GroupResult)``
  1027. - ``TaskSetResult.taskset_id`` -> ``GroupResult.id``
  1028. - Removed ``ResultSet.subtasks``, use ``ResultSet.results`` instead.
  1029. TaskSet
  1030. ~~~~~~~
  1031. TaskSet has been renamed to group and TaskSet will be removed in version 4.0.
  1032. Old::
  1033. >>> from celery.task import TaskSet
  1034. >>> TaskSet(add.subtask((i, i)) for i in xrange(10)).apply_async()
  1035. New::
  1036. >>> from celery import group
  1037. >>> group(add.s(i, i) for i in xrange(10))()
  1038. Events
  1039. ~~~~~~
  1040. - Removals for class :class:`celery.events.state.Worker`:
  1041. - ``Worker._defaults`` attribute.
  1042. Use ``{k: getattr(worker, k) for k in worker._fields}``.
  1043. - ``Worker.update_heartbeat``
  1044. Use ``Worker.event(None, timestamp, received)``
  1045. - ``Worker.on_online``
  1046. Use ``Worker.event('online', timestamp, received, fields)``
  1047. - ``Worker.on_offline``
  1048. Use ``Worker.event('offline', timestamp, received, fields)``
  1049. - ``Worker.on_heartbeat``
  1050. Use ``Worker.event('heartbeat', timestamp, received, fields)``
  1051. - Removals for class :class:`celery.events.state.Task`:
  1052. - ``Task._defaults`` attribute.
  1053. Use ``{k: getattr(task, k) for k in task._fields}``.
  1054. - ``Task.on_sent``
  1055. Use ``Worker.event('sent', timestamp, received, fields)``
  1056. - ``Task.on_received``
  1057. Use ``Task.event('received', timestamp, received, fields)``
  1058. - ``Task.on_started``
  1059. Use ``Task.event('started', timestamp, received, fields)``
  1060. - ``Task.on_failed``
  1061. Use ``Task.event('failed', timestamp, received, fields)``
  1062. - ``Task.on_retried``
  1063. Use ``Task.event('retried', timestamp, received, fields)``
  1064. - ``Task.on_succeeded``
  1065. Use ``Task.event('succeeded', timestamp, received, fields)``
  1066. - ``Task.on_revoked``
  1067. Use ``Task.event('revoked', timestamp, received, fields)``
  1068. - ``Task.on_unknown_event``
  1069. Use ``Task.event(short_type, timestamp, received, fields)``
  1070. - ``Task.update``
  1071. Use ``Task.event(short_type, timestamp, received, fields)``
  1072. - ``Task.merge``
  1073. Contact us if you need this.
  1074. Magic keyword arguments
  1075. ~~~~~~~~~~~~~~~~~~~~~~~
  1076. Support for the very old magic keyword arguments accepted by tasks is
  1077. finally removed in this version.
  1078. If you're still using these you have to rewrite any task still
  1079. using the old ``celery.decorators`` module and depending
  1080. on keyword arguments being passed to the task,
  1081. for example::
  1082. from celery.decorators import task
  1083. @task()
  1084. def add(x, y, task_id=None):
  1085. print('My task id is %r' % (task_id,))
  1086. should be rewritten into::
  1087. from celery import task
  1088. @task(bind=True)
  1089. def add(self, x, y):
  1090. print('My task id is {0.request.id}'.format(self))
  1091. Removed Settings
  1092. ----------------
  1093. The following settings have been removed, and is no longer supported:
  1094. Logging Settings
  1095. ~~~~~~~~~~~~~~~~
  1096. ===================================== =====================================
  1097. **Setting name** **Replace with**
  1098. ===================================== =====================================
  1099. ``CELERYD_LOG_LEVEL`` :option:`celery worker --loglevel`
  1100. ``CELERYD_LOG_FILE`` :option:`celery worker --logfile`
  1101. ``CELERYBEAT_LOG_LEVEL`` :option:`celery beat --loglevel`
  1102. ``CELERYBEAT_LOG_FILE`` :option:`celery beat --loglevel`
  1103. ``CELERYMON_LOG_LEVEL`` celerymon is deprecated, use flower
  1104. ``CELERYMON_LOG_FILE`` celerymon is deprecated, use flower
  1105. ``CELERYMON_LOG_FORMAT`` celerymon is deprecated, use flower
  1106. ===================================== =====================================
  1107. Task Settings
  1108. ~~~~~~~~~~~~~~
  1109. ===================================== =====================================
  1110. **Setting name** **Replace with**
  1111. ===================================== =====================================
  1112. ``CELERY_CHORD_PROPAGATES`` N/A
  1113. ===================================== =====================================
  1114. Changes to internal API
  1115. -----------------------
  1116. - Module ``celery.datastructures`` renamed to :mod:`celery.utils.collections`.
  1117. - Module ``celery.utils.timeutils`` renamed to :mod:`celery.utils.time`.
  1118. - ``celery.utils.datastructures.DependencyGraph`` moved to
  1119. :mod:`celery.utils.graph`.
  1120. - ``celery.utils.jsonify`` is now :func:`celery.utils.serialization.jsonify`.
  1121. - ``celery.utils.strtobool`` is now
  1122. :func:`celery.utils.serialization.strtobool`.
  1123. - ``celery.utils.is_iterable`` has been removed.
  1124. Instead use::
  1125. isinstance(x, collections.Iterable)
  1126. - ``celery.utils.lpmerge`` is now :func:`celery.utils.collections.lpmerge`.
  1127. - ``celery.utils.cry`` is now :func:`celery.utils.debug.cry`.
  1128. - ``celery.utils.isatty`` is now :func:`celery.platforms.isatty`.
  1129. - ``celery.utils.gen_task_name`` is now
  1130. :func:`celery.utils.imports.gen_task_name`.
  1131. - ``celery.utils.deprecated`` is now :func:`celery.utils.deprecated.Callable`
  1132. - ``celery.utils.deprecated_property`` is now
  1133. :func:`celery.utils.deprecated.Property`.
  1134. - ``celery.utils.warn_deprecated`` is now :func:`celery.utils.deprecated.warn`
  1135. .. _v400-deprecations:
  1136. Deprecation Time-line Changes
  1137. =============================
  1138. See the :ref:`deprecation-timeline`.