whatsnew-2.5.rst 16 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519
  1. .. _whatsnew-2.5:
  2. ==========================
  3. What's new in Celery 2.5
  4. ==========================
  5. Celery aims to be a flexible and reliable, best-of-breed solution
  6. to process vast amounts of messages in a distributed fashion, while
  7. providing operations with the tools to maintain such a system.
  8. Celery has a large and diverse community of users and contributors,
  9. you should come join us :ref:`on IRC <irc-channel>`
  10. or :ref:`our mailing-list <mailing-list>`.
  11. To read more about Celery you should visit our `website`_.
  12. While this version is backward compatible with previous versions
  13. it is important that you read the following section.
  14. If you use Celery in combination with Django you must also
  15. read the `django-celery changelog <djcelery:version-2.5.0>` and upgrade to `django-celery 2.5`_.
  16. This version is officially supported on CPython 2.5, 2.6, 2.7, 3.2 and 3.3,
  17. as well as PyPy and Jython.
  18. .. _`website`: http://celeryproject.org/
  19. .. _`django-celery 2.5`: http://pypi.python.org/pypi/django-celery/
  20. .. contents::
  21. :local:
  22. .. _v250-important:
  23. Important Notes
  24. ===============
  25. Broker connection pool now enabled by default
  26. ---------------------------------------------
  27. The default limit is 10 connections, if you have many threads/green-threads
  28. using connections at the same time you may want to tweak this limit
  29. to avoid contention.
  30. See the :setting:`BROKER_POOL_LIMIT` setting for more information.
  31. Also note that publishing tasks will be retried by default, to change
  32. this default or the default retry policy see
  33. :setting:`CELERY_TASK_PUBLISH_RETRY` and
  34. :setting:`CELERY_TASK_PUBLISH_RETRY_POLICY`.
  35. Rabbit Result Backend: Exchange is no longer *auto delete*
  36. ----------------------------------------------------------
  37. The exchange used for results in the Rabbit (AMQP) result backend
  38. used to have the *auto_delete* flag set, which could result in a
  39. race condition leading to an annoying warning.
  40. .. admonition:: For RabbitMQ users
  41. Old exchanges created with the *auto_delete* flag enabled has
  42. to be removed.
  43. The :program:`camqadm` command can be used to delete the
  44. previous exchange:
  45. .. code-block:: console
  46. $ camqadm exchange.delete celeryresults
  47. As an alternative to deleting the old exchange you can
  48. configure a new name for the exchange::
  49. CELERY_RESULT_EXCHANGE = 'celeryresults2'
  50. But you have to make sure that all clients and workers
  51. use this new setting, so they are updated to use the same
  52. exchange name.
  53. Solution for hanging workers (but must be manually enabled)
  54. -----------------------------------------------------------
  55. The `CELERYD_FORCE_EXECV` setting has been added to solve
  56. a problem with deadlocks that originate when threads and fork is mixed
  57. together:
  58. .. code-block:: python
  59. CELERYD_FORCE_EXECV = True
  60. This setting is recommended for all users using the prefork pool,
  61. but especially users also using time limits or a max tasks per child
  62. setting.
  63. - See `Python Issue 6721`_ to read more about this issue, and why
  64. resorting to :func:`~os.execv`` is the only safe solution.
  65. Enabling this option will result in a slight performance penalty
  66. when new child worker processes are started, and it will also increase
  67. memory usage (but many platforms are optimized, so the impact may be
  68. minimal). Considering that it ensures reliability when replacing
  69. lost worker processes, it should be worth it.
  70. - It's already the default behavior on Windows.
  71. - It will be the default behavior for all platforms in a future version.
  72. .. _`Python Issue 6721`: http://bugs.python.org/issue6721#msg140215
  73. .. _v250-optimizations:
  74. Optimization
  75. ============
  76. - The code path used when the worker executes a task has been heavily
  77. optimized, meaning the worker is able to process a great deal
  78. more tasks/second compared to previous versions. As an example the solo
  79. pool can now process up to 15000 tasks/second on a 4 core MacBook Pro
  80. when using the `pylibrabbitmq`_ transport, where it previously
  81. could only do 5000 tasks/second.
  82. - The task error tracebacks are now much shorter.
  83. - Fixed a noticeable delay in task processing when rate limits are enabled.
  84. .. _`pylibrabbitmq`: http://pypi.python.org/pylibrabbitmq/
  85. .. _v250-deprecations:
  86. Deprecation Time-line Changes
  87. =============================
  88. Removals
  89. --------
  90. * The old :class:`TaskSet` signature of ``(task_name, list_of_tasks)``
  91. can no longer be used (originally scheduled for removal in 2.4).
  92. The deprecated ``.task_name`` and ``.task`` attributes has also been
  93. removed.
  94. * The functions ``celery.execute.delay_task``, ``celery.execute.apply``,
  95. and ``celery.execute.apply_async`` has been removed (originally)
  96. scheduled for removal in 2.3).
  97. * The built-in ``ping`` task has been removed (originally scheduled
  98. for removal in 2.3). Please use the ping broadcast command
  99. instead.
  100. * It is no longer possible to import ``subtask`` and ``TaskSet``
  101. from :mod:`celery.task.base`, please import them from :mod:`celery.task`
  102. instead (originally scheduled for removal in 2.4).
  103. Deprecated modules
  104. ------------------
  105. * The :mod:`celery.decorators` module has changed status
  106. from pending deprecation to deprecated, and is scheduled for removal
  107. in version 4.0. The ``celery.task`` module must be used instead.
  108. .. _v250-news:
  109. News
  110. ====
  111. Timezone support
  112. ----------------
  113. Celery can now be configured to treat all incoming and outgoing dates
  114. as UTC, and the local timezone can be configured.
  115. This is not yet enabled by default, since enabling
  116. time zone support means workers running versions pre-2.5
  117. will be out of sync with upgraded workers.
  118. To enable UTC you have to set :setting:`CELERY_ENABLE_UTC`::
  119. CELERY_ENABLE_UTC = True
  120. When UTC is enabled, dates and times in task messages will be
  121. converted to UTC, and then converted back to the local timezone
  122. when received by a worker.
  123. You can change the local timezone using the :setting:`CELERY_TIMEZONE`
  124. setting. Installing the :pypi:`pytz` library is recommended when
  125. using a custom timezone, to keep timezone definition up-to-date,
  126. but it will fallback to a system definition of the timezone if available.
  127. UTC will enabled by default in version 3.0.
  128. .. note::
  129. :pypi:`django-celery` will use the local timezone as specified by the
  130. ``TIME_ZONE`` setting, it will also honor the new `USE_TZ`_ setting
  131. introduced in Django 1.4.
  132. .. _`USE_TZ`: https://docs.djangoproject.com/en/dev/topics/i18n/timezones/
  133. New security serializer using cryptographic signing
  134. ---------------------------------------------------
  135. A new serializer has been added that signs and verifies the signature
  136. of messages.
  137. The name of the new serializer is ``auth``, and needs additional
  138. configuration to work (see :ref:`conf-security`).
  139. .. seealso::
  140. :ref:`guide-security`
  141. Contributed by Mher Movsisyan.
  142. New :setting:`CELERY_ANNOTATIONS` setting
  143. -----------------------------------------
  144. This new setting enables the configuration to modify task classes
  145. and their attributes.
  146. The setting can be a dict, or a list of annotation objects that filter
  147. for tasks and return a map of attributes to change.
  148. As an example, this is an annotation to change the ``rate_limit`` attribute
  149. for the ``tasks.add`` task:
  150. .. code-block:: python
  151. CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
  152. or change the same for all tasks:
  153. .. code-block:: python
  154. CELERY_ANNOTATIONS = {'*': {'rate_limit': '10/s'}}
  155. You can change methods too, for example the ``on_failure`` handler:
  156. .. code-block:: python
  157. def my_on_failure(self, exc, task_id, args, kwargs, einfo):
  158. print('Oh no! Task failed: %r' % (exc,))
  159. CELERY_ANNOTATIONS = {'*': {'on_failure': my_on_failure}}
  160. If you need more flexibility then you can also create objects
  161. that filter for tasks to annotate:
  162. .. code-block:: python
  163. class MyAnnotate(object):
  164. def annotate(self, task):
  165. if task.name.startswith('tasks.'):
  166. return {'rate_limit': '10/s'}
  167. CELERY_ANNOTATIONS = (MyAnnotate(), {other_annotations,})
  168. ``current`` provides the currently executing task
  169. -------------------------------------------------
  170. The new :data:`celery.task.current` proxy will always give the currently
  171. executing task.
  172. **Example**:
  173. .. code-block:: python
  174. from celery.task import current, task
  175. @task
  176. def update_twitter_status(auth, message):
  177. twitter = Twitter(auth)
  178. try:
  179. twitter.update_status(message)
  180. except twitter.FailWhale, exc:
  181. # retry in 10 seconds.
  182. current.retry(countdown=10, exc=exc)
  183. Previously you would have to type ``update_twitter_status.retry(…)``
  184. here, which can be annoying for long task names.
  185. .. note::
  186. This will not work if the task function is called directly, i.e:
  187. ``update_twitter_status(a, b)``. For that to work ``apply`` must
  188. be used: ``update_twitter_status.apply((a, b))``.
  189. In Other News
  190. -------------
  191. - Now depends on Kombu 2.1.0.
  192. - Efficient Chord support for the Memcached backend (Issue #533)
  193. This means Memcached joins Redis in the ability to do non-polling
  194. chords.
  195. Contributed by Dan McGee.
  196. - Adds Chord support for the Rabbit result backend (amqp)
  197. The Rabbit result backend can now use the fallback chord solution.
  198. - Sending :sig:`QUIT` to ``celeryd`` will now cause it cold terminate.
  199. That is, it will not finish executing the tasks it is currently
  200. working on.
  201. Contributed by Alec Clowes.
  202. - New "detailed" mode for the Cassandra backend.
  203. Allows to have a "detailed" mode for the Cassandra backend.
  204. Basically the idea is to keep all states using Cassandra wide columns.
  205. New states are then appended to the row as new columns, the last state
  206. being the last column.
  207. See the :setting:`CASSANDRA_DETAILED_MODE` setting.
  208. Contributed by Steeve Morin.
  209. - The Crontab parser now matches Vixie Cron behavior when parsing ranges
  210. with steps (e.g. 1-59/2).
  211. Contributed by Daniel Hepper.
  212. - ``celerybeat`` can now be configured on the command-line like ``celeryd``.
  213. Additional configuration must be added at the end of the argument list
  214. followed by ``--``, for example:
  215. .. code-block:: console
  216. $ celerybeat -l info -- celerybeat.max_loop_interval=10.0
  217. - Now limits the number of frames in a traceback so that ``celeryd`` does not
  218. crash on maximum recursion limit exceeded exceptions (Issue #615).
  219. The limit is set to the current recursion limit divided by 8 (which
  220. is 125 by default).
  221. To get or set the current recursion limit use
  222. :func:`sys.getrecursionlimit` and :func:`sys.setrecursionlimit`.
  223. - More information is now preserved in the pickleable traceback.
  224. This has been added so that Sentry can show more details.
  225. Contributed by Sean O'Connor.
  226. - CentOS init-script has been updated and should be more flexible.
  227. Contributed by Andrew McFague.
  228. - MongoDB result backend now supports ``forget()``.
  229. Contributed by Andrew McFague
  230. - ``task.retry()`` now re-raises the original exception keeping
  231. the original stack trace.
  232. Suggested by :github_user:`ojii`.
  233. - The `--uid` argument to daemons now uses ``initgroups()`` to set
  234. groups to all the groups the user is a member of.
  235. Contributed by Łukasz Oleś.
  236. - ``celeryctl``: Added ``shell`` command.
  237. The shell will have the current_app (``celery``) and all tasks
  238. automatically added to locals.
  239. - ``celeryctl``: Added ``migrate`` command.
  240. The migrate command moves all tasks from one broker to another.
  241. Note that this is experimental and you should have a backup
  242. of the data before proceeding.
  243. **Examples**:
  244. .. code-block:: console
  245. $ celeryctl migrate redis://localhost amqp://localhost
  246. $ celeryctl migrate amqp://localhost//v1 amqp://localhost//v2
  247. $ python manage.py celeryctl migrate django:// redis://
  248. * Routers can now override the ``exchange`` and ``routing_key`` used
  249. to create missing queues (Issue #577).
  250. By default this will always use the name of the queue,
  251. but you can now have a router return exchange and routing_key keys
  252. to set them.
  253. This is useful when using routing classes which decides a destination
  254. at run-time.
  255. Contributed by Akira Matsuzaki.
  256. - Redis result backend: Adds support for a ``max_connections`` parameter.
  257. It is now possible to configure the maximum number of
  258. simultaneous connections in the Redis connection pool used for
  259. results.
  260. The default max connections setting can be configured using the
  261. :setting:`CELERY_REDIS_MAX_CONNECTIONS` setting,
  262. or it can be changed individually by ``RedisBackend(max_connections=int)``.
  263. Contributed by Steeve Morin.
  264. - Redis result backend: Adds the ability to wait for results without polling.
  265. Contributed by Steeve Morin.
  266. - MongoDB result backend: Now supports save and restore ``taskset``.
  267. Contributed by Julien Poissonnier.
  268. - There's a new :ref:`guide-security` guide in the documentation.
  269. - The init-scripts have been updated, and many bugs fixed.
  270. Contributed by Chris Streeter.
  271. - User (tilde) is now expanded in command-line arguments.
  272. - Can now configure :envvar:`CELERYCTL` environment variable
  273. in :file:`/etc/default/celeryd`.
  274. While not necessary for operation, :program:`celeryctl` is used for the
  275. ``celeryd status`` command, and the path to :program:`celeryctl` must be
  276. configured for that to work.
  277. The daemonization cookbook contains examples.
  278. Contributed by Jude Nagurney.
  279. - The MongoDB result backend can now use Replica Sets.
  280. Contributed by Ivan Metzlar.
  281. - gevent: Now supports autoscaling (Issue #599).
  282. Contributed by Mark Lavin.
  283. - multiprocessing: Mediator thread is now always enabled,
  284. even though rate limits are disabled, as the pool semaphore
  285. is known to block the main thread, causing broadcast commands and
  286. shutdown to depend on the semaphore being released.
  287. Fixes
  288. =====
  289. - Exceptions that are re-raised with a new exception object now keeps
  290. the original stack trace.
  291. - Windows: Fixed the ``no handlers found for multiprocessing`` warning.
  292. - Windows: The ``celeryd`` program can now be used.
  293. Previously Windows users had to launch ``celeryd`` using
  294. ``python -m celery.bin.celeryd``.
  295. - Redis result backend: Now uses ``SETEX`` command to set result key,
  296. and expiry atomically.
  297. Suggested by :github_user:`yaniv-aknin`.
  298. - ``celeryd``: Fixed a problem where shutdown hanged when :kbd:`Control-c`
  299. was used to terminate.
  300. - ``celeryd``: No longer crashes when channel errors occur.
  301. Fix contributed by Roger Hu.
  302. - Fixed memory leak in the eventlet pool, caused by the
  303. use of ``greenlet.getcurrent``.
  304. Fix contributed by Ignas Mikalajūnas.
  305. - Cassandra backend: No longer uses :func:`pycassa.connect` which is
  306. deprecated since :pypi:`pycassa` 1.4.
  307. Fix contributed by Jeff Terrace.
  308. - Fixed unicode decode errors that could occur while sending error emails.
  309. Fix contributed by Seong Wun Mun.
  310. - ``celery.bin`` programs now always defines ``__package__`` as recommended
  311. by PEP-366.
  312. - ``send_task`` now emits a warning when used in combination with
  313. :setting:`CELERY_ALWAYS_EAGER` (Issue #581).
  314. Contributed by Mher Movsisyan.
  315. - ``apply_async`` now forwards the original keyword arguments to ``apply``
  316. when :setting:`CELERY_ALWAYS_EAGER` is enabled.
  317. - ``celeryev`` now tries to re-establish the connection if the connection
  318. to the broker is lost (Issue #574).
  319. - ``celeryev``: Fixed a crash occurring if a task has no associated worker
  320. information.
  321. Fix contributed by Matt Williamson.
  322. - The current date and time is now consistently taken from the current loaders
  323. ``now`` method.
  324. - Now shows helpful error message when given a configuration module ending in
  325. ``.py`` that can't be imported.
  326. - ``celeryctl``: The :option:`--expires <celery call --expires>` and
  327. :option:`--eta <celery call --eta>` arguments to the apply command
  328. can now be an ISO-8601 formatted string.
  329. - ``celeryctl`` now exits with exit status ``EX_UNAVAILABLE`` (69) if no replies
  330. have been received.