whatsnew-3.0.rst 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978
  1. .. _whatsnew-3.0:
  2. ===========================================
  3. What's new in Celery 3.0 (Chiastic Slide)
  4. ===========================================
  5. Celery is a simple, flexible and reliable distributed system to
  6. process vast amounts of messages, while providing operations with
  7. the tools required to maintain such a system.
  8. It's a task queue with focus on real-time processing, while also
  9. supporting task scheduling.
  10. Celery has a large and diverse community of users and contributors,
  11. you should come join us :ref:`on IRC <irc-channel>`
  12. or :ref:`our mailing-list <mailing-list>`.
  13. To read more about Celery you should go read the :ref:`introduction <intro>`.
  14. While this version is backward compatible with previous versions
  15. it's important that you read the following section.
  16. If you use Celery in combination with Django you must also
  17. read the `django-celery changelog`_ and upgrade to `django-celery 3.0`_.
  18. This version is officially supported on CPython 2.5, 2.6, 2.7, 3.2 and 3.3,
  19. as well as PyPy and Jython.
  20. Highlights
  21. ==========
  22. .. topic:: Overview
  23. - A new and improved API, that is both simpler and more powerful.
  24. Everyone must read the new :ref:`first-steps` tutorial,
  25. and the new :ref:`next-steps` tutorial. Oh, and
  26. why not reread the user guide while you're at it :)
  27. There are no current plans to deprecate the old API,
  28. so you don't have to be in a hurry to port your applications.
  29. - The worker is now thread-less, giving great performance improvements.
  30. - The new "Canvas" makes it easy to define complex workflows.
  31. Ever wanted to chain tasks together? This is possible, but
  32. not just that, now you can even chain together groups and chords,
  33. or even combine multiple chains.
  34. Read more in the :ref:`Canvas <guide-canvas>` user guide.
  35. - All of Celery's command line programs are now available from a single
  36. :program:`celery` umbrella command.
  37. - This is the last version to support Python 2.5.
  38. Starting with Celery 3.1, Python 2.6 or later is required.
  39. - Support for the new librabbitmq C client.
  40. Celery will automatically use the :mod:`librabbitmq` module
  41. if installed, which is a very fast and memory-optimized
  42. replacement for the amqplib module.
  43. - Redis support is more reliable with improved ack emulation.
  44. - Celery now always uses UTC
  45. - Over 600 commits, 30k additions/36k deletions.
  46. In comparison 1.0➝ 2.0 had 18k additions/8k deletions.
  47. .. _`website`: http://celeryproject.org/
  48. .. _`django-celery changelog`:
  49. http://github.com/celery/django-celery/tree/master/Changelog
  50. .. _`django-celery 3.0`: http://pypi.python.org/pypi/django-celery/
  51. .. contents::
  52. :local:
  53. :depth: 2
  54. .. _v300-important:
  55. Important Notes
  56. ===============
  57. Broadcast exchanges renamed
  58. ---------------------------
  59. The workers remote control command exchanges has been renamed
  60. (a new pidbox name), this is because the ``auto_delete`` flag on the exchanges
  61. has been removed, and that makes it incompatible with earlier versions.
  62. You can manually delete the old exchanges if you want,
  63. using the :program:`celery amqp` command (previously called ``camqadm``)::
  64. $ celery amqp exchange.delete celeryd.pidbox
  65. $ celery amqp exchange.delete reply.celeryd.pidbox
  66. Eventloop
  67. ---------
  68. The worker is now running *without threads* when used with RabbitMQ (AMQP),
  69. or Redis as a broker, resulting in:
  70. - Much better overall performance.
  71. - Fixes several edge case race conditions.
  72. - Sub-millisecond timer precision.
  73. - Faster shutdown times.
  74. The transports supported are: ``amqplib``, ``librabbitmq``, and ``redis``
  75. Hopefully this can be extended to include additional broker transports
  76. in the future.
  77. For increased reliability the :setting:`CELERY_FORCE_EXECV` setting is enabled
  78. by default if the eventloop is not used.
  79. New ``celery`` umbrella command
  80. -------------------------------
  81. All Celery's command line programs are now available from a single
  82. :program:`celery` umbrella command.
  83. You can see a list of subcommands and options by running::
  84. $ celery help
  85. Commands include:
  86. - ``celery worker`` (previously ``celeryd``).
  87. - ``celery beat`` (previously ``celerybeat``).
  88. - ``celery amqp`` (previously ``camqadm``).
  89. The old programs are still available (``celeryd``, ``celerybeat``, etc),
  90. but you are discouraged from using them.
  91. Now depends on :mod:`billiard`.
  92. -------------------------------
  93. Billiard is a fork of the multiprocessing containing
  94. the no-execv patch by sbt (http://bugs.python.org/issue8713),
  95. and also contains the pool improvements previously located in Celery.
  96. This fork was necessary as changes to the C extension code was required
  97. for the no-execv patch to work.
  98. - Issue #625
  99. - Issue #627
  100. - Issue #640
  101. - `django-celery #122 <http://github.com/celery/django-celery/issues/122`
  102. - `django-celery #124 <http://github.com/celery/django-celery/issues/122`
  103. :mod:`celery.app.task` no longer a package
  104. ------------------------------------------
  105. The :mod:`celery.app.task` module is now a module instead of a package.
  106. The setup.py install script will try to remove the old package,
  107. but if that doesn't work for some reason you have to remove
  108. it manually. This command helps::
  109. $ rm -r $(dirname $(python -c '
  110. import celery;print(celery.__file__)'))/app/task/
  111. If you experience an error like ``ImportError: cannot import name _unpickle_task``,
  112. you just have to remove the old package and everything is fine.
  113. Last version to support Python 2.5
  114. ----------------------------------
  115. The 3.0 series will be last version to support Python 2.5,
  116. and starting from 3.1 Python 2.6 and later will be required.
  117. With several other distributions taking the step to discontinue
  118. Python 2.5 support, we feel that it is time too.
  119. Python 2.6 should be widely available at this point, and we urge
  120. you to upgrade, but if that is not possible you still have the option
  121. to continue using the Celery 3.0, and important bug fixes
  122. introduced in Celery 3.1 will be back-ported to Celery 3.0 upon request.
  123. UTC timezone is now used
  124. ------------------------
  125. This means that ETA/countdown in messages are not compatible with Celery
  126. versions prior to 2.5.
  127. You can disable UTC and revert back to old local time by setting
  128. the :setting:`CELERY_ENABLE_UTC` setting.
  129. Redis: Ack emulation improvements
  130. ---------------------------------
  131. Reducing the possibility of data loss.
  132. Acks are now implemented by storing a copy of the message when the message
  133. is consumed. The copy is not removed until the consumer acknowledges
  134. or rejects it.
  135. This means that unacknowledged messages will be redelivered either
  136. when the connection is closed, or when the visibility timeout is exceeded.
  137. - Visibility timeout
  138. This is a timeout for acks, so that if the consumer
  139. does not ack the message within this time limit, the message
  140. is redelivered to another consumer.
  141. The timeout is set to one hour by default, but
  142. can be changed by configuring a transport option::
  143. BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 18000} # 5 hours
  144. .. note::
  145. Messages that have not been acked will be redelivered
  146. if the visibility timeout is exceeded, for Celery users
  147. this means that ETA/countdown tasks that are scheduled to execute
  148. with a time that exceeds the visibility timeout will be executed
  149. twice (or more). If you plan on using long ETA/countdowns you
  150. should tweak the visibility timeout accordingly.
  151. Setting a long timeout means that it will take a long time
  152. for messages to be redelivered in the event of a power failure,
  153. but if so happens you could temporarily set the visibility timeout lower
  154. to flush out messages when you start up the systems again.
  155. .. _v300-news:
  156. News
  157. ====
  158. Chaining Tasks
  159. --------------
  160. Tasks can now have callbacks and errbacks, and dependencies are recorded
  161. - The task message format have been updated with two new extension keys
  162. Both keys can be empty/undefined or a list of subtasks.
  163. - ``callbacks``
  164. Applied if the task exits successfully, with the result
  165. of the task as an argument.
  166. - ``errbacks``
  167. Applied if an error occurred while executing the task,
  168. with the uuid of the task as an argument. Since it may not be possible
  169. to serialize the exception instance, it passes the uuid of the task
  170. instead. The uuid can then be used to retrieve the exception and
  171. traceback of the task from the result backend.
  172. - ``link`` and ``link_error`` keyword arguments has been added
  173. to ``apply_async``.
  174. These add callbacks and errbacks to the task, and
  175. you can read more about them at :ref:`calling-links`.
  176. - We now track what subtasks a task sends, and some result backends
  177. supports retrieving this information.
  178. - task.request.children
  179. Contains the result instances of the subtasks
  180. the currently executing task has applied.
  181. - AsyncResult.children
  182. Returns the tasks dependencies, as a list of
  183. ``AsyncResult``/``ResultSet`` instances.
  184. - AsyncResult.iterdeps
  185. Recursively iterates over the tasks dependencies,
  186. yielding `(parent, node)` tuples.
  187. Raises IncompleteStream if any of the dependencies
  188. has not returned yet.
  189. - AsyncResult.graph
  190. A ``DependencyGraph`` of the tasks dependencies.
  191. This can also be used to convert to dot format:
  192. .. code-block:: python
  193. with open('graph.dot') as fh:
  194. result.graph.to_dot(fh)
  195. which can than be used to produce an image::
  196. $ dot -Tpng graph.dot -o graph.png
  197. - A new special subtask called ``chain`` is also included:
  198. .. code-block:: python
  199. >>> from celery import chain
  200. # (2 + 2) * 8 / 2
  201. >>> res = chain(add.subtask((2, 2)),
  202. mul.subtask((8, )),
  203. div.subtask((2,))).apply_async()
  204. >>> res.get() == 16
  205. >>> res.parent.get() == 32
  206. >>> res.parent.parent.get() == 4
  207. - Adds :meth:`AsyncResult.get_leaf`
  208. Waits and returns the result of the leaf subtask.
  209. That is the last node found when traversing the graph,
  210. but this means that the graph can be 1-dimensional only (in effect
  211. a list).
  212. - Adds ``subtask.link(subtask)`` + ``subtask.link_error(subtask)``
  213. Shortcut to ``s.options.setdefault('link', []).append(subtask)``
  214. - Adds ``subtask.flatten_links()``
  215. Returns a flattened list of all dependencies (recursively)
  216. Redis: Priority support.
  217. ------------------------
  218. The message's ``priority`` field is now respected by the Redis
  219. transport by having multiple lists for each named queue.
  220. The queues are then consumed by in order of priority.
  221. The priority field is a number in the range of 0 - 9, where
  222. 0 is the default and highest priority.
  223. The priority range is collapsed into four steps by default, since it is
  224. unlikely that nine steps will yield more benefit than using four steps.
  225. The number of steps can be configured by setting the ``priority_steps``
  226. transport option, which must be a list of numbers in **sorted order**::
  227. >>> BROKER_TRANSPORT_OPTIONS = {
  228. ... 'priority_steps': [0, 2, 4, 6, 8, 9],
  229. ... }
  230. Priorities implemented in this way is not as reliable as
  231. priorities on the server side, which is why
  232. the feature is nicknamed "quasi-priorities";
  233. **Using routing is still the suggested way of ensuring
  234. quality of service**, as client implemented priorities
  235. fall short in a number of ways, e.g. if the worker
  236. is busy with long running tasks, has prefetched many messages,
  237. or the queues are congested.
  238. Still, it is possible that using priorities in combination
  239. with routing can be more beneficial than using routing
  240. or priorities alone. Experimentation and monitoring
  241. should be used to prove this.
  242. Contributed by Germán M. Bravo.
  243. Redis: Now cycles queues so that consuming is fair.
  244. ---------------------------------------------------
  245. This ensures that a very busy queue won't block messages
  246. from other queues, and ensures that all queues have
  247. an equal chance of being consumed from.
  248. This used to be the case before, but the behavior was
  249. accidentally changed while switching to using blocking pop.
  250. `group`/`chord`/`chain` are now subtasks
  251. ----------------------------------------
  252. - group is no longer an alias to TaskSet, but new alltogether,
  253. since it was very difficult to migrate the TaskSet class to become
  254. a subtask.
  255. - A new shortcut has been added to tasks:
  256. ::
  257. >>> task.s(arg1, arg2, kw=1)
  258. as a shortcut to::
  259. >>> task.subtask((arg1, arg2), {'kw': 1})
  260. - Tasks can be chained by using the ``|`` operator::
  261. >>> (add.s(2, 2), pow.s(2)).apply_async()
  262. - Subtasks can be "evaluated" using the ``~`` operator:
  263. ::
  264. >>> ~add.s(2, 2)
  265. 4
  266. >>> ~(add.s(2, 2) | pow.s(2))
  267. is the same as::
  268. >>> chain(add.s(2, 2), pow.s(2)).apply_async().get()
  269. - A new subtask_type key has been added to the subtask dicts
  270. This can be the string "chord", "group", "chain", "chunks",
  271. "xmap", or "xstarmap".
  272. - maybe_subtask now uses subtask_type to reconstruct
  273. the object, to be used when using non-pickle serializers.
  274. - The logic for these operations have been moved to dedicated
  275. tasks celery.chord, celery.chain and celery.group.
  276. - subtask no longer inherits from AttributeDict.
  277. It's now a pure dict subclass with properties for attribute
  278. access to the relevant keys.
  279. - The repr's now outputs how the sequence would like imperatively::
  280. >>> from celery import chord
  281. >>> (chord([add.s(i, i) for i in xrange(10)], xsum.s())
  282. | pow.s(2))
  283. tasks.xsum([tasks.add(0, 0),
  284. tasks.add(1, 1),
  285. tasks.add(2, 2),
  286. tasks.add(3, 3),
  287. tasks.add(4, 4),
  288. tasks.add(5, 5),
  289. tasks.add(6, 6),
  290. tasks.add(7, 7),
  291. tasks.add(8, 8),
  292. tasks.add(9, 9)]) | tasks.pow(2)
  293. New remote control commands
  294. ---------------------------
  295. These commands were previously experimental, but they have proven
  296. stable and is now documented as part of the offical API.
  297. - ``add_consumer``/``cancel_consumer``
  298. Tells workers to consume from a new queue, or cancel consuming from a
  299. queue. This command has also been changed so that the worker remembers
  300. the queues added, so that the change will persist even if
  301. the connection is re-connected.
  302. These commands are available programmatically as
  303. :meth:`@control.add_consumer` / :meth:`@control.cancel_consumer`:
  304. .. code-block:: python
  305. >>> celery.control.add_consumer(queue_name,
  306. ... destination=['w1.example.com'])
  307. >>> celery.control.cancel_consumer(queue_name,
  308. ... destination=['w1.example.com'])
  309. or using the :program:`celery control` command::
  310. $ celery control -d w1.example.com add_consumer queue
  311. $ celery control -d w1.example.com cancel_consumer queue
  312. .. note::
  313. Remember that a control command without *destination* will be
  314. sent to **all workers**.
  315. - ``autoscale``
  316. Tells workers with `--autoscale` enabled to change autoscale
  317. max/min concurrency settings.
  318. This command is available programmatically as :meth:`@control.autoscale`:
  319. .. code-block:: python
  320. >>> celery.control.autoscale(max=10, min=5,
  321. ... destination=['w1.example.com'])
  322. or using the :program:`celery control` command::
  323. $ celery control -d w1.example.com autoscale 10 5
  324. - ``pool_grow``/``pool_shrink``
  325. Tells workers to add or remove pool processes.
  326. These commands are available programmatically as
  327. :meth:`@control.pool_grow` / :meth:`@control.pool_shrink`:
  328. .. code-block:: python
  329. >>> celery.control.pool_grow(2, destination=['w1.example.com'])
  330. >>> celery.contorl.pool_shrink(2, destination=['w1.example.com'])
  331. or using the :program:`celery control` command::
  332. $ celery control -d w1.example.com pool_grow 2
  333. $ celery control -d w1.example.com pool_shrink 2
  334. - :program:`celery control` now supports ``rate_limit`` & ``time_limit``
  335. commands.
  336. See ``celery control --help`` for details.
  337. Crontab now supports Day of Month, and Month of Year arguments
  338. --------------------------------------------------------------
  339. See the updated list of examples at :ref:`beat-crontab`.
  340. Immutable subtasks
  341. ------------------
  342. ``subtask``'s can now be immutable, which means that the arguments
  343. will not be modified when calling callbacks::
  344. >>> chain(add.s(2, 2), clear_static_electricity.si())
  345. means it will not receive the argument of the parent task,
  346. and ``.si()`` is a shortcut to::
  347. >>> clear_static_electricity.subtask(immutable=True)
  348. Logging Improvements
  349. --------------------
  350. Logging support now conforms better with best practices.
  351. - Classes used by the worker no longer uses app.get_default_logger, but uses
  352. `celery.utils.log.get_logger` which simply gets the logger not setting the
  353. level, and adds a NullHandler.
  354. - Loggers are no longer passed around, instead every module using logging
  355. defines a module global logger that is used throughout.
  356. - All loggers inherit from a common logger called "celery".
  357. - Before task.get_logger would setup a new logger for every task,
  358. and even set the loglevel. This is no longer the case.
  359. - Instead all task loggers now inherit from a common "celery.task" logger
  360. that is set up when programs call `setup_logging_subsystem`.
  361. - Instead of using LoggerAdapter to augment the formatter with
  362. the task_id and task_name field, the task base logger now use
  363. a special formatter adding these values at runtime from the
  364. currently executing task.
  365. - In fact, ``task.get_logger`` is no longer recommended, it is better
  366. to add a module-level logger to your tasks module.
  367. For example, like this:
  368. .. code-block:: python
  369. from celery.utils.log import get_task_logger
  370. logger = get_task_logger(__name__)
  371. @celery.task()
  372. def add(x, y):
  373. logger.debug('Adding %r + %r' % (x, y))
  374. return x + y
  375. The resulting logger will then inherit from the ``"celery.task"`` logger
  376. so that the current task name and id is included in logging output.
  377. - Redirected output from stdout/stderr is now logged to a "celery.redirected"
  378. logger.
  379. - In addition a few warnings.warn have been replaced with logger.warn.
  380. - Now avoids the 'no handlers for logger multiprocessing' warning
  381. Task registry no longer global
  382. ------------------------------
  383. Every Celery instance now has its own task registry.
  384. You can make apps share registries by specifying it::
  385. >>> app1 = Celery()
  386. >>> app2 = Celery(tasks=app1.tasks)
  387. Note that tasks are shared between registries by default, so that
  388. tasks will be added to every subsequently created task registry.
  389. As an alternative tasks can be private to specific task registries
  390. by setting the ``shared`` argument to the ``@task`` decorator::
  391. @celery.task(shared=False)
  392. def add(x, y):
  393. return x + y
  394. Abstract tasks are now lazily bound.
  395. ------------------------------------
  396. The :class:`~celery.task.Task` class is no longer bound to an app
  397. by default, it will first be bound (and configured) when
  398. a concrete subclass is created.
  399. This means that you can safely import and make task base classes,
  400. without also initializing the app environment::
  401. from celery.task import Task
  402. class DebugTask(Task):
  403. abstract = True
  404. def __call__(self, *args, **kwargs):
  405. print('CALLING %r' % (self, ))
  406. return self.run(*args, **kwargs)
  407. >>> DebugTask
  408. <unbound DebugTask>
  409. >>> @celery1.task(base=DebugTask)
  410. ... def add(x, y):
  411. ... return x + y
  412. >>> add.__class__
  413. <class add of <Celery default:0x101510d10>>
  414. Lazy task decorators
  415. --------------------
  416. The ``@task`` decorator is now lazy when used with custom apps.
  417. That is, if ``accept_magic_kwargs`` is enabled (herby called "compat mode"), the task
  418. decorator executes inline like before, however for custom apps the @task
  419. decorator now returns a special PromiseProxy object that is only evaluated
  420. on access.
  421. All promises will be evaluated when `app.finalize` is called, or implicitly
  422. when the task registry is first used.
  423. Smart `--app` option
  424. --------------------
  425. The :option:`--app` option now 'auto-detects'
  426. - If the provided path is a module it tries to get an
  427. attribute named 'celery'.
  428. - If the provided path is a package it tries
  429. to import a submodule named 'celery',
  430. and get the celery attribute from that module.
  431. E.g. if you have a project named 'proj' where the
  432. celery app is located in 'from proj.celery import celery',
  433. then the following will be equivalent::
  434. $ celery worker --app=proj
  435. $ celery worker --app=proj.celery:
  436. $ celery worker --app=proj.celery:celery
  437. In Other News
  438. -------------
  439. - New :setting:`CELERYD_WORKER_LOST_WAIT` to control the timeout in
  440. seconds before :exc:`billiard.WorkerLostError` is raised
  441. when a worker can not be signalled (Issue #595).
  442. Contributed by Brendon Crawford.
  443. - Redis event monitor queues are now automatically deleted (Issue #436).
  444. - App instance factory methods have been converted to be cached
  445. descriptors that creates a new subclass on access.
  446. This means that e.g. ``celery.Worker`` is an actual class
  447. and will work as expected when::
  448. class Worker(celery.Worker):
  449. ...
  450. - New signal: :signal:`task-success`.
  451. - Multiprocessing logs are now only emitted if the :envvar:`MP_LOG`
  452. environment variable is set.
  453. - The Celery instance can now be created with a broker URL
  454. .. code-block:: python
  455. celery = Celery(broker='redis://')
  456. - Result backends can now be set using an URL
  457. Currently only supported by redis. Example use::
  458. CELERY_RESULT_BACKEND = 'redis://localhost/1'
  459. - Heartbeat frequency now every 5s, and frequency sent with event
  460. The heartbeat frequency is now available in the worker event messages,
  461. so that clients can decide when to consider workers offline based on
  462. this value.
  463. - Module celery.actors has been removed, and will be part of cl instead.
  464. - Introduces new ``celery`` command, which is an entrypoint for all other
  465. commands.
  466. The main for this command can be run by calling ``celery.start()``.
  467. - Annotations now supports decorators if the key startswith '@'.
  468. E.g.:
  469. .. code-block:: python
  470. def debug_args(fun):
  471. @wraps(fun)
  472. def _inner(*args, **kwargs):
  473. print('ARGS: %r' % (args, ))
  474. return _inner
  475. CELERY_ANNOTATIONS = {
  476. 'tasks.add': {'@__call__': debug_args},
  477. }
  478. Also tasks are now always bound by class so that
  479. annotated methods end up being bound.
  480. - Bugreport now available as a command and broadcast command
  481. - Get it from a Python repl::
  482. >>> import celery
  483. >>> print(celery.bugreport())
  484. - Using the ``celery`` command-line program::
  485. $ celery report
  486. - Get it from remote workers::
  487. $ celery inspect report
  488. - Module ``celery.log`` moved to :mod:`celery.app.log`.
  489. - Module ``celery.task.control`` moved to :mod:`celery.app.control`.
  490. - New signal: :signal:`task-revoked`
  491. Sent in the main process when the task is revoked or terminated.
  492. - ``AsyncResult.task_id`` renamed to ``AsyncResult.id``
  493. - ``TasksetResult.taskset_id`` renamed to ``.id``
  494. - ``xmap(task, sequence)`` and ``xstarmap(task, sequence)``
  495. Returns a list of the results applying the task function to every item
  496. in the sequence.
  497. Example::
  498. >>> from celery import xstarmap
  499. >>> xstarmap(add, zip(range(10), range(10)).apply_async()
  500. [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
  501. - ``chunks(task, sequence, chunksize)``
  502. - ``group.skew(start=, stop=, step=)``
  503. Skew will skew the countdown for the individual tasks in a group,
  504. e.g. with a group::
  505. >>> g = group(add.s(i, i) for i in xrange(10))
  506. Skewing the tasks from 0 seconds to 10 seconds::
  507. >>> g.skew(stop=10)
  508. Will have the first task execute in 0 seconds, the second in 1 second,
  509. the third in 2 seconds and so on.
  510. - 99% test Coverage
  511. - :setting:`CELERY_QUEUES` can now be a list/tuple of :class:`~kombu.Queue`
  512. instances.
  513. Internally :attr:`@amqp.queues` is now a mapping of name/Queue instances,
  514. instead of converting on the fly.
  515. - Can now specify connection for :class:`@control.inspect`.
  516. .. code-block:: python
  517. from kombu import Connection
  518. i = celery.control.inspect(connection=Connection('redis://'))
  519. i.active_queues()
  520. - :setting:`CELERY_FORCE_EXECV` is now enabled by default.
  521. If the old behavior is wanted the setting can be set to False,
  522. or the new :option:`--no-execv` to :program:`celery worker`.
  523. - Deprecated module ``celery.conf`` has been removed.
  524. - The :setting:`CELERY_TIMEZONE` now always require the :mod:`pytz`
  525. library to be installed (exept if the timezone is set to `UTC`).
  526. - The Tokyo Tyrant backend has been removed and is no longer supported.
  527. - Now uses :func:`~kombu.common.maybe_declare` to cache queue declarations.
  528. - There is no longer a global default for the
  529. :setting:`CELERYBEAT_MAX_LOOP_INTERVAL` setting, it is instead
  530. set by individual schedulers.
  531. - Worker: now truncates very long message bodies in error reports.
  532. - No longer deepcopies exceptions when trying to serialize errors.
  533. - :envvar:`CELERY_BENCH` environment variable, will now also list
  534. memory usage statistics at worker shutdown.
  535. - Worker: now only ever use a single timer for all timing needs,
  536. and instead set different priorities.
  537. - An exceptions arguments are now safely pickled
  538. Contributed by Matt Long.
  539. - Worker/Celerybeat no longer logs the startup banner.
  540. Previously it would be logged with severity warning,
  541. now it's only written to stdout.
  542. - The ``contrib/`` directory in the distribution has been renamed to
  543. ``extra/``.
  544. - New signal: :signal:`task_revoked`
  545. - celery.contrib.migrate: Many improvements including
  546. filtering, queue migration, and support for acking messages on the broker
  547. migrating from.
  548. Contributed by John Watson.
  549. - Worker: Prefetch count increments are now optimized and grouped together.
  550. - Worker: No longer calls ``consume`` on the remote control command queue
  551. twice.
  552. Probably didn't cause any problems, but was unecessary.
  553. Internals
  554. ---------
  555. - ``app.broker_connection`` is now ``app.connection``
  556. Both names still work.
  557. - Compat modules are now generated dynamically upon use.
  558. These modules are ``celery.messaging``, ``celery.log``,
  559. ``celery.decorators`` and ``celery.registry``.
  560. - :mod:`celery.utils` refactored into multiple modules:
  561. :mod:`celery.utils.text`
  562. :mod:`celery.utils.imports`
  563. :mod:`celery.utils.functional`
  564. - Now using :mod:`kombu.utils.encoding` instead of
  565. :mod:`celery.utils.encoding`.
  566. - Renamed module ``celery.routes`` -> :mod:`celery.app.routes`.
  567. - Renamed package ``celery.db`` -> :mod:`celery.backends.database`.
  568. - Renamed module ``celery.abstract`` -> :mod:`celery.worker.bootsteps`.
  569. - Command-line docs are now parsed from the module docstrings.
  570. - Test suite directory has been reorganized.
  571. - :program:`setup.py` now reads docs from the :file:`requirements/` directory.
  572. - Celery commands no longer wraps output (Issue #700).
  573. Contributed by Thomas Johansson.
  574. .. _v300-experimental:
  575. Experimental
  576. ============
  577. :mod:`celery.contrib.methods`: Task decorator for methods
  578. ----------------------------------------------------------
  579. This is an experimental module containing a task
  580. decorator, and a task decorator filter, that can be used
  581. to create tasks out of methods::
  582. from celery.contrib.methods import task_method
  583. class Counter(object):
  584. def __init__(self):
  585. self.value = 1
  586. @celery.task(name='Counter.increment', filter=task_method)
  587. def increment(self, n=1):
  588. self.value += 1
  589. return self.value
  590. See :mod:`celery.contrib.methods` for more information.
  591. .. _v300-unscheduled-removals:
  592. Unscheduled Removals
  593. ====================
  594. Usually we don't make backward incompatible removals,
  595. but these removals should have no major effect.
  596. - The following settings have been renamed:
  597. - ``CELERYD_ETA_SCHEDULER`` -> ``CELERYD_TIMER``
  598. - ``CELERYD_ETA_SCHEDULER_PRECISION`` -> ``CELERYD_TIMER_PRECISION``
  599. .. _v300-deprecations:
  600. Deprecations
  601. ============
  602. See the :ref:`deprecation-timeline`.
  603. - The ``celery.backends.pyredis`` compat module has been removed.
  604. Use :mod:`celery.backends.redis` instead!
  605. - The following undocumented API's has been moved:
  606. - ``control.inspect.add_consumer`` -> :meth:`@control.add_consumer`.
  607. - ``control.inspect.cancel_consumer`` -> :meth:`@control.cancel_consumer`.
  608. - ``control.inspect.enable_events`` -> :meth:`@control.enable_events`.
  609. - ``control.inspect.disable_events`` -> :meth:`@control.disable_events`.
  610. This way ``inspect()`` is only used for commands that do not
  611. modify anything, while idempotent control commands that make changes
  612. are on the control objects.
  613. Fixes
  614. =====
  615. - Retry sqlalchemy backend operations on DatabaseError/OperationalError
  616. (Issue #634)
  617. - Tasks that called ``retry`` was not acknowledged if acks late was enabled
  618. Fix contributed by David Markey.
  619. - The message priority argument was not properly propagated to Kombu
  620. (Issue #708).
  621. Fix contributed by Eran Rundstein