Browse Source

Updates whatsnew-4.0 document

Ask Solem 9 years ago
parent
commit
f3b1f7fd54
2 changed files with 483 additions and 130 deletions
  1. 0 100
      docs/internals/deprecation.rst
  2. 483 30
      docs/whatsnew-4.0.rst

+ 0 - 100
docs/internals/deprecation.rst

@@ -7,106 +7,6 @@
 .. contents::
     :local:
 
-.. _deprecations-v4.0:
-
-Removals for version 4.0
-========================
-
-- Module ``celery.task.trace`` has been renamed to ``celery.app.trace``
-  as the ``celery.task`` package is being phased out.  The compat module
-  will be removed in version 4.0 so please change any import from::
-
-    from celery.task.trace import …
-
-  to::
-
-    from celery.app.trace import …
-
-- ``AsyncResult.serializable()`` and ``celery.result.from_serializable``
-  will be removed.
-
-    Use instead::
-
-        >>> tup = result.as_tuple()
-        >>> from celery.result import result_from_tuple
-        >>> result = result_from_tuple(tup)
-
-TaskSet
-~~~~~~~
-
-TaskSet has been renamed to group and TaskSet will be removed in version 4.0.
-
-Old::
-
-    >>> from celery.task import TaskSet
-
-    >>> TaskSet(add.subtask((i, i)) for i in xrange(10)).apply_async()
-
-New::
-
-    >>> from celery import group
-    >>> group(add.s(i, i) for i in xrange(10))()
-
-
-Magic keyword arguments
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The magic keyword arguments accepted by tasks will be removed
-in 4.0, so you should start rewriting any tasks
-using the ``celery.decorators`` module and depending
-on keyword arguments being passed to the task,
-for example::
-
-    from celery.decorators import task
-
-    @task()
-    def add(x, y, task_id=None):
-        print("My task id is %r" % (task_id,))
-
-should be rewritten into::
-
-    from celery import task
-
-    @task(bind=True)
-    def add(self, x, y):
-        print("My task id is {0.request.id}".format(self))
-
-:mod:`celery.result`
---------------------
-
-- ``BaseAsyncResult`` -> ``AsyncResult``.
-
-- ``TaskSetResult`` -> ``GroupResult``.
-
-- ``TaskSetResult.total`` -> ``len(GroupResult)``
-
-- ``TaskSetResult.taskset_id`` -> ``GroupResult.id``
-
-:mod:`celery.loader`
---------------------
-
-- ``current_loader()`` -> ``current_app.loader``
-
-- ``load_settings()`` -> ``current_app.conf``
-
-
-Settings
---------
-
-Logging Settings
-~~~~~~~~~~~~~~~~
-
-=====================================  =====================================
-**Setting name**                       **Replace with**
-=====================================  =====================================
-``CELERYD_LOG_LEVEL``                  :option:`--loglevel`
-``CELERYD_LOG_FILE``                   :option:`--logfile``
-``CELERYBEAT_LOG_LEVEL``               :option:`--loglevel`
-``CELERYBEAT_LOG_FILE``                :option:`--loglevel``
-``CELERYMON_LOG_LEVEL``                :option:`--loglevel`
-``CELERYMON_LOG_FILE``                 :option:`--loglevel``
-=====================================  =====================================
-
 .. _deprecations-v5.0:
 
 Removals for version 5.0

+ 483 - 30
docs/whatsnew-4.0.rst

@@ -55,12 +55,14 @@ Dropped support for Python 2.6
 
 Celery now requires Python 2.7 or later.
 
+Dropped support for Python 3.3
+------------------------------
+
+Celery now requires Python3 3.4 or later.
+
 JSON is now the default serializer
 ----------------------------------
 
-Using one logfile per process by default
-----------------------------------------
-
 The Task base class no longer automatically register tasks
 ----------------------------------------------------------
 
@@ -70,6 +72,51 @@ The metaclass has been removed blah blah
 Arguments now verified when calling a task
 ------------------------------------------
 
+Redis Events not backward compatible
+------------------------------------
+
+The Redis ``fanout_patterns`` and ``fanout_prefix`` transport
+options are now enabled by default, which means that workers
+running 4.0 cannot see workers running 3.1 and vice versa.
+
+They should still execute tasks as normally, so this is only
+related to monitoring events.
+
+To avoid this situation you can reconfigure the 3.1 workers (and clients)
+to enable these settings before you mix them with workers and clients
+running 4.x:
+
+.. code-block:: python
+
+    BROKER_TRANSPORT_OPTIONS = {
+        'fanout_patterns': True,
+        'fanout_prefix': True,
+    }
+
+Django: Autodiscover no longer takes arguments.
+-----------------------------------------------
+
+# e436454d02dcbba4f4410868ad109c54047c2c15
+
+Old command-line programs removed
+---------------------------------
+
+Installing Celery will no longer install the ``celeryd``,
+``celerybeat`` and ``celeryd-multi`` programs.
+
+This was announced with the release of Celery 3.1, but you may still
+have scripts pointing to the old names, so make sure you update them
+to use the new umbrella command.
+
++-------------------+--------------+-------------------------------------+
+| Program           | New Status   | Replacement                         |
++===================+==============+=====================================+
+| ``celeryd``       | **REMOVED**  | :program:`celery worker`            |
++-------------------+--------------+-------------------------------------+
+| ``celerybeat``    | **REMOVED**  | :program:`celery beat`              |
++-------------------+--------------+-------------------------------------+
+| ``celeryd-multi`` | **REMOVED**  | :program:`celery multi`             |
++-------------------+--------------+-------------------------------------+
 
 .. _v320-news:
 
@@ -79,6 +126,8 @@ News
 New Task Message Protocol
 =========================
 
+# e71652d384b1b5df2a4e6145df9f0efb456bc71c
+
 
 ``TaskProducer`` replaced by ``app.amqp.create_task_message`` and
 ``app.amqp.send_task_message``.
@@ -86,10 +135,109 @@ New Task Message Protocol
 - Worker stores results for internal errors like ``ContentDisallowed``, and
   exceptions occurring outside of the task function.
 
+- Worker stores results and sends monitoring events for unknown task names
+
+- shadow
+
+- argsrepr
+
+- Support for very long chains
+
+- parent_id / root_id
+
+
+Prefork: Tasks now log from the child process
+=============================================
+
+Logging of task success/failure now happens from the child process
+actually executing the task, which means that logging utilities
+like Sentry can get full information about tasks that fail, including
+variables in the traceback.
+
+Prefork: One logfile per child process
+======================================
+
+Init scrips and :program:`celery multi` now uses the `%I` logfile format
+option (e.g. :file:`/var/log/celery/%n%I.log`) to ensure each child
+process has a separate log file to avoid race conditions.
+
+You are encouraged to upgrade your init scripts and multi arguments
+to do so also.
 
 Canvas Refactor
 ===============
 
+# BLALBLABLA
+d79dcd8e82c5e41f39abd07ffed81ca58052bcd2
+1e9dd26592eb2b93f1cb16deb771cfc65ab79612
+e442df61b2ff1fe855881c1e2ff9acc970090f54
+0673da5c09ac22bdd49ba811c470b73a036ee776
+
+- Now unrolls groups within groups into a single group (Issue #1509).
+- chunks/map/starmap tasks now routes based on the target task
+- chords and chains can now be immutable.
+- Fixed bug where serialized signature were not converted back into
+  signatures (Issue #2078)
+
+    Fix contributed by Ross Deane.
+
+- Fixed problem where chains and groups did not work when using JSON
+  serialization (Issue #2076).
+
+    Fix contributed by Ross Deane.
+
+- Creating a chord no longer results in multiple values for keyword
+  argument 'task_id'" (Issue #2225).
+
+    Fix contributed by Aneil Mallavarapu
+
+- Fixed issue where the wrong result is returned when a chain
+  contains a chord as the penultimate task.
+
+    Fix contributed by Aneil Mallavarapu
+
+- Special case of ``group(A.s() | group(B.s() | C.s()))`` now works.
+
+- Chain: Fixed bug with incorrect id set when a subtask is also a chain.
+
+Schedule tasks based on sunrise, sunset, dawn and dusk.
+=======================================================
+
+See :ref:`beat-solar` for more information.
+
+Contributed by Mark Parncutt.
+
+App can now configure periodic tasks
+====================================
+
+# bc18d0859c1570f5eb59f5a969d1d32c63af764b
+# 132d8d94d38f4050db876f56a841d5a5e487b25b
+
+RabbitMQ Priority queue support
+===============================
+
+# 1d4cbbcc921aa34975bde4b503b8df9c2f1816e0
+
+Contributed by Gerald Manipon.
+
+Prefork: Limits for child process resident memory size.
+=======================================================
+
+This version introduces the new :setting:`worker_max_memory_per_child` setting,
+which BLA BLA BLA
+
+# 5cae0e754128750a893524dcba4ae030c414de33
+
+Contributed by Dave Smith.
+
+Redis: New optimized chord join implementation.
+===============================================
+
+This was an experimental feature introduced in Celery 3.1,
+but is now enabled by default.
+
+?new_join BLABLABLA
+
 Riak Result Backend
 ===================
 
@@ -99,6 +247,11 @@ Bla bla
 
 - blah blah
 
+CouchDB Result Backend
+======================
+
+Contributed by Nathan Van Gheem
+
 New Cassandra Backend
 =====================
 
@@ -112,35 +265,53 @@ to be using the new driver.
 Event Batching
 ==============
 
-Events are now buffered in the worker and sent as a list
+Events are now buffered in the worker and sent as a list, and
+events are sent as transient messages by default so that they are not written
+to disk by RabbitMQ.
+
+03399b4d7c26fb593e61acf34f111b66b340ba4e
 
 
 Task.replace
 ============
- Task.replace changed, removes Task.replace_in_chord.
 
-    The two methods had almost the same functionality, but the old Task.replace
-    would force the new task to inherit the callbacks/errbacks of the existing
-    task.
+Task.replace changed, removes Task.replace_in_chord.
+
+The two methods had almost the same functionality, but the old Task.replace
+would force the new task to inherit the callbacks/errbacks of the existing
+task.
 
-    If you replace a node in a tree, then you would not expect the new node to
-    inherit the children of the old node, so this seems like unexpected
-    behavior.
+If you replace a node in a tree, then you would not expect the new node to
+inherit the children of the old node, so this seems like unexpected
+behavior.
 
-    So self.replace(sig) now works for any task, in addition sig can now
-    be a group.
+So self.replace(sig) now works for any task, in addition sig can now
+be a group.
 
-    Groups are automatically converted to a chord, where the callback
-    will "accumulate" the results of the group tasks.
+Groups are automatically converted to a chord, where the callback
+will "accumulate" the results of the group tasks.
 
-    A new builtin task (`celery.accumulate` was added for this purpose)
+A new builtin task (`celery.accumulate` was added for this purpose)
 
-    Closes #817
+Closes #817
 
 
 Optimized Beat implementation
 =============================
 
+heapq
+20340d79b55137643d5ac0df063614075385daaa
+
+Contributed by Ask Solem and Alexander Koshelev.
+
+
+Task Autoretry Decorator
+========================
+
+75246714dd11e6c463b9dc67f4311690643bff24
+
+Contributed by Dmitry Malinovsky.
+
 In Other News
 -------------
 
@@ -155,21 +326,182 @@ In Other News
 - **Programs**: ``%n`` format for :program:`celery multi` is now synonym with
   ``%N`` to be consistent with :program:`celery worker`.
 
-- **Programs**: celery inspect/control now supports --json argument
+- **Programs**: celery inspect/control now supports ``--json`` argument to
+  give output in json format.
+
+- **Programs**: :program:`celery inspect registered` now ignores built-in
+  tasks.
+
+- **Programs**: New :program:`celery logtool`: Utility for filtering and parsing
+  celery worker logfiles
+
+- **Redis Transport**: The Redis transport now supports the
+  :setting:`broker_use_ssl` option.
 
-- **Programs**: :program:`celery logtool`: Utility for filtering and parsing celery worker logfiles
+- **Worker**: Worker now only starts the remote control command consumer if the
+  broker transport used actually supports them.
 
 - **Worker**: Gossip now sets ``x-message-ttl`` for event queue to heartbeat_interval s.
-  (Iss ue #2005).
+  (Issue #2005).
 
-- **App**: New signals
+- **Worker**: Now preserves exit code (Issue #2024).
+
+- **Worker**: Loglevel for unrecoverable errors changed from ``error`` to
+  ``critical``.
+
+- **Worker**: Improved rate limiting accuracy.
+
+- **Worker**: Account for missing timezone information in task expires field.
+
+    Fix contributed by Albert Wang.
+
+- **Worker**: The worker no longer has a ``Queues`` bootsteps, as it is now
+    superfluous.
+
+- **Tasks**: New :setting:`task_reject_on_worker_lost` setting, and
+  :attr:`~@Task.reject_on_worker_lost` task attribute decides what happens
+  when the child worker process executing a late ack task is terminated.
+
+    Contributed by Michael Permana.
+
+- **App**: New signals for app configuration/finalization:
 
     - :data:`app.on_configure <@on_configure>`
     - :data:`app.on_after_configure <@on_after_configure>`
     - :data:`app.on_after_finalize <@on_after_finalize>`
 
+- **Task**: New task signals for rejected task messages:
+
+    - :data:`celery.signals.task_rejected`.
+    - :data:`celery.signals.task_unknown`.
+
+- **Events**: Event messages now uses the RabbitMQ ``x-message-ttl`` option
+    to ensure older event messages are discarded.
+
+    The default is 5 seconds, but can be changed using the
+    :setting:`event_queue_ttl` setting.
+
+- **Events**: Event monitors now sets the :setting:`event_queue_expires`
+  setting by default.
+
+    The queues will now expire after 60 seconds after the monitor stops
+    consuming from it.
+
 - **Canvas**: ``chunks``/``map``/``starmap`` are now routed based on the target task.
 
+- **Canvas**: ``Signature.link`` now works when argument is scalar (not a list)
+    (Issue #2019).
+
+- **App**: The application can now change how task names are generated using
+    the :meth:`~@gen_task_name` method.
+
+    Contributed by Dmitry Malinovsky.
+
+- **Tasks**: ``Task.subtask`` renamed to ``Task.signature`` with alias.
+
+- **Tasks**: ``Task.subtask_from_request`` renamed to
+  ``Task.signature_from_request`` with alias.
+
+- **Tasks**: The ``delivery_mode`` attribute for :class:`kombu.Queue` is now
+  respected (Issue #1953).
+
+- **Tasks**: Routes in :setting:`task-routes` can now specify a
+  :class:`~kombu.Queue` instance directly.
+
+    Example:
+
+    .. code-block:: python
+
+        task_routes = {'proj.tasks.add': {'queue': Queue('add')}}
+
+- **Tasks**: ``AsyncResult`` now raises :exc:`ValueError` if task_id is None.
+  (Issue #1996).
+
+- **Tasks**: ``result.get()`` now supports an ``on_message`` argument to set a
+  callback to be called for every message received.
+
+- **Tasks**: New abstract classes added:
+
+    - :class:`~celery.utils.abstract.CallableTask`
+
+        Looks like a task.
+
+    - :class:`~celery.utils.abstract.CallableSignature`
+
+        Looks like a task signature.
+
+- **Programs**: :program:`celery multi` now passes through `%i` and `%I` log
+  file formats.
+
+- **Programs**: A new command line option :option:``--executable`` is now
+  available for daemonizing programs.
+
+    Contributed by Bert Vanderbauwhede.
+
+- **Programs**: :program:`celery worker` supports new
+  :option:`--prefetch-multiplier` option.
+
+    Contributed by Mickaël Penhard.
+
+- **Prefork**: Prefork pool now uses ``poll`` instead of ``select`` where
+  available (Issue #2373).
+
+- **Tasks**: New :setting:`email_charset` setting allows for changing
+  the charset used for outgoing error emails.
+
+    Contributed by Vladimir Gorbunov.
+
+- **Worker**: Now respects :setting:`broker_connection_retry` setting.
+
+    Fix contributed by Nat Williams.
+
+- **Worker**: Autoscale did not always update keepalive when scaling down.
+
+    Fix contributed by Philip Garnero.
+
+- **General**: Dates are now always timezone aware even if
+  :setting:`enable_utc` is disabled (Issue #943).
+
+    Fix contributed by Omer Katz.
+
+- **Result Backends**: The redis result backend now has a default socket
+   timeout of 5 seconds.
+
+    The default can be changed using the new :setting:`redis_socket_timeout`
+    setting.
+
+    Contributed by Raghuram Srinivasan.
+
+- **Result Backends**: RPC Backend result queues are now auto delete by
+  default (Issue #2001).
+
+- **Result Backends**: MongoDB now supports setting the
+  :setting:`result_serialzier` setting to ``bson`` to use the MongoDB
+  libraries own serializer.
+
+    Contributed by Davide Quarta.
+
+- **Result Backends**: SQLAlchemy result backend now ignores all result
+   engine options when using NullPool (Issue #1930).
+
+- **Result Backends**: MongoDB URI handling has been improved to use
+    database name, user and password from the URI if provided.
+
+    Contributed by Samuel Jaillet.
+
+- **Result Backends**: Fix problem with rpc/amqp backends where exception
+    was not deserialized properly with the json serializer (Issue #2518).
+
+    Fix contributed by Allard Hoeve.
+
+- **General**: All Celery exceptions/warnings now inherit from common
+  :class:`~celery.exceptions.CeleryException`/:class:`~celery.exceptions.CeleryWarning`.
+  (Issue #2643).
+
+- **Tasks**: Task retry now also throws in eager mode.
+
+    Fix contributed by Feanil Patel.
+
 - Apps can now define how tasks are named (:meth:`@gen_task_name`).
 
     Contributed by Dmitry Malinovsky
@@ -179,16 +511,143 @@ In Other News
 - Beat: ``Scheduler.Publisher``/``.publisher`` renamed to
   ``.Producer``/``.producer``.
 
+Unscheduled Removals
+====================
+
+- The experimental :mod:`celery.contrib.methods` feature has been removed,
+  as there were far many bugs in the implementation to be useful.
+
+- The CentOS init scripts have been removed.
+
+    These did not really add any features over the generic init scripts,
+    so you are encouraged to use them instead, or something like
+    ``supervisord``.
+
 
 .. _v320-removals:
 
 Scheduled Removals
 ==================
 
-- The module ``celery.task.trace`` has been removed as scheduled for this
-  version.
+Modules
+-------
+
+- Module ``celery.worker.job`` has been renamed to :mod:`celery.worker.request`.
+
+    This was an internal module so should not have any effect.
+    It is now part of the public API so should not change again.
+
+- Module ``celery.task.trace`` has been renamed to ``celery.app.trace``
+  as the ``celery.task`` package is being phased out.  The compat module
+  will be removed in version 4.0 so please change any import from::
+
+    from celery.task.trace import …
+
+  to::
+
+    from celery.app.trace import …
+
+- Old compatibility aliases in the :mod:`celery.loaders` module
+  has been removed.
+
+    - Removed ``celery.loaders.current_loader()``, use: ``current_app.loader``
+
+    - Removed ``celery.loaders.load_settings()``, use: ``current_app.conf``
+
+Result
+------
+
+- ``AsyncResult.serializable()`` and ``celery.result.from_serializable``
+    has been removed:
+
+    Use instead:
+
+    .. code-block:: pycon
+
+        >>> tup = result.as_tuple()
+        >>> from celery.result import result_from_tuple
+        >>> result = result_from_tuple(tup)
+
+- Removed ``BaseAsyncResult``, use ``AsyncResult`` for instance checks
+  instead.
+
+- Removed ``TaskSetResult``, use ``GroupResult`` instead.
+
+    - ``TaskSetResult.total`` -> ``len(GroupResult)``
+
+    - ``TaskSetResult.taskset_id`` -> ``GroupResult.id``
+
+
+TaskSet
+-------
+
+TaskSet has been renamed to group and TaskSet will be removed in version 4.0.
+
+Old::
 
-- Magic keyword arguments no longer supported.
+    >>> from celery.task import TaskSet
+
+    >>> TaskSet(add.subtask((i, i)) for i in xrange(10)).apply_async()
+
+New::
+
+    >>> from celery import group
+    >>> group(add.s(i, i) for i in xrange(10))()
+
+
+Magic keyword arguments
+-----------------------
+
+Support for the very old magic keyword arguments accepted by tasks has finally
+been in 4.0.
+
+If you are still using these you have to rewrite any task still
+using the old ``celery.decorators`` module and depending
+on keyword arguments being passed to the task,
+for example::
+
+    from celery.decorators import task
+
+    @task()
+    def add(x, y, task_id=None):
+        print("My task id is %r" % (task_id,))
+
+should be rewritten into::
+
+    from celery import task
+
+    @task(bind=True)
+    def add(self, x, y):
+        print("My task id is {0.request.id}".format(self))
+
+Settings
+--------
+
+The following settings have been removed, and is no longer supported:
+
+Logging Settings
+~~~~~~~~~~~~~~~~
+
+=====================================  =====================================
+**Setting name**                       **Replace with**
+=====================================  =====================================
+``CELERYD_LOG_LEVEL``                  :option:`--loglevel`
+``CELERYD_LOG_FILE``                   :option:`--logfile``
+``CELERYBEAT_LOG_LEVEL``               :option:`--loglevel`
+``CELERYBEAT_LOG_FILE``                :option:`--loglevel``
+``CELERYMON_LOG_LEVEL``                celerymon is deprecated, use flower.
+``CELERYMON_LOG_FILE``                 celerymon is deprecated, use flower.
+``CELERYMON_LOG_FORMAT``               celerymon is deprecated, use flower.
+=====================================  =====================================
+
+Task Settings
+~~~~~~~~~~~~~~
+
+=====================================  =====================================
+**Setting name**                       **Replace with**
+=====================================  =====================================
+``CELERY_CHORD_PROPAGATES``            N/a
+=====================================  =====================================
 
 .. _v320-deprecations:
 
@@ -202,9 +661,3 @@ See the :ref:`deprecation-timeline`.
 Fixes
 =====
 
-.. _v320-internal:
-
-Internal changes
-================
-
-- Module ``celery.worker.job`` has been renamed to :mod:`celery.worker.request`.