Selaa lähdekoodia

Fixes rst error

Ask Solem 13 vuotta sitten
vanhempi
commit
fa5e7f78d8

BIN
_images/math/5bec3f9bd85f65932e771e2f003f923f49ed1a94.png


+ 513 - 0
_sources/userguide/calling.txt

@@ -0,0 +1,513 @@
+.. _guide-calling:
+
+===============
+ Calling Tasks
+===============
+
+.. contents::
+    :local:
+    :depth: 1
+
+
+.. _calling-basics:
+
+Basics
+======
+
+This document describes Celery's uniform "Calling API"
+used by task instances and the :ref:`canvas <guide-canvas>`.
+
+The API defines a standard set of execution options, as well as three methods:
+
+    - ``apply_async(args[, kwargs[, ...]])``
+
+        Sends a task message.
+
+    - ``delay(*args, **kwargs)``
+
+        Shortcut to send a task message, but does not support execution
+        options.
+
+    - ``apply()``
+
+        Does not send a message but executes the task inline instead.
+
+.. _calling-cheat:
+
+.. topic:: Quick Cheat Sheet
+
+    - ``T.delay(arg, kwarg=value)``
+        always a shortcut to ``.apply_async``.
+
+    - ``T.apply_async((arg, ), {"kwarg": value})``
+
+    - ``T.apply_async(countdown=10)``
+        executes 10 seconds from now.
+
+    - ``T.apply_async(eta=now + timedelta(seconds=10))``
+        executes 10 seconds from now, specifed using ``eta``
+
+    - ``T.apply_async(countdown=60, expires=120)``
+        executes in one minute from now, but expires after 2 minutes.
+
+    - ``T.apply_async(expires=now + timedelta(days=2))``
+        expires in 2 days, set using :class:`~datetime.datetime`.
+
+
+Example
+-------
+
+The :meth:`~@Task.delay` method is convenient as it looks like calling a regular
+function:
+
+.. code-block:: python
+
+    task.delay(arg1, arg2, kwarg1="x", kwarg2="y")
+
+Using :meth:`~@Task.apply_async` instead we have to write::
+
+.. code-block:: python
+
+    task.apply_async(args=[arg1, arg2], kwargs={"kwarg1": "x", "kwarg2": "y"})
+
+.. sidebar:: Tip
+
+    If the task is not registered in the current process
+    you can use :meth:`~@send_task` to call the task by name instead.
+
+
+So `delay` is clearly convenient, but it doesn't give you as much control as using
+`apply_async`.  With `apply_async` you can override the execution options
+available as attributes on the :class:`~@Task` class (see :ref:`task-options`).
+In addition you can set countdown/eta, task expiry, provide a custom broker
+connection and more.
+
+The rest of this document will go into the task execution
+options in detail.  All examples use a task
+called `add`, returning the sum of two arguments:
+
+.. code-block:: python
+
+    @celery.task()
+    def add(x, y):
+        return x + y
+
+
+.. topic:: There's another way...
+
+    You will learn more about this later while reading about the :ref:`Canvas
+    <guide-canvas>`, but :class:`~celery.subtask`'s are objects used to pass around
+    the signature of a task invocation, (for example to send it over the
+    network), and they also support the Calling API:
+
+    .. code-block:: python
+
+        task.s(arg1, arg2, kwarg1="x", kwargs2="y").apply_async()
+
+.. _calling-links:
+
+Linking (callbacks/errbacks)
+============================
+
+Celery supports linking tasks together so that one task follows another.
+The callback task will be applied with the result of the parent task
+as a partial argument:
+
+.. code-block:: python
+
+    add.apply_async((2, 2), link=add.s(16))
+
+.. sidebar:: What is ``s``?
+
+    The ``add.s`` call used here is called a subtask, we talk
+    more about subtasks in the :ref:`canvas guide <guide-canvas>`,
+    where you can also learn about :class:`~celery.chain`, which
+    is a simpler way to chain tasks together.
+
+    In practice the ``link`` execution option is considered an internal
+    primitive, and you will probably not use it directly, but
+    rather use chains instead.
+
+Here the result of the first task (4) will be sent to a new
+task that adds 16 to the previous result, forming the expression
+:math:`(2 + 2) + 16 = 20`
+
+
+You can also cause a callback to be applied if task raises an exception
+(*errback*), but this behaves differently from a regular callback
+in that it will be passed the id of the parent task, not the result.
+This is because it may not always be possible to serialize
+the exception raised, and so this way the error callback requires
+a result backend to be enabled, and the task must retrieve the result
+of the task instead.
+
+This is an example error callback:
+
+.. code-block:: python
+
+    @celery.task()
+    def error_handler(uuid):
+        result = AsyncResult(uuid)
+        exc = result.get(propagate=False)
+        print("Task %r raised exception: %r\n%r" % (
+              exc, result.traceback))
+
+it can be added to the task using the ``link_error`` execution
+option:
+
+.. code-block:: python
+
+    add.apply_async((2, 2), link_error=error_handler.s())
+
+
+In addition, both the ``link`` and ``link_error`` options can be expressed
+as a list::
+
+    add.apply_async((2, 2), link=[add.s(16), other_task.s()])
+
+The callbacks/errbacks will then be called in order, and all
+callbacks will be called with the return value of the parent task
+as a partial argument.
+
+.. _calling-eta:
+
+ETA and countdown
+=================
+
+The ETA (estimated time of arrival) lets you set a specific date and time that
+is the earliest time at which your task will be executed.  `countdown` is
+a shortcut to set eta by seconds into the future.
+
+.. code-block:: python
+
+    >>> result = add.apply_async((2, 2), countdown=3)
+    >>> result.get()    # this takes at least 3 seconds to return
+    20
+
+The task is guaranteed to be executed at some time *after* the
+specified date and time, but not necessarily at that exact time.
+Possible reasons for broken deadlines may include many items waiting
+in the queue, or heavy network latency.  To make sure your tasks
+are executed in a timely manner you should monitor the queue for congestion. Use
+Munin, or similar tools, to receive alerts, so appropriate action can be
+taken to ease the workload.  See :ref:`monitoring-munin`.
+
+While `countdown` is an integer, `eta` must be a :class:`~datetime.datetime`
+object, specifying an exact date and time (including millisecond precision,
+and timezone information):
+
+.. code-block:: python
+
+    >>> from datetime import datetime, timedelta
+
+    >>> tomorrow = datetime.utcnow() + timedelta(days=1)
+    >>> add.apply_async((2, 2), eta=tomorrow)
+
+.. _calling-expiration:
+
+Expiration
+==========
+
+The `expires` argument defines an optional expiry time,
+either as seconds after task publish, or a specific date and time using
+:class:`~datetime.datetime`:
+
+.. code-block:: python
+
+    >>> # Task expires after one minute from now.
+    >>> add.apply_async((10, 10), expires=60)
+
+    >>> # Also supports datetime
+    >>> from datetime import datetime, timedelta
+    >>> add.apply_async((10, 10), kwargs,
+    ...                 expires=datetime.now() + timedelta(days=1)
+
+
+When a worker receives an expired task it will mark
+the task as :state:`REVOKED` (:exc:`~@TaskRevokedError`).
+
+.. _calling-retry:
+
+Message Sending Retry
+=====================
+
+Celery will automatically retry sending messages in the event of connection
+failure, and retry behavior can be configured -- like how often to retry, or a maximum
+number of retries -- or disabled all together.
+
+To disable retry you can set the ``retry`` execution option to :const:`False`:
+
+.. code-block:: python
+
+    add.apply_async((2, 2), retry=False)
+
+.. topic:: Related Settings
+
+    .. hlist::
+        :columns: 2
+
+        - :setting:`CELERY_TASK_PUBLISH_RETRY`
+        - :setting:`CELERY_TASK_PUBLISH_RETRY_POLICY`
+
+
+
+Retry Policy
+------------
+
+A retry policy is a mapping that controls how retries behave,
+and can contain the following keys:
+
+- `max_retries`
+
+    Maximum number of retries before giving up, in this case the
+    exception that caused the retry to fail will be raised.
+
+    A value of 0 or :const:`None` means it will retry forever.
+
+    The default is to retry 3 times.
+
+- `interval_start`
+
+    Defines the number of seconds (float or integer) to wait between
+    retries.  Default is 0, which means the first retry will be
+    instantaneous.
+
+- `interval_step`
+
+    On each consecutive retry this number will be added to the retry
+    delay (float or integer).  Default is 0.2.
+
+- `interval_max`
+
+    Maximum number of seconds (float or integer) to wait between
+    retries.  Default is 0.2.
+
+For example, the default policy correlates to:
+
+.. code-block:: python
+
+    add.apply_async((2, 2), retry=True, retry_policy={
+        "max_retries": 3,
+        "interval_start": 0,
+        "interval_step": 0.2,
+        "interval_max": 0.2,
+    })
+
+the maximum time spent retrying will be 0.4 seconds.  It is set relatively
+short by default because a connection failure could lead to a retry pile effect
+if the broker connection is down: e.g. many web server processes waiting
+to retry blocking other incoming requests.
+
+.. _calling-serializers:
+
+Serializers
+===========
+
+.. sidebar::  Security
+
+    The pickle module allows for execution of arbitrary functions,
+    please see the :ref:`security guide <guide-security>`.
+
+    Celery also comes with a special serializer that uses
+    cryptography to sign your messages.
+
+Data transferred between clients and workers needs to be serialized,
+so every message in Celery has a ``content_type`` header that
+describes the serialization method used to encode it.
+
+The default serializer is :mod:`pickle`, but you can
+change this using the :setting:`CELERY_TASK_SERIALIZER` setting,
+or for each individual task, or even per message.
+
+There's built-in support for :mod:`pickle`, `JSON`, `YAML`
+and `msgpack`, and you can also add your own custom serializers by registering
+them into the Kombu serializer registry (see `Kombu: Serialization of Data`_).
+
+.. _`Kombu: Serialization of Data`:
+    http://packages.python.org/kombu/introduction.html#serialization-of-data
+
+Each option has its advantages and disadvantages.
+
+json -- JSON is supported in many programming languages, is now
+    a standard part of Python (since 2.6), and is fairly fast to decode
+    using the modern Python libraries such as :mod:`cjson` or :mod:`simplejson`.
+
+    The primary disadvantage to JSON is that it limits you to the following
+    data types: strings, Unicode, floats, boolean, dictionaries, and lists.
+    Decimals and dates are notably missing.
+
+    Also, binary data will be transferred using Base64 encoding, which will
+    cause the transferred data to be around 34% larger than an encoding which
+    supports native binary types.
+
+    However, if your data fits inside the above constraints and you need
+    cross-language support, the default setting of JSON is probably your
+    best choice.
+
+    See http://json.org for more information.
+
+pickle -- If you have no desire to support any language other than
+    Python, then using the pickle encoding will gain you the support of
+    all built-in Python data types (except class instances), smaller
+    messages when sending binary files, and a slight speedup over JSON
+    processing.
+
+    See http://docs.python.org/library/pickle.html for more information.
+
+yaml -- YAML has many of the same characteristics as json,
+    except that it natively supports more data types (including dates,
+    recursive references, etc.)
+
+    However, the Python libraries for YAML are a good bit slower than the
+    libraries for JSON.
+
+    If you need a more expressive set of data types and need to maintain
+    cross-language compatibility, then YAML may be a better fit than the above.
+
+    See http://yaml.org/ for more information.
+
+msgpack -- msgpack is a binary serialization format that is closer to JSON
+    in features.  It is very young however, and support should be considered
+    experimental at this point.
+
+    See http://msgpack.org/ for more information.
+
+The encoding used is available as a message header, so the worker knows how to
+deserialize any task.  If you use a custom serializer, this serializer must
+be available for the worker.
+
+The following order is used to decide which serializer
+to use when sending a task:
+
+    1. The `serializer` execution option.
+    2. The :attr:`@-Task.serializer` attribute
+    3. The :setting:`CELERY_TASK_SERIALIZER` setting.
+
+
+Example setting a custom serializer for a single task invocation:
+
+.. code-block:: python
+
+    >>> add.apply_async((10, 10), serializer="json")
+
+.. _calling-compression:
+
+Compression
+===========
+
+Celery can compress the messages using either *gzip*, or *bzip2*.
+You can also create your own compression schemes and register
+them in the :func:`kombu compression registry <kombu.compression.register>`.
+
+The following order is used to decide which compression scheme
+to use when sending a task:
+
+    1. The `compression` execution option.
+    2. The :attr:`@-Task.compression` attribute.
+    3. The :setting:`CELERY_MESSAGE_COMPRESSION` attribute.
+
+Example specifying the compression used when calling a task::
+
+    >>> add.apply_async((2, 2), compression="zlib")
+
+.. _calling-connections:
+
+Connections
+===========
+
+.. sidebar:: Automatic Pool Support
+
+    Since version 2.3 there is support for automatic connection pools,
+    so you don't have to manually handle connections and publishers
+    to reuse connections.
+
+    The connection pool is enabled by default since version 2.5.
+
+    See the :setting:`BROKER_POOL_LIMIT` setting for more information.
+
+You can handle the connection manually by creating a
+publisher:
+
+.. code-block:: python
+
+
+    results = []
+    with add.app.pool.acquire(block=True) as connection:
+        with add.get_publisher(connection) as publisher:
+            try:
+                for args in numbers:
+                    res = add.apply_async((2, 2), publisher=publisher)
+                    results.append(res)
+    print([res.get() for res in results])
+
+
+Though this particular example is much better expressed as a group:
+
+.. code-block:: python
+
+    >>> from celery import group
+
+    >>> numbers = [(2, 2), (4, 4), (8, 8), (16, 16)]
+    >>> res = group(add.subtask(n) for i in numbers).apply_async()
+
+    >>> res.get()
+    [4, 8, 16, 32]
+
+.. _calling-routing:
+
+Routing options
+===============
+
+Celery can route tasks to different queues.
+
+Simple routing (name <-> name) is accomplished using the ``queue`` option::
+
+    add.apply_async(queue="priority.high")
+
+You can then assign workers to the ``priority.high`` queue by using
+the workers :option:`-Q` argument::
+
+    $ celery worker -l info -Q celery,priority.high
+
+.. seealso::
+
+    Hard-coding queue names in code is not recommended, the best practice
+    is to use configuration routers (:setting:`CELERY_ROUTES`).
+
+    To find out more about routing, please see :ref:`guide-routing`.
+
+Advanced Options
+----------------
+
+These options are for advanced users who want to take use of
+AMQP's full routing capabilities. Interested parties may read the
+:ref:`routing guide <guide-routing>`.
+
+- exchange
+
+    Name of exchange (or a :class:`kombu.entity.Exchange`) to
+    send the message to.
+
+- routing_key
+
+    Routing key used to determine.
+
+- mandatory
+
+    This sets the delivery to be mandatory.  An exception will be raised
+    if there are no running workers able to take on the task.
+
+    Not supported by :mod:`amqplib`.
+
+- immediate
+
+    Request immediate delivery. Will raise an exception
+    if the task cannot be routed to a worker immediately.
+
+    Not supported by :mod:`amqplib`.
+
+- priority
+
+    A number between `0` and `9`, where `0` is the highest priority.
+
+    Supported by: redis, beanstalk

+ 269 - 0
_sources/userguide/canvas.txt

@@ -0,0 +1,269 @@
+.. _guide-canvas:
+
+============================
+ Canvas: Building Workflows
+============================
+
+.. contents::
+    :local:
+
+.. _canvas-subtasks:
+
+Subtasks
+========
+
+.. versionadded:: 2.0
+
+The :class:`~celery.subtask` type is used to wrap the arguments and
+execution options for a single task invocation:
+
+.. code-block:: python
+
+    from celery import subtask
+
+    subtask(task_name_or_cls, args, kwargs, options)
+
+For convenience every task also has a shortcut to create subtasks:
+
+.. code-block:: python
+
+    task.subtask(args, kwargs, options)
+
+:class:`~celery.subtask` is actually a :class:`dict` subclass,
+which means it can be serialized with JSON or other encodings that doesn't
+support complex Python objects.
+
+Also it can be regarded as a type, as the following usage works::
+
+    >>> s = subtask("tasks.add", args=(2, 2), kwargs={})
+
+    >>> subtask(dict(s))  # coerce dict into subtask
+
+This makes it excellent as a means to pass callbacks around to tasks.
+
+.. _canvas-callbacks:
+
+Callbacks
+---------
+
+Callbacks can be added to any task using the ``link`` argument
+to ``apply_async``:
+
+    add.apply_async((2, 2), link=other_task.subtask())
+
+The callback will only be applied if the task exited successfully,
+and it will be applied with the return value of the parent task as argument.
+
+
+The best thing is that any arguments you add to `subtask`,
+will be prepended to the arguments specified by the subtask itself!
+
+If you have the subtask::
+
+    >>> add.subtask(args=(10, ))
+
+`subtask.delay(result)` becomes::
+
+    >>> add.apply_async(args=(result, 10))
+
+...
+
+Now let's call our ``add`` task with a callback using partial
+arguments::
+
+    >>> add.apply_async((2, 2), link=add.subtask((8, )))
+
+As expected this will first launch one task calculating :math:`2 + 2`, then
+another task calculating :math:`4 + 8`.
+
+.. _canvas-group:
+
+Groups
+======
+
+The :class:`~celery.group` enables easy invocation of several
+tasks at once, and is then able to join the results in the same order as the
+tasks were invoked.
+
+``group`` takes a list of :class:`~celery.subtask`'s::
+
+    >>> from celery import group
+    >>> from tasks import add
+
+    >>> job = group([
+    ...             add.subtask((2, 2)),
+    ...             add.subtask((4, 4)),
+    ...             add.subtask((8, 8)),
+    ...             add.subtask((16, 16)),
+    ...             add.subtask((32, 32)),
+    ... ])
+
+    >>> result = job.apply_async()
+
+    >>> result.ready()  # have all subtasks completed?
+    True
+    >>> result.successful() # were all subtasks successful?
+    True
+    >>> result.join()
+    [4, 8, 16, 32, 64]
+
+The first argument can alternatively be an iterator, like::
+
+    >>> group(add.subtask((i, i)) for i in range(100))
+
+.. _canvas-group-results:
+
+Group Results
+-------------
+
+When a  :class:`~celery.group` is applied it returns a
+:class:`~celery.result.GroupResult` object.
+
+:class:`~celery.result.GroupResult` takes a list of
+:class:`~celery.result.AsyncResult` instances and operates on them as if it was a
+single task.
+
+It supports the following operations:
+
+* :meth:`~celery.result.GroupResult.successful`
+
+    Returns :const:`True` if all of the subtasks finished
+    successfully (e.g. did not raise an exception).
+
+* :meth:`~celery.result.GroupResult.failed`
+
+    Returns :const:`True` if any of the subtasks failed.
+
+* :meth:`~celery.result.GroupResult.waiting`
+
+    Returns :const:`True` if any of the subtasks
+    is not ready yet.
+
+* :meth:`~celery.result.GroupResult.ready`
+
+    Return :const:`True` if all of the subtasks
+    are ready.
+
+* :meth:`~celery.result.GroupResult.completed_count`
+
+    Returns the number of completed subtasks.
+
+* :meth:`~celery.result.GroupResult.revoke`
+
+    Revokes all of the subtasks.
+
+* :meth:`~celery.result.GroupResult.iterate`
+
+    Iterates over the return values of the subtasks
+    as they finish, one by one.
+
+* :meth:`~celery.result.GroupResult.join`
+
+    Gather the results for all of the subtasks
+    and return a list with them ordered by the order of which they
+    were called.
+
+.. _chords:
+
+Chords
+======
+
+.. versionadded:: 2.3
+
+A chord is a task that only executes after all of the tasks in a taskset has
+finished executing.
+
+
+Let's calculate the sum of the expression
+:math:`1 + 1 + 2 + 2 + 3 + 3 ... n + n` up to a hundred digits.
+
+First we need two tasks, :func:`add` and :func:`tsum` (:func:`sum` is
+already a standard function):
+
+.. code-block:: python
+
+    @celery.task()
+    def add(x, y):
+        return x + y
+
+    @celery.task()
+    def tsum(numbers):
+        return sum(numbers)
+
+
+Now we can use a chord to calculate each addition step in parallel, and then
+get the sum of the resulting numbers::
+
+    >>> from celery import chord
+    >>> from tasks import add, tsum
+
+    >>> chord(add.subtask((i, i))
+    ...     for i in xrange(100))(tsum.subtask()).get()
+    9900
+
+
+This is obviously a very contrived example, the overhead of messaging and
+synchronization makes this a lot slower than its Python counterpart::
+
+    sum(i + i for i in xrange(100))
+
+The synchronization step is costly, so you should avoid using chords as much
+as possible. Still, the chord is a powerful primitive to have in your toolbox
+as synchronization is a required step for many parallel algorithms.
+
+Let's break the chord expression down::
+
+    >>> callback = tsum.subtask()
+    >>> header = [add.subtask((i, i)) for i in xrange(100)]
+    >>> result = chord(header)(callback)
+    >>> result.get()
+    9900
+
+Remember, the callback can only be executed after all of the tasks in the
+header has returned.  Each step in the header is executed as a task, in
+parallel, possibly on different nodes.  The callback is then applied with
+the return value of each task in the header.  The task id returned by
+:meth:`chord` is the id of the callback, so you can wait for it to complete
+and get the final return value (but remember to :ref:`never have a task wait
+for other tasks <task-synchronous-subtasks>`)
+
+.. _chord-important-notes:
+
+Important Notes
+---------------
+
+By default the synchronization step is implemented by having a recurring task
+poll the completion of the taskset every second, applying the subtask when
+ready.
+
+Example implementation:
+
+.. code-block:: python
+
+    def unlock_chord(taskset, callback, interval=1, max_retries=None):
+        if taskset.ready():
+            return subtask(callback).delay(taskset.join())
+        raise unlock_chord.retry(countdown=interval, max_retries=max_retries)
+
+
+This is used by all result backends except Redis and Memcached, which increment a
+counter after each task in the header, then applying the callback when the
+counter exceeds the number of tasks in the set. *Note:* chords do not properly
+work with Redis before version 2.2; you will need to upgrade to at least 2.2 to
+use them.
+
+The Redis and Memcached approach is a much better solution, but not easily
+implemented in other backends (suggestions welcome!).
+
+
+.. note::
+
+    If you are using chords with the Redis result backend and also overriding
+    the :meth:`Task.after_return` method, you need to make sure to call the
+    super method or else the chord callback will not be applied.
+
+    .. code-block:: python
+
+        def after_return(self, *args, **kwargs):
+            do_something()
+            super(MyTask, self).after_return(*args, **kwargs)

+ 3 - 8
docs/userguide/calling.rst

@@ -64,7 +64,7 @@ function:
 
     task.delay(arg1, arg2, kwarg1="x", kwarg2="y")
 
-Using :meth:`~@Task.apply_async` instead we have to write::
+Using :meth:`~@Task.apply_async` instead we have to write:
 
 .. code-block:: python
 
@@ -76,11 +76,8 @@ Using :meth:`~@Task.apply_async` instead we have to write::
     you can use :meth:`~@send_task` to call the task by name instead.
 
 
-So `delay` is clearly convenient, but it doesn't give you as much control as using
-`apply_async`.  With `apply_async` you can override the execution options
-available as attributes on the :class:`~@Task` class (see :ref:`task-options`).
-In addition you can set countdown/eta, task expiry, provide a custom broker
-connection and more.
+So `delay` is clearly convenient, but if you want to set additional execution
+options you have to use ``apply_async``.
 
 The rest of this document will go into the task execution
 options in detail.  All examples use a task
@@ -249,8 +246,6 @@ To disable retry you can set the ``retry`` execution option to :const:`False`:
         - :setting:`CELERY_TASK_PUBLISH_RETRY`
         - :setting:`CELERY_TASK_PUBLISH_RETRY_POLICY`
 
-
-
 Retry Policy
 ------------
 

+ 593 - 0
userguide/calling.html

@@ -0,0 +1,593 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Calling Tasks &mdash; Celery 2.6.0rc4 documentation</title>
+    
+    <link rel="stylesheet" href="../_static/celery.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/issuetracker.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '2.6.0rc4',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="top" title="Celery 2.6.0rc4 documentation" href="../index.html" />
+    <link rel="up" title="User Guide" href="index.html" />
+    <link rel="next" title="Workers Guide" href="workers.html" />
+    <link rel="prev" title="Tasks" href="tasks.html" /> 
+  </head>
+  <body>
+    <div class="related">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="../py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="workers.html" title="Workers Guide"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="tasks.html" title="Tasks"
+             accesskey="P">previous</a> |</li>
+        <li><a href="../index.html">Celery 2.6.0rc4 documentation</a> &raquo;</li>
+          <li><a href="index.html" accesskey="U">User Guide</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body">
+            
+<div class="deck">
+
+    
+        <p class="developmentversion">
+        This document is for Celery's development version, which can be
+        significantly different from previous releases. Get old docs here:
+
+        <a href="http://docs.celeryproject.org/en/latest/userguide/calling.html">2.5</a>.
+        </p>
+        
+
+</div>
+    <div class="section" id="calling-tasks">
+<span id="guide-calling"></span><h1>Calling Tasks<a class="headerlink" href="#calling-tasks" title="Permalink to this headline">¶</a></h1>
+<div class="contents local topic" id="contents">
+<ul class="simple">
+<li><a class="reference internal" href="#basics" id="id1">Basics</a></li>
+<li><a class="reference internal" href="#linking-callbacks-errbacks" id="id2">Linking (callbacks/errbacks)</a></li>
+<li><a class="reference internal" href="#eta-and-countdown" id="id3">ETA and countdown</a></li>
+<li><a class="reference internal" href="#expiration" id="id4">Expiration</a></li>
+<li><a class="reference internal" href="#message-sending-retry" id="id5">Message Sending Retry</a></li>
+<li><a class="reference internal" href="#serializers" id="id6">Serializers</a></li>
+<li><a class="reference internal" href="#compression" id="id7">Compression</a></li>
+<li><a class="reference internal" href="#connections" id="id8">Connections</a></li>
+<li><a class="reference internal" href="#routing-options" id="id9">Routing options</a></li>
+</ul>
+</div>
+<div class="section" id="basics">
+<span id="calling-basics"></span><h2><a class="toc-backref" href="#id1">Basics</a><a class="headerlink" href="#basics" title="Permalink to this headline">¶</a></h2>
+<p>This document describes Celery&#8217;s uniform &#8220;Calling API&#8221;
+used by task instances and the <a class="reference internal" href="canvas.html#guide-canvas"><em>canvas</em></a>.</p>
+<p>The API defines a standard set of execution options, as well as three methods:</p>
+<blockquote>
+<div><ul>
+<li><p class="first"><tt class="docutils literal"><span class="pre">apply_async(args[,</span> <span class="pre">kwargs[,</span> <span class="pre">...]])</span></tt></p>
+<blockquote>
+<div><p>Sends a task message.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="docutils literal"><span class="pre">delay(*args,</span> <span class="pre">**kwargs)</span></tt></p>
+<blockquote>
+<div><p>Shortcut to send a task message, but does not support execution
+options.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="docutils literal"><span class="pre">apply()</span></tt></p>
+<blockquote>
+<div><p>Does not send a message but executes the task inline instead.</p>
+</div></blockquote>
+</li>
+</ul>
+</div></blockquote>
+<div class="topic" id="calling-cheat">
+<p class="topic-title first">Quick Cheat Sheet</p>
+<ul>
+<li><dl class="first docutils">
+<dt><tt class="docutils literal"><span class="pre">T.delay(arg,</span> <span class="pre">kwarg=value)</span></tt></dt>
+<dd><p class="first last">always a shortcut to <tt class="docutils literal"><span class="pre">.apply_async</span></tt>.</p>
+</dd>
+</dl>
+</li>
+<li><p class="first"><tt class="docutils literal"><span class="pre">T.apply_async((arg,</span> <span class="pre">),</span> <span class="pre">{&quot;kwarg&quot;:</span> <span class="pre">value})</span></tt></p>
+</li>
+<li><dl class="first docutils">
+<dt><tt class="docutils literal"><span class="pre">T.apply_async(countdown=10)</span></tt></dt>
+<dd><p class="first last">executes 10 seconds from now.</p>
+</dd>
+</dl>
+</li>
+<li><dl class="first docutils">
+<dt><tt class="docutils literal"><span class="pre">T.apply_async(eta=now</span> <span class="pre">+</span> <span class="pre">timedelta(seconds=10))</span></tt></dt>
+<dd><p class="first last">executes 10 seconds from now, specifed using <tt class="docutils literal"><span class="pre">eta</span></tt></p>
+</dd>
+</dl>
+</li>
+<li><dl class="first docutils">
+<dt><tt class="docutils literal"><span class="pre">T.apply_async(countdown=60,</span> <span class="pre">expires=120)</span></tt></dt>
+<dd><p class="first last">executes in one minute from now, but expires after 2 minutes.</p>
+</dd>
+</dl>
+</li>
+<li><dl class="first docutils">
+<dt><tt class="docutils literal"><span class="pre">T.apply_async(expires=now</span> <span class="pre">+</span> <span class="pre">timedelta(days=2))</span></tt></dt>
+<dd><p class="first last">expires in 2 days, set using <a class="reference external" href="http://docs.python.org/dev/library/datetime.html#datetime.datetime" title="(in Python v3.3)"><tt class="xref py py-class docutils literal"><span class="pre">datetime</span></tt></a>.</p>
+</dd>
+</dl>
+</li>
+</ul>
+</div>
+<div class="section" id="example">
+<h3>Example<a class="headerlink" href="#example" title="Permalink to this headline">¶</a></h3>
+<p>The <a class="reference internal" href="../reference/celery.app.task.html#celery.app.task.Task.delay" title="celery.app.task.Task.delay"><tt class="xref py py-meth docutils literal"><span class="pre">delay()</span></tt></a> method is convenient as it looks like calling a regular
+function:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">task</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">,</span> <span class="n">kwarg1</span><span class="o">=</span><span class="s">&quot;x&quot;</span><span class="p">,</span> <span class="n">kwarg2</span><span class="o">=</span><span class="s">&quot;y&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Using <a class="reference internal" href="../reference/celery.app.task.html#celery.app.task.Task.apply_async" title="celery.app.task.Task.apply_async"><tt class="xref py py-meth docutils literal"><span class="pre">apply_async()</span></tt></a> instead we have to write:</p>
+<div class="highlight-python"><pre>.. code-block:: python</pre>
+</div>
+<blockquote>
+<div>task.apply_async(args=[arg1, arg2], kwargs={&#8220;kwarg1&#8221;: &#8220;x&#8221;, &#8220;kwarg2&#8221;: &#8220;y&#8221;})</div></blockquote>
+<div class="sidebar">
+<p class="first sidebar-title">Tip</p>
+<p class="last">If the task is not registered in the current process
+you can use <a class="reference internal" href="../reference/celery.html#celery.Celery.send_task" title="celery.Celery.send_task"><tt class="xref py py-meth docutils literal"><span class="pre">send_task()</span></tt></a> to call the task by name instead.</p>
+</div>
+<p>So <cite>delay</cite> is clearly convenient, but it doesn&#8217;t give you as much control as using
+<cite>apply_async</cite>.  With <cite>apply_async</cite> you can override the execution options
+available as attributes on the <a class="reference internal" href="../reference/celery.app.task.html#celery.app.task.Task" title="celery.app.task.Task"><tt class="xref py py-class docutils literal"><span class="pre">Task</span></tt></a> class (see <a class="reference internal" href="tasks.html#task-options"><em>Task options</em></a>).
+In addition you can set countdown/eta, task expiry, provide a custom broker
+connection and more.</p>
+<p>The rest of this document will go into the task execution
+options in detail.  All examples use a task
+called <cite>add</cite>, returning the sum of two arguments:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="nd">@celery.task</span><span class="p">()</span>
+<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
+</pre></div>
+</div>
+<div class="topic">
+<p class="topic-title first">There&#8217;s another way...</p>
+<p>You will learn more about this later while reading about the <a class="reference internal" href="canvas.html#guide-canvas"><em>Canvas</em></a>, but <a class="reference internal" href="../reference/celery.html#celery.subtask" title="celery.subtask"><tt class="xref py py-class docutils literal"><span class="pre">subtask</span></tt></a>&#8216;s are objects used to pass around
+the signature of a task invocation, (for example to send it over the
+network), and they also support the Calling API:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">task</span><span class="o">.</span><span class="n">s</span><span class="p">(</span><span class="n">arg1</span><span class="p">,</span> <span class="n">arg2</span><span class="p">,</span> <span class="n">kwarg1</span><span class="o">=</span><span class="s">&quot;x&quot;</span><span class="p">,</span> <span class="n">kwargs2</span><span class="o">=</span><span class="s">&quot;y&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">apply_async</span><span class="p">()</span>
+</pre></div>
+</div>
+</div>
+</div>
+</div>
+<div class="section" id="linking-callbacks-errbacks">
+<span id="calling-links"></span><h2><a class="toc-backref" href="#id2">Linking (callbacks/errbacks)</a><a class="headerlink" href="#linking-callbacks-errbacks" title="Permalink to this headline">¶</a></h2>
+<p>Celery supports linking tasks together so that one task follows another.
+The callback task will be applied with the result of the parent task
+as a partial argument:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">link</span><span class="o">=</span><span class="n">add</span><span class="o">.</span><span class="n">s</span><span class="p">(</span><span class="mi">16</span><span class="p">))</span>
+</pre></div>
+</div>
+<div class="sidebar">
+<p class="first sidebar-title">What is <tt class="docutils literal"><span class="pre">s</span></tt>?</p>
+<p>The <tt class="docutils literal"><span class="pre">add.s</span></tt> call used here is called a subtask, we talk
+more about subtasks in the <a class="reference internal" href="canvas.html#guide-canvas"><em>canvas guide</em></a>,
+where you can also learn about <a class="reference internal" href="../reference/celery.html#celery.chain" title="celery.chain"><tt class="xref py py-class docutils literal"><span class="pre">chain</span></tt></a>, which
+is a simpler way to chain tasks together.</p>
+<p class="last">In practice the <tt class="docutils literal"><span class="pre">link</span></tt> execution option is considered an internal
+primitive, and you will probably not use it directly, but
+rather use chains instead.</p>
+</div>
+<p>Here the result of the first task (4) will be sent to a new
+task that adds 16 to the previous result, forming the expression
+<img class="math" src="../_images/math/5bec3f9bd85f65932e771e2f003f923f49ed1a94.png" alt="(2 + 2) + 16 = 20"/></p>
+<p>You can also cause a callback to be applied if task raises an exception
+(<em>errback</em>), but this behaves differently from a regular callback
+in that it will be passed the id of the parent task, not the result.
+This is because it may not always be possible to serialize
+the exception raised, and so this way the error callback requires
+a result backend to be enabled, and the task must retrieve the result
+of the task instead.</p>
+<p>This is an example error callback:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="nd">@celery.task</span><span class="p">()</span>
+<span class="k">def</span> <span class="nf">error_handler</span><span class="p">(</span><span class="n">uuid</span><span class="p">):</span>
+    <span class="n">result</span> <span class="o">=</span> <span class="n">AsyncResult</span><span class="p">(</span><span class="n">uuid</span><span class="p">)</span>
+    <span class="n">exc</span> <span class="o">=</span> <span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">propagate</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+    <span class="k">print</span><span class="p">(</span><span class="s">&quot;Task </span><span class="si">%r</span><span class="s"> raised exception: </span><span class="si">%r</span><span class="se">\n</span><span class="si">%r</span><span class="s">&quot;</span> <span class="o">%</span> <span class="p">(</span>
+          <span class="n">exc</span><span class="p">,</span> <span class="n">result</span><span class="o">.</span><span class="n">traceback</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>it can be added to the task using the <tt class="docutils literal"><span class="pre">link_error</span></tt> execution
+option:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">link_error</span><span class="o">=</span><span class="n">error_handler</span><span class="o">.</span><span class="n">s</span><span class="p">())</span>
+</pre></div>
+</div>
+<p>In addition, both the <tt class="docutils literal"><span class="pre">link</span></tt> and <tt class="docutils literal"><span class="pre">link_error</span></tt> options can be expressed
+as a list:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">link</span><span class="o">=</span><span class="p">[</span><span class="n">add</span><span class="o">.</span><span class="n">s</span><span class="p">(</span><span class="mi">16</span><span class="p">),</span> <span class="n">other_task</span><span class="o">.</span><span class="n">s</span><span class="p">()])</span>
+</pre></div>
+</div>
+<p>The callbacks/errbacks will then be called in order, and all
+callbacks will be called with the return value of the parent task
+as a partial argument.</p>
+</div>
+<div class="section" id="eta-and-countdown">
+<span id="calling-eta"></span><h2><a class="toc-backref" href="#id3">ETA and countdown</a><a class="headerlink" href="#eta-and-countdown" title="Permalink to this headline">¶</a></h2>
+<p>The ETA (estimated time of arrival) lets you set a specific date and time that
+is the earliest time at which your task will be executed.  <cite>countdown</cite> is
+a shortcut to set eta by seconds into the future.</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">countdown</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>    <span class="c"># this takes at least 3 seconds to return</span>
+<span class="go">20</span>
+</pre></div>
+</div>
+<p>The task is guaranteed to be executed at some time <em>after</em> the
+specified date and time, but not necessarily at that exact time.
+Possible reasons for broken deadlines may include many items waiting
+in the queue, or heavy network latency.  To make sure your tasks
+are executed in a timely manner you should monitor the queue for congestion. Use
+Munin, or similar tools, to receive alerts, so appropriate action can be
+taken to ease the workload.  See <a class="reference internal" href="monitoring.html#monitoring-munin"><em>Munin</em></a>.</p>
+<p>While <cite>countdown</cite> is an integer, <cite>eta</cite> must be a <a class="reference external" href="http://docs.python.org/dev/library/datetime.html#datetime.datetime" title="(in Python v3.3)"><tt class="xref py py-class docutils literal"><span class="pre">datetime</span></tt></a>
+object, specifying an exact date and time (including millisecond precision,
+and timezone information):</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">tomorrow</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">eta</span><span class="o">=</span><span class="n">tomorrow</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="expiration">
+<span id="calling-expiration"></span><h2><a class="toc-backref" href="#id4">Expiration</a><a class="headerlink" href="#expiration" title="Permalink to this headline">¶</a></h2>
+<p>The <cite>expires</cite> argument defines an optional expiry time,
+either as seconds after task publish, or a specific date and time using
+<a class="reference external" href="http://docs.python.org/dev/library/datetime.html#datetime.datetime" title="(in Python v3.3)"><tt class="xref py py-class docutils literal"><span class="pre">datetime</span></tt></a>:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="c"># Task expires after one minute from now.</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="n">expires</span><span class="o">=</span><span class="mi">60</span><span class="p">)</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="c"># Also supports datetime</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="n">kwargs</span><span class="p">,</span>
+<span class="gp">... </span>                <span class="n">expires</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>When a worker receives an expired task it will mark
+the task as <a class="reference internal" href="tasks.html#std:state-REVOKED"><tt class="xref std std-state docutils literal"><span class="pre">REVOKED</span></tt></a> (<a class="reference internal" href="../reference/celery.exceptions.html#celery.exceptions.TaskRevokedError" title="celery.exceptions.TaskRevokedError"><tt class="xref py py-exc docutils literal"><span class="pre">TaskRevokedError</span></tt></a>).</p>
+</div>
+<div class="section" id="message-sending-retry">
+<span id="calling-retry"></span><h2><a class="toc-backref" href="#id5">Message Sending Retry</a><a class="headerlink" href="#message-sending-retry" title="Permalink to this headline">¶</a></h2>
+<p>Celery will automatically retry sending messages in the event of connection
+failure, and retry behavior can be configured &#8211; like how often to retry, or a maximum
+number of retries &#8211; or disabled all together.</p>
+<p>To disable retry you can set the <tt class="docutils literal"><span class="pre">retry</span></tt> execution option to <tt class="xref py py-const docutils literal"><span class="pre">False</span></tt>:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">retry</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+</pre></div>
+</div>
+<div class="topic">
+<p class="topic-title first">Related Settings</p>
+<table class="hlist"><tr><td><ul class="simple">
+<li><a class="reference internal" href="../configuration.html#std:setting-CELERY_TASK_PUBLISH_RETRY"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_TASK_PUBLISH_RETRY</span></tt></a></li>
+</ul>
+</td><td><ul class="simple">
+<li><a class="reference internal" href="../configuration.html#std:setting-CELERY_TASK_PUBLISH_RETRY_POLICY"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_TASK_PUBLISH_RETRY_POLICY</span></tt></a></li>
+</ul>
+</td></tr></table>
+</div>
+<div class="section" id="retry-policy">
+<h3>Retry Policy<a class="headerlink" href="#retry-policy" title="Permalink to this headline">¶</a></h3>
+<p>A retry policy is a mapping that controls how retries behave,
+and can contain the following keys:</p>
+<ul>
+<li><p class="first"><cite>max_retries</cite></p>
+<blockquote>
+<div><p>Maximum number of retries before giving up, in this case the
+exception that caused the retry to fail will be raised.</p>
+<p>A value of 0 or <tt class="xref py py-const docutils literal"><span class="pre">None</span></tt> means it will retry forever.</p>
+<p>The default is to retry 3 times.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><cite>interval_start</cite></p>
+<blockquote>
+<div><p>Defines the number of seconds (float or integer) to wait between
+retries.  Default is 0, which means the first retry will be
+instantaneous.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><cite>interval_step</cite></p>
+<blockquote>
+<div><p>On each consecutive retry this number will be added to the retry
+delay (float or integer).  Default is 0.2.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><cite>interval_max</cite></p>
+<blockquote>
+<div><p>Maximum number of seconds (float or integer) to wait between
+retries.  Default is 0.2.</p>
+</div></blockquote>
+</li>
+</ul>
+<p>For example, the default policy correlates to:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">retry</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">retry_policy</span><span class="o">=</span><span class="p">{</span>
+    <span class="s">&quot;max_retries&quot;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
+    <span class="s">&quot;interval_start&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
+    <span class="s">&quot;interval_step&quot;</span><span class="p">:</span> <span class="mf">0.2</span><span class="p">,</span>
+    <span class="s">&quot;interval_max&quot;</span><span class="p">:</span> <span class="mf">0.2</span><span class="p">,</span>
+<span class="p">})</span>
+</pre></div>
+</div>
+<p>the maximum time spent retrying will be 0.4 seconds.  It is set relatively
+short by default because a connection failure could lead to a retry pile effect
+if the broker connection is down: e.g. many web server processes waiting
+to retry blocking other incoming requests.</p>
+</div>
+</div>
+<div class="section" id="serializers">
+<span id="calling-serializers"></span><h2><a class="toc-backref" href="#id6">Serializers</a><a class="headerlink" href="#serializers" title="Permalink to this headline">¶</a></h2>
+<div class="sidebar">
+<p class="first sidebar-title">Security</p>
+<p>The pickle module allows for execution of arbitrary functions,
+please see the <a class="reference internal" href="security.html#guide-security"><em>security guide</em></a>.</p>
+<p class="last">Celery also comes with a special serializer that uses
+cryptography to sign your messages.</p>
+</div>
+<p>Data transferred between clients and workers needs to be serialized,
+so every message in Celery has a <tt class="docutils literal"><span class="pre">content_type</span></tt> header that
+describes the serialization method used to encode it.</p>
+<p>The default serializer is <a class="reference external" href="http://docs.python.org/dev/library/pickle.html#pickle" title="(in Python v3.3)"><tt class="xref py py-mod docutils literal"><span class="pre">pickle</span></tt></a>, but you can
+change this using the <a class="reference internal" href="../configuration.html#std:setting-CELERY_TASK_SERIALIZER"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_TASK_SERIALIZER</span></tt></a> setting,
+or for each individual task, or even per message.</p>
+<p>There&#8217;s built-in support for <a class="reference external" href="http://docs.python.org/dev/library/pickle.html#pickle" title="(in Python v3.3)"><tt class="xref py py-mod docutils literal"><span class="pre">pickle</span></tt></a>, <cite>JSON</cite>, <cite>YAML</cite>
+and <cite>msgpack</cite>, and you can also add your own custom serializers by registering
+them into the Kombu serializer registry (see <a class="reference external" href="http://packages.python.org/kombu/introduction.html#serialization-of-data">Kombu: Serialization of Data</a>).</p>
+<p>Each option has its advantages and disadvantages.</p>
+<dl class="docutils">
+<dt>json &#8211; JSON is supported in many programming languages, is now</dt>
+<dd><p class="first">a standard part of Python (since 2.6), and is fairly fast to decode
+using the modern Python libraries such as <tt class="xref py py-mod docutils literal"><span class="pre">cjson</span></tt> or <tt class="xref py py-mod docutils literal"><span class="pre">simplejson</span></tt>.</p>
+<p>The primary disadvantage to JSON is that it limits you to the following
+data types: strings, Unicode, floats, boolean, dictionaries, and lists.
+Decimals and dates are notably missing.</p>
+<p>Also, binary data will be transferred using Base64 encoding, which will
+cause the transferred data to be around 34% larger than an encoding which
+supports native binary types.</p>
+<p>However, if your data fits inside the above constraints and you need
+cross-language support, the default setting of JSON is probably your
+best choice.</p>
+<p class="last">See <a class="reference external" href="http://json.org">http://json.org</a> for more information.</p>
+</dd>
+<dt>pickle &#8211; If you have no desire to support any language other than</dt>
+<dd><p class="first">Python, then using the pickle encoding will gain you the support of
+all built-in Python data types (except class instances), smaller
+messages when sending binary files, and a slight speedup over JSON
+processing.</p>
+<p class="last">See <a class="reference external" href="http://docs.python.org/library/pickle.html">http://docs.python.org/library/pickle.html</a> for more information.</p>
+</dd>
+<dt>yaml &#8211; YAML has many of the same characteristics as json,</dt>
+<dd><p class="first">except that it natively supports more data types (including dates,
+recursive references, etc.)</p>
+<p>However, the Python libraries for YAML are a good bit slower than the
+libraries for JSON.</p>
+<p>If you need a more expressive set of data types and need to maintain
+cross-language compatibility, then YAML may be a better fit than the above.</p>
+<p class="last">See <a class="reference external" href="http://yaml.org/">http://yaml.org/</a> for more information.</p>
+</dd>
+<dt>msgpack &#8211; msgpack is a binary serialization format that is closer to JSON</dt>
+<dd><p class="first">in features.  It is very young however, and support should be considered
+experimental at this point.</p>
+<p class="last">See <a class="reference external" href="http://msgpack.org/">http://msgpack.org/</a> for more information.</p>
+</dd>
+</dl>
+<p>The encoding used is available as a message header, so the worker knows how to
+deserialize any task.  If you use a custom serializer, this serializer must
+be available for the worker.</p>
+<p>The following order is used to decide which serializer
+to use when sending a task:</p>
+<blockquote>
+<div><ol class="arabic simple">
+<li>The <cite>serializer</cite> execution option.</li>
+<li>The <a class="reference internal" href="../reference/celery.app.task.html#celery.app.task.Task.serializer" title="celery.app.task.Task.serializer"><tt class="xref py py-attr docutils literal"><span class="pre">Task.serializer</span></tt></a> attribute</li>
+<li>The <a class="reference internal" href="../configuration.html#std:setting-CELERY_TASK_SERIALIZER"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_TASK_SERIALIZER</span></tt></a> setting.</li>
+</ol>
+</div></blockquote>
+<p>Example setting a custom serializer for a single task invocation:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">10</span><span class="p">,</span> <span class="mi">10</span><span class="p">),</span> <span class="n">serializer</span><span class="o">=</span><span class="s">&quot;json&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="compression">
+<span id="calling-compression"></span><h2><a class="toc-backref" href="#id7">Compression</a><a class="headerlink" href="#compression" title="Permalink to this headline">¶</a></h2>
+<p>Celery can compress the messages using either <em>gzip</em>, or <em>bzip2</em>.
+You can also create your own compression schemes and register
+them in the <a class="reference external" href="http://kombu.readthedocs.org/en/latest/reference/kombu.compression.html#kombu.compression.register" title="(in Kombu v2.2)"><tt class="xref py py-func docutils literal"><span class="pre">kombu</span> <span class="pre">compression</span> <span class="pre">registry</span></tt></a>.</p>
+<p>The following order is used to decide which compression scheme
+to use when sending a task:</p>
+<blockquote>
+<div><ol class="arabic simple">
+<li>The <cite>compression</cite> execution option.</li>
+<li>The <tt class="xref py py-attr docutils literal"><span class="pre">Task.compression</span></tt> attribute.</li>
+<li>The <a class="reference internal" href="../configuration.html#std:setting-CELERY_MESSAGE_COMPRESSION"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_MESSAGE_COMPRESSION</span></tt></a> attribute.</li>
+</ol>
+</div></blockquote>
+<p>Example specifying the compression used when calling a task:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">compression</span><span class="o">=</span><span class="s">&quot;zlib&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="connections">
+<span id="calling-connections"></span><h2><a class="toc-backref" href="#id8">Connections</a><a class="headerlink" href="#connections" title="Permalink to this headline">¶</a></h2>
+<div class="sidebar">
+<p class="first sidebar-title">Automatic Pool Support</p>
+<p>Since version 2.3 there is support for automatic connection pools,
+so you don&#8217;t have to manually handle connections and publishers
+to reuse connections.</p>
+<p>The connection pool is enabled by default since version 2.5.</p>
+<p class="last">See the <a class="reference internal" href="../configuration.html#std:setting-BROKER_POOL_LIMIT"><tt class="xref std std-setting docutils literal"><span class="pre">BROKER_POOL_LIMIT</span></tt></a> setting for more information.</p>
+</div>
+<p>You can handle the connection manually by creating a
+publisher:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">results</span> <span class="o">=</span> <span class="p">[]</span>
+<span class="k">with</span> <span class="n">add</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">pool</span><span class="o">.</span><span class="n">acquire</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">as</span> <span class="n">connection</span><span class="p">:</span>
+    <span class="k">with</span> <span class="n">add</span><span class="o">.</span><span class="n">get_publisher</span><span class="p">(</span><span class="n">connection</span><span class="p">)</span> <span class="k">as</span> <span class="n">publisher</span><span class="p">:</span>
+        <span class="k">try</span><span class="p">:</span>
+            <span class="k">for</span> <span class="n">args</span> <span class="ow">in</span> <span class="n">numbers</span><span class="p">:</span>
+                <span class="n">res</span> <span class="o">=</span> <span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">publisher</span><span class="o">=</span><span class="n">publisher</span><span class="p">)</span>
+                <span class="n">results</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
+<span class="k">print</span><span class="p">([</span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">results</span><span class="p">])</span>
+</pre></div>
+</div>
+<p>Though this particular example is much better expressed as a group:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">group</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">numbers</span> <span class="o">=</span> <span class="p">[(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span> <span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="mi">8</span><span class="p">),</span> <span class="p">(</span><span class="mi">16</span><span class="p">,</span> <span class="mi">16</span><span class="p">)]</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">res</span> <span class="o">=</span> <span class="n">group</span><span class="p">(</span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">numbers</span><span class="p">)</span><span class="o">.</span><span class="n">apply_async</span><span class="p">()</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">res</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
+<span class="go">[4, 8, 16, 32]</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="routing-options">
+<span id="calling-routing"></span><h2><a class="toc-backref" href="#id9">Routing options</a><a class="headerlink" href="#routing-options" title="Permalink to this headline">¶</a></h2>
+<p>Celery can route tasks to different queues.</p>
+<p>Simple routing (name &lt;-&gt; name) is accomplished using the <tt class="docutils literal"><span class="pre">queue</span></tt> option:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">queue</span><span class="o">=</span><span class="s">&quot;priority.high&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>You can then assign workers to the <tt class="docutils literal"><span class="pre">priority.high</span></tt> queue by using
+the workers <em class="xref std std-option">-Q</em> argument:</p>
+<div class="highlight-python"><pre>$ celery worker -l info -Q celery,priority.high</pre>
+</div>
+<div class="admonition-see-also admonition seealso">
+<p class="first admonition-title">See also</p>
+<p>Hard-coding queue names in code is not recommended, the best practice
+is to use configuration routers (<a class="reference internal" href="../configuration.html#std:setting-CELERY_ROUTES"><tt class="xref std std-setting docutils literal"><span class="pre">CELERY_ROUTES</span></tt></a>).</p>
+<p class="last">To find out more about routing, please see <a class="reference internal" href="routing.html#guide-routing"><em>Routing Tasks</em></a>.</p>
+</div>
+<div class="section" id="advanced-options">
+<h3>Advanced Options<a class="headerlink" href="#advanced-options" title="Permalink to this headline">¶</a></h3>
+<p>These options are for advanced users who want to take use of
+AMQP&#8217;s full routing capabilities. Interested parties may read the
+<a class="reference internal" href="routing.html#guide-routing"><em>routing guide</em></a>.</p>
+<ul>
+<li><p class="first">exchange</p>
+<blockquote>
+<div><p>Name of exchange (or a <a class="reference external" href="http://kombu.readthedocs.org/en/latest/reference/kombu.entity.html#kombu.entity.Exchange" title="(in Kombu v2.2)"><tt class="xref py py-class docutils literal"><span class="pre">kombu.entity.Exchange</span></tt></a>) to
+send the message to.</p>
+</div></blockquote>
+</li>
+<li><p class="first">routing_key</p>
+<blockquote>
+<div><p>Routing key used to determine.</p>
+</div></blockquote>
+</li>
+<li><p class="first">mandatory</p>
+<blockquote>
+<div><p>This sets the delivery to be mandatory.  An exception will be raised
+if there are no running workers able to take on the task.</p>
+<p>Not supported by <tt class="xref py py-mod docutils literal"><span class="pre">amqplib</span></tt>.</p>
+</div></blockquote>
+</li>
+<li><p class="first">immediate</p>
+<blockquote>
+<div><p>Request immediate delivery. Will raise an exception
+if the task cannot be routed to a worker immediately.</p>
+<p>Not supported by <tt class="xref py py-mod docutils literal"><span class="pre">amqplib</span></tt>.</p>
+</div></blockquote>
+</li>
+<li><p class="first">priority</p>
+<blockquote>
+<div><p>A number between <cite>0</cite> and <cite>9</cite>, where <cite>0</cite> is the highest priority.</p>
+<p>Supported by: redis, beanstalk</p>
+</div></blockquote>
+</li>
+</ul>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar">
+        <div class="sphinxsidebarwrapper"><p class="logo"><a href="../index.html">
+  <img class="logo" src="http://cloud.github.com/downloads/celery/celery/celery_128.png" alt="Logo"/>
+</a></p>
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="tasks.html"
+                        title="previous chapter">Tasks</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="workers.html"
+                        title="next chapter">Workers Guide</a></p>
+  <h3>This Page</h3>
+  <ul class="this-page-menu">
+    <li><a href="../_sources/userguide/calling.txt"
+           rel="nofollow">Show Source</a></li>
+  </ul>
+<div id="searchbox" style="display: none">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="../py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="workers.html" title="Workers Guide"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="tasks.html" title="Tasks"
+             >previous</a> |</li>
+        <li><a href="../index.html">Celery 2.6.0rc4 documentation</a> &raquo;</li>
+          <li><a href="index.html" >User Guide</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer">
+        &copy; Copyright 2009-2012, Ask Solem &amp; Contributors.
+    </div>
+  </body>
+</html>

+ 375 - 0
userguide/canvas.html

@@ -0,0 +1,375 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+    
+    <title>Canvas: Building Workflows &mdash; Celery 2.6.0rc4 documentation</title>
+    
+    <link rel="stylesheet" href="../_static/celery.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
+    <link rel="stylesheet" href="../_static/issuetracker.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    '../',
+        VERSION:     '2.6.0rc4',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true
+      };
+    </script>
+    <script type="text/javascript" src="../_static/jquery.js"></script>
+    <script type="text/javascript" src="../_static/underscore.js"></script>
+    <script type="text/javascript" src="../_static/doctools.js"></script>
+    <link rel="top" title="Celery 2.6.0rc4 documentation" href="../index.html" />
+    <link rel="up" title="User Guide" href="index.html" />
+    <link rel="next" title="HTTP Callback Tasks (Webhooks)" href="remote-tasks.html" />
+    <link rel="prev" title="Periodic Tasks" href="periodic-tasks.html" /> 
+  </head>
+  <body>
+    <div class="related">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="../py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="remote-tasks.html" title="HTTP Callback Tasks (Webhooks)"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="periodic-tasks.html" title="Periodic Tasks"
+             accesskey="P">previous</a> |</li>
+        <li><a href="../index.html">Celery 2.6.0rc4 documentation</a> &raquo;</li>
+          <li><a href="index.html" accesskey="U">User Guide</a> &raquo;</li> 
+      </ul>
+    </div>  
+
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body">
+            
+<div class="deck">
+
+    
+        <p class="developmentversion">
+        This document is for Celery's development version, which can be
+        significantly different from previous releases. Get old docs here:
+
+        <a href="http://docs.celeryproject.org/en/latest/userguide/canvas.html">2.5</a>.
+        </p>
+        
+
+</div>
+    <div class="section" id="canvas-building-workflows">
+<span id="guide-canvas"></span><h1>Canvas: Building Workflows<a class="headerlink" href="#canvas-building-workflows" title="Permalink to this headline">¶</a></h1>
+<div class="contents local topic" id="contents">
+<ul class="simple">
+<li><a class="reference internal" href="#subtasks" id="id2">Subtasks</a><ul>
+<li><a class="reference internal" href="#callbacks" id="id3">Callbacks</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#groups" id="id4">Groups</a><ul>
+<li><a class="reference internal" href="#group-results" id="id5">Group Results</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#chords" id="id6">Chords</a><ul>
+<li><a class="reference internal" href="#important-notes" id="id7">Important Notes</a></li>
+</ul>
+</li>
+</ul>
+</div>
+<div class="section" id="subtasks">
+<span id="canvas-subtasks"></span><h2><a class="toc-backref" href="#id2">Subtasks</a><a class="headerlink" href="#subtasks" title="Permalink to this headline">¶</a></h2>
+<p class="versionadded">
+<span class="versionmodified">New in version 2.0.</span></p>
+<p>The <a class="reference internal" href="../reference/celery.html#celery.subtask" title="celery.subtask"><tt class="xref py py-class docutils literal"><span class="pre">subtask</span></tt></a> type is used to wrap the arguments and
+execution options for a single task invocation:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">subtask</span>
+
+<span class="n">subtask</span><span class="p">(</span><span class="n">task_name_or_cls</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">options</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>For convenience every task also has a shortcut to create subtasks:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="n">task</span><span class="o">.</span><span class="n">subtask</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">,</span> <span class="n">options</span><span class="p">)</span>
+</pre></div>
+</div>
+<p><a class="reference internal" href="../reference/celery.html#celery.subtask" title="celery.subtask"><tt class="xref py py-class docutils literal"><span class="pre">subtask</span></tt></a> is actually a <a class="reference external" href="http://docs.python.org/dev/library/stdtypes.html#dict" title="(in Python v3.3)"><tt class="xref py py-class docutils literal"><span class="pre">dict</span></tt></a> subclass,
+which means it can be serialized with JSON or other encodings that doesn&#8217;t
+support complex Python objects.</p>
+<p>Also it can be regarded as a type, as the following usage works:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">s</span> <span class="o">=</span> <span class="n">subtask</span><span class="p">(</span><span class="s">&quot;tasks.add&quot;</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{})</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">subtask</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span><span class="n">s</span><span class="p">))</span>  <span class="c"># coerce dict into subtask</span>
+</pre></div>
+</div>
+<p>This makes it excellent as a means to pass callbacks around to tasks.</p>
+<div class="section" id="callbacks">
+<span id="canvas-callbacks"></span><h3><a class="toc-backref" href="#id3">Callbacks</a><a class="headerlink" href="#callbacks" title="Permalink to this headline">¶</a></h3>
+<p>Callbacks can be added to any task using the <tt class="docutils literal"><span class="pre">link</span></tt> argument
+to <tt class="docutils literal"><span class="pre">apply_async</span></tt>:</p>
+<blockquote>
+<div>add.apply_async((2, 2), link=other_task.subtask())</div></blockquote>
+<p>The callback will only be applied if the task exited successfully,
+and it will be applied with the return value of the parent task as argument.</p>
+<p>The best thing is that any arguments you add to <cite>subtask</cite>,
+will be prepended to the arguments specified by the subtask itself!</p>
+<p>If you have the subtask:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">(</span><span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="p">))</span>
+</pre></div>
+</div>
+<p><cite>subtask.delay(result)</cite> becomes:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span><span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">result</span><span class="p">,</span> <span class="mi">10</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>...</p>
+<p>Now let&#8217;s call our <tt class="docutils literal"><span class="pre">add</span></tt> task with a callback using partial
+arguments:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">add</span><span class="o">.</span><span class="n">apply_async</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">link</span><span class="o">=</span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">8</span><span class="p">,</span> <span class="p">)))</span>
+</pre></div>
+</div>
+<p>As expected this will first launch one task calculating <img class="math" src="../_images/math/2da3a5ecd44fdb54a9a30c87a48a5b1638c139b0.png" alt="2 + 2"/>, then
+another task calculating <img class="math" src="../_images/math/c67562ae31bb2d1017a9b0240c012edc474851c3.png" alt="4 + 8"/>.</p>
+</div>
+</div>
+<div class="section" id="groups">
+<span id="canvas-group"></span><h2><a class="toc-backref" href="#id4">Groups</a><a class="headerlink" href="#groups" title="Permalink to this headline">¶</a></h2>
+<p>The <a class="reference internal" href="../reference/celery.html#celery.group" title="celery.group"><tt class="xref py py-class docutils literal"><span class="pre">group</span></tt></a> enables easy invocation of several
+tasks at once, and is then able to join the results in the same order as the
+tasks were invoked.</p>
+<p><tt class="docutils literal"><span class="pre">group</span></tt> takes a list of <a class="reference internal" href="../reference/celery.html#celery.subtask" title="celery.subtask"><tt class="xref py py-class docutils literal"><span class="pre">subtask</span></tt></a>&#8216;s:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">group</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">tasks</span> <span class="kn">import</span> <span class="n">add</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">job</span> <span class="o">=</span> <span class="n">group</span><span class="p">([</span>
+<span class="gp">... </span>            <span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
+<span class="gp">... </span>            <span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">)),</span>
+<span class="gp">... </span>            <span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">8</span><span class="p">,</span> <span class="mi">8</span><span class="p">)),</span>
+<span class="gp">... </span>            <span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">16</span><span class="p">,</span> <span class="mi">16</span><span class="p">)),</span>
+<span class="gp">... </span>            <span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="mi">32</span><span class="p">,</span> <span class="mi">32</span><span class="p">)),</span>
+<span class="gp">... </span><span class="p">])</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">job</span><span class="o">.</span><span class="n">apply_async</span><span class="p">()</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">ready</span><span class="p">()</span>  <span class="c"># have all subtasks completed?</span>
+<span class="go">True</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">successful</span><span class="p">()</span> <span class="c"># were all subtasks successful?</span>
+<span class="go">True</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
+<span class="go">[4, 8, 16, 32, 64]</span>
+</pre></div>
+</div>
+<p>The first argument can alternatively be an iterator, like:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">group</span><span class="p">(</span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span>
+</pre></div>
+</div>
+<div class="section" id="group-results">
+<span id="canvas-group-results"></span><h3><a class="toc-backref" href="#id5">Group Results</a><a class="headerlink" href="#group-results" title="Permalink to this headline">¶</a></h3>
+<p>When a  <a class="reference internal" href="../reference/celery.html#celery.group" title="celery.group"><tt class="xref py py-class docutils literal"><span class="pre">group</span></tt></a> is applied it returns a
+<a class="reference internal" href="../reference/celery.result.html#celery.result.GroupResult" title="celery.result.GroupResult"><tt class="xref py py-class docutils literal"><span class="pre">GroupResult</span></tt></a> object.</p>
+<p><a class="reference internal" href="../reference/celery.result.html#celery.result.GroupResult" title="celery.result.GroupResult"><tt class="xref py py-class docutils literal"><span class="pre">GroupResult</span></tt></a> takes a list of
+<a class="reference internal" href="../reference/celery.result.html#celery.result.AsyncResult" title="celery.result.AsyncResult"><tt class="xref py py-class docutils literal"><span class="pre">AsyncResult</span></tt></a> instances and operates on them as if it was a
+single task.</p>
+<p>It supports the following operations:</p>
+<ul>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">successful()</span></tt></p>
+<blockquote>
+<div><p>Returns <tt class="xref py py-const docutils literal"><span class="pre">True</span></tt> if all of the subtasks finished
+successfully (e.g. did not raise an exception).</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">failed()</span></tt></p>
+<blockquote>
+<div><p>Returns <tt class="xref py py-const docutils literal"><span class="pre">True</span></tt> if any of the subtasks failed.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">waiting()</span></tt></p>
+<blockquote>
+<div><p>Returns <tt class="xref py py-const docutils literal"><span class="pre">True</span></tt> if any of the subtasks
+is not ready yet.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">ready()</span></tt></p>
+<blockquote>
+<div><p>Return <tt class="xref py py-const docutils literal"><span class="pre">True</span></tt> if all of the subtasks
+are ready.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">completed_count()</span></tt></p>
+<blockquote>
+<div><p>Returns the number of completed subtasks.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">revoke()</span></tt></p>
+<blockquote>
+<div><p>Revokes all of the subtasks.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">iterate()</span></tt></p>
+<blockquote>
+<div><p>Iterates over the return values of the subtasks
+as they finish, one by one.</p>
+</div></blockquote>
+</li>
+<li><p class="first"><tt class="xref py py-meth docutils literal"><span class="pre">join()</span></tt></p>
+<blockquote>
+<div><p>Gather the results for all of the subtasks
+and return a list with them ordered by the order of which they
+were called.</p>
+</div></blockquote>
+</li>
+</ul>
+</div>
+</div>
+<div class="section" id="chords">
+<span id="id1"></span><h2><a class="toc-backref" href="#id6">Chords</a><a class="headerlink" href="#chords" title="Permalink to this headline">¶</a></h2>
+<p class="versionadded">
+<span class="versionmodified">New in version 2.3.</span></p>
+<p>A chord is a task that only executes after all of the tasks in a taskset has
+finished executing.</p>
+<p>Let&#8217;s calculate the sum of the expression
+<img class="math" src="../_images/math/c9bf2df3c397d6399241c37fc9887ca50bc8fa09.png" alt="1 + 1 + 2 + 2 + 3 + 3 ... n + n"/> up to a hundred digits.</p>
+<p>First we need two tasks, <tt class="xref py py-func docutils literal"><span class="pre">add()</span></tt> and <tt class="xref py py-func docutils literal"><span class="pre">tsum()</span></tt> (<a class="reference external" href="http://docs.python.org/dev/library/functions.html#sum" title="(in Python v3.3)"><tt class="xref py py-func docutils literal"><span class="pre">sum()</span></tt></a> is
+already a standard function):</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="nd">@celery.task</span><span class="p">()</span>
+<span class="k">def</span> <span class="nf">add</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span>
+
+<span class="nd">@celery.task</span><span class="p">()</span>
+<span class="k">def</span> <span class="nf">tsum</span><span class="p">(</span><span class="n">numbers</span><span class="p">):</span>
+    <span class="k">return</span> <span class="nb">sum</span><span class="p">(</span><span class="n">numbers</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Now we can use a chord to calculate each addition step in parallel, and then
+get the sum of the resulting numbers:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">chord</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">tasks</span> <span class="kn">import</span> <span class="n">add</span><span class="p">,</span> <span class="n">tsum</span>
+
+<span class="gp">&gt;&gt;&gt; </span><span class="n">chord</span><span class="p">(</span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span>
+<span class="gp">... </span>    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">))(</span><span class="n">tsum</span><span class="o">.</span><span class="n">subtask</span><span class="p">())</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
+<span class="go">9900</span>
+</pre></div>
+</div>
+<p>This is obviously a very contrived example, the overhead of messaging and
+synchronization makes this a lot slower than its Python counterpart:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="nb">sum</span><span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">))</span>
+</pre></div>
+</div>
+<p>The synchronization step is costly, so you should avoid using chords as much
+as possible. Still, the chord is a powerful primitive to have in your toolbox
+as synchronization is a required step for many parallel algorithms.</p>
+<p>Let&#8217;s break the chord expression down:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">callback</span> <span class="o">=</span> <span class="n">tsum</span><span class="o">.</span><span class="n">subtask</span><span class="p">()</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">header</span> <span class="o">=</span> <span class="p">[</span><span class="n">add</span><span class="o">.</span><span class="n">subtask</span><span class="p">((</span><span class="n">i</span><span class="p">,</span> <span class="n">i</span><span class="p">))</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">xrange</span><span class="p">(</span><span class="mi">100</span><span class="p">)]</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">chord</span><span class="p">(</span><span class="n">header</span><span class="p">)(</span><span class="n">callback</span><span class="p">)</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
+<span class="go">9900</span>
+</pre></div>
+</div>
+<p>Remember, the callback can only be executed after all of the tasks in the
+header has returned.  Each step in the header is executed as a task, in
+parallel, possibly on different nodes.  The callback is then applied with
+the return value of each task in the header.  The task id returned by
+<tt class="xref py py-meth docutils literal"><span class="pre">chord()</span></tt> is the id of the callback, so you can wait for it to complete
+and get the final return value (but remember to <a class="reference internal" href="tasks.html#task-synchronous-subtasks"><em>never have a task wait
+for other tasks</em></a>)</p>
+<div class="section" id="important-notes">
+<span id="chord-important-notes"></span><h3><a class="toc-backref" href="#id7">Important Notes</a><a class="headerlink" href="#important-notes" title="Permalink to this headline">¶</a></h3>
+<p>By default the synchronization step is implemented by having a recurring task
+poll the completion of the taskset every second, applying the subtask when
+ready.</p>
+<p>Example implementation:</p>
+<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">unlock_chord</span><span class="p">(</span><span class="n">taskset</span><span class="p">,</span> <span class="n">callback</span><span class="p">,</span> <span class="n">interval</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">max_retries</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+    <span class="k">if</span> <span class="n">taskset</span><span class="o">.</span><span class="n">ready</span><span class="p">():</span>
+        <span class="k">return</span> <span class="n">subtask</span><span class="p">(</span><span class="n">callback</span><span class="p">)</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="n">taskset</span><span class="o">.</span><span class="n">join</span><span class="p">())</span>
+    <span class="k">raise</span> <span class="n">unlock_chord</span><span class="o">.</span><span class="n">retry</span><span class="p">(</span><span class="n">countdown</span><span class="o">=</span><span class="n">interval</span><span class="p">,</span> <span class="n">max_retries</span><span class="o">=</span><span class="n">max_retries</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>This is used by all result backends except Redis and Memcached, which increment a
+counter after each task in the header, then applying the callback when the
+counter exceeds the number of tasks in the set. <em>Note:</em> chords do not properly
+work with Redis before version 2.2; you will need to upgrade to at least 2.2 to
+use them.</p>
+<p>The Redis and Memcached approach is a much better solution, but not easily
+implemented in other backends (suggestions welcome!).</p>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p>If you are using chords with the Redis result backend and also overriding
+the <tt class="xref py py-meth docutils literal"><span class="pre">Task.after_return()</span></tt> method, you need to make sure to call the
+super method or else the chord callback will not be applied.</p>
+<div class="last highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">after_return</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+    <span class="n">do_something</span><span class="p">()</span>
+    <span class="nb">super</span><span class="p">(</span><span class="n">MyTask</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">after_return</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+</div>
+</div>
+
+
+          </div>
+        </div>
+      </div>
+      <div class="sphinxsidebar">
+        <div class="sphinxsidebarwrapper"><p class="logo"><a href="../index.html">
+  <img class="logo" src="http://cloud.github.com/downloads/celery/celery/celery_128.png" alt="Logo"/>
+</a></p>
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="periodic-tasks.html"
+                        title="previous chapter">Periodic Tasks</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="remote-tasks.html"
+                        title="next chapter">HTTP Callback Tasks (Webhooks)</a></p>
+  <h3>This Page</h3>
+  <ul class="this-page-menu">
+    <li><a href="../_sources/userguide/canvas.txt"
+           rel="nofollow">Show Source</a></li>
+  </ul>
+<div id="searchbox" style="display: none">
+  <h3>Quick search</h3>
+    <form class="search" action="../search.html" method="get">
+      <input type="text" name="q" />
+      <input type="submit" value="Go" />
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+    <p class="searchtip" style="font-size: 90%">
+    Enter search terms or a module, class or function name.
+    </p>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
+        </div>
+      </div>
+      <div class="clearer"></div>
+    </div>
+    <div class="related">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="../genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="../py-modindex.html" title="Python Module Index"
+             >modules</a> |</li>
+        <li class="right" >
+          <a href="remote-tasks.html" title="HTTP Callback Tasks (Webhooks)"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="periodic-tasks.html" title="Periodic Tasks"
+             >previous</a> |</li>
+        <li><a href="../index.html">Celery 2.6.0rc4 documentation</a> &raquo;</li>
+          <li><a href="index.html" >User Guide</a> &raquo;</li> 
+      </ul>
+    </div>
+    <div class="footer">
+        &copy; Copyright 2009-2012, Ask Solem &amp; Contributors.
+    </div>
+  </body>
+</html>