Improve docs

docs/getting-started/first-steps-with-celery.rst

@@ -77,6 +77,12 @@ this is needed so that names can be automatically generated, the second
 argument is the broker keyword argument which specifies the URL of the
 message broker we want to use.
 
+The broker argument specifies the URL of the broker we want to use,
+we use RabbitMQ here, which is already the default option,
+but see :ref:`celerytut-broker` above if you want to use something different,
+e.g. for Redis you can use ``redis://localhost``, or MongoDB:
+``mongodb://localhost``.
+
 We defined a single task, called ``add``, which returns the sum of two numbers.
 
 .. _celerytut-running-celeryd:
@@ -115,7 +121,7 @@ Whenever we want to execute our task, we use the
 
 This is a handy shortcut to the :meth:`~@Task.apply_async`
 method which gives greater control of the task execution (see
-:ref:`guide-executing`).
+:ref:`guide-executing`)::
 
     >>> from tasks import add
     >>> add.delay(4, 4)
@@ -136,44 +142,55 @@ Keeping Results
 
 If you want to keep track of the tasks state, Celery needs to store or send
 the states somewhere.  There are several
-built-in backends to choose from: SQLAlchemy/Django ORM, Memcached, Redis,
-AMQP, MongoDB, Tokyo Tyrant and Redis -- or you can define your own.
+built-in result backends to choose from: `SQLAlchemy`_/`Django`_ ORM,
+`Memcached`_, `Redis`_, AMQP (`RabbitMQ`_), and `MongoDB`_ -- or you can define your own.
+
+.. _`Memcached`: http://memcached.org
+.. _`MongoDB`: http://www.mongodb.org
+.. _`SQLAlchemy`: http://www.sqlalchemy.org/
+.. _`Django`: http://djangoproject.com
 
 For this example we will use the `amqp` result backend, which sends states
-as messages.  The backend is configured via the :setting:`CELERY_RESULT_BACKEND`
-setting or using the ``backend`` argument to :class:`Celery`, in addition individual
-result backends may have additional required or optional settings
-to configure::
+as messages.  The backend is specified via the ``backend`` argument to
+:class:`@Celery`, (or via the :setting:`CELERY_RESULT_BACKEND` setting if
+you choose to use a configuration module)::
+
+    celery = Celery("tasks", backend="amqp", broker="amqp://")
+
+or if you want to use Redis as the result backend, but still use RabbitMQ as
+the message broker (a popular combination)::
 
-    celery = Celery(backend="amqp")
+    celery = Celery("tasks", backend="redis://localhost", broker="amqp://")
 
 To read more about result backends please see :ref:`task-result-backends`.
 
 Now with the result backend configured, let's execute the task again.
-This time we'll hold on to the :class:`~@AsyncResult`::
+This time we'll hold on to the :class:`~@AsyncResult` instance returned
+when you apply a task::
 
     >>> result = add.delay(4, 4)
 
-Here's some examples of what you can do when you have results::
+Here's some examples of what you can do with the result instance::
 
-    >>> result.ready() # returns True if the task has finished processing.
+    >>> result.ready()     # returns True if the task has finished processing.
     False
 
-    >>> result.result # task is not ready, so no return value yet.
+    >>> result.result      # task is not ready, so no return value yet.
     None
 
-    >>> result.get()   # Waits until the task is done and returns the retval.
+    >>> result.get()       # waits for the task and returns its retval.
     8
 
-    >>> result.result # direct access to result, doesn't re-raise errors.
+    >>> result.result      # direct access to result, doesn't re-raise errors.
     8
 
     >>> result.successful() # returns True if the task didn't end in failure.
     True
 
-If the task raises an exception, the return value of :meth:`~@AsyncResult.successful`
-will be :const:`False`, and `result.result` will contain the exception instance
-raised by the task.
+If the task raises an exception, the return value of
+:meth:`~@AsyncResult.failed` will be :const:`True`, and `result.result` will
+contain the exception instance raised by the task, and `result.traceback`
+will contain the original traceback as a string.
 
 .. _celerytut-configuration:
 
@@ -190,20 +207,36 @@ are many things to tweak so that Celery works just the way you want it to.
 Reading about the options available is a good idea to get familiar with what
 can be configured, see the :ref:`configuration` reference.
 
-The configuration can be set on the app directly (but not all at runtime)
-or by using a dedicated configuration module.
-As an example you can set the default value for the workers
-``--concurrency`` argument, which is used to decide the number of pool worker
-processes, by changing the :setting:`CELERYD_CONCURRENCY` setting:
+The configuration can be set on the app directly or by using a dedicated
+configuration module.
+As an example you can configure the default serializer used for serializing
+task payloads by changing the :setting:`CELERY_TASK_SERIALIZER` setting:
 
 .. code-block:: python
 
-    celery.conf.CELERY_CONCURRENCY = 10
+    celery.conf.CELERY_TASK_SERIALIZER = "json"
 
-If you are configuring many settings then one practice is to have a separate module
-containing the configuration.  You can tell your Celery instance to use
-this module, historically called ``celeryconfig.py``, with the
-:meth:`config_from_obj` method:
+If you are configuring many settings at once you can use ``update``:
+
+.. code-block:: python
+
+    celery.conf.update(
+        CELERY_TASK_SERIALIZER="json",
+        CELERY_RESULT_SERIALIZER="json",
+        CELERY_TIMEZONE="Europe/Oslo",
+        CELERY_ENABLE_UTC=True,
+    )
+
+For larger projects using a dedicated configuration module is useful,
+in fact you are discouraged from hard coding
+periodic task intervals and task routing options, as it is much
+better to keep this in a centralized location, and especially for libaries
+it makes it possible for users to control how they want your tasks to behave,
+you can also imagine your sysadmin making simple changes to the configuration
+in the event of system trobule.
+
+You can tell your Celery instance to use a configuration module,
+often called ``celeryconfig.py``, with :meth:`config_from_obj` method:
 
 .. code-block:: python
 
@@ -212,9 +245,17 @@ this module, historically called ``celeryconfig.py``, with the
 A module named ``celeryconfig.py`` must then be available to load from the
 current directory or on the Python path, it could look like this:
 
-:file:`celeryconfig.py`::
+:file:`celeryconfig.py`:
+
+.. code-block:: python
+
+    BROKER_URL = "amqp://"
+    CELERY_RESULT_BACKEND = "amqp://"
 
-    CELERY_CONCURRENCY = 10
+    CELERY_TASK_SERIALIZER = "json"
+    CELERY_RESULT_SERIALIZER = "json"
+    CELERY_TIMEZONE = "Europe/Oslo"
+    CELERY_ENABLE_UTC = True
 
 To verify that your configuration file works properly, and does't
 contain any syntax errors, you can try to import it::
@@ -223,6 +264,42 @@ contain any syntax errors, you can try to import it::
 
 For a complete reference of configuration options, see :ref:`configuration`.
 
+To demonstrate the power of configuration files, this how you would
+route a misbehaving task to a dedicated queue:
+
+:file:`celeryconfig.py`:
+
+.. code-block:: python
+
+    CELERY_ROUTES = {
+        "tasks.add": "low-priority",
+    }
+
+Or instead of routing it you could rate limit the task
+instead, so that only 10 tasks of this type can execute in a minute
+(10/m):
+
+:file:`celeryconfig.py`:
+
+.. code-block:: python
+
+    CELERY_ANNOTATIONS = {
+        "tasks.add": {"rate_limit": "10/m"}
+    }
+
+But in fact, if you are using one of RabbitMQ, Redis or MongoDB as the
+broker then you can actually direct the workers to set new rate limit
+for the task at runtime::
+
+    $ python tasks.py rate_limit tasks.add 10/m
+    worker.example.com: OK
+        new rate limit set successfully
+
+See :ref:`guide-routing` to read more about task routing,
+and the :setting:`CELERY_ANNOTATIONS` setting for more about annotations,
+or :ref:`guide-monitoring` for more about remote control commands,
+and how to monitor what your workers are doing.
+
 Where to go from here
 =====================
 

docs/includes/introduction.txt

@@ -58,6 +58,7 @@ integration packages:
 .. _`SQLAlchemy`: http://www.sqlalchemy.org/
 .. _`Django`: http://djangoproject.com/
 .. _`Django ORM`: http://djangoproject.com/
+.. _`Memcached`: http://memcached.org/
 .. _`Eventlet`: http://eventlet.net/
 .. _`gevent`: http://gevent.org/
 .. _`Beanstalk`: http://kr.github.com/beanstalkd/
@@ -128,8 +129,8 @@ Features
 
     +-----------------+----------------------------------------------------+
     | Messaging       | Supported brokers include `RabbitMQ`_, `Redis`_,   |
-    |                 | `Beanstalk`_, `MongoDB`_, `CouchDB`_, and popular  |
-    |                 | SQL databases.                                     |
+    |                 | `MongoDB`_, `Beanstalk`_, SQL databases,           |
+    |                 | Amazon SQS and more.                               |
     +-----------------+----------------------------------------------------+
     | Fault-tolerant  | Excellent configurable error recovery when using   |
     |                 | `RabbitMQ`, ensures your tasks are never lost.     |
@@ -155,7 +156,7 @@ Features
     |                 | result store backend. You can wait for the result, |
     |                 | retrieve it later, or ignore it.                   |
     +-----------------+----------------------------------------------------+
-    | Result Stores   | Database, `MongoDB`_, `Redis`_, `Tokyo Tyrant`,    |
+    | Result Stores   | Database, `MongoDB`_, `Redis`_, `Memcached`_,      |
     |                 | `Cassandra`, or `AMQP`_ (message notification).    |
     +-----------------+----------------------------------------------------+
     | Webhooks        | Your tasks can also be HTTP callbacks, enabling    |