Document SQS transport

docs/getting-started/brokers/sqs.rst

@@ -0,0 +1,127 @@
+.. _broker-sqs:
+
+==================
+ Using Amazon SQS
+==================
+
+.. _broker-sqs-installation:
+
+Installation
+============
+
+For the Amazon SQS support you have to install the `boto`_ library::
+
+    $ pip install -U boto
+
+.. _boto:
+    http://pypi.python.org/pypi/boto
+
+.. _broker-sqs-configuration:
+
+Configuration
+=============
+
+You have to specify SQS in the broker URL::
+
+    BROKER_URL = 'sqs://ABCDEFGHIJKLMNOPQRST:ZYXK7NiynGlTogH8Nj+P9nlE73sq3@'
+
+where the URL format is::
+
+    sqs://aws_access_key_id@aws_secret_access_key@
+
+remember to include the "@" at the end.
+
+The login credentials can also be set using the environment variables
+:envvar:`AWS_ACCESS_KEY_ID` and :envvar:`AWS_SECRET_ACCESS_KEY`,
+in that case the broker url may only be ``sqs://``.
+
+Options
+=======
+
+region
+------
+
+The default region is ``us-east-1`` but you can select another region
+by configuring the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
+
+    BROKER_TRANSPORT_OPTIONS = {'region': 'eu-west-1'}
+
+.. seealso::
+
+    An overview of Amazon Web Services regions can be found here:
+
+        http://aws.amazon.com/about-aws/globalinfrastructure/
+
+visibility_timeout
+------------------
+
+The visibility timeout defines the number of seconds to wait
+for the worker to acknowledge the task before the message is redelivered
+to another worker.  Also see caveats below.
+
+This option is set via the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
+
+    BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600}  # 1 hour.
+
+The default visibility timeout is 120 seconds.
+
+queue_name_prefix
+-----------------
+
+By default Celery will not assign any prefix to the queue names,
+If you have other services using SQS you can configure it do so
+using the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
+
+    BROKER_TRANSPORT_OPTIONS = {'queue_name_prefix': 'celery-'}
+
+
+.. _sqs-caveats:
+
+Caveats
+=======
+
+- If a task is not acknowledged within the ``visibility_timeout``,
+  the task will be redelivered to another worker and executed.
+
+    This causes problems with ETA/countdown/retry tasks where the
+    time to execute exceeds the visibility timeout; in fact if that
+    happens it will be executed again, and again in a loop.
+
+    So you have to increase the visibility timeout to match
+    the time of the longest ETA you are planning to use.
+
+    Note that Celery will redeliver messages at worker shutdown,
+    so having a long visibility timeout will only delay the redelivery
+    of 'lost' tasks in the event of a power failure or forcefully terminated
+    workers.
+
+    Periodic tasks will not be affected by the visibility timeout,
+    as it is a concept separate from ETA/countdown.
+
+    The maximum visibility timeout supported by AWS as of this writing
+    is 12 hours (43200 seconds)::
+
+        BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}
+
+- SQS does not yet support worker remote control commands.
+
+- SQS does not yet support events, and so cannot be used with
+  :program:`celery events`, :program:`celerymon` or the Django Admin
+  monitor.
+
+.. _sqs-results-configuration:
+
+Results
+-------
+
+Multiple products in the Amazon Web Services family could be a good candidate
+to store or publish results with, but there is no such result backend included
+at this point.
+
+.. warning::
+
+    Do not use the ``amqp`` backend with SQS.
+
+    It will create one queue for every task, and the queues will
+    not be collected.  This could cost you money that would be better
+    spent contributing an Amazon WS result store backend back to Celery :)

docs/getting-started/first-steps-with-celery.rst

@@ -75,7 +75,7 @@ for very small installations.  Celery can use the SQLAlchemy and Django ORM.
 
 In addition to the above, there are several other transport implementations
 to choose from, including :ref:`broker-mongodb`, :ref:`broker-django`,
-:ref:`broker-sqlalchemy`, and SQS.
+:ref:`broker-sqlalchemy`, and :ref:`Amazon SQS <broker-sqs>`.
 
 .. _`RabbitMQ`: http://www.rabbitmq.com/
 .. _`Redis`: http://redis.io/

docs/getting-started/intro.rst

@@ -85,7 +85,7 @@ Celery is…
             - :ref:`RabbitMQ <broker-rabbitmq>`, :ref:`Redis <broker-redis>`,
             - :ref:`MongoDB <broker-mongodb>`, :ref:`Beanstalk <broker-beanstalk>`,
             - :ref:`CouchDB <broker-couchdb>`, :ref:`SQLAlchemy <broker-sqlalchemy>`,
-            - :ref:`Django ORM <broker-django>`, and more…
+            - :ref:`Django ORM <broker-django>`, :ref:`Amazon SQS <broker-sqs>`, and more…
 
         - **Concurrency**