|
|
@@ -0,0 +1,127 @@
|
|
|
+.. _broker-sqs:
|
|
|
+
|
|
|
+==================
|
|
|
+ Using Amazon SQS
|
|
|
+==================
|
|
|
+
|
|
|
+.. _broker-sqs-installation:
|
|
|
+
|
|
|
+Installation
|
|
|
+============
|
|
|
+
|
|
|
+For the Amazon SQS support you have to install the `boto`_ library::
|
|
|
+
|
|
|
+ $ pip install -U boto
|
|
|
+
|
|
|
+.. _boto:
|
|
|
+ http://pypi.python.org/pypi/boto
|
|
|
+
|
|
|
+.. _broker-sqs-configuration:
|
|
|
+
|
|
|
+Configuration
|
|
|
+=============
|
|
|
+
|
|
|
+You have to specify SQS in the broker URL::
|
|
|
+
|
|
|
+ BROKER_URL = 'sqs://ABCDEFGHIJKLMNOPQRST:ZYXK7NiynGlTogH8Nj+P9nlE73sq3@'
|
|
|
+
|
|
|
+where the URL format is::
|
|
|
+
|
|
|
+ sqs://aws_access_key_id@aws_secret_access_key@
|
|
|
+
|
|
|
+remember to include the "@" at the end.
|
|
|
+
|
|
|
+The login credentials can also be set using the environment variables
|
|
|
+:envvar:`AWS_ACCESS_KEY_ID` and :envvar:`AWS_SECRET_ACCESS_KEY`,
|
|
|
+in that case the broker url may only be ``sqs://``.
|
|
|
+
|
|
|
+Options
|
|
|
+=======
|
|
|
+
|
|
|
+region
|
|
|
+------
|
|
|
+
|
|
|
+The default region is ``us-east-1`` but you can select another region
|
|
|
+by configuring the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
|
|
|
+
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'region': 'eu-west-1'}
|
|
|
+
|
|
|
+.. seealso::
|
|
|
+
|
|
|
+ An overview of Amazon Web Services regions can be found here:
|
|
|
+
|
|
|
+ http://aws.amazon.com/about-aws/globalinfrastructure/
|
|
|
+
|
|
|
+visibility_timeout
|
|
|
+------------------
|
|
|
+
|
|
|
+The visibility timeout defines the number of seconds to wait
|
|
|
+for the worker to acknowledge the task before the message is redelivered
|
|
|
+to another worker. Also see caveats below.
|
|
|
+
|
|
|
+This option is set via the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
|
|
|
+
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600} # 1 hour.
|
|
|
+
|
|
|
+The default visibility timeout is 120 seconds.
|
|
|
+
|
|
|
+queue_name_prefix
|
|
|
+-----------------
|
|
|
+
|
|
|
+By default Celery will not assign any prefix to the queue names,
|
|
|
+If you have other services using SQS you can configure it do so
|
|
|
+using the :setting:`BROKER_TRANSPORT_OPTIONS` setting::
|
|
|
+
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'queue_name_prefix': 'celery-'}
|
|
|
+
|
|
|
+
|
|
|
+.. _sqs-caveats:
|
|
|
+
|
|
|
+Caveats
|
|
|
+=======
|
|
|
+
|
|
|
+- If a task is not acknowledged within the ``visibility_timeout``,
|
|
|
+ the task will be redelivered to another worker and executed.
|
|
|
+
|
|
|
+ This causes problems with ETA/countdown/retry tasks where the
|
|
|
+ time to execute exceeds the visibility timeout; in fact if that
|
|
|
+ happens it will be executed again, and again in a loop.
|
|
|
+
|
|
|
+ So you have to increase the visibility timeout to match
|
|
|
+ the time of the longest ETA you are planning to use.
|
|
|
+
|
|
|
+ Note that Celery will redeliver messages at worker shutdown,
|
|
|
+ so having a long visibility timeout will only delay the redelivery
|
|
|
+ of 'lost' tasks in the event of a power failure or forcefully terminated
|
|
|
+ workers.
|
|
|
+
|
|
|
+ Periodic tasks will not be affected by the visibility timeout,
|
|
|
+ as it is a concept separate from ETA/countdown.
|
|
|
+
|
|
|
+ The maximum visibility timeout supported by AWS as of this writing
|
|
|
+ is 12 hours (43200 seconds)::
|
|
|
+
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}
|
|
|
+
|
|
|
+- SQS does not yet support worker remote control commands.
|
|
|
+
|
|
|
+- SQS does not yet support events, and so cannot be used with
|
|
|
+ :program:`celery events`, :program:`celerymon` or the Django Admin
|
|
|
+ monitor.
|
|
|
+
|
|
|
+.. _sqs-results-configuration:
|
|
|
+
|
|
|
+Results
|
|
|
+-------
|
|
|
+
|
|
|
+Multiple products in the Amazon Web Services family could be a good candidate
|
|
|
+to store or publish results with, but there is no such result backend included
|
|
|
+at this point.
|
|
|
+
|
|
|
+.. warning::
|
|
|
+
|
|
|
+ Do not use the ``amqp`` backend with SQS.
|
|
|
+
|
|
|
+ It will create one queue for every task, and the queues will
|
|
|
+ not be collected. This could cost you money that would be better
|
|
|
+ spent contributing an Amazon WS result store backend back to Celery :)
|
Ask Solem