|
@@ -69,72 +69,77 @@ Caveats
|
|
|
|
|
|
.. _redis-caveat-fanout-prefix:
|
|
.. _redis-caveat-fanout-prefix:
|
|
|
|
|
|
-- Broadcast messages will be seen by all virtual hosts by default.
|
|
|
|
|
|
+Fanout prefix
|
|
|
|
+-------------
|
|
|
|
|
|
- You have to set a transport option to prefix the messages so that
|
|
|
|
- they will only be received by the active virtual host::
|
|
|
|
|
|
+Broadcast messages will be seen by all virtual hosts by default.
|
|
|
|
|
|
- BROKER_TRANSPORT_OPTIONS = {'fanout_prefix': True}
|
|
|
|
|
|
+You have to set a transport option to prefix the messages so that
|
|
|
|
+they will only be received by the active virtual host::
|
|
|
|
|
|
- Note that you will not be able to communicate with workers running older
|
|
|
|
- versions or workers that does not have this setting enabled.
|
|
|
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'fanout_prefix': True}
|
|
|
|
|
|
- This setting will be the default in the future, so better to migrate
|
|
|
|
- sooner rather than later.
|
|
|
|
|
|
+Note that you will not be able to communicate with workers running older
|
|
|
|
+versions or workers that does not have this setting enabled.
|
|
|
|
+
|
|
|
|
+This setting will be the default in the future, so better to migrate
|
|
|
|
+sooner rather than later.
|
|
|
|
|
|
.. _redis-caveat-fanout-patterns:
|
|
.. _redis-caveat-fanout-patterns:
|
|
|
|
|
|
-- Workers will receive all task related events by default.
|
|
|
|
|
|
+Fanout patterns
|
|
|
|
+---------------
|
|
|
|
|
|
- To avoid this you must set the ``fanout_patterns`` fanout option so that
|
|
|
|
- the workers may only subscribe to worker related events::
|
|
|
|
|
|
+Workers will receive all task related events by default.
|
|
|
|
|
|
- BROKER_TRANSPORT_OPTIONS = {'fanout_patterns': True}
|
|
|
|
|
|
+To avoid this you must set the ``fanout_patterns`` fanout option so that
|
|
|
|
+the workers may only subscribe to worker related events::
|
|
|
|
|
|
- Note that this change is backward incompatible so all workers in the
|
|
|
|
- cluster must have this option enabled, or else they will not be able to
|
|
|
|
- communicate.
|
|
|
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'fanout_patterns': True}
|
|
|
|
|
|
- This option will be enabled by default in the future.
|
|
|
|
|
|
+Note that this change is backward incompatible so all workers in the
|
|
|
|
+cluster must have this option enabled, or else they will not be able to
|
|
|
|
+communicate.
|
|
|
|
|
|
-- If a task is not acknowledged within the :ref:`redis-visibility_timeout`
|
|
|
|
- the task will be redelivered to another worker and executed.
|
|
|
|
|
|
+This option will be enabled by default in the future.
|
|
|
|
|
|
- This causes problems with ETA/countdown/retry tasks where the
|
|
|
|
- time to execute exceeds the visibility timeout; in fact if that
|
|
|
|
- happens it will be executed again, and again in a loop.
|
|
|
|
|
|
+Visibility timeout
|
|
|
|
+------------------
|
|
|
|
|
|
- So you have to increase the visibility timeout to match
|
|
|
|
- the time of the longest ETA you are planning to use.
|
|
|
|
|
|
+If a task is not acknowledged within the :ref:`redis-visibility_timeout`
|
|
|
|
+the task will be redelivered to another worker and executed.
|
|
|
|
|
|
- Note that Celery will redeliver messages at worker shutdown,
|
|
|
|
- so having a long visibility timeout will only delay the redelivery
|
|
|
|
- of 'lost' tasks in the event of a power failure or forcefully terminated
|
|
|
|
- workers.
|
|
|
|
|
|
+This causes problems with ETA/countdown/retry tasks where the
|
|
|
|
+time to execute exceeds the visibility timeout; in fact if that
|
|
|
|
+happens it will be executed again, and again in a loop.
|
|
|
|
|
|
- Periodic tasks will not be affected by the visibility timeout,
|
|
|
|
- as this is a concept separate from ETA/countdown.
|
|
|
|
|
|
+So you have to increase the visibility timeout to match
|
|
|
|
+the time of the longest ETA you are planning to use.
|
|
|
|
|
|
- You can increase this timeout by configuring a transport option
|
|
|
|
- with the same name::
|
|
|
|
|
|
+Note that Celery will redeliver messages at worker shutdown,
|
|
|
|
+so having a long visibility timeout will only delay the redelivery
|
|
|
|
+of 'lost' tasks in the event of a power failure or forcefully terminated
|
|
|
|
+workers.
|
|
|
|
|
|
- BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}
|
|
|
|
|
|
+Periodic tasks will not be affected by the visibility timeout,
|
|
|
|
+as this is a concept separate from ETA/countdown.
|
|
|
|
|
|
- The value must be an int describing the number of seconds.
|
|
|
|
|
|
+You can increase this timeout by configuring a transport option
|
|
|
|
+with the same name::
|
|
|
|
|
|
|
|
+ BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}
|
|
|
|
|
|
-- Monitoring events (as used by flower and other tools) are global
|
|
|
|
- and is not affected by the virtual host setting.
|
|
|
|
|
|
+The value must be an int describing the number of seconds.
|
|
|
|
|
|
- This is caused by a limitation in Redis. The Redis PUB/SUB channels
|
|
|
|
- are global and not affected by the database number.
|
|
|
|
|
|
+Key eviction
|
|
|
|
+------------
|
|
|
|
|
|
-- Redis may evict keys from the database in some situations
|
|
|
|
|
|
+Redis may evict keys from the database in some situations
|
|
|
|
|
|
- If you experience an error like::
|
|
|
|
|
|
+If you experience an error like::
|
|
|
|
|
|
- InconsistencyError, Probably the key ('_kombu.binding.celery') has been
|
|
|
|
- removed from the Redis database.
|
|
|
|
|
|
+ InconsistencyError, Probably the key ('_kombu.binding.celery') has been
|
|
|
|
+ removed from the Redis database.
|
|
|
|
|
|
- you may want to configure the redis-server to not evict keys by setting
|
|
|
|
- the ``timeout`` parameter to 0.
|
|
|
|
|
|
+you may want to configure the redis-server to not evict keys by setting
|
|
|
|
+the ``timeout`` parameter to 0 in the redis configuration file.
|