Revert "Deprecates support for using Redis as a broker (Issue #3274)"

This reverts commit 3c1632f66e0d4ff17f5b85ac7fda5ba15c2406c1.

README.rst

@@ -67,8 +67,9 @@ so we do not support Microsoft Windows.
 Please do not open any issues related to that platform.
 
 *Celery* is usually used with a message broker to send and receive messages.
-The RabbitMQ transports is feature complete, but there's also Qpid and Amazon
-SQS broker support.
+The RabbitMQ, Redis transports are feature complete,
+but there's also experimental support for a myriad of other solutions, including
+using SQLite for local development.
 
 *Celery* can run on a single machine, on multiple machines, or even
 across datacenters.
@@ -137,7 +138,7 @@ It supports...
 
     - **Message Transports**
 
-        - RabbitMQ_, Amazon SQS
+        - RabbitMQ_, Redis_, Amazon SQS
 
     - **Concurrency**
 
@@ -243,7 +244,7 @@ separating them by commas.
 
     $ pip install "celery[librabbitmq]"
 
-    $ pip install "celery[librabbitmq,auth,msgpack]"
+    $ pip install "celery[librabbitmq,redis,auth,msgpack]"
 
 The following bundles are available:
 
@@ -274,6 +275,9 @@ Transports and Backends
 :``celery[librabbitmq]``:
     for using the librabbitmq C library.
 
+:``celery[redis]``:
+    for using Redis as a message transport or as a result backend.
+
 :``celery[sqs]``:
     for using Amazon SQS as a message transport (*experimental*).
 

celery/app/__init__.py

@@ -101,11 +101,11 @@ def shared_task(*args, **kwargs):
         ... def add(x, y):
         ...     return x + y
 
-        >>> app1 = Celery(broker='amqp://A.example.com')
+        >>> app1 = Celery(broker='amqp://')
         >>> add.app is app1
         True
 
-        >>> app2 = Celery(broker='amqp://B.example.com')
+        >>> app2 = Celery(broker='redis://')
         >>> add.app is app2
     """
 

celery/bin/celery.py

@@ -645,7 +645,7 @@ class _RemoteControl(Command):
 class inspect(_RemoteControl):
     """Inspect the worker at runtime.
 
-    Availability: RabbitMQ (AMQP) transport.
+    Availability: RabbitMQ (AMQP) and Redis transports.
 
     Examples:
         .. code-block:: console
@@ -692,7 +692,7 @@ class inspect(_RemoteControl):
 class control(_RemoteControl):
     """Workers remote control.
 
-    Availability: RabbitMQ (AMQP) transport.
+    Availability: RabbitMQ (AMQP), Redis, and MongoDB transports.
 
     Examples:
         .. code-block:: console
@@ -786,6 +786,7 @@ class migrate(Command):
         .. code-block:: console
 
             $ celery migrate amqp://A.example.com amqp://guest@B.example.com//
+            $ celery migrate redis://localhost amqp://guest@localhost//
     """
 
     args = '<source_url> <dest_url>'

docs/faq.rst

@@ -184,9 +184,20 @@ Do I have to use AMQP/RabbitMQ?
 
 **Answer**: No.
 
-Although using RabbitMQ is recommended you can also use SQS or Qpid.
+Although using RabbitMQ is recommended you can also use Redis, SQS or Qpid.
 See :ref:`brokers` for more information.
 
+Redis as a broker won't perform as well as
+an AMQP broker, but the combination RabbitMQ as broker and Redis as a result
+store is commonly used.  If you have strict reliability requirements you are
+encouraged to use RabbitMQ or another AMQP broker. Some transports also uses
+polling, so they are likely to consume more resources. However, if you for
+some reason are not able to use AMQP, feel free to use these alternatives.
+They will probably work fine for most use cases, and note that the above
+points are not specific to Celery; If using Redis/database as a queue worked
+fine for you before, it probably will now. You can always upgrade later
+if you need to.
+
 .. _faq-is-celery-multilingual:
 
 Is Celery multilingual?
@@ -263,7 +274,8 @@ Does it work on FreeBSD?
 
 **Answer:** Depends
 
-When using the RabbitMQ (AMQP) it should work out of the box.
+When using the RabbitMQ (AMQP) and Redis transports it should work
+out of the box.
 
 For other transports the compatibility prefork pool is
 used which requires a working POSIX semaphore implementation,
@@ -538,11 +550,12 @@ What features are not supported when not using an AMQP broker?
 This is an incomplete list of features not available when
 using the virtual transports:
 
-    * Remote control commands.
+    * Remote control commands (supported only by Redis).
 
     * Monitoring with events may not work in all virtual transports.
 
     * The `header` and `fanout` exchange types
+        (`fanout` is supported by Redis).
 
 .. _faq-tasks:
 
@@ -757,6 +770,7 @@ Does celery support task priorities?
 **Answer**: Yes.
 
 RabbitMQ supports priorities since version 3.5.0.
+Redis transport emulates support of priorities.
 
 You can also prioritize work by routing high priority tasks
 to different workers.  In the real world this may actually work better

docs/getting-started/brokers/index.rst

@@ -18,6 +18,7 @@ Broker Instructions
     :maxdepth: 1
 
     rabbitmq
+    redis
     sqs
 
 .. _broker-overview:
@@ -34,6 +35,8 @@ individual transport (see :ref:`broker_toc`).
 +---------------+--------------+----------------+--------------------+
 | *RabbitMQ*    | Stable       | Yes            | Yes                |
 +---------------+--------------+----------------+--------------------+
+| *Redis*       | Stable       | Yes            | Yes                |
++---------------+--------------+----------------+--------------------+
 | *Amazon SQS*  | Stable       | No             | No                 |
 +---------------+--------------+----------------+--------------------+
 | *Zookeeper*   | Experimental | No             | No                 |

docs/getting-started/brokers/redis.rst

@@ -0,0 +1,172 @@
+.. _broker-redis:
+
+=============
+ Using Redis
+=============
+
+.. _broker-redis-installation:
+
+Installation
+============
+
+For the Redis support you have to install additional dependencies.
+You can install both Celery and these dependencies in one go using
+the ``celery[redis]`` :ref:`bundle <bundles>`:
+
+.. code-block:: console
+
+    $ pip install -U celery[redis]
+
+.. _broker-redis-configuration:
+
+Configuration
+=============
+
+Configuration is easy, just configure the location of
+your Redis database:
+
+.. code-block:: python
+
+    app.conf.broker_url = 'redis://localhost:6379/0'
+
+Where the URL is in the format of:
+
+.. code-block:: text
+
+    redis://:password@hostname:port/db_number
+
+all fields after the scheme are optional, and will default to ``localhost``
+on port 6379, using database 0.
+
+If a Unix socket connection should be used, the URL needs to be in the format:
+
+.. code-block:: text
+
+    redis+socket:///path/to/redis.sock
+
+Specifying a different database number when using a Unix socket is possible
+by adding the ``virtual_host`` parameter to the URL:
+
+.. code-block:: text
+
+    redis+socket:///path/to/redis.sock?virtual_host=db_number
+
+.. _redis-visibility_timeout:
+
+Visibility Timeout
+------------------
+
+The visibility timeout defines the number of seconds to wait
+for the worker to acknowledge the task before the message is redelivered
+to another worker.  Be sure to see :ref:`redis-caveats` below.
+
+This option is set via the :setting:`broker_transport_options` setting:
+
+.. code-block:: python
+
+    app.conf.broker_transport_options = {'visibility_timeout': 3600}  # 1 hour.
+
+The default visibility timeout for Redis is 1 hour.
+
+.. _redis-results-configuration:
+
+Results
+-------
+
+If you also want to store the state and return values of tasks in Redis,
+you should configure these settings::
+
+    app.conf.result_backend = 'redis://localhost:6379/0'
+
+For a complete list of options supported by the Redis result backend, see
+:ref:`conf-redis-result-backend`
+
+.. _redis-caveats:
+
+Caveats
+=======
+
+.. _redis-caveat-fanout-prefix:
+
+Fanout prefix
+-------------
+
+Broadcast messages will be seen by all virtual hosts by default.
+
+You have to set a transport option to prefix the messages so that
+they will only be received by the active virtual host:
+
+.. code-block:: python
+
+    app.conf.broker_transport_options = {'fanout_prefix': True}
+
+Note that you will not be able to communicate with workers running older
+versions or workers that does not have this setting enabled.
+
+This setting will be the default in the future, so better to migrate
+sooner rather than later.
+
+.. _redis-caveat-fanout-patterns:
+
+Fanout patterns
+---------------
+
+Workers will receive all task related events by default.
+
+To avoid this you must set the ``fanout_patterns`` fanout option so that
+the workers may only subscribe to worker related events:
+
+.. code-block:: python
+
+    app.conf.broker_transport_options = {'fanout_patterns': True}
+
+Note that this change is backward incompatible so all workers in the
+cluster must have this option enabled, or else they will not be able to
+communicate.
+
+This option will be enabled by default in the future.
+
+Visibility timeout
+------------------
+
+If a task is not acknowledged within the :ref:`redis-visibility_timeout`
+the task will be redelivered to another worker and executed.
+
+This causes problems with ETA/countdown/retry tasks where the
+time to execute exceeds the visibility timeout; in fact if that
+happens it will be executed again, and again in a loop.
+
+So you have to increase the visibility timeout to match
+the time of the longest ETA you are planning to use.
+
+Note that Celery will redeliver messages at worker shutdown,
+so having a long visibility timeout will only delay the redelivery
+of 'lost' tasks in the event of a power failure or forcefully terminated
+workers.
+
+Periodic tasks will not be affected by the visibility timeout,
+as this is a concept separate from ETA/countdown.
+
+You can increase this timeout by configuring a transport option
+with the same name:
+
+.. code-block:: python
+
+    app.conf.broker_transport_options = {'visibility_timeout': 43200}
+
+The value must be an int describing the number of seconds.
+
+Key eviction
+------------
+
+Redis may evict keys from the database in some situations
+
+If you experience an error like:
+
+.. code-block:: text
+
+    InconsistencyError: Probably the key ('_kombu.binding.celery') has been
+    removed from the Redis database.
+
+then you may want to configure the :command:`redis-server` to not evict keys
+by setting the ``timeout`` parameter to 0 in the redis configuration file.

docs/getting-started/brokers/sqs.rst

@@ -100,7 +100,8 @@ setting::
 
 Very frequent polling intervals can cause *busy loops*, which results in the
 worker using a lot of CPU time.  If you need sub-millisecond precision you
-should consider using another transport, like `RabbitMQ <broker-amqp>`.
+should consider using another transport, like `RabbitMQ <broker-amqp>`,
+or `Redis <broker-redis>`.
 
 Queue Prefix
 ------------

docs/getting-started/first-steps-with-celery.rst

@@ -69,6 +69,15 @@ platforms, including Microsoft Windows:
 
     http://www.rabbitmq.com/download.html
 
+Redis
+-----
+
+`Redis`_ is also feature-complete, but is more susceptible to data loss in
+the event of abrupt termination or power failures. Detailed information about using Redis:
+
+    :ref:`broker-redis`
+
+.. _`Redis`: http://redis.io/
 
 Other brokers
 -------------
@@ -119,7 +128,8 @@ this is needed so that names can be automatically generated, the second
 argument is the broker keyword argument which specifies the URL of the
 message broker you want to use, using RabbitMQ here, which is already the
 default option.  See :ref:`celerytut-broker` above for more choices,
-e.g. for RabbitMQ you can use ``amqp://localhost``.
+e.g. for RabbitMQ you can use ``amqp://localhost``, or for Redis you can
+use ``redis://localhost``.
 
 You defined a single task, called ``add``, which returns the sum of two numbers.
 
@@ -208,7 +218,7 @@ you choose to use a configuration module):
 
     app = Celery('tasks', backend='rpc://', broker='amqp://')
 
-Or if you want to use Redis as the result backend, but use RabbitMQ as
+Or if you want to use Redis as the result backend, but still use RabbitMQ as
 the message broker (a popular combination):
 
 .. code-block:: python
@@ -363,8 +373,8 @@ instead, so that only 10 tasks of this type can be processed in a minute
         'tasks.add': {'rate_limit': '10/m'}
     }
 
-If you are using RabbitMQ as broker you can also direct the
-workers to set a new rate limit
+If you are using RabbitMQ or Redis as the
+broker then you can also direct the workers to set a new rate limit
 for the task at runtime:
 
 .. code-block:: console

docs/getting-started/introduction.rst

@@ -58,8 +58,9 @@ What do I need?
     Please do not open any issues related to that platform.
 
 *Celery* requires a message transport to send and receive messages.
-The RabbitMQ broker transport is feature complete,
-but there's also support for SQS and Apache Qpid.
+The RabbitMQ and Redis broker transports are feature complete,
+but there's also support for a myriad of other experimental solutions, including
+using SQLite for local development.
 
 *Celery* can run on a single machine, on multiple machines, or even
 across data centers.
@@ -126,7 +127,7 @@ Celery is…
 
         - **Brokers**
 
-            - :ref:`RabbitMQ <broker-rabbitmq>`
+            - :ref:`RabbitMQ <broker-rabbitmq>`, :ref:`Redis <broker-redis>`,
             - :ref:`Amazon SQS <broker-sqs>` and more…
 
         - **Concurrency**

docs/getting-started/next-steps.rst

@@ -650,7 +650,7 @@ power of AMQP routing, see the :ref:`Routing Guide <guide-routing>`.
 Remote Control
 ==============
 
-If you're using RabbitMQ (AMQP) or Qpid as a broker then
+If you're using RabbitMQ (AMQP), Redis or Qpid as the broker then
 you can control and inspect the worker at runtime.
 
 For example you can see what tasks the worker is currently working on:

docs/glossary.rst

@@ -12,7 +12,7 @@ Glossary
         will cause the message to be redelivered.   Exactly when a
         transaction is considered a failure varies by transport.  In AMQP the
         transaction fails when the connection/channel is closed (or lost),
-        but in SQS the transaction times out after a configurable amount
+        but in Redis/SQS the transaction times out after a configurable amount
         of time (the ``visibility_timeout``).
 
     ack

docs/includes/installation.txt

@@ -34,7 +34,7 @@ separating them by commas.
 
     $ pip install "celery[librabbitmq]"
 
-    $ pip install "celery[librabbitmq,auth,msgpack]"
+    $ pip install "celery[librabbitmq,redis,auth,msgpack]"
 
 The following bundles are available:
 
@@ -65,6 +65,9 @@ Transports and Backends
 :``celery[librabbitmq]``:
     for using the librabbitmq C library.
 
+:``celery[redis]``:
+    for using Redis as a message transport or as a result backend.
+
 :``celery[sqs]``:
     for using Amazon SQS as a message transport (*experimental*).
 

docs/includes/introduction.txt

@@ -59,8 +59,9 @@ so we do not support Microsoft Windows.
 Please do not open any issues related to that platform.
 
 *Celery* is usually used with a message broker to send and receive messages.
-The RabbitMQ transports is feature complete, but there's also Qpid and Amazon
-SQS broker support.
+The RabbitMQ, Redis transports are feature complete,
+but there's also experimental support for a myriad of other solutions, including
+using SQLite for local development.
 
 *Celery* can run on a single machine, on multiple machines, or even
 across datacenters.
@@ -129,7 +130,7 @@ It supports…
 
     - **Message Transports**
 
-        - RabbitMQ_, Amazon SQS
+        - RabbitMQ_, Redis_, Amazon SQS
 
     - **Concurrency**
 

docs/userguide/calling.rst

@@ -500,4 +500,4 @@ AMQP's full routing capabilities. Interested parties may read the
 
     A number between `0` and `255`, where `255` is the highest priority.
 
-    Supported by: RabbitMQ
+    Supported by: RabbitMQ, Redis (priority reversed, 0 is highest).

docs/userguide/configuration.rst

@@ -589,7 +589,7 @@ Default is to expire after 1 day.
 .. note::
 
     For the moment this only works with the AMQP, database, cache,
-    and Redis result backends.
+    and Redis backends.
 
     When using the database backend, `celery beat` must be
     running for the results to be expired.
@@ -1689,7 +1689,7 @@ A dict of additional options passed to the underlying transport.
 
 See your transport user manual for supported options (if any).
 
-Example setting the visibility timeout (supported by SQS
+Example setting the visibility timeout (supported by Redis and SQS
 transports):
 
 .. code-block:: python

docs/userguide/monitoring.rst

@@ -173,7 +173,7 @@ Commands
 
     .. code-block:: console
 
-        $ celery -A proj migrate amqp://A.example.com amqp://B.example.cmo
+        $ celery -A proj migrate redis://localhost amqp://localhost
 
   This command will migrate all the tasks on one broker to another.
   As this command is new and experimental you should be sure to have
@@ -299,6 +299,8 @@ Broker URL can also be passed through the
 .. code-block:: console
 
     $ celery flower --broker=amqp://guest:guest@localhost:5672//
+    or
+    $ celery flower --broker=redis://guest:guest@localhost:6379/0
 
 Then, you can visit flower in your web browser :
 
@@ -415,6 +417,48 @@ Finding the amount of memory allocated to a queue:
 :Tip: Adding the ``-q`` option to `rabbitmqctl(1)`_ makes the output
       easier to parse.
 
+
+.. _monitoring-redis:
+
+Redis
+=====
+
+If you're using Redis as the broker, you can monitor the Celery cluster using
+the `redis-cli(1)` command to list lengths of queues.
+
+.. _monitoring-redis-queues:
+
+Inspecting queues
+-----------------
+
+Finding the number of tasks in a queue:
+
+.. code-block:: console
+
+    $ redis-cli -h HOST -p PORT -n DATABASE_NUMBER llen QUEUE_NAME
+
+The default queue is named `celery`. To get all available queues, invoke:
+
+.. code-block:: console
+
+    $ redis-cli -h HOST -p PORT -n DATABASE_NUMBER keys \*
+
+.. note::
+
+    Queue keys only exists when there are tasks in them, so if a key
+    does not exist it simply means there are no messages in that queue.
+    This is because in Redis a list with no elements in it is automatically
+    removed, and hence it won't show up in the `keys` command output,
+    and `llen` for that list returns 0.
+
+    Also, if you're using Redis for other purposes, the
+    output of the `keys` command will include unrelated values stored in
+    the database.  The recommended way around this is to use a
+    dedicated `DATABASE_NUMBER` for Celery, you can also use
+    database numbers to separate Celery applications from each other (virtual
+    hosts), but this will not affect the monitoring events used by e.g. Flower
+    as Redis pub/sub commands are global rather than database based.
+
 .. _monitoring-munin:
 
 Munin

docs/userguide/routing.rst

@@ -111,7 +111,7 @@ A queue named `"video"` will be created with the following settings:
      'exchange_type': 'direct',
      'routing_key': 'video'}
 
-The non-AMQP transports like `SQS` do not support exchanges,
+The non-AMQP backends like `Redis` or `SQS` do not support exchanges,
 so they require the exchange to have the same name as the queue. Using this
 design ensures it will work for them as well.
 

docs/userguide/workers.rst

@@ -233,7 +233,7 @@ Remote control
     listed below.  See :ref:`monitoring-control` for more information.
 
 :pool support: *prefork, eventlet, gevent*, blocking:*solo* (see note)
-:broker support: *amqp*
+:broker support: *amqp, redis*
 
 Workers have the ability to be remote controlled using a high-priority
 broadcast message queue.  The commands can be directed to all, or a specific
@@ -313,7 +313,7 @@ Commands
 ``revoke``: Revoking tasks
 --------------------------
 :pool support: all, terminate only supported by prefork
-:broker support: *amqp*
+:broker support: *amqp, redis*
 :command: :program:`celery -A proj control revoke <task_id>`
 
 All worker nodes keeps a memory of revoked task ids, either in-memory or
@@ -412,8 +412,8 @@ name:
 See also :ref:`worker-files`
 
 Note that remote control commands must be working for revokes to work.
-Remote control commands are only supported by the RabbitMQ (amqp) at this
-point.
+Remote control commands are only supported by the RabbitMQ (amqp) and Redis
+at this point.
 
 .. _worker-time-limits:
 
@@ -467,7 +467,7 @@ Changing time limits at run-time
 --------------------------------
 .. versionadded:: 2.3
 
-:broker support: *amqp*
+:broker support: *amqp, redis*
 
 There is a remote control command that enables you to change both soft
 and hard time limits for a task — named ``time_limit``.
@@ -852,12 +852,22 @@ The output will include the following fields:
 
     * ``transport``
 
-        Name of transport used (e.g. ``amqp``)
+        Name of transport used (e.g. ``amqp`` or ``redis``)
 
     * ``transport_options``
 
         Options passed to transport.
 
+    * ``uri_prefix``
+
+        Some transports expects the host name to be a URL.
+
+        .. code-block:: text
+
+            redis+socket:///tmp/redis.sock
+
+        In this example the URI-prefix will be ``redis``.
+
     * ``userid``
 
         User id used to connect to the broker with.

docs/whatsnew-4.0.rst

@@ -382,6 +382,27 @@ even asynchronously:
         check_arguments(*(args or ()), **(kwargs or {}))
     TypeError: add() takes exactly 2 arguments (1 given)
 
+Redis Events not backward compatible
+------------------------------------
+
+The Redis ``fanout_patterns`` and ``fanout_prefix`` transport
+options are now enabled by default, which means that workers
+running 4.0 cannot see workers running 3.1 on the default configuration,
+and vice versa.
+
+This is only related to monitor event messages, the workers should still
+execute tasks as normally.
+
+You can avoid this situation by configuring the 3.1 workers (and clients)
+to enable these settings, before upgrading to 4.0:
+
+.. code-block:: python
+
+    BROKER_TRANSPORT_OPTIONS = {
+        'fanout_patterns': True,
+        'fanout_prefix': True,
+    }
+
 Django: Auto-discover now supports Django app configurations
 ------------------------------------------------------------
 
@@ -680,7 +701,7 @@ Amazon SQS transport now officially supported.
 ----------------------------------------------
 
 The SQS broker transport has been rewritten to use async I/O and as such
-joins RabbitMQ and Qpid as officially supported transports.
+joins RabbitMQ and Redis as officially supported transports.
 
 The new implementation also takes advantage of long polling,
 and closes several issues related to using SQS as a broker.
@@ -984,7 +1005,8 @@ Tasks
 
 - The "anon-exchange" is now used for simple name-name direct routing.
 
-  This increases performance as it completely bypasses the routing table.
+  This increases performance as it completely bypasses the routing table,
+  in addition it also improves reliability for the Redis broker transport.
 
 - An empty ResultSet now evaluates to True.
 
@@ -1104,6 +1126,11 @@ Execution Pools
 
     Contributed by **Alexander Oblovatniy**.
 
+Transports
+~~~~~~~~~~
+
+- **Redis Transport**: The Redis transport now supports the
+  :setting:`broker_use_ssl` option.
 
 Programs
 ~~~~~~~~