Merge branch 'master' into gossip2

Changelog

@@ -26,10 +26,35 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
 ======
 :release-date: TBA
 
+- Now depends on kombu 2.4.8
+
+    - [Redis] New and improved fair queue cycle algorithm (Kevin McCarthy).
+    - [Redis] Now uses a Redis-based mutex when restoring messages.
+    - [Redis] Number of messages that can be restored in one interval is no
+              longer limited (but can be set using the
+              ``unacked_restore_limit`` transport option.)
+    - Heartbeat value can be specified in broker URLs (Mher Movsisyan).
+    - Fixed problem with msgpack on Python 3 (Jasper Bryant-Greene).
+
+- Now depends on billiard 2.7.3.18
+
+- Celery can now be used with static analysis tools like PyDev/PyCharm/pylint
+  etc.
+
+- Development documentation has moved to Read The Docs.
+
+    The new URL is: http://docs.celeryproject.org/en/master
+
 - Event state's ``tasks_by_name`` applied limit before filtering by name.
 
     Fix contributed by Alexander A. Sosnovskiy.
 
+- New :setting:`CELERY_QUEUE_HA_POLICY` setting used to set the default
+  HA policy for queues when using RabbitMQ.
+
+- New method ``Task.subtask_from_request`` returns a subtask using the current
+  request.
+
 - Results get_many method did not respect timeout argument.
 
     Fix contributed by Remigiusz Modrzejewski
@@ -56,6 +81,8 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
 
 - :mod:`celery.contrib.batches` now works again.
 
+- Fixed missing whitespace in ``bdist_rpm`` requirements (Issue #1046).
+
 .. _version-3.0.11:
 
 3.0.11
@@ -115,10 +142,10 @@ If you're looking for versions prior to 3.0.x you should go to :ref:`history`.
 - Terminating a task now properly updates the state of the task to revoked,
   and sends a ``task-revoked`` event.
 
-- Multi: No longer parses --app option (Issue #1008).
-
 - Generic worker init script now waits for workers to shutdown by default.
 
+- Multi: No longer parses --app option (Issue #1008).
+
 - Multi: stop_verify command renamed to stopwait.
 
 - Daemonization: Now delays trying to create pidfile/logfile until after

docs/getting-started/next-steps.rst

@@ -115,8 +115,8 @@ Eventlet, Gevent, and threads (see :ref:`concurrency`).
 -- *Events* is an option that when enabled causes Celery to send
 monitoring messages (events) for actions occurring in the worker.
 These can be used by monitor programs like ``celery events``,
-celerymon and the Django-Celery admin monitor that you can read
-about in the :ref:`Monitoring and Management guide <guide-monitoring>`.
+and Flower - the real-time Celery monitor, which you can read about in
+the :ref:`Monitoring and Management guide <guide-monitoring>`.
 
 -- *Queues* is the list of queues that the worker will consume
 tasks from.  The worker can be told to consume from several queues

docs/history/changelog-2.1.rst

@@ -275,8 +275,9 @@ Important Notes
   if the database result backend is used.
 
 * django-celery now comes with a monitor for the Django Admin interface.
-  This can also be used if you're not a Django user.  See
-  :ref:`monitoring-django-admin` and :ref:`monitoring-nodjango` for more information.
+  This can also be used if you're not a Django user.
+  (Update: Django-Admin monitor has been replaced with Flower, see the
+  Monitoring guide).
 
 * If you get an error after upgrading saying:
   `AttributeError: 'module' object has no attribute 'system'`,

docs/userguide/monitoring.rst

@@ -171,30 +171,40 @@ You can specify a single, or a list of workers by using the
 
 .. _monitoring-flower:
 
-Celery Flower: Web interface
-----------------------------
+Flower: Real-time Celery web-monitor
+------------------------------------
 
-Celery Flower is a web based, real-time monitor and administration tool.
+Flower is a real-time web based monitor and administration tool for Celery.
+It is under active development, but is already an essential tool.
+Being the recommended monitor for Celery, it obsoletes the Django-Admin
+monitor, celerymon and the ncurses based monitor.
+
+Flower is pronounced like "flow", but you can also use the botanical version
+if you prefer.
 
 Features
 ~~~~~~~~
 
-- Shutdown or restart workers
-- View workers status (completed, running tasks, etc.)
-- View worker pool options (timeouts, processes, etc.)
-- Control worker pool size
-- View message broker options
-- View active queues, add or cancel queues
-- View processed task stats by type
-- View currently running tasks
-- View scheduled tasks
-- View reserved and revoked tasks
-- Apply time and rate limits
-- View all active configuration options
-- View all tasks (by type, by worker, etc.)
-- View all task options (arguments, start time, runtime, etc.)
-- Revoke or terminate tasks
-- View real-time execution graphs
+- Real-time monitoring using Celery Events
+
+    - Task progress and history.
+    - Ability to show task details (arguments, start time, runtime, and more)
+    - Graphs and statistics
+
+- Remote Control
+
+    - View worker status and statistics.
+    - Shutdown and restart worker instances.
+    - Control worker pool size and autoscale settings.
+    - View and modify the queues a worker instance consumes from.
+    - View currently running tasks
+    - View scheduled tasks (ETA/countdown)
+    - View reserved and revoked tasks
+    - Apply time and rate limits
+    - Configuration viewer
+    - Revoke or terminate tasks
+
+- HTTP API
 
 **Screenshots**
 
@@ -211,203 +221,22 @@ More screenshots_:
 Usage
 ~~~~~
 
-Install Celery Flower:
+You can use pip to install Flower:
 
 .. code-block:: bash
 
     $ pip install flower
 
-Launch Celery Flower and open http://localhost:5555 in browser:
+Running the flower command will start a web-server that you can visit:
 
 .. code-block:: bash
 
     $ celery flower
 
-.. _monitoring-django-admin:
-
-Django Admin Monitor
---------------------
-
-.. versionadded:: 2.1
-
-When you add `django-celery`_ to your Django project you will
-automatically get a monitor section as part of the Django admin interface.
-
-This can also be used if you're not using Celery with a Django project.
-
-*Screenshot*
-
-.. figure:: ../images/djangoceleryadmin2.jpg
-   :width: 700px
-
-.. _`django-celery`: http://pypi.python.org/pypi/django-celery
-
-
-.. _monitoring-django-starting:
-
-Starting the monitor
-~~~~~~~~~~~~~~~~~~~~
-
-The Celery section will already be present in your admin interface,
-but you won't see any data appearing until you start the snapshot camera.
-
-The camera takes snapshots of the events your workers sends at regular
-intervals, storing them in your database (See :ref:`monitoring-snapshots`).
-
-To start the camera run:
-
-.. code-block:: bash
-
-    $ python manage.py celerycam
-
-If you haven't already enabled the sending of events you need to do so:
-
-.. code-block:: bash
-
-    $ python manage.py celery control enable_events
-
-:Tip: You can enable events when the worker starts using the `-E` argument.
-
-Now that the camera has been started, and events have been enabled
-you should be able to see your workers and the tasks in the admin interface
-(it may take some time for workers to show up).
-
-The admin interface shows tasks, worker nodes, and even
-lets you perform some actions, like revoking and rate limiting tasks,
-or shutting down worker nodes.
-
-.. _monitoring-django-frequency:
-
-Shutter frequency
-~~~~~~~~~~~~~~~~~
-
-By default the camera takes a snapshot every second, if this is too frequent
-or you want to have higher precision, then you can change this using the
-``--frequency`` argument.  This is a float describing how often, in seconds,
-it should wake up to check if there are any new events:
-
-.. code-block:: bash
-
-    $ python manage.py celerycam --frequency=3.0
-
-The camera also supports rate limiting using the ``--maxrate`` argument.
-While the frequency controls how often the camera thread wakes up,
-the rate limit controls how often it will actually take a snapshot.
-
-The rate limits can be specified in seconds, minutes or hours
-by appending `/s`, `/m` or `/h` to the value.
-Example: ``--maxrate=100/m``, means "hundred writes a minute".
-
-The rate limit is off by default, which means it will take a snapshot
-for every ``--frequency`` seconds.
-
-The events also expire after some time, so the database doesn't fill up.
-Successful tasks are deleted after 1 day, failed tasks after 3 days,
-and tasks in other states after 5 days.
-
-.. _monitoring-django-reset:
-
-Resetting monitor data
-~~~~~~~~~~~~~~~~~~~~~~
+The default port is http://localhost:5555, but you can change this using the
+:option:`--port` argument::
 
-To reset the monitor data you need to clear out two models::
-
-    >>> from djcelery.models import WorkerState, TaskState
-
-    # delete worker history
-    >>> WorkerState.objects.all().delete()
-
-    # delete task history
-    >>> TaskState.objects.all().update(hidden=True)
-    >>> TaskState.objects.purge()
-
-.. _monitoring-django-expiration:
-
-Expiration
-~~~~~~~~~~
-
-By default monitor data for successful tasks will expire in 1 day,
-failed tasks in 3 days and pending tasks in 5 days.
-
-You can change the expiry times for each of these using
-adding the following settings to your :file:`settings.py`:
-
-.. code-block:: python
-
-    from datetime import timedelta
-
-    CELERYCAM_EXPIRE_SUCCESS = timedelta(hours=1)
-    CELERYCAM_EXPIRE_ERROR = timedelta(hours=2)
-    CELERYCAM_EXPIRE_PENDING = timedelta(hours=2)
-
-.. _monitoring-nodjango:
-
-Using outside of Django
-~~~~~~~~~~~~~~~~~~~~~~~
-
-`django-celery` also installs the :program:`djcelerymon` program. This
-can be used by non-Django users, and runs both a web server and a snapshot
-camera in the same process.
-
-**Installing**
-
-Using :program:`pip`:
-
-.. code-block:: bash
-
-    $ pip install -U django-celery
-
-or using :program:`easy_install`:
-
-.. code-block:: bash
-
-    $ easy_install -U django-celery
-
-**Running**
-
-:program:`djcelerymon` reads configuration from your Celery configuration
-module, and sets up the Django environment using the same settings:
-
-.. code-block:: bash
-
-    $ djcelerymon
-
-Database tables will be created the first time the monitor is run.
-By default an `sqlite3` database file named
-:file:`djcelerymon.db` is used, so make sure this file is writeable by the
-user running the monitor.
-
-If you want to store the events in a different database, e.g. MySQL,
-then you can configure the `DATABASE*` settings directly in your Celery
-config module.  See http://docs.djangoproject.com/en/dev/ref/settings/#databases
-for more information about the database options available.
-
-You will also be asked to create a superuser (and you need to create one
-to be able to log into the admin later)::
-
-    Creating table auth_permission
-    Creating table auth_group_permissions
-    [...]
-
-    You just installed Django's auth system, which means you don't
-    have any superusers defined.  Would you like to create
-    one now? (yes/no): yes
-    Username (Leave blank to use 'username'): username
-    Email address: me@example.com
-    Password: ******
-    Password (again): ******
-    Superuser created successfully.
-
-    [...]
-    Django version 1.2.1, using settings 'celeryconfig'
-    Development server is running at http://127.0.0.1:8000/
-    Quit the server with CONTROL-C.
-
-Now that the service is started you can visit the monitor
-at http://127.0.0.1:8000, and log in using the user you created.
-
-For a list of the command-line options supported by :program:`djcelerymon`,
-please see ``djcelerymon --help``.
+    $ open http://localhost:5555
 
 .. _monitoring-celeryev:
 
@@ -419,7 +248,8 @@ celery events: Curses Monitor
 `celery events` is a simple curses monitor displaying
 task and worker history.  You can inspect the result and traceback of tasks,
 and it also supports some management commands like rate limiting and shutting
-down workers.
+down workers.  This monitor was started as a proof of concept, and you
+probably want to use Flower instead.
 
 Starting:
 
@@ -451,23 +281,6 @@ For a complete list of options use ``--help``:
 
     $ celery events --help
 
-
-.. _monitoring-celerymon:
-
-celerymon: Web monitor
-----------------------
-
-`celerymon`_ is the ongoing work to create a web monitor.
-It's far from complete yet, and does currently only support
-a JSON API.  Help is desperately needed for this project, so if you,
-or someone you know would like to contribute templates, design, code
-or help this project in any way, please get in touch!
-
-:Tip: The Django admin monitor can be used even though you're not using
-      Celery with a Django project.  See :ref:`monitoring-nodjango`.
-
-.. _`celerymon`: http://github.com/celery/celerymon/
-
 .. _monitoring-rabbitmq:
 
 RabbitMQ
@@ -590,7 +403,7 @@ Events
 ======
 
 The worker has the ability to send a message whenever some event
-happens.  These events are then captured by tools like :program:`celerymon`
+happens.  These events are then captured by tools like Flower,
 and :program:`celery events` to monitor the cluster.
 
 .. _monitoring-snapshots:

requirements/default.txt

@@ -1,3 +1,3 @@
 pytz
-billiard>=2.7.3.17
-kombu>=2.4.7,<3.0
+billiard>=2.7.3.18
+kombu>=2.4.8,<3.0

setup.cfg

@@ -15,5 +15,5 @@ upload-dir = docs/.build/html
 
 [bdist_rpm]
 requires = pytz
-           billiard >= 2.7.3.17
-           kombu >= 2.4.7
+           billiard >= 2.7.3.18
+           kombu >= 2.4.8