Intro document

docs/getting-started/intro.rst

@@ -9,79 +9,142 @@
 What are Task Queues?
 =====================
 
+.. compound::
 
-Celery Features
-===============
+    "The quick brown fox jumps over the lazy dog"
+
+        said the Farmer.
 
-- Messaging Transports (Brokers)
+            but little did *he* know...
 
-Celery requires a message broker to send and receive messages,
+
+What do I need?
+===============
+
+*Celery* requires a message broker to send and receive messages,
 but this term has been stretched to include everything from
-financial-grade messaging systems to your fridge (no offense)
+financial-grade messaging systems to your fridge.
+
+You can run *Celery* on a single or multiple machines, or even
+across datacenters.
+
+Celery runs on Python 2.6/2.7/3.2, PyPy and Jython.
+
+
+Celery is…
+==========
+
+.. topic:: ”
+
+    - **Simple**
+
+        Bla bla bla., yaddi blabla, bla bli bla do re mi, bla bi do,
+        re mi bla do blah blah yadda blah blah blah blah.
+
+    - **Fast**
+
+        Bla bla bla. librabbitmq, yaddi blabla lightweight, bla bli bla do re mi, bla bi do,
+        re mi bla do blah blah yadda blah blah blah blah.
+
+    - **Highly Available**
+
+        Workers and clients will automatically retry in the event
+        of connection loss or failure, and some brokers support
+        HA in way of *Master/Master* or -- *Master/Slave* replication.
+
+    - **Flexible**
+
+        Almost every part of *Celery* can be extended or used on its own,
+        Custom pool implementations, serializers, compression schemes, logging,
+        schedulers, consumers, producers, autoscalers, broker transorts and much more.
+
+
+.. topic:: It supports
+
+    .. hlist::
+        :columns: 2
+
+        - **Brokers**
+
+            :ref:`RabbitMQ <broker-rabbitmq>`, :ref:`Redis <broker-redis>`,
+            :ref:`MongoDB <broker-mongodb>`, :ref:`Beanstalk <broker-beanstalk>`,
+            :ref:`CouchDB <broker-couchdb>`, or
+            :ref:`SQLAlchemy <broker-sqlalchemy>`/:ref:`Django ORM <broker-django>`.
+
+        - **Concurrency**
+
+            multiprocessing, Eventlet_, gevent_ and threads.
+
+        - **Serialization & Compression**
+
+            Messages can be serialized using *pickle*, *json*, *yaml*, *msgpack*,
+            and optionally compressed using *zlib* or *bzip2*
+
+        - **Result Stores**
 
-RabbitMQ, Redis, MongoDB, Amazon SQS, CouchDB, Beanstalk, Apache ZooKeeper,
-or Databases (SQLAlchemy/Django ORM).
+            AMQP, Redis, memcached, MongoDB, SQLAlchemy/Django ORM, Apache Cassandra.
 
-- HA
 
-Both clients and workers will automatically retry in the event
-of connection loss or failure, and some brokers support
-HA in way of Master/Master or Master/Slave replication.
+.. topic:: Features
 
-- Multiple Serializers
+    .. hlist::
+        :columns: 2
 
-Messages can be serialized using pickle, json, yaml, msgpack or
-even custom serializers.  In addition Celery ships with a special
-serializer that signs messages using cryptographic hashes.
+        - **Monitoring**
 
-- Compression
+            The stream of monitoring events emit by the worker are used
+            by built-in and external tools to tell you what your cluster
+            is doing in real-time.
 
-Messages can be compressed using zlib, bzip2 or custom
-compression schemes defined by the user.
+            :ref:`Read more… <guide-monitoring>`.
 
-Worker
-------
+        - **Time Limits & Rate Limits**
 
-- Monitoring
+            You can control how many tasks can be executed per second/minute/hour,
+            or how long a task can be allowed to run, and this can be set as
+            a default, for a specific worker or individually for each task type.
 
-Workers emit a stream of monitoring events, that is used
-by monitoring tools like `celery events`, `celerymon` and
-the Django Admin monitor.  Users can write custom event consumers
-to analyze what the workers are doing in real-time.
+            :ref:`Read more… <worker-time-limits>`.
 
-- Time Limits
+        - **Autoreloading**
 
-Tasks can be enforced a strict time to run, and this can be set as a default
-for all tasks, for a specific worker, or individually for each task.
+            While in development workers can be configured to automatically reload source
+            code as it changes.
 
-.. sidebar:: Soft, or hard?
+            :ref:`Read more… <worker-autoreloading>`.
 
-    The time limit is set in two values, `soft` and `hard`.
-    The soft time limit allows the task to catch an exception
-    to clean up before it is killed: the hard timeout is not catchable
-    and force terminates the task.
+        - **Autoscaling**
 
-- Autoreloading
+            Dynamically resizing the worker pool depending on load,
+            or custom metrics specified by the user, used to limit
+            memory usage in shared hosting/cloud environment or to
+            enforce a given quality of service.
 
-While developing the worker can be set to automatically reload
-when the source code for a task changes.
+            :ref:`Read more… <worker-autoscaling>`.
 
-- Autoscaling
+        - **Resource Leak Protection**
 
-The worker pool can be dynamically resized based on worker load,
-and autoscaling rules can be customized by the user.
+            The :option:`--maxtasksperchild` option is used for user tasks
+            leaking resources, like memory or file descriptors, that
+            are out simply out of your control.
 
-- Memory Leak Cleanup
+            :ref:`Read more… <worker-maxtasksperchild>`.
 
-Sometimes tasks contain memory leaks that are out of the
-developers control, or the task allocated other resources
-that cannot be cleaned up.  In this case the worker supports
-a :option:`--maxtasksperchild` argument that defines how
-many task a given pool process can execute before it's
-replaced by a fresh process.
+        - **User Components**
 
-- User components
+            Each worker component can be customized, and additional components
+            can be defined by the user.  The worker is built up using "boot steps" — a
+            dependency graph enabling fine grained control of the workers
+            internals.
 
-Each worker component can be customized, and additional components
-can be defined by the user simply by defining a new boot steps
-that will be loaded as part of the workers dependency graph.
+.. _`RabbitMQ`: http://www.rabbitmq.com/
+.. _`Redis`: http://code.google.com/p/redis/
+.. _`SQLAlchemy`: http://www.sqlalchemy.org/
+.. _`Django ORM`: http://djangoproject.com/
+.. _`Eventlet`: http://eventlet.net/
+.. _`gevent`: http://gevent.org/
+.. _`Beanstalk`: http://kr.github.com/beanstalkd/
+.. _`MongoDB`: http://mongodb.org/
+.. _`CouchDB`: http://couchdb.apache.org/
+.. _`Amazon SQS`: http://aws.amazon.com/sqs/
+.. _`Apache ZooKeeper`: http://zookeeper.apache.org/

docs/userguide/workers.rst

@@ -6,6 +6,7 @@
 
 .. contents::
     :local:
+    :depth: 1
 
 .. _worker-starting:
 
@@ -72,10 +73,11 @@ arguments as it was started with.
 
 .. note::
 
-    This will only work if the worker is running in the background as
-    a daemon (it does not have a controlling terminal).
+    Restarting by :sig:`HUP` only works if the worker is running
+    in the background as a daemon (it does not have a controlling
+    terminal).
 
-    Restarting by HUP is disabled on OS X because of a limitation on
+    :sig:`HUP` is disabled on OS X because of a limitation on
     that platform.
 
 
@@ -135,12 +137,19 @@ transports at this point.
 
 .. _worker-time-limits:
 
-Time limits
+Time Limits
 ===========
 
 .. versionadded:: 2.0
 
-:supported pools: processes
+pool support: *processes*
+
+.. sidebar:: Soft, or hard?
+
+    The time limit is set in two values, `soft` and `hard`.
+    The soft time limit allows the task to catch an exception
+    to clean up before it is killed: the hard timeout is not catchable
+    and force terminates the task.
 
 A single task can potentially run forever, if you have lots of tasks
 waiting for some event that will never happen you will block the worker
@@ -178,8 +187,10 @@ Changing time limits at runtime
 -------------------------------
 .. versionadded:: 2.3
 
-You can change the soft and hard time limits for a task by using the
-``time_limit`` remote control command.
+broker support: *amqp, redis, mongodb*
+
+There is a remote control command that enables you to change both soft
+and hard time limits for a task — named ``time_limit``.
 
 Example changing the time limit for the ``tasks.crawl_the_web`` task
 to have a soft time limit of one minute, and a hard time limit of
@@ -198,7 +209,7 @@ Max tasks per child setting
 
 .. versionadded:: 2.0
 
-:supported pools: processes
+pool support: *processes*
 
 With this option you can configure the maximum number of tasks
 a worker can execute before it's replaced by a new process.
@@ -209,14 +220,42 @@ for example from closed source C extensions.
 The option can be set using the workers `--maxtasksperchild` argument
 or using the :setting:`CELERYD_MAX_TASKS_PER_CHILD` setting.
 
-.. _worker-autoreload:
+.. _worker-autoscaling:
+
+Autoscaling
+===========
+
+.. versionadded:: 2.2
+pool support: *processes*, *gevent*
+
+The *autoscaler* component is used to dynamically resize the pool
+based on load:
+
+- The autoscaler adds more pool processes when there is work to do,
+    - and starts removing processes when the workload is low.
+
+It's enabled by the :option:`--autoscale` option, which needs two
+numbers: the maximum and minumum number of pool processes::
+
+        --autoscale=AUTOSCALE
+             Enable autoscaling by providing
+             max_concurrency,min_concurrency.  Example:
+               --autoscale=10,3 (always keep 3 processes, but grow to
+              10 if necessary).
+
+You can also define your own rules for the autoscaler by subclassing
+:class:`~celery.worker.autoscaler.Autoscaler`.
+Some ideas for metrics include load average or the amount of memory available.
+You can specify a custom autoscaler with the :setting:`CELERYD_AUTOSCALER` setting.
+
+.. _worker-autoreloading:
 
 Autoreloading
 =============
 
 .. versionadded:: 2.5
 
-:supported pools: processes, eventlet, gevent, threads, solo
+pool support: *processes, eventlet, gevent, threads, solo*
 
 Starting :program:`celery worker` with the :option:`--autoreload` option will
 enable the worker to watch for file system changes to all imported task
@@ -267,8 +306,8 @@ Remote control
 
 .. versionadded:: 2.0
 
-:supported pools: processes, eventlet, gevent, blocking:threads/solo (see note)
-:supported transports: amqp, redis, mongodb
+pool support: *processes, eventlet, gevent*, blocking:*threads/solo* (see note)
+broker support: *amqp, redis, mongodb*
 
 Workers have the ability to be remote controlled using a high-priority
 broadcast message queue.  The commands can be directed to all, or a specific
@@ -446,8 +485,10 @@ a worker using :program:`celery events`/:program:`celerymon`.
     >>> celery.control.broadcast("enable_events")
     >>> celery.control.broadcast("disable_events")
 
-Adding/Reloading modules
-------------------------
+.. _worker-autoreload:
+
+Autoreloading
+-------------
 
 .. versionadded:: 2.5