celery/docs/getting-started/intro.rst

=============================
 Introduction to Task Queues
=============================

.. contents::
    :local:
    :depth: 1

What are Task Queues?
=====================

.. compound::

    "The quick brown fox jumps over the lazy dog"

        said the Farmer.

            but little did *he* know...


What do I need?
===============

.. sidebar:: Version Requirements
    :subtitle: Celery version 2.6 runs on

    - Python ❨2.5, 2.6, 2.7, 3.2, 3.3❩
    - PyPy ❨1.8, 1.9❩
    - Jython ❨2.5, 2.7❩.

    This is the last version to support Python 2.5,
    and from the next version Python 2.6 or newer is required.
    The last version to support Python 2.4 was Celery series 2.2.

*Celery* requires a message broker to send and receive messages,
but this term has been stretched to include everything from
financial-grade messaging systems to your fridge.

You can run *Celery* on a single or multiple machines, or even
across datacenters.



Celery is…
==========

.. topic:: ”

    - **Simple**

        Bla bla bla., yaddi blabla, bla bli bla do re mi, bla bi do,
        re mi bla do blah blah yadda blah blah blah blah.

    - **Fast**

        Bla bla bla. librabbitmq, yaddi blabla lightweight, bla bli bla do re mi, bla bi do,
        re mi bla do blah blah yadda blah blah blah blah.

    - **Highly Available**

        Workers and clients will automatically retry in the event
        of connection loss or failure, and some brokers support
        HA in way of *Master/Master* or -- *Master/Slave* replication.

    - **Flexible**

        Almost every part of *Celery* can be extended or used on its own,
        Custom pool implementations, serializers, compression schemes, logging,
        schedulers, consumers, producers, autoscalers, broker transorts and much more.


.. topic:: It supports

    .. hlist::
        :columns: 2

        - **Brokers**

            - :ref:`RabbitMQ <broker-rabbitmq>`, :ref:`Redis <broker-redis>`,
            - :ref:`MongoDB <broker-mongodb>`, :ref:`Beanstalk <broker-beanstalk>`,
            - :ref:`CouchDB <broker-couchdb>`, :ref:`SQLAlchemy <broker-sqlalchemy>`,
            - :ref:`Django ORM <broker-django>`, and more…

        - **Concurrency**

            - multiprocessing,
            - Eventlet_, gevent_
            - threads
            - single

        - **Result Stores**

            - AMQP, Redis
            - memcached, MongoDB,
            - SQLAlchemy/Django ORM,
            - Apache Cassandra.

        - **Serialization & Compression**

            - *pickle*, *json*, *yaml*, *msgpack*.
            - *zlib*, *bzip2*, or uncompressed.
            - Cryptographic message signing.
            - Fine-grained serialization settings.



.. topic:: Features

    .. hlist::
        :columns: 2

        - **Monitoring**

            The stream of monitoring events emit by the worker are used
            by built-in and external tools to tell you what your cluster
            is doing in real-time.

            :ref:`Read more… <guide-monitoring>`.

        - **Time Limits & Rate Limits**

            You can control how many tasks can be executed per second/minute/hour,
            or how long a task can be allowed to run, and this can be set as
            a default, for a specific worker or individually for each task type.

            :ref:`Read more… <worker-time-limits>`.

        - **Autoreloading**

            While in development workers can be configured to automatically reload source
            code as it changes.

            :ref:`Read more… <worker-autoreloading>`.

        - **Autoscaling**

            Dynamically resizing the worker pool depending on load,
            or custom metrics specified by the user, used to limit
            memory usage in shared hosting/cloud environment or to
            enforce a given quality of service.

            :ref:`Read more… <worker-autoscaling>`.

        - **Resource Leak Protection**

            The :option:`--maxtasksperchild` option is used for user tasks
            leaking resources, like memory or file descriptors, that
            are out simply out of your control.

            :ref:`Read more… <worker-maxtasksperchild>`.

        - **User Components**

            Each worker component can be customized, and additional components
            can be defined by the user.  The worker is built up using "boot steps" — a
            dependency graph enabling fine grained control of the workers
            internals.

.. _`RabbitMQ`: http://www.rabbitmq.com/
.. _`Redis`: http://code.google.com/p/redis/
.. _`SQLAlchemy`: http://www.sqlalchemy.org/
.. _`Django ORM`: http://djangoproject.com/
.. _`Eventlet`: http://eventlet.net/
.. _`gevent`: http://gevent.org/
.. _`Beanstalk`: http://kr.github.com/beanstalkd/
.. _`MongoDB`: http://mongodb.org/
.. _`CouchDB`: http://couchdb.apache.org/
.. _`Amazon SQS`: http://aws.amazon.com/sqs/
.. _`Apache ZooKeeper`: http://zookeeper.apache.org/