============================ Configuration and defaults ============================ This document describes the configuration options available. If you're using celery in a Django project these settings should be defined in your projects ``settings.py`` file. In a regular Python environment using the default loader you must create the ``celeryconfig.py`` module and make sure it is available on the Python path. Example configuration file ========================== This is an example configuration file to get you started, it should contain all you need to run a basic celery set-up. .. code-block:: python CELERY_BACKEND = "database" DATABASE_ENGINE = "sqlite3" DATABASE_NAME = "mydatabase.db" BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_VHOST = "/" BROKER_USER = "guest" BROKER_PASSWORD = "guest" ## If you're doing mostly I/O you can have more processes, ## but if mostly spending CPU, try to keep it close to the ## number of CPUs on your machine. If not set, the number of CPUs/cores ## available will be used. # CELERYD_CONCURRENCY = 8 # CELERYD_LOG_FILE = "celeryd.log" # CELERYD_LOG_LEVEL = "INFO" # CELERYD_PID_FILE = "celeryd.pid" Concurrency settings ==================== * CELERYD_CONCURRENCY The number of concurrent worker processes, executing tasks simultaneously. Defaults to the number of CPUs/cores available. Task result backend settings ============================ * CELERY_BACKEND The backend used to store task results (tombstones). Can be one of the following: * database (default) Use a relational database supported by the Django ORM. * cache Use `memcached`_ to store the results. * mongodb Use `MongoDB`_ to store the results. * pyredis Use `Redis`_ to store the results. * tyrant Use `Tokyo Tyrant`_ to store the results. * amqp Send results back as AMQP messages (**WARNING** While very fast, you must make sure you only try to receive the result once). .. _`memcached`: http://memcached.org .. _`MongoDB`: http://mongodb.org .. _`Redis`: http://code.google.com/p/redis/ .. _`Tokyo Tyrant`: http://1978th.net/tokyotyrant/ Database backend settings ========================= Please see the Django ORM database settings documentation: http://docs.djangoproject.com/en/dev/ref/settings/#database-engine If you use this backend make sure to initialize the database tables after configuration. When using celery with a Django project this means executing:: $ python manage.py syncdb When using celery in a regular Python environment you have to execute:: $ celeryinit Example configuration --------------------- .. code-block:: python CELERY_BACKEND = "database" DATABASE_ENGINE = "mysql" DATABASE_USER = "myusername" DATABASE_PASSWORD = "mypassword" DATABASE_NAME = "mydatabase" DATABASE_HOST = "localhost" AMQP backend settings ===================== The AMQP backend does not have any settings yet. Example configuration --------------------- CELERY_BACKEND = "amqp" Cache backend settings ====================== Please see the documentation for the Django cache framework settings: http://docs.djangoproject.com/en/dev/topics/cache/#memcached To use a custom cache backend for Celery, while using another for Django, you should use the ``CELERY_CACHE_BACKEND`` setting instead of the regular django ``CACHE_BACKEND`` setting. Example configuration --------------------- Using a single memcached server: .. code-block:: python CACHE_BACKEND = 'memcached://127.0.0.1:11211/' Using multiple memcached servers: .. code-block:: python CELERY_BACKEND = "cache" CACHE_BACKEND = 'memcached://172.19.26.240:11211;172.19.26.242:11211/' Tokyo Tyrant backend settings ============================= **NOTE** The Tokyo Tyrant backend requires the :mod:`pytyrant` library: http://pypi.python.org/pypi/pytyrant/ This backend requires the following configuration directives to be set: * TT_HOST Hostname of the Tokyo Tyrant server. * TT_PORT The port the Tokyo Tyrant server is listening to. Example configuration --------------------- .. code-block:: python CELERY_BACKEND = "tyrant" TT_HOST = "localhost" TT_PORT = 1978 Redis backend settings ====================== **NOTE** The Redis backend requires the :mod:`redis` library: http://pypi.python.org/pypi/redis/0.5.5 To install the redis package use ``pip`` or ``easy_install``:: $ pip install redis This backend requires the following configuration directives to be set: * REDIS_HOST Hostname of the Redis database server. e.g. ``"localhost"``. * REDIS_PORT Port to the Redis database server. e.g. ``6379``. Also, the following optional configuration directives are available: * REDIS_DB Name of the database to use. Default is ``celery_results``. * REDIS_TIMEOUT Timeout in seconds before we give up establishing a connection to the Redis server. * REDIS_CONNECT_RETRY Retry connecting if an connection could not be established. Default is false. Example configuration --------------------- .. code-block:: python CELERY_BACKEND = "pyredis" REDIS_HOST = "localhost" REDIS_PORT = 6739 REDIS_DATABASE = "celery_results" REDIS_CONNECT_RETRY=True MongoDB backend settings ======================== **NOTE** The MongoDB backend requires the :mod:`pymongo` library: http://github.com/mongodb/mongo-python-driver/tree/master * CELERY_MONGODB_BACKEND_SETTINGS This is a dict supporting the following keys: * host Hostname of the MongoDB server. Defaults to "localhost". * port The port the MongoDB server is listening to. Defaults to 27017. * user Username to authenticate to the MongoDB server as (optional). * password Password to authenticate to the MongoDB server (optional). * database The database name to connect to. Defaults to "celery". * taskmeta_collection The collection name to store task metadata. Defaults to "celery_taskmeta". Example configuration --------------------- .. code-block:: python CELERY_BACKEND = "mongodb" CELERY_MONGODB_BACKEND_SETTINGS = { "host": "192.168.1.100", "port": 30000, "database": "mydb", "taskmeta_collection": "my_taskmeta_collection", } Messaging settings ================== Routing ------- * CELERY_QUEUES The mapping of queues the worker consumes from. This is a dictionary of queue name/options. See :doc:`userguide/routing` for more information. The default is a queue/exchange/binding key of ``"celery"``, with exchange type ``direct``. You don't have to care about this unless you want custom routing facilities. * CELERY_DEFAULT_QUEUE The queue used by default, if no custom queue is specified. This queue must be listed in ``CELERY_QUEUES``. The default is: ``celery``. * CELERY_DEFAULT_EXCHANGE Name of the default exchange to use when no custom exchange is specified. The default is: ``celery``. * CELERY_DEFAULT_EXCHANGE_TYPE Default exchange type used when no custom exchange is specified. The default is: ``direct``. * CELERY_DEFAULT_ROUTING_KEY The default routing key used when sending tasks. The default is: ``celery``. Connection ---------- * CELERY_BROKER_CONNECTION_TIMEOUT The timeout in seconds before we give up establishing a connection to the AMQP server. Default is 4 seconds. * CELERY_BROKER_CONNECTION_RETRY Automatically try to re-establish the connection to the AMQP broker if it's lost. The time between retries is increased for each retry, and is not exhausted before ``CELERY_BROKER_CONNECTION_MAX_RETRIES`` is exceeded. This behaviour is on by default. * CELERY_BROKER_CONNECTION_MAX_RETRIES Maximum number of retries before we give up re-establishing a connection to the AMQP broker. If this is set to ``0`` or ``None``, we will retry forever. Default is 100 retries. Task execution settings ======================= * CELERY_ALWAYS_EAGER If this is ``True``, all tasks will be executed locally by blocking until it is finished. ``apply_async`` and ``Task.delay`` will return a :class:`celery.result.EagerResult` which emulates the behaviour of :class:`celery.result.AsyncResult`, except the result has already been evaluated. Tasks will never be sent to the queue, but executed locally instead. * CELERY_IGNORE_RESULT Wheter to store the task return values or not (tombstones). If you still want to store errors, just not successful return values, you can set ``CELERY_STORE_ERRORS_EVEN_IF_IGNORED``. * CELERY_TASK_RESULT_EXPIRES Time (in seconds, or a :class:`datetime.timedelta` object) for when after stored task tombstones are deleted. **NOTE**: For the moment this only works for the database and MongoDB backends., except the result has already been evaluated. * CELERY_TASK_SERIALIZER A string identifying the default serialization method to use. Can be ``pickle`` (default), ``json``, ``yaml``, or any custom serialization methods that have been registered with :mod:`carrot.serialization.registry`. Default is ``pickle``. Worker: celeryd =============== * CELERY_IMPORTS A sequence of modules to import when the celery daemon starts. This is useful to add tasks if you are not using django or cannot use task autodiscovery. * CELERY_SEND_EVENTS Send events so the worker can be monitored by tools like ``celerymon``. * CELERY_SEND_TASK_ERROR_EMAILS If set to ``True``, errors in tasks will be sent to admins by e-mail. If unset, it will send the e-mails if ``settings.DEBUG`` is False. * CELERY_STORE_ERRORS_EVEN_IF_IGNORED If set, the worker stores all task errors in the result store even if ``Task.ignore_result`` is on. Logging ------- * CELERYD_LOG_FILE The default filename the worker daemon logs messages to, can be overridden using the `--logfile`` option to ``celeryd``. The default is to log using ``stderr`` if running in the foreground, when running in the background, detached as a daemon, the default logfile is ``celeryd.log``. Can also be set via the ``--logfile`` argument. * CELERYD_LOG_LEVEL Worker log level, can be any of ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``, ``CRITICAL``. Can also be set via the ``--loglevel`` argument. See the :mod:`logging` module for more information. * CELERYD_LOG_FORMAT The format to use for log messages. Can be overridden using the ``--loglevel`` option to ``celeryd``. Default is ``[%(asctime)s: %(levelname)s/%(processName)s] %(message)s`` See the Python :mod:`logging` module for more information about log formats. Process ------- * CELERYD_PID_FILE Full path to the pid file. Default is ``celeryd.pid``. Can also be set via the ``--pidfile`` argument. Periodic Task Server: celerybeat ================================ * CELERYBEAT_SCHEDULE_FILENAME Name of the file celerybeat stores the current schedule in. Can be a relative or absolute path, but be aware that the suffix ``.db`` will be appended to the filename. Can also be set via the ``--schedule`` argument. * CELERYBEAT_MAX_LOOP_INTERVAL The maximum number of seconds celerybeat can sleep between checking the schedule. Default is 300 seconds (5 minutes). * CELERYBEAT_LOG_FILE The default filename to log messages to, can be overridden using the `--logfile`` option. The default is to log using ``stderr`` if running in the foreground, when running in the background, detached as a daemon, the default logfile is ``celerybeat.log``. Can also be set via the ``--logfile`` argument. * CELERYBEAT_LOG_LEVEL Logging level. Can be any of ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``, or ``CRITICAL``. Can also be set via the ``--loglevel`` argument. See the :mod:`logging` module for more information. * CELERYBEAT_PID_FILE Full path to celerybeat's pid file. Default is ``celerybat.pid``. Can also be set via the ``--pidfile`` argument. Monitor Server: celerymon ========================= * CELERYMON_LOG_FILE The default filename to log messages to, can be overridden using the `--logfile`` option. The default is to log using ``stderr`` if running in the foreground, when running in the background, detached as a daemon, the default logfile is ``celerymon.log``. * CELERYMON_LOG_LEVEL Logging level. Can be any of ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``, or ``CRITICAL``. See the :mod:`logging` module for more information. * CELERYMON_PID_FILE Full path to celerymon's pid file. Default is ``celerymon.pid``. Can be overridden using the ``--pidfile`` option to ``celerymon``.