Merge branch 'master' into consumerset

Conflicts:
	celery/task/base.py
	celery/worker.py

AUTHORS

@@ -1,4 +1,7 @@
-Ask Solem <askh@opera.com>
-Grégoire Cachet <gregoire@audacy.fr>
-Vitaly Babiy <vbabiy86@gmail.com>
-Brian Rosner <brosner@gmail.com>
+Ordered by date of first contribution:
+  Ask Solem <askh@opera.com>
+  Grégoire Cachet <gregoire@audacy.fr>
+  Vitaly Babiy <vbabiy86@gmail.com>
+  Brian Rosner <brosner@gmail.com>
+  Sean Creeley <sean.creeley@gmail.com>
+  Ben Firshman <ben@firshman.co.uk>

Changelog

@@ -2,124 +2,259 @@
 Change history
 ==============
 
-x.x.x [xxxx-xx-xx xx:xx P.M CET] 
------------------------------------------------
+0.6.0 [2009-08-07 06:54 A.M CET]
+--------------------------------
 
-	*	**IMPORTANT** The ``subtask_ids`` attribute on the ``TaskSetResult``
-		instance has been removed. To get this information instead use:
+**IMPORTANT CHANGES**
 
-			>>> subtask_ids = [subtask.task_id for subtask in ts_res.subtasks]
+* Fixed a bug where tasks raising unpickleable exceptions crashed pool
+	workers. So if you've had pool workers mysteriously dissapearing, or
+	problems with celeryd stopping working, this has been fixed in this
+	version.
 
-	*	Taskset.run() now respects extra message options from the task class.
+* Fixed a race condition with periodic tasks.
 
-	* Task: Add attribute ``ignore_result``: Don't store the status and
-		return value. This means you can't use the
-		:class:`celery.result.AsyncResult` to check if the task is
-		done, or get its return value. Only use if you need the performance
-		and is able live without these features. Any exceptions raised will
-		store the return value/status as usual.
+* The task pool is now supervised, so if a pool worker crashes,
+	goes away or stops responding, it is automatically replaced with
+	a new one.
 
-	* Task: Add attribute ``disable_error_emails`` to disable sending error
-		emails for that task.
+* Task.name is now automatically generated out of class module+name, e.g.
+	``"djangotwitter.tasks.UpdateStatusesTask"``. Very convenient. No idea why
+	we didn't do this before. Some documentation is updated to not manually
+	specify a task name.
 
-	* Should now work on Windows (although running in the background won't
-		work, so using the ``--detach`` argument results in an exception
-		being raised.)
+**NEWS**
 
-	* Added support for statistics for profiling and monitoring.
-	    To start sending statistics start ``celeryd`` with the
-	    ``--statistics`` option. Then after a while you can dump the results
-	    by running ``python manage.py celerystats``. See
-	    :mod:`celery.monitoring` for more information.
+* Tested with Django 1.1
+
+* New Tutorial: Creating a click counter using carrot and celery
+
+* Database entries for periodic tasks are now created at ``celeryd``
+	startup instead of for each check (which has been a forgotten TODO/XXX
+	in the code for a long time)
+
+* New settings variable: ``CELERY_TASK_RESULT_EXPIRES``
+	Time (in seconds, or a `datetime.timedelta` object) for when after
+	stored task results are deleted. For the moment this only works for the
+	database backend.
+
+* ``celeryd`` now emits a debug log message for which periodic tasks
+	has been launched.
+
+* The periodic task table is now locked for reading while getting
+	periodic task status. (MySQL only so far, seeking patches for other
+	engines)
+
+* A lot more debugging information is now available by turning on the
+	``DEBUG`` loglevel (``--loglevel=DEBUG``).
+
+* Functions/methods with a timeout argument now works correctly.
+
+* New: ``celery.strategy.even_time_distribution``: 
+	With an iterator yielding task args, kwargs tuples, evenly distribute
+	the processing of its tasks throughout the time window available.
+
+* Log message ``Unknown task ignored...`` now has loglevel ``ERROR``
+
+* Log message ``"Got task from broker"`` is now emitted for all tasks, even if
+	the task has an ETA (estimated time of arrival). Also the message now
+	includes the ETA for the task (if any).
+
+* Acknowledgement now happens in the pool callback. Can't do ack in the job
+	target, as it's not pickleable (can't share AMQP connection, etc)).
+
+* Added note about .delay hanging in README
+
+* Tests now passing in Django 1.1
+
+* Fixed discovery to make sure app is in INSTALLED_APPS
+
+* Previously overrided pool behaviour (process reap, wait until pool worker
+	available, etc.) is now handled by ``multiprocessing.Pool`` itself.
+
+* Convert statistics data to unicode for use as kwargs. Thanks Lucy!
+
+0.4.1 [2009-07-02 01:42 P.M CET]
+--------------------------------
+
+* Fixed a bug with parsing the message options (``mandatory``,
+  ``routing_key``, ``priority``, ``immediate``)
+
+0.4.0 [2009-07-01 07:29 P.M CET] 
+--------------------------------
+
+* Adds eager execution. ``celery.execute.apply``|``Task.apply`` executes the
+  function blocking until the task is done, for API compatiblity it
+  returns an ``celery.result.EagerResult`` instance. You can configure
+  celery to always run tasks locally by setting the
+  ``CELERY_ALWAYS_EAGER`` setting to ``True``.
+
+* Now depends on ``anyjson``.
+
+* 99% coverage using python ``coverage`` 3.0.
+
+0.3.20 [2009-06-25 08:42 P.M CET] 
+---------------------------------
+
+* New arguments to ``apply_async`` (the advanced version of
+  ``delay_task``), ``countdown`` and ``eta``;
+
+	>>> # Run 10 seconds into the future.
+	>>> res = apply_async(MyTask, countdown=10);
+
+	>>> # Run 1 day from now
+	>>> res = apply_async(MyTask, eta=datetime.now() + 
+	...									timedelta(days=1)
+
+* Now unlinks the pidfile if it's stale.
+
+* Lots of more tests.
+
+* Now compatible with carrot >= 0.5.0.
+
+* **IMPORTANT** The ``subtask_ids`` attribute on the ``TaskSetResult``
+  instance has been removed. To get this information instead use:
+
+		>>> subtask_ids = [subtask.task_id for subtask in ts_res.subtasks]
+
+*	``Taskset.run()`` now respects extra message options from the task class.
+
+* Task: Add attribute ``ignore_result``: Don't store the status and
+  return value. This means you can't use the
+  ``celery.result.AsyncResult`` to check if the task is
+  done, or get its return value. Only use if you need the performance
+  and is able live without these features. Any exceptions raised will
+  store the return value/status as usual.
+
+* Task: Add attribute ``disable_error_emails`` to disable sending error
+  emails for that task.
+
+* Should now work on Windows (although running in the background won't
+  work, so using the ``--detach`` argument results in an exception
+  being raised.)
+
+* Added support for statistics for profiling and monitoring.
+  To start sending statistics start ``celeryd`` with the
+  ``--statistics`` option. Then after a while you can dump the results
+  by running ``python manage.py celerystats``. See
+  ``celery.monitoring`` for more information.
+
+* The celery daemon can now be supervised (i.e it is automatically
+  restarted if it crashes). To use this start celeryd with the
+  ``--supervised`` option (or alternatively ``-S``).
+
+* views.apply: View applying a task. Example::
+
+	http://e.com/celery/apply/task_name/arg1/arg2//?kwarg1=a&kwarg2=b
+
+  **NOTE** Use with caution, preferably not make this publicly
+  accessible without ensuring your code is safe!
+
+* Refactored ``celery.task``. It's now split into three modules:
+
+	* celery.task
+
+		Contains ``apply_async``, ``delay_task``, ``discard_all``, and task
+		shortcuts, plus imports objects from ``celery.task.base`` and
+		``celery.task.builtins``
+
+	* celery.task.base
+
+		Contains task base classes: ``Task``, ``PeriodicTask``,
+		``TaskSet``, ``AsynchronousMapTask``, ``ExecuteRemoteTask``.
+
+	* celery.task.builtins
+
+		Built-in tasks: ``PingTask``, ``DeleteExpiredTaskMetaTask``.
 
 
 0.3.7 [2008-06-16 11:41 P.M CET] 
------------------------------------------------
+--------------------------------
 
-	* **IMPORTANT** Now uses AMQP's ``basic.consume`` instead of
-		``basic.get``. This means we're no longer polling the broker for
-		new messages.
+* **IMPORTANT** Now uses AMQP's ``basic.consume`` instead of
+  ``basic.get``. This means we're no longer polling the broker for
+  new messages.
 
-	* **IMPORTANT** Default concurrency limit is now set to the number of CPUs
-		available on the system.
+* **IMPORTANT** Default concurrency limit is now set to the number of CPUs
+  available on the system.
 
-	* **IMPORTANT** ``tasks.register``: Renamed ``task_name`` argument to
-		``name``, so
+* **IMPORTANT** ``tasks.register``: Renamed ``task_name`` argument to
+  ``name``, so
 
-			>>> tasks.register(func, task_name="mytask")
+		>>> tasks.register(func, task_name="mytask")
 
-		has to be replaced with:
+  has to be replaced with:
 
-			>>> tasks.register(func, name="mytask")
+		>>> tasks.register(func, name="mytask")
 
-	* The daemon now correctly runs if the pidlock is stale.
+* The daemon now correctly runs if the pidlock is stale.
 
-	* Now compatible with carrot 0.4.5
+* Now compatible with carrot 0.4.5
 
-	* Default AMQP connnection timeout is now 4 seconds.
-	* ``AsyncResult.read()`` was always returning ``True``.
+* Default AMQP connnection timeout is now 4 seconds.
+* ``AsyncResult.read()`` was always returning ``True``.
 
-	*  Only use README as long_description if the file exists so easy_install
-		doesn't break.
+*  Only use README as long_description if the file exists so easy_install
+   doesn't break.
 
-	* ``celery.view``: JSON responses now properly set its mime-type. 
+* ``celery.view``: JSON responses now properly set its mime-type. 
 
-	* ``apply_async`` now has a ``connection`` keyword argument so you
-		can re-use the same AMQP connection if you want to execute
-		more than one task.
+* ``apply_async`` now has a ``connection`` keyword argument so you
+  can re-use the same AMQP connection if you want to execute
+  more than one task.
 
-	* Handle failures in task_status view such that it won't throw 500s.
+* Handle failures in task_status view such that it won't throw 500s.
 
-	* Fixed typo ``AMQP_SERVER`` in documentation to ``AMQP_HOST``.
+* Fixed typo ``AMQP_SERVER`` in documentation to ``AMQP_HOST``.
 
-	* Worker exception e-mails sent to admins now works properly.
+* Worker exception e-mails sent to admins now works properly.
 
-	* No longer depends on ``django``, so installing ``celery`` won't affect
-		the preferred Django version installed.
+* No longer depends on ``django``, so installing ``celery`` won't affect
+  the preferred Django version installed.
 
-	* Now works with PostgreSQL (psycopg2) again by registering the
-		``PickledObject`` field.
+* Now works with PostgreSQL (psycopg2) again by registering the
+  ``PickledObject`` field.
 
-	* ``celeryd``: Added ``--detach`` option as an alias to ``--daemon``, and
-		it's the term used in the documentation from now on.
+* ``celeryd``: Added ``--detach`` option as an alias to ``--daemon``, and
+  it's the term used in the documentation from now on.
 
-	* Make sure the pool and periodic task worker thread is terminated
-		properly at exit. (So ``Ctrl-C`` works again).
+* Make sure the pool and periodic task worker thread is terminated
+  properly at exit. (So ``Ctrl-C`` works again).
 
-	* Now depends on ``python-daemon``.
+* Now depends on ``python-daemon``.
 
-	* Removed dependency to ``simplejson``
+* Removed dependency to ``simplejson``
 
-	* Cache Backend: Re-establishes connection for every task process
-		if the Django cache backend is memcached/libmemcached.
+* Cache Backend: Re-establishes connection for every task process
+  if the Django cache backend is memcached/libmemcached.
 
-	* Tyrant Backend: Now re-establishes the connection for every task
-		executed.
+* Tyrant Backend: Now re-establishes the connection for every task
+  executed.
 
 0.3.3 [2009-06-08 01:07 P.M CET] 
------------------------------------------------
+--------------------------------
 
 	* The ``PeriodicWorkController`` now sleeps for 1 second between checking
 		for periodic tasks to execute.
 
 0.3.2 [2009-06-08 01:07 P.M CET]
------------------------------------------------
+--------------------------------
 
-	* celeryd: Added option ``--discard``: Discard (delete!) all waiting
-		messages in the queue.
+* celeryd: Added option ``--discard``: Discard (delete!) all waiting
+  messages in the queue.
 
-	* celeryd: The ``--wakeup-after`` option was not handled as a float.
+* celeryd: The ``--wakeup-after`` option was not handled as a float.
 
 0.3.1 [2009-06-08 01:07 P.M CET]
------------------------------------------------
+--------------------------------
 
-	* The `PeriodicTask`` worker is now running in its own thread instead
-		of blocking the ``TaskController`` loop.
+* The `PeriodicTask`` worker is now running in its own thread instead
+  of blocking the ``TaskController`` loop.
 
-	* Default ``QUEUE_WAKEUP_AFTER`` has been lowered to ``0.1`` (was ``0.3``)
+* Default ``QUEUE_WAKEUP_AFTER`` has been lowered to ``0.1`` (was ``0.3``)
 
 0.3.0 [2009-06-08 12:41 P.M CET]
------------------------------------------------
+--------------------------------
 
 **NOTE** This is a development version, for the stable release, please
 see versions 0.2.x.
@@ -127,257 +262,258 @@ see versions 0.2.x.
 **VERY IMPORTANT:** Pickle is now the encoder used for serializing task
 arguments, so be sure to flush your task queue before you upgrade.
 
-	* **IMPORTANT** TaskSet.run() now returns a celery.result.TaskSetResult
-		instance, which lets you inspect the status and return values of a
-		taskset as it was a single entity.
+* **IMPORTANT** TaskSet.run() now returns a celery.result.TaskSetResult
+  instance, which lets you inspect the status and return values of a
+  taskset as it was a single entity.
 
-	* **IMPORTANT** Celery now depends on carrot >= 0.4.1.
+* **IMPORTANT** Celery now depends on carrot >= 0.4.1.
 
-	* The celery daemon now sends task errors to the registered admin e-mails.
-		To turn off this feature, set ``SEND_CELERY_TASK_ERROR_EMAILS`` to
-		``False`` in your ``settings.py``. Thanks to Grégoire Cachet.
+* The celery daemon now sends task errors to the registered admin e-mails.
+  To turn off this feature, set ``SEND_CELERY_TASK_ERROR_EMAILS`` to
+  ``False`` in your ``settings.py``. Thanks to Grégoire Cachet.
 
-	* You can now run the celery daemon by using ``manage.py``::
-		
-			$ python manage.py celeryd
+* You can now run the celery daemon by using ``manage.py``::
 
-		Thanks to Grégoire Cachet.
+		$ python manage.py celeryd
 
-	* Added support for message priorities, topic exchanges, custom routing
-		keys for tasks. This means we have introduced
-		``celery.task.apply_async``, a new way of executing tasks.
+  Thanks to Grégoire Cachet.
 
-		You can use ``celery.task.delay`` and ``celery.Task.delay`` like usual, but
-		if you want greater control over the message sent, you want
-		``celery.task.apply_async`` and ``celery.Task.apply_async``.
+* Added support for message priorities, topic exchanges, custom routing
+  keys for tasks. This means we have introduced
+  ``celery.task.apply_async``, a new way of executing tasks.
 
-		This also means the AMQP configuration has changed. Some settings has
-		been renamed, while others are new::
+  You can use ``celery.task.delay`` and ``celery.Task.delay`` like usual, but
+  if you want greater control over the message sent, you want
+  ``celery.task.apply_async`` and ``celery.Task.apply_async``.
 
-			CELERY_AMQP_EXCHANGE
-			CELERY_AMQP_PUBLISHER_ROUTING_KEY
-			CELERY_AMQP_CONSUMER_ROUTING_KEY
-			CELERY_AMQP_CONSUMER_QUEUE
-			CELERY_AMQP_EXCHANGE_TYPE
+  This also means the AMQP configuration has changed. Some settings has
+  been renamed, while others are new::
 
-		See the entry `Can I send some tasks to only some servers?`_ in the
-		`FAQ`_ for more information.
+		CELERY_AMQP_EXCHANGE
+		CELERY_AMQP_PUBLISHER_ROUTING_KEY
+		CELERY_AMQP_CONSUMER_ROUTING_KEY
+		CELERY_AMQP_CONSUMER_QUEUE
+		CELERY_AMQP_EXCHANGE_TYPE
+
+  See the entry `Can I send some tasks to only some servers?`_ in the
+  `FAQ`_ for more information.
 
 .. _`Can I send some tasks to only some servers?`:
 		http://bit.ly/celery_AMQP_routing
 .. _`FAQ`: http://ask.github.com/celery/faq.html
 
-	* Task errors are now logged using loglevel ``ERROR`` instead of ``INFO``,
-		and backtraces are dumped. Thanks to Grégoire Cachet.
+* Task errors are now logged using loglevel ``ERROR`` instead of ``INFO``,
+  and backtraces are dumped. Thanks to Grégoire Cachet.
 
-	* Make every new worker process re-establish it's Django DB connection,
-		this solving the "MySQL connection died?" exceptions.
-		Thanks to Vitaly Babiy and Jirka Vejrazka.
+* Make every new worker process re-establish it's Django DB connection,
+  this solving the "MySQL connection died?" exceptions.
+  Thanks to Vitaly Babiy and Jirka Vejrazka.
 
-	* **IMOPORTANT** Now using pickle to encode task arguments. This means you
-		now can pass complex python objects to tasks as arguments.
+* **IMOPORTANT** Now using pickle to encode task arguments. This means you
+  now can pass complex python objects to tasks as arguments.
 
-	* Removed dependency on ``yadayada``.
+* Removed dependency to ``yadayada``.
 
-	* Added a FAQ, see ``docs/faq.rst``.
+* Added a FAQ, see ``docs/faq.rst``.
 
-	* Now converts any unicode keys in task ``kwargs`` to regular strings.
-		Thanks Vitaly Babiy.
+* Now converts any unicode keys in task ``kwargs`` to regular strings.
+  Thanks Vitaly Babiy.
 
-	* Renamed the ``TaskDaemon`` to ``WorkController``.
+* Renamed the ``TaskDaemon`` to ``WorkController``.
 
-	* ``celery.datastructures.TaskProcessQueue`` is now renamed to
-		``celery.pool.TaskPool``.
+* ``celery.datastructures.TaskProcessQueue`` is now renamed to
+  ``celery.pool.TaskPool``.
 
-	* The pool algorithm has been refactored for greater performance and
-		stability.
+* The pool algorithm has been refactored for greater performance and
+  stability.
 
 0.2.0 [2009-05-20 05:14 P.M CET]
-------------------------------------------------
+--------------------------------
 
-	* Final release of 0.2.0
+* Final release of 0.2.0
 
-	* Compatible with carrot version 0.4.0.
+* Compatible with carrot version 0.4.0.
 
-	* Fixes some syntax errors related to fetching results
-	  from the database backend.
+* Fixes some syntax errors related to fetching results
+  from the database backend.
 
 0.2.0-pre3 [2009-05-20 05:14 P.M CET]
-----------------------------------------------------
+-------------------------------------
 
-	 * *Internal release*. Improved handling of unpickled exceptions,
-	 	get_result() now tries to recreate something looking like the
-	 	original exception.
+* *Internal release*. Improved handling of unpickled exceptions,
+  ``get_result`` now tries to recreate something looking like the
+  original exception.
 
 0.2.0-pre2 [2009-05-20 01:56 P.M CET]
-----------------------------------------------------
+-------------------------------------
 
-	* Now handles unpickleable exceptions (like the dynimically generated
-	  subclasses of ``django.core.exception.MultipleObjectsReturned``).
+* Now handles unpickleable exceptions (like the dynimically generated
+  subclasses of ``django.core.exception.MultipleObjectsReturned``).
 
 0.2.0-pre1 [2009-05-20 12:33 P.M CET]
-----------------------------------------------------
+-------------------------------------
 
-	* It's getting quite stable, with a lot of new features, so bump
-	  version to 0.2. This is a pre-release.
+* It's getting quite stable, with a lot of new features, so bump
+  version to 0.2. This is a pre-release.
 
-	* ``celery.task.mark_as_read()`` and ``celery.task.mark_as_failure()`` has
-	  been removed. Use ``celery.backends.default_backend.mark_as_read()``, 
-	  and ``celery.backends.default_backend.mark_as_failure()`` instead.
+* ``celery.task.mark_as_read()`` and ``celery.task.mark_as_failure()`` has
+  been removed. Use ``celery.backends.default_backend.mark_as_read()``, 
+  and ``celery.backends.default_backend.mark_as_failure()`` instead.
 
 0.1.15 [2009-05-19 04:13 P.M CET]
-------------------------------------------------
+---------------------------------
 
-	* The celery daemon was leaking AMQP connections, this should be fixed,
-	  if you have any problems with too many files open (like ``emfile``
-	  errors in ``rabbit.log``, please contact us!
+* The celery daemon was leaking AMQP connections, this should be fixed,
+  if you have any problems with too many files open (like ``emfile``
+  errors in ``rabbit.log``, please contact us!
 
 0.1.14 [2009-05-19 01:08 P.M CET]
-------------------------------------------------
+---------------------------------
 
-	* Fixed a syntax error in the ``TaskSet`` class.  (No such variable
-	  ``TimeOutError``).
+* Fixed a syntax error in the ``TaskSet`` class.  (No such variable
+  ``TimeOutError``).
 
 0.1.13 [2009-05-19 12:36 P.M CET]
-------------------------------------------------
+---------------------------------
+
+* Forgot to add ``yadayada`` to install requirements.
 
-	* Forgot to add ``yadayada`` to install requirements.
+* Now deletes all expired task results, not just those marked as done.
 
-	* Now deletes all expired task results, not just those marked as done.
+* Able to load the Tokyo Tyrant backend class without django
+  configuration, can specify tyrant settings directly in the class
+  constructor.
 
-	* Able to load the Tokyo Tyrant backend class without django
-	  configuration, can specify tyrant settings directly in the class
-	  constructor.
+* Improved API documentation
 
-	* Improved API documentation
+* Now using the Sphinx documentation system, you can build
+  the html documentation by doing ::
 
-	* Now using the Sphinx documentation system, you can build
-	  the html documentation by doing ::
+		$ cd docs
+		$ make html
 
-			$ cd docs
-			$ make html
-	
-	  and the result will be in ``docs/.build/html``.
+  and the result will be in ``docs/.build/html``.
 
 0.1.12 [2009-05-18 04:38 P.M CET]
-------------------------------------------------
+---------------------------------
 
-    * ``delay_task()`` etc. now returns ``celery.task.AsyncResult`` object,
-      which lets you check the result and any failure that might have
-      happened.  It kind of works like the ``multiprocessing.AsyncResult``
-      class returned by ``multiprocessing.Pool.map_async``.
+* ``delay_task()`` etc. now returns ``celery.task.AsyncResult`` object,
+  which lets you check the result and any failure that might have
+  happened.  It kind of works like the ``multiprocessing.AsyncResult``
+  class returned by ``multiprocessing.Pool.map_async``.
 
-    * Added dmap() and dmap_async(). This works like the 
-      ``multiprocessing.Pool`` versions except they are tasks
-      distributed to the celery server. Example:
+* Added dmap() and dmap_async(). This works like the 
+  ``multiprocessing.Pool`` versions except they are tasks
+  distributed to the celery server. Example:
 
-        >>> from celery.task import dmap
-        >>> import operator
-        >>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]])
-        >>> [4, 8, 16]
+		>>> from celery.task import dmap
+		>>> import operator
+		>>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]])
+		>>> [4, 8, 16]
         
-        >>> from celery.task import dmap_async
-        >>> import operator
-        >>> result = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]])
-        >>> result.ready()
-        False
-        >>> time.sleep(1)
-        >>> result.ready()
-        True
-        >>> result.result
-        [4, 8, 16]
-
-    * Refactored the task metadata cache and database backends, and added a new backend for Tokyo Tyrant. You can set the backend in your django settings file. e.g
-
-        CELERY_BACKEND = "database"; # Uses the database
-
-        CELERY_BACKEND = "cache"; # Uses the django cache framework
-
-        CELERY_BACKEND = "tyrant"; # Uses Tokyo Tyrant
-        TT_HOST = "localhost"; # Hostname for the Tokyo Tyrant server.
-        TT_PORT = 6657; # Port of the Tokyo Tyrant server.
+		>>> from celery.task import dmap_async
+		>>> import operator
+		>>> result = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]])
+		>>> result.ready()
+		False
+		>>> time.sleep(1)
+		>>> result.ready()
+		True
+		>>> result.result
+		[4, 8, 16]
+
+* Refactored the task metadata cache and database backends, and added
+  a new backend for Tokyo Tyrant. You can set the backend in your django
+  settings file. e.g::
+
+		CELERY_BACKEND = "database"; # Uses the database
+		CELERY_BACKEND = "cache"; # Uses the django cache framework
+		CELERY_BACKEND = "tyrant"; # Uses Tokyo Tyrant
+		TT_HOST = "localhost"; # Hostname for the Tokyo Tyrant server.
+		TT_PORT = 6657; # Port of the Tokyo Tyrant server.
 
 0.1.11 [2009-05-12 02:08 P.M CET]
--------------------------------------------------
+---------------------------------
 
-	* The logging system was leaking file descriptors, resulting in
-	  servers stopping with the EMFILES (too many open files) error. (fixed)
+* The logging system was leaking file descriptors, resulting in
+  servers stopping with the EMFILES (too many open files) error. (fixed)
 
 0.1.10 [2009-05-11 12:46 P.M CET]
--------------------------------------------------
+---------------------------------
 
-	* Tasks now supports both positional arguments and keyword arguments.
+* Tasks now supports both positional arguments and keyword arguments.
 
-	* Requires carrot 0.3.8.
+* Requires carrot 0.3.8.
 
-	* The daemon now tries to reconnect if the connection is lost.
+* The daemon now tries to reconnect if the connection is lost.
 
 0.1.8 [2009-05-07 12:27 P.M CET]
-------------------------------------------------
+--------------------------------
 
-	* Better test coverage
-	* More documentation
-	* celeryd doesn't emit ``Queue is empty`` message if
-	  ``settings.CELERYD_EMPTY_MSG_EMIT_EVERY`` is 0.
+* Better test coverage
+* More documentation
+* celeryd doesn't emit ``Queue is empty`` message if
+  ``settings.CELERYD_EMPTY_MSG_EMIT_EVERY`` is 0.
 
 0.1.7 [2009-04-30 1:50 P.M CET]
------------------------------------------------
+-------------------------------
 
-	* Added some unittests
+* Added some unittests
 
-	* Can now use the database for task metadata (like if the task has
-	  been executed or not). Set ``settings.CELERY_TASK_META``
+* Can now use the database for task metadata (like if the task has
+  been executed or not). Set ``settings.CELERY_TASK_META``
 
-	* Can now run ``python setup.py test`` to run the unittests from
-	  within the ``testproj`` project.
+* Can now run ``python setup.py test`` to run the unittests from
+  within the ``testproj`` project.
 
-	* Can set the AMQP exchange/routing key/queue using
-	  ``settings.CELERY_AMQP_EXCHANGE``, ``settings.CELERY_AMQP_ROUTING_KEY``,
-	  and ``settings.CELERY_AMQP_CONSUMER_QUEUE``.
+* Can set the AMQP exchange/routing key/queue using
+  ``settings.CELERY_AMQP_EXCHANGE``, ``settings.CELERY_AMQP_ROUTING_KEY``,
+  and ``settings.CELERY_AMQP_CONSUMER_QUEUE``.
 
 0.1.6 [2009-04-28 2:13 P.M CET]
------------------------------------------------
+-------------------------------
 
-	* Introducing ``TaskSet``. A set of subtasks is executed and you can
-	  find out how many, or if all them, are done (excellent for progress bars and such)
+* Introducing ``TaskSet``. A set of subtasks is executed and you can
+  find out how many, or if all them, are done (excellent for progress
+  bars and such)
 
-	* Now catches all exceptions when running ``Task.__call__``, so the
-	  daemon doesn't die. This does't happen for pure functions yet, only
-	  ``Task`` classes.
+* Now catches all exceptions when running ``Task.__call__``, so the
+  daemon doesn't die. This does't happen for pure functions yet, only
+  ``Task`` classes.
 
-	* ``autodiscover()`` now works with zipped eggs.
+* ``autodiscover()`` now works with zipped eggs.
 
-	* celeryd: Now adds curernt working directory to ``sys.path`` for
-	  convenience.
+* celeryd: Now adds curernt working directory to ``sys.path`` for
+  convenience.
 
-	* The ``run_every`` attribute of ``PeriodicTask`` classes can now be a
-	  ``datetime.timedelta()`` object.
+* The ``run_every`` attribute of ``PeriodicTask`` classes can now be a
+  ``datetime.timedelta()`` object.
 
-	* celeryd: You can now set the ``DJANGO_PROJECT_DIR`` variable
-	  for ``celeryd`` and it will add that to ``sys.path`` for easy launching.
+* celeryd: You can now set the ``DJANGO_PROJECT_DIR`` variable
+  for ``celeryd`` and it will add that to ``sys.path`` for easy launching.
 
-	* Can now check if a task has been executed or not via HTTP.
+* Can now check if a task has been executed or not via HTTP.
 
-	You can do this by including the celery ``urls.py`` into your project,
+* You can do this by including the celery ``urls.py`` into your project,
 
 		>>> url(r'^celery/$', include("celery.urls"))
 
-	then visiting the following url,::
+  then visiting the following url,::
 
 		http://mysite/celery/$task_id/done/
 
-	this will return a JSON dictionary like e.g:
+  this will return a JSON dictionary like e.g:
 
 		>>> {"task": {"id": $task_id, "executed": true}}
 
-	* ``delay_task`` now returns string id, not ``uuid.UUID`` instance.
+* ``delay_task`` now returns string id, not ``uuid.UUID`` instance.
 
-	* Now has ``PeriodicTasks``, to have ``cron`` like functionality.
+* Now has ``PeriodicTasks``, to have ``cron`` like functionality.
 
-	* Project changed name from ``crunchy`` to ``celery``. The details of
-	  the name change request is in ``docs/name_change_request.txt``.
+* Project changed name from ``crunchy`` to ``celery``. The details of
+  the name change request is in ``docs/name_change_request.txt``.
 
 0.1.0 [2009-04-24 11:28 A.M CET]
-------------------------------------------------
+--------------------------------
 
-	* Initial release
+* Initial release

FAQ

@@ -26,6 +26,21 @@ celeryd is not doing anything, just hanging
 --------------------------------------------
 
 **Answer:** See `MySQL is throwing deadlock errors, what can I do?`_.
+            or `Why is Task.delay/apply\* just hanging?`.
+
+Why is Task.delay/apply\* just hanging?`
+----------------------------------------
+
+**Answer:** :mod:`amqplib` hangs if it isn't able to authenticate to the
+AMQP server, so make sure you are able to access the configured vhost using
+the user and password.
+
+Why won't celeryd run on FreeBSD?
+---------------------------------
+
+**Answer:** multiprocessing.Pool requires a working POSIX semaphore
+implementation which isn't enabled in FreeBSD by default. You have to enable
+POSIX semaphores in the kernel and manually recompile multiprocessing.
 
 I'm having ``IntegrityError: Duplicate Key`` errors. Why?
 ----------------------------------------------------------
@@ -52,15 +67,43 @@ if it's able to find the task, or if some other error is happening.
 Why won't my Periodic Task run?
 -------------------------------
 
-See `Why won't my Task run?`_.
+**Answer:** See `Why won't my Task run?`_.
+
+How do I discard all waiting tasks?
+------------------------------------
+
+**Answer:** Use ``celery.task.discard_all()``, like this:
+
+    >>> from celery.task import discard_all
+    >>> discard_all()
+    1753
+
+The number ``1753`` is the number of messages deleted.
+
+You can also start celeryd with the ``--discard`` argument which will
+accomplish the same thing.
+
+I've discarded messages, but there are still messages left in the queue?
+------------------------------------------------------------------------
+
+**Answer:** Tasks are acknowledged (removed from the queue) as soon
+as they are actually executed. After the worker has received a task, it will
+take some time until it is actually executed, especially if there are a lot
+of tasks already waiting for execution. Messages that are not acknowledged are
+hold on to by the worker until it closes the connection to the broker (AMQP
+server). When that connection is closed (e.g because the worker was stopped)
+the tasks will be re-sent by the broker to the next available worker (or the
+same worker when it has been restarted), so to properly purge the queue of
+waiting tasks you have to stop all the workers, and then discard the tasks
+using ``discard_all``.
 
 Can I send some tasks to only some servers?
 --------------------------------------------
 
-As of now there is only one use-case that works like this, and that is
-tasks of type ``A`` can be sent to servers ``x`` and ``y``, while tasks
-of type ``B`` can be sent to server ``z``. One server can't handle more than
-one routing_key, but this is coming in a later release.
+**Answer:** As of now there is only one use-case that works like this,
+and that is tasks of type ``A`` can be sent to servers ``x`` and ``y``,
+while tasks of type ``B`` can be sent to server ``z``. One server can't
+handle more than one routing_key, but this is coming in a later release.
 
 Say you have two servers, ``x``, and ``y`` that handles regular tasks,
 and one server ``z``, that only handles feed related tasks, you can use this
@@ -123,3 +166,4 @@ You can also override this using the ``routing_key`` argument to
     >>> from myapp.tasks import RefreshFeedTask
     >>> apply_async(RefreshFeedTask, args=["http://cnn.com/rss"],
     ...             routing_key="feed.importer")
+

MANIFEST.in

@@ -1,6 +1,7 @@
 include AUTHORS
 include Changelog
 include README
+include README.rst
 include MANIFEST.in
 include LICENSE
 include TODO
@@ -8,6 +9,7 @@ include THANKS
 recursive-include celery *.py
 recursive-include docs *
 recursive-include testproj *
+recursive-include contrib *
 prune testproj/*.pyc
 prune docs/*.pyc
 prune contrib/*.pyc

Makefile

@@ -0,0 +1,39 @@
+PEP8=pep8
+
+pep8:
+	(find . -name "*.py" | xargs pep8 | perl -nle'\
+		print; $$a=1 if $$_}{exit($$a)')
+
+cycomplex:
+	find celery -type f -name "*.py" | xargs pygenie.py complexity
+
+ghdocs:
+	contrib/doc2ghpages
+
+autodoc:
+	contrib/doc4allmods celery
+
+bump:
+	contrib/bump -c celery
+
+coverage2:
+	[ -d testproj/temp ] || mkdir -p testproj/temp
+	(cd testproj; python manage.py test --figleaf)
+
+coverage:
+	[ -d testproj/temp ] || mkdir -p testproj/temp
+	(cd testproj; python manage.py test --coverage)
+
+test:
+	(cd testproj; python manage.py test)
+
+testverbose:
+	(cd testproj; python manage.py test --verbosity=2)
+
+releaseok: pep8 autodoc test
+
+removepyc:
+	find . -name "*.pyc" | xargs rm
+
+release: releaseok ghdocs removepyc
+

README

@@ -1 +0,0 @@
-README.rst

README

@@ -0,0 +1,369 @@
+=============================================
+ celery - Distributed Task Queue for Django.
+=============================================
+
+:Version: 0.6.0
+
+Introduction
+============
+
+``celery`` is a distributed task queue framework for Django.
+
+It is used for executing tasks *asynchronously*, routed to one or more
+worker servers, running concurrently using multiprocessing.
+
+It is designed to solve certain problems related to running websites
+demanding high-availability and performance.
+
+It is perfect for filling caches, posting updates to twitter, mass
+downloading data like syndication feeds or web scraping. Use-cases are
+plentiful. Implementing these features asynchronously using ``celery`` is
+easy and fun, and the performance improvements can make it more than
+worthwhile.
+
+Overview
+========
+
+This is a high level overview of the architecture.
+
+.. image:: http://cloud.github.com/downloads/ask/celery/Celery-Overview-v4.jpg
+
+The broker is an AMQP server pushing tasks to the worker servers.
+A worker server is a networked machine running ``celeryd``. This can be one or
+more machines, depending on the workload. See `A look inside the worker`_ to
+see how the worker server works.
+
+The result of the task can be stored for later retrieval (called its
+"tombstone").
+
+Features
+========
+
+    * Uses AMQP messaging (RabbitMQ, ZeroMQ) to route tasks to the
+      worker servers.
+
+    * You can run as many worker servers as you want, and still
+      be *guaranteed that the task is only executed once.*
+
+    * Tasks are executed *concurrently* using the Python 2.6
+      ``multiprocessing`` module (also available as a back-port
+      to older python versions)
+
+    * Supports *periodic tasks*, which makes it a (better) replacement
+      for cronjobs.
+
+    * When a task has been executed, the return value can be stored using
+      either a MySQL/Oracle/PostgreSQL/SQLite database, Memcached,
+      or Tokyo Tyrant back-end.
+
+    * If the task raises an exception, the exception instance is stored,
+      instead of the return value.
+
+    * All tasks has a Universally Unique Identifier (UUID), which is the
+      task id, used for querying task status and return values.
+
+    * Supports *task-sets*, which is a task consisting of several sub-tasks.
+      You can find out how many, or if all of the sub-tasks has been executed.
+      Excellent for progress-bar like functionality.
+
+    * Has a ``map`` like function that uses tasks, called ``dmap``.
+
+    * However, you rarely want to wait for these results in a web-environment.
+      You'd rather want to use Ajax to poll the task status, which is
+      available from a URL like ``celery/<task_id>/status/``. This view
+      returns a JSON-serialized data structure containing the task status,
+      and the return value if completed, or exception on failure.
+
+    * The worker can collect statistics, like, how many tasks has been
+      executed by type, and the time it took to process them. Very useful
+      for monitoring and profiling.
+
+    * Pool workers are supervised, so if for some reason a worker crashes
+        it is automatically replaced by a new worker.
+
+API Reference Documentation
+===========================
+
+The `API Reference`_ is hosted at Github
+(http://ask.github.com/celery)
+
+.. _`API Reference`: http://ask.github.com/celery/
+
+Installation
+=============
+
+You can install ``celery`` either via the Python Package Index (PyPI)
+or from source.
+
+To install using ``pip``,::
+
+    $ pip install celery
+
+To install using ``easy_install``,::
+
+    $ easy_install celery
+
+Downloading and installing from source
+--------------------------------------
+
+Download the latest version of ``celery`` from
+http://pypi.python.org/pypi/celery/
+
+You can install it by doing the following,::
+
+    $ tar xvfz celery-0.0.0.tar.gz
+    $ cd celery-0.0.0
+    $ python setup.py build
+    # python setup.py install # as root
+
+Using the development version
+------------------------------
+
+
+You can clone the repository by doing the following::
+
+    $ git clone git://github.com/ask/celery.git celery
+
+
+Usage
+=====
+
+Installing RabbitMQ
+-------------------
+
+See `Installing RabbitMQ`_ over at RabbitMQ's website. For Mac OS X
+see `Installing RabbitMQ on OS X`_.
+
+.. _`Installing RabbitMQ`: http://www.rabbitmq.com/install.html
+.. _`Installing RabbitMQ on OS X`:
+    http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/
+
+
+Setting up RabbitMQ
+-------------------
+
+To use celery we need to create a RabbitMQ user, a virtual host and
+allow that user access to that virtual host::
+
+    $ rabbitmqctl add_user myuser mypassword
+
+    $ rabbitmqctl add_vhost myvhost
+
+From RabbitMQ version 1.6.0 and onward you have to use the new ACL features
+to allow access::
+
+    $ rabbitmqctl set_permissions -p myvhost myuser "" ".*" ".*"
+
+See the RabbitMQ `Admin Guide`_ for more information about `access control`_.
+
+.. _`Admin Guide`: http://www.rabbitmq.com/admin-guide.html
+
+.. _`access control`: http://www.rabbitmq.com/admin-guide.html#access-control
+
+
+If you are still using version 1.5.0 or below, please use ``map_user_vhost``::
+
+    $ rabbitmqctl map_user_vhost myuser myvhost
+
+
+Configuring your Django project to use Celery
+---------------------------------------------
+
+You only need three simple steps to use celery with your Django project.
+
+    1. Add ``celery`` to ``INSTALLED_APPS``.
+
+    2. Create the celery database tables::
+
+            $ python manage.py syncdb
+
+    3. Configure celery to use the AMQP user and virtual host we created
+        before, by adding the following to your ``settings.py``::
+
+            AMQP_SERVER = "localhost"
+            AMQP_PORT = 5672
+            AMQP_USER = "myuser"
+            AMQP_PASSWORD = "mypassword"
+            AMQP_VHOST = "myvhost"
+
+
+That's it.
+
+There are more options available, like how many processes you want to process
+work in parallel (the ``CELERY_CONCURRENCY`` setting), and the backend used
+for storing task statuses. But for now, this should do. For all of the options
+available, please consult the `API Reference`_
+
+**Note**: If you're using SQLite as the Django database back-end,
+``celeryd`` will only be able to process one task at a time, this is
+because SQLite doesn't allow concurrent writes.
+
+Running the celery worker server
+--------------------------------
+
+To test this we'll be running the worker server in the foreground, so we can
+see what's going on without consulting the logfile::
+
+    $ python manage.py celeryd
+
+
+However, in production you probably want to run the worker in the
+background, as a daemon:: 
+
+    $ python manage.py celeryd --detach
+
+
+For a complete listing of the command line arguments available, with a short
+description, you can use the help command::
+
+    $ python manage.py help celeryd
+
+
+Defining and executing tasks
+----------------------------
+
+**Please note** All of these tasks has to be stored in a real module, they can't
+be defined in the python shell or ipython/bpython. This is because the celery
+worker server needs access to the task function to be able to run it.
+So while it looks like we use the python shell to define the tasks in these
+examples, you can't do it this way. Put them in the ``tasks`` module of your
+Django application. The worker server will automatically load any ``tasks.py``
+file for all of the applications listed in ``settings.INSTALLED_APPS``.
+Executing tasks using ``delay`` and ``apply_async`` can be done from the
+python shell, but keep in mind that since arguments are pickled, you can't
+use custom classes defined in the shell session.
+
+While you can use regular functions, the recommended way is to define
+a task class. This way you can cleanly upgrade the task to use the more
+advanced features of celery later.
+
+This is a task that basically does nothing but take some arguments,
+and return a value:
+
+    >>> from celery.task import Task
+    >>> from celery.registry import tasks
+    >>> class MyTask(Task):
+    ...     def run(self, some_arg, **kwargs):
+    ...         logger = self.get_logger(**kwargs)
+    ...         logger.info("Did something: %s" % some_arg)
+    ...         return 42
+    >>> tasks.register(MyTask)
+
+Now if we want to execute this task, we can use the ``delay`` method of the
+task class (this is a handy shortcut to the ``apply_async`` method which gives
+you greater control of the task execution).
+
+    >>> from myapp.tasks import MyTask
+    >>> MyTask.delay(some_arg="foo")
+
+At this point, the task has been sent to the message broker. The message
+broker will hold on to the task until a celery worker server has successfully
+picked it up.
+
+*Note* If everything is just hanging when you execute ``delay``, please check
+that RabbitMQ is running, and that the user/password has access to the virtual
+host you configured earlier.
+
+Right now we have to check the celery worker logfiles to know what happened with
+the task. This is because we didn't keep the ``AsyncResult`` object returned
+by ``delay``.
+
+The ``AsyncResult`` lets us find the state of the task, wait for the task to
+finish and get its return value (or exception if the task failed).
+
+So, let's execute the task again, but this time we'll keep track of the task:
+
+    >>> result = MyTask.delay("do_something", some_arg="foo bar baz")
+    >>> result.ready() # returns True if the task has finished processing.
+    False
+    >>> result.result # task is not ready, so no return value yet.
+    None
+    >>> result.get()   # Waits until the task is done and return the retval.
+    42
+    >>> result.result
+    42
+    >>> result.successful() # returns True if the task didn't end in failure.
+    True
+
+
+If the task raises an exception, the ``result.success()`` will be ``False``,
+and ``result.result`` will contain the exception instance raised.
+
+Auto-discovery of tasks
+-----------------------
+
+``celery`` has an auto-discovery feature like the Django Admin, that
+automatically loads any ``tasks.py`` module in the applications listed
+in ``settings.INSTALLED_APPS``. This autodiscovery is used by the celery
+worker to find registered tasks for your Django project.
+
+Periodic Tasks
+---------------
+
+Periodic tasks are tasks that are run every ``n`` seconds. 
+Here's an example of a periodic task:
+
+    >>> from celery.task import PeriodicTask
+    >>> from celery.registry import tasks
+    >>> from datetime import timedelta
+    >>> class MyPeriodicTask(PeriodicTask):
+    ...     run_every = timedelta(seconds=30)
+    ...
+    ...     def run(self, **kwargs):
+    ...         logger = self.get_logger(**kwargs)
+    ...         logger.info("Running periodic task!")
+    ...
+    >>> tasks.register(MyPeriodicTask)
+
+**Note:** Periodic tasks does not support arguments, as this doesn't
+really make sense.
+
+
+A look inside the worker
+========================
+
+.. image:: http://cloud.github.com/downloads/ask/celery/InsideTheWorker-v2.jpg
+
+Getting Help
+============
+
+Mailing list
+------------
+
+For discussions about the usage, development, and future of celery,
+please join the `celery-users`_ mailing list. 
+
+.. _`celery-users`: http://groups.google.com/group/celery-users/
+
+IRC
+---
+
+Come chat with us on IRC. The `#celery`_ channel is located at the `Freenode`_
+network.
+
+.. _`#celery`: irc://irc.freenode.net/celery
+.. _`Freenode`: http://freenode.net
+
+
+Bug tracker
+===========
+
+If you have any suggestions, bug reports or annoyances please report them
+to our issue tracker at http://github.com/ask/celery/issues/
+
+Contributing
+============
+
+Development of ``celery`` happens at Github: http://github.com/ask/celery
+
+You are highly encouraged to participate in the development
+of ``celery``. If you don't like Github (for some reason) you're welcome
+to send regular patches.
+
+License
+=======
+
+This software is licensed under the ``New BSD License``. See the ``LICENSE``
+file in the top distribution directory for the full license text.
+
+.. # vim: syntax=rst expandtab tabstop=4 shiftwidth=4 shiftround

README.rst

@@ -2,7 +2,7 @@
 celery - Distributed Task Queue for Django.
 ============================================
 
-:Version: 0.3.12
+:Version: 0.3.20
 
 Introduction
 ============
@@ -237,7 +237,8 @@ advanced features of celery later.
 This is a task that basically does nothing but take some arguments,
 and return a value:
 
-    >>> from celery.task import Task, tasks
+    >>> from celery.task import Task
+    >>> from celery.registry import tasks
     >>> class MyTask(Task):
     ...     name = "myapp.mytask"
     ...     def run(self, some_arg, **kwargs):
@@ -296,7 +297,8 @@ Periodic Tasks
 Periodic tasks are tasks that are run every ``n`` seconds. 
 Here's an example of a periodic task:
 
-    >>> from celery.task import tasks, PeriodicTask
+    >>> from celery.task import PeriodicTask
+    >>> from celery.registry import tasks
     >>> from datetime import timedelta
     >>> class MyPeriodicTask(PeriodicTask):
     ...     name = "foo.my-periodic-task"

bin/celeryd

@@ -4,4 +4,4 @@ from celery.bin.celeryd import run_worker, parse_options
 
 if __name__ == "__main__":
     options = parse_options(sys.argv[1:])
-    run_worker(**options)
+    run_worker(**vars(options))

celery/__init__.py

@@ -1,5 +1,5 @@
 """Distributed Task Queue for Django"""
-VERSION = (0, 3, 12)
+VERSION = (0, 6, 0)
 __version__ = ".".join(map(str, VERSION))
 __author__ = "Ask Solem"
 __contact__ = "askh@opera.com"

celery/backends/base.py

@@ -1,11 +1,12 @@
 """celery.backends.base"""
 import time
+import operator
+from functools import partial as curry
+from celery.serialization import pickle
 
-from celery.timer import TimeoutTimer
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
+
+class TimeoutError(Exception):
+    """The operation timed out."""
 
 
 def find_nearest_pickleable_exception(exc):
@@ -23,13 +24,19 @@ def find_nearest_pickleable_exception(exc):
     :rtype: :exc:`Exception`
 
     """
-    for supercls in exc.__class__.mro():
-        if supercls is Exception:
+
+    unwanted = (Exception, BaseException, object)
+    is_unwanted = lambda exc: any(map(curry(operator.is_, exc), unwanted))
+
+    mro_ = getattr(exc.__class__, "mro", lambda: [])
+    for supercls in mro_():
+        if is_unwanted(supercls):
             # only BaseException and object, from here on down,
             # we don't care about these.
             return None
         try:
-            superexc = supercls(*exc.args)
+            exc_args = getattr(exc, "args", [])
+            superexc = supercls(*exc_args)
             pickle.dumps(superexc)
         except:
             pass
@@ -83,6 +90,7 @@ class BaseBackend(object):
 
     capabilities = []
     UnpickleableExceptionWrapper = UnpickleableExceptionWrapper
+    TimeoutError = TimeoutError
 
     def store_result(self, task_id, result, status):
         """Store the result and status of a task."""
@@ -115,10 +123,8 @@ class BaseBackend(object):
             excwrapper = UnpickleableExceptionWrapper(
                             exc.__class__.__module__,
                             exc.__class__.__name__,
-                            exc.args)
+                            getattr(exc, "args", []))
             return excwrapper
-        else:
-            return exc
 
     def exception_to_python(self, exc):
         """Convert serialized exception to Python exception."""
@@ -128,10 +134,6 @@ class BaseBackend(object):
             return exc_cls(*exc.exc_args)
         return exc
 
-    def mark_as_retry(self, task_id, exc):
-        """Mark task for retry."""
-        return self.store_result(task_id, exc, status="RETRY")
-
     def get_status(self, task_id):
         """Get the status of a task."""
         raise NotImplementedError(
@@ -168,15 +170,21 @@ class BaseBackend(object):
         longer than ``timeout`` seconds.
 
         """
-        timeout_timer = TimeoutTimer(timeout)
+
+        sleep_inbetween = 0.5
+        time_elapsed = 0.0
+
         while True:
             status = self.get_status(task_id)
             if status == "DONE":
                 return self.get_result(task_id)
             elif status == "FAILURE":
                 raise self.get_result(task_id)
-            time.sleep(0.5) # avoid hammering the CPU checking status.
-            timeout_timer.tick()
+            # avoid hammering the CPU checking status.
+            time.sleep(sleep_inbetween)
+            time_elapsed += sleep_inbetween
+            if timeout and time_elapsed >= timeout:
+                raise TimeoutError("The operation timed out.")
 
     def process_cleanup(self):
         """Cleanup actions to do at the end of a task worker process.
@@ -185,3 +193,60 @@ class BaseBackend(object):
 
         """
         pass
+
+
+class KeyValueStoreBackend(BaseBackend):
+
+    capabilities = ["ResultStore"]
+
+    def __init__(self, *args, **kwargs):
+        super(KeyValueStoreBackend, self).__init__()
+        self._cache = {}
+
+    def get_cache_key_for_task(self, task_id):
+        """Get the cache key for a task by id."""
+        return "celery-task-meta-%s" % task_id
+
+    def get(self, key):
+        raise NotImplementedError("Must implement the get method.")
+
+    def set(self, key, value):
+        raise NotImplementedError("Must implement the set method.")
+
+    def store_result(self, task_id, result, status):
+        """Store task result and status."""
+        if status == "DONE":
+            result = self.prepare_result(result)
+        elif status == "FAILURE":
+            result = self.prepare_exception(result)
+        meta = {"status": status, "result": result}
+        self.set(self.get_cache_key_for_task(task_id), pickle.dumps(meta))
+        return result
+
+    def get_status(self, task_id):
+        """Get the status of a task."""
+        return self._get_task_meta_for(task_id)["status"]
+
+    def get_result(self, task_id):
+        """Get the result of a task."""
+        meta = self._get_task_meta_for(task_id)
+        if meta["status"] == "FAILURE":
+            return self.exception_to_python(meta["result"])
+        else:
+            return meta["result"]
+
+    def is_done(self, task_id):
+        """Returns ``True`` if the task executed successfully."""
+        return self.get_status(task_id) == "DONE"
+
+    def _get_task_meta_for(self, task_id):
+        """Get task metadata for a task by id."""
+        if task_id in self._cache:
+            return self._cache[task_id]
+        meta = self.get(self.get_cache_key_for_task(task_id))
+        if not meta:
+            return {"status": "PENDING", "result": None}
+        meta = pickle.loads(str(meta))
+        if meta.get("status") == "DONE":
+            self._cache[task_id] = meta
+        return meta

celery/backends/cache.py

@@ -1,58 +1,13 @@
 """celery.backends.cache"""
 from django.core.cache import cache
-from celery.backends.base import BaseBackend
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
+from celery.backends.base import KeyValueStoreBackend
 
 
-class Backend(BaseBackend):
+class Backend(KeyValueStoreBackend):
     """Backend using the Django cache framework to store task metadata."""
 
-    capabilities = ["ResultStore"]
+    def get(self, key):
+        return cache.get(key)
 
-    def __init__(self, *args, **kwargs):
-        super(Backend, self).__init__(*args, **kwargs)
-        self._cache = {}
-
-    def _cache_key(self, task_id):
-        """Get the cache key for a task by id."""
-        return "celery-task-meta-%s" % task_id
-
-    def store_result(self, task_id, result, status):
-        """Store task result and status."""
-        if status == "DONE":
-            result = self.prepare_result(result)
-        elif status == "FAILURE":
-            result = self.prepare_exception(result)
-        meta = {"status": status, "result": pickle.dumps(result)}
-        cache.set(self._cache_key(task_id), meta)
-
-    def get_status(self, task_id):
-        """Get the status of a task."""
-        return self._get_task_meta_for(task_id)["status"]
-
-    def get_result(self, task_id):
-        """Get the result of a task."""
-        meta = self._get_task_meta_for(task_id)
-        if meta["status"] == "FAILURE":
-            return self.exception_to_python(meta["result"])
-        else:
-            return meta["result"]
-
-    def is_done(self, task_id):
-        """Returns ``True`` if the task has been executed successfully."""
-        return self.get_status(task_id) == "DONE"
-
-    def _get_task_meta_for(self, task_id):
-        """Get the task metadata for a task by id."""
-        if task_id in self._cache:
-            return self._cache[task_id]
-        meta = cache.get(self._cache_key(task_id))
-        if not meta:
-            return {"status": "PENDING", "result": None}
-        meta["result"] = pickle.loads(meta.get("result", None))
-        if meta.get("status") == "DONE":
-            self._cache[task_id] = meta
-        return meta
+    def set(self, key, value):
+        cache.set(key, value)

celery/backends/database.py

@@ -12,11 +12,23 @@ class Backend(BaseBackend):
         super(Backend, self).__init__(*args, **kwargs)
         self._cache = {}
 
+    def init_periodic_tasks(self):
+        """Create entries for all periodic tasks in the database."""
+        PeriodicTaskMeta.objects.init_entries()
+
     def run_periodic_tasks(self):
-        """Run all waiting periodic tasks."""
+        """Run all waiting periodic tasks.
+
+        :returns: a list of ``(task, task_id)`` tuples containing
+            the task class and id for the resulting tasks applied.
+
+        """
         waiting_tasks = PeriodicTaskMeta.objects.get_waiting_tasks()
+        task_id_tuples = []
         for waiting_task in waiting_tasks:
-            waiting_task.delay()
+            task_id = waiting_task.delay()
+            task_id_tuples.append((waiting_task, task_id))
+        return task_id_tuples
 
     def store_result(self, task_id, result, status):
         """Mark task as done (executed)."""
@@ -24,7 +36,8 @@ class Backend(BaseBackend):
             result = self.prepare_result(result)
         elif status == "FAILURE":
             result = self.prepare_exception(result)
-        return TaskMeta.objects.store_result(task_id, result, status)
+        TaskMeta.objects.store_result(task_id, result, status)
+        return result
 
     def is_done(self, task_id):
         """Returns ``True`` if task with ``task_id`` has been executed."""

celery/backends/tyrant.py

@@ -7,16 +7,11 @@ except ImportError:
     raise ImproperlyConfigured(
             "The Tokyo Tyrant backend requires the pytyrant library.")
 
-from celery.backends.base import BaseBackend
+from celery.backends.base import KeyValueStoreBackend
 from django.conf import settings
-from carrot.messaging import serialize, deserialize
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
 
 
-class Backend(BaseBackend):
+class Backend(KeyValueStoreBackend):
     """Tokyo Cabinet based task backend store.
 
     .. attribute:: tyrant_host
@@ -31,8 +26,6 @@ class Backend(BaseBackend):
     tyrant_host = None
     tyrant_port = None
 
-    capabilities = ["ResultStore"]
-
     def __init__(self, tyrant_host=None, tyrant_port=None):
         """Initialize Tokyo Tyrant backend instance.
 
@@ -44,68 +37,41 @@ class Backend(BaseBackend):
                             getattr(settings, "TT_HOST", self.tyrant_host)
         self.tyrant_port = tyrant_port or \
                             getattr(settings, "TT_PORT", self.tyrant_port)
+        if self.tyrant_port:
+            self.tyrant_port = int(self.tyrant_port)
         if not self.tyrant_host or not self.tyrant_port:
             raise ImproperlyConfigured(
                 "To use the Tokyo Tyrant backend, you have to "
                 "set the TT_HOST and TT_PORT settings in your settings.py")
         super(Backend, self).__init__()
-        self._cache = {}
         self._connection = None
 
     def open(self):
         """Get :class:`pytyrant.PyTyrant`` instance with the current
-        server configuration."""
-        if not self._connection:
+        server configuration.
+
+        The connection is then cached until you do an
+        explicit :meth:`close`.
+
+        """
+        # connection overrides bool()
+        if self._connection is None:
             self._connection = pytyrant.PyTyrant.open(self.tyrant_host,
                                                       self.tyrant_port)
         return self._connection
 
     def close(self):
-        if self._connection:
+        """Close the tyrant connection and remove the cache."""
+        # connection overrides bool()
+        if self._connection is not None:
             self._connection.close()
             self._connection = None
 
     def process_cleanup(self):
         self.close()
 
-    def _cache_key(self, task_id):
-        """Get the cache key for a task by id."""
-        return "celery-task-meta-%s" % task_id
-
-    def store_result(self, task_id, result, status):
-        """Store task result and status."""
-        if status == "DONE":
-            result = self.prepare_result(result)
-        elif status == "FAILURE":
-            result = self.prepare_exception(result)
-        meta = {"status": status, "result": pickle.dumps(result)}
-        self.open()[self._cache_key(task_id)] = serialize(meta)
-
-    def get_status(self, task_id):
-        """Get the status for a task."""
-        return self._get_task_meta_for(task_id)["status"]
-
-    def get_result(self, task_id):
-        """Get the result of a task."""
-        meta = self._get_task_meta_for(task_id)
-        if meta["status"] == "FAILURE":
-            return self.exception_to_python(meta["result"])
-        else:
-            return meta["result"]
-
-    def is_done(self, task_id):
-        """Returns ``True`` if the task executed successfully."""
-        return self.get_status(task_id) == "DONE"
-
-    def _get_task_meta_for(self, task_id):
-        """Get task metadata for a task by id."""
-        if task_id in self._cache:
-            return self._cache[task_id]
-        meta = self.open().get(self._cache_key(task_id))
-        if not meta:
-            return {"status": "PENDING", "result": None}
-        meta = deserialize(meta)
-        meta["result"] = pickle.loads(meta.get("result", None))
-        if meta.get("status") == "DONE":
-            self._cache[task_id] = meta
-        return meta
+    def get(self, key):
+        return self.open().get(key)
+
+    def set(self, key, value):
+        self.open()[key] = value

celery/bin/celeryd.py

@@ -30,6 +30,10 @@
 
     Run in the background as a daemon.
 
+.. cmdoption:: -S, --supervised
+
+    Restart the worker server if it dies.
+
 .. cmdoption:: --discard
 
     Discard all waiting tasks before the daemon is started.
@@ -71,6 +75,7 @@ if django_project_dir:
 
 from django.conf import settings
 from celery import __version__
+from celery.supervisor import OFASupervisor
 from celery.log import emergency_error
 from celery.conf import LOG_LEVELS, DAEMON_LOG_FILE, DAEMON_LOG_LEVEL
 from celery.conf import DAEMON_CONCURRENCY, DAEMON_PID_FILE
@@ -123,6 +128,9 @@ OPTION_LIST = (
     optparse.make_option('-d', '--detach', '--daemon', default=False,
             action="store_true", dest="detach",
             help="Run in the background as a daemon."),
+    optparse.make_option('-S', '--supervised', default=False,
+            action="store_true", dest="supervised",
+            help="Restart the worker server if it dies."),
     optparse.make_option('-u', '--uid', default=None,
             action="store", dest="uid",
             help="User-id to run celeryd as when in daemon mode."),
@@ -163,7 +171,7 @@ def acquire_pidlock(pidfile):
     except os.error, exc:
         if exc.errno == errno.ESRCH:
             sys.stderr.write("Stale pidfile exists. Removing it.\n")
-            pidlock.release()
+            os.unlink(pidfile)
             return PIDLockFile(pidfile)
     else:
         raise SystemExit(
@@ -176,7 +184,8 @@ def acquire_pidlock(pidfile):
 def run_worker(concurrency=DAEMON_CONCURRENCY, detach=False,
         loglevel=DAEMON_LOG_LEVEL, logfile=DAEMON_LOG_FILE, discard=False,
         pidfile=DAEMON_PID_FILE, umask=0, uid=None, gid=None,
-        working_directory=None, chroot=None, statistics=None, **kwargs):
+        supervised=False, working_directory=None, chroot=None,
+        statistics=None, **kwargs):
     """Starts the celery worker server."""
 
     print("Celery %s is starting." % __version__)
@@ -248,18 +257,25 @@ def run_worker(concurrency=DAEMON_CONCURRENCY, detach=False,
         context.open()
 
     discovery.autodiscover()
-    worker = WorkController(concurrency=concurrency,
-                            loglevel=loglevel,
-                            logfile=logfile,
-                            is_detached=detach)
 
-    try:
-        worker.run()
-    except Exception, e:
-        emergency_error(logfile, "celeryd raised exception %s: %s\n%s" % (
+    def run_worker():
+        worker = WorkController(concurrency=concurrency,
+                                loglevel=loglevel,
+                                logfile=logfile,
+                                is_detached=detach)
+        try:
+            worker.start()
+        except Exception, e:
+            emergency_error(logfile, "celeryd raised exception %s: %s\n%s" % (
                             e.__class__, e, traceback.format_exc()))
+
+    try:
+        if supervised:
+            OFASupervisor(target=run_worker).start()
+        else:
+            run_worker()
     except:
-        if daemon:
+        if detach:
             context.close()
         raise
 
@@ -273,4 +289,4 @@ def parse_options(arguments):
 
 if __name__ == "__main__":
     options = parse_options(sys.argv[1:])
-    run_worker(**options)
+    run_worker(**vars(options))

celery/conf.py

@@ -1,5 +1,6 @@
 """celery.conf"""
 from django.conf import settings
+from datetime import timedelta
 import logging
 
 DEFAULT_AMQP_EXCHANGE = "celery"
@@ -15,6 +16,8 @@ DEFAULT_DAEMON_LOG_FILE = "celeryd.log"
 DEFAULT_AMQP_CONNECTION_TIMEOUT = 4
 DEFAULT_STATISTICS = False
 DEFAULT_STATISTICS_COLLECT_INTERVAL = 60 * 5
+DEFAULT_ALWAYS_EAGER = False
+DEFAULT_TASK_RESULT_EXPIRES = timedelta(days=5)
 
 """
 .. data:: LOG_LEVELS
@@ -182,3 +185,29 @@ SEND_CELERY_TASK_ERROR_EMAILS = getattr(settings,
 STATISTICS_COLLECT_INTERVAL = getattr(settings,
                                 "CELERY_STATISTICS_COLLECT_INTERVAL",
                                 DEFAULT_STATISTICS_COLLECT_INTERVAL)
+
+"""
+.. data:: ALWAYS_EAGER
+
+    If this is ``True``, all tasks will be executed locally by blocking
+    until it is finished. ``apply_async`` and ``delay_task`` will return
+    a :class:`celery.result.EagerResult` which emulates the behaviour of
+    an :class:`celery.result.AsyncResult`.
+
+"""
+ALWAYS_EAGER = getattr(settings, "CELERY_ALWAYS_EAGER",
+                       DEFAULT_ALWAYS_EAGER)
+
+"""
+.. data: TASK_RESULT_EXPIRES
+
+    Time (in seconds, or a :class:`datetime.timedelta` object) for when after
+    stored task results are deleted. For the moment this only works for the
+    database backend.
+"""
+TASK_RESULT_EXPIRES = getattr(settings, "CELERY_TASK_RESULT_EXPIRES",
+                              DEFAULT_TASK_RESULT_EXPIRES)
+
+# Make sure TASK_RESULT_EXPIRES is a timedelta.
+if isinstance(TASK_RESULT_EXPIRES, int):
+    TASK_RESULT_EXPIRES = timedelta(seconds=TASK_RESULT_EXPIRES)

celery/execute.py

@@ -0,0 +1,145 @@
+from carrot.connection import DjangoAMQPConnection
+from celery.conf import AMQP_CONNECTION_TIMEOUT
+from celery.result import AsyncResult, EagerResult
+from celery.messaging import TaskPublisher
+from celery.registry import tasks
+from celery.utils import gen_unique_id
+from functools import partial as curry
+from datetime import datetime, timedelta
+import inspect
+
+
+def apply_async(task, args=None, kwargs=None, routing_key=None,
+        immediate=None, mandatory=None, connection=None,
+        connect_timeout=AMQP_CONNECTION_TIMEOUT, priority=None,
+        countdown=None, eta=None, **opts):
+    """Run a task asynchronously by the celery daemon(s).
+
+    :param task: The task to run (a callable object, or a :class:`Task`
+        instance
+
+    :param args: The positional arguments to pass on to the task (a ``list``).
+
+    :param kwargs: The keyword arguments to pass on to the task (a ``dict``)
+
+    :param countdown: Number of seconds into the future that the task should
+        execute. Defaults to immediate delivery (Do not confuse that with
+        the ``immediate`` setting, they are unrelated).
+
+    :param eta: A :class:`datetime.datetime` object that describes the
+        absolute time when the task should execute. May not be specified
+        if ``countdown`` is also supplied. (Do not confuse this with the
+        ``immediate`` setting, they are unrelated).
+
+    :keyword routing_key: The routing key used to route the task to a worker
+        server.
+
+    :keyword immediate: Request immediate delivery. Will raise an exception
+        if the task cannot be routed to a worker immediately.
+        (Do not confuse this parameter with the ``countdown`` and ``eta``
+        settings, as they are unrelated).
+
+    :keyword mandatory: Mandatory routing. Raises an exception if there's
+        no running workers able to take on this task.
+
+    :keyword connection: Re-use existing AMQP connection.
+        The ``connect_timeout`` argument is not respected if this is set.
+
+    :keyword connect_timeout: The timeout in seconds, before we give up
+        on establishing a connection to the AMQP server.
+
+    :keyword priority: The task priority, a number between ``0`` and ``9``.
+
+    """
+    args = args or []
+    kwargs = kwargs or {}
+    routing_key = routing_key or getattr(task, "routing_key", None)
+    immediate = immediate or getattr(task, "immediate", None)
+    mandatory = mandatory or getattr(task, "mandatory", None)
+    priority = priority or getattr(task, "priority", None)
+    taskset_id = opts.get("taskset_id")
+    publisher = opts.get("publisher")
+    if countdown:
+        eta = datetime.now() + timedelta(seconds=countdown)
+
+    from celery.conf import ALWAYS_EAGER
+    if ALWAYS_EAGER:
+        return apply(task, args, kwargs)
+
+    need_to_close_connection = False
+    if not publisher:
+        if not connection:
+            connection = DjangoAMQPConnection(connect_timeout=connect_timeout)
+            need_to_close_connection = True
+        publisher = TaskPublisher(connection=connection)
+
+    delay_task = publisher.delay_task
+    if taskset_id:
+        delay_task = curry(publisher.delay_task_in_set, taskset_id)
+
+    task_id = delay_task(task.name, args, kwargs,
+                         routing_key=routing_key, mandatory=mandatory,
+                         immediate=immediate, priority=priority,
+                         eta=eta)
+
+    if need_to_close_connection:
+        publisher.close()
+        connection.close()
+
+    return AsyncResult(task_id)
+
+
+def delay_task(task_name, *args, **kwargs):
+    """Delay a task for execution by the ``celery`` daemon.
+
+    :param task_name: the name of a task registered in the task registry.
+
+    :param \*args: positional arguments to pass on to the task.
+
+    :param \*\*kwargs: keyword arguments to pass on to the task.
+
+    :raises celery.registry.NotRegistered: exception if no such task
+        has been registered in the task registry.
+
+    :rtype: :class:`celery.result.AsyncResult`.
+
+    Example
+
+        >>> r = delay_task("update_record", name="George Constanza", age=32)
+        >>> r.ready()
+        True
+        >>> r.result
+        "Record was updated"
+
+    """
+    if task_name not in tasks:
+        raise tasks.NotRegistered(
+                "Task with name %s not registered in the task registry." % (
+                    task_name))
+    task = tasks[task_name]
+    return apply_async(task, args, kwargs)
+
+
+def apply(task, args, kwargs, **ignored):
+    """Apply the task locally.
+
+    This will block until the task completes, and returns a
+    :class:`celery.result.EagerResult` instance.
+
+    """
+    args = args or []
+    kwargs = kwargs or {}
+    task_id = gen_unique_id()
+
+    # If it's a Task class we need to have to instance
+    # for it to be callable.
+    task = inspect.isclass(task) and task() or task
+
+    try:
+        ret_value = task(*args, **kwargs)
+        status = "DONE"
+    except Exception, exc:
+        ret_value = exc
+        status = "FAILURE"
+
+    return EagerResult(task_id, ret_value, status)

celery/fields.py

@@ -5,11 +5,7 @@ Custom Django Model Fields.
 """
 from django.db import models
 from django.conf import settings
-
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
+from celery.serialization import pickle
 
 
 class PickledObject(str):

celery/managers.py

@@ -1,7 +1,10 @@
 """celery.managers"""
 from django.db import models
+from django.db import connection
 from celery.registry import tasks
+from celery.conf import TASK_RESULT_EXPIRES
 from datetime import datetime, timedelta
+from django.conf import settings
 import random
 
 # server_drift can be negative, but timedelta supports addition on
@@ -9,6 +12,54 @@ import random
 SERVER_DRIFT = timedelta(seconds=random.vonmisesvariate(1, 4))
 
 
+class TableLock(object):
+    """Base class for database table locks. Also works as a NOOP lock."""
+
+    def __init__(self, table, type="read"):
+        self.table = table
+        self.type = type
+        self.cursor = None
+
+    def lock_table(self):
+        """Lock the table."""
+        pass
+
+    def unlock_table(self):
+        """Release previously locked tables."""
+        pass
+
+    @classmethod
+    def acquire(cls, table, type=None):
+        """Acquire table lock."""
+        lock = cls(table, type)
+        lock.lock_table()
+        return lock
+
+    def release(self):
+        """Release the lock."""
+        self.unlock_table()
+        if self.cursor:
+            self.cursor.close()
+            self.cursor = None
+
+
+class MySQLTableLock(TableLock):
+    """Table lock support for MySQL."""
+
+    def lock_table(self):
+        """Lock MySQL table."""
+        self.cursor = connection.cursor()
+        self.cursor.execute("LOCK TABLES %s %s" % (
+            self.table, self.type.upper()))
+
+    def unlock_table(self):
+        """Unlock MySQL table."""
+        self.cursor.execute("UNLOCK TABLES")
+
+TABLE_LOCK_FOR_ENGINE = {"mysql": MySQLTableLock}
+table_lock = TABLE_LOCK_FOR_ENGINE.get(settings.DATABASE_ENGINE, TableLock)
+
+
 class TaskManager(models.Manager):
     """Manager for :class:`celery.models.Task` models."""
 
@@ -23,8 +74,7 @@ class TaskManager(models.Manager):
 
     def get_all_expired(self):
         """Get all expired task results."""
-        # TODO Make the timedelta configurable
-        return self.filter(date_done__lt=datetime.now() - timedelta(days=5))
+        return self.filter(date_done__lt=datetime.now() - TASK_RESULT_EXPIRES)
 
     def delete_expired(self):
         """Delete all expired task results."""
@@ -55,20 +105,54 @@ class TaskManager(models.Manager):
 class PeriodicTaskManager(models.Manager):
     """Manager for :class:`celery.models.PeriodicTask` models."""
 
+    def init_entries(self):
+        """Add entries for all registered periodic tasks.
+
+        Should be run at worker start.
+        """
+        periodic_tasks = tasks.get_all_periodic()
+        for task_name in periodic_tasks.keys():
+            task_meta, created = self.get_or_create(name=task_name)
+
+    def is_time(self, last_run_at, run_every):
+        """Check if if it is time to run the periodic task.
+
+        :param last_run_at: Last time the periodic task was run.
+        :param run_every: How often to run the periodic task.
+
+        :rtype bool:
+
+        """
+        run_every_drifted = run_every + SERVER_DRIFT
+        run_at = last_run_at + run_every_drifted
+        if datetime.now() > run_at:
+            return True
+        return False
+
     def get_waiting_tasks(self):
         """Get all waiting periodic tasks.
 
         :returns: list of :class:`celery.models.PeriodicTaskMeta` objects.
         """
         periodic_tasks = tasks.get_all_periodic()
+        db_table = self.model._meta.db_table
+
+        # Find all periodic tasks to be run.
         waiting = []
-        # XXX This will become a lot of queries. Maybe just only create
-        # the rows at init, and then select all later.
-        for task_name, task in periodic_tasks.items():
-            task_meta, created = self.get_or_create(name=task_name)
-            # task_run.every must be a timedelta object.
-            run_every_drifted = task.run_every + SERVER_DRIFT
-            run_at = task_meta.last_run_at + run_every_drifted
-            if datetime.now() > run_at:
-                waiting.append(task_meta)
+        for task_meta in self.all():
+            if task_meta.name in periodic_tasks:
+                task = periodic_tasks[task_meta.name]
+                run_every = task.run_every
+                if self.is_time(task_meta.last_run_at, run_every):
+                    # Get the object again to be sure noone else
+                    # has already taken care of it.
+                    lock = table_lock.acquire(db_table, "write")
+                    try:
+                        secure = self.get(pk=task_meta.pk)
+                        if self.is_time(secure.last_run_at, run_every):
+                            secure.last_run_at = datetime.now()
+                            secure.save()
+                            waiting.append(secure)
+                    finally:
+                        lock.release()
         return waiting

celery/messaging.py

@@ -5,12 +5,17 @@ Sending and Receiving Messages
 """
 from carrot.messaging import Publisher, Consumer, ConsumerSet
 from celery import conf
-import uuid
+from celery.utils import gen_unique_id
+from celery.utils import mitemgetter
+from celery.serialization import pickle
 
-try:
-    import cPickle as pickle
-except ImportError:
-    import pickle
+
+MSG_OPTIONS = ("mandatory", "priority",
+               "immediate", "routing_key")
+
+get_msg_options = mitemgetter(*MSG_OPTIONS)
+
+extract_msg_options = lambda d: dict(zip(MSG_OPTIONS, get_msg_options(d)))
 
 
 class TaskPublisher(Publisher):
@@ -18,6 +23,7 @@ class TaskPublisher(Publisher):
     exchange = conf.AMQP_EXCHANGE
     exchange_type = conf.AMQP_EXCHANGE_TYPE
     routing_key = conf.AMQP_PUBLISHER_ROUTING_KEY
+    serializer = "pickle"
     encoder = pickle.dumps
 
     def delay_task(self, task_name, task_args, task_kwargs, **kwargs):
@@ -32,35 +38,29 @@ class TaskPublisher(Publisher):
                                 task_args=task_args, task_kwargs=task_kwargs,
                                 **kwargs)
 
-    def requeue_task(self, task_name, task_id, task_args, task_kwargs,
-            part_of_set=None, **kwargs):
-        """Requeue a failed task."""
-        return self._delay_task(task_name=task_name, part_of_set=part_of_set,
-                                task_id=task_id, task_args=task_args,
-                                task_kwargs=task_kwargs, **kwargs)
+    def retry_task(self, task_name, task_id, delivery_info, **kwargs):
+        kwargs["routing_key"] = delivery_info.get("routing_key")
+        kwargs["retries"] = kwargs.get("retries", 0) + 1
+        self._delay_task(task_name, task_id, **kwargs)
 
     def _delay_task(self, task_name, task_id=None, part_of_set=None,
             task_args=None, task_kwargs=None, **kwargs):
         """INTERNAL"""
-        priority = kwargs.get("priority")
-        immediate = kwargs.get("immediate")
-        mandatory = kwargs.get("mandatory")
-        routing_key = kwargs.get("routing_key")
-    
-        task_args = task_args or []
-        task_kwargs = task_kwargs or {}
-        task_id = task_id or str(uuid.uuid4())
+
+        task_id = task_id or gen_unique_id()
+
         message_data = {
-            "id": task_id,
             "task": task_name,
-            "args": task_args,
-            "kwargs": task_kwargs,
+            "id": task_id,
+            "args": task_args or [],
+            "kwargs": task_kwargs or {},
+            "retries": kwargs.get("retries", 0),
+            "eta": kwargs.get("eta"),
         }
         if part_of_set:
             message_data["taskset"] = part_of_set
-        self.send(message_data,
-                routing_key=routing_key, priority=priority,
-                immediate=immediate, mandatory=mandatory)
+
+        self.send(message_data, **extract_msg_options(kwargs))
         return task_id
 
 
@@ -92,6 +92,3 @@ class StatsConsumer(Consumer):
     exchange_type = "direct"
     decoder = pickle.loads
     no_ack=True
-
-    def receive(self, message_data, message):
-        pass

celery/models.py

@@ -8,6 +8,7 @@ from celery.registry import tasks
 from celery.managers import TaskManager, PeriodicTaskManager
 from celery.fields import PickledObjectField
 from django.utils.translation import ugettext_lazy as _
+from datetime import datetime
 
 TASK_STATUS_PENDING = "PENDING"
 TASK_STATUS_RETRY = "RETRY"
@@ -41,7 +42,8 @@ class PeriodicTaskMeta(models.Model):
     """Information about a Periodic Task."""
     name = models.CharField(_(u"name"), max_length=255, unique=True)
     last_run_at = models.DateTimeField(_(u"last time run"),
-                                       auto_now=True, blank=True)
+                                       blank=True,
+                                       default=datetime.fromtimestamp(0))
     total_run_count = models.PositiveIntegerField(_(u"total run count"),
                                                   default=0)
 

celery/monitoring.py

@@ -14,7 +14,7 @@ DEFAULT_CACHE_KEY_PREFIX = "celery-statistics"
 
 class Statistics(object):
     """Base class for classes publishing celery statistics.
-    
+
     .. attribute:: type
 
         **REQUIRED** The type of statistics this class handles.
@@ -97,7 +97,7 @@ class TimerStats(Statistics):
         self.args = args
         self.kwargs = kwargs
         self.time_start = time.time()
-    
+
     def on_finish(self):
         """What to do when the timers :meth:`stop` method is called.
 
@@ -120,7 +120,7 @@ class TaskTimerStats(TimerStats):
 
 class StatsCollector(object):
     """Collect and report Celery statistics.
-    
+
     **NOTE**: Please run only one collector at any time, or your stats
         will be skewed.
 
@@ -142,14 +142,14 @@ class StatsCollector(object):
         instance.
 
     .. attribute:: total_task_time_running_by_type
-        
+
         A dictionary of task names and their total running time in seconds,
         counting all the tasks that has been run since the first time
         :meth:`collect` was executed on this class instance.
 
     **NOTE**: You have to run :meth:`collect` for these attributes
         to be filled.
-        
+
 
     """
 
@@ -171,8 +171,11 @@ class StatsCollector(object):
             stats_entry = message.decode()
             stat_type = stats_entry["type"]
             if stat_type in self.allowed_types:
+                # Decode keys to unicode for use as kwargs.
+                data = dict((key.encode("utf-8"), value)
+                                for key, value in stats_entry["data"].items())
                 handler = getattr(self, stat_type)
-                handler(**stats_entry["data"])
+                handler(**data)
 
     def dump_to_cache(self, cache_key_prefix=DEFAULT_CACHE_KEY_PREFIX):
         """Store collected statistics in the cache."""
@@ -221,7 +224,7 @@ class StatsCollector(object):
             * Total task processing time.
 
             * Total number of tasks executed
-        
+
         """
         print("Total processing time by task type:")
         for task_name, nsecs in self.total_task_time_running_by_type.items():

celery/pool.py

@@ -3,18 +3,183 @@
 Process Pools.
 
 """
+import os
+import time
+import errno
 import multiprocessing
-import itertools
-import threading
-import uuid
 
-from multiprocessing.pool import RUN as POOL_STATE_RUN
+from multiprocessing.pool import Pool, worker
 from celery.datastructures import ExceptionInfo
+from celery.utils import gen_unique_id
+from functools import partial as curry
+from operator import isNumberType
+
+
+def pid_is_dead(pid):
+    """Check if a process is not running by PID.
+
+    :rtype bool:
+
+    """
+    try:
+        return os.kill(pid, 0)
+    except OSError, err:
+        if err.errno == errno.ESRCH:
+            return True # No such process.
+        elif err.errno == errno.EPERM:
+            return False # Operation not permitted.
+        else:
+            raise
+
+
+def reap_process(pid):
+    """Reap process if the process is a zombie.
+
+    :returns: ``True`` if process was reaped or is not running,
+        ``False`` otherwise.
+
+    """
+    if pid_is_dead(pid):
+        return True
+
+    try:
+        is_dead, _ = os.waitpid(pid, os.WNOHANG)
+    except OSError, err:
+        if err.errno == errno.ECHILD:
+            return False # No child processes.
+        raise
+    return is_dead
+
+
+def process_is_dead(process):
+    """Check if process is not running anymore.
+
+    First it finds out if the process is running by sending
+    signal 0. Then if the process is a child process, and is running
+    it finds out if it's a zombie process and reaps it.
+    If the process is running and is not a zombie it tries to send
+    a ping through the process pipe.
+
+    :param process: A :class:`multiprocessing.Process` instance.
+
+    :returns: ``True`` if the process is not running, ``False`` otherwise.
+
+    """
+
+    # Try to see if the process is actually running,
+    # and reap zombie proceses while we're at it.
+
+    if reap_process(process.pid):
+        return True
+
+    # Then try to ping the process using its pipe.
+    try:
+        proc_is_alive = process.is_alive()
+    except OSError:
+        return True
+    else:
+        return not proc_is_alive
+
+
+class DynamicPool(Pool):
+    """Version of :class:`multiprocessing.Pool` that can dynamically grow
+    in size."""
+
+    def __init__(self, processes=None, initializer=None, initargs=()):
+
+        if processes is None:
+            try:
+                processes = cpu_count()
+            except NotImplementedError:
+                processes = 1
+
+        super(DynamicPool, self).__init__(processes=processes,
+                                          initializer=initializer,
+                                          initargs=initargs)
+        self._initializer = initializer
+        self._initargs = initargs
+        self._size = processes
+        self.logger = multiprocessing.get_logger()
+
+    def _my_cleanup(self):
+        from multiprocessing.process import _current_process
+        for p in list(_current_process._children):
+            discard = False
+            try:
+                status = p._popen.poll()
+            except OSError:
+                discard = True
+            else:
+                if status is not None:
+                    discard = True
+            if discard:
+                _current_process._children.discard(p)
+
+    def add_worker(self):
+        """Add another worker to the pool."""
+        self._my_cleanup()
+        w = self.Process(target=worker,
+                         args=(self._inqueue, self._outqueue,
+                               self._initializer, self._initargs))
+        w.name = w.name.replace("Process", "PoolWorker")
+        w.daemon = True
+        w.start()
+        self._pool.append(w)
+        self.logger.debug(
+            "DynamicPool: Started pool worker %s (PID: %s, Poolsize: %d)" %(
+                w.name, w.pid, len(self._pool)))
+
+    def grow(self, size=1):
+        """Add workers to the pool.
+
+        :keyword size: Number of workers to add (default: 1)
+
+        """
+        [self.add_worker() for i in range(size)]
+
+    def _is_dead(self, process):
+        """Try to find out if the process is dead.
+
+        :rtype bool:
+
+        """
+        if process_is_dead(process):
+            self.logger.info("DynamicPool: Found dead process (PID: %s)" % (
+                process.pid))
+            return True
+        return False
+
+    def _bring_out_the_dead(self):
+        """Sort out dead process from pool.
+
+        :returns: Tuple of two lists, the first list with dead processes,
+            the second with active processes.
+
+        """
+        dead, alive = [], []
+        for process in self._pool:
+            if process and process.pid and isNumberType(process.pid):
+                dest = dead if self._is_dead(process) else alive
+                dest.append(process)
+        return dead, alive
+
+    def replace_dead_workers(self):
+        """Replace dead workers in the pool by spawning new ones.
+
+        :returns: number of dead processes replaced, or ``None`` if all
+            processes are alive and running.
+
+        """
+        dead, alive = self._bring_out_the_dead()
+        if dead:
+            dead_count = len(dead)
+            self._pool = alive
+            self.grow(self._size if dead_count > self._size else dead_count)
+            return dead_count
 
 
 class TaskPool(object):
-    """Pool of running child processes, which starts waiting for the
-    processes to finish when the queue limit has been reached.
+    """Process Pool for processing tasks in parallel.
 
     :param limit: see :attr:`limit` attribute.
     :param logger: see :attr:`logger` attribute.
@@ -22,8 +187,7 @@ class TaskPool(object):
 
     .. attribute:: limit
 
-        The number of processes that can run simultaneously until
-        we start collecting results.
+        The number of processes that can run simultaneously.
 
     .. attribute:: logger
 
@@ -34,56 +198,31 @@ class TaskPool(object):
     def __init__(self, limit, logger=None):
         self.limit = limit
         self.logger = logger or multiprocessing.get_logger()
-        self._process_counter = itertools.count(1)
-        self._processed_total = 0
         self._pool = None
-        self._processes = None
 
-    def run(self):
+    def start(self):
         """Run the task pool.
 
         Will pre-fork all workers so they're ready to accept tasks.
 
         """
-        self._start()
+        self._pool = DynamicPool(processes=self.limit)
 
-    def _start(self):
-        """INTERNAL: Starts the pool. Used by :meth:`run`."""
-        self._processes = {}
-        self._pool = multiprocessing.Pool(processes=self.limit)
-
-    def terminate(self):
+    def stop(self):
         """Terminate the pool."""
         self._pool.terminate()
+        self._pool = None
 
-    def _terminate_and_restart(self):
-        """INTERNAL: Terminate and restart the pool."""
-        try:
-            self.terminate()
-        except OSError:
-            pass
-        self._start()
-
-    def _restart(self):
-        """INTERNAL: Close and restart the pool."""
-        self.logger.info("Closing and restarting the pool...")
-        self._pool.close()
-        timeout_thread = threading.Timer(30.0, self._terminate_and_restart)
-        timeout_thread.start()
-        self._pool.join()
-        timeout_thread.cancel()
-        self._start()
-
-    def _pool_is_running(self):
-        """Check if the pool is in the run state.
-
-        :returns: ``True`` if the pool is running.
-
-        """
-        return self._pool._state == POOL_STATE_RUN
+    def replace_dead_workers(self):
+        self.logger.debug("TaskPool: Finding dead pool processes...")
+        dead_count = self._pool.replace_dead_workers()
+        if dead_count:
+            self.logger.info(
+                "TaskPool: Replaced %d dead pool workers..." % (
+                    dead_count))
 
     def apply_async(self, target, args=None, kwargs=None, callbacks=None,
-            errbacks=None, on_acknowledge=None, meta=None):
+            errbacks=None, on_ack=None, meta=None):
         """Equivalent of the :func:``apply`` built-in function.
 
         All ``callbacks`` and ``errbacks`` should complete immediately since
@@ -95,106 +234,35 @@ class TaskPool(object):
         callbacks = callbacks or []
         errbacks = errbacks or []
         meta = meta or {}
-        tid = str(uuid.uuid4())
 
-        if not self._pool_is_running():
-            self._start()
+        on_return = curry(self.on_return, callbacks, errbacks,
+                          on_ack, meta)
 
-        self._processed_total = self._process_counter.next()
 
-        on_return = lambda r: self.on_return(r, tid, callbacks, errbacks, meta)
+        self.logger.debug("TaskPool: Apply %s (args:%s kwargs:%s)" % (
+            target, args, kwargs))
 
+        self.replace_dead_workers()
 
-        if self.full():
-            self.wait_for_result()
-        result = self._pool.apply_async(target, args, kwargs,
-                                           callback=on_return)
-        if on_acknowledge:
-            on_acknowledge()
-        self.add(result, callbacks, errbacks, tid, meta)
+        return self._pool.apply_async(target, args, kwargs,
+                                        callback=on_return)
 
-        return result
-
-    def on_return(self, ret_val, tid, callbacks, errbacks, meta):
+    def on_return(self, callbacks, errbacks, on_ack, meta, ret_value):
         """What to do when the process returns."""
-        try:
-            del(self._processes[tid])
-        except KeyError:
-            pass
-        else:
-            self.on_ready(ret_val, callbacks, errbacks, meta)
-
-    def add(self, result, callbacks, errbacks, tid, meta):
-        """Add a process to the queue.
-
-        If the queue is full, it will wait for the first task to finish,
-        collects its result and remove it from the queue, so it's ready
-        to accept new processes.
 
-        :param result: A :class:`multiprocessing.AsyncResult` instance, as
-            returned by :meth:`multiprocessing.Pool.apply_async`.
-
-        :option callbacks: List of callbacks to execute if the task was
-            successful. Must have the function signature:
-            ``mycallback(result, meta)``
-
-        :option errbacks: List of errbacks to execute if the task raised
-            and exception. Must have the function signature:
-            ``myerrback(exc, meta)``.
-
-        :option tid: The tid for this task (unqiue pool id).
-
-        """
-
-        self._processes[tid] = [result, callbacks, errbacks, meta]
-
-    def full(self):
-        """Is the pool full?
-
-        :returns: ``True`` if the maximum number of concurrent processes
-            has been reached.
-
-        """
-        return len(self._processes.values()) >= self.limit
-
-    def wait_for_result(self):
-        """Waits for the first process in the pool to finish.
-
-        This operation is blocking.
-
-        """
-        while True:
-            if self.reap():
-                break
-
-    def reap(self):
-        """Reap finished tasks."""
-        self.logger.debug("Reaping processes...")
-        processes_reaped = 0
-        for process_no, entry in enumerate(self._processes.items()):
-            tid, process_info = entry
-            result, callbacks, errbacks, meta = process_info
-            try:
-                ret_value = result.get(timeout=0.3)
-            except multiprocessing.TimeoutError:
-                continue
-            else:
-                self.on_return(ret_value, tid, callbacks, errbacks, meta)
-                processes_reaped += 1
-        return processes_reaped
+        # Acknowledge the task as being processed.
+        if on_ack:
+            on_ack()
 
-    def get_worker_pids(self):
-        """Returns the process id's of all the pool workers."""
-        return [process.pid for process in self._pool._pool]
+        self.on_ready(callbacks, errbacks, meta, ret_value)
 
-    def on_ready(self, ret_value, callbacks, errbacks, meta):
+    def on_ready(self, callbacks, errbacks, meta, ret_value):
         """What to do when a worker task is ready and its return value has
         been collected."""
 
         if isinstance(ret_value, ExceptionInfo):
-            if isinstance(ret_value.exception, KeyboardInterrupt) or \
-                    isinstance(ret_value.exception, SystemExit):
-                self.terminate()
+            if isinstance(ret_value.exception, (
+                    SystemExit, KeyboardInterrupt)):
                 raise ret_value.exception
             for errback in errbacks:
                 errback(ret_value, meta)

celery/registry.py

@@ -1,5 +1,6 @@
 """celery.registry"""
 from celery import discovery
+from celery.utils import get_full_cls_name
 from UserDict import UserDict
 
 
@@ -38,7 +39,8 @@ class TaskRegistry(UserDict):
 
         """
         is_class = hasattr(task, "run")
-
+        if is_class:
+            task = task() # instantiate Task class
         if not name:
             name = getattr(task, "name")
 
@@ -46,12 +48,11 @@ class TaskRegistry(UserDict):
             raise self.AlreadyRegistered(
                     "Task with name %s is already registered." % name)
 
-        if is_class:
-            self.data[name] = task() # instantiate Task class
-        else:
+        if not is_class:
             task.name = name
             task.type = "regular"
-            self.data[name] = task
+
+        self.data[name] = task
 
     def unregister(self, name):
         """Unregister task by name.
@@ -75,9 +76,9 @@ class TaskRegistry(UserDict):
 
     def filter_types(self, type):
         """Return all tasks of a specific type."""
-        return dict([(task_name, task)
+        return dict((task_name, task)
                         for task_name, task in self.data.items()
-                            if task.type == type])
+                            if task.type == type)
 
     def get_all_regular(self):
         """Get all regular task types."""

celery/result.py

@@ -5,8 +5,12 @@ Asynchronous result types.
 """
 from celery.backends import default_backend
 from celery.datastructures import PositionQueue
-from celery.timer import TimeoutTimer
 from itertools import imap
+import time
+
+
+class TimeoutError(Exception):
+    """The operation timed out."""
 
 
 class BaseAsyncResult(object):
@@ -28,6 +32,8 @@ class BaseAsyncResult(object):
 
     """
 
+    TimeoutError = TimeoutError
+
     def __init__(self, task_id, backend):
         self.task_id = task_id
         self.backend = backend
@@ -50,7 +56,7 @@ class BaseAsyncResult(object):
         :keyword timeout: How long to wait in seconds, before the
             operation times out.
 
-        :raises celery.timer.TimeoutError: if ``timeout`` is not ``None`` and
+        :raises TimeoutError: if ``timeout`` is not ``None`` and
             the result does not arrive within ``timeout`` seconds.
 
         If the remote call raised an exception then that
@@ -235,8 +241,8 @@ class TaskSetResult(object):
         :raises: The exception if any of the tasks raised an exception.
 
         """
-        results = dict([(subtask.task_id, AsyncResult(subtask.task_id))
-                            for subtask in self.subtasks])
+        results = dict((subtask.task_id, AsyncResult(subtask.task_id))
+                            for subtask in self.subtasks)
         while results:
             for task_id, pending_result in results.items():
                 if pending_result.status == "DONE":
@@ -253,7 +259,7 @@ class TaskSetResult(object):
         :keyword timeout: The time in seconds, how long
             it will wait for results, before the operation times out.
 
-        :raises celery.timer.TimeoutError: if ``timeout`` is not ``None``
+        :raises TimeoutError: if ``timeout`` is not ``None``
             and the operation takes longer than ``timeout`` seconds.
 
         If any of the tasks raises an exception, the exception
@@ -262,7 +268,12 @@ class TaskSetResult(object):
         :returns: list of return values for all tasks in the taskset.
 
         """
-        timeout_timer = TimeoutTimer(timeout)
+
+        time_start = time.time()
+
+        def on_timeout():
+            raise TimeoutError("The operation timed out.")
+
         results = PositionQueue(length=self.total)
 
         while True:
@@ -275,11 +286,49 @@ class TaskSetResult(object):
                 # Make list copy, so the returned type is not a position
                 # queue.
                 return list(results)
-
-            # This raises TimeoutError when timed out.
-            timeout_timer.tick()
+            else:
+                if time.time() >= time_start + timeout:
+                    on_timeout()
 
     @property
     def total(self):
         """The total number of tasks in the :class:`celery.task.TaskSet`."""
         return len(self.subtasks)
+
+
+class EagerResult(BaseAsyncResult):
+    """Result that we know has already been executed.  """
+    TimeoutError = TimeoutError
+
+    def __init__(self, task_id, ret_value, status):
+        self.task_id = task_id
+        self._result = ret_value
+        self._status = status
+
+    def is_done(self):
+        """Returns ``True`` if the task executed without failure."""
+        return self.status == "DONE"
+
+    def is_ready(self):
+        """Returns ``True`` if the task has been executed."""
+        return True
+
+    def wait(self, timeout=None):
+        """Wait until the task has been executed and return its result."""
+        if self.status == "DONE":
+            return self.result
+        elif self.status == "FAILURE":
+            raise self.result
+
+    @property
+    def result(self):
+        """The tasks return value"""
+        return self._result
+
+    @property
+    def status(self):
+        """The tasks status"""
+        return self._status
+
+    def __repr__(self):
+        return "<EagerResult: %s>" % self.task_id

celery/serialization.py

@@ -0,0 +1,4 @@
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle

celery/supervisor.py

@@ -0,0 +1,120 @@
+import multiprocessing
+import time
+from multiprocessing import TimeoutError
+
+JOIN_TIMEOUT = 2
+CHECK_INTERVAL = 2
+MAX_RESTART_FREQ = 3
+MAX_RESTART_FREQ_TIME = 10
+
+
+class MaxRestartsExceededError(Exception):
+    """Restarts exceeded the maximum restart frequency."""
+
+
+class OFASupervisor(object):
+    """Process supervisor using the `one_for_all`_ strategy.
+
+    .. _`one_for_all`:
+        http://erlang.org/doc/design_principles/sup_princ.html#5.3.2
+
+    However, instead of registering a list of processes, you have one
+    process which runs a pool. Makes for an easy implementation.
+
+    :param target: see :attr:`target`.
+    :param args: see :attr:`args`.
+    :param kwargs: see :attr:`kwargs`.
+    :param max_restart_freq: see :attr:`max_restart_freq`.
+    :param max_restart_freq_time: see :attr:`max_restart_freq_time`.
+    :param check_interval: see :attr:`max_restart_freq_time`.
+
+    .. attribute:: target
+
+        The target callable to be launched in a new process.
+
+    .. attribute:: args
+
+        The positional arguments to apply to :attr:`target`.
+
+    .. attribute:: kwargs
+
+        The keyword arguments to apply to :attr:`target`.
+
+    .. attribute:: max_restart_freq
+
+        Limit the number of restarts which can occur in a given time interval.
+
+        The max restart frequency is the number of restarts that can occur
+        within the interval :attr:`max_restart_freq_time`.
+
+        The restart mechanism prevents situations where the process repeatedly
+        dies for the same reason. If this happens both the process and the
+        supervisor is terminated.
+
+    .. attribute:: max_restart_freq_time
+
+        See :attr:`max_restart_freq`.
+
+    .. attribute:: check_interval
+
+        The time in seconds, between process pings.
+
+    """
+    Process = multiprocessing.Process
+
+    def __init__(self, target, args=None, kwargs=None,
+            max_restart_freq=MAX_RESTART_FREQ,
+            join_timeout=JOIN_TIMEOUT,
+            max_restart_freq_time=MAX_RESTART_FREQ_TIME,
+            check_interval=CHECK_INTERVAL):
+        self.target = target
+        self.join_timeout = join_timeout
+        self.args = args or []
+        self.kwargs = kwargs or {}
+        self.check_interval = check_interval
+        self.max_restart_freq = max_restart_freq
+        self.max_restart_freq_time = max_restart_freq_time
+        self.restarts_in_frame = 0
+
+    def start(self):
+        """Launches the :attr:`target` in a seperate process and starts
+        supervising it."""
+        target = self.target
+
+        def _start_supervised_process():
+            """Start the :attr:`target` in a new process."""
+            process = self.Process(target=target,
+                                   args=self.args, kwargs=self.kwargs)
+            process.start()
+            return process
+
+        def _restart(process):
+            """Terminate the process and restart."""
+            process.join(timeout=self.join_timeout)
+            process.terminate()
+            self.restarts_in_frame += 1
+            process = _start_supervised_process()
+
+        process = _start_supervised_process()
+        try:
+            restart_frame = 0
+            while True:
+                if restart_frame > self.max_restart_freq_time:
+                    if self.restarts_in_frame >= self.max_restart_freq:
+                        raise MaxRestartsExceededError(
+                                "Supervised: Max restart frequency reached")
+                restart_frame = 0
+                self.restarts_in_frame = 0
+
+                try:
+                    proc_is_alive = process.is_alive()
+                except TimeoutError:
+                    proc_is_alive = False
+
+                if not proc_is_alive:
+                    _restart(process)
+
+                time.sleep(self.check_interval)
+                restart_frame += self.check_interval
+        finally:
+            process.join()

celery/task/__init__.py

@@ -0,0 +1,109 @@
+"""
+
+Working with tasks and task sets.
+
+"""
+from carrot.connection import DjangoAMQPConnection
+from celery.messaging import TaskConsumer
+from celery.conf import AMQP_CONNECTION_TIMEOUT
+from celery.registry import tasks
+from celery.backends import default_backend
+from celery.task.base import Task, TaskSet, PeriodicTask
+from celery.task.base import ExecuteRemoteTask
+from celery.task.base import AsynchronousMapTask
+from celery.task.builtins import DeleteExpiredTaskMetaTask, PingTask
+from celery.execute import apply_async, delay_task
+from celery.serialization import pickle
+
+
+def discard_all(connect_timeout=AMQP_CONNECTION_TIMEOUT):
+    """Discard all waiting tasks.
+
+    This will ignore all tasks waiting for execution, and they will
+    be deleted from the messaging server.
+
+    :returns: the number of tasks discarded.
+
+    :rtype: int
+
+    """
+    amqp_connection = DjangoAMQPConnection(connect_timeout=connect_timeout)
+    consumer = TaskConsumer(connection=amqp_connection)
+    discarded_count = consumer.discard_all()
+    amqp_connection.close()
+    return discarded_count
+
+
+def is_done(task_id):
+    """Returns ``True`` if task with ``task_id`` has been executed.
+
+    :rtype: bool
+
+    """
+    return default_backend.is_done(task_id)
+
+
+def dmap(func, args, timeout=None):
+    """Distribute processing of the arguments and collect the results.
+
+    Example
+
+        >>> from celery.task import map
+        >>> import operator
+        >>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]])
+        [4, 8, 16]
+
+    """
+    return TaskSet.map(func, args, timeout=timeout)
+
+
+def dmap_async(func, args, timeout=None):
+    """Distribute processing of the arguments and collect the results
+    asynchronously.
+
+    :returns: :class:`celery.result.AsyncResult` object.
+
+    Example
+
+        >>> from celery.task import dmap_async
+        >>> import operator
+        >>> presult = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]])
+        >>> presult
+        <AsyncResult: 373550e8-b9a0-4666-bc61-ace01fa4f91d>
+        >>> presult.status
+        'DONE'
+        >>> presult.result
+        [4, 8, 16]
+
+    """
+    return TaskSet.map_async(func, args, timeout=timeout)
+
+
+def execute_remote(func, *args, **kwargs):
+    """Execute arbitrary function/object remotely.
+
+    :param func: A callable function or object.
+
+    :param \*args: Positional arguments to apply to the function.
+
+    :param \*\*kwargs: Keyword arguments to apply to the function.
+
+    The object must be picklable, so you can't use lambdas or functions
+    defined in the REPL (the objects must have an associated module).
+
+    :returns: class:`celery.result.AsyncResult`.
+
+    """
+    return ExecuteRemoteTask.delay(pickle.dumps(func), args, kwargs)
+
+
+def ping():
+    """Test if the server is alive.
+
+    Example:
+
+        >>> from celery.task import ping
+        >>> ping()
+        'pong'
+    """
+    return PingTask.apply_async().get()

celery/task.py → celery/task/base.py

@@ -1,142 +1,13 @@
-"""
-
-Working with tasks and task sets.
-
-"""
 from carrot.connection import DjangoAMQPConnection
 from celery.conf import AMQP_CONNECTION_TIMEOUT
-from celery.conf import STATISTICS_COLLECT_INTERVAL
 from celery.messaging import TaskPublisher, TaskConsumer
 from celery.log import setup_logger
-from celery.registry import tasks
+from celery.result import TaskSetResult
+from celery.execute import apply_async, delay_task, apply
+from celery.utils import gen_unique_id, get_full_cls_name
 from datetime import timedelta
-from celery.backends import default_backend
-from celery.result import AsyncResult, TaskSetResult
-from django.utils.functional import curry
-import uuid
-import pickle
-
-
-def apply_async(task, args=None, kwargs=None, routing_key=None,
-        immediate=None, mandatory=None, connection=None,
-        connect_timeout=AMQP_CONNECTION_TIMEOUT, priority=None, 
-        exchange=None, **opts):
-    """Run a task asynchronously by the celery daemon(s).
-
-    :param task: The task to run (a callable object, or a :class:`Task`
-        instance
-
-    :param args: The positional arguments to pass on to the task (a ``list``).
-
-    :param kwargs: The keyword arguments to pass on to the task (a ``dict``)
-
-
-    :keyword routing_key: The routing key used to route the task to a worker
-        server.
-
-    :keyword immediate: Request immediate delivery. Will raise an exception
-        if the task cannot be routed to a worker immediately.
-
-    :keyword mandatory: Mandatory routing. Raises an exception if there's
-        no running workers able to take on this task.
-
-    :keyword connection: Re-use existing AMQP connection.
-        The ``connect_timeout`` argument is not respected if this is set.
-
-    :keyword connect_timeout: The timeout in seconds, before we give up
-        on establishing a connection to the AMQP server.
-
-    :keyword priority: The task priority, a number between ``0`` and ``9``.
-
-    """
-    args = args or []
-    kwargs = kwargs or {}
-    exchange = exchange or getattr(task, "exchange", None)
-    routing_key = routing_key or getattr(task, "routing_key", None)
-    immediate = immediate or getattr(task, "immediate", None)
-    mandatory = mandatory or getattr(task, "mandatory", None)
-    priority = priority or getattr(task, "priority", None)
-    taskset_id = opts.get("taskset_id")
-    publisher = opts.get("publisher")
-
-    need_to_close_connection = False
-    if not publisher:
-        if not connection:
-            connection = DjangoAMQPConnection(connect_timeout=connect_timeout)
-            need_to_close_connection = True
-        publisher = TaskPublisher(connection=connection, exchange=exchange)
-
-    delay_task = publisher.delay_task
-    if taskset_id:
-        delay_task = curry(publisher.delay_task_in_set, taskset_id)
-        
-    task_id = delay_task(task.name, args, kwargs,
-                         routing_key=routing_key, mandatory=mandatory,
-                         immediate=immediate, priority=priority)
-
-    if need_to_close_connection:
-        publisher.close()
-        connection.close()
-
-    return AsyncResult(task_id)
-
-
-def delay_task(task_name, *args, **kwargs):
-    """Delay a task for execution by the ``celery`` daemon.
-
-    :param task_name: the name of a task registered in the task registry.
-
-    :param \*args: positional arguments to pass on to the task.
-
-    :param \*\*kwargs: keyword arguments to pass on to the task.
-
-    :raises celery.registry.NotRegistered: exception if no such task
-        has been registered in the task registry.
-
-    :rtype: :class:`celery.result.AsyncResult`.
-
-    Example
-
-        >>> r = delay_task("update_record", name="George Constanza", age=32)
-        >>> r.ready()
-        True
-        >>> r.result
-        "Record was updated"
-
-    """
-    if task_name not in tasks:
-        raise tasks.NotRegistered(
-                "Task with name %s not registered in the task registry." % (
-                    task_name))
-    task = tasks[task_name]
-    return apply_async(task, args, kwargs)
-
-
-def discard_all(connect_timeout=AMQP_CONNECTION_TIMEOUT):
-    """Discard all waiting tasks.
-
-    This will ignore all tasks waiting for execution, and they will
-    be deleted from the messaging server.
-
-    :returns: the number of tasks discarded.
-
-    :rtype: int
-
-    """
-    amqp_connection = DjangoAMQPConnection(connect_timeout=connect_timeout)
-    consumer = TaskConsumer(connection=amqp_connection)
-    discarded_count = consumer.discard_all()
-    amqp_connection.close()
-    return discarded_count
-
-
-def is_done(task_id):
-    """Returns ``True`` if task with ``task_id`` has been executed.
-
-    :rtype: bool
-
-    """
-    return default_backend.is_done(task_id)
+from celery.registry import tasks
+from celery.serialization import pickle
 
 
 class Task(object):
@@ -171,7 +42,7 @@ class Task(object):
         instead.
 
     .. attribute:: immediate:
-            
+
         Request immediate delivery. If the message cannot be routed to a
         task worker immediately, an exception will be raised. This is
         instead of the default behaviour, where the broker will accept and
@@ -179,7 +50,7 @@ class Task(object):
         be consumed.
 
     .. attribute:: priority:
-    
+
         The message priority. A number from ``0`` to ``9``.
 
     .. attribute:: ignore_result
@@ -242,8 +113,8 @@ class Task(object):
     disable_error_emails = False
 
     def __init__(self):
-        if not self.name:
-            raise NotImplementedError("Tasks must define a name attribute.")
+        if not self.__class__.name:
+            self.__class__.name = get_full_cls_name(self.__class__)
 
     def __call__(self, *args, **kwargs):
         return self.run(*args, **kwargs)
@@ -313,7 +184,7 @@ class Task(object):
 
         :rtype: :class:`celery.result.AsyncResult`
 
-        See :func:`delay_task`.
+        See :func:`celery.execute.delay_task`.
 
         """
         return apply_async(cls, args, kwargs)
@@ -328,11 +199,65 @@ class Task(object):
 
         :rtype: :class:`celery.result.AsyncResult`
 
-        See :func:`apply_async`.
+        See :func:`celery.execute.apply_async`.
 
         """
         return apply_async(cls, args, kwargs, **options)
 
+    @classmethod
+    def apply(cls, args=None, kwargs=None, **options):
+        """Execute this task at once, by blocking until the task
+        has finished executing.
+
+        :param args: positional arguments passed on to the task.
+
+        :param kwargs: keyword arguments passed on to the task.
+
+        :rtype: :class:`celery.result.EagerResult`
+
+        See :func:`celery.execute.apply`.
+
+        """
+        return apply(cls, args, kwargs, **options)
+
+
+class ExecuteRemoteTask(Task):
+    """Execute an arbitrary function or object.
+
+    *Note* You probably want :func:`execute_remote` instead, which this
+    is an internal component of.
+
+    The object must be pickleable, so you can't use lambdas or functions
+    defined in the REPL (that is the python shell, or ``ipython``).
+
+    """
+    name = "celery.execute_remote"
+
+    def run(self, ser_callable, fargs, fkwargs, **kwargs):
+        """
+        :param ser_callable: A pickled function or callable object.
+
+        :param fargs: Positional arguments to apply to the function.
+
+        :param fkwargs: Keyword arguments to apply to the function.
+
+        """
+        callable_ = pickle.loads(ser_callable)
+        return callable_(*fargs, **fkwargs)
+tasks.register(ExecuteRemoteTask)
+
+
+class AsynchronousMapTask(Task):
+    """Task used internally by :func:`dmap_async` and
+    :meth:`TaskSet.map_async`.  """
+    name = "celery.map_async"
+
+    def run(self, serfunc, args, **kwargs):
+        """The method run by ``celeryd``."""
+        timeout = kwargs.get("timeout")
+        return TaskSet.map(pickle.loads(serfunc), args, timeout=timeout)
+tasks.register(AsynchronousMapTask)
+
 
 class TaskSet(object):
     """A task containing several subtasks, making it possible
@@ -414,7 +339,14 @@ class TaskSet(object):
             [True, True]
 
         """
-        taskset_id = str(uuid.uuid4())
+        taskset_id = gen_unique_id()
+
+        from celery.conf import ALWAYS_EAGER
+        if ALWAYS_EAGER:
+            subtasks = [apply(self.task, args, kwargs)
+                            for args, kwargs in self.arguments]
+            return TaskSetResult(taskset_id, subtasks)
+
         conn = DjangoAMQPConnection(connect_timeout=connect_timeout)
         publisher = TaskPublisher(connection=conn,
                                   exchange=self.task.exchange)
@@ -425,15 +357,6 @@ class TaskSet(object):
         conn.close()
         return TaskSetResult(taskset_id, subtasks)
 
-    def iterate(self):
-        """Iterate over the results returned after calling :meth:`run`.
-
-        If any of the tasks raises an exception, the exception will
-        be re-raised.
-
-        """
-        return iter(self.run())
-
     def join(self, timeout=None):
         """Gather the results for all of the tasks in the taskset,
         and return a list with them ordered by the order of which they
@@ -442,7 +365,7 @@ class TaskSet(object):
         :keyword timeout: The time in seconds, how long
             it will wait for results, before the operation times out.
 
-        :raises celery.timer.TimeoutError: if ``timeout`` is not ``None``
+        :raises TimeoutError: if ``timeout`` is not ``None``
             and the operation takes longer than ``timeout`` seconds.
 
         If any of the tasks raises an exception, the exception
@@ -479,54 +402,6 @@ class TaskSet(object):
         return AsynchronousMapTask.delay(serfunc, args, timeout=timeout)
 
 
-def dmap(func, args, timeout=None):
-    """Distribute processing of the arguments and collect the results.
-
-    Example
-
-        >>> from celery.task import map
-        >>> import operator
-        >>> dmap(operator.add, [[2, 2], [4, 4], [8, 8]])
-        [4, 8, 16]
-
-    """
-    return TaskSet.map(func, args, timeout=timeout)
-
-
-class AsynchronousMapTask(Task):
-    """Task used internally by :func:`dmap_async` and
-    :meth:`TaskSet.map_async`.  """
-    name = "celery.map_async"
-
-    def run(self, serfunc, args, **kwargs):
-        """The method run by ``celeryd``."""
-        timeout = kwargs.get("timeout")
-        return TaskSet.map(pickle.loads(serfunc), args, timeout=timeout)
-tasks.register(AsynchronousMapTask)
-
-
-def dmap_async(func, args, timeout=None):
-    """Distribute processing of the arguments and collect the results
-    asynchronously.
-
-    :returns: :class:`celery.result.AsyncResult` object.
-
-    Example
-
-        >>> from celery.task import dmap_async
-        >>> import operator
-        >>> presult = dmap_async(operator.add, [[2, 2], [4, 4], [8, 8]])
-        >>> presult
-        <AsyncResult: 373550e8-b9a0-4666-bc61-ace01fa4f91d>
-        >>> presult.status
-        'DONE'
-        >>> presult.result
-        [4, 8, 16]
-
-    """
-    return TaskSet.map_async(func, args, timeout=timeout)
-
-
 class PeriodicTask(Task):
     """A periodic task is a task that behaves like a :manpage:`cron` job.
 
@@ -568,87 +443,3 @@ class PeriodicTask(Task):
             self.run_every = timedelta(seconds=self.run_every)
 
         super(PeriodicTask, self).__init__()
-
-
-class ExecuteRemoteTask(Task):
-    """Execute an arbitrary function or object.
-
-    *Note* You probably want :func:`execute_remote` instead, which this
-    is an internal component of.
-
-    The object must be pickleable, so you can't use lambdas or functions
-    defined in the REPL (that is the python shell, or ``ipython``).
-
-    """
-    name = "celery.execute_remote"
-
-    def run(self, ser_callable, fargs, fkwargs, **kwargs):
-        """
-        :param ser_callable: A pickled function or callable object.
-
-        :param fargs: Positional arguments to apply to the function.
-
-        :param fkwargs: Keyword arguments to apply to the function.
-
-        """
-        callable_ = pickle.loads(ser_callable)
-        return callable_(*fargs, **fkwargs)
-tasks.register(ExecuteRemoteTask)
-
-
-def execute_remote(func, *args, **kwargs):
-    """Execute arbitrary function/object remotely.
-
-    :param func: A callable function or object.
-
-    :param \*args: Positional arguments to apply to the function.
-
-    :param \*\*kwargs: Keyword arguments to apply to the function.
-
-    The object must be picklable, so you can't use lambdas or functions
-    defined in the REPL (the objects must have an associated module).
-
-    :returns: class:`celery.result.AsyncResult`.
-
-    """
-    return ExecuteRemoteTask.delay(pickle.dumps(func), args, kwargs)
-
-
-class DeleteExpiredTaskMetaTask(PeriodicTask):
-    """A periodic task that deletes expired task metadata every day.
-
-    This runs the current backend's
-    :meth:`celery.backends.base.BaseBackend.cleanup` method.
-
-    """
-    name = "celery.delete_expired_task_meta"
-    run_every = timedelta(days=1)
-
-    def run(self, **kwargs):
-        """The method run by ``celeryd``."""
-        logger = self.get_logger(**kwargs)
-        logger.info("Deleting expired task meta objects...")
-        default_backend.cleanup()
-tasks.register(DeleteExpiredTaskMetaTask)
-
-
-class PingTask(Task):
-    """The task used by :func:`ping`."""
-    name = "celery.ping"
-
-    def run(self, **kwargs):
-        """:returns: the string ``"pong"``."""
-        return "pong"
-tasks.register(PingTask)
-
-
-def ping():
-    """Test if the server is alive.
-
-    Example:
-
-        >>> from celery.task import ping
-        >>> ping()
-        'pong'
-    """
-    return PingTask.apply_async().get()

celery/task/builtins.py

@@ -0,0 +1,33 @@
+from celery.task.base import Task, TaskSet, PeriodicTask
+from celery.registry import tasks
+from celery.backends import default_backend
+from datetime import timedelta
+from celery.serialization import pickle
+
+
+class DeleteExpiredTaskMetaTask(PeriodicTask):
+    """A periodic task that deletes expired task metadata every day.
+
+    This runs the current backend's
+    :meth:`celery.backends.base.BaseBackend.cleanup` method.
+
+    """
+    name = "celery.delete_expired_task_meta"
+    run_every = timedelta(days=1)
+
+    def run(self, **kwargs):
+        """The method run by ``celeryd``."""
+        logger = self.get_logger(**kwargs)
+        logger.info("Deleting expired task meta objects...")
+        default_backend.cleanup()
+tasks.register(DeleteExpiredTaskMetaTask)
+
+
+class PingTask(Task):
+    """The task used by :func:`ping`."""
+    name = "celery.ping"
+
+    def run(self, **kwargs):
+        """:returns: the string ``"pong"``."""
+        return "pong"
+tasks.register(PingTask)

celery/task/strategy.py

@@ -0,0 +1,52 @@
+from carrot.connection import DjangoAMQPConnection
+from celery.utils import chunks
+
+
+def even_time_distribution(task, size, time_window, iterable, **apply_kwargs):
+    """With an iterator yielding task args, kwargs tuples, evenly distribute
+    the processing of its tasks throughout the time window available.
+
+    :param task: The kind of task (a :class:`celery.task.base.Task`.)
+    :param size: Total number of elements the iterator gives.
+    :param time_window: Total time available, in minutes.
+    :param iterable: Iterable yielding task args, kwargs tuples.
+    :param \*\*apply_kwargs: Additional keyword arguments to be passed on to
+        :func:`celery.execute.apply_async`.
+
+    Example
+
+        >>> class RefreshAllFeeds(Task):
+        ...
+        ...     def run(self, **kwargs):
+        ...         feeds = Feed.objects.all()
+        ...         total = feeds.count()
+        ...
+        ...         time_window = REFRESH_FEEDS_EVERY_INTERVAL_MINUTES
+        ...
+        ...         def iter_feed_task_args(iterable):
+        ...             for feed in iterable:
+        ...                 yield ([feed.feed_url], {}) # args, kwargs tuple
+        ...
+        ...         it = iter_feed_task_args(feeds.iterator())
+        ...
+        ...         even_time_distribution(RefreshFeedTask, total,
+        ...                                time_window, it)
+
+    """
+
+    bucketsize = size / time_window
+    buckets = chunks(iterable, int(bucketsize))
+
+    connection = DjangoAMQPConnection()
+    try:
+        for bucket_count, bucket in enumerate(buckets):
+            # Skew the countdown for items in this bucket by one.
+            seconds_eta = (60 * bucket_count if bucket_count else None)
+
+            for args, kwargs in bucket:
+                task.apply_async(args=args, kwargs=kwargs,
+                                 connection=connection,
+                                 countdown=seconds_eta,
+                                 **apply_kwargs)
+    finally:
+        connection.close()

celery/tests/test_backends/test_base.py

@@ -0,0 +1,83 @@
+import unittest
+import types
+from celery.backends.base import find_nearest_pickleable_exception as fnpe
+from celery.backends.base import BaseBackend, KeyValueStoreBackend
+from celery.backends.base import UnpickleableExceptionWrapper
+from django.db.models.base import subclass_exception
+
+
+class wrapobject(object):
+
+    def __init__(self, *args, **kwargs):
+        self.args = args
+
+
+Oldstyle = types.ClassType("Oldstyle", (), {})
+Unpickleable = subclass_exception("Unpickleable", KeyError, "foo.module")
+Impossible = subclass_exception("Impossible", object, "foo.module")
+Lookalike = subclass_exception("Lookalike", wrapobject, "foo.module")
+b = BaseBackend()
+
+
+class TestBaseBackendInterface(unittest.TestCase):
+
+    def test_get_status(self):
+        self.assertRaises(NotImplementedError,
+                b.is_done, "SOMExx-N0Nex1stant-IDxx-")
+
+    def test_store_result(self):
+        self.assertRaises(NotImplementedError,
+                b.store_result, "SOMExx-N0nex1stant-IDxx-", 42, "DONE")
+
+    def test_get_result(self):
+        self.assertRaises(NotImplementedError,
+                b.get_result, "SOMExx-N0nex1stant-IDxx-")
+
+
+class TestPickleException(unittest.TestCase):
+
+    def test_oldstyle(self):
+        self.assertTrue(fnpe(Oldstyle()) is None)
+
+    def test_BaseException(self):
+        self.assertTrue(fnpe(Exception()) is None)
+
+    def test_unpickleable(self):
+        self.assertTrue(isinstance(fnpe(Unpickleable()), KeyError))
+        self.assertEquals(fnpe(Impossible()), None)
+
+
+class TestPrepareException(unittest.TestCase):
+
+    def test_unpickleable(self):
+        x = b.prepare_exception(Unpickleable(1, 2, "foo"))
+        self.assertTrue(isinstance(x, KeyError))
+        y = b.exception_to_python(x)
+        self.assertTrue(isinstance(y, KeyError))
+
+    def test_impossible(self):
+        x = b.prepare_exception(Impossible())
+        self.assertTrue(isinstance(x, UnpickleableExceptionWrapper))
+        y = b.exception_to_python(x)
+        self.assertEquals(y.__class__.__name__, "Impossible")
+        self.assertEquals(y.__class__.__module__, "foo.module")
+
+    def test_regular(self):
+        x = b.prepare_exception(KeyError("baz"))
+        self.assertTrue(isinstance(x, KeyError))
+        y = b.exception_to_python(x)
+        self.assertTrue(isinstance(y, KeyError))
+
+
+class TestKeyValueStoreBackendInterface(unittest.TestCase):
+
+    def test_get(self):
+        self.assertRaises(NotImplementedError, KeyValueStoreBackend().get,
+                "a")
+
+    def test_set(self):
+        self.assertRaises(NotImplementedError, KeyValueStoreBackend().set,
+                "a", 1)
+
+    def test_cleanup(self):
+        self.assertFalse(KeyValueStoreBackend().cleanup())

celery/tests/test_backends/test_cache.py

@@ -0,0 +1,61 @@
+import sys
+import unittest
+import errno
+import socket
+from celery.backends.cache import Backend as CacheBackend
+from celery.utils import gen_unique_id
+from django.conf import settings
+
+
+class SomeClass(object):
+
+    def __init__(self, data):
+        self.data = data
+
+
+class TestCacheBackend(unittest.TestCase):
+
+    def test_mark_as_done(self):
+        cb = CacheBackend()
+
+        tid = gen_unique_id()
+
+        self.assertFalse(cb.is_done(tid))
+        self.assertEquals(cb.get_status(tid), "PENDING")
+        self.assertEquals(cb.get_result(tid), None)
+
+        cb.mark_as_done(tid, 42)
+        self.assertTrue(cb.is_done(tid))
+        self.assertEquals(cb.get_status(tid), "DONE")
+        self.assertEquals(cb.get_result(tid), 42)
+        self.assertTrue(cb._cache.get(tid))
+        self.assertTrue(cb.get_result(tid), 42)
+
+    def test_is_pickled(self):
+        cb = CacheBackend()
+
+        tid2 = gen_unique_id()
+        result = {"foo": "baz", "bar": SomeClass(12345)}
+        cb.mark_as_done(tid2, result)
+        # is serialized properly.
+        rindb = cb.get_result(tid2)
+        self.assertEquals(rindb.get("foo"), "baz")
+        self.assertEquals(rindb.get("bar").data, 12345)
+
+    def test_mark_as_failure(self):
+        cb = CacheBackend()
+
+        tid3 = gen_unique_id()
+        try:
+            raise KeyError("foo")
+        except KeyError, exception:
+            pass
+        cb.mark_as_failure(tid3, exception)
+        self.assertFalse(cb.is_done(tid3))
+        self.assertEquals(cb.get_status(tid3), "FAILURE")
+        self.assertTrue(isinstance(cb.get_result(tid3), KeyError))
+
+    def test_process_cleanup(self):
+        cb = CacheBackend()
+
+        cb.process_cleanup()

celery/tests/test_backends/test_database.py

@@ -1,6 +1,10 @@
 import unittest
 from celery.backends.database import Backend
-import uuid
+from celery.utils import gen_unique_id
+from celery.task import PeriodicTask
+from celery import registry
+from celery.models import PeriodicTaskMeta
+from datetime import datetime, timedelta
 
 
 class SomeClass(object):
@@ -9,11 +13,31 @@ class SomeClass(object):
         self.data = data
 
 
+class MyPeriodicTask(PeriodicTask):
+    name = "c.u.my-periodic-task-244"
+    run_every = timedelta(seconds=1)
+
+    def run(self, **kwargs):
+        return 42
+registry.tasks.register(MyPeriodicTask)
+
+
 class TestDatabaseBackend(unittest.TestCase):
 
+    def test_run_periodic_tasks(self):
+        #obj, created = PeriodicTaskMeta.objects.get_or_create(
+        #                    name=MyPeriodicTask.name,
+        #                    defaults={"last_run_at": datetime.now() -
+        #                        timedelta(days=-4)})
+        #if not created:
+        #    obj.last_run_at = datetime.now() - timedelta(days=4)
+        #    obj.save()
+        b = Backend()
+        b.run_periodic_tasks()
+
     def test_backend(self):
         b = Backend()
-        tid = str(uuid.uuid4())
+        tid = gen_unique_id()
 
         self.assertFalse(b.is_done(tid))
         self.assertEquals(b.get_status(tid), "PENDING")
@@ -26,7 +50,7 @@ class TestDatabaseBackend(unittest.TestCase):
         self.assertTrue(b._cache.get(tid))
         self.assertTrue(b.get_result(tid), 42)
 
-        tid2 = str(uuid.uuid4())
+        tid2 = gen_unique_id()
         result = {"foo": "baz", "bar": SomeClass(12345)}
         b.mark_as_done(tid2, result)
         # is serialized properly.
@@ -34,7 +58,7 @@ class TestDatabaseBackend(unittest.TestCase):
         self.assertEquals(rindb.get("foo"), "baz")
         self.assertEquals(rindb.get("bar").data, 12345)
 
-        tid3 = str(uuid.uuid4())
+        tid3 = gen_unique_id()
         try:
             raise KeyError("foo")
         except KeyError, exception:

celery/tests/test_backends/test_tyrant.py

@@ -0,0 +1,103 @@
+import sys
+import unittest
+import errno
+import socket
+from celery.backends.tyrant import Backend as TyrantBackend
+from django.conf import settings
+from celery.utils import gen_unique_id
+from django.core.exceptions import ImproperlyConfigured
+
+_no_tyrant_msg = "* Tokyo Tyrant not running. Will not execute related tests."
+_no_tyrant_msg_emitted = False
+
+
+class SomeClass(object):
+
+    def __init__(self, data):
+        self.data = data
+
+
+def get_tyrant_or_None():
+    try:
+        tb = TyrantBackend()
+        try:
+            tb.open()
+        except socket.error, exc:
+            if exc.errno == errno.ECONNREFUSED:
+                if not _no_tyrant_msg_emitted:
+                    sys.stderr.write("\n" + _no_tyrant_msg + "\n")
+                return None
+            else:
+                raise
+        return tb
+    except ImproperlyConfigured, exc:
+        return None
+
+
+class TestTyrantBackend(unittest.TestCase):
+
+    def test_cached_connection(self):
+        tb = get_tyrant_or_None()
+        if not tb:
+            return # Skip test
+
+        self.assertTrue(tb._connection is not None)
+        tb.close()
+        self.assertTrue(tb._connection is None)
+        tb.open()
+        self.assertTrue(tb._connection is not None)
+
+    def test_mark_as_done(self):
+        tb = get_tyrant_or_None()
+        if not tb:
+            return
+
+        tid = gen_unique_id()
+
+        self.assertFalse(tb.is_done(tid))
+        self.assertEquals(tb.get_status(tid), "PENDING")
+        self.assertEquals(tb.get_result(tid), None)
+
+        tb.mark_as_done(tid, 42)
+        self.assertTrue(tb.is_done(tid))
+        self.assertEquals(tb.get_status(tid), "DONE")
+        self.assertEquals(tb.get_result(tid), 42)
+        self.assertTrue(tb._cache.get(tid))
+        self.assertTrue(tb.get_result(tid), 42)
+
+    def test_is_pickled(self):
+        tb = get_tyrant_or_None()
+        if not tb:
+            return
+
+        tid2 = gen_unique_id()
+        result = {"foo": "baz", "bar": SomeClass(12345)}
+        tb.mark_as_done(tid2, result)
+        # is serialized properly.
+        rindb = tb.get_result(tid2)
+        self.assertEquals(rindb.get("foo"), "baz")
+        self.assertEquals(rindb.get("bar").data, 12345)
+
+    def test_mark_as_failure(self):
+        tb = get_tyrant_or_None()
+        if not tb:
+            return
+
+        tid3 = gen_unique_id()
+        try:
+            raise KeyError("foo")
+        except KeyError, exception:
+            pass
+        tb.mark_as_failure(tid3, exception)
+        self.assertFalse(tb.is_done(tid3))
+        self.assertEquals(tb.get_status(tid3), "FAILURE")
+        self.assertTrue(isinstance(tb.get_result(tid3), KeyError))
+
+    def test_process_cleanup(self):
+        tb = get_tyrant_or_None()
+        if not tb:
+            return
+
+        tb.process_cleanup()
+
+        self.assertTrue(tb._connection is None)

celery/tests/test_celery.py

@@ -0,0 +1,21 @@
+import unittest
+import celery
+
+
+class TestInitFile(unittest.TestCase):
+
+    def test_version(self):
+        self.assertTrue(celery.VERSION)
+        self.assertEquals(len(celery.VERSION), 3)
+        celery.VERSION = (0, 3, 0)
+        self.assertFalse(celery.is_stable_release())
+        self.assertEquals(celery.__version__.count("."), 2)
+        self.assertTrue("(unstable)" in celery.version_with_meta())
+        celery.VERSION = (0, 4, 0)
+        self.assertTrue(celery.is_stable_release())
+        self.assertTrue("(stable)" in celery.version_with_meta())
+
+    def test_meta(self):
+        for m in ("__author__", "__contact__", "__homepage__",
+                "__docformat__"):
+            self.assertTrue(getattr(celery, m, None))

celery/tests/test_datastructures.py

@@ -0,0 +1,51 @@
+import unittest
+import sys
+
+from celery.datastructures import PositionQueue, ExceptionInfo
+
+
+class TestPositionQueue(unittest.TestCase):
+
+    def test_position_queue_unfilled(self):
+        q = PositionQueue(length=10)
+        for position in q.data:
+            self.assertTrue(isinstance(position, q.UnfilledPosition))
+
+        self.assertEquals(q.filled, [])
+        self.assertEquals(len(q), 0)
+        self.assertFalse(q.full())
+
+    def test_position_queue_almost(self):
+        q = PositionQueue(length=10)
+        q[3] = 3
+        q[6] = 6
+        q[9] = 9
+
+        self.assertEquals(q.filled, [3, 6, 9])
+        self.assertEquals(len(q), 3)
+        self.assertFalse(q.full())
+
+    def test_position_queue_full(self):
+        q = PositionQueue(length=10)
+        for i in xrange(10):
+            q[i] = i
+        self.assertEquals(q.filled, list(xrange(10)))
+        self.assertEquals(len(q), 10)
+        self.assertTrue(q.full())
+
+
+class TestExceptionInfo(unittest.TestCase):
+
+    def test_exception_info(self):
+
+        try:
+            raise LookupError("The quick brown fox jumps...")
+        except LookupError:
+            exc_info = sys.exc_info()
+
+        einfo = ExceptionInfo(exc_info)
+        self.assertEquals(str(einfo), "The quick brown fox jumps...")
+        self.assertTrue(isinstance(einfo.exception, LookupError))
+        self.assertEquals(einfo.exception.args,
+                ("The quick brown fox jumps...", ))
+        self.assertTrue(einfo.traceback)

celery/tests/test_discovery.py

@@ -16,3 +16,9 @@ class TestDiscovery(unittest.TestCase):
     def test_discovery(self):
         if "someapp" in settings.INSTALLED_APPS:
             self.assertDiscovery()
+
+    def test_discovery_with_broken(self):
+        if "someapp" in settings.INSTALLED_APPS:
+            settings.INSTALLED_APPS = settings.INSTALLED_APPS + \
+                    ["xxxnot.aexist"]
+            self.assertDiscovery()

celery/tests/test_log.py

@@ -1,10 +1,13 @@
-import unittest
-
+from __future__ import with_statement
+import os
 import sys
 import logging
+import unittest
 import multiprocessing
 from StringIO import StringIO
 from celery.log import setup_logger, emergency_error
+from celery.tests.utils import OverrideStdout
+from tempfile import mktemp
 
 
 class TestLog(unittest.TestCase):
@@ -48,3 +51,35 @@ class TestLog(unittest.TestCase):
         emergency_error(sio, "Testing emergency error facility")
         self.assertEquals(sio.getvalue().rpartition(":")[2].strip(),
                              "Testing emergency error facility")
+
+    def test_setup_logger_no_handlers_stream(self):
+        from multiprocessing import get_logger
+        l = get_logger()
+        l.handlers = []
+        with OverrideStdout() as outs:
+            stdout, stderr = outs
+            l = setup_logger(logfile=stderr, loglevel=logging.INFO)
+            l.info("The quick brown fox...")
+            self.assertTrue("The quick brown fox..." in stderr.getvalue())
+
+    def test_setup_logger_no_handlers_file(self):
+        from multiprocessing import get_logger
+        l = get_logger()
+        l.handlers = []
+        tempfile = mktemp(suffix="unittest", prefix="celery")
+        l = setup_logger(logfile=tempfile, loglevel=0)
+        self.assertTrue(isinstance(l.handlers[0], logging.FileHandler))
+
+    def test_emergency_error_stderr(self):
+        with OverrideStdout() as outs:
+            stdout, stderr = outs
+            emergency_error(None, "The lazy dog crawls under the fast fox")
+            self.assertTrue("The lazy dog crawls under the fast fox" in \
+                                stderr.getvalue())
+
+    def test_emergency_error_file(self):
+        tempfile = mktemp(suffix="unittest", prefix="celery")
+        emergency_error(tempfile, "Vandelay Industries")
+        with open(tempfile, "r") as tempfilefh:
+            self.assertTrue("Vandelay Industries" in "".join(tempfilefh))
+        os.unlink(tempfile)

celery/tests/test_messaging.py

@@ -0,0 +1,14 @@
+import unittest
+from celery.messaging import MSG_OPTIONS, get_msg_options, extract_msg_options
+
+
+class TestMsgOptions(unittest.TestCase):
+
+    def test_MSG_OPTIONS(self):
+        self.assertTrue(MSG_OPTIONS)
+
+    def test_extract_msg_options(self):
+        testing = {"mandatory": True, "routing_key": "foo.xuzzy"}
+        result = extract_msg_options(testing)
+        self.assertEquals(result["mandatory"], True)
+        self.assertEquals(result["routing_key"], "foo.xuzzy")

celery/tests/test_models.py

@@ -1,9 +1,9 @@
 import unittest
-import uuid
 from datetime import datetime, timedelta
 from celery.models import TaskMeta, PeriodicTaskMeta
 from celery.task import PeriodicTask
 from celery.registry import tasks
+from celery.utils import gen_unique_id
 
 
 class TestPeriodicTask(PeriodicTask):
@@ -14,12 +14,13 @@ class TestPeriodicTask(PeriodicTask):
 class TestModels(unittest.TestCase):
 
     def createTaskMeta(self):
-        id = str(uuid.uuid4())
+        id = gen_unique_id()
         taskmeta, created = TaskMeta.objects.get_or_create(task_id=id)
         return taskmeta
 
     def createPeriodicTaskMeta(self, name):
-        ptaskmeta, created = PeriodicTaskMeta.objects.get_or_create(name=name)
+        ptaskmeta, created = PeriodicTaskMeta.objects.get_or_create(name=name,
+                defaults={"last_run_at": datetime.now()})
         return ptaskmeta
 
     def test_taskmeta(self):
@@ -56,10 +57,9 @@ class TestModels(unittest.TestCase):
         # check that repr works.
         self.assertTrue(unicode(p).startswith("<PeriodicTask:"))
         self.assertFalse(p in PeriodicTaskMeta.objects.get_waiting_tasks())
-        # Have to avoid save() because it applies the auto_now=True.
-        PeriodicTaskMeta.objects.filter(name=p.name).update(
-                last_run_at=datetime.now() - (TestPeriodicTask.run_every +
-                timedelta(seconds=10)))
+        p.last_run_at = datetime.now() - (TestPeriodicTask.run_every +
+                timedelta(seconds=10))
+        p.save()
         self.assertTrue(p in PeriodicTaskMeta.objects.get_waiting_tasks())
         self.assertTrue(isinstance(p.task, TestPeriodicTask))
 

celery/tests/test_monitoring.py

@@ -0,0 +1,97 @@
+from __future__ import with_statement
+import unittest
+import time
+from celery.monitoring import TaskTimerStats, Statistics, StatsCollector
+from carrot.connection import DjangoAMQPConnection
+from celery.messaging import StatsConsumer
+from celery.tests.utils import OverrideStdout
+
+
+class PartialStatistics(Statistics):
+    type = "c.u.partial"
+
+
+class TestStatisticsInterface(unittest.TestCase):
+
+    def test_must_have_type(self):
+        self.assertRaises(NotImplementedError, Statistics)
+
+    def test_must_have_on_start(self):
+        self.assertRaises(NotImplementedError, PartialStatistics().on_start)
+
+    def test_must_have_on_stop(self):
+        self.assertRaises(NotImplementedError, PartialStatistics().on_stop)
+
+
+class TestTaskTimerStats(unittest.TestCase):
+
+    def test_time(self):
+        self.assertTimeElapsed(0.5, 1, 0, "0.5")
+        self.assertTimeElapsed(0.002, 0.05, 0, "0.0")
+        self.assertTimeElapsed(0.1, 0.5, 0, "0.1")
+
+    def test_not_enabled(self):
+        t = TaskTimerStats()
+        t.enabled = False
+        self.assertFalse(t.publish(isnot="enabled"))
+        self.assertFalse(getattr(t, "time_start", None))
+        t.run("foo", "bar", [], {})
+        t.stop()
+
+    def assertTimeElapsed(self, time_sleep, max_appx, min_appx, appx):
+        t = TaskTimerStats()
+        t.enabled = True
+        t.run("foo", "bar", [], {})
+        self.assertTrue(t.time_start)
+        time.sleep(time_sleep)
+        time_stop = t.stop()
+        self.assertTrue(time_stop)
+        self.assertFalse(time_stop > max_appx)
+        self.assertFalse(time_stop <= min_appx)
+
+        strstop = str(time_stop)[0:3]
+        # Time elapsed is approximately 0.1 seconds.
+        self.assertTrue(strstop == appx)
+
+
+class TestStatsCollector(unittest.TestCase):
+
+    def setUp(self):
+        conn = DjangoAMQPConnection()
+        consumer = StatsConsumer(connection=conn)
+        consumer.discard_all()
+        conn.close()
+        consumer.close()
+        self.s = StatsCollector()
+        self.assertEquals(self.s.total_tasks_processed, 0)
+        self.assertEquals(self.s.total_tasks_processed_by_type, {})
+        self.assertEquals(self.s.total_task_time_running, 0.0)
+        self.assertEquals(self.s.total_task_time_running_by_type, {})
+
+    def test_collect_report_dump(self):
+        timer1 = TaskTimerStats()
+        timer1.enabled = True
+        timer1.run("foo", "bar", [], {})
+        timer2 = TaskTimerStats()
+        timer2.enabled = True
+        timer2.run("foo", "bar", [], {})
+        timer3 = TaskTimerStats()
+        timer3.enabled = True
+        timer3.run("foo", "bar", [], {})
+        for timer in (timer1, timer2, timer3):
+            timer.stop()
+
+
+        # Collect
+        self.s.collect()
+        self.assertEquals(self.s.total_tasks_processed, 3)
+
+        # Report
+        with OverrideStdout() as outs:
+            stdout, stderr = outs
+            self.s.report()
+            self.assertTrue(
+                "Total processing time by task type:" in stdout.getvalue())
+
+        # Dump to cache
+        self.s.dump_to_cache()

celery/tests/test_pickle.py

@@ -1,5 +1,5 @@
 import unittest
-import pickle
+from celery.serialization import pickle
 
 
 class RegularException(Exception):

celery/tests/test_pool.py

@@ -0,0 +1,93 @@
+import unittest
+import logging
+import itertools
+import time
+from celery.pool import TaskPool
+from celery.datastructures import ExceptionInfo
+import sys
+
+
+def do_something(i):
+    return i * i
+
+
+def long_something():
+    import time
+    time.sleep(1)
+
+
+def raise_something(i):
+    try:
+        raise KeyError("FOO EXCEPTION")
+    except KeyError:
+        return ExceptionInfo(sys.exc_info())
+
+
+class TestTaskPool(unittest.TestCase):
+
+    def test_attrs(self):
+        p = TaskPool(limit=2)
+        self.assertEquals(p.limit, 2)
+        self.assertTrue(isinstance(p.logger, logging.Logger))
+        self.assertTrue(p._pool is None)
+
+    def x_start_stop(self):
+        p = TaskPool(limit=2)
+        p.start()
+        self.assertTrue(p._pool)
+        p.stop()
+        self.assertTrue(p._pool is None)
+
+    def x_apply(self):
+        p = TaskPool(limit=2)
+        p.start()
+        scratchpad = {}
+        proc_counter = itertools.count().next
+
+        def mycallback(ret_value, meta):
+            process = proc_counter()
+            scratchpad[process] = {}
+            scratchpad[process]["ret_value"] = ret_value
+            scratchpad[process]["meta"] = meta
+
+        myerrback = mycallback
+
+        res = p.apply_async(do_something, args=[10], callbacks=[mycallback],
+                            meta={"foo": "bar"})
+        res2 = p.apply_async(raise_something, args=[10], errbacks=[myerrback],
+                            meta={"foo2": "bar2"})
+        res3 = p.apply_async(do_something, args=[20], callbacks=[mycallback],
+                            meta={"foo3": "bar3"})
+
+        self.assertEquals(res.get(), 100)
+        time.sleep(0.5)
+        self.assertTrue(scratchpad.get(0))
+        self.assertEquals(scratchpad[0]["ret_value"], 100)
+        self.assertEquals(scratchpad[0]["meta"], {"foo": "bar"})
+
+        self.assertTrue(isinstance(res2.get(), ExceptionInfo))
+        self.assertTrue(scratchpad.get(1))
+        time.sleep(1)
+        #self.assertEquals(scratchpad[1]["ret_value"], "FOO")
+        self.assertTrue(isinstance(scratchpad[1]["ret_value"],
+                          ExceptionInfo))
+        self.assertEquals(scratchpad[1]["ret_value"].exception.args,
+                          ("FOO EXCEPTION", ))
+        self.assertEquals(scratchpad[1]["meta"], {"foo2": "bar2"})
+
+        self.assertEquals(res3.get(), 400)
+        time.sleep(0.5)
+        self.assertTrue(scratchpad.get(2))
+        self.assertEquals(scratchpad[2]["ret_value"], 400)
+        self.assertEquals(scratchpad[2]["meta"], {"foo3": "bar3"})
+
+        res3 = p.apply_async(do_something, args=[30], callbacks=[mycallback],
+                            meta={"foo4": "bar4"})
+
+        self.assertEquals(res3.get(), 900)
+        time.sleep(0.5)
+        self.assertTrue(scratchpad.get(3))
+        self.assertEquals(scratchpad[3]["ret_value"], 900)
+        self.assertEquals(scratchpad[3]["meta"], {"foo4": "bar4"})
+
+        p.stop()

celery/tests/test_result.py

@@ -0,0 +1,194 @@
+import unittest
+from celery.backends import default_backend
+from celery.result import AsyncResult
+from celery.result import TaskSetResult
+from celery.result import TimeoutError
+from celery.utils import gen_unique_id
+
+
+def mock_task(name, status, result):
+    return dict(id=gen_unique_id(), name=name, status=status, result=result)
+
+
+def save_result(task):
+    if task["status"] == "DONE":
+        default_backend.mark_as_done(task["id"], task["result"])
+    else:
+        default_backend.mark_as_failure(task["id"], task["result"])
+
+
+def make_mock_taskset(size=10):
+    tasks = [mock_task("ts%d" % i, "DONE", i) for i in xrange(size)]
+    [save_result(task) for task in tasks]
+    return [AsyncResult(task["id"]) for task in tasks]
+
+
+class TestAsyncResult(unittest.TestCase):
+
+    def setUp(self):
+        self.task1 = mock_task("task1", "DONE", "the")
+        self.task2 = mock_task("task2", "DONE", "quick")
+        self.task3 = mock_task("task3", "FAILURE", KeyError("brown"))
+
+        for task in (self.task1, self.task2, self.task3):
+            save_result(task)
+
+    def test_is_done(self):
+        ok_res = AsyncResult(self.task1["id"])
+        nok_res = AsyncResult(self.task3["id"])
+
+        self.assertTrue(ok_res.is_done())
+        self.assertFalse(nok_res.is_done())
+
+    def test_sucessful(self):
+        ok_res = AsyncResult(self.task1["id"])
+        nok_res = AsyncResult(self.task3["id"])
+
+        self.assertTrue(ok_res.successful())
+        self.assertFalse(nok_res.successful())
+
+    def test_str(self):
+        ok_res = AsyncResult(self.task1["id"])
+        ok2_res = AsyncResult(self.task2["id"])
+        nok_res = AsyncResult(self.task3["id"])
+        self.assertEquals(str(ok_res), self.task1["id"])
+        self.assertEquals(str(ok2_res), self.task2["id"])
+        self.assertEquals(str(nok_res), self.task3["id"])
+
+    def test_repr(self):
+        ok_res = AsyncResult(self.task1["id"])
+        ok2_res = AsyncResult(self.task2["id"])
+        nok_res = AsyncResult(self.task3["id"])
+        self.assertEquals(repr(ok_res), "<AsyncResult: %s>" % (
+                self.task1["id"]))
+        self.assertEquals(repr(ok2_res), "<AsyncResult: %s>" % (
+                self.task2["id"]))
+        self.assertEquals(repr(nok_res), "<AsyncResult: %s>" % (
+                self.task3["id"]))
+
+    def test_get(self):
+        ok_res = AsyncResult(self.task1["id"])
+        ok2_res = AsyncResult(self.task2["id"])
+        nok_res = AsyncResult(self.task3["id"])
+
+        self.assertEquals(ok_res.get(), "the")
+        self.assertEquals(ok2_res.get(), "quick")
+        self.assertRaises(KeyError, nok_res.get)
+
+    def test_ready(self):
+        oks = (AsyncResult(self.task1["id"]),
+               AsyncResult(self.task2["id"]),
+               AsyncResult(self.task3["id"]))
+        [self.assertTrue(ok.ready()) for ok in oks]
+
+
+class TestTaskSetResult(unittest.TestCase):
+
+    def setUp(self):
+        self.size = 10
+        self.ts = TaskSetResult(gen_unique_id(), make_mock_taskset(self.size))
+
+    def test_total(self):
+        self.assertEquals(self.ts.total, self.size)
+
+    def test_itersubtasks(self):
+
+        it = self.ts.itersubtasks()
+
+        for i, t in enumerate(it):
+            self.assertEquals(t.get(), i)
+
+    def test___iter__(self):
+
+        it = iter(self.ts)
+
+        results = sorted(list(it))
+        self.assertEquals(results, list(xrange(self.size)))
+
+    def test_join(self):
+        joined = self.ts.join()
+        self.assertEquals(joined, list(xrange(self.size)))
+
+    def test_successful(self):
+        self.assertTrue(self.ts.successful())
+
+    def test_failed(self):
+        self.assertFalse(self.ts.failed())
+
+    def test_waiting(self):
+        self.assertFalse(self.ts.waiting())
+
+    def test_ready(self):
+        self.assertTrue(self.ts.ready())
+
+    def test_completed_count(self):
+        self.assertEquals(self.ts.completed_count(), self.ts.total)
+
+
+class TestPendingAsyncResult(unittest.TestCase):
+
+    def setUp(self):
+        self.task = AsyncResult(gen_unique_id())
+
+    def test_result(self):
+        self.assertTrue(self.task.result is None)
+
+
+class TestFailedTaskSetResult(TestTaskSetResult):
+
+    def setUp(self):
+        self.size = 11
+        subtasks = make_mock_taskset(10)
+        failed = mock_task("ts11", "FAILED", KeyError("Baz"))
+        save_result(failed)
+        failed_res = AsyncResult(failed["id"])
+        self.ts = TaskSetResult(gen_unique_id(), subtasks + [failed_res])
+
+    def test_itersubtasks(self):
+
+        it = self.ts.itersubtasks()
+
+        for i in xrange(self.size - 1):
+            t = it.next()
+            self.assertEquals(t.get(), i)
+        self.assertRaises(KeyError, it.next().get)
+
+    def test_completed_count(self):
+        self.assertEquals(self.ts.completed_count(), self.ts.total - 1)
+
+    def test___iter__(self):
+        it = iter(self.ts)
+
+        def consume():
+            return list(it)
+
+        self.assertRaises(KeyError, consume)
+
+    def test_join(self):
+        self.assertRaises(KeyError, self.ts.join)
+
+    def test_successful(self):
+        self.assertFalse(self.ts.successful())
+
+    def test_failed(self):
+        self.assertTrue(self.ts.failed())
+
+
+class TestTaskSetPending(unittest.TestCase):
+
+    def setUp(self):
+        self.ts = TaskSetResult(gen_unique_id(), [
+                                        AsyncResult(gen_unique_id()),
+                                        AsyncResult(gen_unique_id())])
+
+    def test_completed_count(self):
+        self.assertEquals(self.ts.completed_count(), 0)
+
+    def test_ready(self):
+        self.assertFalse(self.ts.ready())
+
+    def test_waiting(self):
+        self.assertTrue(self.ts.waiting())
+
+    def x_join(self):
+        self.assertRaises(TimeoutError, self.ts.join, timeout=0.001)

celery/tests/test_serialization.py

@@ -0,0 +1,14 @@
+from __future__ import with_statement
+import sys
+import unittest
+
+
+class TestAAPickle(unittest.TestCase):
+
+    def test_no_cpickle(self):
+        from celery.tests.utils import mask_modules
+        del(sys.modules["celery.serialization"])
+        with mask_modules("cPickle"):
+            from celery.serialization import pickle
+            import pickle as orig_pickle
+            self.assertTrue(pickle.dumps is orig_pickle.dumps)

celery/tests/test_supervisor.py

@@ -0,0 +1,66 @@
+import unittest
+from celery.supervisor import OFASupervisor
+from celery.supervisor import TimeoutError, MaxRestartsExceededError
+
+
+def target_one(x, y, z):
+    return x * y * z
+
+
+class MockProcess(object):
+    _started = False
+    _stopped = False
+    _terminated = False
+    _joined = False
+    alive = True
+    timeout_on_is_alive = False
+
+    def __init__(self, target, args, kwargs):
+        self.target = target
+        self.args = args
+        self.kwargs = kwargs
+
+    def start(self):
+        self._stopped = False
+        self._started = True
+
+    def stop(self):
+        self._stopped = True
+        self._started = False
+
+    def terminate(self):
+        self._terminated = False
+
+    def is_alive(self):
+        if self._started and self.alive:
+            if self.timeout_on_is_alive:
+                raise TimeoutError("Supervised: timed out.")
+            return True
+        return False
+
+    def join(self, timeout=None):
+        self._joined = True
+
+
+class TestOFASupervisor(unittest.TestCase):
+
+    def test_init(self):
+        s = OFASupervisor(target=target_one, args=[2, 4, 8], kwargs={})
+        s.Process = MockProcess
+
+    def test_start(self):
+        MockProcess.alive = False
+        s = OFASupervisor(target=target_one, args=[2, 4, 8], kwargs={},
+                          max_restart_freq=0, max_restart_freq_time=0)
+        s.Process = MockProcess
+        self.assertRaises(MaxRestartsExceededError, s.start)
+        MockProcess.alive = True
+
+    def test_start_is_alive_timeout(self):
+        MockProcess.alive = True
+        MockProcess.timeout_on_is_alive = True
+        s = OFASupervisor(target=target_one, args=[2, 4, 8], kwargs={},
+                          max_restart_freq=0, max_restart_freq_time=0)
+        s.Process = MockProcess
+        self.assertRaises(MaxRestartsExceededError, s.start)
+        MockProcess.timeout_on_is_alive = False

celery/tests/test_task.py

@@ -7,12 +7,15 @@ from celery import task
 from celery import registry
 from celery.log import setup_logger
 from celery import messaging
+from celery.result import EagerResult
 from celery.backends import default_backend
+from datetime import datetime, timedelta
 
 
 def return_True(self, **kwargs):
     # Task run functions can't be closures/lambdas, as they're pickled.
     return True
+registry.tasks.register(return_True, "cu.return-true")
 
 
 def raise_exception(self, **kwargs):
@@ -23,9 +26,17 @@ class IncrementCounterTask(task.Task):
     name = "c.unittest.increment_counter_task"
     count = 0
 
-    def run(self, increment_by, **kwargs):
+    def run(self, increment_by=1, **kwargs):
         increment_by = increment_by or 1
         self.__class__.count += increment_by
+        return self.__class__.count
+
+
+class RaisingTask(task.Task):
+    name = "c.unittest.raising_task"
+
+    def run(self, **kwargs):
+        raise KeyError("foo")
 
 
 class TestCeleryTasks(unittest.TestCase):
@@ -38,13 +49,46 @@ class TestCeleryTasks(unittest.TestCase):
         cls.run = return_True
         return cls
 
+    def test_ping(self):
+        from celery import conf
+        conf.ALWAYS_EAGER = True
+        self.assertEquals(task.ping(), 'pong')
+        conf.ALWAYS_EAGER = False
+
+    def test_execute_remote(self):
+        from celery import conf
+        conf.ALWAYS_EAGER = True
+        self.assertEquals(task.execute_remote(return_True, ["foo"]).get(),
+                          True)
+        conf.ALWAYS_EAGER = False
+
+    def test_dmap(self):
+        from celery import conf
+        import operator
+        conf.ALWAYS_EAGER = True
+        res = task.dmap(operator.add, zip(xrange(10), xrange(10)))
+        self.assertTrue(res, sum([operator.add(x, x)
+                                    for x in xrange(10)]))
+        conf.ALWAYS_EAGER = False
+
+    def test_dmap_async(self):
+        from celery import conf
+        import operator
+        conf.ALWAYS_EAGER = True
+        res = task.dmap_async(operator.add, zip(xrange(10), xrange(10)))
+        self.assertTrue(res.get(), sum([operator.add(x, x)
+                                            for x in xrange(10)]))
+        conf.ALWAYS_EAGER = False
+
     def assertNextTaskDataEquals(self, consumer, presult, task_name,
-            **kwargs):
+            test_eta=False, **kwargs):
         next_task = consumer.fetch()
-        task_data = consumer.decoder(next_task.body)
+        task_data = next_task.decode()
         self.assertEquals(task_data["id"], presult.task_id)
         self.assertEquals(task_data["task"], task_name)
         task_kwargs = task_data.get("kwargs", {})
+        if test_eta:
+            self.assertTrue(isinstance(task_data.get("eta"), datetime))
         for arg_name, arg_value in kwargs.items():
             self.assertEquals(task_kwargs.get(arg_name), arg_value)
 
@@ -64,9 +108,9 @@ class TestCeleryTasks(unittest.TestCase):
         self.assertTrue(T1()(),
                 "Task class runs run() when called")
 
-        # task without name raises NotImplementedError
+        # task name generated out of class module + name.
         T2 = self.createTaskCls("T2")
-        self.assertRaises(NotImplementedError, T2)
+        self.assertEquals(T2().name, "celery.tests.test_task.T2")
 
         registry.tasks.register(T1)
         t1 = T1()
@@ -84,6 +128,18 @@ class TestCeleryTasks(unittest.TestCase):
         self.assertNextTaskDataEquals(consumer, presult2, t1.name,
                 name="George Constanza")
 
+        # With eta.
+        presult2 = task.apply_async(t1, kwargs=dict(name="George Constanza"),
+                                    eta=datetime.now() + timedelta(days=1))
+        self.assertNextTaskDataEquals(consumer, presult2, t1.name,
+                name="George Constanza", test_eta=True)
+
+        # With countdown.
+        presult2 = task.apply_async(t1, kwargs=dict(name="George Constanza"),
+                                    countdown=10)
+        self.assertNextTaskDataEquals(consumer, presult2, t1.name,
+                name="George Constanza", test_eta=True)
+
         self.assertRaises(registry.tasks.NotRegistered, task.delay_task,
                 "some.task.that.should.never.exist.X.X.X.X.X")
 
@@ -103,10 +159,28 @@ class TestCeleryTasks(unittest.TestCase):
         publisher = t1.get_publisher()
         self.assertTrue(isinstance(publisher, messaging.TaskPublisher))
 
+    def test_get_logger(self):
+        T1 = self.createTaskCls("T1", "c.unittest.t.t1")
+        t1 = T1()
+        logfh = StringIO()
+        logger = t1.get_logger(logfile=logfh, loglevel=0)
+        self.assertTrue(logger)
+
 
 class TestTaskSet(unittest.TestCase):
 
+    def test_function_taskset(self):
+        from celery import conf
+        conf.ALWAYS_EAGER = True
+        ts = task.TaskSet("cu.return-true", [
+            [[1], {}], [[2], {}], [[3], {}], [[4], {}], [[5], {}]])
+        res = ts.run()
+        self.assertEquals(res.join(), [True, True, True, True, True])
+
+        conf.ALWAYS_EAGER = False
+
     def test_counter_taskset(self):
+        IncrementCounterTask.count = 0
         ts = task.TaskSet(IncrementCounterTask, [
             [[], {}],
             [[], {"increment_by": 2}],
@@ -134,3 +208,38 @@ class TestTaskSet(unittest.TestCase):
             IncrementCounterTask().run(
                     increment_by=m.get("kwargs", {}).get("increment_by"))
         self.assertEquals(IncrementCounterTask.count, sum(xrange(1, 10)))
+
+
+class TestTaskApply(unittest.TestCase):
+
+    def test_apply(self):
+        IncrementCounterTask.count = 0
+
+        e = IncrementCounterTask.apply()
+        self.assertTrue(isinstance(e, EagerResult))
+        self.assertEquals(e.get(), 1)
+
+        e = IncrementCounterTask.apply(args=[1])
+        self.assertEquals(e.get(), 2)
+
+        e = IncrementCounterTask.apply(kwargs={"increment_by": 4})
+        self.assertEquals(e.get(), 6)
+
+        self.assertTrue(e.is_done())
+        self.assertTrue(e.is_ready())
+        self.assertTrue(repr(e).startswith("<EagerResult:"))
+
+        f = RaisingTask.apply()
+        self.assertTrue(f.is_ready())
+        self.assertFalse(f.is_done())
+        self.assertRaises(KeyError, f.get)
+
+
+class TestPeriodicTask(unittest.TestCase):
+
+    def test_interface(self):
+
+        class MyPeriodicTask(task.PeriodicTask):
+            run_every = None
+
+        self.assertRaises(NotImplementedError, MyPeriodicTask)

celery/tests/test_task_builtins.py

@@ -0,0 +1,28 @@
+import unittest
+from celery.task.builtins import PingTask, DeleteExpiredTaskMetaTask
+from celery.task.base import ExecuteRemoteTask
+from celery.serialization import pickle
+
+
+def some_func(i):
+    return i * i
+
+
+class TestPingTask(unittest.TestCase):
+
+    def test_ping(self):
+        self.assertEquals(PingTask.apply().get(), 'pong')
+
+
+class TestRemoteExecuteTask(unittest.TestCase):
+
+    def test_execute_remote(self):
+        self.assertEquals(ExecuteRemoteTask.apply(
+                            args=[pickle.dumps(some_func), [10], {}]).get(),
+                          100)
+
+
+class TestDeleteExpiredTaskMetaTask(unittest.TestCase):
+
+    def test_run(self):
+        DeleteExpiredTaskMetaTask.apply()

celery/tests/test_utils.py

@@ -0,0 +1,23 @@
+import sys
+import unittest
+from celery.utils import chunks
+
+
+class TestChunks(unittest.TestCase):
+
+    def test_chunks(self):
+
+        # n == 2
+        x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 2)
+        self.assertEquals(list(x),
+            [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10]])
+
+        # n == 3
+        x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 3)
+        self.assertEquals(list(x),
+            [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10]])
+
+        # n == 2 (exact)
+        x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), 2)
+        self.assertEquals(list(x),
+            [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]])

celery/tests/test_worker.py

@@ -0,0 +1,212 @@
+import unittest
+from Queue import Queue, Empty
+from carrot.connection import AMQPConnection
+from celery.messaging import TaskConsumer
+from celery.worker.job import TaskWrapper
+from celery.worker import AMQPListener, WorkController
+from multiprocessing import get_logger
+from carrot.backends.base import BaseMessage
+from celery import registry
+from celery.serialization import pickle
+from celery.utils import gen_unique_id
+from datetime import datetime, timedelta
+
+
+def foo_task(x, y, z, **kwargs):
+    return x * y * z
+registry.tasks.register(foo_task, name="c.u.foo")
+
+
+class MockLogger(object):
+
+    def critical(self, *args, **kwargs):
+        pass
+
+    def info(self, *args, **kwargs):
+        pass
+
+    def error(self, *args, **kwargs):
+        pass
+
+    def debug(self, *args, **kwargs):
+        pass
+
+
+class MockBackend(object):
+    _acked = False
+
+    def ack(self, delivery_tag):
+        self._acked = True
+
+
+class MockPool(object):
+
+    def __init__(self, *args, **kwargs):
+        self.raise_regular = kwargs.get("raise_regular", False)
+        self.raise_base = kwargs.get("raise_base", False)
+
+    def apply_async(self, *args, **kwargs):
+        if self.raise_regular:
+            raise KeyError("some exception")
+        if self.raise_base:
+            raise KeyboardInterrupt("Ctrl+c")
+
+    def start(self):
+        pass
+
+    def stop(self):
+        pass
+        return True
+
+
+class MockController(object):
+
+    def __init__(self, w, *args, **kwargs):
+        self._w = w
+        self._stopped = False
+
+    def start(self):
+        self._w["started"] = True
+        self._stopped = False
+
+    def stop(self):
+        self._stopped = True
+
+
+def create_message(backend, **data):
+    data["id"] = gen_unique_id()
+    return BaseMessage(backend, body=pickle.dumps(dict(**data)),
+                       content_type="application/x-python-serialize",
+                       content_encoding="binary")
+
+
+class TestAMQPListener(unittest.TestCase):
+
+    def setUp(self):
+        self.bucket_queue = Queue()
+        self.hold_queue = Queue()
+        self.logger = get_logger()
+        self.logger.setLevel(0)
+
+    def test_connection(self):
+        l = AMQPListener(self.bucket_queue, self.hold_queue, self.logger)
+
+        c = l.reset_connection()
+        self.assertTrue(isinstance(c, TaskConsumer))
+        self.assertTrue(c is l.task_consumer)
+        self.assertTrue(isinstance(l.amqp_connection, AMQPConnection))
+
+        l.close_connection()
+        self.assertTrue(l.amqp_connection is None)
+        self.assertTrue(l.task_consumer is None)
+
+        c = l.reset_connection()
+        self.assertTrue(isinstance(c, TaskConsumer))
+        self.assertTrue(c is l.task_consumer)
+        self.assertTrue(isinstance(l.amqp_connection, AMQPConnection))
+
+        l.stop()
+        self.assertTrue(l.amqp_connection is None)
+        self.assertTrue(l.task_consumer is None)
+
+    def test_receieve_message(self):
+        l = AMQPListener(self.bucket_queue, self.hold_queue, self.logger)
+        backend = MockBackend()
+        m = create_message(backend, task="c.u.foo", args=[2, 4, 8], kwargs={})
+
+        l.receive_message(m.decode(), m)
+
+        in_bucket = self.bucket_queue.get_nowait()
+        self.assertTrue(isinstance(in_bucket, TaskWrapper))
+        self.assertEquals(in_bucket.task_name, "c.u.foo")
+        self.assertEquals(in_bucket.execute(), 2 * 4 * 8)
+        self.assertRaises(Empty, self.hold_queue.get_nowait)
+
+    def test_receieve_message_not_registered(self):
+        l = AMQPListener(self.bucket_queue, self.hold_queue, self.logger)
+        backend = MockBackend()
+        m = create_message(backend, task="x.X.31x", args=[2, 4, 8], kwargs={})
+
+        self.assertFalse(l.receive_message(m.decode(), m))
+        self.assertRaises(Empty, self.bucket_queue.get_nowait)
+        self.assertRaises(Empty, self.hold_queue.get_nowait)
+
+    def test_receieve_message_eta(self):
+        l = AMQPListener(self.bucket_queue, self.hold_queue, self.logger)
+        backend = MockBackend()
+        m = create_message(backend, task="c.u.foo", args=[2, 4, 8], kwargs={},
+                           eta=datetime.now() + timedelta(days=1))
+
+        l.receive_message(m.decode(), m)
+
+        in_hold = self.hold_queue.get_nowait()
+        self.assertEquals(len(in_hold), 2)
+        task, eta = in_hold
+        self.assertTrue(isinstance(task, TaskWrapper))
+        self.assertTrue(isinstance(eta, datetime))
+        self.assertEquals(task.task_name, "c.u.foo")
+        self.assertEquals(task.execute(), 2 * 4 * 8)
+        self.assertRaises(Empty, self.bucket_queue.get_nowait)
+
+
+class TestWorkController(unittest.TestCase):
+
+    def setUp(self):
+        self.worker = WorkController(concurrency=1, loglevel=0,
+                                     is_detached=False)
+        self.worker.logger = MockLogger()
+
+    def test_attrs(self):
+        worker = self.worker
+        self.assertTrue(isinstance(worker.bucket_queue, Queue))
+        self.assertTrue(isinstance(worker.hold_queue, Queue))
+        self.assertTrue(worker.periodic_work_controller)
+        self.assertTrue(worker.pool)
+        self.assertTrue(worker.amqp_listener)
+        self.assertTrue(worker.mediator)
+        self.assertTrue(worker.components)
+
+    def test_safe_process_task(self):
+        worker = self.worker
+        worker.pool = MockPool()
+        backend = MockBackend()
+        m = create_message(backend, task="c.u.foo", args=[4, 8, 10],
+                           kwargs={})
+        task = TaskWrapper.from_message(m, m.decode())
+        worker.safe_process_task(task)
+        worker.pool.stop()
+
+    def test_safe_process_task_raise_base(self):
+        worker = self.worker
+        worker.pool = MockPool(raise_base=True)
+        backend = MockBackend()
+        m = create_message(backend, task="c.u.foo", args=[4, 8, 10],
+                           kwargs={})
+        task = TaskWrapper.from_message(m, m.decode())
+        worker.safe_process_task(task)
+        worker.pool.stop()
+
+    def test_safe_process_task_raise_regular(self):
+        worker = self.worker
+        worker.pool = MockPool(raise_regular=True)
+        backend = MockBackend()
+        m = create_message(backend, task="c.u.foo", args=[4, 8, 10],
+                           kwargs={})
+        task = TaskWrapper.from_message(m, m.decode())
+        worker.safe_process_task(task)
+        worker.pool.stop()
+
+    def test_start_stop(self):
+        worker = self.worker
+        w1 = {"started": False}
+        w2 = {"started": False}
+        w3 = {"started": False}
+        w4 = {"started": False}
+        worker.components = [MockController(w1), MockController(w2),
+                             MockController(w3), MockController(w4)]
+
+        worker.start()
+        for w in (w1, w2, w3, w4):
+            self.assertTrue(w["started"])
+        for component in worker.components:
+            self.assertTrue(component._stopped)

celery/tests/test_worker_controllers.py

@@ -0,0 +1,109 @@
+import unittest
+import time
+import multiprocessing
+from Queue import Queue, Empty
+from datetime import datetime, timedelta
+
+from celery.worker.controllers import Mediator, PeriodicWorkController
+from celery.worker.controllers import BackgroundThread
+
+
+class MockTask(object):
+    task_id = 1234
+    task_name = "mocktask"
+
+    def __init__(self, value, **kwargs):
+        self.value = value
+
+
+class MyBackgroundThread(BackgroundThread):
+
+    def on_iteration(self):
+        import time
+        time.sleep(1)
+
+
+class TestBackgroundThread(unittest.TestCase):
+
+    def test_on_iteration(self):
+        self.assertRaises(NotImplementedError,
+                BackgroundThread().on_iteration)
+
+    def test_run(self):
+        t = MyBackgroundThread()
+        t._shutdown.set()
+        t.run()
+        self.assertTrue(t._stopped.isSet())
+
+    def test_start_stop(self):
+        t = MyBackgroundThread()
+        t.start()
+        self.assertFalse(t._shutdown.isSet())
+        self.assertFalse(t._stopped.isSet())
+        t.stop()
+        self.assertTrue(t._shutdown.isSet())
+        self.assertTrue(t._stopped.isSet())
+
+
+class TestMediator(unittest.TestCase):
+
+    def test_mediator_start__stop(self):
+        bucket_queue = Queue()
+        m = Mediator(bucket_queue, lambda t: t)
+        m.start()
+        self.assertFalse(m._shutdown.isSet())
+        self.assertFalse(m._stopped.isSet())
+        m.stop()
+        m.join()
+        self.assertTrue(m._shutdown.isSet())
+        self.assertTrue(m._stopped.isSet())
+
+    def test_mediator_on_iteration(self):
+        bucket_queue = Queue()
+        got = {}
+
+        def mycallback(value):
+            got["value"] = value.value
+
+        m = Mediator(bucket_queue, mycallback)
+        bucket_queue.put(MockTask("George Constanza"))
+
+        m.on_iteration()
+
+        self.assertEquals(got["value"], "George Constanza")
+
+
+class TestPeriodicWorkController(unittest.TestCase):
+
+    def test_process_hold_queue(self):
+        bucket_queue = Queue()
+        hold_queue = Queue()
+        m = PeriodicWorkController(bucket_queue, hold_queue)
+
+        m.process_hold_queue()
+
+        hold_queue.put((MockTask("task1"),
+                        datetime.now() - timedelta(days=1)))
+
+        m.process_hold_queue()
+        self.assertRaises(Empty, hold_queue.get_nowait)
+        self.assertEquals(bucket_queue.get_nowait().value, "task1")
+        tomorrow = datetime.now() + timedelta(days=1)
+        hold_queue.put((MockTask("task2"), tomorrow))
+        m.process_hold_queue()
+        self.assertRaises(Empty, bucket_queue.get_nowait)
+        value, eta = hold_queue.get_nowait()
+        self.assertEquals(value.value, "task2")
+        self.assertEquals(eta, tomorrow)
+
+    def test_run_periodic_tasks(self):
+        bucket_queue = Queue()
+        hold_queue = Queue()
+        m = PeriodicWorkController(bucket_queue, hold_queue)
+        m.run_periodic_tasks()
+
+    def test_on_iteration(self):
+        bucket_queue = Queue()
+        hold_queue = Queue()
+        m = PeriodicWorkController(bucket_queue, hold_queue)
+        m.on_iteration()

celery/tests/test_worker_job.py

@@ -0,0 +1,236 @@
+# -*- coding: utf-8 -*-
+import sys
+import unittest
+from celery.worker.job import jail
+from celery.worker.job import TaskWrapper
+from celery.datastructures import ExceptionInfo
+from celery.models import TaskMeta
+from celery.registry import tasks, NotRegistered
+from celery.pool import TaskPool
+from celery.utils import gen_unique_id
+from carrot.backends.base import BaseMessage
+from StringIO import StringIO
+from celery.log import setup_logger
+from django.core import cache
+import simplejson
+import logging
+
+scratch = {"ACK": False}
+
+
+def on_ack():
+    scratch["ACK"] = True
+
+
+def mytask(i, **kwargs):
+    return i ** i
+tasks.register(mytask, name="cu.mytask")
+
+
+def mytask_raising(i, **kwargs):
+    raise KeyError(i)
+tasks.register(mytask_raising, name="cu.mytask-raising")
+
+
+def get_db_connection(i, **kwargs):
+    from django.db import connection
+    return id(connection)
+get_db_connection.ignore_result = True
+
+
+class TestJail(unittest.TestCase):
+
+    def test_execute_jail_success(self):
+        ret = jail(gen_unique_id(), gen_unique_id(), mytask, [2], {})
+        self.assertEquals(ret, 4)
+
+    def test_execute_jail_failure(self):
+        ret = jail(gen_unique_id(), gen_unique_id(), mytask_raising, [4], {})
+        self.assertTrue(isinstance(ret, ExceptionInfo))
+        self.assertEquals(ret.exception.args, (4, ))
+
+    def test_django_db_connection_is_closed(self):
+        from django.db import connection
+        connection._was_closed = False
+        old_connection_close = connection.close
+
+        def monkeypatched_connection_close(*args, **kwargs):
+            connection._was_closed = True
+            return old_connection_close(*args, **kwargs)
+
+        connection.close = monkeypatched_connection_close
+
+        ret = jail(gen_unique_id(), gen_unique_id(),
+                   get_db_connection, [2], {})
+        self.assertTrue(connection._was_closed)
+
+        connection.close = old_connection_close
+
+    def test_django_cache_connection_is_closed(self):
+        old_cache_close = getattr(cache.cache, "close", None)
+        old_backend = cache.settings.CACHE_BACKEND
+        cache.settings.CACHE_BACKEND = "libmemcached"
+        cache._was_closed = False
+        old_cache_parse_backend = getattr(cache, "parse_backend_uri", None)
+        if old_cache_parse_backend: # checks to make sure attr exists
+            delattr(cache, 'parse_backend_uri')
+
+        def monkeypatched_cache_close(*args, **kwargs):
+            cache._was_closed = True
+
+        cache.cache.close = monkeypatched_cache_close
+
+        jail(gen_unique_id(), gen_unique_id(), mytask, [4], {})
+        self.assertTrue(cache._was_closed)
+        cache.cache.close = old_cache_close
+        cache.settings.CACHE_BACKEND = old_backend
+        if old_cache_parse_backend:
+            cache.parse_backend_uri = old_cache_parse_backend
+
+    def test_django_cache_connection_is_closed_django_1_1(self):
+        old_cache_close = getattr(cache.cache, "close", None)
+        old_backend = cache.settings.CACHE_BACKEND
+        cache.settings.CACHE_BACKEND = "libmemcached"
+        cache._was_closed = False
+        old_cache_parse_backend = getattr(cache, "parse_backend_uri", None)
+        cache.parse_backend_uri = lambda uri: ["libmemcached", "1", "2"]
+
+        def monkeypatched_cache_close(*args, **kwargs):
+            cache._was_closed = True
+
+        cache.cache.close = monkeypatched_cache_close
+
+        jail(gen_unique_id(), gen_unique_id(), mytask, [4], {})
+        self.assertTrue(cache._was_closed)
+        cache.cache.close = old_cache_close
+        cache.settings.CACHE_BACKEND = old_backend
+        if old_cache_parse_backend:
+            cache.parse_backend_uri = old_cache_parse_backend
+        else:
+            del(cache.parse_backend_uri)
+
+
+class TestTaskWrapper(unittest.TestCase):
+
+    def test_task_wrapper_attrs(self):
+        tw = TaskWrapper(gen_unique_id(), gen_unique_id(),
+                         mytask, [1], {"f": "x"})
+        for attr in ("task_name", "task_id", "args", "kwargs", "logger"):
+            self.assertTrue(getattr(tw, attr, None))
+
+    def test_task_wrapper_repr(self):
+        tw = TaskWrapper(gen_unique_id(), gen_unique_id(),
+                         mytask, [1], {"f": "x"})
+        self.assertTrue(repr(tw))
+
+    def test_task_wrapper_mail_attrs(self):
+        tw = TaskWrapper(gen_unique_id(), gen_unique_id(), mytask, [], {})
+        x = tw.success_msg % {"name": tw.task_name,
+                              "id": tw.task_id,
+                              "return_value": 10}
+        self.assertTrue(x)
+        x = tw.fail_msg % {"name": tw.task_name,
+                           "id": tw.task_id,
+                           "exc": "FOOBARBAZ",
+                           "traceback": "foobarbaz"}
+        self.assertTrue(x)
+        x = tw.fail_email_subject % {"name": tw.task_name,
+                                     "id": tw.task_id,
+                                     "exc": "FOOBARBAZ",
+                                     "hostname": "lana"}
+        self.assertTrue(x)
+
+    def test_from_message(self):
+        body = {"task": "cu.mytask", "id": gen_unique_id(),
+                "args": [2], "kwargs": {u"æØåveéðƒeæ": "bar"}}
+        m = BaseMessage(body=simplejson.dumps(body), backend="foo",
+                        content_type="application/json",
+                        content_encoding="utf-8")
+        tw = TaskWrapper.from_message(m, m.decode())
+        self.assertTrue(isinstance(tw, TaskWrapper))
+        self.assertEquals(tw.task_name, body["task"])
+        self.assertEquals(tw.task_id, body["id"])
+        self.assertEquals(tw.args, body["args"])
+        self.assertEquals(tw.kwargs.keys()[0],
+                          u"æØåveéðƒeæ".encode("utf-8"))
+        self.assertFalse(isinstance(tw.kwargs.keys()[0], unicode))
+        self.assertEquals(id(mytask), id(tw.task_func))
+        self.assertTrue(tw.logger)
+
+    def test_from_message_nonexistant_task(self):
+        body = {"task": "cu.mytask.doesnotexist", "id": gen_unique_id(),
+                "args": [2], "kwargs": {u"æØåveéðƒeæ": "bar"}}
+        m = BaseMessage(body=simplejson.dumps(body), backend="foo",
+                        content_type="application/json",
+                        content_encoding="utf-8")
+        self.assertRaises(NotRegistered, TaskWrapper.from_message,
+                          m, m.decode())
+
+    def test_execute(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask", tid, mytask, [4], {"f": "x"})
+        self.assertEquals(tw.execute(), 256)
+        meta = TaskMeta.objects.get(task_id=tid)
+        self.assertEquals(meta.result, 256)
+        self.assertEquals(meta.status, "DONE")
+
+    def test_execute_ack(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask", tid, mytask, [4], {"f": "x"},
+                        on_ack=on_ack)
+        self.assertEquals(tw.execute(), 256)
+        meta = TaskMeta.objects.get(task_id=tid)
+        self.assertTrue(scratch["ACK"])
+        self.assertEquals(meta.result, 256)
+        self.assertEquals(meta.status, "DONE")
+
+    def test_execute_fail(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask-raising", tid, mytask_raising, [4],
+                         {"f": "x"})
+        self.assertTrue(isinstance(tw.execute(), ExceptionInfo))
+        meta = TaskMeta.objects.get(task_id=tid)
+        self.assertEquals(meta.status, "FAILURE")
+        self.assertTrue(isinstance(meta.result, KeyError))
+
+    def test_execute_using_pool(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask", tid, mytask, [4], {"f": "x"})
+        p = TaskPool(2)
+        p.start()
+        asyncres = tw.execute_using_pool(p)
+        self.assertTrue(asyncres.get(), 256)
+        p.stop()
+
+    def test_default_kwargs(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask", tid, mytask, [4], {"f": "x"})
+        self.assertEquals(tw.extend_with_default_kwargs(10, "some_logfile"), {
+            "f": "x",
+            "logfile": "some_logfile",
+            "loglevel": 10,
+            "task_id": tw.task_id,
+            "task_name": tw.task_name})
+
+    def test_on_failure(self):
+        tid = gen_unique_id()
+        tw = TaskWrapper("cu.mytask", tid, mytask, [4], {"f": "x"})
+        try:
+            raise Exception("Inside unit tests")
+        except Exception:
+            exc_info = ExceptionInfo(sys.exc_info())
+
+        logfh = StringIO()
+        tw.logger.handlers = []
+        tw.logger = setup_logger(logfile=logfh, loglevel=logging.INFO)
+
+        from celery import conf
+        conf.SEND_CELERY_TASK_ERROR_EMAILS = True
+
+        tw.on_failure(exc_info, {"task_id": tid, "task_name": "cu.mytask"})
+        logvalue = logfh.getvalue()
+        self.assertTrue("cu.mytask" in logvalue)
+        self.assertTrue(tid in logvalue)
+        self.assertTrue("ERROR" in logvalue)
+
+        conf.SEND_CELERY_TASK_ERROR_EMAILS = False

celery/tests/utils.py

@@ -0,0 +1,55 @@
+from __future__ import with_statement
+from contextlib import contextmanager
+from StringIO import StringIO
+import os
+import sys
+import __builtin__
+
+
+@contextmanager
+def mask_modules(*modnames):
+    """Ban some modules from being importable inside the context
+
+    For example:
+
+        >>> with missing_modules("sys"):
+        ...     try:
+        ...         import sys
+        ...     except ImportError:
+        ...         print "sys not found"
+        sys not found
+
+        >>> import sys
+        >>> sys.version
+        (2, 5, 2, 'final', 0)
+
+    """
+
+    realimport = __builtin__.__import__
+
+    def myimp(name, *args, **kwargs):
+        if name in modnames:
+            raise ImportError("No module named %s" % name)
+        else:
+            return realimport(name, *args, **kwargs)
+
+    __builtin__.__import__ = myimp
+    yield
+    __builtin__.__import__ = realimport
+
+
+class OverrideStdout(object):
+    """Override ``sys.stdout`` and ``sys.stderr`` with ``StringIO``."""
+
+    def __enter__(self):
+        mystdout = StringIO()
+        mystderr = StringIO()
+        sys.stdout = mystdout
+        sys.stderr = mystderr
+        return mystdout, mystderr
+
+    def __exit__(self, e_type, e_value, e_trace):
+        if e_type:
+            raise e_type(e_value)
+        sys.stdout = sys.__stdout__
+        sys.stderr = sys.__stderr__

celery/timer.py

@@ -1,79 +0,0 @@
-"""
-
-Managing time and events
-
-"""
-import time
-
-
-class TimeoutError(Exception):
-    """The event has timed out."""
-
-
-class EventTimer(object):
-    """Do something at an interval.
-
-    .. attribute:: interval
-
-        How often we call the event (in seconds).
-
-    .. attribute:: event
-
-        The event callable to run every ``interval`` seconds.
-
-    .. attribute:: last_triggered
-
-        The last time, in unix timestamp format, the event was executed.
-
-    """
-
-    def __init__(self, event, interval=None):
-        self.event = event
-        self.interval = interval
-        self.last_triggered = None
-
-    def tick(self):
-        """Run a event timer clock tick.
-
-        When the interval has run, the event will be triggered.
-        If interval is not set, the event will never be triggered.
-
-        """
-        if not self.interval: # never trigger if no interval.
-            return
-        if not self.last_triggered or \
-                time.time() > self.last_triggered + self.interval:
-            self.event()
-            self.last_triggered = time.time()
-
-
-class TimeoutTimer(object):
-    """A timer that raises :exc:`TimeoutError` exception when the
-    time has run out.
-
-    .. attribute:: timeout
-
-        The timeout in seconds.
-
-    .. attribute:: time_start
-
-        The time when the timeout timer instance was constructed.
-
-    """
-
-    def __init__(self, timeout, timeout_msg="The operation timed out"):
-        self.timeout = timeout
-        self.timeout_msg = timeout_msg
-        self.time_start = time.time()
-
-    def tick(self):
-        """Run a timeout timer clock tick.
-
-        :raises TimeoutError: when :attr:`timeout` seconds has passed.
-            If :attr:`timeout` is not set, it will never time out.
-
-        """
-        if not self.timeout:
-            return
-        if time.time() > self.time_start + self.timeout:
-            raise TimeoutError(self.timeout_msg)

celery/utils.py

@@ -0,0 +1,51 @@
+"""
+
+Utility functions
+
+"""
+import uuid
+
+
+def chunks(it, n):
+    """Split an iterator into chunks with ``n`` elements each.
+
+    Examples
+
+        # n == 2
+        >>> x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 2)
+        >>> list(x)
+        [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10]]
+
+        # n == 3
+        >>> x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 3)
+        >>> list(x)
+        [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10]]
+
+    """
+    acc = []
+    for i, item in enumerate(it):
+        if i and not i % n:
+            yield acc
+            acc = []
+        acc.append(item)
+    yield acc
+
+
+def gen_unique_id():
+    """Generate a unique id, having - hopefully - a very small chance of
+    collission.
+
+    For now this is provided by :func:`uuid.uuid4`.
+    """
+    return str(uuid.uuid4())
+
+
+def mitemgetter(*keys):
+    """Like :func:`operator.itemgetter` but returns `None` on missing keys
+    instead of raising :exc:`KeyError`."""
+    return lambda dict_: map(dict_.get, keys)
+
+
+def get_full_cls_name(cls):
+    return ".".join([cls.__module__,
+                     cls.__name__])

celery/views.py

@@ -1,8 +1,30 @@
 """celery.views"""
-from django.http import HttpResponse
-from celery.task import is_done, delay_task
+from django.http import HttpResponse, Http404
+from celery.task import tasks, is_done, apply_async
 from celery.result import AsyncResult
-from carrot.serialization import serialize as JSON_dump
+from anyjson import serialize as JSON_dump
+
+
+def apply(request, task_name, *args):
+    """View applying a task.
+
+    Example:
+        http://e.com/celery/apply/task_name/arg1/arg2//?kwarg1=a&kwarg2=b
+
+    **NOTE** Use with caution, preferably not make this publicly accessible
+    without ensuring your code is safe!
+
+    """
+    kwargs = request.method == "POST" and \
+            request.POST.copy() or request.GET.copy()
+    kwargs = dict((key.encode("utf-8"), value)
+                    for key, value in kwargs.items())
+    if task_name not in tasks:
+        raise Http404("apply: no such task")
+
+    task = tasks[task_name]
+    result = apply_async(task, args=args, kwargs=kwargs)
+    return JSON_dump({"ok": "true", "task_id": result.task_id})
 
 
 def is_task_done(request, task_id):

celery/worker.py

@@ -1,442 +0,0 @@
-"""celery.worker"""
-from carrot.connection import DjangoAMQPConnection
-from celery.messaging import get_consumer_set
-from celery.conf import DAEMON_CONCURRENCY, DAEMON_LOG_FILE
-from celery.conf import SEND_CELERY_TASK_ERROR_EMAILS
-from celery.log import setup_logger
-from celery.registry import tasks
-from celery.pool import TaskPool
-from celery.datastructures import ExceptionInfo
-from celery.backends import default_backend, default_periodic_status_backend
-from celery.timer import EventTimer
-from django.core.mail import mail_admins
-from celery.monitoring import TaskTimerStats
-import multiprocessing
-import traceback
-import threading
-import logging
-import signal
-import socket
-import time
-import sys
-
-
-# pep8.py borks on a inline signature separator and
-# says "trailing whitespace" ;)
-EMAIL_SIGNATURE_SEP = "-- "
-TASK_FAIL_EMAIL_BODY = """
-Task %%(name)s with id %%(id)s raised exception: %%(exc)s
-
-The contents of the full traceback was:
-
-%%(traceback)s
-
-%(EMAIL_SIGNATURE_SEP)s
-Just thought I'd let you know!
-celeryd at %%(hostname)s.
-""" % {"EMAIL_SIGNATURE_SEP": EMAIL_SIGNATURE_SEP}
-
-
-class UnknownTask(Exception):
-    """Got an unknown task in the queue. The message is requeued and
-    ignored."""
-
-
-def jail(task_id, task_name, func, args, kwargs):
-    """Wraps the task in a jail, which catches all exceptions, and
-    saves the status and result of the task execution to the task
-    meta backend.
-
-    If the call was successful, it saves the result to the task result
-    backend, and sets the task status to ``"DONE"``.
-
-    If the call results in an exception, it saves the exception as the task
-    result, and sets the task status to ``"FAILURE"``.
-
-    :param task_id: The id of the task.
-    :param task_name: The name of the task.
-    :param func: Callable object to execute.
-    :param args: List of positional args to pass on to the function.
-    :param kwargs: Keyword arguments mapping to pass on to the function.
-
-    :returns: the function return value on success, or
-        the exception instance on failure.
-
-    """
-    ignore_result = getattr(func, "ignore_result", False)
-    timer_stat = TaskTimerStats.start(task_id, task_name, args, kwargs)
-
-     Backend process cleanup
-    default_backend.process_cleanup()
-
-    try:
-        result = func(*args, **kwargs)
-    except (SystemExit, KeyboardInterrupt):
-        raise
-    except Exception, exc:
-        default_backend.mark_as_failure(task_id, exc)
-        retval = ExceptionInfo(sys.exc_info())
-    else:
-        if not ignore_result:
-            default_backend.mark_as_done(task_id, result)
-        retval = result
-    finally:
-        timer_stat.stop()
-
-    return retval
-
-
-class TaskWrapper(object):
-    """Class wrapping a task to be run.
-
-    :param task_name: see :attr:`task_name`.
-
-    :param task_id: see :attr:`task_id`.
-
-    :param task_func: see :attr:`task_func`
-
-    :param args: see :attr:`args`
-
-    :param kwargs: see :attr:`kwargs`.
-
-    .. attribute:: task_name
-
-        Kind of task. Must be a name registered in the task registry.
-
-    .. attribute:: task_id
-
-        UUID of the task.
-
-    .. attribute:: task_func
-
-        The tasks callable object.
-
-    .. attribute:: args
-
-        List of positional arguments to apply to the task.
-
-    .. attribute:: kwargs
-
-        Mapping of keyword arguments to apply to the task.
-
-    .. attribute:: message
-
-        The original message sent. Used for acknowledging the message.
-
-    """
-    success_msg = "Task %(name)s[%(id)s] processed: %(return_value)s"
-    fail_msg = """
-        Task %(name)s[%(id)s] raised exception: %(exc)s\n%(traceback)s
-    """
-    fail_email_subject = """
-        [celery@%(hostname)s] Error: Task %(name)s (%(id)s): %(exc)s
-    """
-    fail_email_body = TASK_FAIL_EMAIL_BODY
-
-    def __init__(self, task_name, task_id, task_func, args, kwargs,
-            on_acknowledge=None, **opts):
-        self.task_name = task_name
-        self.task_id = task_id
-        self.task_func = task_func
-        self.args = args
-        self.kwargs = kwargs
-        self.logger = kwargs.get("logger")
-        self.on_acknowledge = on_acknowledge
-        for opt in ("success_msg", "fail_msg", "fail_email_subject",
-                "fail_email_body"):
-            setattr(self, opt, opts.get(opt, getattr(self, opt, None)))
-        if not self.logger:
-            self.logger = multiprocessing.get_logger()
-
-    def __repr__(self):
-        return '<%s: {name:"%s", id:"%s", args:"%s", kwargs:"%s"}>' % (
-                self.__class__.__name__,
-                self.task_name, self.task_id,
-                self.args, self.kwargs)
-
-    @classmethod
-    def from_message(cls, message, message_data, logger):
-        """Create a :class:`TaskWrapper` from a task message sent by
-        :class:`celery.messaging.TaskPublisher`.
-
-        :raises UnknownTask: if the message does not describe a task,
-            the message is also rejected.
-
-        :returns: :class:`TaskWrapper` instance.
-
-        """
-        task_name = message_data["task"]
-        task_id = message_data["id"]
-        args = message_data["args"]
-        kwargs = message_data["kwargs"]
-
-        # Convert any unicode keys in the keyword arguments to ascii.
-        kwargs = dict([(key.encode("utf-8"), value)
-                    for key, value in kwargs.items()])
-
-        if task_name not in tasks:
-            raise UnknownTask(task_name)
-        task_func = tasks[task_name]
-        return cls(task_name, task_id, task_func, args, kwargs,
-                    on_acknowledge=message.ack, logger=logger)
-
-    def extend_with_default_kwargs(self, loglevel, logfile):
-        """Extend the tasks keyword arguments with standard task arguments.
-
-        These are ``logfile``, ``loglevel``, ``task_id`` and ``task_name``.
-
-        """
-        task_func_kwargs = {"logfile": logfile,
-                            "loglevel": loglevel,
-                            "task_id": self.task_id,
-                            "task_name": self.task_name}
-        task_func_kwargs.update(self.kwargs)
-        return task_func_kwargs
-
-    def execute(self, loglevel=None, logfile=None):
-        """Execute the task in a :func:`jail` and store return value
-        and status in the task meta backend.
-
-        :keyword loglevel: The loglevel used by the task.
-
-        :keyword logfile: The logfile used by the task.
-
-        """
-        task_func_kwargs = self.extend_with_default_kwargs(loglevel, logfile)
-        if self.on_acknowledge:
-            self.on_acknowledge()
-        return jail(self.task_id, self.task_name, [
-                        self.task_func, self.args, task_func_kwargs])
-
-    def on_success(self, ret_value, meta):
-        """The handler used if the task was successfully processed (
-        without raising an exception)."""
-        task_id = meta.get("task_id")
-        task_name = meta.get("task_name")
-        msg = self.success_msg.strip() % {
-                "id": task_id,
-                "name": task_name,
-                "return_value": ret_value}
-        self.logger.info(msg)
-
-    def on_failure(self, exc_info, meta):
-        """The handler used if the task raised an exception."""
-        task_id = meta.get("task_id")
-        task_name = meta.get("task_name")
-        context = {
-            "hostname": socket.gethostname(),
-            "id": task_id,
-            "name": task_name,
-            "exc": exc_info.exception,
-            "traceback": exc_info.traceback,
-        }
-        self.logger.error(self.fail_msg.strip() % context)
-
-        task_obj = tasks.get(task_name, object)
-        send_error_email = SEND_CELERY_TASK_ERROR_EMAILS and not \
-                getattr(task_obj, "disable_error_emails", False)
-        if send_error_email:
-            subject = self.fail_email_subject.strip() % context
-            body = self.fail_email_body.strip() % context
-            mail_admins(subject, body, fail_silently=True)
-
-    def execute_using_pool(self, pool, loglevel=None, logfile=None):
-        """Like :meth:`execute`, but using the :mod:`multiprocessing` pool.
-
-        :param pool: A :class:`multiprocessing.Pool` instance.
-
-        :keyword loglevel: The loglevel used by the task.
-
-        :keyword logfile: The logfile used by the task.
-
-        :returns :class:`multiprocessing.AsyncResult` instance.
-
-        """
-        task_func_kwargs = self.extend_with_default_kwargs(loglevel, logfile)
-        jail_args = [self.task_id, self.task_name, self.task_func,
-                     self.args, task_func_kwargs]
-        return pool.apply_async(jail, args=jail_args,
-                callbacks=[self.on_success], errbacks=[self.on_failure],
-                on_acknowledge=self.on_acknowledge,
-                meta={"task_id": self.task_id, "task_name": self.task_name})
-
-
-class PeriodicWorkController(threading.Thread):
-    """A thread that continuously checks if there are
-    :class:`celery.task.PeriodicTask` tasks waiting for execution,
-    and executes them.
-
-    Example:
-
-        >>> PeriodicWorkController().start()
-
-    """
-
-    def __init__(self):
-        super(PeriodicWorkController, self).__init__()
-        self._shutdown = threading.Event()
-        self._stopped = threading.Event()
-
-    def run(self):
-        """Run when you use :meth:`Thread.start`"""
-        while True:
-            if self._shutdown.isSet():
-                break
-            default_periodic_status_backend.run_periodic_tasks()
-            time.sleep(1)
-        self._stopped.set() # indicate that we are stopped
-
-    def stop(self):
-        """Shutdown the thread."""
-        self._shutdown.set()
-        self._stopped.wait() # block until this thread is done
-
-
-class WorkController(object):
-    """Executes tasks waiting in the task queue.
-
-    :param concurrency: see :attr:`concurrency`.
-
-    :param logfile: see :attr:`logfile`.
-
-    :param loglevel: see :attr:`loglevel`.
-
-
-    .. attribute:: concurrency
-
-        The number of simultaneous processes doing work (default:
-        :const:`celery.conf.DAEMON_CONCURRENCY`)
-
-    .. attribute:: loglevel
-
-        The loglevel used (default: :const:`logging.INFO`)
-
-    .. attribute:: logfile
-
-        The logfile used, if no logfile is specified it uses ``stderr``
-        (default: :const:`celery.conf.DAEMON_LOG_FILE`).
-
-    .. attribute:: logger
-
-        The :class:`logging.Logger` instance used for logging.
-
-    .. attribute:: pool
-
-        The :class:`multiprocessing.Pool` instance used.
-
-    .. attribute:: task_consumer
-
-        The :class:`carrot.messaging.ConsumerSet` instance used.
-
-    """
-    loglevel = logging.ERROR
-    concurrency = DAEMON_CONCURRENCY
-    logfile = DAEMON_LOG_FILE
-    _state = None
-
-    def __init__(self, concurrency=None, logfile=None, loglevel=None,
-            is_detached=False):
-        self.loglevel = loglevel or self.loglevel
-        self.concurrency = concurrency or self.concurrency
-        self.logfile = logfile or self.logfile
-        self.logger = setup_logger(loglevel, logfile)
-        self.pool = TaskPool(self.concurrency, logger=self.logger)
-        self.periodicworkcontroller = PeriodicWorkController()
-        self.is_detached = is_detached
-        self.amqp_connection = None
-        self.task_consumer = None
-
-    def close_connection(self):
-        """Close the AMQP connection."""
-        if self.task_consumer:
-            self.task_consumer.close()
-        if self.amqp_connection:
-            self.amqp_connection.close()
-
-    def reset_connection(self):
-        """Reset the AMQP connection, and reinitialize the
-        :class:`celery.messaging.TaskConsumer` instance.
-
-        Resets the task consumer in :attr:`task_consumer`.
-
-        """
-        self.close_connection()
-        self.amqp_connection = DjangoAMQPConnection()
-        self.task_consumer = get_consumer_set(connection=self.amqp_connection)
-        self.task_consumer.register_callback(self._message_callback)
-        return self.task_consumer
-
-    def connection_diagnostics(self):
-        """Diagnose the AMQP connection, and reset connection if
-        necessary."""
-        connection = self.task_consumer.backend.channel.connection
-
-        if not connection:
-            self.logger.info(
-                    "AMQP Connection has died, restoring connection.")
-            self.reset_connection()
-
-    def _message_callback(self, message_data, message):
-        """The method called when we receive a message."""
-        try:
-            try:
-                self.process_task(message_data, message)
-            except ValueError:
-                # execute_next_task didn't return a r/name/id tuple,
-                # probably because it got an exception.
-                pass
-            except UnknownTask, exc:
-                self.logger.info("Unknown task ignored: %s" % (exc))
-            except Exception, exc:
-                self.logger.critical("Message queue raised %s: %s\n%s" % (
-                                exc.__class__, exc, traceback.format_exc()))
-        except (SystemExit, KeyboardInterrupt):
-            self.shutdown()
-
-    def process_task(self, message_data, message):
-        """Process task message by passing it to the pool of workers."""
-        task = TaskWrapper.from_message(message, message_data,
-                                        logger=self.logger)
-        self.logger.info("Got task from broker: %s[%s]" % (
-            task.task_name, task.task_id))
-        self.logger.debug("Got a task: %s. Trying to execute it..." % task)
-
-        result = task.execute_using_pool(self.pool, self.loglevel,
-                                         self.logfile)
-
-        self.logger.debug("Task %s has been executed asynchronously." % task)
-
-        return result
-
-    def shutdown(self):
-        """Make sure ``celeryd`` exits cleanly."""
-        # shut down the periodic work controller thread
-        if self._state != "RUN":
-            return
-        self._state = "TERMINATE"
-        self.periodicworkcontroller.stop()
-        self.pool.terminate()
-        self.close_connection()
-
-    def run(self):
-        """Starts the workers main loop."""
-        self._state = "RUN"
-        task_consumer = self.reset_connection()
-        it = task_consumer.iterconsume(limit=None)
-
-        self.pool.run()
-        self.periodicworkcontroller.start()
-
-        # If not running as daemon, and DEBUG logging level is enabled,
-        # print pool PIDs and sleep for a second before we start.
-        if self.logger.isEnabledFor(logging.DEBUG):
-            self.logger.debug("Pool child processes: [%s]" % (
-                "|".join(map(str, self.pool.get_worker_pids()))))
-            if not self.is_detached:
-                time.sleep(1)
-
-        try:
-            while True:
-                it.next()
-        except (SystemExit, KeyboardInterrupt):
-            self.shutdown()

celery/worker/__init__.py

@@ -0,0 +1,249 @@
+"""
+
+The Multiprocessing Worker Server
+
+Documentation for this module is in ``docs/reference/celery.worker.rst``.
+
+"""
+from carrot.connection import DjangoAMQPConnection
+from celery.worker.controllers import Mediator, PeriodicWorkController
+from celery.worker.job import TaskWrapper
+from celery.registry import NotRegistered
+from celery.messaging import TaskConsumer
+from celery.conf import DAEMON_CONCURRENCY, DAEMON_LOG_FILE
+from celery.log import setup_logger
+from celery.pool import TaskPool
+from Queue import Queue
+import traceback
+import logging
+
+
+class AMQPListener(object):
+    """Listen for messages received from the AMQP broker and
+    move them the the bucket queue for task processing.
+
+    :param bucket_queue: See :attr:`bucket_queue`.
+    :param hold_queue: See :attr:`hold_queue`.
+
+    .. attribute:: bucket_queue
+
+        The queue that holds tasks ready for processing immediately.
+
+    .. attribute:: hold_queue
+
+        The queue that holds paused tasks. Reasons for being paused include
+        a countdown/eta or that it's waiting for retry.
+
+    .. attribute:: logger
+
+        The logger used.
+
+    """
+
+    def __init__(self, bucket_queue, hold_queue, logger):
+        self.amqp_connection = None
+        self.task_consumer = None
+        self.bucket_queue = bucket_queue
+        self.hold_queue = hold_queue
+        self.logger = logger
+
+    def start(self):
+        """Start processing AMQP messages."""
+        task_consumer = self.reset_connection()
+
+        self.logger.debug("AMQPListener: Starting message consumer...")
+        it = task_consumer.iterconsume(limit=None)
+
+        self.logger.debug("AMQPListener: Ready to accept tasks!")
+
+        while True:
+            it.next()
+
+    def stop(self):
+        """Stop processing AMQP messages and close the connection
+        to the broker."""
+        self.close_connection()
+
+    def receive_message(self, message_data, message):
+        """The callback called when a new message is received.
+
+        If the message has an ``eta`` we move it to the hold queue,
+        otherwise we move it the bucket queue for immediate processing.
+
+        """
+        try:
+            task = TaskWrapper.from_message(message, message_data,
+                                            logger=self.logger)
+        except NotRegistered, exc:
+            self.logger.error("Unknown task ignored: %s" % (exc))
+            return
+
+        eta = message_data.get("eta")
+        if eta:
+            self.logger.info("Got task from broker: %s[%s] eta:[%s]" % (
+                    task.task_name, task.task_id, eta))
+            self.hold_queue.put((task, eta))
+        else:
+            self.logger.info("Got task from broker: %s[%s]" % (
+                    task.task_name, task.task_id))
+            self.bucket_queue.put(task)
+
+    def close_connection(self):
+        """Close the AMQP connection."""
+        if self.task_consumer:
+            self.task_consumer.close()
+            self.task_consumer = None
+        if self.amqp_connection:
+            self.logger.debug(
+                    "AMQPListener: Closing connection to the broker...")
+            self.amqp_connection.close()
+            self.amqp_connection = None
+
+    def reset_connection(self):
+        """Reset the AMQP connection, and reinitialize the
+        :class:`celery.messaging.TaskConsumer` instance.
+
+        Resets the task consumer in :attr:`task_consumer`.
+
+        """
+        self.logger.debug(
+                "AMQPListener: Re-establishing connection to the broker...")
+        self.close_connection()
+        self.amqp_connection = DjangoAMQPConnection()
+        self.task_consumer = TaskConsumer(connection=self.amqp_connection)
+        self.task_consumer.register_callback(self.receive_message)
+        return self.task_consumer
+
+
+class WorkController(object):
+    """Executes tasks waiting in the task queue.
+
+    :param concurrency: see :attr:`concurrency`.
+    :param logfile: see :attr:`logfile`.
+    :param loglevel: see :attr:`loglevel`.
+
+
+    .. attribute:: concurrency
+
+        The number of simultaneous processes doing work (default:
+        :const:`celery.conf.DAEMON_CONCURRENCY`)
+
+    .. attribute:: loglevel
+
+        The loglevel used (default: :const:`logging.INFO`)
+
+    .. attribute:: logfile
+
+        The logfile used, if no logfile is specified it uses ``stderr``
+        (default: :const:`celery.conf.DAEMON_LOG_FILE`).
+
+    .. attribute:: logger
+
+        The :class:`logging.Logger` instance used for logging.
+
+    .. attribute:: is_detached
+
+        Flag describing if the worker is running as a daemon or not.
+
+    .. attribute:: pool
+
+        The :class:`multiprocessing.Pool` instance used.
+
+    .. attribute:: bucket_queue
+
+        The :class:`Queue.Queue` that holds tasks ready for immediate
+        processing.
+
+    .. attribute:: hold_queue
+
+        The :class:`Queue.Queue` that holds paused tasks. Reasons for holding
+        back the task include waiting for ``eta`` to pass or the task is being
+        retried.
+
+    .. attribute:: periodic_work_controller
+
+        Instance of :class:`celery.worker.controllers.PeriodicWorkController`.
+
+    .. attribute:: mediator
+
+        Instance of :class:`celery.worker.controllers.Mediator`.
+
+    .. attribute:: amqp_listener
+
+        Instance of :class:`AMQPListener`.
+
+    """
+    loglevel = logging.ERROR
+    concurrency = DAEMON_CONCURRENCY
+    logfile = DAEMON_LOG_FILE
+    _state = None
+
+    def __init__(self, concurrency=None, logfile=None, loglevel=None,
+            is_detached=False):
+
+        # Options
+        self.loglevel = loglevel or self.loglevel
+        self.concurrency = concurrency or self.concurrency
+        self.logfile = logfile or self.logfile
+        self.is_detached = is_detached
+        self.logger = setup_logger(loglevel, logfile)
+
+        # Queues
+        self.bucket_queue = Queue()
+        self.hold_queue = Queue()
+
+        self.logger.debug("Instantiating thread components...")
+
+        # Threads+Pool
+        self.periodic_work_controller = PeriodicWorkController(
+                                                    self.bucket_queue,
+                                                    self.hold_queue)
+        self.pool = TaskPool(self.concurrency, logger=self.logger)
+        self.amqp_listener = AMQPListener(self.bucket_queue, self.hold_queue,
+                                          logger=self.logger)
+        self.mediator = Mediator(self.bucket_queue, self.safe_process_task)
+
+        # The order is important here;
+        #   the first in the list is the first to start,
+        # and they must be stopped in reverse order.
+        self.components = [self.pool,
+                           self.mediator,
+                           self.periodic_work_controller,
+                           self.amqp_listener]
+
+    def start(self):
+        """Starts the workers main loop."""
+        self._state = "RUN"
+
+        try:
+            for component in self.components:
+                self.logger.debug("Starting thread %s..." % \
+                        component.__class__.__name__)
+                component.start()
+        finally:
+            self.stop()
+
+    def safe_process_task(self, task):
+        """Same as :meth:`process_task`, but catches all exceptions
+        the task raises and log them as errors, to make sure the
+        worker doesn't die."""
+        try:
+            try:
+                self.process_task(task)
+            except Exception, exc:
+                self.logger.critical("Internal error %s: %s\n%s" % (
+                                exc.__class__, exc, traceback.format_exc()))
+        except (SystemExit, KeyboardInterrupt):
+            self.stop()
+
+    def process_task(self, task):
+        """Process task by sending it to the pool of workers."""
+        task.execute_using_pool(self.pool, self.loglevel, self.logfile)
+
+    def stop(self):
+        """Gracefully shutdown the worker server."""
+        # shut down the periodic work controller thread
+        if self._state != "RUN":
+            return
+
+        [component.stop() for component in reversed(self.components)]

celery/worker/controllers.py

@@ -0,0 +1,157 @@
+"""
+
+Worker Controller Threads
+
+"""
+from celery.backends import default_periodic_status_backend
+from Queue import Empty as QueueEmpty
+from datetime import datetime
+from multiprocessing import get_logger
+import threading
+import time
+
+
+class BackgroundThread(threading.Thread):
+    """Thread running an infinite loop which for every iteration
+    calls its :meth:`on_iteration` method.
+
+    This also implements graceful shutdown of the thread by providing
+    the :meth:`stop` method.
+
+    """
+    is_infinite = True
+
+    def __init__(self):
+        super(BackgroundThread, self).__init__()
+        self._shutdown = threading.Event()
+        self._stopped = threading.Event()
+        self.setDaemon(True)
+
+    def run(self):
+        """This is the body of the thread.
+
+        To start the thread use :meth:`start` instead.
+
+        """
+        self.on_start()
+
+        while self.is_infinite:
+            if self._shutdown.isSet():
+                break
+            self.on_iteration()
+        self._stopped.set() # indicate that we are stopped
+
+    def on_start(self):
+        """This handler is run at thread start, just before the infinite
+        loop."""
+        pass
+
+    def on_iteration(self):
+        """This is the method called for every iteration and must be
+        implemented by every subclass of :class:`BackgroundThread`."""
+        raise NotImplementedError(
+                "InfiniteThreads must implement on_iteration")
+
+    def on_stop(self):
+        """This handler is run when the thread is shutdown."""
+        pass
+
+    def stop(self):
+        """Gracefully shutdown the thread."""
+        self.on_stop()
+        self._shutdown.set()
+        self._stopped.wait() # block until this thread is done
+
+
+class Mediator(BackgroundThread):
+    """Thread continuously sending tasks in the queue to the pool.
+
+    .. attribute:: bucket_queue
+
+        The task queue, a :class:`Queue.Queue` instance.
+
+    .. attribute:: callback
+
+        The callback used to process tasks retrieved from the
+        :attr:`bucket_queue`.
+
+    """
+
+    def __init__(self, bucket_queue, callback):
+        super(Mediator, self).__init__()
+        self.bucket_queue = bucket_queue
+        self.callback = callback
+
+    def on_iteration(self):
+        logger = get_logger()
+        try:
+            logger.debug("Mediator: Trying to get message from bucket_queue")
+            # This blocks until there's a message in the queue.
+            task = self.bucket_queue.get(timeout=1)
+        except QueueEmpty:
+            logger.debug("Mediator: Bucket queue is empty.")
+            pass
+        else:
+            logger.debug("Mediator: Running callback for task: %s[%s]" % (
+                task.task_name, task.task_id))
+            self.callback(task)
+
+
+class PeriodicWorkController(BackgroundThread):
+    """A thread that continuously checks if there are
+    :class:`celery.task.PeriodicTask` tasks waiting for execution,
+    and executes them. It also finds tasks in the hold queue that is
+    ready for execution and moves them to the bucket queue.
+
+    (Tasks in the hold queue are tasks waiting for retry, or with an
+    ``eta``/``countdown``.)
+
+    """
+
+    def __init__(self, bucket_queue, hold_queue):
+        super(PeriodicWorkController, self).__init__()
+        self.hold_queue = hold_queue
+        self.bucket_queue = bucket_queue
+
+    def on_start(self):
+        """Do backend-specific periodic task initialization."""
+        default_periodic_status_backend.init_periodic_tasks()
+
+    def on_iteration(self):
+        logger = get_logger()
+        logger.debug("PeriodicWorkController: Running periodic tasks...")
+        self.run_periodic_tasks()
+        logger.debug("PeriodicWorkController: Processing hold queue...")
+        self.process_hold_queue()
+        logger.debug("PeriodicWorkController: Going to sleep...")
+        time.sleep(1)
+
+    def run_periodic_tasks(self):
+        logger = get_logger()
+        applied = default_periodic_status_backend.run_periodic_tasks()
+        for task, task_id in applied:
+            logger.debug(
+                "PeriodicWorkController: Periodic task %s applied (%s)" % (
+                    task.name, task_id))
+
+    def process_hold_queue(self):
+        """Finds paused tasks that are ready for execution and move
+        them to the :attr:`bucket_queue`."""
+        logger = get_logger()
+        try:
+            logger.debug(
+                "PeriodicWorkController: Getting next task from hold queue..")
+            task, eta = self.hold_queue.get_nowait()
+        except QueueEmpty:
+            logger.debug("PeriodicWorkController: Hold queue is empty")
+            return
+        if datetime.now() >= eta:
+            logger.debug(
+                "PeriodicWorkController: Time to run %s[%s] (%s)..." % (
+                    task.task_name, task.task_id, eta))
+            self.bucket_queue.put(task)
+        else:
+            logger.debug(
+                "PeriodicWorkController: ETA not ready for %s[%s] (%s)..." % (
+                    task.task_name, task.task_id, eta))
+            self.hold_queue.put((task, eta))

celery/worker/job.py

@@ -0,0 +1,278 @@
+"""
+
+Jobs Executable by the Worker Server.
+
+"""
+from celery.registry import tasks, NotRegistered
+from celery.datastructures import ExceptionInfo
+from celery.backends import default_backend
+from django.core.mail import mail_admins
+from celery.monitoring import TaskTimerStats
+import multiprocessing
+import traceback
+import socket
+import sys
+
+
+# pep8.py borks on a inline signature separator and
+# says "trailing whitespace" ;)
+EMAIL_SIGNATURE_SEP = "-- "
+TASK_FAIL_EMAIL_BODY = """
+Task %%(name)s with id %%(id)s raised exception: %%(exc)s
+
+
+Task was called with args:%%(args)s kwargs:%%(kwargs)s.
+The contents of the full traceback was:
+
+%%(traceback)s
+
+%(EMAIL_SIGNATURE_SEP)s
+Just thought I'd let you know!
+celeryd at %%(hostname)s.
+""" % {"EMAIL_SIGNATURE_SEP": EMAIL_SIGNATURE_SEP}
+
+
+def jail(task_id, task_name, func, args, kwargs):
+    """Wraps the task in a jail, which catches all exceptions, and
+    saves the status and result of the task execution to the task
+    meta backend.
+
+    If the call was successful, it saves the result to the task result
+    backend, and sets the task status to ``"DONE"``.
+
+    If the call results in an exception, it saves the exception as the task
+    result, and sets the task status to ``"FAILURE"``.
+
+    :param task_id: The id of the task.
+    :param task_name: The name of the task.
+    :param func: Callable object to execute.
+    :param args: List of positional args to pass on to the function.
+    :param kwargs: Keyword arguments mapping to pass on to the function.
+
+    :returns: the function return value on success, or
+        the exception instance on failure.
+
+    """
+    ignore_result = getattr(func, "ignore_result", False)
+    timer_stat = TaskTimerStats.start(task_id, task_name, args, kwargs)
+
+    # See: http://groups.google.com/group/django-users/browse_thread/
+    #       thread/78200863d0c07c6d/38402e76cf3233e8?hl=en&lnk=gst&
+    #       q=multiprocessing#38402e76cf3233e8
+    from django.db import connection
+    connection.close()
+
+    # Reset cache connection only if using memcached/libmemcached
+    from django.core import cache
+    # XXX At Opera we use a custom memcached backend that uses libmemcached
+    # instead of libmemcache (cmemcache). Should find a better solution for
+    # this, but for now "memcached" should probably be unique enough of a
+    # string to not make problems.
+    cache_backend = cache.settings.CACHE_BACKEND
+    if hasattr(cache, "parse_backend_uri"):
+        cache_scheme = cache.parse_backend_uri(cache_backend)[0]
+    else:
+        # Django <= 1.0.2
+        cache_scheme = cache_backend.split(":", 1)[0]
+    if "memcached" in cache_scheme:
+        cache.cache.close()
+
+    # Backend process cleanup
+    default_backend.process_cleanup()
+
+    try:
+        result = func(*args, **kwargs)
+    except (SystemExit, KeyboardInterrupt):
+        raise
+    except Exception, exc:
+        stored_exc = default_backend.mark_as_failure(task_id, exc)
+        type_, _, tb = sys.exc_info()
+        retval = ExceptionInfo((type_, stored_exc, tb))
+    else:
+        if not ignore_result:
+            default_backend.mark_as_done(task_id, result)
+        retval = result
+    finally:
+        timer_stat.stop()
+
+    return retval
+
+
+class TaskWrapper(object):
+    """Class wrapping a task to be run.
+
+    :param task_name: see :attr:`task_name`.
+
+    :param task_id: see :attr:`task_id`.
+
+    :param task_func: see :attr:`task_func`
+
+    :param args: see :attr:`args`
+
+    :param kwargs: see :attr:`kwargs`.
+
+    .. attribute:: task_name
+
+        Kind of task. Must be a name registered in the task registry.
+
+    .. attribute:: task_id
+
+        UUID of the task.
+
+    .. attribute:: task_func
+
+        The tasks callable object.
+
+    .. attribute:: args
+
+        List of positional arguments to apply to the task.
+
+    .. attribute:: kwargs
+
+        Mapping of keyword arguments to apply to the task.
+
+    .. attribute:: message
+
+        The original message sent. Used for acknowledging the message.
+
+    """
+    success_msg = "Task %(name)s[%(id)s] processed: %(return_value)s"
+    fail_msg = """
+        Task %(name)s[%(id)s] raised exception: %(exc)s\n%(traceback)s
+    """
+    fail_email_subject = """
+        [celery@%(hostname)s] Error: Task %(name)s (%(id)s): %(exc)s
+    """
+    fail_email_body = TASK_FAIL_EMAIL_BODY
+
+    def __init__(self, task_name, task_id, task_func, args, kwargs,
+            on_ack=None, **opts):
+        self.task_name = task_name
+        self.task_id = task_id
+        self.task_func = task_func
+        self.args = args
+        self.kwargs = kwargs
+        self.logger = kwargs.get("logger")
+        self.on_ack = on_ack
+        for opt in ("success_msg", "fail_msg", "fail_email_subject",
+                "fail_email_body"):
+            setattr(self, opt, opts.get(opt, getattr(self, opt, None)))
+        if not self.logger:
+            self.logger = multiprocessing.get_logger()
+
+    def __repr__(self):
+        return '<%s: {name:"%s", id:"%s", args:"%s", kwargs:"%s"}>' % (
+                self.__class__.__name__,
+                self.task_name, self.task_id,
+                self.args, self.kwargs)
+
+    @classmethod
+    def from_message(cls, message, message_data, logger=None):
+        """Create a :class:`TaskWrapper` from a task message sent by
+        :class:`celery.messaging.TaskPublisher`.
+
+        :raises UnknownTaskError: if the message does not describe a task,
+            the message is also rejected.
+
+        :returns: :class:`TaskWrapper` instance.
+
+        """
+        task_name = message_data["task"]
+        task_id = message_data["id"]
+        args = message_data["args"]
+        kwargs = message_data["kwargs"]
+
+        # Convert any unicode keys in the keyword arguments to ascii.
+        kwargs = dict((key.encode("utf-8"), value)
+                        for key, value in kwargs.items())
+
+        if task_name not in tasks:
+            raise NotRegistered(task_name)
+        task_func = tasks[task_name]
+        return cls(task_name, task_id, task_func, args, kwargs,
+                    on_ack=message.ack, logger=logger)
+
+    def extend_with_default_kwargs(self, loglevel, logfile):
+        """Extend the tasks keyword arguments with standard task arguments.
+
+        These are ``logfile``, ``loglevel``, ``task_id`` and ``task_name``.
+
+        """
+        task_func_kwargs = {"logfile": logfile,
+                            "loglevel": loglevel,
+                            "task_id": self.task_id,
+                            "task_name": self.task_name}
+        task_func_kwargs.update(self.kwargs)
+        return task_func_kwargs
+
+    def execute(self, loglevel=None, logfile=None):
+        """Execute the task in a :func:`jail` and store return value
+        and status in the task meta backend.
+
+        :keyword loglevel: The loglevel used by the task.
+
+        :keyword logfile: The logfile used by the task.
+
+        """
+        task_func_kwargs = self.extend_with_default_kwargs(loglevel, logfile)
+        # acknowledge task as being processed.
+        if self.on_ack:
+            self.on_ack()
+        return jail(self.task_id, self.task_name, self.task_func,
+                    self.args, task_func_kwargs)
+
+    def on_success(self, ret_value, meta):
+        """The handler used if the task was successfully processed (
+        without raising an exception)."""
+        task_id = meta.get("task_id")
+        task_name = meta.get("task_name")
+        msg = self.success_msg.strip() % {
+                "id": task_id,
+                "name": task_name,
+                "return_value": ret_value}
+        self.logger.info(msg)
+
+    def on_failure(self, exc_info, meta):
+        """The handler used if the task raised an exception."""
+        from celery.conf import SEND_CELERY_TASK_ERROR_EMAILS
+
+        task_id = meta.get("task_id")
+        task_name = meta.get("task_name")
+        context = {
+            "hostname": socket.gethostname(),
+            "id": task_id,
+            "name": task_name,
+            "exc": exc_info.exception,
+            "traceback": exc_info.traceback,
+            "args": self.args,
+            "kwargs": self.kwargs,
+        }
+        self.logger.error(self.fail_msg.strip() % context)
+
+        task_obj = tasks.get(task_name, object)
+        send_error_email = SEND_CELERY_TASK_ERROR_EMAILS and not \
+                getattr(task_obj, "disable_error_emails", False)
+        if send_error_email:
+            subject = self.fail_email_subject.strip() % context
+            body = self.fail_email_body.strip() % context
+            mail_admins(subject, body, fail_silently=True)
+
+    def execute_using_pool(self, pool, loglevel=None, logfile=None):
+        """Like :meth:`execute`, but using the :mod:`multiprocessing` pool.
+
+        :param pool: A :class:`multiprocessing.Pool` instance.
+
+        :keyword loglevel: The loglevel used by the task.
+
+        :keyword logfile: The logfile used by the task.
+
+        :returns :class:`multiprocessing.AsyncResult` instance.
+
+        """
+        task_func_kwargs = self.extend_with_default_kwargs(loglevel, logfile)
+        jail_args = [self.task_id, self.task_name, self.task_func,
+                     self.args, task_func_kwargs]
+        return pool.apply_async(jail, args=jail_args,
+                callbacks=[self.on_success], errbacks=[self.on_failure],
+                on_ack=self.on_ack,
+                meta={"task_id": self.task_id, "task_name": self.task_name})

contrib/bump

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Bump version of python package in current directory.
+# Updates version in package/__init__.py, and version embedded as
+# reStructuredtext in README.
+#
+# Usage: BUMP package_name [new_version] [-c]
+# If new_version is not specified the release part of the version will
+# be incremented.
+# if -c is set it will be commited and pushed.
+
+bump_version () {
+    commit=0
+    while getopts "c" flag; do
+        case $flag in
+            c)
+                commit=1
+            ;;
+        esac
+    done
+    shift $(($OPTIND - 1))
+    package="$1"
+    new_version="$2"
+    [ $commit ] && git pull origin master
+    current=$(python -c "
+import $package
+print($package.__version__)
+    ")
+    cur_major=$(echo "$current" | cut -d. -f 1)
+    cur_minor=$(echo "$current" | cut -d. -f 2)
+    cur_release=$(echo "$current" | cut -d. -f 3)
+    if [ -z "$new_version" ]; then
+        new_version="$cur_major.$cur_minor.$(($cur_release + 1))";
+        new_as_tuple="($cur_major, $cur_minor, $(($cur_release + 1)))";
+    fi
+    new_major=$(echo "$new_version" | cut -d. -f 1)
+    new_minor=$(echo "$new_version" | cut -d. -f 2)
+    new_release=$(echo "$new_version" | cut -d. -f 3)
+
+    new_as_tuple="($new_major, $new_minor, $new_release)"
+
+    echo "$package: $current -> $new_version"
+
+    perl -pi -e"s/(VERSION\s*=\s*)\((.+?)\);?/\$1$new_as_tuple/" \
+        "$package/__init__.py"
+    perl -pi -e"s/(:Version:)\s*(.+?)(\s*$)/\$1 $new_version\$3/i" README
+
+    [ $commit ] && (
+        git commit "$package/__init__.py" README \
+            -m "Bumped version to $new_version";
+        git push;
+    )
+    
+}
+
+if [ -z "$1" ]; then
+    echo "Usage: $(basename $0) package_name [new_version]"
+    exit 1
+fi
+
+bump_version $*
+

contrib/doc4allmods

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+PACKAGE="$1"
+SKIP_PACKAGES="$PACKAGE tests management urls"
+SKIP_FILES="celery.bin.rst celery.serialization.rst"
+
+modules=$(find "$PACKAGE" -name "*.py")
+
+failed=0
+for module in $modules; do
+    dotted=$(echo $module | sed 's/\//\./g')
+    name=${dotted%.__init__.py}
+    name=${name%.py}
+    rst=$name.rst
+    skip=0
+    for skip_package in $SKIP_PACKAGES; do
+        [ $(echo "$name" | cut -d. -f 2) == "$skip_package" ] && skip=1
+    done
+    for skip_file in $SKIP_FILES; do
+        [ "$skip_file" == "$rst" ] && skip=1
+    done
+    
+    if [ $skip -eq 0 ]; then
+        if [ ! -f "docs/reference/$rst" ]; then
+            echo $rst :: FAIL
+            failed=1
+        fi
+    fi
+done
+
+exit $failed

contrib/find-unprocessed-tasks.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+#--------------------------------------------------------------------#
+# Find all currently unprocessed tasks by searching the celeryd
+# log file.
+#
+# Please note that this will also include tasks that raised an exception,
+# or is just under active processing (will finish soon).
+#
+# Usage:
+#
+#     # Using default log file /var/log/celeryd.log
+#     $ bash find-unprocessed-tasks.sh
+#
+#     # Using a custom logfile
+#     # bash find-unprocessed-tasks.sh ./celeryd.log
+# 
+#--------------------------------------------------------------------#
+
+DEFAULT_LOGFILE=/var/log/celeryd.log
+export CELERYD_LOGFILE=${1:-$DEFAULT_LOGFILE}
+
+
+get_start_date_by_task_id() {
+    task_id="$1"
+    grep Apply $CELERYD_LOGFILE | \
+        grep "$task_id" | \
+        perl -nle'
+            /^\[(.+?): DEBUG/; print $1' | \
+        sed 's/\s*$//'
+}
+
+
+get_end_date_by_task_id() {
+    task_id="$1"
+    grep processed $CELERYD_LOGFILE | \
+        grep "$task_id" | \
+        perl -nle'
+            /^\[(.+?): INFO/; print $1 ' | \
+        sed 's/\s*$//'
+}
+
+
+get_all_task_ids() {
+    grep Apply $CELERYD_LOGFILE | perl -nle"/'task_id': '(.+?)'/; print \$1"
+}
+
+
+search_logs_for_task_id() {
+    grep "$task_id" $CELERYD_LOGFILE
+}
+
+
+report_unprocessed_task() {
+    task_id="$1"
+    date_start="$2"
+
+    cat <<EOFTEXT
+"---------------------------------------------------------------------------------"
+| UNFINISHED TASK: $task_id [$date_start]
+"---------------------------------------------------------------------------------"
+Related logs:
+EOFTEXT
+	search_logs_for_task_id "$task_id"
+}
+
+for task_id in $(get_all_task_ids); do
+    date_start=$(get_start_date_by_task_id "$task_id")
+    date_end=$(get_end_date_by_task_id "$task_id")
+    if [ -z "$date_end" ]; then
+        report_unprocessed_task "$task_id" "$date_start"
+    fi
+done

contrib/periodic-task-runtimes.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+#---------------------------------------------------------------------------#
+# 
+# Tool to find race conditions in the Periodic Task system.
+# Outputs times of all runs of a certain task (by searching for task name
+# using a search query).
+#
+# Usage:
+#   
+#   $ bash periodic-task-runtimes.sh query host1 [host2 ... hostN]
+#
+# Example usage:
+#
+#   $ bash periodic-task-runtimes.sh refresh_all_feeds host1 host2 host3
+#
+# The output is sorted.
+#
+#---------------------------------------------------------------------------#
+
+USER="root"
+CELERYD_LOGFILE="/var/log/celeryd.log"
+
+query="$1"
+shift
+hosts="$*"
+
+usage () {
+    echo "$(basename $0) task_name_query host1 [host2 ... hostN]"
+    exit 1
+}
+
+[ -z "$query" -o -z "$hosts" ] && usage
+
+
+get_processed_date_for_task () {
+    host="$1"
+    ssh "$USER@$host" "
+        grep '$query' $CELERYD_LOGFILE | \
+            grep 'Got task from broker:' | \
+            perl -nle'
+                /^\[(.+?): INFO.+?Got task from broker:(.+?)\s*/;
+                print \"[\$1] $host \$2\"' | \
+            sed 's/\s*$//'
+    "
+}
+
+get_processed_for_all_hosts () {
+    for_hosts="$*"
+    for host in $for_hosts; do
+        get_processed_date_for_task $host
+    done
+}
+
+get_processed_for_all_hosts $hosts | sort

contrib/queuelog.py

@@ -0,0 +1,82 @@
+import sys
+from multiprocessing.queues import SimpleQueue
+from multiprocessing.process import Process
+from multiprocessing.pool import Pool
+
+
+class Logwriter(Process):
+
+    def start(self, log_queue, logfile="process.log"):
+        self.log_queue = log_queue
+        self.logfile = logfile
+        super(Logwriter, self).start()
+
+    def run(self):
+        self.process_logs(self.log_queue, self.logfile)
+
+    def process_logs(self, log_queue, logfile):
+        need_to_close_fh = False
+        logfh = logfile
+        if isinstance(logfile, basestring):
+            need_to_close_fh = True
+            logfh = open(logfile, "a")
+
+        logfh = open(logfile, "a")
+        while 1:
+            message = log_queue.get()
+            if message is None: # received sentinel
+                break
+            logfh.write(message)
+
+        log_queue.put(None) # cascade sentinel
+
+        if need_to_close_fh:
+            logfh.close()
+
+
+class QueueLogger(object):
+
+    def __init__(self, log_queue, log_process):
+        self.log_queue = log_queue
+        self.log_process = log_process
+
+    @classmethod
+    def start(cls):
+        log_queue = SimpleQueue()
+        log_process = Logwriter()
+        log_process.start(log_queue)
+        return cls(log_queue, log_process)
+
+    def write(self, message):
+        self.log_queue.put(message)
+
+    def stop(self):
+        self.log_queue.put(None) # send sentinel
+
+    def flush(self):
+        pass
+
+
+def some_process_body():
+    sys.stderr.write("Vandelay industries!\n")
+
+
+def setup_redirection():
+    queue_logger = QueueLogger.start()
+    sys.stderr = queue_logger
+    return queue_logger
+
+
+def main():
+    queue_logger = setup_redirection()
+    queue_logger.write("ABCDEF\n")
+    try:
+        p = Pool(10)
+        results = [p.apply_async(some_process_body) for i in xrange(20)]
+        [result.get() for result in results]
+        p.close()
+    finally:
+        queue_logger.stop()
+
+if __name__ == "__main__":
+    main()

contrib/testdynpool.py

@@ -0,0 +1,48 @@
+from celery.pool import DynamicPool
+from multiprocessing import get_logger, log_to_stderr
+import logging
+
+
+def setup_logger():
+    log_to_stderr()
+    logger = get_logger()
+    logger.setLevel(logging.DEBUG)
+    return logger
+
+
+def target(n):
+    r = n * n
+    setup_logger().info("%d * %d = %d" % (n, n, r))
+    return r
+
+
+def exit_process():
+    setup_logger().error("EXITING NOW!")
+    import os
+    os._exit(0)
+
+
+def send_exit(pool):
+    pool.apply_async(exit_process)
+
+
+def do_work(pool):
+    results = [pool.apply_async(target, args=[i]) for i in range(10)]
+    [result.get() for result in results]
+
+
+def workpool():
+    pool = DynamicPool(2)
+    do_work(pool)
+    print("GROWING")
+    pool.grow(1)
+    do_work(pool)
+    send_exit(pool)
+    import time
+    time.sleep(2)
+    pool.replace_dead_workers()
+    do_work(pool)
+
+
+if __name__ == "__main__":
+    workpool()

docs/Makefile

@@ -32,6 +32,12 @@ html:
 	@echo
 	@echo "Build finished. The HTML pages are in .build/html."
 
+coverage:
+	mkdir -p .build/coverage .build/doctrees
+	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) .build/coverage
+	@echo
+	@echo "Build finished."
+
 pickle:
 	mkdir -p .build/pickle .build/doctrees
 	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) .build/pickle

docs/conf.py

@@ -17,7 +17,7 @@ sys.path.append(os.path.join(os.path.dirname(__file__), "_ext"))
 # General configuration
 # ---------------------
 
-extensions = ['sphinx.ext.autodoc', 'djangodocs']
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.coverage', 'djangodocs']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['.templates']

docs/index.rst

@@ -12,6 +12,7 @@ Contents:
     :maxdepth: 3
 
     introduction
+    tutorials/index
     faq
     reference/index
     changelog

docs/reference/celery.execute.rst

@@ -0,0 +1,8 @@
+==================================
+ Executing Tasks - celery.execute
+==================================
+
+.. currentmodule:: celery.execute
+
+.. automodule:: celery.execute
+    :members:

docs/reference/celery.fields.rst

@@ -0,0 +1,8 @@
+===============================
+ Django Fields - celery.fields
+===============================
+
+.. currentmodule:: celery.fields
+
+.. automodule:: celery.fields
+    :members:

docs/reference/celery.supervisor.rst

@@ -0,0 +1,8 @@
+========================================
+ Process Supervisor - celery.supervisor
+========================================
+
+.. currentmodule:: celery.supervisor
+
+.. automodule:: celery.supervisor
+    :members:

docs/reference/celery.task.base.rst

@@ -0,0 +1,8 @@
+===================================
+ Defining Tasks - celery.task.base
+===================================
+
+.. currentmodule:: celery.task.base
+
+.. automodule:: celery.task.base
+    :members:

docs/reference/celery.task.builtins.rst

@@ -0,0 +1,8 @@
+==============================================
+ Built-in Task Classes - celery.task.builtins
+==============================================
+
+.. currentmodule:: celery.task.builtins
+
+.. automodule:: celery.task.builtins
+    :members:

docs/reference/celery.task.rst

@@ -1,6 +1,6 @@
-=================================
-Tasks - celery.task
-=================================
+==============================================
+ Task Information and Utilities - celery.task
+==============================================
 
 .. currentmodule:: celery.task
 

docs/reference/celery.task.strategy.rst

@@ -0,0 +1,8 @@
+===============================================
+ Common Task Strategies - celery.task.strategy
+===============================================
+
+.. currentmodule:: celery.task.strategy
+
+.. automodule:: celery.task.strategy
+    :members:

docs/reference/celery.timer.rst

@@ -1,8 +0,0 @@
-======================
-Timers - celery.timer
-======================
-
-.. currentmodule:: celery.timer
-
-.. automodule:: celery.timer
-    :members:

docs/reference/celery.utils.rst

@@ -0,0 +1,8 @@
+========================================
+Multiprocessing Worker - celery.worker
+========================================
+
+.. currentmodule:: celery.utils
+
+.. automodule:: celery.utils
+    :members:

docs/reference/celery.views.rst

@@ -0,0 +1,8 @@
+========================================
+Django Views - celery.views
+========================================
+
+.. currentmodule:: celery.views
+
+.. automodule:: celery.views
+    :members:

docs/reference/celery.worker.controllers.rst

@@ -0,0 +1,8 @@
+=======================================================
+ Worker Controller Threads - celery.worker.controllers
+=======================================================
+
+.. currentmodule:: celery.worker.controllers
+
+.. automodule:: celery.worker.controllers
+    :members:

docs/reference/celery.worker.job.rst

@@ -0,0 +1,8 @@
+=====================================
+ Executable Jobs - celery.worker.job
+=====================================
+
+.. currentmodule:: celery.worker.job
+
+.. automodule:: celery.worker.job
+    :members:

docs/reference/index.rst

@@ -7,14 +7,22 @@
 
 .. toctree::
     :maxdepth: 2
-    
-    celery.task
+
+    celery.task.base
+    celery.execute 
     celery.result
+    celery.task
     celery.registry
+    celery.task.builtins
+    celery.task.strategy
     celery.discovery
     celery.monitoring
+    celery.messaging
     celery.worker
+    celery.worker.job
+    celery.worker.controllers
     celery.pool
+    celery.supervisor
     celery.backends
     celery.backends.base
     celery.backends.database
@@ -23,8 +31,9 @@
     celery.conf
     celery.datastructures
     celery.log
+    celery.utils
+    celery.views
     celery.managers
     celery.models
-    celery.messaging
-    celery.timer
+    celery.fields
     celery.bin.celeryd

docs/tutorials/clickcounter.rst

@@ -0,0 +1,238 @@
+============================================================
+ Tutorial: Creating a click counter using carrot and celery
+============================================================
+
+Introduction
+============
+
+A click counter should be easy, right? Just a simple view that increments
+a click in the DB and forwards you to the real destination.
+
+This would work well for most sites, but when traffic starts to increase,
+you are likely to bump into problems. One database write for every click is
+not good if you have millions of clicks a day.
+
+So what can you do? In this tutorial we will send the individual clicks as
+messages using ``carrot``, and then process them later with a ``celery``
+periodic task.
+
+Celery and carrot is excellent in tandem, and while this might not be
+the perfect example, you'll at least see one example how of they can be used
+to solve a task.
+
+The model
+=========
+
+The model is simple, ``Click`` has the URL as primary key and a number of
+clicks for that URL. Its manager, ``ClickManager`` implements the
+``increment_clicks`` method, which takes a URL and by how much to increment
+its count by.
+
+
+*clickmuncher/models.py*:
+
+.. code-block:: python
+
+    from django.db import models
+    from django.utils.translation import ugettext_lazy as _
+
+
+    class ClickManager(models.Manager):
+
+        def increment_clicks(self, for_url, increment_by=1):
+            """Increment the click count for an URL.
+
+                >>> Click.objects.increment_clicks("http://google.com", 10)
+
+            """
+            click, created = self.get_or_create(url=for_url,
+                                    defaults={"click_count": increment_by})
+            if not created:
+                click.click_count += increment_by
+                click.save()
+
+            return click.click_count
+
+
+    class Click(models.Model):
+        url = models.URLField(_(u"URL"), verify_exists=False, unique=True)
+        click_count = models.PositiveIntegerField(_(u"click_count"),
+                                                  default=0)
+
+        objects = ClickManager()
+
+        class Meta:
+            verbose_name = _(u"URL clicks")
+            verbose_name_plural = _(u"URL clicks")
+
+Using carrot to send clicks as messages
+========================================
+
+The model is normal django stuff, nothing new there. But now we get on to
+the messaging. It has been a tradition for me to put the projects messaging
+related code in its own ``messaging.py`` module, and I will continue to do so
+here so maybe you can adopt this practice. In this module we have two
+functions:
+
+* ``send_increment_clicks``
+
+  This function sends a simple message to the broker. The message body only
+  contains the URL we want to increment as plain-text, so the exchange and
+  routing key play a role here. We use an exchange called ``clicks``, with a
+  routing key of ``increment_click``, so any consumer binding a queue to
+  this exchange using this routing key will receive these messages.
+
+* ``process_clicks``
+
+  This function processes all currently gathered clicks sent using
+  ``send_increment_clicks``. Instead of issuing one database query for every
+  click it processes all of the messages first, calculates the new click count
+  and issues one update per URL. A message that has been received will not be
+  deleted from the broker until it has been acknowledged by the receiver, so
+  if the reciever dies in the middle of processing the message, it will be
+  re-sent at a later point in time. This guarantees delivery and we respect
+  this feature here by not acknowledging the message until the clicks has
+  actually been written to disk.
+  
+  **Note**: This could probably be optimized further with
+  some hand-written SQL, but it will do for now. Let's say it's an excersise
+  left for the picky reader, albeit a discouraged one if you can survive
+  without doing it.
+
+On to the code...
+
+*clickmuncher/messaging.py*:
+
+.. code-block:: python
+
+    from carrot.connection import DjangoAMQPConnection
+    from carrot.messaging import Publisher, Consumer
+    from clickmuncher.models import Click
+
+
+    def send_increment_clicks(for_url):
+        """Send a message for incrementing the click count for an URL."""
+        connection = DjangoAMQPConnection()
+        publisher = Publisher(connection=connection,
+                              exchange="clicks",
+                              routing_key="increment_click",
+                              exchange_type="direct")
+
+        publisher.send(for_url)
+
+        publisher.close()
+        connection.close()
+
+
+    def process_clicks():
+        """Process all currently gathered clicks by saving them to the
+        database."""
+        connection = DjangoAMQPConnection()
+        consumer = Consumer(connection=connection,
+                            queue="clicks",
+                            exchange="clicks",
+                            routing_key="increment_click",
+                            exchange_type="direct")
+
+        # First process the messages: save the number of clicks
+        # for every URL.
+        clicks_for_url = {}
+        messages_for_url = {}
+        for message in consumer.iterqueue():
+            url = message.body
+            clicks_for_url[url] = clicks_for_url.get(url, 0) + 1
+            # We also need to keep the message objects so we can ack the
+            # messages as processed when we are finished with them.
+            if url in messages_for_url:
+                messages_for_url[url].append(message)
+            else:
+                messages_for_url[url] = [message]
+    
+        # Then increment the clicks in the database so we only need
+        # one UPDATE/INSERT for each URL.
+        for url, click_count in clicks_for_urls.items():
+            Click.objects.increment_clicks(url, click_count)
+            # Now that the clicks has been registered for this URL we can
+            # acknowledge the messages
+            [message.ack() for message in messages_for_url[url]]
+        
+        consumer.close()
+        connection.close()
+
+
+View and URLs
+=============
+
+This is also simple stuff, don't think I have to explain this code to you.
+The interface is as follows, if you have a link to http://google.com you
+would want to count the clicks for, you replace the URL with:
+
+    http://mysite/clickmuncher/count/?u=http://google.com
+
+and the ``count`` view will send off an increment message and forward you to
+that site.
+
+*clickmuncher/views.py*:
+
+.. code-block:: python
+
+    from django.http import HttpResponseRedirect
+    from clickmuncher.messaging import send_increment_clicks
+
+
+    def count(request):
+        url = request.GET["u"]
+        send_increment_clicks(url)
+        return HttpResponseRedirect(url)
+
+
+*clickmuncher/urls.py*:
+
+.. code-block:: python
+
+    from django.conf.urls.defaults import patterns, url
+    from clickmuncher import views
+
+    urlpatterns = patterns("",
+        url(r'^$', views.count, name="clickmuncher-count"),
+    )
+
+
+Creating the periodic task
+==========================
+
+Processing the clicks every 30 minutes is easy using celery periodic tasks.
+
+*clickmuncher/tasks.py*:
+
+.. code-block:: python
+
+    from celery.task import PeriodicTask
+    from celery.registry import tasks
+    from clickmuncher.messaging import process_clicks
+    from datetime import timedelta
+
+
+    class ProcessClicksTask(PeriodicTask):
+        run_every = timedelta(minutes=30)
+    
+        def run(self, \*\*kwargs):
+            process_clicks()
+    tasks.register(ProcessClicksTask)
+
+We subclass from :class:`celery.task.base.PeriodicTask`, set the ``run_every``
+attribute and in the body of the task just call the ``process_clicks``
+function we wrote earlier. Finally, we register the task in the task registry
+so the celery workers is able to recognize and find it.
+
+
+Finishing
+=========
+
+There are still ways to improve this application. The URLs could be cleaned
+so the url http://google.com and http://google.com/ is the same. Maybe it's
+even possible to update the click count using a single UPDATE query?
+
+If you have any questions regarding this tutorial, please send a mail to the
+mailing-list or come join us in the #celery IRC channel at Freenode:
+http://celeryq.org/introduction.html#getting-help

docs/tutorials/index.rst

@@ -0,0 +1,11 @@
+===========
+ Tutorials
+===========
+
+:Release: |version|
+:Date: |today|
+
+.. toctree::
+    :maxdepth: 2
+
+    clickcounter

setup.py

@@ -40,7 +40,10 @@ class RunTests(Command):
         os.chdir(this_dir)
 
 
-install_requires = ["carrot"]
+install_requires = ["django-unittest-depth",
+                    "anyjson",
+                    "carrot>=0.5.0",
+                    "python-daemon"]
 py_version_info = sys.version_info
 py_major_version = py_version_info[0]
 py_minor_version = py_version_info[1]
@@ -48,8 +51,8 @@ py_minor_version = py_version_info[1]
 if (py_major_version == 2 and py_minor_version <=5) or py_major_version < 2:
     install_requires.append("multiprocessing")
 
-if os.path.exists("README"):
-    long_description = codecs.open("README", "r", "utf-8").read()
+if os.path.exists("README.rst"):
+    long_description = codecs.open("README.rst", "r", "utf-8").read()
 else:
     long_description = "See http://pypi.python.org/pypi/celery"
 
@@ -65,13 +68,13 @@ setup(
     packages=find_packages(exclude=['ez_setup']),
     scripts=["bin/celeryd"],
     zip_safe=False,
-    install_requires=[
-        'carrot>=0.4.5',
-        'python-daemon',
-    ],
+    install_requires=install_requires,
+    extra_requires={
+        "Tyrant": ["pytyrant"],
+    },
     cmdclass = {"test": RunTests},
     classifiers=[
-        "Development Status :: 4 - Beta",
+        "Development Status :: 5 - Production/Stable",
         "Framework :: Django",
         "Operating System :: OS Independent",
         "Programming Language :: Python",

testproj/settings.py

@@ -5,9 +5,13 @@ import sys
 # import source code dir
 sys.path.insert(0, os.path.join(os.getcwd(), os.pardir))
 
+SITE_ID = 300
+
 DEBUG = True
 TEMPLATE_DEBUG = DEBUG
 
+ROOT_URLCONF = "urls"
+
 ADMINS = (
     # ('Your Name', 'your_email@domain.com'),
 )
@@ -23,6 +27,9 @@ AMQP_VHOST = "/"
 AMQP_USER = "guest"
 AMQP_PASSWORD = "guest"
 
+TT_HOST = "localhost"
+TT_PORT = 1978
+
 CELERY_AMQP_EXCHANGE = "testcelery"
 CELERY_AMQP_ROUTING_KEY = "testcelery"
 CELERY_AMQP_CONSUMER_QUEUE = "testcelery"
@@ -51,4 +58,7 @@ try:
 except ImportError:
     pass
 else:
-    INSTALLED_APPS += ("test_extensions", )
+    pass
+    #INSTALLED_APPS += ("test_extensions", )
+
+SEND_CELERY_TASK_ERROR_EMAILS = False