Added new documentation: Internals

docs/index.rst

@@ -17,6 +17,7 @@ Contents:
     tutorials/index
     faq
     reference/index
+    internals/index
     changelog
     links
 

docs/internals/index.rst

@@ -0,0 +1,11 @@
+===========
+ Internals
+===========
+
+:Release: |version|
+:Date: |today|
+
+.. toctree::
+    :maxdepth: 2
+
+    worker

docs/internals/workers.rst

@@ -0,0 +1,63 @@
+=======================
+ Internals: The worker
+=======================
+
+**NOTE** This describes the internals of the development version, not the
+current release.
+
+The worker consists of 4 main components: the broker listener, the scheduler,
+the mediator and the task pool. All these components runs in parallel working
+with two data structures: the ready queue and the ETA schedule.
+
+
+Data structures
+===============
+
+ready_queue
+-----------
+
+The ready queue is either an instance of :class:`Queue.Queue`, or
+`celery.buckets.TaskBucket`. The latter if rate limiting is enabled.
+
+eta_schedule
+------------
+
+The ETA schedule is a heap queue sorted by time.
+
+
+Components
+==========
+
+
+CarrotListener
+--------------
+
+Receives messages from the broker using ``carrot``.
+
+When a message is received it's converted into a
+:class:`celery.worker.job.TaskWrapper` object.
+
+Tasks with an ETA are entered into the ``eta_schedule``, messages that can
+be immediately processed are moved directly to the ``ready_queue``.
+
+ScheduleController
+------------------
+
+The schedule controller is running the ``eta_schedule``.
+If the scheduled tasks eta has passed it is moved to the ``ready_queue``,
+otherwise the thread sleeps until the eta is met (remember that the schedule
+is sorted by time).
+
+Mediator
+--------
+The mediator simply moves tasks in the ``ready_queue`` over to the
+task pool for execution using
+:meth:`celery.worker.job.TaskWrapper.execute_using_pool`.
+
+TaskPool
+--------
+
+This is a slightly modified :class:`multiprocessing.Pool`.
+It mostly works the same way, except it makes sure all of the workers
+are running at all times. If a worker is missing, it replaces
+it with a new one.