| 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970 |
- .. _internals-worker:
- =======================
- Internals: The worker
- =======================
- .. contents::
- :local:
- Introduction
- ============
- The worker consists of 4 main components: the broker listener, the scheduler,
- the mediator and the task pool. All these components runs in parallel working
- with two data structures: the ready queue and the ETA schedule.
- .. image:: ../images/Celery1.0-inside-worker.jpg
- Data structures
- ===============
- ready_queue
- -----------
- The ready queue is either an instance of :class:`Queue.Queue`, or
- `celery.buckets.TaskBucket`. The latter if rate limiting is enabled.
- eta_schedule
- ------------
- The ETA schedule is a heap queue sorted by time.
- Components
- ==========
- Listener
- --------------
- Receives messages from the broker using `Kombu`_.
- .. _`Kombu`: http://pypi.python.org/pypi/kombu
- When a message is received it's converted into a
- :class:`celery.worker.job.TaskRequest` object.
- Tasks with an ETA are entered into the ``eta_schedule``, messages that can
- be immediately processed are moved directly to the ``ready_queue``.
- ScheduleController
- ------------------
- The schedule controller is running the ``eta_schedule``.
- If the scheduled tasks eta has passed it is moved to the ``ready_queue``,
- otherwise the thread sleeps until the eta is met (remember that the schedule
- is sorted by time).
- Mediator
- --------
- The mediator simply moves tasks in the ``ready_queue`` over to the
- task pool for execution using
- :meth:`celery.worker.job.TaskRequest.execute_using_pool`.
- TaskPool
- --------
- This is a slightly modified :class:`multiprocessing.Pool`.
- It mostly works the same way, except it makes sure all of the workers
- are running at all times. If a worker is missing, it replaces
- it with a new one.
|
