| 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768 |
- .. _internals-worker:
- =======================
- Internals: The worker
- =======================
- .. contents::
- :local:
- Introduction
- ============
- The worker consists of 4 main components: the consumer, the scheduler,
- the mediator and the task pool. All these components runs in parallel working
- with two data structures: the ready queue and the ETA schedule.
- Data structures
- ===============
- ready_queue
- -----------
- The ready queue is either an instance of :class:`Queue.Queue`, or
- :class:`celery.buckets.TaskBucket`. The latter if rate limiting is enabled.
- eta_schedule
- ------------
- The ETA schedule is a heap queue sorted by time.
- Components
- ==========
- Consumer
- --------
- Receives messages from the broker using `Kombu`_.
- .. _`Kombu`: http://pypi.python.org/pypi/kombu
- When a message is received it's converted into a
- :class:`celery.worker.job.TaskRequest` object.
- Tasks with an ETA are entered into the `eta_schedule`, messages that can
- be immediately processed are moved directly to the `ready_queue`.
- ScheduleController
- ------------------
- The schedule controller is running the `eta_schedule`.
- If the scheduled tasks eta has passed it is moved to the `ready_queue`,
- otherwise the thread sleeps until the eta is met (remember that the schedule
- is sorted by time).
- Mediator
- --------
- The mediator simply moves tasks in the `ready_queue` over to the
- task pool for execution using
- :meth:`celery.worker.job.TaskRequest.execute_using_pool`.
- TaskPool
- --------
- This is a slightly modified :class:`multiprocessing.Pool`.
- It mostly works the same way, except it makes sure all of the workers
- are running at all times. If a worker is missing, it replaces
- it with a new one.
|
