Task Pool - celery.pool¶
Process Pools.
- class celery.pool.TaskPool(limit, reap_timeout=None, logger=None)¶
Pool of running child processes, which starts waiting for the processes to finish when the queue limit has been reached.
Parameters: - limit¶
- The number of processes that can run simultaneously until we start collecting results.
- logger¶
- The logger used for debugging.
- add(result, callbacks, errbacks, tid, meta)¶
Add a process to the queue.
If the queue is full, it will wait for the first task to finish, collects its result and remove it from the queue, so it’s ready to accept new processes.
Parameter: result – A multiprocessing.AsyncResult instance, as returned by multiprocessing.Pool.apply_async().
Option callbacks: List of callbacks to execute if the task was successful. Must have the function signature:
mycallback(result, meta)
Option errbacks: List of errbacks to execute if the task raised and exception. Must have the function signature:
myerrback(exc, meta).
Option tid: The tid for this task (unqiue pool id).
- apply_async(target, args=None, kwargs=None, callbacks=None, errbacks=None, on_acknowledge=None, meta=None)¶
Equivalent of the :func:apply built-in function.
All callbacks and errbacks should complete immediately since otherwise the thread which handles the result will get blocked.
- full()¶
Is the pool full?
Returns: True if the maximum number of concurrent processes has been reached.
- get_worker_pids()¶
Returns the process id’s of all the pool workers.
Return type: list
- on_ready(ret_value, callbacks, errbacks, meta)¶
- What to do when a worker task is ready and its return value has been collected.
- on_return(ret_val, tid, callbacks, errbacks, meta)¶
- What to do when the process returns.
- reap()¶
- Reap finished tasks.
- run()¶
Run the task pool.
Will launch all worker processes so they are ready for processing tasks.
- terminate()¶
- Terminate the pool.
- wait_for_result()¶
Waits for the first process in the pool to finish.
This operation is blocking.