Datastructures - celery.datastructures¶

Custom Datastructures

class celery.datastructures.PositionQueue(length)¶

A positional queue of a specific length, with slots that are either filled or unfilled. When all of the positions are filled, the queue is considered full().

Parameter:	length – see length.

length¶

The number of items required for the queue to be considered full.

class UnfilledPosition(position)¶

Describes an unfilled slot.

PositionQueue.filled¶

Returns the filled slots as a list.

PositionQueue.full()¶

Returns True if all of the slots has been filled.

class celery.datastructures.TaskProcessQueue(limit, reap_timeout=None, logger=None, done_msg=None)¶

Queue of running child processes, which starts waiting for the processes to finish when the queue limit has been reached.

Parameters:	limit – see limit attribute. logger – see logger attribute. done_msg – see done_msg attribute.

limit¶

The number of processes that can run simultaneously until we start collecting results.

logger¶

The logger used to print the done_msg.

done_msg¶

Message logged when a tasks result has been collected. The message is logged with loglevel logging.INFO.

add(result, task_name, task_id)¶

Add a process to the queue.

If the queue is full, it will wait for the first task to finish, collects its result and remove it from the queue, so it’s ready to accept new processes.

Parameters:	result – A multiprocessing.AsyncResult instance, as returned by multiprocessing.Pool.apply_async(). task_name – Name of the task executed. task_id – Id of the task executed.

get_worker_pids()¶

Returns the process id’s of all the pool workers.

Return type:	list

on_ready(ret_value, task_name, task_id)¶

What to do when a worker returns with a result.

If done_msg is defined, it will log this format string, with level logging.INFO, using these format variables:

• %(name)

  The name of the task completed

• %(id)

  The UUID of the task completed.

• %(return_value)

  Return value of the task function.

wait_for_result()¶

Waits for the first process in the pool to finish.

This operation is blocking.