Multiprocessing Worker - celery.worker¶

celery.worker

exception celery.worker.EmptyQueue¶

The message queue is currently empty.

class celery.worker.PeriodicWorkController(group=None, target=None, name=None, args=(), kwargs=None, verbose=None)¶

A thread that continuously checks if there are :class:`celery.task.PeriodicTask`s waiting for execution, and executes them.

Example

>>> PeriodicWorkController().start()

run()¶

Don’t use run(). use start().

class celery.worker.TaskWrapper(task_name, task_id, task_func, args, kwargs, **opts)¶

Class wrapping a task to be run.

Parameters:	task_name – see task_name. task_id – see task_id. task_func – see task_func args – see args kwargs – see kwargs.

task_name¶

Kind of task. Must be a name registered in the task registry.

task_id¶

UUID of the task.

task_func¶

The tasks callable object.

args¶

List of positional arguments to apply to the task.

kwargs¶

Mapping of keyword arguments to apply to the task.

execute(loglevel=None, logfile=None)¶

Execute the task in a jail() and store return value and status in the task meta backend.

Parameters:	loglevel – The loglevel used by the task. logfile – The logfile used by the task.

execute_using_pool(pool, loglevel=None, logfile=None)¶

Like execute(), but using the multiprocessing pool.

Parameters:	pool – A multiprocessing.Pool instance. loglevel – The loglevel used by the task. logfile – The logfile used by the task.

:returns multiprocessing.AsyncResult instance.

extend_with_default_kwargs(loglevel, logfile)¶

Extend the tasks keyword arguments with standard task arguments.

These are logfile, loglevel, task_id and task_name.

classmethod from_message(message, logger)¶

Create a TaskWrapper from a task message sent by celery.messaging.TaskPublisher.

Raises UnknownTask:
	if the message does not describe a task, the message is also rejected.
Returns:	TaskWrapper instance.

on_failure(exc_info, meta)¶

The handler used if the task raised an exception.

on_success(ret_value, meta)¶

The handler used if the task was successfully processed ( without raising an exception).

exception celery.worker.UnknownTask¶

Got an unknown task in the queue. The message is requeued and ignored.

class celery.worker.WorkController(concurrency=None, logfile=None, loglevel=None, queue_wakeup_after=None, is_detached=False)¶

Executes tasks waiting in the task queue.

Parameters:	concurrency – see concurrency. logfile – see logfile. loglevel – see loglevel. queue_wakeup_after – see queue_wakeup_after.

concurrency¶

The number of simultaneous processes doing work (default: celery.conf.DAEMON_CONCURRENCY)

loglevel¶

The loglevel used (default: logging.INFO)

logfile¶

The logfile used, if no logfile is specified it uses stderr (default: celery.conf.DAEMON_LOG_FILE).

queue_wakeup_after¶

The time it takes for the daemon to wake up after the queue is empty, so it can check for more work (default: celery.conf.QUEUE_WAKEUP_AFTER).

empty_msg_emit_every¶

How often the daemon emits the "Waiting for queue..." message. If this is None, the message will never be logged. (default: celery.conf.EMPTY_MSG_EMIT_EVERY)

logger¶

The logging.Logger instance used for logging.

pool¶

The multiprocessing.Pool instance used.

task_consumer¶

The celery.messaging.TaskConsumer instance used.

connection_diagnostics()¶

Diagnose the AMQP connection, and reset connection if necessary.

execute_next_task()¶

Execute the next task on the queue using the multiprocessing pool.

Catches all exceptions and logs them with level logging.CRITICAL.

Raises EmptyQueue exception if there is no message waiting on the queue.

process_task(message)¶

Process task message by passing it to the pool of workers.

receive_message()¶

Receive the next message from the message broker.

Tries to reset the AMQP connection if not available. Returns None if no message is waiting on the queue.

Return type:	carrot.messaging.Message instance.

reset_connection()¶

Reset the AMQP connection, and reinitialize the celery.messaging.TaskConsumer instance.

Resets the task consumer in task_consumer.

run()¶

Starts the workers main loop.

schedule_retry_tasks()¶

Reschedule all requeued tasks waiting for retry.

celery.worker.jail(task_id, func, args, kwargs)¶

Wraps the task in a jail, which catches all exceptions, and saves the status and result of the task execution to the task meta backend.

If the call was successful, it saves the result to the task result backend, and sets the task status to "DONE".

If the call results in an exception, it saves the exception as the task result, and sets the task status to "FAILURE".

Parameters:	task_id – The id of the task. func – Callable object to execute. args – List of positional args to pass on to the function. kwargs – Keyword arguments mapping to pass on to the function.
Returns:	the function return value on success, or the exception instance on failure.