Multiprocessing Worker - celery.worker¶

The Multiprocessing Worker Server

Documentation for this module is in docs/reference/celery.worker.rst.

class celery.worker.AMQPListener(bucket_queue, hold_queue, logger, initial_prefetch_count=2)¶

Listen for messages received from the AMQP broker and move them the the bucket queue for task processing.

Parameters:	bucket_queue – See bucket_queue. hold_queue – See hold_queue.

bucket_queue¶

The queue that holds tasks ready for processing immediately.

hold_queue¶

The queue that holds paused tasks. Reasons for being paused include a countdown/eta or that it’s waiting for retry.

logger¶

The logger used.

close_connection()¶

Close the AMQP connection.

consume_messages()¶

Consume messages forever (or until an exception is raised).

receive_message(message_data, message)¶

The callback called when a new message is received.

If the message has an eta we move it to the hold queue, otherwise we move it the bucket queue for immediate processing.

reset_connection()¶

Reset the AMQP connection, and reinitialize the carrot.messaging.ConsumerSet instance.

Resets the task consumer in task_consumer.

start()¶

Start the consumer.

If the connection is lost, it tries to re-establish the connection over time and restart consuming messages.

stop()¶

Stop processing AMQP messages and close the connection to the broker.

class celery.worker.WorkController(concurrency=None, logfile=None, loglevel=None, is_detached=False)¶

Executes tasks waiting in the task queue.

Parameters:	concurrency – see concurrency. logfile – see logfile. loglevel – see loglevel.

concurrency¶

The number of simultaneous processes doing work (default: celery.conf.DAEMON_CONCURRENCY)

loglevel¶

The loglevel used (default: logging.INFO)

logfile¶

The logfile used, if no logfile is specified it uses stderr (default: celery.conf.DAEMON_LOG_FILE).

logger¶

The logging.Logger instance used for logging.

is_detached¶

Flag describing if the worker is running as a daemon or not.

pool¶

The multiprocessing.Pool instance used.

bucket_queue¶

The Queue.Queue that holds tasks ready for immediate processing.

hold_queue¶

The Queue.Queue that holds paused tasks. Reasons for holding back the task include waiting for eta to pass or the task is being retried.

periodic_work_controller¶

Instance of celery.worker.controllers.PeriodicWorkController.

mediator¶

Instance of celery.worker.controllers.Mediator.

amqp_listener¶

Instance of AMQPListener.

process_task(task)¶

Process task by sending it to the pool of workers.

safe_process_task(task)¶

Same as process_task(), but catches all exceptions the task raises and log them as errors, to make sure the worker doesn’t die.

start()¶

Starts the workers main loop.

stop()¶

Gracefully shutdown the worker server.