Browse Source

Optimizing User Guide: Runes edit

Ask Solem 14 years ago
parent
commit
2e8b4de810
1 changed files with 48 additions and 49 deletions
  1. 48 49
      docs/userguide/optimizing.rst

+ 48 - 49
docs/userguide/optimizing.rst

@@ -6,17 +6,14 @@
 
 
 Introduction
 Introduction
 ============
 ============
+The default configuration makes a lot of compromises.  It's not optimal for
+any single case, but works well enough for most situations.
 
 
-The default configuration, like any good default, is full of compromises.
-It is not tweaked to be optimal for any single use case, but tries to
-find middle ground that works *well enough* for most situations.
+There are optimizations that can be applied based on specific use cases.
 
 
-There are key optimizations to be done if your application is mainly
-processing lots of short tasks, and also if you have fewer but very
-long tasks.
-
-Optimization here does not necessarily mean optimizing for runtime, but also
-optimizing resource usage and ensuring responsiveness at times of high load.
+Optimizations can apply to different properties of the running environment,
+be it the time tasks take to execute, the amount of memory used, or
+responsiveness at times of high load.
 
 
 Ensuring Operations
 Ensuring Operations
 ===================
 ===================
@@ -26,22 +23,23 @@ back-of-the-envelope calculations by asking the question;
 
 
     ❝ How much water flows out of the Mississippi River in a day? ❞
     ❝ How much water flows out of the Mississippi River in a day? ❞
 
 
-The point of this exercise[*] is to demonstrate that there is a limit
-to how much data a system can process in a timely manner, and teaches
-back of the envelope calculations as a means to plan for this ahead of time.
+The point of this exercise[*] is to show that there is a limit
+to how much data a system can process in a timely manner.
+Back of the envelope calculations can be used as a means to plan for this
+ahead of time.
 
 
-This is very relevant to Celery; If a task takes 10 minutes to complete,
-and there are 10 new tasks coming in every minute, then this means
-the queue will *never be processed*.  This is why it's very important
+In Celery; If a task takes 10 minutes to complete,
+and there are 10 new tasks coming in every minute, the queue will never
+be empty.  This is why it's very important
 that you monitor queue lengths!
 that you monitor queue lengths!
 
 
-One way to do this is by :ref:`using Munin <monitoring-munin>`.
-You should set up alerts, so you are notified as soon as any queue has
-reached an unacceptable size, this way you can take appropriate action like
-adding new worker nodes, or revoking unnecessary tasks.
+A way to do this is by :ref:`using Munin <monitoring-munin>`.
+You should set up alerts, that will notify you as soon as any queue has
+reached an unacceptable size.  This way you can take appropriate action
+like adding new worker nodes, or revoking unnecessary tasks.
 
 
 .. [*] The chapter is available to read for free here:
 .. [*] The chapter is available to read for free here:
-       `The back of the envelope`_.  This book is a classic text, highly
+       `The back of the envelope`_.  The book is a classic text. Highly
        recommended.
        recommended.
 
 
 .. _`Programming Pearls`: http://www.cs.bell-labs.com/cm/cs/pearls/
 .. _`Programming Pearls`: http://www.cs.bell-labs.com/cm/cs/pearls/
@@ -62,36 +60,36 @@ Prefetch Limits
 *Prefetch* is a term inherited from AMQP that is often misunderstood
 *Prefetch* is a term inherited from AMQP that is often misunderstood
 by users.
 by users.
 
 
-The prefetch limit is a **limit** for the number of messages (tasks) a worker
-can reserve in advance.  If this is set to zero, the worker will keep
-consuming messages *ad infinitum*, not respecting that there may be other
+The prefetch limit is a **limit** for the number of tasks (messages) a worker
+can reserve for itself.  If it is zero, the worker will keep
+consuming messages, not respecting that there may be other
 available worker nodes that may be able to process them sooner[#],
 available worker nodes that may be able to process them sooner[#],
 or that the messages may not even fit in memory.
 or that the messages may not even fit in memory.
 
 
-The workers initial prefetch count is set by multiplying
-the :setting:`CELERYD_PREFETCH_MULTIPLIER` setting by the number
-of child worker processes[#].  The default is 4 messages per child process.
+The workers' default prefetch count is the
+:setting:`CELERYD_PREFETCH_MULTIPLIER` setting multiplied by the number
+of child worker processes[#].
 
 
-If you have many expensive tasks with a long duration you would want
+If you have many tasks with a long duration you want
 the multiplier value to be 1, which means it will only reserve one
 the multiplier value to be 1, which means it will only reserve one
-unacknowledged task per worker process at a time.
+task per worker process at a time.
 
 
-However -- If you have lots of short tasks, and throughput/roundtrip latency
-is important to you, then you want this number to be large.  Say 64, or 128
-for example, as the worker is able to process a lot more *tasks/s* if the
-messages have already been prefetched in memory.  You may have to experiment
-to find the best value that works for you.
+However -- If you have many short-running tasks, and throughput/roundtrip
+latency[#] is important to you, this number should be large. The worker is
+able to process more tasks per second if the messages have already been
+prefetched, and is available in memory.  You may have to experiment to find
+the best value that works for you.  Values like 50 or 150 might make sense in
+these circumstances. Say 64, or 128.
 
 
-If you have a combination of both very long and short tasks, then the best
-option is to use two worker nodes that is configured individually, and route
-the tasks accordingly (see :ref:`guide-routing`).
+If you have a combination of long- and short-running tasks, the best option
+is to use two worker nodes that are configured separatly, and route
+the tasks according to the run-time. (see :ref:`guide-routing`).
 
 
-.. [*] RabbitMQ and other brokers will deliver the messages in round-robin,
-       so this doesn't apply to an active system.  But if there is no prefetch
+.. [*] RabbitMQ and other brokers deliver messages round-robin,
+       so this doesn't apply to an active system.  If there is no prefetch
        limit and you restart the cluster, there will be timing delays between
        limit and you restart the cluster, there will be timing delays between
-       nodes starting, so if there are 3 offline nodes and one active node,
-       then all messages will be delivered to the active node while the others
-       are offline.
+       nodes starting. If there are 3 offline nodes and one active node,
+       all messages will be delivered to the active node.
 
 
 .. [*] This is the concurrency setting; :setting:`CELERYD_CONCURRENCY` or the
 .. [*] This is the concurrency setting; :setting:`CELERYD_CONCURRENCY` or the
        :option:`-c` option to :program:`celeryd`.
        :option:`-c` option to :program:`celeryd`.
@@ -104,18 +102,18 @@ When using early acknowledgement (default), a prefetch multiplier of 1
 means the worker will reserve at most one extra task for every active
 means the worker will reserve at most one extra task for every active
 worker process.
 worker process.
 
 
-Often when users ask if it's possible to disable "prefetching of tasks",
-what they really want is to have a worker only reserve as many tasks
-as there are child processes at a time.
+When users ask if it's possible to disable "prefetching of tasks", often
+what they really want is to have a worker only reserve as many tasks as there
+are child processes.
 
 
-Sadly, this requirement is not possible without enabling late
+But this is not possible without enabling late acknowledgements
 acknowledgements; A task that has been started, will be
 acknowledgements; A task that has been started, will be
 retried if the worker crashes mid execution so the task must be `reentrant`_
 retried if the worker crashes mid execution so the task must be `reentrant`_
 (see also notes at :ref:`faq-acks_late-vs-retry`).
 (see also notes at :ref:`faq-acks_late-vs-retry`).
 
 
 .. _`reentrant`: http://en.wikipedia.org/wiki/Reentrant_(subroutine)
 .. _`reentrant`: http://en.wikipedia.org/wiki/Reentrant_(subroutine)
 
 
-You can enable this behavior by using the following configuration:
+You can enable this behavior by using the following configuration options:
 
 
 .. code-block:: python
 .. code-block:: python
 
 
@@ -127,9 +125,10 @@ You can enable this behavior by using the following configuration:
 Rate Limits
 Rate Limits
 -----------
 -----------
 
 
-The subsystem responsible for enforcing rate limits introduces extra
-complexity, so if you're not using rate limits it may be a good idea to
-disable them completely.
+The system responsible for enforcing rate limits introduces some overhead,
+so if you're not using rate limits it may be a good idea to
+disable them completely.  This will disable one thread, and it won't
+spend as many CPU cycles when the queue is inactive.
 
 
 Set the :setting:`CELERY_DISABLE_RATE_LIMITS` setting to disable
 Set the :setting:`CELERY_DISABLE_RATE_LIMITS` setting to disable
 the rate limit subsystem:
 the rate limit subsystem: