|
|
@@ -0,0 +1,61 @@
|
|
|
+.. _concurrency-eventlet:
|
|
|
+
|
|
|
+===========================
|
|
|
+ Concurrency with Eventlet
|
|
|
+===========================
|
|
|
+
|
|
|
+.. _eventlet-introduction:
|
|
|
+
|
|
|
+Introduction
|
|
|
+============
|
|
|
+
|
|
|
+The `Eventlet`_ homepage describes eventlet as,
|
|
|
+a concurrent networking library for Python that allows you to
|
|
|
+change how you run your code, not how you write it.
|
|
|
+
|
|
|
+ * It uses epoll or libevent for `highly scalable non-blocking I/O`_.
|
|
|
+ * `Coroutines`_ ensure that the developer uses a blocking style of
|
|
|
+ programming that is similar to threading, but provide the benefits of
|
|
|
+ non-blocking I/O.
|
|
|
+ * The event dispatch is implicit, which means you can easily use Eventlet
|
|
|
+ from the Python interpreter, or as a small part of a larger application.
|
|
|
+
|
|
|
+Celery supports Eventlet as an alternative execution pool implementation.
|
|
|
+It is in some cases superior to multiprocessing, but you need to ensure
|
|
|
+your tasks do not perform any blocking calls, as this will halt all
|
|
|
+other operations in the worker.
|
|
|
+
|
|
|
+The multiprocessing pool can take use of many processes, but it is often
|
|
|
+limited to a few processes per CPU. With eventlet you can efficiently spawn
|
|
|
+hundreds of concurrent couroutines. In an informal test with a feed hub
|
|
|
+system the Eventlet pool could fetch and process hundreds of feeds every
|
|
|
+second, while the multiprocessing pool spent 14 seconds processing 100 feeds.
|
|
|
+But this is exactly the kind of application evented I/O is good for.
|
|
|
+You may want a a mix of both eventlet and multiprocessing workers,
|
|
|
+depending on the needs of your tasks.
|
|
|
+
|
|
|
+Enabling Eventlet
|
|
|
+=================
|
|
|
+
|
|
|
+You can enable the Eventlet pool by using the `-P` option to
|
|
|
+:program:`celeryd`::
|
|
|
+
|
|
|
+ $ celeryd -P eventlet -c 1000
|
|
|
+
|
|
|
+.. _eventlet-examples:
|
|
|
+
|
|
|
+Examples
|
|
|
+========
|
|
|
+
|
|
|
+See the `Eventlet examples`_ directory in the Celery distribution for
|
|
|
+some examples taking use of Eventlet support.
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+.. _`Eventlet`: http://eventlet.net
|
|
|
+.. _`highly scalable non-blocking I/O`:
|
|
|
+ http://en.wikipedia.org/wiki/Asynchronous_I/O#Select.28.2Fpoll.29_loops
|
|
|
+.. _`Coroutines`: http://en.wikipedia.org/wiki/Coroutine
|
|
|
+.. _`Eventlet examples`:
|
|
|
+ https://github.com/ask/celery/tree/master/examples/eventlet
|
|
|
+
|
Ask Solem