| 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061 |
- .. _concurrency-eventlet:
- ===========================
- Concurrency with Eventlet
- ===========================
- .. _eventlet-introduction:
- Introduction
- ============
- The `Eventlet`_ homepage describes eventlet as,
- a concurrent networking library for Python that allows you to
- change how you run your code, not how you write it.
- * It uses epoll or libevent for `highly scalable non-blocking I/O`_.
- * `Coroutines`_ ensure that the developer uses a blocking style of
- programming that is similar to threading, but provide the benefits of
- non-blocking I/O.
- * The event dispatch is implicit, which means you can easily use Eventlet
- from the Python interpreter, or as a small part of a larger application.
- Celery supports Eventlet as an alternative execution pool implementation.
- It is in some cases superior to multiprocessing, but you need to ensure
- your tasks do not perform any blocking calls, as this will halt all
- other operations in the worker.
- The multiprocessing pool can take use of many processes, but it is often
- limited to a few processes per CPU. With eventlet you can efficiently spawn
- hundreds of concurrent couroutines. In an informal test with a feed hub
- system the Eventlet pool could fetch and process hundreds of feeds every
- second, while the multiprocessing pool spent 14 seconds processing 100 feeds.
- But this is exactly the kind of application evented I/O is good for.
- You may want a a mix of both eventlet and multiprocessing workers,
- depending on the needs of your tasks.
- Enabling Eventlet
- =================
- You can enable the Eventlet pool by using the `-P` option to
- :program:`celeryd`::
- $ celeryd -P eventlet -c 1000
- .. _eventlet-examples:
- Examples
- ========
- See the `Eventlet examples`_ directory in the Celery distribution for
- some examples taking use of Eventlet support.
- .. _`Eventlet`: http://eventlet.net
- .. _`highly scalable non-blocking I/O`:
- http://en.wikipedia.org/wiki/Asynchronous_I/O#Select.28.2Fpoll.29_loops
- .. _`Coroutines`: http://en.wikipedia.org/wiki/Coroutine
- .. _`Eventlet examples`:
- https://github.com/ask/celery/tree/master/examples/eventlet
|
