eventlet.rst 2.3 KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465
  1. .. _concurrency-eventlet:
  2. ===========================
  3. Concurrency with Eventlet
  4. ===========================
  5. .. _eventlet-introduction:
  6. Introduction
  7. ============
  8. The `Eventlet`_ homepage describes it as;
  9. A concurrent networking library for Python that allows you to
  10. change how you run your code, not how you write it.
  11. * It uses `epoll(4)`_ or `libevent`_ for
  12. `highly scalable non-blocking I/O`_.
  13. * `Coroutines`_ ensure that the developer uses a blocking style of
  14. programming that is similar to threading, but provide the benefits of
  15. non-blocking I/O.
  16. * The event dispatch is implicit, which means you can easily use Eventlet
  17. from the Python interpreter, or as a small part of a larger application.
  18. Celery supports Eventlet as an alternative execution pool implementation.
  19. It is in some cases superior to prefork, but you need to ensure
  20. your tasks do not perform blocking calls, as this will halt all
  21. other operations in the worker until the blocking call returns.
  22. The prefork pool can take use of multiple processes, but how many is
  23. often limited to a few processes per CPU. With Eventlet you can efficiently
  24. spawn hundreds, or thousands of green threads. In an informal test with a
  25. feed hub system the Eventlet pool could fetch and process hundreds of feeds
  26. every second, while the prefork pool spent 14 seconds processing 100
  27. feeds. Note that this is one of the applications async I/O is especially good
  28. at (asynchronous HTTP requests). You may want a mix of both Eventlet and
  29. prefork workers, and route tasks according to compatibility or
  30. what works best.
  31. Enabling Eventlet
  32. =================
  33. You can enable the Eventlet pool by using the :option:`celery worker -P`
  34. worker option.
  35. .. code-block:: console
  36. $ celery -A proj worker -P eventlet -c 1000
  37. .. _eventlet-examples:
  38. Examples
  39. ========
  40. See the `Eventlet examples`_ directory in the Celery distribution for
  41. some examples taking use of Eventlet support.
  42. .. _`Eventlet`: http://eventlet.net
  43. .. _`epoll(4)`: http://linux.die.net/man/4/epoll
  44. .. _`libevent`: http://monkey.org/~provos/libevent/
  45. .. _`highly scalable non-blocking I/O`:
  46. https://en.wikipedia.org/wiki/Asynchronous_I/O#Select.28.2Fpoll.29_loops
  47. .. _`Coroutines`: https://en.wikipedia.org/wiki/Coroutine
  48. .. _`Eventlet examples`:
  49. https://github.com/celery/celery/tree/master/examples/eventlet