FAQ 7.9 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240
  1. ============================
  2. Frequently Asked Questions
  3. ============================
  4. Questions
  5. =========
  6. MySQL is throwing deadlock errors, what can I do?
  7. -------------------------------------------------
  8. **Answer:** MySQL has default isolation level set to ``REPEATABLE-READ``,
  9. if you don't really need that, set it to ``READ-COMMITTED``.
  10. You can do that by adding the following to your ``my.cnf``::
  11. [mysqld]
  12. transaction-isolation = READ-COMMITTED
  13. For more information about InnoDBs transaction model see `MySQL - The InnoDB
  14. Transaction Model and Locking`_ in the MySQL user manual.
  15. (Thanks to Honza Kral and Anton Tsigularov for this solution)
  16. .. _`MySQL - The InnoDB Transaction Model and Locking`: http://dev.mysql.com/doc/refman/5.1/en/innodb-transaction-model.html
  17. celeryd is not doing anything, just hanging
  18. --------------------------------------------
  19. **Answer:** See `MySQL is throwing deadlock errors, what can I do?`_.
  20. or `Why is Task.delay/apply\* just hanging?`.
  21. Why is Task.delay/apply\* just hanging?`
  22. ----------------------------------------
  23. **Answer:** :mod:`amqplib` hangs if it isn't able to authenticate to the
  24. AMQP server, so make sure you are able to access the configured vhost using
  25. the user and password.
  26. Why won't celeryd run on FreeBSD?
  27. ---------------------------------
  28. **Answer:** multiprocessing.Pool requires a working POSIX semaphore
  29. implementation which isn't enabled in FreeBSD by default. You have to enable
  30. POSIX semaphores in the kernel and manually recompile multiprocessing.
  31. I'm having ``IntegrityError: Duplicate Key`` errors. Why?
  32. ----------------------------------------------------------
  33. **Answer:** See `MySQL is throwing deadlock errors, what can I do?`_.
  34. Thanks to howsthedotcom.
  35. Why won't my Task run?
  36. ----------------------
  37. **Answer:** Did you register the task in the applications ``tasks.py`` module?
  38. (or in some other module Django loads by default, like ``models.py``?).
  39. Also there might be syntax errors preventing the tasks module being imported.
  40. You can find out if the celery daemon is able to run the task by executing the
  41. task manually:
  42. >>> from myapp.tasks import MyPeriodicTask
  43. >>> MyPeriodicTask.delay()
  44. Watch celery daemons logfile (or output if not running as a daemon), to see
  45. if it's able to find the task, or if some other error is happening.
  46. Why won't my Periodic Task run?
  47. -------------------------------
  48. **Answer:** See `Why won't my Task run?`_.
  49. How do I discard all waiting tasks?
  50. ------------------------------------
  51. **Answer:** Use ``celery.task.discard_all()``, like this:
  52. >>> from celery.task import discard_all
  53. >>> discard_all()
  54. 1753
  55. The number ``1753`` is the number of messages deleted.
  56. You can also start celeryd with the ``--discard`` argument which will
  57. accomplish the same thing.
  58. I've discarded messages, but there are still messages left in the queue?
  59. ------------------------------------------------------------------------
  60. **Answer:** Tasks are acknowledged (removed from the queue) as soon
  61. as they are actually executed. After the worker has received a task, it will
  62. take some time until it is actually executed, especially if there are a lot
  63. of tasks already waiting for execution. Messages that are not acknowledged are
  64. hold on to by the worker until it closes the connection to the broker (AMQP
  65. server). When that connection is closed (e.g because the worker was stopped)
  66. the tasks will be re-sent by the broker to the next available worker (or the
  67. same worker when it has been restarted), so to properly purge the queue of
  68. waiting tasks you have to stop all the workers, and then discard the tasks
  69. using ``discard_all``.
  70. Can I use celery with ActiveMQ/STOMP?
  71. -------------------------------------
  72. **Answer**: Yes. But this is very experimental for now.
  73. First you have to use the ``master`` branch of ``celery``::
  74. $ git clone git://github.com/ask/celery.git
  75. $ cd celery
  76. $ sudo python setup.py install
  77. $ cd ..
  78. Then you need to install the ``stompbackend`` branch of ``carrot``::
  79. $ git clone git://github.com/ask/carrot.git
  80. $ cd carrot
  81. $ git checkout stompbackend
  82. $ sudo python setup.py install
  83. $ cd ..
  84. And my fork of ``python-stomp`` which adds non-blocking support::
  85. $ hg clone http://bitbucket.org/asksol/python-stomp/
  86. $ cd python-stomp
  87. $ sudo python setup.py install
  88. $ cd ..
  89. In this example we will use a queue called ``celery`` which we created in
  90. the ActiveMQ web admin interface.
  91. **Note**: For ActiveMQ the queue name has to have ``"/queue/"`` prepended to
  92. it. i.e. the queue ``celery`` becomes ``/queue/celery``.
  93. Since a STOMP queue is a single named entity and it doesn't have the
  94. routing capabilities of AMQP you need to set both the ``queue``, and
  95. ``exchange`` settings to your queue name. This is a minor inconvenience since
  96. carrot needs to maintain the same interface for both AMQP and STOMP (obviously
  97. the one with the most capabilities won).
  98. Use the following specific settings in your ``settings.py``:
  99. .. code-block:: python
  100. # Makes python-stomp the default backend for carrot.
  101. CARROT_BACKEND = "pystomp"
  102. # STOMP hostname and port settings.
  103. AMQP_HOST = "localhost"
  104. AMQP_PORT = 61613
  105. # The queue name to use (both queue and exchange must be set to the
  106. # same queue name when using STOMP!!!)
  107. CELERY_AMQP_CONSUMER_QUEUE = "/queue/celery"
  108. CELERY_AMQP_EXCHANGE = "/queue/celery"
  109. Now you can go on reading the tutorial in the README, ignoring any AMQP
  110. specific options.
  111. Which features are not supported when using STOMP?
  112. --------------------------------------------------
  113. This is a (possible incomplete) list of features not available when
  114. using the STOMP backend:
  115. * routing keys
  116. * exchange types (direct, topic, headers, etc)
  117. * immediate
  118. * mandatory
  119. Can I send some tasks to only some servers?
  120. --------------------------------------------
  121. **Answer:** As of now there is only one use-case that works like this,
  122. and that is tasks of type ``A`` can be sent to servers ``x`` and ``y``,
  123. while tasks of type ``B`` can be sent to server ``z``. One server can't
  124. handle more than one routing_key, but this is coming in a later release.
  125. Say you have two servers, ``x``, and ``y`` that handles regular tasks,
  126. and one server ``z``, that only handles feed related tasks, you can use this
  127. configuration:
  128. * Servers ``x`` and ``y``: settings.py:
  129. .. code-block:: python
  130. AMQP_SERVER = "rabbit"
  131. AMQP_PORT = 5678
  132. AMQP_USER = "myapp"
  133. AMQP_PASSWORD = "secret"
  134. AMQP_VHOST = "myapp"
  135. CELERY_AMQP_CONSUMER_QUEUE = "regular_tasks"
  136. CELERY_AMQP_EXCHANGE = "tasks"
  137. CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular"
  138. CELERY_AMQP_CONSUMER_ROUTING_KEY = "task.#"
  139. CELERY_AMQP_EXCHANGE_TYPE = "topic"
  140. * Server ``z``: settings.py:
  141. .. code-block:: python
  142. AMQP_SERVER = "rabbit"
  143. AMQP_PORT = 5678
  144. AMQP_USER = "myapp"
  145. AMQP_PASSWORD = "secret"
  146. AMQP_VHOST = "myapp"
  147. CELERY_AMQP_EXCHANGE = "tasks"
  148. CELERY_AMQP_PUBLISHER_ROUTING_KEY = "task.regular"
  149. CELERY_AMQP_EXCHANGE_TYPE = "topic"
  150. # This is the settings different for this server:
  151. CELERY_AMQP_CONSUMER_QUEUE = "feed_tasks"
  152. CELERY_AMQP_CONSUMER_ROUTING_KEY = "feed.#"
  153. Now to make a Task run on the ``z`` server you need to set its
  154. ``routing_key`` attribute so it starts with the words ``"task.feed."``:
  155. .. code-block:: python
  156. from feedaggregator.models import Feed
  157. from celery.task import Task
  158. class FeedImportTask(Task):
  159. routing_key = "feed.importer"
  160. def run(self, feed_url):
  161. # something importing the feed
  162. Feed.objects.import_feed(feed_url)
  163. You can also override this using the ``routing_key`` argument to
  164. :func:`celery.task.apply_async`:
  165. >>> from celery.task import apply_async
  166. >>> from myapp.tasks import RefreshFeedTask
  167. >>> apply_async(RefreshFeedTask, args=["http://cnn.com/rss"],
  168. ... routing_key="feed.importer")