introduction.html 22 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312
  1. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  2. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  3. <html xmlns="http://www.w3.org/1999/xhtml">
  4. <head>
  5. <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  6. <title>celery - Distributed Task Queue for Django. &mdash; Celery v0.2.0-pre3 documentation</title>
  7. <link rel="stylesheet" href="static/agogo.css" type="text/css" />
  8. <link rel="stylesheet" href="static/pygments.css" type="text/css" />
  9. <script type="text/javascript">
  10. var DOCUMENTATION_OPTIONS = {
  11. URL_ROOT: '',
  12. VERSION: '0.2.0-pre3',
  13. COLLAPSE_MODINDEX: false,
  14. FILE_SUFFIX: '.html',
  15. HAS_SOURCE: true
  16. };
  17. </script>
  18. <script type="text/javascript" src="static/jquery.js"></script>
  19. <script type="text/javascript" src="static/doctools.js"></script>
  20. <link rel="top" title="Celery v0.2.0-pre3 documentation" href="index.html" />
  21. <link rel="next" title="Tasks - celery.task" href="reference/celery.task.html" />
  22. <link rel="prev" title="Welcome to Celery’s documentation!" href="index.html" />
  23. </head>
  24. <body>
  25. <div class="header-wrapper">
  26. <div class="header">
  27. <h1><a href="index.html">Celery v0.2.0-pre3 documentation</a></h1>
  28. <div class="rel">
  29. <a href="genindex.html" title="General Index"
  30. accesskey="I">index</a> |
  31. <a href="modindex.html" title="Global Module Index"
  32. accesskey="M">modules</a> |
  33. <a href="reference/celery.task.html" title="Tasks - celery.task"
  34. accesskey="N">next</a> |
  35. <a href="index.html" title="Welcome to Celery’s documentation!"
  36. accesskey="P">previous</a>
  37. </div>
  38. </div>
  39. </div>
  40. <div class="content-wrapper">
  41. <div class="content">
  42. <div class="document">
  43. <div class="documentwrapper">
  44. <div class="bodywrapper">
  45. <div class="body">
  46. <div class="section" id="celery-distributed-task-queue-for-django">
  47. <h1>celery - Distributed Task Queue for Django.<a class="headerlink" href="#celery-distributed-task-queue-for-django" title="Permalink to this headline">¶</a></h1>
  48. <table class="docutils field-list" frame="void" rules="none">
  49. <col class="field-name" />
  50. <col class="field-body" />
  51. <tbody valign="top">
  52. <tr class="field"><th class="field-name">Authors:</th><td class="field-body">Ask Solem (<a class="reference external" href="mailto:askh&#37;&#52;&#48;opera&#46;com">askh<span>&#64;</span>opera<span>&#46;</span>com</a>)</td>
  53. </tr>
  54. <tr class="field"><th class="field-name">Version:</th><td class="field-body">0.2.0-pre3</td>
  55. </tr>
  56. </tbody>
  57. </table>
  58. <div class="section" id="introduction">
  59. <h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
  60. <p><tt class="docutils literal"><span class="pre">celery</span></tt> is a distributed task queue framework for Django.</p>
  61. <p>It is used for executing tasks <em>asynchronously</em>, routed to one or more
  62. worker servers, running concurrently using multiprocessing.</p>
  63. <p>It is designed to solve certain problems related to running websites
  64. demanding high-availability and performance.</p>
  65. <p>It is perfect for filling caches, posting updates to twitter, mass
  66. downloading data like syndication feeds or web scraping. Use-cases are
  67. plentiful. Implementing these features asynchronously using <tt class="docutils literal"><span class="pre">celery</span></tt> is
  68. easy and fun, and the performance improvements can make it more than
  69. worthwhile.</p>
  70. </div>
  71. <div class="section" id="features">
  72. <h2>Features<a class="headerlink" href="#features" title="Permalink to this headline">¶</a></h2>
  73. <ul class="simple">
  74. <li>Uses AMQP messaging (RabbitMQ, ZeroMQ) to route tasks to the
  75. worker servers.</li>
  76. <li>You can run as many worker servers as you want, and still
  77. be <em>guaranteed that the task is only executed once.</em></li>
  78. <li>Tasks are executed <em>concurrently</em> using the Python 2.6
  79. <tt class="docutils literal"><span class="pre">multiprocessing</span></tt> module (also available as a backport
  80. to older python versions)</li>
  81. <li>Supports <em>periodic tasks</em>, which makes it a (better) replacement
  82. for cronjobs.</li>
  83. <li>When a task has been executed, the return value is stored using either
  84. a MySQL/Oracle/PostgreSQL/SQLite database, memcached,
  85. or Tokyo Tyrant backend.</li>
  86. <li>If the task raises an exception, the exception instance is stored,
  87. instead of the return value.</li>
  88. <li>All tasks has a Universaly Unique Identifier (UUID), which is the
  89. task id, used for querying task status and return values.</li>
  90. <li>Supports <em>tasksets</em>, which is a task consisting of several subtasks.
  91. You can find out how many, or if all of the subtasks has been executed.
  92. Excellent for progress-bar like functionality.</li>
  93. <li>Has a <tt class="docutils literal"><span class="pre">map</span></tt> like function that uses tasks, called <tt class="docutils literal"><span class="pre">dmap</span></tt>.</li>
  94. <li>However, you rarely want to wait for these results in a web-environment.
  95. You&#8217;d rather want to use Ajax to poll the task status, which is
  96. available from a URL like <tt class="docutils literal"><span class="pre">celery/&lt;task_id&gt;/status/</span></tt>. This view
  97. returns a JSON-serialized data structure containing the task status,
  98. and the return value if completed, or exception on failure.</li>
  99. </ul>
  100. </div>
  101. <div class="section" id="api-reference-documentation">
  102. <h2>API Reference Documentation<a class="headerlink" href="#api-reference-documentation" title="Permalink to this headline">¶</a></h2>
  103. <p>The <a class="reference internal" href="#api-reference-documentation">API Reference Documentation</a> is hosted at Github.</p>
  104. <div class="section" id="installation">
  105. <h3>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h3>
  106. <p>You can install <tt class="docutils literal"><span class="pre">celery</span></tt> either via the Python Package Index (PyPI)
  107. or from source.</p>
  108. <p>To install using <tt class="docutils literal"><span class="pre">pip</span></tt>,:</p>
  109. <div class="highlight-python"><pre>$ pip install celery</pre>
  110. </div>
  111. <p>To install using <tt class="docutils literal"><span class="pre">easy_install</span></tt>,:</p>
  112. <div class="highlight-python"><pre>$ easy_install celery</pre>
  113. </div>
  114. <p>If you have downloaded a source tarball you can install it
  115. by doing the following,:</p>
  116. <div class="highlight-python"><pre>$ python setup.py build
  117. # python setup.py install # as root</pre>
  118. </div>
  119. </div>
  120. <div class="section" id="usage">
  121. <h3>Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h3>
  122. <p>Have to write a cool tutorial, but here is some simple usage info.</p>
  123. <p><em>Note</em> You need to have a AMQP message broker running, like <a class="reference external" href="http://www.rabbitmq.com">RabbitMQ</a>,
  124. and you need to have the amqp server setup in your settings file, as described
  125. in the <a class="reference external" href="http://pypi.python.org/pypi/carrot/0.3.3">carrot distribution README</a>.</p>
  126. <p><em>Note</em> If you&#8217;re running <tt class="docutils literal"><span class="pre">SQLite</span></tt> as the database backend, <tt class="docutils literal"><span class="pre">celeryd</span></tt> will
  127. only be able to process one message at a time, this is because <tt class="docutils literal"><span class="pre">SQLite</span></tt>
  128. doesn&#8217;t allow concurrent writes.</p>
  129. </div>
  130. </div>
  131. <div class="section" id="defining-tasks">
  132. <h2>Defining tasks<a class="headerlink" href="#defining-tasks" title="Permalink to this headline">¶</a></h2>
  133. <div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">tasks</span>
  134. <span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.log</span> <span class="kn">import</span> <span class="n">setup_logger</span>
  135. <span class="gp">&gt;&gt;&gt; </span><span class="k">def</span> <span class="nf">do_something</span><span class="p">(</span><span class="n">some_arg</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
  136. <span class="gp">... </span> <span class="n">logger</span> <span class="o">=</span> <span class="n">setup_logger</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
  137. <span class="gp">... </span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">&quot;Did something: </span><span class="si">%s</span><span class="s">&quot;</span> <span class="o">%</span> <span class="n">some_arg</span><span class="p">)</span>
  138. <span class="gp">... </span> <span class="k">return</span> <span class="mf">42</span>
  139. <span class="gp">&gt;&gt;&gt; </span><span class="n">task</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">do_something</span><span class="p">,</span> <span class="s">&quot;do_something&quot;</span><span class="p">)</span>
  140. </pre></div>
  141. </div>
  142. </div>
  143. <div class="section" id="tell-the-celery-daemon-to-run-a-task">
  144. <h2>Tell the celery daemon to run a task<a class="headerlink" href="#tell-the-celery-daemon-to-run-a-task" title="Permalink to this headline">¶</a></h2>
  145. <div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">delay_task</span>
  146. <span class="gp">&gt;&gt;&gt; </span><span class="n">delay_task</span><span class="p">(</span><span class="s">&quot;do_something&quot;</span><span class="p">,</span> <span class="n">some_arg</span><span class="o">=</span><span class="s">&quot;foo bar baz&quot;</span><span class="p">)</span>
  147. </pre></div>
  148. </div>
  149. </div>
  150. <div class="section" id="execute-a-task-and-get-its-return-value">
  151. <h2>Execute a task, and get its return value.<a class="headerlink" href="#execute-a-task-and-get-its-return-value" title="Permalink to this headline">¶</a></h2>
  152. <div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">delay_task</span>
  153. <span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">delay_task</span><span class="p">(</span><span class="s">&quot;do_something&quot;</span><span class="p">,</span> <span class="n">some_arg</span><span class="o">=</span><span class="s">&quot;foo bar baz&quot;</span><span class="p">)</span>
  154. <span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">ready</span><span class="p">()</span>
  155. <span class="go">False</span>
  156. <span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">()</span> <span class="c"># Waits until the task is done.</span>
  157. <span class="go">42</span>
  158. <span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">status</span><span class="p">()</span>
  159. <span class="go">&#39;DONE&#39;</span>
  160. </pre></div>
  161. </div>
  162. <p>If the task raises an exception, the tasks status will be <tt class="docutils literal"><span class="pre">FAILURE</span></tt>, and
  163. <tt class="docutils literal"><span class="pre">result.result</span></tt> will contain the exception instance raised.</p>
  164. </div>
  165. <div class="section" id="running-the-celery-daemon">
  166. <h2>Running the celery daemon<a class="headerlink" href="#running-the-celery-daemon" title="Permalink to this headline">¶</a></h2>
  167. <div class="highlight-python"><pre>$ cd mydjangoproject
  168. $ env DJANGO_SETTINGS_MODULE=settings celeryd
  169. [....]
  170. [2009-04-23 17:44:05,115: INFO/Process-1] Did something: foo bar baz
  171. [2009-04-23 17:44:05,118: INFO/MainProcess] Waiting for queue.</pre>
  172. </div>
  173. </div>
  174. <div class="section" id="autodiscovery-of-tasks">
  175. <h2>Autodiscovery of tasks<a class="headerlink" href="#autodiscovery-of-tasks" title="Permalink to this headline">¶</a></h2>
  176. <p><tt class="docutils literal"><span class="pre">celery</span></tt> has an autodiscovery feature like the Django Admin, that
  177. automatically loads any <tt class="docutils literal"><span class="pre">tasks.py</span></tt> module in the applications listed
  178. in <tt class="docutils literal"><span class="pre">settings.INSTALLED_APPS</span></tt>.</p>
  179. <p>A good place to add this command could be in your <tt class="docutils literal"><span class="pre">urls.py</span></tt>,</p>
  180. <div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">tasks</span>
  181. <span class="n">tasks</span><span class="o">.</span><span class="n">autodiscover</span><span class="p">()</span>
  182. </pre></div>
  183. </div>
  184. <p>Then you can add new tasks in your applications <tt class="docutils literal"><span class="pre">tasks.py</span></tt> module,</p>
  185. <div class="highlight-python"><pre>from celery.task import tasks
  186. from celery.log import setup_logger
  187. from clickcounter.models import ClickCount
  188. def increment_click(for_url, **kwargs):
  189. logger = setup_logger(**kwargs)
  190. clicks_for_url, cr = ClickCount.objects.get_or_create(url=for_url)
  191. clicks_for_url.clicks = clicks_for_url.clicks + 1
  192. clicks_for_url.save()
  193. logger.info("Incremented click count for %s (not at %d)" % (
  194. for_url, clicks_for_url.clicks)
  195. tasks.register(increment_click, "increment_click")</pre>
  196. </div>
  197. </div>
  198. <div class="section" id="periodic-tasks">
  199. <h2>Periodic Tasks<a class="headerlink" href="#periodic-tasks" title="Permalink to this headline">¶</a></h2>
  200. <p>Periodic tasks are tasks that are run every <tt class="docutils literal"><span class="pre">n</span></tt> seconds. They don&#8217;t
  201. support extra arguments. Here&#8217;s an example of a periodic task:</p>
  202. <div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">tasks</span><span class="p">,</span> <span class="n">PeriodicTask</span>
  203. <span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
  204. <span class="gp">&gt;&gt;&gt; </span><span class="k">class</span> <span class="nc">MyPeriodicTask</span><span class="p">(</span><span class="n">PeriodicTask</span><span class="p">):</span>
  205. <span class="gp">... </span> <span class="n">name</span> <span class="o">=</span> <span class="s">&quot;foo.my-periodic-task&quot;</span>
  206. <span class="gp">... </span> <span class="n">run_every</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mf">30</span><span class="p">)</span>
  207. <span class="gp">...</span>
  208. <span class="gp">... </span> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
  209. <span class="gp">... </span> <span class="n">logger</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_logger</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
  210. <span class="gp">... </span> <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">&quot;Running periodic task!&quot;</span><span class="p">)</span>
  211. <span class="gp">...</span>
  212. <span class="gp">&gt;&gt;&gt; </span><span class="n">tasks</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyPeriodicTask</span><span class="p">)</span>
  213. </pre></div>
  214. </div>
  215. <p>For periodic tasks to work you need to add <tt class="docutils literal"><span class="pre">celery</span></tt> to <tt class="docutils literal"><span class="pre">INSTALLED_APPS</span></tt>,
  216. and issue a <tt class="docutils literal"><span class="pre">syncdb</span></tt>.</p>
  217. <div class="section" id="license">
  218. <h3>License<a class="headerlink" href="#license" title="Permalink to this headline">¶</a></h3>
  219. <p>This software is licensed under the <tt class="docutils literal"><span class="pre">New</span> <span class="pre">BSD</span> <span class="pre">License</span></tt>. See the <tt class="docutils literal"><span class="pre">LICENSE</span></tt>
  220. file in the top distribution directory for the full license text.</p>
  221. </div>
  222. </div>
  223. </div>
  224. </div>
  225. </div>
  226. </div>
  227. </div>
  228. <div class="sidebar">
  229. <h3>Contents</h3>
  230. <ul class="current">
  231. <li class="toctree-l1 current"><a class="current reference external" href="">celery - Distributed Task Queue for Django.</a><ul>
  232. <li class="toctree-l2"><a class="reference external" href="#introduction">Introduction</a></li>
  233. <li class="toctree-l2"><a class="reference external" href="#features">Features</a></li>
  234. <li class="toctree-l2"><a class="reference external" href="#api-reference-documentation">API Reference Documentation</a></li>
  235. <li class="toctree-l2"><a class="reference external" href="#defining-tasks">Defining tasks</a></li>
  236. <li class="toctree-l2"><a class="reference external" href="#tell-the-celery-daemon-to-run-a-task">Tell the celery daemon to run a task</a></li>
  237. <li class="toctree-l2"><a class="reference external" href="#execute-a-task-and-get-its-return-value">Execute a task, and get its return value.</a></li>
  238. <li class="toctree-l2"><a class="reference external" href="#running-the-celery-daemon">Running the celery daemon</a></li>
  239. <li class="toctree-l2"><a class="reference external" href="#autodiscovery-of-tasks">Autodiscovery of tasks</a></li>
  240. <li class="toctree-l2"><a class="reference external" href="#periodic-tasks">Periodic Tasks</a></li>
  241. </ul>
  242. </li>
  243. <li class="toctree-l1"><a class="reference external" href="reference/celery.task.html">Tasks - celery.task</a></li>
  244. <li class="toctree-l1"><a class="reference external" href="reference/celery.result.html">Task Result - celery.result</a></li>
  245. <li class="toctree-l1"><a class="reference external" href="reference/celery.registry.html">Task Registry - celery.registry</a></li>
  246. <li class="toctree-l1"><a class="reference external" href="reference/celery.discovery.html">Task Discovery - celery.discovery</a></li>
  247. <li class="toctree-l1"><a class="reference external" href="reference/celery.worker.html">Multiprocessing Worker - celery.worker</a></li>
  248. <li class="toctree-l1"><a class="reference external" href="reference/celery.backends.html">Backends - celery.backends</a></li>
  249. <li class="toctree-l1"><a class="reference external" href="reference/celery.backends.base.html">Backend: Base - celery.backends.base</a></li>
  250. <li class="toctree-l1"><a class="reference external" href="reference/celery.backends.database.html">Backend: Database - celery.backends.database</a></li>
  251. <li class="toctree-l1"><a class="reference external" href="reference/celery.backends.cache.html">Backend: Cache - celery.backends.cache</a></li>
  252. <li class="toctree-l1"><a class="reference external" href="reference/celery.backends.tyrant.html">Backend: Tokyo Tyrant - celery.backends.tyrant</a></li>
  253. <li class="toctree-l1"><a class="reference external" href="reference/celery.conf.html">Configuration - celery.conf</a></li>
  254. <li class="toctree-l1"><a class="reference external" href="reference/celery.datastructures.html">Datastructures - celery.datastructures</a></li>
  255. <li class="toctree-l1"><a class="reference external" href="reference/celery.log.html">Logging - celery.log</a></li>
  256. <li class="toctree-l1"><a class="reference external" href="reference/celery.managers.html">Django Model Managers - celery.managers</a></li>
  257. <li class="toctree-l1"><a class="reference external" href="reference/celery.models.html">Django Models - celery.models</a></li>
  258. <li class="toctree-l1"><a class="reference external" href="reference/celery.messaging.html">Messaging - celery.messaging</a></li>
  259. <li class="toctree-l1"><a class="reference external" href="reference/celery.platform.html">Platform Specific - celery.platform</a></li>
  260. <li class="toctree-l1"><a class="reference external" href="reference/celery.timer.html">Timers - celery.timer</a></li>
  261. <li class="toctree-l1"><a class="reference external" href="reference/celery.bin.celeryd.html">Celery Worker Daemon - celery.bin.celeryd</a></li>
  262. </ul>
  263. <h3 style="margin-top: 1.5em;">Search</h3>
  264. <form class="search" action="search.html" method="get">
  265. <input type="text" name="q" size="18" />
  266. <input type="submit" value="Go" />
  267. <input type="hidden" name="check_keywords" value="yes" />
  268. <input type="hidden" name="area" value="default" />
  269. </form>
  270. <p class="searchtip" style="font-size: 90%">
  271. Enter search terms or a module, class or function name.
  272. </p>
  273. </div>
  274. <div class="clearer"></div>
  275. </div>
  276. </div>
  277. <div class="footer-wrapper">
  278. <div class="footer">
  279. <div class="left">
  280. <a href="genindex.html" title="General Index"
  281. >index</a> |
  282. <a href="modindex.html" title="Global Module Index"
  283. >modules</a> |
  284. <a href="reference/celery.task.html" title="Tasks - celery.task"
  285. >next</a> |
  286. <a href="index.html" title="Welcome to Celery’s documentation!"
  287. >previous</a>
  288. <br/>
  289. <a href="sources/introduction.txt"
  290. rel="nofollow">Show Source</a>
  291. </div>
  292. <div class="right">
  293. &copy; Copyright 2009, Ask Solem.<br/>
  294. Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 0.6.1.
  295. </div>
  296. <div class="clearer"></div>
  297. </div>
  298. </div>
  299. </body>
  300. </html>