yangck
/
celery


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346
							<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>celery - Distributed Task Queue for Django. &mdash; Celery v0.3.2 (unstable) documentation</title>
    <link rel="stylesheet" href="static/agogo.css" type="text/css" />
    <link rel="stylesheet" href="static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
      URL_ROOT:    '',
      VERSION:     '0.3.2 (unstable)',
      COLLAPSE_MODINDEX: false,
      FILE_SUFFIX: '.html',
      HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="static/jquery.js"></script>
    <script type="text/javascript" src="static/doctools.js"></script>
    <link rel="top" title="Celery v0.3.2 (unstable) documentation" href="index.html" />
    <link rel="next" title="Frequently Asked Questions" href="faq.html" />
    <link rel="prev" title="Welcome to Celery’s documentation!" href="index.html" /> 
  </head>
  <body>

    <div class="header-wrapper">
      <div class="header">
	<h1><a href="index.html">Celery v0.3.2 (unstable) documentation</a></h1>
	<div class="rel">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a> |
          <a href="modindex.html" title="Global Module Index"
             accesskey="M">modules</a> |
          <a href="faq.html" title="Frequently Asked Questions"
             accesskey="N">next</a> |
          <a href="index.html" title="Welcome to Celery’s documentation!"
             accesskey="P">previous</a>
	</div>
      </div>
    </div>

    <div class="content-wrapper">
      <div class="content">
	<div class="document">
	  <div class="documentwrapper">
	    <div class="bodywrapper">
	      <div class="body">
		
  <div class="section" id="celery-distributed-task-queue-for-django">
<h1>celery - Distributed Task Queue for Django.<a class="headerlink" href="#celery-distributed-task-queue-for-django" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field"><th class="field-name">Version:</th><td class="field-body">0.3.2</td>
</tr>
</tbody>
</table>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p><tt class="docutils literal"><span class="pre">celery</span></tt> is a distributed task queue framework for Django.</p>
<p>It is used for executing tasks <em>asynchronously</em>, routed to one or more
worker servers, running concurrently using multiprocessing.</p>
<p>It is designed to solve certain problems related to running websites
demanding high-availability and performance.</p>
<p>It is perfect for filling caches, posting updates to twitter, mass
downloading data like syndication feeds or web scraping. Use-cases are
plentiful. Implementing these features asynchronously using <tt class="docutils literal"><span class="pre">celery</span></tt> is
easy and fun, and the performance improvements can make it more than
worthwhile.</p>
</div>
<div class="section" id="features">
<h2>Features<a class="headerlink" href="#features" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>Uses AMQP messaging (RabbitMQ, ZeroMQ) to route tasks to the
worker servers.</li>
<li>You can run as many worker servers as you want, and still
be <em>guaranteed that the task is only executed once.</em></li>
<li>Tasks are executed <em>concurrently</em> using the Python 2.6
<tt class="docutils literal"><span class="pre">multiprocessing</span></tt> module (also available as a back-port
to older python versions)</li>
<li>Supports <em>periodic tasks</em>, which makes it a (better) replacement
for cronjobs.</li>
<li>When a task has been executed, the return value is stored using either
a MySQL/Oracle/PostgreSQL/SQLite database, memcached,
or Tokyo Tyrant back-end.</li>
<li>If the task raises an exception, the exception instance is stored,
instead of the return value.</li>
<li>All tasks has a Universally Unique Identifier (UUID), which is the
task id, used for querying task status and return values.</li>
<li>Supports <em>task-sets</em>, which is a task consisting of several sub-tasks.
You can find out how many, or if all of the sub-tasks has been executed.
Excellent for progress-bar like functionality.</li>
<li>Has a <tt class="docutils literal"><span class="pre">map</span></tt> like function that uses tasks, called <tt class="docutils literal"><span class="pre">dmap</span></tt>.</li>
<li>However, you rarely want to wait for these results in a web-environment.
You&#8217;d rather want to use Ajax to poll the task status, which is
available from a URL like <tt class="docutils literal"><span class="pre">celery/&lt;task_id&gt;/status/</span></tt>. This view
returns a JSON-serialized data structure containing the task status,
and the return value if completed, or exception on failure.</li>
</ul>
</div>
<div class="section" id="api-reference-documentation">
<h2>API Reference Documentation<a class="headerlink" href="#api-reference-documentation" title="Permalink to this headline">¶</a></h2>
<p>The <a class="reference external" href="http://ask.github.com/celery/">API Reference</a> is hosted at Github
(<a class="reference external" href="http://ask.github.com/celery">http://ask.github.com/celery</a>)</p>
</div>
<div class="section" id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h2>
<p>You can install <tt class="docutils literal"><span class="pre">celery</span></tt> either via the Python Package Index (PyPI)
or from source.</p>
<p>To install using <tt class="docutils literal"><span class="pre">pip</span></tt>,:</p>
<div class="highlight-python"><pre>$ pip install celery</pre>
</div>
<p>To install using <tt class="docutils literal"><span class="pre">easy_install</span></tt>,:</p>
<div class="highlight-python"><pre>$ easy_install celery</pre>
</div>
<p>If you have downloaded a source tarball you can install it
by doing the following,:</p>
<div class="highlight-python"><pre>$ python setup.py build
# python setup.py install # as root</pre>
</div>
</div>
<div class="section" id="usage">
<h2>Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h2>
<div class="section" id="installing-rabbitmq">
<h3>Installing RabbitMQ<a class="headerlink" href="#installing-rabbitmq" title="Permalink to this headline">¶</a></h3>
<p>See <a class="reference external" href="http://www.rabbitmq.com/install.html">Installing RabbitMQ</a> over at RabbitMQ&#8217;s website. For Mac OS X
see <a class="reference external" href="http://playtype.net/past/2008/10/9/installing_rabbitmq_on_osx/">Installing RabbitMQ on OS X</a>.</p>
</div>
<div class="section" id="setting-up-rabbitmq">
<h3>Setting up RabbitMQ<a class="headerlink" href="#setting-up-rabbitmq" title="Permalink to this headline">¶</a></h3>
<p>To use celery we need to create a RabbitMQ user, a virtual host and
allow that user access to that virtual host:</p>
<div class="highlight-python"><pre>$ rabbitmqctl add_user myuser mypassword

$ rabbitmqctl add_vhost myvhost

$ rabbitmqctl map_user_vhost myuser myvhost</pre>
</div>
</div>
<div class="section" id="configuring-your-django-project-to-use-celery">
<h3>Configuring your Django project to use Celery<a class="headerlink" href="#configuring-your-django-project-to-use-celery" title="Permalink to this headline">¶</a></h3>
<p>You only need three simple steps to use celery with your Django project.</p>
<ol class="arabic">
<li><p class="first">Add <tt class="docutils literal"><span class="pre">celery</span></tt> to <tt class="docutils literal"><span class="pre">INSTALLED_APPS</span></tt>.</p>
</li>
<li><p class="first">Create the celery database tables:</p>
<div class="highlight-python"><pre>$ python manage.py syncdb</pre>
</div>
</li>
<li><dl class="first docutils">
<dt>Configure celery to use the AMQP user and virtual host we created</dt>
<dd><p class="first">before, by adding the following to your <tt class="docutils literal"><span class="pre">settings.py</span></tt>:</p>
<div class="last highlight-python"><div class="highlight"><pre><span class="n">AMQP_HOST</span> <span class="o">=</span> <span class="s">&quot;localhost&quot;</span>
<span class="n">AMQP_PORT</span> <span class="o">=</span> <span class="mf">5672</span>
<span class="n">AMQP_USER</span> <span class="o">=</span> <span class="s">&quot;myuser&quot;</span>
<span class="n">AMQP_PASSWORD</span> <span class="o">=</span> <span class="s">&quot;mypassword&quot;</span>
<span class="n">AMQP_VHOST</span> <span class="o">=</span> <span class="s">&quot;myvhost&quot;</span>
</pre></div>
</div>
</dd>
</dl>
</li>
</ol>
<p>That&#8217;s it.</p>
<p>There are more options available, like how many processes you want to process
work in parallel (the <tt class="docutils literal"><span class="pre">CELERY_CONCURRENCY</span></tt> setting), and the backend used
for storing task statuses. But for now, this should do. For all of the options
available, please consult the <a class="reference external" href="http://ask.github.com/celery/">API Reference</a></p>
<p><strong>Note</strong>: If you&#8217;re using SQLite as the Django database back-end,
<tt class="docutils literal"><span class="pre">celeryd</span></tt> will only be able to process one task at a time, this is
because SQLite doesn&#8217;t allow concurrent writes.</p>
</div>
<div class="section" id="running-the-celery-worker-daemon">
<h3>Running the celery worker daemon<a class="headerlink" href="#running-the-celery-worker-daemon" title="Permalink to this headline">¶</a></h3>
<p>To test this we&#8217;ll be running the worker daemon in the foreground, so we can
see what&#8217;s going on without consulting the logfile:</p>
<div class="highlight-python"><pre>$ python manage.py celeryd</pre>
</div>
<p>However, in production you&#8217;ll probably want to run the worker in the
background as a daemon instead:</p>
<div class="highlight-python"><pre>$ python manage.py celeryd --daemon</pre>
</div>
<p>For help on command line arguments to the worker daemon, you can execute the
help command:</p>
<div class="highlight-python"><pre>$ python manage.py help celeryd</pre>
</div>
</div>
<div class="section" id="defining-and-executing-tasks">
<h3>Defining and executing tasks<a class="headerlink" href="#defining-and-executing-tasks" title="Permalink to this headline">¶</a></h3>
<p><strong>Please note</strong> All of these tasks has to be stored in a real module, they can&#8217;t
be defined in the python shell or ipython/bpython. This is because the celery
worker server needs access to the task function to be able to run it.
So while it looks like we use the python shell to define the tasks in these
examples, you can&#8217;t do it this way. Put them in the <tt class="docutils literal"><span class="pre">tasks</span></tt> module of your
Django application. The worker daemon will automatically load any <tt class="docutils literal"><span class="pre">tasks.py</span></tt>
file for all of the applications listed in <tt class="docutils literal"><span class="pre">settings.INSTALLED_APPS</span></tt>.
Executing tasks using <tt class="docutils literal"><span class="pre">delay</span></tt> and <tt class="docutils literal"><span class="pre">apply_async</span></tt> can be done from the
python shell, but keep in mind that since arguments are pickled, you can&#8217;t
use custom classes defined in the shell session.</p>
<p>While you can use regular functions, the recommended way is to define
a task class. With this way you can cleanly upgrade the task to use the more
advanced features of celery later.</p>
<p>This is a task that basically does nothing but take some arguments,
and return a value:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">class</span> <span class="nc">MyTask</span><span class="p">(</span><span class="n">Task</span><span class="p">):</span>
<span class="gp">... </span>    <span class="n">name</span> <span class="o">=</span> <span class="s">&quot;myapp.mytask&quot;</span>
<span class="gp">... </span>    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">some_arg</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="gp">... </span>        <span class="n">logger</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_logger</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="gp">... </span>        <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">&quot;Did something: </span><span class="si">%s</span><span class="s">&quot;</span> <span class="o">%</span> <span class="n">some_arg</span><span class="p">)</span>
<span class="gp">... </span>        <span class="k">return</span> <span class="mf">42</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">tasks</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyTask</span><span class="p">)</span>
</pre></div>
</div>
<p>Now if we want to execute this task, we can use the <tt class="docutils literal"><span class="pre">delay</span></tt> method of the
task class (this is a handy shortcut to the <tt class="docutils literal"><span class="pre">apply_async</span></tt> method which gives
you greater control of the task execution).</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">myapp.tasks</span> <span class="kn">import</span> <span class="n">MyTask</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">MyTask</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="n">some_arg</span><span class="o">=</span><span class="s">&quot;foo&quot;</span><span class="p">)</span>
</pre></div>
</div>
<p>At this point, the task has been sent to the message broker. The message
broker will hold on to the task until a celery worker server has successfully
picked it up.</p>
<p>Right now we have to check the celery worker logfiles to know what happened with
the task. This is because we didn&#8217;t keep the <tt class="docutils literal"><span class="pre">AsyncResult</span></tt> object returned
by <tt class="docutils literal"><span class="pre">delay</span></tt>.</p>
<p>The <tt class="docutils literal"><span class="pre">AsyncResult</span></tt> lets us find the state of the task, wait for the task to
finish and get its return value (or exception if the task failed).</p>
<p>So, let&#8217;s execute the task again, but this time we&#8217;ll keep track of the task:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">result</span> <span class="o">=</span> <span class="n">MyTask</span><span class="o">.</span><span class="n">delay</span><span class="p">(</span><span class="s">&quot;do_something&quot;</span><span class="p">,</span> <span class="n">some_arg</span><span class="o">=</span><span class="s">&quot;foo bar baz&quot;</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">ready</span><span class="p">()</span> <span class="c"># returns True if the task has finished processing.</span>
<span class="go">False</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">result</span> <span class="c"># task is not ready, so no return value yet.</span>
<span class="go">None</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>   <span class="c"># Waits until the task is done and return the retval.</span>
<span class="go">42</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">result</span>
<span class="go">42</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">result</span><span class="o">.</span><span class="n">success</span><span class="p">()</span> <span class="c"># returns True if the task didn&#39;t end in failure.</span>
<span class="go">True</span>
</pre></div>
</div>
<p>If the task raises an exception, the <tt class="docutils literal"><span class="pre">result.success()</span></tt> will be <tt class="xref docutils literal"><span class="pre">False</span></tt>,
and <tt class="docutils literal"><span class="pre">result.result</span></tt> will contain the exception instance raised.</p>
</div>
<div class="section" id="auto-discovery-of-tasks">
<h3>Auto-discovery of tasks<a class="headerlink" href="#auto-discovery-of-tasks" title="Permalink to this headline">¶</a></h3>
<p><tt class="docutils literal"><span class="pre">celery</span></tt> has an auto-discovery feature like the Django Admin, that
automatically loads any <tt class="docutils literal"><span class="pre">tasks.py</span></tt> module in the applications listed
in <tt class="docutils literal"><span class="pre">settings.INSTALLED_APPS</span></tt>. This autodiscovery is used by the celery
worker to find registered tasks for your Django project.</p>
</div>
<div class="section" id="periodic-tasks">
<h3>Periodic Tasks<a class="headerlink" href="#periodic-tasks" title="Permalink to this headline">¶</a></h3>
<p>Periodic tasks are tasks that are run every <tt class="docutils literal"><span class="pre">n</span></tt> seconds.
Here&#8217;s an example of a periodic task:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">celery.task</span> <span class="kn">import</span> <span class="n">tasks</span><span class="p">,</span> <span class="n">PeriodicTask</span>
<span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">timedelta</span>
<span class="gp">&gt;&gt;&gt; </span><span class="k">class</span> <span class="nc">MyPeriodicTask</span><span class="p">(</span><span class="n">PeriodicTask</span><span class="p">):</span>
<span class="gp">... </span>    <span class="n">name</span> <span class="o">=</span> <span class="s">&quot;foo.my-periodic-task&quot;</span>
<span class="gp">... </span>    <span class="n">run_every</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="mf">30</span><span class="p">)</span>
<span class="gp">...</span>
<span class="gp">... </span>    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="gp">... </span>        <span class="n">logger</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_logger</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="gp">... </span>        <span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">&quot;Running periodic task!&quot;</span><span class="p">)</span>
<span class="gp">...</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">tasks</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">MyPeriodicTask</span><span class="p">)</span>
</pre></div>
</div>
<p><strong>Note:</strong> Periodic tasks does not support arguments, as this doesn&#8217;t
really make sense.</p>
</div>
</div>
<div class="section" id="license">
<h2>License<a class="headerlink" href="#license" title="Permalink to this headline">¶</a></h2>
<p>This software is licensed under the <tt class="docutils literal"><span class="pre">New</span> <span class="pre">BSD</span> <span class="pre">License</span></tt>. See the <tt class="docutils literal"><span class="pre">LICENSE</span></tt>
file in the top distribution directory for the full license text.</p>
</div>
</div>


	      </div>
	    </div>
	  </div>
	</div>
	<div class="sidebar">
	  <h3>Contents</h3>
	  <ul class="current">
<li class="toctree-l1 current"><a class="current reference external" href="">celery - Distributed Task Queue for Django.</a><ul>
<li class="toctree-l2"><a class="reference external" href="#introduction">Introduction</a></li>
<li class="toctree-l2"><a class="reference external" href="#features">Features</a></li>
<li class="toctree-l2"><a class="reference external" href="#api-reference-documentation">API Reference Documentation</a></li>
<li class="toctree-l2"><a class="reference external" href="#installation">Installation</a></li>
<li class="toctree-l2"><a class="reference external" href="#usage">Usage</a></li>
<li class="toctree-l2"><a class="reference external" href="#license">License</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference external" href="faq.html">Frequently Asked Questions</a></li>
<li class="toctree-l1"><a class="reference external" href="reference/index.html">Module API Reference</a></li>
<li class="toctree-l1"><a class="reference external" href="changelog.html">Change history</a></li>
</ul>

	  <h3 style="margin-top: 1.5em;">Search</h3>
	  <form class="search" action="search.html" method="get">
            <input type="text" name="q" size="18" />
            <input type="submit" value="Go" />
            <input type="hidden" name="check_keywords" value="yes" />
            <input type="hidden" name="area" value="default" />
          </form>
          <p class="searchtip" style="font-size: 90%">
            Enter search terms or a module, class or function name.
          </p>
	</div>
	<div class="clearer"></div>
      </div>
    </div>

    <div class="footer-wrapper">
      <div class="footer">
	<div class="left">
          <a href="genindex.html" title="General Index"
             >index</a> |
          <a href="modindex.html" title="Global Module Index"
             >modules</a> |
          <a href="faq.html" title="Frequently Asked Questions"
             >next</a> |
          <a href="index.html" title="Welcome to Celery’s documentation!"
             >previous</a>
            <br/>
            <a href="sources/introduction.txt"
               rel="nofollow">Show Source</a>
	</div>

	<div class="right">
	  &copy; Copyright 2009, Ask Solem.<br/>
	  Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 0.6.1.
	</div>
	<div class="clearer"></div>
      </div>
    </div>

  </body>
</html>