tasks.rst 30 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999100010011002100310041005100610071008100910101011101210131014
  1. .. _guide-tasks:
  2. =======
  3. Tasks
  4. =======
  5. .. contents::
  6. :local:
  7. This guide gives an overview of how tasks are defined. For a complete
  8. listing of task attributes and methods, please see the
  9. :class:`API reference <celery.task.base.Task>`.
  10. .. _task-basics:
  11. Basics
  12. ======
  13. A task is a class that encapsulates a function and its execution options.
  14. Given a function ``create_user``, that takes two arguments: ``username`` and
  15. ``password``, you can create a task like this:
  16. .. code-block:: python
  17. from celery.task import Task
  18. from django.contrib.auth import User
  19. class CreateUserTask(Task):
  20. def run(self, username, password):
  21. User.objects.create(username=username, password=password)
  22. For convenience there is a shortcut decorator that turns any function into
  23. a task:
  24. .. code-block:: python
  25. from celery.decorators import task
  26. from django.contrib.auth import User
  27. @task
  28. def create_user(username, password):
  29. User.objects.create(username=username, password=password)
  30. The task decorator takes the same execution options as the
  31. :class:`~celery.task.base.Task` class does:
  32. .. code-block:: python
  33. @task(serializer="json")
  34. def create_user(username, password):
  35. User.objects.create(username=username, password=password)
  36. .. _task-keyword-arguments:
  37. Default keyword arguments
  38. =========================
  39. Celery supports a set of default arguments that can be forwarded to any task.
  40. Tasks can choose not to take these, or list the ones they want.
  41. The worker will do the right thing.
  42. The current default keyword arguments are:
  43. :task_id: The unique id of the executing task.
  44. :task_name: Name of the currently executing task.
  45. :task_retries: How many times the current task has been retried.
  46. An integer starting at ``0``.
  47. :task_is_eager: Set to :const:`True` if the task is executed locally in
  48. the client, kand not by a worker.
  49. :logfile: The log file, can be passed on to
  50. :meth:`~celery.task.base.Task.get_logger` to gain access to
  51. the workers log file. See `Logging`_.
  52. :loglevel: The current loglevel used.
  53. :delivery_info: Additional message delivery information. This is a mapping
  54. containing the exchange and routing key used to deliver this
  55. task. It's used by e.g. :meth:`~celery.task.base.Task.retry`
  56. to resend the task to the same destination queue.
  57. **NOTE** As some messaging backends doesn't have advanced routing
  58. capabilities, you can't trust the availability of keys in this mapping.
  59. .. _task-logging:
  60. Logging
  61. =======
  62. You can use the workers logger to add diagnostic output to
  63. the worker log:
  64. .. code-block:: python
  65. class AddTask(Task):
  66. def run(self, x, y, **kwargs):
  67. logger = self.get_logger(**kwargs)
  68. logger.info("Adding %s + %s" % (x, y))
  69. return x + y
  70. or using the decorator syntax:
  71. .. code-block:: python
  72. @task()
  73. def add(x, y, **kwargs):
  74. logger = add.get_logger(**kwargs)
  75. logger.info("Adding %s + %s" % (x, y))
  76. return x + y
  77. There are several logging levels available, and the workers ``loglevel``
  78. setting decides whether or not they will be written to the log file.
  79. Of course, you can also simply use ``print`` as anything written to standard
  80. out/-err will be written to the logfile as well.
  81. .. _task-retry:
  82. Retrying a task if something fails
  83. ==================================
  84. Simply use :meth:`~celery.task.base.Task.retry` to re-send the task.
  85. It will do the right thing, and respect the
  86. :attr:`~celery.task.base.Task.max_retries` attribute:
  87. .. code-block:: python
  88. @task()
  89. def send_twitter_status(oauth, tweet, **kwargs):
  90. try:
  91. twitter = Twitter(oauth)
  92. twitter.update_status(tweet)
  93. except (Twitter.FailWhaleError, Twitter.LoginError), exc:
  94. send_twitter_status.retry(args=[oauth, tweet], kwargs=kwargs, exc=exc)
  95. Here we used the ``exc`` argument to pass the current exception to
  96. :meth:`~celery.task.base.Task.retry`. At each step of the retry this exception
  97. is available as the tombstone (result) of the task. When
  98. :attr:`~celery.task.base.Task.max_retries` has been exceeded this is the
  99. exception raised. However, if an ``exc`` argument is not provided the
  100. :exc:`~celery.exceptions.RetryTaskError` exception is raised instead.
  101. **Important note:** The task has to take the magic keyword arguments
  102. in order for max retries to work properly, this is because it keeps track
  103. of the current number of retries using the ``task_retries`` keyword argument
  104. passed on to the task. In addition, it also uses the ``task_id`` keyword
  105. argument to use the same task id, and ``delivery_info`` to route the
  106. retried task to the same destination.
  107. .. _task-retry-custom-delay:
  108. Using a custom retry delay
  109. --------------------------
  110. When a task is to be retried, it will wait for a given amount of time
  111. before doing so. The default delay is in the
  112. :attr:`~celery.task.base.Task.default_retry_delay`
  113. attribute on the task. By default this is set to 3 minutes. Note that the
  114. unit for setting the delay is in seconds (int or float).
  115. You can also provide the ``countdown`` argument to
  116. :meth:`~celery.task.base.Task.retry` to override this default.
  117. .. code-block:: python
  118. class MyTask(Task):
  119. default_retry_delay = 30 * 60 # retry in 30 minutes
  120. def run(self, x, y, **kwargs):
  121. try:
  122. ...
  123. except Exception, exc:
  124. self.retry([x, y], kwargs, exc=exc,
  125. countdown=60) # override the default and
  126. # - retry in 1 minute
  127. .. _task-options:
  128. Task options
  129. ============
  130. General
  131. -------
  132. .. _task-general-options:
  133. .. attribute:: Task.name
  134. The name the task is registered as.
  135. You can set this name manually, or just use the default which is
  136. automatically generated using the module and class name. See
  137. :ref:`task-names`.
  138. .. attribute:: Task.abstract
  139. Abstract classes are not registered, but are used as the
  140. superclass when making new task types by subclassing.
  141. .. attribute:: Task.max_retries
  142. The maximum number of attempted retries before giving up.
  143. If this exceeds the :exc:`~celery.exceptions.MaxRetriesExceeded`
  144. an exception will be raised. *NOTE:* You have to :meth:`retry`
  145. manually, it's not something that happens automatically.
  146. .. attribute:: Task.default_retry_delay
  147. Default time in seconds before a retry of the task
  148. should be executed. Can be either :class:`int` or :class:`float`.
  149. Default is a 3 minute delay.
  150. .. attribute:: Task.rate_limit
  151. Set the rate limit for this task type, i.e. how many times in
  152. a given period of time is the task allowed to run.
  153. If this is :const:`None` no rate limit is in effect.
  154. If it is an integer, it is interpreted as "tasks per second".
  155. The rate limits can be specified in seconds, minutes or hours
  156. by appending ``"/s"``, ``"/m"`` or ``"/h"`` to the value.
  157. Example: ``"100/m"`` (hundred tasks a minute). Default is the
  158. :setting:`CELERY_DEFAULT_RATE_LIMIT` setting, which if not specified means
  159. rate limiting for tasks is disabled by default.
  160. .. attribute:: Task.ignore_result
  161. Don't store task state. Note that this means you can't use
  162. :class:`~celery.result.AsyncResult` to check if the task is ready,
  163. or get its return value.
  164. .. attribute:: Task.store_errors_even_if_ignored
  165. If :const:`True`, errors will be stored even if the task is configured
  166. to ignore results.
  167. .. attribute:: Task.send_error_emails
  168. Send an e-mail whenever a task of this type fails.
  169. Defaults to the :setting:`CELERY_SEND_TASK_ERROR_EMAILS` setting.
  170. See :ref:`conf-error-mails` for more information.
  171. .. attribute:: Task.error_whitelist
  172. If the sending of error e-emails is enabled for this task, then
  173. this is a whitelist of exceptions to actually send e-mails about.
  174. .. attribute:: Task.serializer
  175. A string identifying the default serialization
  176. method to use. Defaults to the :setting:`CELERY_TASK_SERIALIZER`
  177. setting. Can be ``pickle`` ``json``, ``yaml``, or any custom
  178. serialization methods that have been registered with
  179. :mod:`carrot.serialization.registry`.
  180. Please see :ref:`executing-serializers` for more information.
  181. .. attribute:: Task.backend
  182. The result store backend to use for this task. Defaults to the
  183. :setting:`CELERY_RESULT_BACKEND` setting.
  184. .. attribute:: Task.acks_late
  185. If set to :const:`True` messages for this task will be acknowledged
  186. **after** the task has been executed, not *just before*, which is
  187. the default behavior.
  188. Note that this means the task may be executed twice if the worker
  189. crashes in the middle of execution, which may be acceptable for some
  190. applications.
  191. The global default can be overriden by the :setting:`CELERY_ACKS_LATE`
  192. setting.
  193. .. _task-track-started:
  194. .. attribute:: Task.track_started
  195. If :const:`True` the task will report its status as "started"
  196. when the task is executed by a worker.
  197. The default value is :const:`False` as the normal behaviour is to not
  198. report that level of granularity. Tasks are either pending, finished,
  199. or waiting to be retried. Having a "started" status can be useful for
  200. when there are long running tasks and there is a need to report which
  201. task is currently running.
  202. The hostname and pid of the worker executing the task
  203. will be avaiable in the state metadata (e.g. ``result.info["pid"]``)
  204. The global default can be overridden by the
  205. :setting:`CELERY_TRACK_STARTED` setting.
  206. .. seealso::
  207. The API reference for :class:`~celery.task.base.Task`.
  208. .. _task-message-options:
  209. Message and routing options
  210. ---------------------------
  211. .. attribute:: Task.queue
  212. Use the routing settings from a queue defined in :setting:`CELERY_QUEUES`.
  213. If defined the :attr:`exchange` and :attr:`routing_key` options will be
  214. ignored.
  215. .. attribute:: Task.exchange
  216. Override the global default ``exchange`` for this task.
  217. .. attribute:: Task.routing_key
  218. Override the global default ``routing_key`` for this task.
  219. .. attribute:: Task.mandatory
  220. If set, the task message has mandatory routing. By default the task
  221. is silently dropped by the broker if it can't be routed to a queue.
  222. However -- If the task is mandatory, an exception will be raised
  223. instead.
  224. Not supported by amqplib.
  225. .. attribute:: Task.immediate
  226. Request immediate delivery. If the task cannot be routed to a
  227. task worker immediately, an exception will be raised. This is
  228. instead of the default behavior, where the broker will accept and
  229. queue the task, but with no guarantee that the task will ever
  230. be executed.
  231. Not supported by amqplib.
  232. .. attribute:: Task.priority
  233. The message priority. A number from 0 to 9, where 0 is the
  234. highest priority.
  235. Not supported by RabbitMQ.
  236. .. seealso::
  237. :ref:`executing-routing` for more information about message options,
  238. and :ref:`guide-routing`.
  239. .. _task-names:
  240. Task names
  241. ==========
  242. The task type is identified by the *task name*.
  243. If not provided a name will be automatically generated using the module
  244. and class name.
  245. For example:
  246. .. code-block:: python
  247. >>> @task(name="sum-of-two-numbers")
  248. >>> def add(x, y):
  249. ... return x + y
  250. >>> add.name
  251. 'sum-of-two-numbers'
  252. The best practice is to use the module name as a prefix to classify the
  253. tasks using namespaces. This way the name won't collide with the name from
  254. another module::
  255. .. code-block:: python
  256. >>> @task(name="tasks.add")
  257. >>> def add(x, y):
  258. ... return x + y
  259. >>> add.name
  260. 'tasks.add'
  261. Which is exactly the name that is automatically generated for this
  262. task if the module name is "tasks.py":
  263. .. code-block:: python
  264. >>> @task()
  265. >>> def add(x, y):
  266. ... return x + y
  267. >>> add.name
  268. 'tasks.add'
  269. .. _task-naming-relative-imports:
  270. Automatic naming and relative imports
  271. -------------------------------------
  272. Relative imports and automatic name generation does not go well together,
  273. so if you're using relative imports you should set the name explicitly.
  274. For example if the client imports the module "myapp.tasks" as ".tasks", and
  275. the worker imports the module as "myapp.tasks", the generated names won't match
  276. and an :exc:`~celery.exceptions.NotRegistered` error will be raised by the worker.
  277. This is also the case if using Django and using ``project.myapp``::
  278. INSTALLED_APPS = ("project.myapp", )
  279. The worker will have the tasks registered as "project.myapp.tasks.*",
  280. while this is what happens in the client if the module is imported as
  281. "myapp.tasks":
  282. .. code-block:: python
  283. >>> from myapp.tasks import add
  284. >>> add.name
  285. 'myapp.tasks.add'
  286. For this reason you should never use "project.app", but rather
  287. add the project directory to the Python path::
  288. import os
  289. import sys
  290. sys.path.append(os.getcwd())
  291. INSTALLED_APPS = ("myapp", )
  292. This makes more sense from the reusable app perspective anyway.
  293. .. _task-states:
  294. Task States
  295. ===========
  296. During its lifetime a task will transition through several possible states,
  297. and each state may have arbitrary metadata attached to it. When a task
  298. moves into a new state the previous state is
  299. forgotten about, but some transitions can be deducted, (e.g. a task now
  300. in the :state:`FAILED` state, is implied to have been in the
  301. :state:`STARTED` state at some point).
  302. There are also sets of states, like the set of
  303. :state:`failure states <FAILURE_STATES>`, and the set of
  304. :state:`ready states <READY_STATES>`.
  305. The client uses the membership of these sets to decide whether
  306. the exception should be re-raised (:state:`PROPAGATE_STATES`), or whether
  307. the result can be cached (it can if the task is ready).
  308. You can also define :ref:`custom-states`.
  309. .. _task-builtin-states:
  310. Built-in States
  311. ---------------
  312. .. state:: PENDING
  313. PENDING
  314. ~~~~~~~
  315. Task is waiting for execution or unknown.
  316. Any task id that is not know is implied to be in the pending state.
  317. .. state:: STARTED
  318. STARTED
  319. ~~~~~~~
  320. Task has been started.
  321. Not reported by default, to enable please see :ref:`task-track-started`.
  322. :metadata: ``pid`` and ``hostname`` of the worker process executing
  323. the task.
  324. .. state:: SUCCESS
  325. SUCCESS
  326. ~~~~~~~
  327. Task has been successfully executed.
  328. :metadata: ``result`` contains the return value of the task.
  329. :propagates: Yes
  330. :ready: Yes
  331. .. state:: FAILURE
  332. FAILURE
  333. ~~~~~~~
  334. Task execution resulted in failure.
  335. :metadata: ``result`` contains the exception occured, and ``traceback``
  336. contains the backtrace of the stack at the point when the
  337. exception was raised.
  338. :propagates: Yes
  339. .. state:: RETRY
  340. RETRY
  341. ~~~~~
  342. Task is being retried.
  343. :metadata: ``result`` contains the exception that caused the retry,
  344. and ``traceback`` contains the backtrace of the stack at the point
  345. when the exceptions was raised.
  346. :propagates: No
  347. .. state:: REVOKED
  348. REVOKED
  349. ~~~~~~~
  350. Task has been revoked.
  351. :propagates: Yes
  352. Custom states
  353. -------------
  354. You can easily define your own states, all you need is a unique name.
  355. The name of the state is usually an uppercase string. As an example
  356. you could have a look at :mod:`abortable tasks <~celery.contrib.abortable>`
  357. wich defines its own custom :state:`ABORTED` state.
  358. Use :meth:`Task.update_state <celery.task.base.Task.update_state>` to
  359. update a tasks state::
  360. @task
  361. def upload_files(filenames, **kwargs):
  362. for i, file in enumerate(filenames):
  363. upload_files.update_state(kwargs["task_id"], "PROGRESS",
  364. {"current": i, "total": len(filenames)})
  365. Here we created the state ``"PROGRESS"``, which tells any application
  366. aware of this state that the task is currently in progress, and also where
  367. it is in the process by having ``current`` and ``total`` counts as part of the
  368. state metadata. This can then be used to create e.g. progress bars.
  369. .. _task-how-they-work:
  370. How it works
  371. ============
  372. Here comes the technical details, this part isn't something you need to know,
  373. but you may be interested.
  374. All defined tasks are listed in a registry. The registry contains
  375. a list of task names and their task classes. You can investigate this registry
  376. yourself:
  377. .. code-block:: python
  378. >>> from celery import registry
  379. >>> from celery import task
  380. >>> registry.tasks
  381. {'celery.delete_expired_task_meta':
  382. <PeriodicTask: celery.delete_expired_task_meta (periodic)>,
  383. 'celery.task.http.HttpDispatchTask':
  384. <Task: celery.task.http.HttpDispatchTask (regular)>,
  385. 'celery.execute_remote':
  386. <Task: celery.execute_remote (regular)>,
  387. 'celery.map_async':
  388. <Task: celery.map_async (regular)>,
  389. 'celery.ping':
  390. <Task: celery.ping (regular)>}
  391. This is the list of tasks built-in to celery. Note that we had to import
  392. ``celery.task`` first for these to show up. This is because the tasks will
  393. only be registered when the module they are defined in is imported.
  394. The default loader imports any modules listed in the
  395. :setting:`CELERY_IMPORTS` setting.
  396. The entity responsible for registering your task in the registry is a
  397. meta class, :class:`~celery.task.base.TaskType`. This is the default
  398. meta class for :class:`~celery.task.base.Task`.
  399. If you want to register your task manually you can set mark the
  400. task as :attr:`~celery.task.base.Task.abstract`:
  401. .. code-block:: python
  402. class MyTask(Task):
  403. abstract = True
  404. This way the task won't be registered, but any task subclassing it will be.
  405. When tasks are sent, we don't send any actual function code, just the name
  406. of the task to execute. When the worker then receives the message it can look
  407. up th ename in its task registry to find the execution code.
  408. This means that your workers should always be updated with the same software
  409. as the client. This is a drawback, but the alternative is a technical
  410. challenge that has yet to be solved.
  411. .. _task-best-practices:
  412. Tips and Best Practices
  413. =======================
  414. .. _task-ignore_results:
  415. Ignore results you don't want
  416. -----------------------------
  417. If you don't care about the results of a task, be sure to set the
  418. :attr:`~celery.task.base.Task.ignore_result` option, as storing results
  419. wastes time and resources.
  420. .. code-block:: python
  421. @task(ignore_result=True)
  422. def mytask(...)
  423. something()
  424. Results can even be disabled globally using the :setting:`CELERY_IGNORE_RESULT`
  425. setting.
  426. .. _task-disable-rate-limits:
  427. Disable rate limits if they're not used
  428. ---------------------------------------
  429. Disabling rate limits altogether is recommended if you don't have
  430. any tasks using them. This is because the rate limit subsystem introduces
  431. quite a lot of complexity.
  432. Set the :setting:`CELERY_DISABLE_RATE_LIMITS` setting to globally disable
  433. rate limits:
  434. .. code-block:: python
  435. CELERY_DISABLE_RATE_LIMITS = True
  436. .. _task-synchronous-subtasks:
  437. Avoid launching synchronous subtasks
  438. ------------------------------------
  439. Having a task wait for the result of another task is really inefficient,
  440. and may even cause a deadlock if the worker pool is exhausted.
  441. Make your design asynchronous instead, for example by using *callbacks*.
  442. **Bad**:
  443. .. code-block:: python
  444. @task()
  445. def update_page_info(url):
  446. page = fetch_page.delay(url).get()
  447. info = parse_page.delay(url, page).get()
  448. store_page_info.delay(url, info)
  449. @task()
  450. def fetch_page(url):
  451. return myhttplib.get(url)
  452. @task()
  453. def parse_page(url, page):
  454. return myparser.parse_document(page)
  455. @task()
  456. def store_page_info(url, info):
  457. return PageInfo.objects.create(url, info)
  458. **Good**:
  459. .. code-block:: python
  460. @task(ignore_result=True)
  461. def update_page_info(url):
  462. # fetch_page -> parse_page -> store_page
  463. fetch_page.delay(url, callback=subtask(parse_page,
  464. callback=subtask(store_page_info)))
  465. @task(ignore_result=True)
  466. def fetch_page(url, callback=None):
  467. page = myhttplib.get(url)
  468. if callback:
  469. # The callback may have been serialized with JSON,
  470. # so best practice is to convert the subtask dict back
  471. # into a subtask object.
  472. subtask(callback).delay(url, page)
  473. @task(ignore_result=True)
  474. def parse_page(url, page, callback=None):
  475. info = myparser.parse_document(page)
  476. if callback:
  477. subtask(callback).delay(url, info)
  478. @task(ignore_result=True)
  479. def store_page_info(url, info):
  480. PageInfo.objects.create(url, info)
  481. We use :class:`~celery.task.sets.subtask` here to safely pass
  482. around the callback task. :class:`~celery.task.sets.subtask` is a
  483. subclass of dict used to wrap the arguments and execution options
  484. for a single task invocation.
  485. .. seealso::
  486. :ref:`sets-subtasks` for more information about subtasks.
  487. .. _task-performance-and-strategies:
  488. Performance and Strategies
  489. ==========================
  490. .. _task-granularity:
  491. Granularity
  492. -----------
  493. The task granularity is the amount of computation needed by each subtask.
  494. In general it is better to split the problem up into many small tasks, than
  495. have a few long running tasks.
  496. With smaller tasks you can process more tasks in parallel and the tasks
  497. won't run long enough to block the worker from processing other waiting tasks.
  498. However, executing a task does have overhead. A message needs to be sent, data
  499. may not be local, etc. So if the tasks are too fine-grained the additional
  500. overhead may not be worth it in the end.
  501. .. seealso::
  502. The book `Art of Concurrency`_ has a whole section dedicated to the topic
  503. of task granularity.
  504. .. _`Art of Concurrency`: http://oreilly.com/catalog/9780596521547
  505. .. _task-data-locality:
  506. Data locality
  507. -------------
  508. The worker processing the task should be as close to the data as
  509. possible. The best would be to have a copy in memory, the worst would be a
  510. full transfer from another continent.
  511. If the data is far away, you could try to run another worker at location, or
  512. if that's not possible - cache often used data, or preload data you know
  513. is going to be used.
  514. The easiest way to share data between workers is to use a distributed cache
  515. system, like `memcached`_.
  516. .. seealso::
  517. The paper `Distributed Computing Economics`_ by Jim Gray is an excellent
  518. introduction to the topic of data locality.
  519. .. _`Distributed Computing Economics`:
  520. http://research.microsoft.com/pubs/70001/tr-2003-24.pdf
  521. .. _`memcached`: http://memcached.org/
  522. .. _task-state:
  523. State
  524. -----
  525. Since celery is a distributed system, you can't know in which process, or
  526. on what machine the task will be executed. You can't even know if the task will
  527. run in a timely manner.
  528. The ancient async sayings tells us that “asserting the world is the
  529. responsibility of the task”. What this means is that the world view may
  530. have changed since the task was requested, so the task is responsible for
  531. making sure the world is how it should be; If you have a task
  532. that reindexes a search engine, and the search engine should only be reindexed
  533. at maximum every 5 minutes, then it must be the tasks responsibility to assert
  534. that, not the callers.
  535. Another gotcha is Django model objects. They shouldn't be passed on as arguments
  536. to tasks. It's almost always better to re-fetch the object from the
  537. database when the task is running instead, as using old data may lead
  538. to race conditions.
  539. Imagine the following scenario where you have an article and a task
  540. that automatically expands some abbreviations in it:
  541. .. code-block:: python
  542. class Article(models.Model):
  543. title = models.CharField()
  544. body = models.TextField()
  545. @task
  546. def expand_abbreviations(article):
  547. article.body.replace("MyCorp", "My Corporation")
  548. article.save()
  549. First, an author creates an article and saves it, then the author
  550. clicks on a button that initiates the abbreviation task.
  551. >>> article = Article.objects.get(id=102)
  552. >>> expand_abbreviations.delay(model_object)
  553. Now, the queue is very busy, so the task won't be run for another 2 minutes.
  554. In the meantime another author makes changes to the article, so
  555. when the task is finally run, the body of the article is reverted to the old
  556. version because the task had the old body in its argument.
  557. Fixing the race condition is easy, just use the article id instead, and
  558. re-fetch the article in the task body:
  559. .. code-block:: python
  560. @task
  561. def expand_abbreviations(article_id):
  562. article = Article.objects.get(id=article_id)
  563. article.body.replace("MyCorp", "My Corporation")
  564. article.save()
  565. >>> expand_abbreviations(article_id)
  566. There might even be performance benefits to this approach, as sending large
  567. messages may be expensive.
  568. .. _task-database-transactions:
  569. Database transactions
  570. ---------------------
  571. Let's have a look at another example:
  572. .. code-block:: python
  573. from django.db import transaction
  574. @transaction.commit_on_success
  575. def create_article(request):
  576. article = Article.objects.create(....)
  577. expand_abbreviations.delay(article.pk)
  578. This is a Django view creating an article object in the database,
  579. then passing the primary key to a task. It uses the `commit_on_success`
  580. decorator, which will commit the transaction when the view returns, or
  581. roll back if the view raises an exception.
  582. There is a race condition if the task starts executing
  583. before the transaction has been committed; The database object does not exist
  584. yet!
  585. The solution is to *always commit transactions before sending tasks
  586. depending on state from the current transaction*:
  587. .. code-block:: python
  588. @transaction.commit_manually
  589. def create_article(request):
  590. try:
  591. article = Article.objects.create(...)
  592. except:
  593. transaction.rollback()
  594. raise
  595. else:
  596. transaction.commit()
  597. expand_abbreviations.delay(article.pk)
  598. .. _task-example:
  599. Example
  600. =======
  601. Let's take a real wold example; A blog where comments posted needs to be
  602. filtered for spam. When the comment is created, the spam filter runs in the
  603. background, so the user doesn't have to wait for it to finish.
  604. We have a Django blog application allowing comments
  605. on blog posts. We'll describe parts of the models/views and tasks for this
  606. application.
  607. blog/models.py
  608. --------------
  609. The comment model looks like this:
  610. .. code-block:: python
  611. from django.db import models
  612. from django.utils.translation import ugettext_lazy as _
  613. class Comment(models.Model):
  614. name = models.CharField(_("name"), max_length=64)
  615. email_address = models.EmailField(_("e-mail address"))
  616. homepage = models.URLField(_("home page"),
  617. blank=True, verify_exists=False)
  618. comment = models.TextField(_("comment"))
  619. pub_date = models.DateTimeField(_("Published date"),
  620. editable=False, auto_add_now=True)
  621. is_spam = models.BooleanField(_("spam?"),
  622. default=False, editable=False)
  623. class Meta:
  624. verbose_name = _("comment")
  625. verbose_name_plural = _("comments")
  626. In the view where the comment is posted, we first write the comment
  627. to the database, then we launch the spam filter task in the background.
  628. .. _task-example-blog-views:
  629. blog/views.py
  630. -------------
  631. .. code-block:: python
  632. from django import forms
  633. from django.http import HttpResponseRedirect
  634. from django.template.context import RequestContext
  635. from django.shortcuts import get_object_or_404, render_to_response
  636. from blog import tasks
  637. from blog.models import Comment
  638. class CommentForm(forms.ModelForm):
  639. class Meta:
  640. model = Comment
  641. def add_comment(request, slug, template_name="comments/create.html"):
  642. post = get_object_or_404(Entry, slug=slug)
  643. remote_addr = request.META.get("REMOTE_ADDR")
  644. if request.method == "post":
  645. form = CommentForm(request.POST, request.FILES)
  646. if form.is_valid():
  647. comment = form.save()
  648. # Check spam asynchronously.
  649. tasks.spam_filter.delay(comment_id=comment.id,
  650. remote_addr=remote_addr)
  651. return HttpResponseRedirect(post.get_absolute_url())
  652. else:
  653. form = CommentForm()
  654. context = RequestContext(request, {"form": form})
  655. return render_to_response(template_name, context_instance=context)
  656. To filter spam in comments we use `Akismet`_, the service
  657. used to filter spam in comments posted to the free weblog platform
  658. `Wordpress`. `Akismet`_ is free for personal use, but for commercial use you
  659. need to pay. You have to sign up to their service to get an API key.
  660. To make API calls to `Akismet`_ we use the `akismet.py`_ library written by
  661. `Michael Foord`_.
  662. .. _task-example-blog-tasks:
  663. blog/tasks.py
  664. -------------
  665. .. code-block:: python
  666. from akismet import Akismet
  667. from celery.decorators import task
  668. from django.core.exceptions import ImproperlyConfigured
  669. from django.contrib.sites.models import Site
  670. from blog.models import Comment
  671. @task
  672. def spam_filter(comment_id, remote_addr=None, **kwargs):
  673. logger = spam_filter.get_logger(**kwargs)
  674. logger.info("Running spam filter for comment %s" % comment_id)
  675. comment = Comment.objects.get(pk=comment_id)
  676. current_domain = Site.objects.get_current().domain
  677. akismet = Akismet(settings.AKISMET_KEY, "http://%s" % domain)
  678. if not akismet.verify_key():
  679. raise ImproperlyConfigured("Invalid AKISMET_KEY")
  680. is_spam = akismet.comment_check(user_ip=remote_addr,
  681. comment_content=comment.comment,
  682. comment_author=comment.name,
  683. comment_author_email=comment.email_address)
  684. if is_spam:
  685. comment.is_spam = True
  686. comment.save()
  687. return is_spam
  688. .. _`Akismet`: http://akismet.com/faq/
  689. .. _`akismet.py`: http://www.voidspace.org.uk/downloads/akismet.py
  690. .. _`Michael Foord`: http://www.voidspace.org.uk/