periodic-tasks.rst 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466
  1. .. _guide-beat:
  2. ================
  3. Periodic Tasks
  4. ================
  5. .. contents::
  6. :local:
  7. Introduction
  8. ============
  9. :program:`celery beat` is a scheduler; It kicks off tasks at regular intervals,
  10. that are then executed by available worker nodes in the cluster.
  11. By default the entries are taken from the :setting:`beat_schedule` setting,
  12. but custom stores can also be used, like storing the entries in a SQL database.
  13. You have to ensure only a single scheduler is running for a schedule
  14. at a time, otherwise you'd end up with duplicate tasks. Using
  15. a centralized approach means the schedule doesn't have to be synchronized,
  16. and the service can operate without using locks.
  17. .. _beat-timezones:
  18. Time Zones
  19. ==========
  20. The periodic task schedules uses the UTC time zone by default,
  21. but you can change the time zone used using the :setting:`timezone`
  22. setting.
  23. An example time zone could be `Europe/London`:
  24. .. code-block:: python
  25. timezone = 'Europe/London'
  26. This setting must be added to your app, either by configuration it directly
  27. using (``app.conf.timezone = 'Europe/London'``), or by adding
  28. it to your configuration module if you have set one up using
  29. ``app.config_from_object``. See :ref:`celerytut-configuration` for
  30. more information about configuration options.
  31. The default scheduler (storing the schedule in the :file:`celerybeat-schedule`
  32. file) will automatically detect that the time zone has changed, and so will
  33. reset the schedule itself, but other schedulers may not be so smart (e.g., the
  34. Django database scheduler, see below) and in that case you'll have to reset the
  35. schedule manually.
  36. .. admonition:: Django Users
  37. Celery recommends and is compatible with the new ``USE_TZ`` setting introduced
  38. in Django 1.4.
  39. For Django users the time zone specified in the ``TIME_ZONE`` setting
  40. will be used, or you can specify a custom time zone for Celery alone
  41. by using the :setting:`timezone` setting.
  42. The database scheduler won't reset when timezone related settings
  43. change, so you must do this manually:
  44. .. code-block:: console
  45. $ python manage.py shell
  46. >>> from djcelery.models import PeriodicTask
  47. >>> PeriodicTask.objects.update(last_run_at=None)
  48. Django-Celery only supports Celery 4.0 and below, for Celery 4.0 and above, do as follow:
  49. .. code-block:: console
  50. $ python manage.py shell
  51. >>> from django_celery_beat.models import PeriodicTask
  52. >>> PeriodicTask.objects.update(last_run_at=None)
  53. .. _beat-entries:
  54. Entries
  55. =======
  56. To call a task periodically you have to add an entry to the
  57. beat schedule list.
  58. .. code-block:: python
  59. from celery import Celery
  60. from celery.schedules import crontab
  61. app = Celery()
  62. @app.on_after_configure.connect
  63. def setup_periodic_tasks(sender, **kwargs):
  64. # Calls test('hello') every 10 seconds.
  65. sender.add_periodic_task(10.0, test.s('hello'), name='add every 10')
  66. # Calls test('world') every 30 seconds
  67. sender.add_periodic_task(30.0, test.s('world'), expires=10)
  68. # Executes every Monday morning at 7:30 a.m.
  69. sender.add_periodic_task(
  70. crontab(hour=7, minute=30, day_of_week=1),
  71. test.s('Happy Mondays!'),
  72. )
  73. @app.task
  74. def test(arg):
  75. print(arg)
  76. Setting these up from within the :data:`~@on_after_configure` handler means
  77. that we'll not evaluate the app at module level when using ``test.s()``.
  78. The :meth:`~@add_periodic_task` function will add the entry to the
  79. :setting:`beat_schedule` setting behind the scenes, and the same setting
  80. can also be used to set up periodic tasks manually:
  81. Example: Run the `tasks.add` task every 30 seconds.
  82. .. code-block:: python
  83. app.conf.beat_schedule = {
  84. 'add-every-30-seconds': {
  85. 'task': 'tasks.add',
  86. 'schedule': 30.0,
  87. 'args': (16, 16)
  88. },
  89. }
  90. app.conf.timezone = 'UTC'
  91. .. note::
  92. If you're wondering where these settings should go then
  93. please see :ref:`celerytut-configuration`. You can either
  94. set these options on your app directly or you can keep
  95. a separate module for configuration.
  96. If you want to use a single item tuple for `args`, don't forget
  97. that the constructor is a comma, and not a pair of parentheses.
  98. Using a :class:`~datetime.timedelta` for the schedule means the task will
  99. be sent in 30 second intervals (the first task will be sent 30 seconds
  100. after `celery beat` starts, and then every 30 seconds
  101. after the last run).
  102. A Crontab like schedule also exists, see the section on `Crontab schedules`_.
  103. Like with :command:`cron`, the tasks may overlap if the first task doesn't complete
  104. before the next. If that's a concern you should use a locking
  105. strategy to ensure only one instance can run at a time (see for example
  106. :ref:`cookbook-task-serial`).
  107. .. _beat-entry-fields:
  108. Available Fields
  109. ----------------
  110. * `task`
  111. The name of the task to execute.
  112. * `schedule`
  113. The frequency of execution.
  114. This can be the number of seconds as an integer, a
  115. :class:`~datetime.timedelta`, or a :class:`~celery.schedules.crontab`.
  116. You can also define your own custom schedule types, by extending the
  117. interface of :class:`~celery.schedules.schedule`.
  118. * `args`
  119. Positional arguments (:class:`list` or :class:`tuple`).
  120. * `kwargs`
  121. Keyword arguments (:class:`dict`).
  122. * `options`
  123. Execution options (:class:`dict`).
  124. This can be any argument supported by
  125. :meth:`~celery.task.base.Task.apply_async` --
  126. `exchange`, `routing_key`, `expires`, and so on.
  127. * `relative`
  128. If `relative` is true :class:`~datetime.timedelta` schedules are scheduled
  129. "by the clock." This means the frequency is rounded to the nearest
  130. second, minute, hour or day depending on the period of the
  131. :class:`~datetime.timedelta`.
  132. By default `relative` is false, the frequency isn't rounded and will be
  133. relative to the time when :program:`celery beat` was started.
  134. .. _beat-crontab:
  135. Crontab schedules
  136. =================
  137. If you want more control over when the task is executed, for
  138. example, a particular time of day or day of the week, you can use
  139. the :class:`~celery.schedules.crontab` schedule type:
  140. .. code-block:: python
  141. from celery.schedules import crontab
  142. app.conf.beat_schedule = {
  143. # Executes every Monday morning at 7:30 a.m.
  144. 'add-every-monday-morning': {
  145. 'task': 'tasks.add',
  146. 'schedule': crontab(hour=7, minute=30, day_of_week=1),
  147. 'args': (16, 16),
  148. },
  149. }
  150. The syntax of these Crontab expressions are very flexible.
  151. Some examples:
  152. +-----------------------------------------+--------------------------------------------+
  153. | **Example** | **Meaning** |
  154. +-----------------------------------------+--------------------------------------------+
  155. | ``crontab()`` | Execute every minute. |
  156. +-----------------------------------------+--------------------------------------------+
  157. | ``crontab(minute=0, hour=0)`` | Execute daily at midnight. |
  158. +-----------------------------------------+--------------------------------------------+
  159. | ``crontab(minute=0, hour='*/3')`` | Execute every three hours: |
  160. | | midnight, 3am, 6am, 9am, |
  161. | | noon, 3pm, 6pm, 9pm. |
  162. +-----------------------------------------+--------------------------------------------+
  163. | ``crontab(minute=0,`` | Same as previous. |
  164. | ``hour='0,3,6,9,12,15,18,21')`` | |
  165. +-----------------------------------------+--------------------------------------------+
  166. | ``crontab(minute='*/15')`` | Execute every 15 minutes. |
  167. +-----------------------------------------+--------------------------------------------+
  168. | ``crontab(day_of_week='sunday')`` | Execute every minute (!) at Sundays. |
  169. +-----------------------------------------+--------------------------------------------+
  170. | ``crontab(minute='*',`` | Same as previous. |
  171. | ``hour='*',`` | |
  172. | ``day_of_week='sun')`` | |
  173. +-----------------------------------------+--------------------------------------------+
  174. | ``crontab(minute='*/10',`` | Execute every ten minutes, but only |
  175. | ``hour='3,17,22',`` | between 3-4 am, 5-6 pm, and 10-11 pm on |
  176. | ``day_of_week='thu,fri')`` | Thursdays or Fridays. |
  177. +-----------------------------------------+--------------------------------------------+
  178. | ``crontab(minute=0, hour='*/2,*/3')`` | Execute every even hour, and every hour |
  179. | | divisible by three. This means: |
  180. | | at every hour *except*: 1am, |
  181. | | 5am, 7am, 11am, 1pm, 5pm, 7pm, |
  182. | | 11pm |
  183. +-----------------------------------------+--------------------------------------------+
  184. | ``crontab(minute=0, hour='*/5')`` | Execute hour divisible by 5. This means |
  185. | | that it is triggered at 3pm, not 5pm |
  186. | | (since 3pm equals the 24-hour clock |
  187. | | value of "15", which is divisible by 5). |
  188. +-----------------------------------------+--------------------------------------------+
  189. | ``crontab(minute=0, hour='*/3,8-17')`` | Execute every hour divisible by 3, and |
  190. | | every hour during office hours (8am-5pm). |
  191. +-----------------------------------------+--------------------------------------------+
  192. | ``crontab(0, 0, day_of_month='2')`` | Execute on the second day of every month. |
  193. | | |
  194. +-----------------------------------------+--------------------------------------------+
  195. | ``crontab(0, 0,`` | Execute on every even numbered day. |
  196. | ``day_of_month='2-30/3')`` | |
  197. +-----------------------------------------+--------------------------------------------+
  198. | ``crontab(0, 0,`` | Execute on the first and third weeks of |
  199. | ``day_of_month='1-7,15-21')`` | the month. |
  200. +-----------------------------------------+--------------------------------------------+
  201. | ``crontab(0, 0, day_of_month='11',`` | Execute on the eleventh of May every year. |
  202. | ``month_of_year='5')`` | |
  203. +-----------------------------------------+--------------------------------------------+
  204. | ``crontab(0, 0,`` | Execute on the first month of every |
  205. | ``month_of_year='*/3')`` | quarter. |
  206. +-----------------------------------------+--------------------------------------------+
  207. See :class:`celery.schedules.crontab` for more documentation.
  208. .. _beat-solar:
  209. Solar schedules
  210. =================
  211. If you have a task that should be executed according to sunrise,
  212. sunset, dawn or dusk, you can use the
  213. :class:`~celery.schedules.solar` schedule type:
  214. .. code-block:: python
  215. from celery.schedules import solar
  216. app.conf.beat_schedule = {
  217. # Executes at sunset in Melbourne
  218. 'add-at-melbourne-sunset': {
  219. 'task': 'tasks.add',
  220. 'schedule': solar('sunset', -37.81753, 144.96715),
  221. 'args': (16, 16),
  222. },
  223. }
  224. The arguments are simply: ``solar(event, latitude, longitude)``
  225. Be sure to use the correct sign for latitude and longitude:
  226. +---------------+-------------------+----------------------+
  227. | **Sign** | **Argument** | **Meaning** |
  228. +---------------+-------------------+----------------------+
  229. | ``+`` | ``latitude`` | North |
  230. +---------------+-------------------+----------------------+
  231. | ``-`` | ``latitude`` | South |
  232. +---------------+-------------------+----------------------+
  233. | ``+`` | ``longitude`` | East |
  234. +---------------+-------------------+----------------------+
  235. | ``-`` | ``longitude`` | West |
  236. +---------------+-------------------+----------------------+
  237. Possible event types are:
  238. +-----------------------------------------+--------------------------------------------+
  239. | **Event** | **Meaning** |
  240. +-----------------------------------------+--------------------------------------------+
  241. | ``dawn_astronomical`` | Execute at the moment after which the sky |
  242. | | is no longer completely dark. This is when |
  243. | | the sun is 18 degrees below the horizon. |
  244. +-----------------------------------------+--------------------------------------------+
  245. | ``dawn_nautical`` | Execute when there's enough sunlight for |
  246. | | the horizon and some objects to be |
  247. | | distinguishable; formally, when the sun is |
  248. | | 12 degrees below the horizon. |
  249. +-----------------------------------------+--------------------------------------------+
  250. | ``dawn_civil`` | Execute when there's enough light for |
  251. | | objects to be distinguishable so that |
  252. | | outdoor activities can commence; |
  253. | | formally, when the Sun is 6 degrees below |
  254. | | the horizon. |
  255. +-----------------------------------------+--------------------------------------------+
  256. | ``sunrise`` | Execute when the upper edge of the sun |
  257. | | appears over the eastern horizon in the |
  258. | | morning. |
  259. +-----------------------------------------+--------------------------------------------+
  260. | ``solar_noon`` | Execute when the sun is highest above the |
  261. | | horizon on that day. |
  262. +-----------------------------------------+--------------------------------------------+
  263. | ``sunset`` | Execute when the trailing edge of the sun |
  264. | | disappears over the western horizon in the |
  265. | | evening. |
  266. +-----------------------------------------+--------------------------------------------+
  267. | ``dusk_civil`` | Execute at the end of civil twilight, when |
  268. | | objects are still distinguishable and some |
  269. | | stars and planets are visible. Formally, |
  270. | | when the sun is 6 degrees below the |
  271. | | horizon. |
  272. +-----------------------------------------+--------------------------------------------+
  273. | ``dusk_nautical`` | Execute when the sun is 12 degrees below |
  274. | | the horizon. Objects are no longer |
  275. | | distinguishable, and the horizon is no |
  276. | | longer visible to the naked eye. |
  277. +-----------------------------------------+--------------------------------------------+
  278. | ``dusk_astronomical`` | Execute at the moment after which the sky |
  279. | | becomes completely dark; formally, when |
  280. | | the sun is 18 degrees below the horizon. |
  281. +-----------------------------------------+--------------------------------------------+
  282. All solar events are calculated using UTC, and are therefore
  283. unaffected by your timezone setting.
  284. In polar regions, the sun may not rise or set every day. The scheduler
  285. is able to handle these cases (i.e., a ``sunrise`` event won't run on a day
  286. when the sun doesn't rise). The one exception is ``solar_noon``, which is
  287. formally defined as the moment the sun transits the celestial meridian,
  288. and will occur every day even if the sun is below the horizon.
  289. Twilight is defined as the period between dawn and sunrise; and between
  290. sunset and dusk. You can schedule an event according to "twilight"
  291. depending on your definition of twilight (civil, nautical, or astronomical),
  292. and whether you want the event to take place at the beginning or end
  293. of twilight, using the appropriate event from the list above.
  294. See :class:`celery.schedules.solar` for more documentation.
  295. .. _beat-starting:
  296. Starting the Scheduler
  297. ======================
  298. To start the :program:`celery beat` service:
  299. .. code-block:: console
  300. $ celery -A proj beat
  301. You can also embed `beat` inside the worker by enabling the
  302. workers :option:`-B <celery worker -B>` option, this is convenient if you'll
  303. never run more than one worker node, but it's not commonly used and for that
  304. reason isn't recommended for production use:
  305. .. code-block:: console
  306. $ celery -A proj worker -B
  307. Beat needs to store the last run times of the tasks in a local database
  308. file (named `celerybeat-schedule` by default), so it needs access to
  309. write in the current directory, or alternatively you can specify a custom
  310. location for this file:
  311. .. code-block:: console
  312. $ celery -A proj beat -s /home/celery/var/run/celerybeat-schedule
  313. .. note::
  314. To daemonize beat see :ref:`daemonizing`.
  315. .. _beat-custom-schedulers:
  316. Using custom scheduler classes
  317. ------------------------------
  318. Custom scheduler classes can be specified on the command-line (the
  319. :option:`--scheduler <celery beat --scheduler>` argument).
  320. The default scheduler is the :class:`celery.beat.PersistentScheduler`,
  321. that simply keeps track of the last run times in a local :mod:`shelve`
  322. database file.
  323. There's also the :pypi:`django-celery-beat` extension that stores the schedule
  324. in the Django database, and presents a convenient admin interface to manage
  325. periodic tasks at runtime.
  326. To install and use this extension:
  327. #. Use :command:`pip` to install the package:
  328. .. code-block:: console
  329. $ pip install django-celery-beat
  330. #. Add the ``django_celery_beat`` module to ``INSTALLED_APPS`` in your
  331. Django project' :file:`settings.py`::
  332. INSTALLED_APPS = (
  333. ...,
  334. 'django_celery_beat',
  335. )
  336. Note that there is no dash in the module name, only underscores.
  337. #. Apply Django database migrations so that the necessary tables are created:
  338. .. code-block:: console
  339. $ python manage.py migrate
  340. #. Start the :program:`celery beat` service using the ``django_celery_beat.schedulers:DatabaseScheduler`` scheduler:
  341. .. code-block:: console
  342. $ celery -A proj beat -l info --scheduler django_celery_beat.schedulers:DatabaseScheduler
  343. Note: You may also add this as an settings option directly.
  344. #. Visit the Django-Admin interface to set up some periodic tasks.