public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more)
@ 2021-11-08 13:07 Dominik Csapak
  2021-11-08 13:07 ` [pve-devel] [PATCH cluster v2 1/1] add 'jobs.cfg' to observed files Dominik Csapak
                   ` (10 more replies)
  0 siblings, 11 replies; 16+ messages in thread
From: Dominik Csapak @ 2021-11-08 13:07 UTC (permalink / raw)
  To: pve-devel

with this series, we implement a new daemon (pvescheduler) that takes
over from pvesrs' systemd timer (original patch from thomas[0]) and
extends it with a generic job handling mechanism

then i convert the vzdump cron jobs to these jobs, the immediate
gain is that users can use calendarevent schedules instead of
dow + starttime

for now, this 'jobs.cfg' only handles vzdump jobs, but should be easily
extendable for other type of recurring jobs (like auth realm sync, etc.)

also, i did not yet convert the replication jobs to this job system,
but that could probably be done without too much effort (though
i did not look too deeply into it)

if some version of this gets applied, the further plan would be
to remove the vzdump.cron part completely with 8.0, but until then
we must at least list/parse that

whats currently missing but not too hard to add is a calculated
'next-run' column in the gui

changes from v1:
* do not log replication into the syslog
* readjust the loop to start at the full minute every 1000 loops
* rework locking state locking/handling:
  - i introduces a new 'starting' state that is set before we start
    and we set it to started after the start.
    we sadly cannot start the job while we hold the lock, since the open
    file descriptor will be still open in the worker, and then we cannot
    get the flock again. now it's more modeled after how we do qm/ct
    long running locks (by writing 'starting' locked into the state)
  - the stop check is now its own call at the beginning of the job handling
  - handle created/removed jobs properly:
    i did not think of state handling on other nodes in my previous
    iteration. now on every loop, i sync the statefiles with the config
    (create/remvoe) so that the file gets created/removed on all nodes
* incorporated fabians feedback for the api (thanks!)

0: https://lists.proxmox.com/pipermail/pve-devel/2018-April/031357.html

pve-cluster:

Dominik Csapak (1):
  add 'jobs.cfg' to observed files

 data/PVE/Cluster.pm | 1 +
 data/src/status.c   | 1 +
 2 files changed, 2 insertions(+)

pve-manager:

Dominik Csapak (5):
  add PVE/Jobs to handle VZDump jobs
  pvescheduler: run jobs from jobs.cfg
  api/backup: refactor string for all days
  api/backup: handle new vzdump jobs
  ui: dc/backup: show id+schedule instead of dow+starttime

Thomas Lamprecht (1):
  replace systemd timer with pvescheduler daemon

 PVE/API2/Backup.pm                 | 235 +++++++++++++++++++-----
 PVE/API2/Cluster/BackupInfo.pm     |   9 +
 PVE/Jobs.pm                        | 286 +++++++++++++++++++++++++++++
 PVE/Jobs/Makefile                  |  16 ++
 PVE/Jobs/Plugin.pm                 |  61 ++++++
 PVE/Jobs/VZDump.pm                 |  54 ++++++
 PVE/Makefile                       |   3 +-
 PVE/Service/Makefile               |   2 +-
 PVE/Service/pvescheduler.pm        | 131 +++++++++++++
 bin/Makefile                       |   6 +-
 bin/pvescheduler                   |  28 +++
 debian/postinst                    |   3 +-
 services/Makefile                  |   3 +-
 services/pvescheduler.service      |  16 ++
 services/pvesr.service             |   8 -
 services/pvesr.timer               |  12 --
 www/manager6/dc/Backup.js          |  46 +++--
 www/manager6/dc/BackupJobDetail.js |  10 +-
 18 files changed, 823 insertions(+), 106 deletions(-)
 create mode 100644 PVE/Jobs.pm
 create mode 100644 PVE/Jobs/Makefile
 create mode 100644 PVE/Jobs/Plugin.pm
 create mode 100644 PVE/Jobs/VZDump.pm
 create mode 100755 PVE/Service/pvescheduler.pm
 create mode 100755 bin/pvescheduler
 create mode 100644 services/pvescheduler.service
 delete mode 100644 services/pvesr.service
 delete mode 100644 services/pvesr.timer

-- 
2.30.2





^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2021-11-11 11:47 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-08 13:07 [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more) Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH cluster v2 1/1] add 'jobs.cfg' to observed files Dominik Csapak
2021-11-09 17:18   ` [pve-devel] applied: " Thomas Lamprecht
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 1/6] replace systemd timer with pvescheduler daemon Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 2/6] add PVE/Jobs to handle VZDump jobs Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 3/6] pvescheduler: run jobs from jobs.cfg Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 4/6] api/backup: refactor string for all days Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 5/6] api/backup: handle new vzdump jobs Dominik Csapak
2021-11-08 13:07 ` [pve-devel] [PATCH manager v2 6/6] ui: dc/backup: show id+schedule instead of dow+starttime Dominik Csapak
2021-11-09 14:17 ` [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more) Dylan Whyte
2021-11-10  9:38   ` Dominik Csapak
2021-11-10 11:05     ` Thomas Lamprecht
2021-11-09 16:55 ` Aaron Lauterer
2021-11-10 20:42 ` [pve-devel] applied-series: " Thomas Lamprecht
2021-11-11 11:35 ` [pve-devel] " Fabian Ebner
2021-11-11 11:46   ` Dominik Csapak

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal