Re: [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more)

[Fabian Ebner <f.ebner@proxmox.com>]

Am 08.11.21 um 14:07 schrieb Dominik Csapak:
> with this series, we implement a new daemon (pvescheduler) that takes
> over from pvesrs' systemd timer (original patch from thomas[0]) and
> extends it with a generic job handling mechanism
> 
> then i convert the vzdump cron jobs to these jobs, the immediate
> gain is that users can use calendarevent schedules instead of
> dow + starttime
> 

There's a problem for mixed-version clusters:

If I edit/create a cluster-wide job (or job for another node) while 
running the new version, a node that doesn't have the pvescheduler yet, 
won't see that job anymore.

> for now, this 'jobs.cfg' only handles vzdump jobs, but should be easily
> extendable for other type of recurring jobs (like auth realm sync, etc.)
> 
> also, i did not yet convert the replication jobs to this job system,
> but that could probably be done without too much effort (though
> i did not look too deeply into it)
> 
> if some version of this gets applied, the further plan would be
> to remove the vzdump.cron part completely with 8.0, but until then
> we must at least list/parse that
> 
> whats currently missing but not too hard to add is a calculated
> 'next-run' column in the gui
> 
> changes from v1:
> * do not log replication into the syslog
> * readjust the loop to start at the full minute every 1000 loops
> * rework locking state locking/handling:
>    - i introduces a new 'starting' state that is set before we start
>      and we set it to started after the start.
>      we sadly cannot start the job while we hold the lock, since the open
>      file descriptor will be still open in the worker, and then we cannot
>      get the flock again. now it's more modeled after how we do qm/ct
>      long running locks (by writing 'starting' locked into the state)
>    - the stop check is now its own call at the beginning of the job handling
>    - handle created/removed jobs properly:
>      i did not think of state handling on other nodes in my previous
>      iteration. now on every loop, i sync the statefiles with the config
>      (create/remvoe) so that the file gets created/removed on all nodes
> * incorporated fabians feedback for the api (thanks!)
> 
> 0: https://lists.proxmox.com/pipermail/pve-devel/2018-April/031357.html
> 
> pve-cluster:
> 
> Dominik Csapak (1):
>    add 'jobs.cfg' to observed files
> 
>   data/PVE/Cluster.pm | 1 +
>   data/src/status.c   | 1 +
>   2 files changed, 2 insertions(+)
> 
> pve-manager:
> 
> Dominik Csapak (5):
>    add PVE/Jobs to handle VZDump jobs
>    pvescheduler: run jobs from jobs.cfg
>    api/backup: refactor string for all days
>    api/backup: handle new vzdump jobs
>    ui: dc/backup: show id+schedule instead of dow+starttime
> 
> Thomas Lamprecht (1):
>    replace systemd timer with pvescheduler daemon
> 
>   PVE/API2/Backup.pm                 | 235 +++++++++++++++++++-----
>   PVE/API2/Cluster/BackupInfo.pm     |   9 +
>   PVE/Jobs.pm                        | 286 +++++++++++++++++++++++++++++
>   PVE/Jobs/Makefile                  |  16 ++
>   PVE/Jobs/Plugin.pm                 |  61 ++++++
>   PVE/Jobs/VZDump.pm                 |  54 ++++++
>   PVE/Makefile                       |   3 +-
>   PVE/Service/Makefile               |   2 +-
>   PVE/Service/pvescheduler.pm        | 131 +++++++++++++
>   bin/Makefile                       |   6 +-
>   bin/pvescheduler                   |  28 +++
>   debian/postinst                    |   3 +-
>   services/Makefile                  |   3 +-
>   services/pvescheduler.service      |  16 ++
>   services/pvesr.service             |   8 -
>   services/pvesr.timer               |  12 --
>   www/manager6/dc/Backup.js          |  46 +++--
>   www/manager6/dc/BackupJobDetail.js |  10 +-
>   18 files changed, 823 insertions(+), 106 deletions(-)
>   create mode 100644 PVE/Jobs.pm
>   create mode 100644 PVE/Jobs/Makefile
>   create mode 100644 PVE/Jobs/Plugin.pm
>   create mode 100644 PVE/Jobs/VZDump.pm
>   create mode 100755 PVE/Service/pvescheduler.pm
>   create mode 100755 bin/pvescheduler
>   create mode 100644 services/pvescheduler.service
>   delete mode 100644 services/pvesr.service
>   delete mode 100644 services/pvesr.timer
>