From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 46DB17D0FB for ; Mon, 8 Nov 2021 14:08:33 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3C76E251A7 for ; Mon, 8 Nov 2021 14:08:03 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id B3D1A2516E for ; Mon, 8 Nov 2021 14:07:59 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8A2EC422F3 for ; Mon, 8 Nov 2021 14:07:59 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Mon, 8 Nov 2021 14:07:51 +0100 Message-Id: <20211108130758.160914-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.236 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more) X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Nov 2021 13:08:33 -0000 with this series, we implement a new daemon (pvescheduler) that takes over from pvesrs' systemd timer (original patch from thomas[0]) and extends it with a generic job handling mechanism then i convert the vzdump cron jobs to these jobs, the immediate gain is that users can use calendarevent schedules instead of dow + starttime for now, this 'jobs.cfg' only handles vzdump jobs, but should be easily extendable for other type of recurring jobs (like auth realm sync, etc.) also, i did not yet convert the replication jobs to this job system, but that could probably be done without too much effort (though i did not look too deeply into it) if some version of this gets applied, the further plan would be to remove the vzdump.cron part completely with 8.0, but until then we must at least list/parse that whats currently missing but not too hard to add is a calculated 'next-run' column in the gui changes from v1: * do not log replication into the syslog * readjust the loop to start at the full minute every 1000 loops * rework locking state locking/handling: - i introduces a new 'starting' state that is set before we start and we set it to started after the start. we sadly cannot start the job while we hold the lock, since the open file descriptor will be still open in the worker, and then we cannot get the flock again. now it's more modeled after how we do qm/ct long running locks (by writing 'starting' locked into the state) - the stop check is now its own call at the beginning of the job handling - handle created/removed jobs properly: i did not think of state handling on other nodes in my previous iteration. now on every loop, i sync the statefiles with the config (create/remvoe) so that the file gets created/removed on all nodes * incorporated fabians feedback for the api (thanks!) 0: https://lists.proxmox.com/pipermail/pve-devel/2018-April/031357.html pve-cluster: Dominik Csapak (1): add 'jobs.cfg' to observed files data/PVE/Cluster.pm | 1 + data/src/status.c | 1 + 2 files changed, 2 insertions(+) pve-manager: Dominik Csapak (5): add PVE/Jobs to handle VZDump jobs pvescheduler: run jobs from jobs.cfg api/backup: refactor string for all days api/backup: handle new vzdump jobs ui: dc/backup: show id+schedule instead of dow+starttime Thomas Lamprecht (1): replace systemd timer with pvescheduler daemon PVE/API2/Backup.pm | 235 +++++++++++++++++++----- PVE/API2/Cluster/BackupInfo.pm | 9 + PVE/Jobs.pm | 286 +++++++++++++++++++++++++++++ PVE/Jobs/Makefile | 16 ++ PVE/Jobs/Plugin.pm | 61 ++++++ PVE/Jobs/VZDump.pm | 54 ++++++ PVE/Makefile | 3 +- PVE/Service/Makefile | 2 +- PVE/Service/pvescheduler.pm | 131 +++++++++++++ bin/Makefile | 6 +- bin/pvescheduler | 28 +++ debian/postinst | 3 +- services/Makefile | 3 +- services/pvescheduler.service | 16 ++ services/pvesr.service | 8 - services/pvesr.timer | 12 -- www/manager6/dc/Backup.js | 46 +++-- www/manager6/dc/BackupJobDetail.js | 10 +- 18 files changed, 823 insertions(+), 106 deletions(-) create mode 100644 PVE/Jobs.pm create mode 100644 PVE/Jobs/Makefile create mode 100644 PVE/Jobs/Plugin.pm create mode 100644 PVE/Jobs/VZDump.pm create mode 100755 PVE/Service/pvescheduler.pm create mode 100755 bin/pvescheduler create mode 100644 services/pvescheduler.service delete mode 100644 services/pvesr.service delete mode 100644 services/pvesr.timer -- 2.30.2