From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <f.ebner@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F3CC97EB51 for <pve-devel@lists.proxmox.com>; Thu, 11 Nov 2021 12:36:13 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E0D91A9B1 for <pve-devel@lists.proxmox.com>; Thu, 11 Nov 2021 12:35:43 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 05477A9A3 for <pve-devel@lists.proxmox.com>; Thu, 11 Nov 2021 12:35:42 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C7548434A6 for <pve-devel@lists.proxmox.com>; Thu, 11 Nov 2021 12:35:42 +0100 (CET) To: pve-devel@lists.proxmox.com, Dominik Csapak <d.csapak@proxmox.com> References: <20211108130758.160914-1-d.csapak@proxmox.com> From: Fabian Ebner <f.ebner@proxmox.com> Message-ID: <2dfd72a6-12b8-3549-bd28-25c8c49092b5@proxmox.com> Date: Thu, 11 Nov 2021 12:35:19 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211108130758.160914-1-d.csapak@proxmox.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 2.202 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.999 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH cluster/manager v2] add scheduling daemon for pvesr + vzdump (and more) X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 11 Nov 2021 11:36:14 -0000 Am 08.11.21 um 14:07 schrieb Dominik Csapak: > with this series, we implement a new daemon (pvescheduler) that takes > over from pvesrs' systemd timer (original patch from thomas[0]) and > extends it with a generic job handling mechanism > > then i convert the vzdump cron jobs to these jobs, the immediate > gain is that users can use calendarevent schedules instead of > dow + starttime > There's a problem for mixed-version clusters: If I edit/create a cluster-wide job (or job for another node) while running the new version, a node that doesn't have the pvescheduler yet, won't see that job anymore. > for now, this 'jobs.cfg' only handles vzdump jobs, but should be easily > extendable for other type of recurring jobs (like auth realm sync, etc.) > > also, i did not yet convert the replication jobs to this job system, > but that could probably be done without too much effort (though > i did not look too deeply into it) > > if some version of this gets applied, the further plan would be > to remove the vzdump.cron part completely with 8.0, but until then > we must at least list/parse that > > whats currently missing but not too hard to add is a calculated > 'next-run' column in the gui > > changes from v1: > * do not log replication into the syslog > * readjust the loop to start at the full minute every 1000 loops > * rework locking state locking/handling: > - i introduces a new 'starting' state that is set before we start > and we set it to started after the start. > we sadly cannot start the job while we hold the lock, since the open > file descriptor will be still open in the worker, and then we cannot > get the flock again. now it's more modeled after how we do qm/ct > long running locks (by writing 'starting' locked into the state) > - the stop check is now its own call at the beginning of the job handling > - handle created/removed jobs properly: > i did not think of state handling on other nodes in my previous > iteration. now on every loop, i sync the statefiles with the config > (create/remvoe) so that the file gets created/removed on all nodes > * incorporated fabians feedback for the api (thanks!) > > 0: https://lists.proxmox.com/pipermail/pve-devel/2018-April/031357.html > > pve-cluster: > > Dominik Csapak (1): > add 'jobs.cfg' to observed files > > data/PVE/Cluster.pm | 1 + > data/src/status.c | 1 + > 2 files changed, 2 insertions(+) > > pve-manager: > > Dominik Csapak (5): > add PVE/Jobs to handle VZDump jobs > pvescheduler: run jobs from jobs.cfg > api/backup: refactor string for all days > api/backup: handle new vzdump jobs > ui: dc/backup: show id+schedule instead of dow+starttime > > Thomas Lamprecht (1): > replace systemd timer with pvescheduler daemon > > PVE/API2/Backup.pm | 235 +++++++++++++++++++----- > PVE/API2/Cluster/BackupInfo.pm | 9 + > PVE/Jobs.pm | 286 +++++++++++++++++++++++++++++ > PVE/Jobs/Makefile | 16 ++ > PVE/Jobs/Plugin.pm | 61 ++++++ > PVE/Jobs/VZDump.pm | 54 ++++++ > PVE/Makefile | 3 +- > PVE/Service/Makefile | 2 +- > PVE/Service/pvescheduler.pm | 131 +++++++++++++ > bin/Makefile | 6 +- > bin/pvescheduler | 28 +++ > debian/postinst | 3 +- > services/Makefile | 3 +- > services/pvescheduler.service | 16 ++ > services/pvesr.service | 8 - > services/pvesr.timer | 12 -- > www/manager6/dc/Backup.js | 46 +++-- > www/manager6/dc/BackupJobDetail.js | 10 +- > 18 files changed, 823 insertions(+), 106 deletions(-) > create mode 100644 PVE/Jobs.pm > create mode 100644 PVE/Jobs/Makefile > create mode 100644 PVE/Jobs/Plugin.pm > create mode 100644 PVE/Jobs/VZDump.pm > create mode 100755 PVE/Service/pvescheduler.pm > create mode 100755 bin/pvescheduler > create mode 100644 services/pvescheduler.service > delete mode 100644 services/pvesr.service > delete mode 100644 services/pvesr.timer >