From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BAD158813 for ; Tue, 22 Aug 2023 11:20:52 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9B444A1CF for ; Tue, 22 Aug 2023 11:20:22 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 22 Aug 2023 11:20:22 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DA6ED4328C for ; Tue, 22 Aug 2023 11:20:21 +0200 (CEST) From: Filip Schauer To: pve-devel@lists.proxmox.com Date: Tue, 22 Aug 2023 11:20:15 +0200 Message-Id: <20230822092015.23709-1-f.schauer@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.000 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [jobs.pm, pvescheduler.pm] Subject: [pve-devel] [PATCH manager] fix #3731: pvescheduler: Improve fork behaviour X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Aug 2023 09:20:52 -0000 Instead of forking every minute, check whether a job has to be run and only fork if there is a pending job. Signed-off-by: Filip Schauer --- Backup jobs are registered in the jobs.cfg file, while replication jobs are registered in the replication.cfg file. Integrating replication into PVE::Jobs would move replication jobs into jobs.cfg, breaking backwards compatibility. PVE/Jobs.pm | 21 ++++++++++++++++++++- PVE/Service/pvescheduler.pm | 11 ++++------- 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/PVE/Jobs.pm b/PVE/Jobs.pm index bd323332..5643bd15 100644 --- a/PVE/Jobs.pm +++ b/PVE/Jobs.pm @@ -258,13 +258,14 @@ sub get_last_runtime { return $state->{time} // 0; } -sub run_jobs { +sub get_pending { my ($first_run) = @_; synchronize_job_states_with_config(); my $jobs_cfg = cfs_read_file('jobs.cfg'); my $nodename = PVE::INotify::nodename(); + my @pending_jobs = (); foreach my $id (sort keys %{$jobs_cfg->{ids}}) { my $cfg = $jobs_cfg->{ids}->{$id}; @@ -292,6 +293,24 @@ sub run_jobs { next if !defined($next_sync) || time() < $next_sync; # not yet its (next) turn + push @pending_jobs, $id; + } + + return @pending_jobs; +} + +sub run_jobs { + my (@scheduled_jobs) = @_; + + synchronize_job_states_with_config(); + + my $jobs_cfg = cfs_read_file('jobs.cfg'); + + foreach my $id (@scheduled_jobs) { + my $cfg = $jobs_cfg->{ids}->{$id} or next; + my $type = $cfg->{type}; + my $schedule = delete $cfg->{schedule}; + my $plugin = PVE::Job::Registry->lookup($type); if (starting_job($id, $type)) { PVE::Cluster::cfs_update(); diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm index 40be5977..7b6b3e99 100755 --- a/PVE/Service/pvescheduler.pm +++ b/PVE/Service/pvescheduler.pm @@ -100,19 +100,16 @@ sub run { my $first_run = 1; my $run_jobs = sub { - # TODO: actually integrate replication in PVE::Jobs and do not always fork here, we could - # do the state lookup and check if there's new work scheduled before doing so, e.g., by - # extending the PVE::Jobs interfacae e.g.; - # my $scheduled_jobs = PVE::Jobs::get_pending() or return; - # forked { PVE::Jobs::run_jobs($scheduled_jobs) } + # TODO: actually integrate replication in PVE::Jobs and do not always fork here $fork->('replication', sub { PVE::API2::Replication::run_jobs(undef, sub {}, 0, 1); }); + my @pending_jobs = PVE::Jobs::get_pending($first_run); $fork->('jobs', sub { - PVE::Jobs::run_jobs($first_run); - }); + PVE::Jobs::run_jobs(@pending_jobs); + }) if (@pending_jobs); $first_run = 0; }; -- 2.39.2