From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1439E8099A for ; Thu, 18 Nov 2021 14:29:04 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id F22791C403 for ; Thu, 18 Nov 2021 14:28:33 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id CCC021C3F2 for ; Thu, 18 Nov 2021 14:28:32 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 64BA5448B1 for ; Thu, 18 Nov 2021 14:28:32 +0100 (CET) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Thu, 18 Nov 2021 14:28:30 +0100 Message-Id: <20211118132831.839774-2-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211118132831.839774-1-d.csapak@proxmox.com> References: <20211118132831.839774-1-d.csapak@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.212 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH manager 2/3] pvescheduler: reworking child pid tracking X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Nov 2021 13:29:04 -0000 previously, systemd timers were responsible for running replication jobs. those timers would not restart if the previous one is still running. though trying again while it is running does no harm really, it spams the log with errors about not being able to acquire the correct lock to fix this, we rework the handling of child processes such that we only start one per loop if there is currently none running. for that, introduce the types of forks we do and allow one child process per type (for now, we have 'jobs' and 'replication' as types) Signed-off-by: Dominik Csapak --- PVE/Service/pvescheduler.pm | 42 ++++++++++++++++++++++++------------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm index d4f73702..466cc599 100755 --- a/PVE/Service/pvescheduler.pm +++ b/PVE/Service/pvescheduler.pm @@ -17,12 +17,16 @@ my $cmdline = [$0, @ARGV]; my %daemon_options = (stop_wait_time => 180, max_workers => 0); my $daemon = __PACKAGE__->new('pvescheduler', $cmdline, %daemon_options); +my @types = qw(replication jobs); + my $finish_jobs = sub { my ($self) = @_; - foreach my $cpid (keys %{$self->{jobs}}) { - my $waitpid = waitpid($cpid, WNOHANG); - if (defined($waitpid) && ($waitpid == $cpid)) { - delete ($self->{jobs}->{$cpid}); + for my $type (@types) { + if (my $cpid = $self->{jobs}->{$type}) { + my $waitpid = waitpid($cpid, WNOHANG); + if (defined($waitpid) && ($waitpid == $cpid)) { + $self->{jobs}->{$type} = undef; + } } } }; @@ -41,7 +45,11 @@ sub run { }; my $fork = sub { - my ($sub) = @_; + my ($type, $sub) = @_; + + # don't fork again if the previous iteration still runs + return if defined($self->{jobs}->{$type}); + my $child = fork(); if (!defined($child)) { die "fork failed: $!\n"; @@ -56,16 +64,16 @@ sub run { POSIX::_exit(0); } - $jobs->{$child} = 1; + $jobs->{$type} = $child; }; my $run_jobs = sub { - $fork->(sub { + $fork->('replication', sub { PVE::API2::Replication::run_jobs(undef, sub {}, 0, 1); }); - $fork->(sub { + $fork->('jobs', sub { PVE::Jobs::run_jobs(); }); }; @@ -92,14 +100,16 @@ sub run { } } - # jobs have a lock timeout of 60s, wait a bit more for graceful termination + # replication jobs have a lock timeout of 60s, wait a bit more for graceful termination my $timeout = 0; - while (keys %$jobs > 0 && $timeout < 75) { - kill 'TERM', keys %$jobs; - $timeout += sleep(5); + for my $type (@types) { + while (defined($jobs->{$type}) && $timeout < 75) { + kill 'TERM', $jobs->{$type}; + $timeout += sleep(5); + } + # ensure the rest gets stopped + kill 'KILL', $jobs->{$type} if defined($jobs->{$type}); } - # ensure the rest gets stopped - kill 'KILL', keys %$jobs if (keys %$jobs > 0); } sub shutdown { @@ -107,7 +117,9 @@ sub shutdown { syslog('info', 'got shutdown request, signal running jobs to stop'); - kill 'TERM', keys %{$self->{jobs}}; + for my $type (@types) { + kill 'TERM', $self->{jobs}->{$type} if $self->{jobs}->{$type}; + } $self->{shutdown_request} = 1; } -- 2.30.2