From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4681780FFC for ; Mon, 22 Nov 2021 09:46:24 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 431E31A54B for ; Mon, 22 Nov 2021 09:46:24 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 9C43E1A53D for ; Mon, 22 Nov 2021 09:46:23 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 719FB43DA3 for ; Mon, 22 Nov 2021 09:46:23 +0100 (CET) Message-ID: <59fb5326-3f84-6a83-acb2-f31a163ae8f2@proxmox.com> Date: Mon, 22 Nov 2021 09:46:22 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.2.1 Content-Language: en-US To: pve-devel@lists.proxmox.com, Dominik Csapak References: <20211118132831.839774-1-d.csapak@proxmox.com> <20211118132831.839774-2-d.csapak@proxmox.com> From: Fabian Ebner In-Reply-To: <20211118132831.839774-2-d.csapak@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 1.690 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -3.022 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH manager 2/3] pvescheduler: reworking child pid tracking X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2021 08:46:24 -0000 On 18.11.21 14:28, Dominik Csapak wrote: > previously, systemd timers were responsible for running replication jobs. > those timers would not restart if the previous one is still running. > > though trying again while it is running does no harm really, it spams > the log with errors about not being able to acquire the correct lock > > to fix this, we rework the handling of child processes such that we only > start one per loop if there is currently none running. for that, > introduce the types of forks we do and allow one child process per type > (for now, we have 'jobs' and 'replication' as types) > > Signed-off-by: Dominik Csapak > --- > PVE/Service/pvescheduler.pm | 42 ++++++++++++++++++++++++------------- > 1 file changed, 27 insertions(+), 15 deletions(-) > > diff --git a/PVE/Service/pvescheduler.pm b/PVE/Service/pvescheduler.pm > index d4f73702..466cc599 100755 > --- a/PVE/Service/pvescheduler.pm > +++ b/PVE/Service/pvescheduler.pm > @@ -17,12 +17,16 @@ my $cmdline = [$0, @ARGV]; > my %daemon_options = (stop_wait_time => 180, max_workers => 0); > my $daemon = __PACKAGE__->new('pvescheduler', $cmdline, %daemon_options); > > +my @types = qw(replication jobs); > + > my $finish_jobs = sub { > my ($self) = @_; > - foreach my $cpid (keys %{$self->{jobs}}) { > - my $waitpid = waitpid($cpid, WNOHANG); > - if (defined($waitpid) && ($waitpid == $cpid)) { > - delete ($self->{jobs}->{$cpid}); > + for my $type (@types) { > + if (my $cpid = $self->{jobs}->{$type}) { > + my $waitpid = waitpid($cpid, WNOHANG); > + if (defined($waitpid) && ($waitpid == $cpid)) { > + $self->{jobs}->{$type} = undef; > + } > } > } > }; > @@ -41,7 +45,11 @@ sub run { > }; > > my $fork = sub { > - my ($sub) = @_; > + my ($type, $sub) = @_; > + > + # don't fork again if the previous iteration still runs > + return if defined($self->{jobs}->{$type}); > + > my $child = fork(); > if (!defined($child)) { > die "fork failed: $!\n"; > @@ -56,16 +64,16 @@ sub run { > POSIX::_exit(0); > } > > - $jobs->{$child} = 1; > + $jobs->{$type} = $child; > }; > > my $run_jobs = sub { > > - $fork->(sub { > + $fork->('replication', sub { > PVE::API2::Replication::run_jobs(undef, sub {}, 0, 1); > }); > > - $fork->(sub { > + $fork->('jobs', sub { > PVE::Jobs::run_jobs(); > }); > }; > @@ -92,14 +100,16 @@ sub run { > } > } > > - # jobs have a lock timeout of 60s, wait a bit more for graceful termination > + # replication jobs have a lock timeout of 60s, wait a bit more for graceful termination > my $timeout = 0; > - while (keys %$jobs > 0 && $timeout < 75) { > - kill 'TERM', keys %$jobs; > - $timeout += sleep(5); > + for my $type (@types) { > + while (defined($jobs->{$type}) && $timeout < 75) { > + kill 'TERM', $jobs->{$type}; > + $timeout += sleep(5); > + } > + # ensure the rest gets stopped > + kill 'KILL', $jobs->{$type} if defined($jobs->{$type}); > } > - # ensure the rest gets stopped > - kill 'KILL', keys %$jobs if (keys %$jobs > 0); > } Nit: this changes the behavior a bit, as it can happen that the timeout is "used up" for one job type and all following ones only get the KILL singal. Because of the code below, each child still gets at least one TERM, so not a big deal. > > sub shutdown { > @@ -107,7 +117,9 @@ sub shutdown { > > syslog('info', 'got shutdown request, signal running jobs to stop'); > > - kill 'TERM', keys %{$self->{jobs}}; > + for my $type (@types) { > + kill 'TERM', $self->{jobs}->{$type} if $self->{jobs}->{$type}; > + } > $self->{shutdown_request} = 1; > } > >