From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D0E7762A8C for ; Tue, 24 Nov 2020 11:58:52 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 52086A869 for ; Tue, 24 Nov 2020 11:58:21 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id BE99EA7C2 for ; Tue, 24 Nov 2020 11:58:17 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 814AC40A8E for ; Tue, 24 Nov 2020 11:58:17 +0100 (CET) From: Alwin Antreich To: pve-devel@lists.proxmox.com Date: Tue, 24 Nov 2020 11:58:11 +0100 Message-Id: <20201124105811.1416723-9-a.antreich@proxmox.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20201124105811.1416723-1-a.antreich@proxmox.com> References: <20201124105811.1416723-1-a.antreich@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.018 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [tools.pm, ceph.com] Subject: [pve-devel] [PATCH manager v2 8/8] fix: ceph: always set pool size first X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2020 10:58:52 -0000 Since Ceph Nautilus 14.2.10 and Octopus 15.2.2 the min_size of a pool is calculated by the size (round(size / 2)). When size is applied after min_size to the pool, the manual specified min_size will be overwritten. With that a race condition can occur if the setting was set but is not active yet. Run an extra rados command to verify the current setting. Signed-off-by: Alwin Antreich --- PVE/Ceph/Tools.pm | 49 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm index d95e8676..9505f0bf 100644 --- a/PVE/Ceph/Tools.pm +++ b/PVE/Ceph/Tools.pm @@ -203,16 +203,51 @@ sub check_ceph_enabled { sub set_pool { my ($pool, $param) = @_; + # by default pool size always sets min_size + # https://tracker.ceph.com/issues/44862 + my $rados = PVE::RADOS->new(); + if ($param->{size}) { + my $value = $param->{size}; + eval { $rados->mon_command({ + prefix => "osd pool set", + pool => "$pool", + var => "size", + val => "$value", + format => 'plain', + }); + }; + if ($@) { + print "$@"; + } else { + my $result; + eval { $result = $rados->mon_command({ + prefix => "osd pool get", + pool => "$pool", + var => "size", + }); + }; + if (!$@ && $result->{size} eq $value) { + delete $param->{size}; + } + } + } + foreach my $setting (keys %$param) { my $value = $param->{$setting}; + next if $setting eq 'size'; my $command; + my $verify; if ($setting eq 'application') { $command = { prefix => "osd pool application enable", pool => "$pool", app => "$value", }; + $verify = { + prefix => "osd pool application get", + pool => "$pool", + }; } else { $command = { prefix => "osd pool set", @@ -221,14 +256,24 @@ sub set_pool { val => "$value", format => 'plain', }; + + $verify = { + prefix => "osd pool get", + pool => "$pool", + var => "$setting", + }; } - my $rados = PVE::RADOS->new(); + $rados = PVE::RADOS->new(); eval { $rados->mon_command($command); }; if ($@) { print "$@"; } else { - delete $param->{$setting}; + my $result; + eval { $result = $rados->mon_command($verify); }; + if (!$@ && $result->{$setting} eq $value) { + delete $param->{$setting}; + } } } -- 2.27.0