From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <f.ebner@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 574676D766
 for <pve-devel@lists.proxmox.com>; Tue, 28 Sep 2021 13:40:18 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 18003DFB8
 for <pve-devel@lists.proxmox.com>; Tue, 28 Sep 2021 13:40:17 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
 [94.136.29.106])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id 13214DCE6
 for <pve-devel@lists.proxmox.com>; Tue, 28 Sep 2021 13:40:07 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
 by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DE47D423AF
 for <pve-devel@lists.proxmox.com>; Tue, 28 Sep 2021 13:40:06 +0200 (CEST)
From: Fabian Ebner <f.ebner@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Tue, 28 Sep 2021 13:39:55 +0200
Message-Id: <20210928114001.164081-16-f.ebner@proxmox.com>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20210928114001.164081-1-f.ebner@proxmox.com>
References: <20210928114001.164081-1-f.ebner@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results:  0
 AWL 0.302 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
 URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
 http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
 information. [osd.pm]
Subject: [pve-devel] [PATCH manager 3/9] api: ceph: create osd: work around
 udev bug
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 28 Sep 2021 11:40:18 -0000

There is a udev bug [0] which can ultimately lead to the udev database
for certain devices not being actively updated. The Diskmanage package
relies upon lsblk for certain info, and lsblk queries the udev
database. Ensure the information is updated by manually calling
'udevadm trigger' for the changed devices.

Without the fix, and a bit of bad luck, a cleaned up disk could still
show up as an 'LVM2_member' for example.

[0]: https://github.com/systemd/systemd/issues/18525

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 PVE/API2/Ceph/OSD.pm | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm
index 97393912..fbbc2139 100644
--- a/PVE/API2/Ceph/OSD.pm
+++ b/PVE/API2/Ceph/OSD.pm
@@ -371,6 +371,9 @@ __PACKAGE__->register_method ({
 	    file_set_contents($ceph_bootstrap_osd_keyring, $bindata);
 	};
 
+	# See FIXME below
+	my @udev_trigger_devs = ();
+
 	my $create_part_or_lv = sub {
 	    my ($dev, $size, $type) = @_;
 
@@ -393,6 +396,8 @@ __PACKAGE__->register_method ({
 		PVE::Storage::LVMPlugin::lvm_create_volume_group($dev->{devpath}, $vg);
 		PVE::Storage::LVMPlugin::lvcreate($vg, $lv, "${size}k");
 
+		push @udev_trigger_devs, $dev->{devpath};
+
 		return "$vg/$lv";
 
 	    } elsif ($dev->{used} eq 'LVM') {
@@ -423,7 +428,9 @@ __PACKAGE__->register_method ({
 	    } elsif ($dev->{used} eq 'partitions' && $dev->{gpt}) {
 		# create new partition at the end
 
-		return PVE::Diskmanage::append_partition($dev->{devpath}, $size * 1024);
+		my $part = PVE::Diskmanage::append_partition($dev->{devpath}, $size * 1024);
+		push @udev_trigger_devs, $part;
+		return $part;
 	    }
 
 	    die "cannot use '$dev->{devpath}' for '$type'\n";
@@ -445,6 +452,8 @@ __PACKAGE__->register_method ({
 		my $devpath = $disklist->{$devname}->{devpath};
 		print "create OSD on $devpath (bluestore)\n";
 
+		push @udev_trigger_devs, $devpath;
+
 		my $osd_size = $disklist->{$devname}->{size};
 		my $size_map = {
 		    db => int($osd_size / 10), # 10% of OSD
@@ -476,6 +485,12 @@ __PACKAGE__->register_method ({
 		PVE::Diskmanage::wipe_blockdev($devpath);
 
 		run_command($cmd);
+
+		# FIXME: Remove once we depend on systemd >= v249.
+		# Work around udev bug https://github.com/systemd/systemd/issues/18525 to ensure the
+		# udev database is updated.
+		eval { run_command(['udevadm', 'trigger', @udev_trigger_devs]); };
+		warn $@ if $@;
 	    });
 	};
 
@@ -583,6 +598,9 @@ __PACKAGE__->register_method ({
 	    # try to unmount from standard mount point
 	    my $mountpoint = "/var/lib/ceph/osd/ceph-$osdid";
 
+	    # See FIXME below
+	    my $udev_trigger_devs = {};
+
 	    my $remove_partition = sub {
 		my ($part) = @_;
 
@@ -590,6 +608,8 @@ __PACKAGE__->register_method ({
 		my $partnum = PVE::Diskmanage::get_partnum($part);
 		my $devpath = PVE::Diskmanage::get_blockdev($part);
 
+		$udev_trigger_devs->{$devpath} = 1;
+
 		PVE::Diskmanage::wipe_blockdev($part);
 		print "remove partition $part (disk '${devpath}', partnum $partnum)\n";
 		eval { run_command(['/sbin/sgdisk', '-d', $partnum, "${devpath}"]); };
@@ -611,6 +631,8 @@ __PACKAGE__->register_method ({
 
 			    eval { run_command(['/sbin/pvremove', $dev], errfunc => sub {}) };
 			    warn $@ if $@;
+
+			    $udev_trigger_devs->{$dev} = 1;
 			}
 		    }
 		}
@@ -648,6 +670,14 @@ __PACKAGE__->register_method ({
 		    }
 		}
 	    }
+
+	    # FIXME: Remove once we depend on systemd >= v249.
+	    # Work around udev bug https://github.com/systemd/systemd/issues/18525 to ensure the
+	    # udev database is updated.
+	    if ($cleanup) {
+		eval { run_command(['udevadm', 'trigger', keys $udev_trigger_devs->%*]); };
+		warn $@ if $@;
+	    }
 	};
 
 	return $rpcenv->fork_worker('cephdestroyosd', $osdsection,  $authuser, $worker);
-- 
2.30.2