From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v8 storage 17/43] diskmanage: read the udev data of all disks with one call
Date: Thu, 19 Nov 2026 20:46:22 +0100 [thread overview]
Message-ID: <20261119194655.3163258-18-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261119194655.3163258-1-t.lamprecht@proxmox.com>
The disk list asks udevadm about each disk on its own, which takes a
few milliseconds per disk. That adds up on a node with many disks, for
example with a storage plugin that uses one LUN per guest disk. The
SAN LUN inventory that follows lists every SAN disk outside of a
multipath map. On a lab node with 1000 NVMe/TCP namespaces, it took
6.6 seconds.
Read the udev database of all block devices with one call instead. Its
entries have the format of the query for a single device, also for a
device that udev has not processed yet. On that node, the inventory
then took 0.9 seconds.
But reading the database gets slower with every block device of a
node, also with those that are not listed. On a node with 1000
multipath maps over 2000 iSCSI paths, reading it took 0.47 seconds,
while asking about the 14 disks of its list took 0.1 seconds. So the
disk list asks about single disks until that took about as long as
reading the database would, counting one query as 40 block devices.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
src/PVE/Diskmanage.pm | 63 +++++++++++++++++++++++++++++--------
src/test/disklist_test.pm | 66 ++++++++++++++++++++++++++++++++++++++-
2 files changed, 115 insertions(+), 14 deletions(-)
diff --git a/src/PVE/Diskmanage.pm b/src/PVE/Diskmanage.pm
index c4b4ce68..9c16e6a6 100644
--- a/src/PVE/Diskmanage.pm
+++ b/src/PVE/Diskmanage.pm
@@ -370,21 +370,50 @@ sub get_ceph_volume_infos {
return $result;
}
+# Returns the udev information of all block devices by their kernel name, in the format that
+# 'udevadm info --query all' prints for a single device. One call for all devices takes a fraction
+# of the time that a call per device takes on a node with many disks.
+sub get_udev_db {
+ my $db = {};
+ my $entry = '';
+ my $add = sub {
+ my ($name) = $entry =~ m|^P: .*/([^/\n]+)$|m;
+ $db->{$name} = $entry if defined($name);
+ $entry = '';
+ };
+ eval {
+ run_command(
+ ['udevadm', 'info', '--export-db', '--subsystem-match=block'],
+ outfunc => sub {
+ my ($line) = @_;
+ return $add->() if $line eq '';
+ $entry .= "$line\n";
+ },
+ );
+ };
+ # every entry ends with an empty line, so one that a failed command cut off is left out
+ warn $@ if $@;
+ return $db;
+}
+
sub get_udev_info {
- my ($dev) = @_;
+ my ($dev, $udev_db) = @_;
- my $info = "";
+ # a device that showed up after the database got read is not in there
+ my $info = $udev_db ? $udev_db->{ basename($dev) } // '' : '';
my $data = {};
- eval {
- run_command(
- ['udevadm', 'info', '-p', $dev, '--query', 'all'],
- outfunc => sub {
- my ($line) = @_;
- $info .= "$line\n";
- },
- );
- };
- warn $@ if $@;
+ if (!$info) {
+ eval {
+ run_command(
+ ['udevadm', 'info', '-p', $dev, '--query', 'all'],
+ outfunc => sub {
+ my ($line) = @_;
+ $info .= "$line\n";
+ },
+ );
+ };
+ warn $@ if $@;
+ }
return if !$info;
return if $info !~ m/^E: DEVTYPE=(disk|partition)$/m;
@@ -594,6 +623,13 @@ sub get_disks {
# through the same match, so perl would compile that regex again for every disk.
$wanted = { map { $_ => 1 } @$disks };
}
+ # Reading the database of all block devices takes about as long as single queries for a 40th
+ # of the devices that lsblk lists, and a node with multipath maps or many volumes has far more
+ # of those than disks to list. So ask about single disks until that took as long as reading the
+ # database would, and about the first few in any case.
+ my ($udev_db, $udev_queries) = (undef, 0);
+ my $udev_single = scalar(keys %$lsblk_info) / 40;
+ $udev_single = 4 if $udev_single < 4;
dir_glob_foreach(
'/sys/block',
@@ -615,7 +651,8 @@ sub get_disks {
return if !$san_inventory && is_iscsi_device($dev);
- my $data = get_udev_info("/sys/block/$dev") // return;
+ $udev_db //= get_udev_db() if ++$udev_queries > $udev_single;
+ my $data = get_udev_info("/sys/block/$dev", $udev_db) // return;
my $devpath = $data->{devpath};
my $sysdir = "/sys/block/$dev";
diff --git a/src/test/disklist_test.pm b/src/test/disklist_test.pm
index 1ad1cf66..305bab8e 100644
--- a/src/test/disklist_test.pm
+++ b/src/test/disklist_test.pm
@@ -19,6 +19,7 @@ my $testcount = 0; # testcount for TAP::Harness
my $diskmanage_module; # mockmodule for PVE::Diskmanage
my $multipath_module; # mockmodule for PVE::Multipath
my $print = 0;
+my $udev_single_queries = 0; # udevadm calls for one device
my $multipath_aliases = {
'/dev/mapper/mpatha' => '/dev/dm-0',
@@ -33,7 +34,18 @@ sub mocked_run_command {
my $outputlines = [];
if (my $ref = ref($cmd)) {
- if ($cmd->[0] =~ m/udevadm/i) {
+ if ($cmd->[0] =~ m/udevadm/i && grep { $_ eq '--export-db' } @$cmd) {
+ # simulate the database of all devices, an empty line ends an entry
+ for my $file (sort glob("disk_tests/$testcasedir/*_udevadm")) {
+ my ($name) = $file =~ m|/([^/]+)$|;
+ my @lines = split(/\n/, read_test_file($name));
+ # the database is keyed by the device path, which older fixtures lack
+ unshift @lines, 'P: /devices/virtual/block/' . ($name =~ s/_udevadm$//r)
+ if !grep { m/^P: / } @lines;
+ push @$outputlines, @lines, '';
+ }
+ } elsif ($cmd->[0] =~ m/udevadm/i) {
+ $udev_single_queries++;
# simulate udevadm output
my $dev = $cmd->[3];
$dev =~ s|/sys/block/||;
@@ -279,6 +291,58 @@ sub test_disk_list {
is_deeply($status, $expected_status, 'nvme path status should be the same');
}
+ if ($testcasedir eq 'multipath') {
+ my $db = PVE::Diskmanage::get_udev_db();
+ $testcount++;
+ is_deeply(
+ [sort keys %$db],
+ [qw(nvme8n1 nvme9n1 sda sdb sdc sdd)],
+ 'udev database has one entry per device',
+ );
+ $udev_single_queries = 0;
+ my $info = PVE::Diskmanage::get_udev_info('/sys/block/sdb', $db);
+ $testcount++;
+ is($info->{serial}, 'SSERb', 'disk found in the udev database');
+ $testcount++;
+ is($udev_single_queries, 0, 'without a query of its own');
+ PVE::Diskmanage::get_udev_info('/sys/block/sdb', {});
+ $testcount++;
+ is(
+ $udev_single_queries,
+ 1,
+ 'a disk that is not in the database is asked for on its own',
+ );
+
+ $udev_single_queries = 0;
+ my $listed = PVE::Diskmanage::get_disks();
+ $testcount++;
+ is(scalar(keys %$listed), 6, 'all disks listed');
+ $testcount++;
+ is(
+ $udev_single_queries,
+ 4,
+ 'the database replaces the queries after the first disks',
+ );
+
+ # six disks: 240 block devices are just enough to ask about each of them on its own
+ for my $count (240, 239) {
+ $diskmanage_module->mock(
+ get_lsblk_info => sub {
+ return { map { ("/dev/dm-$_" => {}) } 1 .. $count };
+ },
+ );
+ $udev_single_queries = 0;
+ PVE::Diskmanage::get_disks();
+ $diskmanage_module->unmock('get_lsblk_info');
+ $testcount++;
+ is(
+ $udev_single_queries,
+ $count == 240 ? 6 : 5,
+ "single queries for six disks among $count block devices",
+ );
+ }
+ }
+
done_testing($testcount);
};
}
next prev parent reply other threads:[~2026-11-19 19:50 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 19:46 [PATCH v8 cluster,storage,docs,manager 00/43] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 cluster 01/43] cfs: observe the multipath configuration files Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 02/43] diskmanage: collect disk transport type from lsblk Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 03/43] diskmanage: qualify NVMe over Fabrics transport Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 04/43] diskmanage: optionally include iSCSI session devices in disk enumeration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 05/43] diskmanage: collect native NVMe multipath path state Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 06/43] iscsi: factor out session device map from device list Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 07/43] multipath: add a library around multipathd Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 08/43] diskmanage: add helper to list multipath devices Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 09/43] multipath: add a cluster-wide configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 10/43] iscsi: add an option to manage storage LUNs with multipath Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 11/43] lvm: activate a missing base volume before using the volume group Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 12/43] multipath: apply the configuration on every node Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 13/43] multipath: publish per-node status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 14/43] api: add endpoints for the cluster-wide multipath configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 15/43] api: multipath: add the cluster-wide health status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 16/43] diskmanage: look up the requested disks by name Thomas Lamprecht
2026-11-19 19:46 ` Thomas Lamprecht [this message]
2026-11-19 19:46 ` [PATCH v8 storage 18/43] api: scan: add a SAN LUN inventory Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 19/43] pvesm: expose the " Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 20/43] api: add node-level iSCSI status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 21/43] api: scan: name the portal of an iSCSI path Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 22/43] disks: lvm: drop unused imports Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 23/43] disks: lvm: allow volume groups on SAN LUNs and multipath maps Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 24/43] disks: lvm: refuse to destroy the volume group of a shared storage Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 25/43] disks: allow wiping a SAN LUN that no storage uses Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 storage 26/43] api: scan: report the process that has a SAN LUN open Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 docs 27/43] storage: document the cluster-wide multipath management Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 docs 28/43] storage: multipath: describe the use shown for an open LUN Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 29/43] pvestatd: apply the multipath configuration and report the node status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 30/43] report: add the managed multipath configuration Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 31/43] api: cluster: add the multipath endpoints Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 32/43] api: nodes: expose the iSCSI status Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 33/43] ui: dc: add a SAN view Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 34/43] ui: storage: iscsi: add the multipath option Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 35/43] ui: lvm: enable shared when a base storage is selected Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 36/43] ui: node: add SAN LUNs panel Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 37/43] ui: node: san: wipe a LUN that no storage uses Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 38/43] ui: node: san: show iSCSI targets and sessions Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 39/43] ui: dc: san: add a guided setup for shared LVM on iSCSI Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 40/43] ui: dc: san: add a guided setup for shared LVM on Fibre Channel Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 41/43] ui: storage: point the iSCSI and LVM add dialogs to the SAN view Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 42/43] ui: san: show the guest or process that has a LUN open Thomas Lamprecht
2026-11-19 19:46 ` [PATCH v8 manager 43/43] ui: node: san: add a column for the target or subsystem of a LUN Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119194655.3163258-18-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox