From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 94D771FF0B0 for ; Tue, 27 Oct 2026 10:32:09 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5B3F9216A7; Tue, 27 Oct 2026 10:32:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=straightec.de; s=dkim; t=1792964312; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=jFURGNnqR1tVUmjuNryqzEERlEXHbl+qkBnVio9Bj+I=; b=lcPzsLfeeqYnmMjwuGqmYpYfmDKxnDSIqIZpCEO3khnibTtSrm5fFenQunBXU+zY1nIb5O Z7O0NjtWlRD5SkV5Nho78xBc3OSZtxOvvY/h5E+xb18Luyd51pQ9hQ8bS1tn5geZwOGq4e J1f4jCXkPz9s4Q6WW6A/fpLzr3sLwscgB3j2+zbr41aDnV4QHR46i/6u9DY4NfRPsyueH6 GOs290u26MImarfx+yjb1C+PWzqr0dzOT8Wp0Kj/IKttqoG9fBLxYumxESa3ZUVqqM6hPp nXXWaXON3wdnSNRDPThnhdjRpUpUw/JwK6Tok9hxra98Cvm3hT0hNYFAA2cE6Q== From: Carsten Haerle To: pve-devel@lists.proxmox.com Subject: [PATCH storage] fix #8230: zfspool: serialize import and mount in activate_storage Date: Sun, 25 Oct 2026 22:38:30 +0100 Message-ID: <20261025213830.2432528-1-carsten.haerle@straightec.de> X-Mailer: git-send-email 2.53.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: carsten.haerle@straightec.de X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: YV7BZ4WVD6YYMT5TRVDNHAETVKM64YMI X-Message-ID-Hash: YV7BZ4WVD6YYMT5TRVDNHAETVKM64YMI X-Mailman-Approved-At: Tue, 27 Oct 2026 10:31:50 +0100 CC: Carsten Haerle X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: activate_storage() imports a pool on demand and then runs 'zfs mount -a'. Neither step is serialized against other callers on the same node. At boot, pvestatd and pve-guests (startall) call it for the same storage within about a second of each other, which fails in two ways: Import: while a 'zpool import' is running, the pool is neither listed by 'zpool list' nor offered by 'zpool import', so the second caller gets "cannot import '': no such pool available". The existing re-check only covers imports that finish within milliseconds. For pools that take seconds (many datasets, dedup tables) the onboot guest start fails and startall does not retry. Mount: both callers then run 'zfs mount -a' and collide on the same datasets ("mountpoint or dataset is busy"). 'zfs mount -a' exits non-zero if any dataset fails, so the worker dies even though the storage's own dataset is mounted. Datasets unrelated to the storage on the same pool cause the same failure. Observed with a raidz2 pool whose import takes ~40 s: 54 s pvestatd: zpool import data1 (runs until 93 s) 55 s pve-guests: Starting VM 124 58 s Starting VM 124 failed: could not activate storage 'vmdata1zfs', zfs error: cannot import 'data1': no such pool available Serialize check, import and mount per pool with a node-local lock. A caller that obtains the lock after another importer re-checks first and does not import again. Every caller tries the lock for one second first. If it is busy, non-workers (pvestatd, API requests) fail with an explicit "import of pool '' in progress", so pvestatd reports the storage as inactive for one round instead of blocking. Workers (guest start, backup, migration) log that they are waiting and then wait for the lock with a long timeout. A failed 'zfs mount -a' is only fatal if the storage's dataset is not mounted afterwards, using the same heuristic the function already uses for its early return. While at it, log which caller imports the pool ("importing ZFS pool '' for storage ''") and drop the warning that the re-check emitted for the expected "no such pool" case, so the task log of a guest start shows what actually happened. Tested on PVE 9.2.21 / libpve-storage-perl 9.1.13 with 12 reboots on four nodes: the onboot guests started every time, in both orders (pvestatd imports and the worker waits at the lock; the worker imports and pvestatd reports "in progress"). Signed-off-by: Carsten Haerle --- src/PVE/Storage/ZFSPoolPlugin.pm | 66 +++++++++++++++++++++++++++----- 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/src/PVE/Storage/ZFSPoolPlugin.pm b/src/PVE/Storage/ZFSPoolPlug= in.pm index e7cd1de..0cb2d1e 100644 --- a/src/PVE/Storage/ZFSPoolPlugin.pm +++ b/src/PVE/Storage/ZFSPoolPlugin.pm @@ -713,22 +713,70 @@ sub activate_storage { my $pool_imported =3D sub { my @param =3D ('-o', 'name', '-H', $pool); my $res =3D eval { $class->zfs_request($scfg, undef, 'zpool_list',= @param) }; - warn "$@\n" if $@; + # a not yet imported pool is the expected case here, not worth a w= arning + warn "$@\n" if $@ && $@ !~ m/no such pool/; =20 return defined($res) && $res =3D~ m/$pool/; }; =20 - if (!$pool_imported->()) { - # import can only be done if not yet imported! - my @param =3D ('-d', '/dev/disk/by-id/', '-o', 'cachefile=3Dnone',= $pool); - eval { $class->zfs_request($scfg, undef, 'zpool_import', @param) }= ; + my $activate =3D sub { + # whoever held the lock before us may have done the work already + return 1 if dataset_mounted_heuristic($dataset); + + if (!$pool_imported->()) { + print STDERR "importing ZFS pool '$pool' for storage '$storeid= '\n"; + # import can only be done if not yet imported! + my @param =3D ('-d', '/dev/disk/by-id/', '-o', 'cachefile=3Dno= ne', $pool); + eval { $class->zfs_request($scfg, undef, 'zpool_import', @para= m) }; + if (my $err =3D $@) { + # an import outside of PVE (manual, zfs-import units) can = still race + die "could not activate storage '$storeid', $err\n" if !$p= ool_imported->(); + } + } + + # 'zfs mount -a' covers all datasets of all pools. Datasets unrela= ted to + # this storage may fail to mount (e.g. "mountpoint or dataset is b= usy" + # because another caller is mounting, or a previous run was killed= by + # its timeout). That must not fail the activation as long as our + # dataset is mounted. + eval { $class->zfs_request($scfg, undef, 'mount', '-a') }; if (my $err =3D $@) { - # just could've raced with another import, so recheck if it is= imported - die "could not activate storage '$storeid', $err\n" if !$pool_= imported->(); + die "could not activate storage '$storeid', $err\n" + if !dataset_mounted_heuristic($dataset); + warn "storage '$storeid': mounting all ZFS datasets failed, $e= rr"; + } + return 1; + }; + + # Serialize import and mount of the same pool on this node. While a + # 'zpool import' is running, the pool is neither listed by 'zpool list= ' + # nor offered by 'zpool import', so a concurrent caller fails with + # "no such pool available". The import can take a long time (large poo= ls, + # dedup tables), so the immediate re-check alone is not sufficient. + # Workers (guest start, backup, migration) wait for the import to fini= sh, + # non-workers (pvestatd, API requests) only wait briefly and report th= e + # storage as not yet active. + my $lockfile =3D "/run/lock/pve-zpool-import-$pool.lck"; + my $lock_busy =3D sub { return $_[0] =3D~ m/can't lock file .* got tim= eout/ }; + + # try briefly first, so that a worker can tell what it is waiting for + my $res =3D PVE::Tools::lock_file($lockfile, 1, $activate); + if (!$res) { + my $err =3D $@; + die $err if !$lock_busy->($err); + die "could not activate storage '$storeid', import of pool '$pool'= in progress\n" + if !PVE::RPCEnvironment->is_worker(); + + print STDERR "pool '$pool' is being imported or mounted by another= process, waiting...\n"; + $res =3D PVE::Tools::lock_file($lockfile, 60 * 10, $activate); + if (!$res) { + $err =3D $@; + die "could not activate storage '$storeid', " + . "timeout waiting for import of pool '$pool'\n" + if $lock_busy->($err); + die $err; } } - eval { $class->zfs_request($scfg, undef, 'mount', '-a') }; - die "could not activate storage '$storeid', $@\n" if $@; return 1; } =20 --=20 2.53.0