From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C756A1FF09C for ; Mon, 05 Oct 2026 02:26:37 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DD93E21655; Mon, 05 Oct 2026 02:26:25 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=neatech-ar.20251104.gappssmtp.com; s=20251104; t=1791159981; x=1791764781; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WoSYsyNQR5lmVrZLQJ1F4SAZPXERLDQiuL2TjAjHUAo=; b=eooOpeHE4D3JleML07VBA+E/XwaVTB+Q9gmM70eM5YZwYF4/hqeGs/frt8rBASTBOV HO3ce0eMd4/Vzx+pMQPUrfiv+5hPC1g2Ii3IHrOjLXOdT/ApVXI3BU6P3UG+sf9nzFXi isfCuz7/Go3qcFu5SxuwXZMMwTuQV8f6WCu/19/4Ie90s/H3lJZkYTkqG+rFIb8qNCcq MqwMs0xPdARejTuGEpZcFh3lVYkX7YdZdj7FxvEyEFDBBQa402dofm5xmJJ4qvz/pjHO wrEHWuIopbpthvh5dCOdwihgKwKXXRAfR6JOeD0KmhAurrcZbPC1zgzQ5xDlhlph2zKk pAaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791159981; x=1791764781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=WoSYsyNQR5lmVrZLQJ1F4SAZPXERLDQiuL2TjAjHUAo=; b=fEXnnxONoluxuLeH3qbmRfG9UhDTAWMQQ5xatg13Z8mgGzb2XG9/eHQ/Heq6mADq9a 573zUdqOuZetygixeqZzLQWLOVKBdLniuwniF75vU7sE4OXLBJy3K+HeTRx691UAjm3H m5mlPgmP1mxjSmp6Kis4a2CpZ50uyezXX6uOOw2jHQqP9V2oI+pvL9AOLovrvF+u/iud DvKMWrSj+//7VUEsikPKfZFRIA5CObSG9LxtPJercA7W74e6JA2CgjHsNt26GqECx6o6 NBleyl5S2GdtkNHmaLZBNXru1liY8kJDZXpbyCr/Bn6yKG7JjVVORG5kp/9JUE6AYp40 th4Q== X-Gm-Message-State: AFq9FYKhkldanL1jIw8ps6PsuW4sBleUkpl7WWCTHKXTKeIU3l0enpoa CYEjImfCsEg9p85By4eG/bJZBVH8SbZUmft7wwTrlP1wOrciX9ogFXANZwOgIFWjRzRe3nBRc7v uZniqdCQ= X-Gm-Gg: AYBFou1GySaJGGRmwo8FgGUkbJIBXjlfq19dGC4dB8fdkRHD8Jwnl9IWZTMCyX32N/S cs1Vp281hszU4QXgyjL0tWJttTLvA1oJkZbO2SV40HgKwN7iFeKRlQjiyP2JYuJpPnBorSheJIa juKrINY3ToWkaPhJ2YjGtmhldZksbBkPBb19WkMkN+jPodRvBcYeiskq0V2E5Yzy2jD3lrafTVk DDF5AX/aCNdgEwietjSoyeLrAdGQKQOyBkcOnSNEbdED52bgJJNY98EppTCHDN61J7VG4grEeNy x+bn/U4wiMV3EhQUuxo5BcEfd3MynRdg8AXilF1CQXuF72G+BJaDAcoa1xOW5ftt4GCbnTIp4zd GKOAGhT9MLPpd/86tYFWTXqJ5YdVukK7HRkR46MF9MCfngjanr9DtgL9lAHfgfxMv6G2KS4YQGz DX0gLKIVZpbdNUPHnXcQoho/un9/hyL+5wBmmrmQ8ReL1VRb34tqBbcY3C4QlpP8WN5jpxku+hO qhHZwu2BbF16a/KfdtDKI0jWIRaHqMdVCpnw3zj9hYER2t1u4o0Sm75ec5yVwHWYH1WjmqkE0jW X-Received: by 2002:a05:6122:6b8c:10b0:5c9:f6b0:e5e0 with SMTP id 71dfb90a1353d-5daafe48815mr1283422e0c.3.1791159980733; Sun, 04 Oct 2026 17:26:20 -0700 (PDT) From: Joaquin Varela To: pve-devel@lists.proxmox.com Subject: [PATCH storage v3 4/4] zfsnvme: wait up to 30 seconds for the shared storage lock Date: Sun, 4 Oct 2026 21:26:07 -0300 Message-ID: <20261005002609.571-5-joaquinvarela@neatech.ar> X-Mailer: git-send-email 2.54.0.windows.1 In-Reply-To: <20261005002609.571-1-joaquinvarela@neatech.ar> References: <20261005002609.571-1-joaquinvarela@neatech.ar> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_PASS -0.1 DMARC pass policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: XCEDTGVC77L554J33H3CYHXAXK2SQOWA X-Message-ID-Hash: XCEDTGVC77L554J33H3CYHXAXK2SQOWA X-MailFrom: joaquinvarela@neatech.ar X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The core takes the shared storage lock without a timeout around vdisk_alloc, vdisk_free, vdisk_clone, vdisk_create_base, rename_volume and rename_snapshot, so cfs_lock applies its default of 10 seconds for acquiring it. A zfsnvme allocation holds that lock for several SSH round trips (read the target state, reserve the NSID, create the zvol, wait for its device, export and verify the namespace), and keeps it while it waits for the target lock, which a destroy or rollback on the same target can hold for several seconds. When two guests get disks on one storage at the same time, the second one can then fail with "got lock request timeout" although nothing went wrong. Override cluster_lock_storage() to wait up to 30 seconds for the shared lock when the caller passes no timeout, the same acquisition budget as for the target lock. An explicit timeout is passed on unchanged (for 0, cfs_lock still applies its own default of 10 seconds), and so are the execution timeout, error handling and exclusion of the parent method. Signed-off-by: Joaquin Varela --- src/PVE/Storage/ZFSNVMePlugin.pm | 9 ++++++++ src/test/zfsnvme_test.pm | 35 +++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/src/PVE/Storage/ZFSNVMePlugin.pm b/src/PVE/Storage/ZFSNVMePlugin.pm index dfd43d43..67595350 100644 --- a/src/PVE/Storage/ZFSNVMePlugin.pm +++ b/src/PVE/Storage/ZFSNVMePlugin.pm @@ -2452,6 +2452,15 @@ sub clone_image($class, $scfg, $storeid, $volname, $vmid, $snap = undef) { return "$basename/$name"; } +sub cluster_lock_storage($class, $storeid, $shared, $timeout, $func, @param) { + # Remote lifecycle operations take several SSH round trips while the core + # holds this lock. Allow a bounded queue of such operations, using the same + # acquisition budget as the target lock. Keep caller-supplied deadlines and + # the parent's execution timeout, error handling and exclusion unchanged. + $timeout = $nvmet_lock_wait if $shared && !defined($timeout); + return $class->SUPER::cluster_lock_storage($storeid, $shared, $timeout, $func, @param); +} + sub alloc_image($class, $storeid, $scfg, $vmid, $fmt, $name, $size) { die "unsupported format '$fmt'" if $fmt ne 'raw'; die "illegal name '$name' - should be 'vm-$vmid-*'\n" diff --git a/src/test/zfsnvme_test.pm b/src/test/zfsnvme_test.pm index 8a963bfe..7a218bf5 100644 --- a/src/test/zfsnvme_test.pm +++ b/src/test/zfsnvme_test.pm @@ -1108,6 +1108,39 @@ my @ownership = ( # The shared storage lock # --------------------------------------------------------------------------- +subtest 'cluster lock timeout defaults and passthrough' => sub { + my @base_calls; + $parent_mock->redefine( + cluster_lock_storage => sub($class, $storeid, $shared, $timeout, $func, @param) { + push @base_calls, [$class, $storeid, $shared, $timeout, @param]; + return $func->(@param); + }, + ); + for my $case ( + ['shared default', 1, undef, 30], + ['shared explicit', 1, 12, 12], + ['shared zero (the parent applies its own default)', 1, 0, 0], + ['local default', 0, undef, undef], + ) { + my ($label, $shared, $timeout, $expected) = $case->@*; + my $result = $PLUGIN->cluster_lock_storage( + 'nvmetest', + $shared, + $timeout, + sub(@args) { return \@args }, + 'left', + 'right', + ); + is_deeply( + $base_calls[-1], + [$PLUGIN, 'nvmetest', $shared, $expected, 'left', 'right'], + "$label: the parent gets its arguments", + ); + is_deeply($result, ['left', 'right'], "$label: the callback runs with its parameters"); + } + $parent_mock->unmock('cluster_lock_storage'); +}; + subtest 'vdisk_alloc dispatches under the shared storage lock' => sub { my $cluster_mock = Test::MockModule->new('PVE::Cluster'); my (@locks, @allocations); @@ -1131,7 +1164,7 @@ subtest 'vdisk_alloc dispatches under the shared storage lock' => sub { 'nvmetest:vm-105-disk-0', 'core allocation returns the allocated volume ID', ); - is_deeply(\@locks, [['nvmetest', undef]], 'cfs_lock_storage gets the timeout of the core'); + is_deeply(\@locks, [['nvmetest', 30]], 'cfs_lock_storage waits the bounded default'); is_deeply( \@allocations, [['nvmetest', 105, 'raw', 'vm-105-disk-0', 131_072]],