* [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option
@ 2025-05-27 13:34 Fiona Ebner
2025-05-27 13:34 ` [pve-devel] [PATCH v2 storage 2/2] zfs over iscsi: on-add hook: dynamically determine base path Fiona Ebner
2025-06-05 11:02 ` [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Christoph Heiss
0 siblings, 2 replies; 4+ messages in thread
From: Fiona Ebner @ 2025-05-27 13:34 UTC (permalink / raw)
To: pve-devel
Use '/dev/zvol' as a base path for new storages for providers 'iet'
and 'LIO', because that is what modern distributions use.
This is a breaking change regarding the addition of new storages on
older distributions, but it's enough to specify the base path '/dev'
explicitly for setups that require it.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
Changes in v2:
* Cleaner/more robust approach, add storage configuration setting.
src/PVE/Storage/LunCmd/Comstar.pm | 3 ++-
src/PVE/Storage/LunCmd/Iet.pm | 5 ++--
src/PVE/Storage/LunCmd/Istgt.pm | 5 ++--
src/PVE/Storage/LunCmd/LIO.pm | 5 ++--
src/PVE/Storage/ZFSPlugin.pm | 39 +++++++++++++++++++++++++++----
5 files changed, 46 insertions(+), 11 deletions(-)
diff --git a/src/PVE/Storage/LunCmd/Comstar.pm b/src/PVE/Storage/LunCmd/Comstar.pm
index 527e4ba..5a1d2e6 100644
--- a/src/PVE/Storage/LunCmd/Comstar.pm
+++ b/src/PVE/Storage/LunCmd/Comstar.pm
@@ -32,7 +32,8 @@ my $get_lun_cmd_map = sub {
};
sub get_base {
- return '/dev/zvol/rdsk';
+ my ($scfg) = @_;
+ return $scfg->{'zfs-base-path'} || '/dev/zvol/rdsk';
}
sub run_lun_command {
diff --git a/src/PVE/Storage/LunCmd/Iet.pm b/src/PVE/Storage/LunCmd/Iet.pm
index 5b09b88..7482d66 100644
--- a/src/PVE/Storage/LunCmd/Iet.pm
+++ b/src/PVE/Storage/LunCmd/Iet.pm
@@ -133,7 +133,7 @@ my $parser = sub {
my $line = 0;
- my $base = get_base;
+ my $base = get_base($scfg);
my $config = $get_config->($scfg);
my @cfgfile = split "\n", $config;
@@ -471,7 +471,8 @@ sub run_lun_command {
}
sub get_base {
- return '/dev';
+ my ($scfg) = @_;
+ return $scfg->{'zfs-base-path'} || '/dev';
}
1;
diff --git a/src/PVE/Storage/LunCmd/Istgt.pm b/src/PVE/Storage/LunCmd/Istgt.pm
index 2f758f9..a53e25e 100644
--- a/src/PVE/Storage/LunCmd/Istgt.pm
+++ b/src/PVE/Storage/LunCmd/Istgt.pm
@@ -304,7 +304,7 @@ my $parser = sub {
$CONFIG =~ s/\n$//;
die "$scfg->{target}: Target not found" unless $SETTINGS->{targets};
my $max = $SETTINGS->{targets};
- my $base = get_base;
+ my $base = get_base($scfg);
for (my $i = 1; $i <= $max; $i++) {
my $target = $SETTINGS->{nodebase}.':'.$SETTINGS->{"LogicalUnit$i"}->{TargetName};
@@ -595,7 +595,8 @@ sub run_lun_command {
}
sub get_base {
- return '/dev/zvol';
+ my ($scfg) = @_;
+ return $scfg->{'zfs-base-path'} || '/dev/zvol';
}
1;
diff --git a/src/PVE/Storage/LunCmd/LIO.pm b/src/PVE/Storage/LunCmd/LIO.pm
index 9264e46..7207c05 100644
--- a/src/PVE/Storage/LunCmd/LIO.pm
+++ b/src/PVE/Storage/LunCmd/LIO.pm
@@ -223,7 +223,7 @@ my $extract_volname = sub {
my ($scfg, $lunpath) = @_;
my $volname = undef;
- my $base = get_base;
+ my $base = get_base($scfg);
if ($lunpath =~ /^$base\/$scfg->{pool}\/([\w\-]+)$/) {
$volname = $1;
my $prefix = $get_backstore_prefix->($scfg);
@@ -414,7 +414,8 @@ sub run_lun_command {
}
sub get_base {
- return '/dev';
+ my ($scfg) = @_;
+ return $scfg->{'zfs-base-path'} || '/dev';
}
1;
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 94cb11f..64f1eda 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -39,13 +39,13 @@ my $zfs_get_base = sub {
my ($scfg) = @_;
if ($scfg->{iscsiprovider} eq 'comstar') {
- return PVE::Storage::LunCmd::Comstar::get_base;
+ return PVE::Storage::LunCmd::Comstar::get_base($scfg);
} elsif ($scfg->{iscsiprovider} eq 'istgt') {
- return PVE::Storage::LunCmd::Istgt::get_base;
+ return PVE::Storage::LunCmd::Istgt::get_base($scfg);
} elsif ($scfg->{iscsiprovider} eq 'iet') {
- return PVE::Storage::LunCmd::Iet::get_base;
+ return PVE::Storage::LunCmd::Iet::get_base($scfg);
} elsif ($scfg->{iscsiprovider} eq 'LIO') {
- return PVE::Storage::LunCmd::LIO::get_base;
+ return PVE::Storage::LunCmd::LIO::get_base($scfg);
} else {
$zfs_unknown_scsi_provider->($scfg->{iscsiprovider});
}
@@ -204,6 +204,12 @@ sub properties {
description => "target portal group for Linux LIO targets",
type => 'string',
},
+ 'zfs-base-path' => {
+ description => "Base path where to look for the created ZFS block devices. Set"
+ ." automatically during creation if not specificed. Usually '/dev/zvol'.",
+ type => 'string',
+ format => 'pve-storage-path',
+ },
};
}
@@ -223,11 +229,36 @@ sub options {
lio_tpg => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
+ 'zfs-base-path' => { optional => 1 },
};
}
# Storage implementation
+sub on_add_hook {
+ my ($class, $storeid, $scfg, %param) = @_;
+
+ if (!$scfg->{'zfs-base-path'}) {
+ my $base_path;
+ if ($scfg->{iscsiprovider} eq 'comstar') {
+ $base_path = PVE::Storage::LunCmd::Comstar::get_base($scfg);
+ } elsif ($scfg->{iscsiprovider} eq 'istgt') {
+ $base_path = PVE::Storage::LunCmd::Istgt::get_base($scfg);
+ } elsif ($scfg->{iscsiprovider} eq 'iet' || $scfg->{iscsiprovider} eq 'LIO') {
+ # Provider implementations hard-code '/dev/', which does not work for distributions like
+ # Debian 12. Keep that implementation as-is for backwards compatibility, but use
+ # '/dev/zvol' here.
+ $base_path = '/dev/zvol';
+ } else {
+ $zfs_unknown_scsi_provider->($scfg->{iscsiprovider});
+ }
+
+ $scfg->{'zfs-base-path'} = $base_path;
+ }
+
+ return;
+}
+
sub path {
my ($class, $scfg, $volname, $storeid, $snapname) = @_;
--
2.39.5
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* [pve-devel] [PATCH v2 storage 2/2] zfs over iscsi: on-add hook: dynamically determine base path
2025-05-27 13:34 [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Fiona Ebner
@ 2025-05-27 13:34 ` Fiona Ebner
2025-06-05 11:02 ` [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Christoph Heiss
1 sibling, 0 replies; 4+ messages in thread
From: Fiona Ebner @ 2025-05-27 13:34 UTC (permalink / raw)
To: pve-devel
This reduces the potential breakage from commit "fix #5071: zfs over
iscsi: add 'zfs-base-path' configuration option". Only setups where
'/dev/zvol' exists, but is not a valid base, will still be affected.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
This one is optional, not sure if it's worth it.
Changes in v2:
* Actually check remote host.
src/PVE/Storage/ZFSPlugin.pm | 26 ++++++++++++++++++++++----
1 file changed, 22 insertions(+), 4 deletions(-)
diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
index 64f1eda..343f1ec 100644
--- a/src/PVE/Storage/ZFSPlugin.pm
+++ b/src/PVE/Storage/ZFSPlugin.pm
@@ -3,9 +3,10 @@ package PVE::Storage::ZFSPlugin;
use strict;
use warnings;
use IO::File;
-use POSIX;
+use POSIX qw(ENOENT);
use PVE::Tools qw(run_command);
use PVE::Storage::ZFSPoolPlugin;
+use PVE::RESTEnvironment qw(log_warn);
use PVE::RPCEnvironment;
use base qw(PVE::Storage::ZFSPoolPlugin);
@@ -246,9 +247,26 @@ sub on_add_hook {
$base_path = PVE::Storage::LunCmd::Istgt::get_base($scfg);
} elsif ($scfg->{iscsiprovider} eq 'iet' || $scfg->{iscsiprovider} eq 'LIO') {
# Provider implementations hard-code '/dev/', which does not work for distributions like
- # Debian 12. Keep that implementation as-is for backwards compatibility, but use
- # '/dev/zvol' here.
- $base_path = '/dev/zvol';
+ # Debian 12. Keep that implementation as-is for backwards compatibility, but use custom
+ # logic here.
+ my $target = 'root@' . $scfg->{portal};
+ my $cmd = [@ssh_cmd, '-i', "$id_rsa_path/$scfg->{portal}_id_rsa", $target];
+ push $cmd->@*, 'ls', '/dev/zvol';
+
+ my $rc = eval { run_command($cmd, timeout => 10, noerr => 1, quiet => 1) };
+ my $err = $@;
+ if (defined($rc) && $rc == 0) {
+ $base_path = '/dev/zvol';
+ } elsif (defined($rc) && $rc == ENOENT) {
+ $base_path = '/dev';
+ } else {
+ my $message = $err ? $err : "remote command failed";
+ chomp($message);
+ $message .= " ($rc)" if defined($rc);
+ $message .= " - check 'zfs-base-path' setting manually!";
+ log_warn($message);
+ $base_path = '/dev/zvol';
+ }
} else {
$zfs_unknown_scsi_provider->($scfg->{iscsiprovider});
}
--
2.39.5
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option
2025-05-27 13:34 [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Fiona Ebner
2025-05-27 13:34 ` [pve-devel] [PATCH v2 storage 2/2] zfs over iscsi: on-add hook: dynamically determine base path Fiona Ebner
@ 2025-06-05 11:02 ` Christoph Heiss
2025-06-05 11:17 ` Fiona Ebner
1 sibling, 1 reply; 4+ messages in thread
From: Christoph Heiss @ 2025-06-05 11:02 UTC (permalink / raw)
To: Fiona Ebner; +Cc: Proxmox VE development discussion
Tested the series by setting up a iSCSI target using targetcli(d) (on a
separate PVE 8.4 system as a base, due to ZFS goodies) and then adding a
ZFS-over-iSCSI storage using the LIO provider to a test cluster.
Confirmed that
- `zfs-base-path` is correctly detected when adding the storage
- the iSCSI storage is seen correctly after setting up and that VM disks
can be (live-)migrated to the ZFS-over-iSCSI storage w/o problems.
One small comment inline, just a typo in the API description.
Please consider the series in any case
Tested-by: Christoph Heiss <c.heiss@proxmox.com>
--
One unrelated thing I noticed during testing, but wanted to note for
reference:
When one hits the error due to a bad `zfs-base-path` (e.g. as currently
happens):
`TASK ERROR: storage migration failed: Could not open /dev/<poolname>/vm-100-disk-0`
the target zvol isn't cleaned up, e.g. the above would result in
`<poolname>/vm-100-disk-0` still being present on the remote zpool.
Fortunately this doesn't really break anything, as the next available
disk number (in this case, `vm-100-disk-1`), is chosen automatically
anyway when creating a new disk.
On Tue May 27, 2025 at 3:34 PM CEST, Fiona Ebner wrote:
> Use '/dev/zvol' as a base path for new storages for providers 'iet'
> and 'LIO', because that is what modern distributions use.
>
> This is a breaking change regarding the addition of new storages on
> older distributions, but it's enough to specify the base path '/dev'
> explicitly for setups that require it.
>
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> ---
[..]
> diff --git a/src/PVE/Storage/ZFSPlugin.pm b/src/PVE/Storage/ZFSPlugin.pm
> index 94cb11f..64f1eda 100644
> --- a/src/PVE/Storage/ZFSPlugin.pm
> +++ b/src/PVE/Storage/ZFSPlugin.pm
> @@ -39,13 +39,13 @@ my $zfs_get_base = sub {
> my ($scfg) = @_;
>
> if ($scfg->{iscsiprovider} eq 'comstar') {
> - return PVE::Storage::LunCmd::Comstar::get_base;
> + return PVE::Storage::LunCmd::Comstar::get_base($scfg);
> } elsif ($scfg->{iscsiprovider} eq 'istgt') {
> - return PVE::Storage::LunCmd::Istgt::get_base;
> + return PVE::Storage::LunCmd::Istgt::get_base($scfg);
> } elsif ($scfg->{iscsiprovider} eq 'iet') {
> - return PVE::Storage::LunCmd::Iet::get_base;
> + return PVE::Storage::LunCmd::Iet::get_base($scfg);
> } elsif ($scfg->{iscsiprovider} eq 'LIO') {
> - return PVE::Storage::LunCmd::LIO::get_base;
> + return PVE::Storage::LunCmd::LIO::get_base($scfg);
> } else {
> $zfs_unknown_scsi_provider->($scfg->{iscsiprovider});
> }
> @@ -204,6 +204,12 @@ sub properties {
> description => "target portal group for Linux LIO targets",
> type => 'string',
> },
> + 'zfs-base-path' => {
> + description => "Base path where to look for the created ZFS block devices. Set"
> + ." automatically during creation if not specificed. Usually '/dev/zvol'.",
small typo :^)
s/specificed/specified/g
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option
2025-06-05 11:02 ` [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Christoph Heiss
@ 2025-06-05 11:17 ` Fiona Ebner
0 siblings, 0 replies; 4+ messages in thread
From: Fiona Ebner @ 2025-06-05 11:17 UTC (permalink / raw)
To: Christoph Heiss; +Cc: Proxmox VE development discussion
Am 05.06.25 um 13:02 schrieb Christoph Heiss:
> Tested the series by setting up a iSCSI target using targetcli(d) (on a
> separate PVE 8.4 system as a base, due to ZFS goodies) and then adding a
> ZFS-over-iSCSI storage using the LIO provider to a test cluster.
>
> Confirmed that
>
> - `zfs-base-path` is correctly detected when adding the storage
>
> - the iSCSI storage is seen correctly after setting up and that VM disks
> can be (live-)migrated to the ZFS-over-iSCSI storage w/o problems.
>
> One small comment inline, just a typo in the API description.
>
> Please consider the series in any case
>
> Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Thank you for testing! Superseded by v3 with the typo fixed:
https://lore.proxmox.com/pve-devel/20250605111109.52712-1-f.ebner@proxmox.com/
> One unrelated thing I noticed during testing, but wanted to note for
> reference:
>
> When one hits the error due to a bad `zfs-base-path` (e.g. as currently
> happens):
>
> `TASK ERROR: storage migration failed: Could not open /dev/<poolname>/vm-100-disk-0`
>
> the target zvol isn't cleaned up, e.g. the above would result in
> `<poolname>/vm-100-disk-0` still being present on the remote zpool.
>
> Fortunately this doesn't really break anything, as the next available
> disk number (in this case, `vm-100-disk-1`), is chosen automatically
> anyway when creating a new disk.
There actually is already error handling for freeing up allocated disks
in this context. But the storage plugin itself already fails during
allocation, so the new volume ID is never returned as a result, so
qemu-server doesn't know about it. I'll send a patch to improve cleanup
handling inside the plugin itself.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-06-05 11:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-05-27 13:34 [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Fiona Ebner
2025-05-27 13:34 ` [pve-devel] [PATCH v2 storage 2/2] zfs over iscsi: on-add hook: dynamically determine base path Fiona Ebner
2025-06-05 11:02 ` [pve-devel] [PATCH v2 storage 1/2] fix #5071: zfs over iscsi: add 'zfs-base-path' configuration option Christoph Heiss
2025-06-05 11:17 ` Fiona Ebner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal