From: Joaquin Varela <joaquinvarela@neatech.ar>
To: pve-devel@lists.proxmox.com
Subject: [PATCH storage v3 2/4] test: add zfsnvme plugin tests
Date: Sun, 4 Oct 2026 21:26:05 -0300 [thread overview]
Message-ID: <20261005002609.571-3-joaquinvarela@neatech.ar> (raw)
In-Reply-To: <20261005002609.571-1-joaquinvarela@neatech.ar>
Add two test files to the plugin test harness and register them in
run_plugin_tests.pl.
zfsnvme_test.pm covers the host side: option and portal parsing, the
schema and the config checks, the key handling (the add and update
hooks with /etc/pve/priv/storage, and the 'dhchap-key' file mapping of
'pvesm add' and 'pvesm set'), the SSH runner and its error
classification, the /dev/nvme-fabrics option builder and the connect
child (against a pseudoterminal that answers like the fabrics device;
skipped without /dev/ptmx), the sysfs-based rescan and controller
deletion, path selection and activation, vdisk_alloc under the core
storage lock, and the volume path and QEMU blockdev options. Nothing in
it opens a connection or touches a local NVMe device.
zfsnvme_target_test.pm covers the target side: the parsers and planners
on fixture reads of an nvmet target (Linux 6.8, ZFS 2.2; addresses and
NQNs replaced by documentation values), the command renderer and
chunking, and every target flow against an in-memory target
(FakeTarget) that interprets exactly the commands the plugin sends and
records anything else as a protocol violation. A fault sweep fails
every mutating call of every flow in each way an SSH call can fail
(before or after it ran, a connection cut or lost in the middle of a
chain, or a call the target runs to its end after ssh gave up on it)
and checks that identities stay unique, nothing of other storages
changes, every change ran under one domain lock, and the next
activation converges. The rendered chains also run with /bin/sh on a
configfs stand-in in a temporary directory, which checks the quoting
and the POSIX command shapes; these tests bail out unless /bin/sh,
find, grep, sha256sum, dd and tee are available.
Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
src/test/run_plugin_tests.pl | 2 +
.../zfsnvme_fixtures/configfs_snapshot.txt | 38 +
src/test/zfsnvme_fixtures/zfs_inventory.txt | 15 +
src/test/zfsnvme_target_test.pm | 4849 +++++++++++++++++
src/test/zfsnvme_test.pm | 2353 ++++++++
5 files changed, 7257 insertions(+)
create mode 100644 src/test/zfsnvme_fixtures/configfs_snapshot.txt
create mode 100644 src/test/zfsnvme_fixtures/zfs_inventory.txt
create mode 100644 src/test/zfsnvme_target_test.pm
create mode 100644 src/test/zfsnvme_test.pm
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 8bce9d3b..d063a3d2 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -17,6 +17,8 @@ my $res = $harness->runtests(
"get_subdir_test.pm",
"filesystem_path_test.pm",
"prune_backups_test.pm",
+ "zfsnvme_test.pm",
+ "zfsnvme_target_test.pm",
);
exit -1 if !$res || $res->{failed} || $res->{parse_errors};
diff --git a/src/test/zfsnvme_fixtures/configfs_snapshot.txt b/src/test/zfsnvme_fixtures/configfs_snapshot.txt
new file mode 100644
index 00000000..1b7672a0
--- /dev/null
+++ b/src/test/zfsnvme_fixtures/configfs_snapshot.txt
@@ -0,0 +1,38 @@
+D /sys/kernel/config/nvmet
+D /sys/kernel/config/nvmet/hosts
+D /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:0a0a0a0a-0a0a-4a0a-8a0a-0a0a0a0a0a0a
+D /sys/kernel/config/nvmet/hosts/nqn.2014-08.org.nvmexpress:uuid:0b0b0b0b-0b0b-4b0b-8b0b-0b0b0b0b0b0b
+D /sys/kernel/config/nvmet/ports
+D /sys/kernel/config/nvmet/ports/2
+D /sys/kernel/config/nvmet/ports/2/subsystems
+D /sys/kernel/config/nvmet/ports/1
+D /sys/kernel/config/nvmet/ports/1/subsystems
+D /sys/kernel/config/nvmet/subsystems
+D /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme
+D /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/allowed_hosts
+D /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces
+D /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/6
+D /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/5
+L /sys/kernel/config/nvmet/ports/2/subsystems/nqn.2026-01.com.example:zfsnvme
+L /sys/kernel/config/nvmet/ports/1/subsystems/nqn.2026-01.com.example:zfsnvme
+L /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/allowed_hosts/nqn.2014-08.org.nvmexpress:uuid:0a0a0a0a-0a0a-4a0a-8a0a-0a0a0a0a0a0a
+L /sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/allowed_hosts/nqn.2014-08.org.nvmexpress:uuid:0b0b0b0b-0b0b-4b0b-8b0b-0b0b0b0b0b0b
+/sys/kernel/config/nvmet/ports/2/addr_trtype:tcp
+/sys/kernel/config/nvmet/ports/2/addr_trsvcid:4420
+/sys/kernel/config/nvmet/ports/2/addr_traddr:192.0.2.22
+/sys/kernel/config/nvmet/ports/2/addr_adrfam:ipv4
+/sys/kernel/config/nvmet/ports/1/addr_trtype:tcp
+/sys/kernel/config/nvmet/ports/1/addr_trsvcid:4420
+/sys/kernel/config/nvmet/ports/1/addr_traddr:192.0.2.21
+/sys/kernel/config/nvmet/ports/1/addr_adrfam:ipv4
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/6/buffered_io:0
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/6/enable:1
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/6/device_uuid:0d0d0d0d-0d0d-4d0d-8d0d-0d0d0d0d0d0d
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/6/device_path:/dev/zvol/tank/vm-100-cloudinit
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/5/buffered_io:0
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/5/enable:1
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/5/device_uuid:0c0c0c0c-0c0c-4c0c-8c0c-0c0c0c0c0c0c
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/namespaces/5/device_path:/dev/zvol/tank/vm-100-disk-0
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/attr_model:Proxmox ZFS NVMe
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/attr_serial:PVEZFS51b2c3c90ea254
+/sys/kernel/config/nvmet/subsystems/nqn.2026-01.com.example:zfsnvme/attr_allow_any_host:0
diff --git a/src/test/zfsnvme_fixtures/zfs_inventory.txt b/src/test/zfsnvme_fixtures/zfs_inventory.txt
new file mode 100644
index 00000000..69ba9238
--- /dev/null
+++ b/src/test/zfsnvme_fixtures/zfs_inventory.txt
@@ -0,0 +1,15 @@
+tank type filesystem -
+tank proxmox:nvme-subsys - -
+tank proxmox:nvme-nsid - -
+tank proxmox:nvme-uuid - -
+tank proxmox:nvme-last-nsid - -
+tank/vm-100-cloudinit type volume -
+tank/vm-100-cloudinit proxmox:nvme-subsys nqn.2026-01.com.example:zfsnvme local
+tank/vm-100-cloudinit proxmox:nvme-nsid 6 local
+tank/vm-100-cloudinit proxmox:nvme-uuid 0d0d0d0d-0d0d-4d0d-8d0d-0d0d0d0d0d0d local
+tank/vm-100-cloudinit proxmox:nvme-last-nsid - -
+tank/vm-100-disk-0 type volume -
+tank/vm-100-disk-0 proxmox:nvme-subsys nqn.2026-01.com.example:zfsnvme local
+tank/vm-100-disk-0 proxmox:nvme-nsid 5 local
+tank/vm-100-disk-0 proxmox:nvme-uuid 0c0c0c0c-0c0c-4c0c-8c0c-0c0c0c0c0c0c local
+tank/vm-100-disk-0 proxmox:nvme-last-nsid - -
diff --git a/src/test/zfsnvme_target_test.pm b/src/test/zfsnvme_target_test.pm
new file mode 100644
index 00000000..991d2084
--- /dev/null
+++ b/src/test/zfsnvme_target_test.pm
@@ -0,0 +1,4849 @@
+# Target side of the zfsnvme storage plugin: the parsers and planners on the
+# fixture reads, the command renderer, and every target flow against an
+# in-memory NVMe target (FakeTarget) that interprets exactly the commands the
+# plugin sends. Nothing here opens a connection or touches local NVMe devices.
+
+use v5.36;
+
+use lib qw(..);
+
+use Compress::Zlib qw(crc32);
+use Digest::SHA qw(sha256_hex);
+use File::Temp qw(tempdir);
+use FindBin;
+use IPC::Open3;
+use MIME::Base64 qw(encode_base64);
+use POSIX ();
+use Storable qw(dclone freeze);
+use Symbol qw(gensym);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Cluster;
+use PVE::Storage;
+use PVE::Storage::ZFSNVMePlugin;
+
+my $PLUGIN = 'PVE::Storage::ZFSNVMePlugin';
+my $ROOT = '/sys/kernel/config/nvmet';
+my $MARKER = 'ZFSNVME-CONFIGFS';
+my $MODEL = 'Proxmox ZFS NVMe';
+my $FIXTURES = "$FindBin::Bin/zfsnvme_fixtures";
+my $UUID_RE = qr/\A[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}\z/i;
+my $EMPTY_KEY_SHA = sha256_hex("\n");
+
+# The fixtures are the ZFS inventory and the configfs read of a target (Linux
+# 6.8 nvmet, ZFS 2.2) that serves one subsystem with two namespaces to two
+# cluster nodes, as the plugin reads them.
+my $FIXTURE_NQN = 'nqn.2026-01.com.example:zfsnvme';
+my @FIXTURE_HOSTS = (
+ 'nqn.2014-08.org.nvmexpress:uuid:0a0a0a0a-0a0a-4a0a-8a0a-0a0a0a0a0a0a',
+ 'nqn.2014-08.org.nvmexpress:uuid:0b0b0b0b-0b0b-4b0b-8b0b-0b0b0b0b0b0b',
+);
+my $FIXTURE_UUID5 = '0c0c0c0c-0c0c-4c0c-8c0c-0c0c0c0c0c0c';
+my $FIXTURE_UUID6 = '0d0d0d0d-0d0d-4d0d-8d0d-0d0d0d0d0d0d';
+
+my $NQN = 'nqn.2026-01.com.example:zfsnvme-test';
+my $FOREIGN_NQN = 'nqn.2026-01.com.example:foreign';
+my @HOSTS =
+ map { sprintf('nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-%012d', $_) } 1, 2;
+
+# A portal as parse_nvme_portals() returns it.
+sub portal($family, $address, $port) {
+ return { family => $family, address => $address, port => $port };
+}
+my $PORTALS = [portal('ipv4', '192.0.2.21', 4420), portal('ipv4', '192.0.2.22', 4420)];
+my %U = map {
+ my $d = $_;
+ ($d => join('-', $d x 8, $d x 4, '4' . $d x 3, '8' . $d x 3, $d x 12))
+} 1 .. 9;
+
+sub make_key($hash, $length, $seed = 1) {
+ my $raw = join('', map { chr(($_ * 131 + $seed * 17) % 256) } 1 .. $length);
+ return "DHHC-1:$hash:" . encode_base64($raw . pack('V', crc32($raw)), '') . ':';
+}
+my $KEY = make_key('00', 32);
+my $KEY_B = make_key('00', 32, 2);
+my $KEY_SHA = sha256_hex("$KEY\n");
+
+sub scfg(%override) {
+ return {
+ type => 'zfsnvme',
+ server => '192.0.2.10',
+ pool => 'tank',
+ subsysnqn => $NQN,
+ blocksize => '16k',
+ sparse => 1,
+ 'nvme-portals' => '192.0.2.21,192.0.2.22',
+ 'nvme-host-ifaces' => 'ens19,ens20',
+ 'nvme-host-nqns' => join(',', @HOSTS),
+ %override,
+ };
+}
+
+sub slurp($path) {
+ open(my $fh, '<', $path) or die "cannot open '$path': $!\n";
+ local $/;
+ return scalar(<$fh>);
+}
+
+# Calls a plugin package sub by name, so a mocked sub is the one called.
+sub nv($name, @args) {
+ my $code = $PLUGIN->can($name) // die "plugin has no sub '$name'\n";
+ return $code->(@args);
+}
+
+sub same($x, $y) {
+ local $Storable::canonical = 1;
+ return freeze([$x]) eq freeze([$y]);
+}
+
+# ---------------------------------------------------------------------------
+# Mocks: the only way to the target is _nvmet_run, and it reaches $FAKE.
+# The pmxcfs lock, quorum, clock, waits, UUID source, local files and local
+# devices are mocked. Waits advance $NOW.
+# ---------------------------------------------------------------------------
+
+our $FAKE;
+our $NOW = 1_000_000;
+our $SLEPT = 0;
+our $QUORATE = 1;
+our $LOCK_HOOK;
+our (%LOCK_HELD, @LOCKS, @NESTED_LOCKS, @QUORUM, @WARNINGS, @SYSFS_WRITES, %FILES, %CONFIG);
+our (%CORPUS, @VIOLATIONS, @WRITES, @UNLINKED, %BLOCK, @UUIDS);
+
+my $uuid_seq = 0;
+
+sub lock_id($scfg) {
+ return 'zfsnvme-' . ($scfg->{server} =~ s/[^A-Za-z0-9.-]/_/gr);
+}
+
+sub lock_held($scfg) {
+ return (($LOCK_HELD{ lock_id($scfg) } // 0) == $$) ? 1 : 0;
+}
+
+# Whether a step changes the target, by the vocabulary of the fake target,
+# independently of the plugin.
+sub mutating_step($step) {
+ return 1 if ref($step) ne 'ARRAY';
+ my ($command, $subcommand) = $step->@*;
+ return 0 if grep { $command eq $_ } qw(test grep cat printf env);
+ return 0 if $command eq 'zfs' && ($subcommand eq 'get' || $subcommand eq 'list');
+ return 1;
+}
+
+# Whether steps change the target, not counting the module load of a locked read.
+sub changing($steps) {
+ return scalar(grep { mutating_step($_) && !step_is($_, 'modprobe') } $steps->@*);
+}
+
+my $plugin_mock = Test::MockModule->new($PLUGIN);
+my $cluster_mock = Test::MockModule->new('PVE::Cluster');
+my $storage_mock = Test::MockModule->new('PVE::Storage');
+my $sysfs_mock = Test::MockModule->new('PVE::SysFSTools');
+my $tools_mock = Test::MockModule->new('PVE::Tools');
+
+$plugin_mock->redefine(
+ _nvmet_run => sub($scfg, $steps, %opts) {
+ die "test error: no fake target\n" if !$FAKE;
+ my $rendered = nv('_nvmet_render', $steps);
+ $CORPUS{$rendered} //= $opts{op} // '';
+ return $FAKE->run($scfg, $steps, $rendered, %opts);
+ },
+);
+# The target is the only place that runs commands, and only through _nvmet_run.
+# A command is also a violation, in case the caller handles the error.
+my $no_command = sub($cmd, %opts) {
+ my $what = 'unexpected command ' . (ref($cmd) ? $cmd->[0] : $cmd);
+ push @VIOLATIONS, $what;
+ die "test error: $what\n";
+};
+$plugin_mock->redefine(run_command => $no_command);
+$tools_mock->redefine(run_command => $no_command);
+$plugin_mock->redefine(_now => sub () { return $NOW });
+$plugin_mock->redefine(
+ _sleep => sub($seconds) {
+ $NOW += $seconds;
+ $SLEPT += $seconds;
+ return;
+ },
+);
+$plugin_mock->redefine(_block_device => sub($path) { return $BLOCK{$path} });
+$plugin_mock->redefine(
+ _unlink_file => sub($path) {
+ push @UNLINKED, $path;
+ delete $FILES{$path};
+ return 1;
+ },
+);
+$plugin_mock->redefine(log_warn => sub($message) { push @WARNINGS, $message; return });
+my $sysfs_write = sub($path, $data, $allow_existing = undef) {
+ push @SYSFS_WRITES, [$path, $data];
+ return 1;
+};
+$sysfs_mock->redefine(file_write => $sysfs_write);
+# The host side runs its kernel-facing steps in a bounded child; here they
+# run inline.
+$tools_mock->redefine(
+ run_fork_with_timeout => sub($timeout, $code, $opts = undef) {
+ my $res = $code->();
+ return wantarray ? ($res, 0) : $res;
+ },
+);
+# UUIDs come from @UUIDS first, then from a sequence.
+$plugin_mock->redefine(
+ file_read_firstline => sub($path) {
+ if ($path eq '/proc/sys/kernel/random/uuid') {
+ return shift(@UUIDS) if @UUIDS;
+ $uuid_seq++;
+ return sprintf('%08x-7e57-4000-8000-%012x', $uuid_seq, $uuid_seq);
+ }
+ return $FILES{$path};
+ },
+);
+$plugin_mock->redefine(
+ file_set_contents => sub($path, $data, $perm = undef, @rest) {
+ push @WRITES, { path => $path, data => $data, perm => $perm };
+ return;
+ },
+);
+$plugin_mock->redefine(make_path => sub(@args) { push @WRITES, { make_path => [@args] }; return });
+$plugin_mock->redefine(_namespace_openers => sub($nqn) { return [] });
+
+$cluster_mock->redefine(
+ check_cfs_quorum => sub($noerr = undef) {
+ push @QUORUM, $noerr;
+ die "cluster not ready - no quorum?\n" if !$QUORATE && !$noerr;
+ return $QUORATE;
+ },
+);
+# Keeps the cfs_lock contract: the code's result with $@ cleared, or undef
+# with $@ set. pmxcfs locks are not re-entrant: a second request for a held
+# lock times out, like a mkdir on /etc/pve/priv/lock would.
+$cluster_mock->redefine(
+ cfs_lock_domain => sub($name, $timeout, $code, @param) {
+ push @LOCKS, { name => $name, timeout => $timeout, pid => $$, now => $NOW };
+ if ($LOCK_HELD{$name}) {
+ push @NESTED_LOCKS, $name;
+ $@ = "cfs-lock 'domain-$name' error: got lock request timeout\n";
+ return undef;
+ }
+ if (my $hook = $LOCK_HOOK) {
+ local $LOCK_HOOK;
+ $hook->($name);
+ }
+ local $LOCK_HELD{$name} = $$;
+ my $res = eval { $code->(@param) };
+ my $err = $@;
+ if ($err) {
+ $@ = $err;
+ return undef;
+ }
+ $@ = undef;
+ return $res;
+ },
+);
+$storage_mock->redefine(config => sub () { return { ids => {%CONFIG} } });
+
+# ---------------------------------------------------------------------------
+# FakeTarget: ZFS datasets and an nvmet configfs tree with the kernel rules
+# the plugin depends on. It accepts only the command vocabulary of the
+# plugin; anything else is recorded as a protocol violation.
+# ---------------------------------------------------------------------------
+
+package FakeTarget {
+ use Digest::SHA qw(sha256_hex);
+ use Storable qw(dclone);
+
+ my $RR = quotemeta($ROOT);
+
+ # fail => [$match, $error, $times]: the steps that match (an argv prefix,
+ # { write => $suffix, value => $value } or a sub of the step) fail $times
+ # times with $error.
+ sub new($class, %opts) {
+ my $pools = delete($opts{pools}) // ['tank'];
+ if (my $fail = delete($opts{fail})) {
+ my ($match, $error, $times) = $fail->@*;
+ my $matches =
+ ref($match) eq 'CODE' ? $match
+ : ref($match) eq 'HASH'
+ ? sub($step) { main::write_to($step, $match->{write}, $match->{value}) }
+ : sub($step) { main::step_is($step, $match->@*) };
+ my $count = 0;
+ $opts{step_fault} = sub($step, $call) {
+ return undef if !$matches->($step) || $count >= ($times // 99);
+ $count++;
+ return $error;
+ };
+ }
+ my $self = bless {
+ m => {
+ ds => {},
+ snaps => {},
+ seq => 0,
+ subsystems => {},
+ ports => {},
+ hosts => {},
+ mounted => 1,
+ loaded => 1,
+ available => '1073741824',
+ used => '1048576',
+ },
+ calls => [],
+ violations => [],
+ faults => {}, # call index => before | after | late | cut:<step> | lost:<step>
+ late => [], # calls that ssh gave up on, applied by land()
+ # { after => call index, mode => before | unreachable | failed }
+ read_fault => undef,
+ step_fault => undef, # sub ($step, $call) returning an error text
+ before_call => undef,
+ after_call => undef,
+ udev_delay => 0, # failing `test -b` polls after a zvol (re)appears
+ unbindable => {}, # port id => 1: its address is missing on the target
+ unreachable => 0,
+ secrets => [$KEY, $KEY_B],
+ %opts,
+ }, $class;
+ $self->{m}->{ds}->{$_} = { type => 'filesystem', props => {} } for $pools->@*;
+ return $self;
+ }
+
+ sub from_state($class, $state, %opts) {
+ my $self = $class->new(%opts);
+ $self->{m} = dclone($state);
+ return $self;
+ }
+
+ # --- building a state -------------------------------------------------
+
+ sub add_zvol($self, $name, %o) {
+ my %props;
+ @props{qw(proxmox:nvme-subsys proxmox:nvme-nsid proxmox:nvme-uuid)} = $o{identity}->@*
+ if $o{identity};
+ $self->{m}->{ds}->{$name} = {
+ type => 'volume',
+ props => \%props,
+ volsize => $o{volsize} // 1073741824,
+ origin => $o{origin} // '-',
+ devwait => 0,
+ };
+ return $self;
+ }
+
+ sub add_snapshot($self, $snapshot, %o) {
+ my ($name) = split /\@/, $snapshot;
+ my $ds = $self->{m}->{ds}->{$name};
+ $self->{m}->{snaps}->{$snapshot} = {
+ props => $o{props} // dclone($ds->{props}),
+ volsize => $ds->{volsize},
+ seq => ++$self->{m}->{seq},
+ };
+ return $self;
+ }
+
+ sub add_subsystem($self, $nqn, %o) {
+ $self->{m}->{subsystems}->{$nqn} = {
+ attr => {
+ attr_model => $o{model} // $MODEL,
+ attr_serial => $o{serial} // PVE::Storage::ZFSNVMePlugin::_nvmet_serial($nqn),
+ attr_allow_any_host => $o{allow_any_host} // '0',
+ },
+ acl => { map { $_ => 1 } ($o{acl} // [])->@* },
+ ns => {},
+ };
+ return $self;
+ }
+
+ sub add_namespace($self, $nqn, $nsid, $uuid, $dev, $enable = 1) {
+ $self->{m}->{subsystems}->{$nqn}->{ns}->{$nsid} = {
+ enable => "$enable",
+ device_path => $dev,
+ device_uuid => $uuid,
+ buffered_io => '0',
+ };
+ return $self;
+ }
+
+ sub add_port($self, $id, $address, $service, %o) {
+ $self->{m}->{ports}->{$id} = {
+ attr => {
+ addr_trtype => 'tcp',
+ addr_adrfam => $o{family} // 'ipv4',
+ addr_traddr => $address,
+ addr_trsvcid => "$service",
+ },
+ links => { map { $_ => 1 } ($o{links} // [])->@* },
+ };
+ return $self;
+ }
+
+ # nvmet creates the key attributes of a host world-readable
+ sub host_entry($key = undef) {
+ return { key => $key, mode => { map { $_ => '0644' } qw(dhchap_key dhchap_ctrl_key) } };
+ }
+
+ sub add_host($self, $hostnqn, $key = undef) {
+ $self->{m}->{hosts}->{$hostnqn} = host_entry($key);
+ return $self;
+ }
+
+ # A target restart: configfs is empty and, unless $loaded, nvmet is not
+ # loaded yet.
+ sub reboot($self, $loaded = 0) {
+ my $m = $self->{m};
+ $m->{$_} = {} for qw(subsystems ports hosts);
+ $m->{loaded} = $loaded;
+ $_->{devwait} = 0 for grep { $_->{type} eq 'volume' } values $m->{ds}->%*;
+ return $self;
+ }
+
+ # --- execution --------------------------------------------------------
+
+ sub run($self, $scfg, $steps, $rendered, %opts) {
+ my $call = {
+ index => scalar($self->{calls}->@*),
+ op => $opts{op},
+ steps => dclone($steps),
+ rendered => $rendered,
+ timeout => $opts{timeout},
+ input => defined($opts{input}) ? 1 : 0,
+ locked => main::lock_held($scfg),
+ mutating => (grep { main::mutating_step($_) } $steps->@*) ? 1 : 0,
+ changing => main::changing($steps),
+ now => $main::NOW,
+ };
+ push $self->{calls}->@*, $call;
+ $self->audit($call, $opts{input});
+ $self->{before_call}->($self, $call) if $self->{before_call};
+ my $res = $self->execute($call, $steps, $opts{input});
+ $call->{rc} = $res->{rc};
+ $call->{err} = $res->{err};
+ $self->audit_text($call, $res->{err}, $res->{out}->@*);
+ $self->{after_call}->($self, $call) if $self->{after_call};
+ return $res;
+ }
+
+ # Violations are recorded for the fake and for the whole run, which the
+ # last subtest checks.
+ sub flag($self, @what) {
+ push $self->{violations}->@*, @what;
+ push @main::VIOLATIONS, @what;
+ }
+
+ sub violation($self, $what) {
+ $self->flag($what);
+ die "FAKEERR:protocol violation: $what\n";
+ }
+
+ sub err($message) {
+ die "FAKEERR:$message\n";
+ }
+
+ sub audit($self, $call, $input) {
+ my @bad;
+ my $op = $call->{op} // '';
+ push @bad, 'call without an operation label' if $op eq '';
+ for my $secret ($self->{secrets}->@*) {
+ push @bad, "key in the command of '$op'" if index($call->{rendered}, $secret) >= 0;
+ }
+ my $keys = grep { ref($_) eq 'HASH' && exists($_->{key}) } $call->{steps}->@*;
+ push @bad, "key write without stdin in '$op'" if $keys && !defined($input);
+ push @bad, "stdin without a key write in '$op'" if defined($input) && !$keys;
+ push @bad, "stdin is not a configured key in '$op'"
+ if defined($input) && !grep { $input eq "$_\n" } $self->{secrets}->@*;
+ push @bad, "target change without the target lock in '$op'"
+ if $call->{mutating} && !$call->{locked};
+ $self->flag(@bad);
+ }
+
+ sub audit_text($self, $call, @texts) {
+ for my $text (grep { defined } @texts) {
+ for my $secret ($self->{secrets}->@*) {
+ $self->flag("key in the output of '$call->{op}'") if index($text, $secret) >= 0;
+ }
+ }
+ }
+
+ # Applies the calls that ssh gave up on, as the target finally runs them:
+ # each chain stops at its first failing step.
+ sub land($self) {
+ for my $late (splice($self->{late}->@*)) {
+ for my $step ($late->{steps}->@*) {
+ last if !eval { $self->step($step, $late->{input}); 1 };
+ }
+ }
+ return $self;
+ }
+
+ # A fault mode fails a call before it runs (rc 1), after it ran (the
+ # connection breaks: 255), at one of its steps (cut: the connection breaks
+ # and the remote shell stops; lost: it breaks and the rest of the chain
+ # still runs, later), or ssh gives up on it while it runs to its end later
+ # (late: -1).
+ sub execute($self, $call, $steps, $input) {
+ my $closed = 'Connection to 192.0.2.10 closed by remote host.';
+ my $broken = { rc => 255, out => [], err => $closed };
+ my $no_route = 'ssh: connect to host 192.0.2.10: No route to host';
+ return { rc => 255, out => [], err => $no_route } if $self->{unreachable};
+ my $read_fault = $self->{read_fault};
+ if ($read_fault && !$call->{mutating} && $call->{index} > $read_fault->{after}) {
+ $self->{read_fault} = undef;
+ $call->{fault} = "read $read_fault->{mode}";
+ return { rc => 255, out => [], err => $no_route }
+ if $read_fault->{mode} eq 'unreachable';
+ return { rc => -1, out => [], err => 'ssh failed' } if $read_fault->{mode} eq 'failed';
+ return { rc => 1, out => [], err => 'injected read failure' };
+ }
+ my $mode = $self->{faults}->{ $call->{index} } // '';
+ $call->{fault} = $mode if $mode;
+ return { rc => 1, out => [], err => 'injected failure' } if $mode eq 'before';
+ if ($mode eq 'late') {
+ push $self->{late}->@*, { steps => dclone($steps), input => $input };
+ return { rc => -1, out => [], err => 'timeout' };
+ }
+ my ($cut) = $mode =~ /\Acut:([0-9]+)\z/;
+ my ($lost) = $mode =~ /\Alost:([0-9]+)\z/;
+ my @out;
+ for my $index (0 .. $steps->$#*) {
+ return $broken if defined($cut) && $index == $cut;
+ if (defined($lost) && $index == $lost) {
+ my @rest = $steps->@[$index .. $steps->$#*];
+ push $self->{late}->@*, { steps => dclone(\@rest), input => $input };
+ return $broken;
+ }
+ my $step = $steps->[$index];
+ my $error = $self->{step_fault} ? $self->{step_fault}->($step, $call) : undef;
+ if (!defined($error)) {
+ my @lines = eval { $self->step($step, $input) };
+ my $e = $@;
+ if (!$e) {
+ push @out, @lines;
+ next;
+ }
+ if ($e !~ s/\AFAKEERR://) {
+ $self->flag("fake target error: $e");
+ return { rc => 2, out => \@out, err => 'fake target error' };
+ }
+ chomp($error = $e);
+ }
+ return { rc => 1, out => \@out, err => $error eq '' ? 'exit code 1' : $error };
+ }
+ return $broken if $mode eq 'after';
+ return { rc => 0, out => \@out, err => '' };
+ }
+
+ sub step($self, $step, $input) {
+ if (ref($step) eq 'HASH' && exists($step->{write})) {
+ $self->write_attr($step->{write}, $step->{value});
+ return ();
+ }
+ if (ref($step) eq 'HASH') {
+ err('dd: error reading standard input') if !defined($input);
+ (my $key = $input) =~ s/\n\z//;
+ my @failed;
+ for my $path ($step->{key}->@*) {
+ eval { $self->write_key($path, $key) };
+ push @failed, $@ if $@;
+ }
+ die $failed[0] if @failed;
+ return ();
+ }
+ my ($command, @args) = $step->@*;
+ my $method = "cmd_$command";
+ $self->violation("command '$command'") if !$self->can($method);
+ return $self->$method(@args);
+ }
+
+ # --- configfs ---------------------------------------------------------
+
+ sub available($self) {
+ err("find: '$ROOT': No such file or directory")
+ if !$self->{m}->{mounted} || !$self->{m}->{loaded};
+ }
+
+ sub parts($self, $path) {
+ return () if $path !~ m{\A$RR/(.+)\z};
+ return split m{/}, $1;
+ }
+
+ sub zvol_name($self, $dev) {
+ return $dev =~ m{\A/dev/zvol/(.+)\z} ? $1 : undef;
+ }
+
+ sub block_device($self, $dev, $poll) {
+ my $name = $self->zvol_name($dev) // return 0;
+ my $ds = $self->{m}->{ds}->{$name} // return 0;
+ return 0 if $ds->{type} ne 'volume';
+ if ($ds->{devwait} > 0) {
+ $ds->{devwait}-- if $poll;
+ return 0;
+ }
+ return 1;
+ }
+
+ sub busy($self, $name) {
+ for my $subsys (values $self->{m}->{subsystems}->%*) {
+ return 1
+ if grep { $_->{enable} eq '1' && $_->{device_path} eq "/dev/zvol/$name" }
+ values $subsys->{ns}->%*;
+ }
+ return 0;
+ }
+
+ sub new_uuid($self) {
+ my $seq = ++$self->{m}->{seq};
+ return sprintf('%08x-dead-4000-8000-%012x', $seq, $seq);
+ }
+
+ sub write_attr($self, $path, $value) {
+ $self->available;
+ my @p = $self->parts($path) or $self->violation("write '$path'");
+ my $m = $self->{m};
+ if ($p[0] eq 'subsystems' && @p == 3) {
+ my $subsys = $m->{subsystems}->{ $p[1] } // err("$path: No such file or directory");
+ $self->violation("write '$path'")
+ if $p[2] !~ /\A(?:attr_model|attr_serial|attr_allow_any_host)\z/;
+ if ($p[2] eq 'attr_allow_any_host') {
+ err("$path: Invalid argument") if $value !~ /\A[01]\z/;
+ err("$path: Invalid argument") if $value eq '1' && $subsys->{acl}->%*;
+ }
+ $subsys->{attr}->{ $p[2] } = "$value";
+ } elsif ($p[0] eq 'subsystems' && @p == 5 && $p[2] eq 'namespaces') {
+ my $subsys = $m->{subsystems}->{ $p[1] } // err("$path: No such file or directory");
+ my $ns = $subsys->{ns}->{ $p[3] } // err("$path: No such file or directory");
+ my $attr = $p[4];
+ if ($attr eq 'enable') {
+ err("$path: Invalid argument") if $value !~ /\A[01]\z/;
+ err("$path: No such device")
+ if $value eq '1'
+ && $ns->{enable} ne '1'
+ && !$self->block_device($ns->{device_path}, 0);
+ $ns->{enable} = "$value";
+ } elsif ($attr =~ /\A(?:device_path|device_uuid|buffered_io)\z/) {
+ err("$path: Device or resource busy") if $ns->{enable} eq '1';
+ if ($attr eq 'device_uuid') {
+ err("$path: Invalid argument") if $value !~ $UUID_RE;
+ $value = lc($value);
+ }
+ $ns->{$attr} = "$value";
+ } elsif ($attr eq 'revalidate_size') {
+ err("$path: Invalid argument") if $ns->{enable} ne '1' || $value ne '1';
+ $self->{revalidated}->{"$p[1]/$p[3]"}++;
+ } else {
+ $self->violation("write '$path'");
+ }
+ } elsif ($p[0] eq 'ports' && @p == 3) {
+ my $port = $m->{ports}->{ $p[1] } // err("$path: No such file or directory");
+ $self->violation("write '$path'")
+ if $p[2] !~ /\Aaddr_(?:trtype|adrfam|traddr|trsvcid)\z/;
+ err("$path: Permission denied") if $port->{links}->%*;
+ $port->{attr}->{ $p[2] } = "$value";
+ } else {
+ $self->violation("write '$path'");
+ }
+ }
+
+ sub write_key($self, $path, $key) {
+ $self->available;
+ my @p = $self->parts($path);
+ $self->violation("key write to '$path'")
+ if @p != 3 || $p[0] ne 'hosts' || $p[2] ne 'dhchap_key';
+ my $host = $self->{m}->{hosts}->{ $p[1] } // err("tee: $path: No such file or directory");
+ err("tee: $path: Invalid argument") if $key !~ /\ADHHC-1:/;
+ $self->violation("key write to the world-readable '$path'")
+ if ($host->{mode}->{ $p[2] } // '') ne '0600';
+ $host->{key} = $key;
+ }
+
+ sub cmd_mkdir($self, @paths) {
+ $self->available;
+ my $m = $self->{m};
+ for my $path (@paths) {
+ my @p = $self->parts($path) or $self->violation("mkdir '$path'");
+ my $exists = "mkdir: cannot create directory '$path': File exists";
+ my $missing = "mkdir: cannot create directory '$path': No such file or directory";
+ if ($p[0] eq 'subsystems' && @p == 2) {
+ err($exists) if $m->{subsystems}->{ $p[1] };
+ $m->{subsystems}->{ $p[1] } = {
+ attr => {
+ attr_model => 'Linux',
+ attr_serial => substr(sha256_hex("serial $p[1]"), 0, 16),
+ attr_allow_any_host => '0',
+ },
+ acl => {},
+ ns => {},
+ };
+ } elsif ($p[0] eq 'subsystems' && @p == 4 && $p[2] eq 'namespaces') {
+ my $subsys = $m->{subsystems}->{ $p[1] } // err($missing);
+ err("mkdir: cannot create directory '$path': Invalid argument")
+ if $p[3] !~ /\A[0-9]+\z/ || $p[3] == 0 || $p[3] > 0xfffffffe;
+ err($exists) if $subsys->{ns}->{ $p[3] };
+ $subsys->{ns}->{ $p[3] } = {
+ enable => '0',
+ device_path => '(null)',
+ device_uuid => $self->new_uuid,
+ buffered_io => '0',
+ };
+ } elsif ($p[0] eq 'ports' && @p == 2 && $p[1] =~ /\A[0-9]+\z/) {
+ err($exists) if $m->{ports}->{ $p[1] };
+ $m->{ports}->{ $p[1] } = {
+ attr => { map { ("addr_$_" => '') } qw(trtype adrfam traddr trsvcid) },
+ links => {},
+ };
+ } elsif ($p[0] eq 'hosts' && @p == 2) {
+ err($exists) if $m->{hosts}->{ $p[1] };
+ $m->{hosts}->{ $p[1] } = host_entry();
+ } else {
+ $self->violation("mkdir '$path'");
+ }
+ }
+ return ();
+ }
+
+ sub cmd_chmod($self, $mode, @paths) {
+ $self->available;
+ $self->violation("chmod '$mode'") if $mode ne '0600';
+ for my $path (@paths) {
+ my @p = $self->parts($path);
+ $self->violation("chmod '$path'")
+ if @p != 3 || $p[0] ne 'hosts' || $p[2] !~ /\Adhchap_(?:ctrl_)?key\z/;
+ my $host = $self->{m}->{hosts}->{ $p[1] }
+ // err("chmod: cannot access '$path': No such file or directory");
+ $host->{mode}->{ $p[2] } = $mode;
+ }
+ return ();
+ }
+
+ sub cmd_rmdir($self, @paths) {
+ $self->available;
+ my $m = $self->{m};
+ my @errors;
+ for my $path (@paths) {
+ my @p = $self->parts($path) or $self->violation("rmdir '$path'");
+ my $fail = sub($why) { push @errors, "rmdir: failed to remove '$path': $why" };
+ if ($p[0] eq 'subsystems' && @p == 4 && $p[2] eq 'namespaces') {
+ my $subsys = $m->{subsystems}->{ $p[1] };
+ $fail->('No such file or directory')
+ if !$subsys || !delete($subsys->{ns}->{ $p[3] });
+ } elsif ($p[0] eq 'subsystems' && @p == 2) {
+ my $subsys = $m->{subsystems}->{ $p[1] };
+ if (!$subsys) {
+ $fail->('No such file or directory');
+ } elsif ($subsys->{ns}->%* || $subsys->{acl}->%*) {
+ $fail->('Directory not empty');
+ } elsif (grep { $_->{links}->{ $p[1] } } values $m->{ports}->%*) {
+ $fail->('Device or resource busy');
+ } else {
+ delete $m->{subsystems}->{ $p[1] };
+ }
+ } elsif ($p[0] eq 'hosts' && @p == 2) {
+ if (!$m->{hosts}->{ $p[1] }) {
+ $fail->('No such file or directory');
+ } elsif (grep { $_->{acl}->{ $p[1] } } values $m->{subsystems}->%*) {
+ $fail->('Device or resource busy');
+ } else {
+ delete $m->{hosts}->{ $p[1] };
+ }
+ } else {
+ $self->violation("rmdir '$path'");
+ }
+ }
+ err($errors[0]) if @errors;
+ return ();
+ }
+
+ sub cmd_rm($self, @paths) {
+ $self->available;
+ my $m = $self->{m};
+ my @errors;
+ for my $path (@paths) {
+ my @p = $self->parts($path);
+ my $missing = "rm: cannot remove '$path': No such file or directory";
+ if (@p == 4 && $p[0] eq 'ports' && $p[2] eq 'subsystems') {
+ my $port = $m->{ports}->{ $p[1] };
+ push @errors, $missing if !$port || !delete($port->{links}->{ $p[3] });
+ } elsif (@p == 4 && $p[0] eq 'subsystems' && $p[2] eq 'allowed_hosts') {
+ my $subsys = $m->{subsystems}->{ $p[1] };
+ push @errors, $missing if !$subsys || !delete($subsys->{acl}->{ $p[3] });
+ } else {
+ $self->violation("rm '$path'");
+ }
+ }
+ err($errors[0]) if @errors;
+ return ();
+ }
+
+ sub cmd_ln($self, $flag, $target, $link) {
+ $self->available;
+ my $m = $self->{m};
+ my @t = $self->parts($target);
+ my @l = $self->parts($link);
+ my $fail = sub($why) { err("ln: failed to create symbolic link '$link': $why") };
+ if (@l == 4 && $l[0] eq 'ports' && $l[2] eq 'subsystems') {
+ $self->violation("ln '$target' '$link'")
+ if @t != 2 || $t[0] ne 'subsystems' || $t[1] ne $l[3];
+ my $port = $m->{ports}->{ $l[1] } // $fail->('No such file or directory');
+ $fail->('No such file or directory') if !$m->{subsystems}->{ $t[1] };
+ $fail->('File exists') if $port->{links}->{ $l[3] };
+ my $a = $port->{attr};
+ $fail->('Invalid argument')
+ if $a->{addr_trtype} ne 'tcp'
+ || $a->{addr_adrfam} !~ /\Aipv[46]\z/
+ || $a->{addr_traddr} eq ''
+ || $a->{addr_trsvcid} eq '';
+ if (!$port->{links}->%*) { # the first link enables the port
+ $fail->('Cannot assign requested address') if $self->{unbindable}->{ $l[1] };
+ my @fields = qw(addr_trtype addr_adrfam addr_traddr addr_trsvcid);
+ my $address = join(' ', $a->@{@fields});
+ for my $id (keys $m->{ports}->%*) {
+ my $other = $m->{ports}->{$id};
+ next if $id eq $l[1] || !$other->{links}->%*;
+ $fail->('Address already in use')
+ if join(' ', $other->{attr}->@{@fields}) eq $address;
+ }
+ }
+ $port->{links}->{ $l[3] } = 1;
+ } elsif (@l == 4 && $l[0] eq 'subsystems' && $l[2] eq 'allowed_hosts') {
+ $self->violation("ln '$target' '$link'")
+ if @t != 2 || $t[0] ne 'hosts' || $t[1] ne $l[3];
+ my $subsys = $m->{subsystems}->{ $l[1] } // $fail->('No such file or directory');
+ $fail->('No such file or directory') if !$m->{hosts}->{ $t[1] };
+ $fail->('Invalid argument') if $subsys->{attr}->{attr_allow_any_host} eq '1';
+ $fail->('File exists') if $subsys->{acl}->{ $l[3] };
+ $subsys->{acl}->{ $l[3] } = 1;
+ } else {
+ $self->violation("ln '$target' '$link'");
+ }
+ return ();
+ }
+
+ sub cmd_test($self, $flag, $path) {
+ if ($flag eq '-b') {
+ err('') if !$self->block_device($path, 1);
+ return ();
+ }
+ $self->available;
+ my $m = $self->{m};
+ my @p = $self->parts($path);
+ my $found;
+ if (@p == 4 && $p[0] eq 'subsystems' && $p[2] eq 'allowed_hosts') {
+ $found = ($m->{subsystems}->{ $p[1] } // {})->{acl}->{ $p[3] };
+ } elsif (@p == 4 && $p[0] eq 'ports' && $p[2] eq 'subsystems') {
+ $found = ($m->{ports}->{ $p[1] } // {})->{links}->{ $p[3] };
+ } else {
+ $self->violation("test $flag '$path'");
+ }
+ err('') if !$found;
+ return ();
+ }
+
+ sub cmd_grep($self, $flag, $value, $path) {
+ $self->available;
+ my @p = $self->parts($path);
+ $self->violation("grep '$path'")
+ if @p != 3 || $p[0] ne 'subsystems' || $p[2] ne 'attr_allow_any_host';
+ my $subsys = $self->{m}->{subsystems}->{ $p[1] }
+ // err("grep: $path: No such file or directory");
+ err('') if $subsys->{attr}->{attr_allow_any_host} ne $value;
+ return ();
+ }
+
+ sub cmd_cat($self, $path) {
+ $self->violation("cat '$path'") if $path ne '/proc/mounts';
+ return (
+ 'sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0',
+ 'proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0',
+ (
+ $self->{m}->{mounted}
+ ? ('configfs /sys/kernel/config configfs rw,relatime 0 0')
+ : ()
+ ),
+ );
+ }
+
+ sub cmd_mount($self, @args) {
+ err('mount: /sys/kernel/config: none already mounted on /sys/kernel/config.')
+ if $self->{m}->{mounted};
+ $self->{m}->{mounted} = 1;
+ return ();
+ }
+
+ sub cmd_modprobe($self, $module) {
+ $self->violation("modprobe '$module'") if $module ne 'nvmet_tcp';
+ $self->{m}->{loaded} = 1;
+ return ();
+ }
+
+ sub cmd_printf($self, @args) {
+ $self->violation('printf') if @args != 2 || $args[0] ne '%s\n' || $args[1] ne $MARKER;
+ return ($MARKER);
+ }
+
+ # The configfs read; 'golden commands' pins its argv.
+ sub cmd_env($self, @args) {
+ my @hosts = map { $args[$_ + 1] =~ m{\A$RR/hosts/([^/]+)/dhchap_key\z} ? ($1) : () }
+ grep { $args[$_] eq '-path' } 0 .. $#args;
+ $self->available;
+ return $self->configfs_lines(\@hosts);
+ }
+
+ sub configfs_lines($self, $hosts) {
+ my $m = $self->{m};
+ my (@dirs, @links, @attrs, @digests);
+ push @dirs, "D $ROOT", map { "D $ROOT/$_" } qw(hosts ports subsystems);
+ push @dirs, map { "D $ROOT/hosts/$_" } sort keys $m->{hosts}->%*;
+ for my $id (sort { $a <=> $b } keys $m->{ports}->%*) {
+ my $port = $m->{ports}->{$id};
+ push @dirs, "D $ROOT/ports/$id", "D $ROOT/ports/$id/subsystems";
+ push @links, map { "L $ROOT/ports/$id/subsystems/$_" } sort keys $port->{links}->%*;
+ push @attrs,
+ map { "$ROOT/ports/$id/$_:$port->{attr}->{$_}" } sort keys $port->{attr}->%*;
+ }
+ for my $nqn (sort keys $m->{subsystems}->%*) {
+ my $subsys = $m->{subsystems}->{$nqn};
+ my $S = "$ROOT/subsystems/$nqn";
+ push @dirs, "D $S", "D $S/allowed_hosts", "D $S/namespaces";
+ push @links, map { "L $S/allowed_hosts/$_" } sort keys $subsys->{acl}->%*;
+ push @attrs, map { "$S/$_:$subsys->{attr}->{$_}" } sort keys $subsys->{attr}->%*;
+ for my $id (sort { $a <=> $b } keys $subsys->{ns}->%*) {
+ my $ns = $subsys->{ns}->{$id};
+ push @dirs, "D $S/namespaces/$id";
+ push @attrs,
+ map { "$S/namespaces/$id/$_:$ns->{$_}" }
+ qw(buffered_io enable device_uuid device_path);
+ }
+ }
+ for my $hostnqn ($hosts->@*) {
+ my $host = $m->{hosts}->{$hostnqn} // next;
+ push @digests,
+ sha256_hex(($host->{key} // '') . "\n") . " $ROOT/hosts/$hostnqn/dhchap_key";
+ }
+ return (@dirs, @links, @attrs, @digests);
+ }
+
+ # --- zfs --------------------------------------------------------------
+
+ sub cmd_zfs($self, $subcommand, @args) {
+ my $method = "zfs_$subcommand";
+ $self->violation("zfs $subcommand") if !$self->can($method);
+ return $self->$method(@args);
+ }
+
+ # Options with values are listed with 1, flags with 0.
+ sub options($self, $args, %spec) {
+ my %o;
+ my @rest = $args->@*;
+ while (@rest && $rest[0] =~ /\A-/) {
+ my $option = shift @rest;
+ if ($option eq '-Hp') {
+ $o{H} = $o{p} = 1;
+ next;
+ }
+ if ($option =~ /\A-d([0-9]+)\z/) {
+ $o{d} = [$1];
+ next;
+ }
+ my $name = substr($option, 1);
+ $self->violation("zfs option '$option'") if !exists($spec{$name});
+ if ($spec{$name}) {
+ push $o{$name}->@*, shift(@rest);
+ } else {
+ $o{$name} = 1;
+ }
+ }
+ return (\%o, @rest);
+ }
+
+ sub properties_of($self, $assignments) {
+ my %props;
+ for my $assignment (($assignments // [])->@*) {
+ $self->violation("zfs property '$assignment'") if $assignment !~ /\A([^=]+)=(.*)\z/;
+ $props{$1} = $2;
+ }
+ return \%props;
+ }
+
+ sub datasets_under($self, $target, $depth) {
+ my $m = $self->{m};
+ return () if !$m->{ds}->{$target};
+ return sort grep {
+ my $name = $_;
+ $name eq $target
+ || (index($name, "$target/") == 0
+ && (() = substr($name, length($target)) =~ m{/}g) <= $depth)
+ } keys $m->{ds}->%*;
+ }
+
+ sub property($self, $name, $prop) {
+ my $m = $self->{m};
+ my $ds = $m->{ds}->{$name};
+ return ($ds->{type}, '-') if $prop eq 'type';
+ if ($prop =~ /:/) {
+ return ($ds->{props}->{$prop}, 'local') if defined($ds->{props}->{$prop});
+ my $parent = $name;
+ while ($parent =~ s{/[^/]+\z}{}) {
+ my $value = ($m->{ds}->{$parent} // last)->{props}->{$prop};
+ return ($value, "inherited from $parent") if defined($value);
+ }
+ return ('-', '-');
+ }
+ return ($m->{available}, '-') if $prop eq 'available';
+ return ($m->{used}, '-') if $prop eq 'used';
+ return ($ds->{volsize} // '-', '-') if $prop eq 'volsize';
+ return ('4096', '-') if $prop eq 'usedbydataset';
+ $self->violation("zfs property '$prop'");
+ }
+
+ sub zfs_get($self, @args) {
+ my ($o, $props, @targets) = $self->options(\@args, H => 0, p => 0, d => 1, t => 1, o => 1);
+ $self->violation('zfs get') if @targets != 1 || !$o->{H};
+ $self->violation('zfs get inventory')
+ if $props =~ /\Atype,proxmox:/
+ && !main::same(['get', @args], main::expected_inventory_read($targets[0]));
+ my $target = $targets[0];
+ my @fields = split /,/, ($o->{o} // ['name,property,value,source'])->[0];
+ my %types = map { $_ => 1 } split /,/, ($o->{t} // ['filesystem,volume'])->[0];
+ my @datasets = $self->datasets_under($target, $o->{d} ? $o->{d}->[0] : 0)
+ or err("cannot open '$target': dataset does not exist");
+ my @out;
+
+ for my $name (grep { $types{ $self->{m}->{ds}->{$_}->{type} } } @datasets) {
+ for my $prop (split /,/, $props) {
+ my ($value, $source) = $self->property($name, $prop);
+ my %row = (name => $name, property => $prop, value => $value, source => $source);
+ push @out, join("\t", map { $row{$_} } @fields);
+ }
+ }
+ return @out;
+ }
+
+ sub list_field($self, $row, $field) {
+ my $m = $self->{m};
+ return $row if $field eq 'name';
+ if (my $snap = $m->{snaps}->{$row}) {
+ return 1000 + $snap->{seq} if $field eq 'guid';
+ return 1700000000 + $snap->{seq} if $field eq 'creation';
+ $self->violation("zfs list snapshot field '$field'");
+ }
+ my $ds = $m->{ds}->{$row};
+ return $ds->{type} eq 'volume' ? $ds->{volsize} : '-' if $field eq 'volsize';
+ return $ds->{origin} // '-' if $field eq 'origin';
+ return $ds->{type} if $field eq 'type';
+ $self->violation("zfs list field '$field'");
+ }
+
+ sub zfs_list($self, @args) {
+ my ($o, @targets) =
+ $self->options(\@args, H => 0, p => 0, r => 0, d => 1, t => 1, o => 1, s => 1);
+ $self->violation('zfs list') if @targets != 1 || !$o->{H};
+ my $m = $self->{m};
+ my $target = $targets[0];
+ my @fields = split /,/, ($o->{o} // ['name'])->[0];
+ my %types = map { $_ => 1 } split /,/, ($o->{t} // ['filesystem,volume'])->[0];
+ my @rows;
+ if ($target =~ /\@/) {
+ err("cannot open '$target': dataset does not exist") if !$m->{snaps}->{$target};
+ @rows = ($target);
+ } else {
+ my $depth = $o->{d} ? $o->{d}->[0] : $o->{r} ? 1000 : 0;
+ my @datasets = $self->datasets_under($target, $depth)
+ or err("cannot open '$target': dataset does not exist");
+ if ($types{snapshot}) {
+ my %under = map { $_ => 1 } @datasets;
+ push @rows, sort { $m->{snaps}->{$a}->{seq} <=> $m->{snaps}->{$b}->{seq} }
+ grep { $under{ (split /\@/)[0] } } keys $m->{snaps}->%*;
+ }
+ push @rows, grep { $types{ $m->{ds}->{$_}->{type} } } @datasets;
+ }
+ return map {
+ my $row = $_;
+ join("\t", map { $self->list_field($row, $_) } @fields)
+ } @rows;
+ }
+
+ sub zfs_create($self, @args) {
+ my ($o, @targets) = $self->options(\@args, s => 0, b => 1, o => 1, V => 1);
+ $self->violation('zfs create') if @targets != 1 || !$o->{V};
+ my $m = $self->{m};
+ my $name = $targets[0];
+ my ($parent) = $name =~ m{\A(.+)/[^/]+\z} or $self->violation('zfs create name');
+ $self->violation('zfs create size') if $o->{V}->[0] !~ /\A([0-9]+)k\z/;
+ my $size = $1 * 1024;
+ err("cannot create '$name': parent does not exist") if !$m->{ds}->{$parent};
+ err("cannot create '$name': dataset already exists") if $m->{ds}->{$name};
+ $m->{ds}->{$name} = {
+ type => 'volume',
+ props => $self->properties_of($o->{o}),
+ volsize => $size,
+ origin => '-',
+ devwait => $self->{udev_delay},
+ sparse => $o->{s} ? 1 : 0,
+ blocksize => $o->{b} ? $o->{b}->[0] : undef,
+ };
+ return ();
+ }
+
+ sub zfs_clone($self, @args) {
+ my ($o, $origin, $name, @extra) = $self->options(\@args, o => 1);
+ $self->violation('zfs clone') if @extra || !defined($name);
+ my $m = $self->{m};
+ my $snap = $m->{snaps}->{$origin} // err("cannot open '$origin': dataset does not exist");
+ err("cannot create '$name': dataset already exists") if $m->{ds}->{$name};
+ $m->{ds}->{$name} = {
+ type => 'volume',
+ props => $self->properties_of($o->{o}),
+ volsize => $snap->{volsize},
+ origin => $origin,
+ devwait => $self->{udev_delay},
+ };
+ return ();
+ }
+
+ sub zfs_destroy($self, @args) {
+ my ($o, $target, @extra) = $self->options(\@args, r => 0);
+ $self->violation('zfs destroy') if @extra || !defined($target);
+ my $m = $self->{m};
+ my $clones = sub($snapshot) {
+ return grep { ($_->{origin} // '-') eq $snapshot } values $m->{ds}->%*;
+ };
+ if ($target =~ /\@/) {
+ $self->violation('zfs destroy -r of a snapshot') if $o->{r};
+ err("could not find any snapshots to destroy; check snapshot names.")
+ if !$m->{snaps}->{$target};
+ err("cannot destroy snapshot $target: snapshot has dependent clones")
+ if $clones->($target);
+ delete $m->{snaps}->{$target};
+ return ();
+ }
+ $self->violation('zfs destroy without -r') if !$o->{r};
+ err("cannot open '$target': dataset does not exist") if !$m->{ds}->{$target};
+ err("cannot destroy '$target': dataset is busy") if $self->busy($target);
+ my @snaps = grep { index($_, "$target\@") == 0 } keys $m->{snaps}->%*;
+ err("cannot destroy '$target': filesystem has dependent clones")
+ if grep { $clones->($_) } @snaps;
+ delete $m->{snaps}->{$_} for @snaps;
+ delete $m->{ds}->{$target};
+ return ();
+ }
+
+ # Stricter than ZFS: user properties revert to their value at snapshot
+ # time, so the plugin must set the identity again after a rollback.
+ sub zfs_rollback($self, @args) {
+ $self->violation('zfs rollback') if @args != 1 || $args[0] !~ /\@/;
+ my $m = $self->{m};
+ my ($name) = split /\@/, $args[0];
+ my $snap = $m->{snaps}->{ $args[0] }
+ // err("cannot open '$args[0]': dataset does not exist");
+ err("cannot rollback to '$args[0]': more recent snapshots or bookmarks exist")
+ if grep { index($_, "$name\@") == 0 && $m->{snaps}->{$_}->{seq} > $snap->{seq} }
+ keys $m->{snaps}->%*;
+ err("cannot rollback '$name': dataset is busy") if $self->busy($name);
+ my $ds = $m->{ds}->{$name};
+ $ds->{props} = dclone($snap->{props});
+ $ds->{volsize} = $snap->{volsize};
+ $ds->{devwait} = $self->{udev_delay};
+ return ();
+ }
+
+ sub zfs_set($self, @args) {
+ my $target = pop @args;
+ $self->violation('zfs set') if !@args;
+ my $m = $self->{m};
+ my $object = $target =~ /\@/ ? $m->{snaps}->{$target} : $m->{ds}->{$target};
+ err("cannot open '$target': dataset does not exist") if !$object;
+ for my $assignment (@args) {
+ $self->violation("zfs set '$assignment'") if $assignment !~ /\A([^=]+)=(.*)\z/;
+ my ($prop, $value) = ($1, $2);
+ if ($prop eq 'volsize') {
+ $self->violation('zfs set volsize') if $value !~ /\A([0-9]+)k\z/;
+ $object->{volsize} = $1 * 1024;
+ } elsif ($prop =~ /:/) {
+ $object->{props}->{$prop} = $value;
+ } else {
+ $self->violation("zfs set '$prop'");
+ }
+ }
+ return ();
+ }
+
+ # Like OpenZFS, which renames the minor of a zvol that is open: an
+ # exported zvol is renamed under its namespace.
+ sub zfs_rename($self, @args) {
+ $self->violation('zfs rename') if @args != 2;
+ my ($old, $new) = @args;
+ my $m = $self->{m};
+ err("cannot open '$old': dataset does not exist") if !$m->{ds}->{$old};
+ err("cannot rename to '$new': dataset already exists") if $m->{ds}->{$new};
+ $m->{ds}->{$new} = delete $m->{ds}->{$old};
+ for my $snap (grep { index($_, "$old\@") == 0 } keys $m->{snaps}->%*) {
+ my $renamed = $new . substr($snap, length($old));
+ $m->{snaps}->{$renamed} = delete $m->{snaps}->{$snap};
+ for my $ds (values $m->{ds}->%*) {
+ $ds->{origin} = $renamed if ($ds->{origin} // '-') eq $snap;
+ }
+ }
+ $m->{ds}->{$new}->{devwait} = $self->{udev_delay};
+ return ();
+ }
+
+ sub zfs_snapshot($self, @args) {
+ $self->violation('zfs snapshot') if @args != 1 || $args[0] !~ /\A([^@]+)\@(.+)\z/;
+ my ($name, $snap) = ($1, $2);
+ err("cannot create snapshot '$args[0]': invalid character in name")
+ if $snap !~ /\A[A-Za-z0-9_.: -]+\z/;
+ my $m = $self->{m};
+ my $ds = $m->{ds}->{$name} // err("cannot open '$name': dataset does not exist");
+ err("cannot create snapshot '$args[0]': dataset already exists")
+ if $m->{snaps}->{ $args[0] };
+ $m->{snaps}->{ $args[0] } =
+ { props => dclone($ds->{props}), volsize => $ds->{volsize}, seq => ++$m->{seq} };
+ return ();
+ }
+}
+
+# The inventory read written down independently of the plugin code.
+my $INVENTORY_PROPS =
+ 'type,proxmox:nvme-subsys,proxmox:nvme-nsid,proxmox:nvme-uuid,proxmox:nvme-last-nsid';
+
+sub expected_inventory_read($pool) {
+ return [
+ 'get',
+ '-H',
+ '-p',
+ '-d',
+ '1',
+ '-t',
+ 'filesystem,volume',
+ '-o',
+ 'name,property,value,source',
+ $INVENTORY_PROPS,
+ $pool,
+ ];
+}
+
+# ---------------------------------------------------------------------------
+# Flow helpers and invariants
+# ---------------------------------------------------------------------------
+
+# Runs $code against $fake and returns what happened during the run.
+sub flow($fake, $code) {
+ local $FAKE = $fake;
+ local @LOCKS = ();
+ local @NESTED_LOCKS = ();
+ local @WARNINGS = ();
+ local @SYSFS_WRITES = ();
+
+ my $start = scalar($fake->{calls}->@*);
+ my $result = eval { $code->() };
+ my $error = $@;
+ my @calls = $fake->{calls}->@*;
+ return {
+ error => $error,
+ result => $result,
+ calls => [@calls[$start .. $#calls]],
+ locks => [@LOCKS],
+ nested => [@NESTED_LOCKS],
+ warnings => [@WARNINGS],
+ sysfs_writes => [@SYSFS_WRITES],
+ };
+}
+
+sub ops($res) {
+ return [map { $_->{op} } $res->{calls}->@*];
+}
+
+sub changes($res) {
+ return scalar(grep { $_->{changing} } $res->{calls}->@*);
+}
+
+# Our subsystem published on two ports to two hosts, no volumes yet.
+sub target_fake(%opts) {
+ my $fake = FakeTarget->new(pools => ['tank', 'other'], %opts);
+ $fake->add_host($_, $KEY) for @HOSTS;
+ $fake->add_subsystem($NQN, acl => [@HOSTS]);
+ $fake->add_port(1, '192.0.2.21', 4420, links => [$NQN]);
+ $fake->add_port(2, '192.0.2.22', 4420, links => [$NQN]);
+ return $fake;
+}
+
+sub add_volume($fake, $name, $nsid, $uuid, %o) {
+ $fake->add_zvol("tank/$name", identity => [$NQN, $nsid, $uuid]);
+ $fake->add_namespace($NQN, $nsid, $uuid, "/dev/zvol/tank/$name", $o{enable} // 1)
+ if !$o{unexported};
+ return $fake;
+}
+
+# A converged target with three owned volumes, an unowned leftover, a volume
+# of another subsystem in our pool, and a foreign subsystem that shares the
+# first host and port 1.
+sub lifecycle_fake(%opts) {
+ my $fake = target_fake(%opts);
+ add_volume($fake, 'vm-100-disk-0', 1, $U{1});
+ add_volume($fake, 'vm-101-disk-0', 2, $U{2});
+ add_volume($fake, 'base-102-disk-0', 3, $U{3});
+ $fake->add_snapshot('tank/vm-100-disk-0@snap1');
+ $fake->add_snapshot('tank/base-102-disk-0@__base__');
+ $fake->add_zvol('tank/vm-900-disk-0');
+ $fake->add_zvol('tank/vm-901-disk-0', identity => [$FOREIGN_NQN, 7, $U{7}]);
+ $fake->add_zvol('other/foreign-disk', identity => [$FOREIGN_NQN, 1, $U{8}]);
+ $fake->add_subsystem(
+ $FOREIGN_NQN,
+ model => 'Linux',
+ serial => '0123456789abcdef',
+ acl => [$HOSTS[0]],
+ );
+ $fake->add_namespace($FOREIGN_NQN, 1, $U{8}, '/dev/zvol/other/foreign-disk');
+ $fake->{m}->{ports}->{1}->{links}->{$FOREIGN_NQN} = 1;
+ return $fake;
+}
+
+sub owned_volumes($m, $nqn = $NQN, $pool = 'tank') {
+ my %owned;
+ for my $name (sort keys $m->{ds}->%*) {
+ my $ds = $m->{ds}->{$name};
+ next if $ds->{type} ne 'volume' || $name !~ m{\A\Q$pool\E/[^/]+\z};
+ next if ($ds->{props}->{'proxmox:nvme-subsys'} // '') ne $nqn;
+ $owned{$name} = [map { $ds->{props}->{"proxmox:nvme-$_"} } qw(nsid uuid)];
+ }
+ return \%owned;
+}
+
+# Everything on the target that does not belong to the storage under test.
+# The last NSID handed out, on its pool, belongs to the storage.
+sub foreign_view($m, $port_ids, $nqn, $hostnqns, $pool = 'tank') {
+ my %ours = map { $_ => 1 } $hostnqns->@*;
+ my $dataset = sub($name) {
+ my $ds = $m->{ds}->{$name};
+ return $ds if $name ne $pool;
+ my %props = $ds->{props}->%*;
+ delete $props{'proxmox:nvme-last-nsid'};
+ return { $ds->%*, props => \%props };
+ };
+ return {
+ subsystems =>
+ { map { $_ => $m->{subsystems}->{$_} } grep { $_ ne $nqn } keys $m->{subsystems}->%* },
+ datasets => {
+ map { $_ => $dataset->($_) }
+ grep { ($m->{ds}->{$_}->{props}->{'proxmox:nvme-subsys'} // '') ne $nqn }
+ keys $m->{ds}->%*
+ },
+ hosts => { map { $_ => $m->{hosts}->{$_} } grep { !$ours{$_} } keys $m->{hosts}->%* },
+ ports => {
+ map {
+ my $port = $m->{ports}->{$_};
+ my @links = $port ? sort grep { $_ ne $nqn } keys $port->{links}->%* : ();
+ ($_ => $port ? [$port->{attr}, \@links] : undef)
+ } $port_ids->@*
+ },
+ };
+}
+
+# (i), (ii), (vi) and (vii), checked after every call.
+sub online_violations($fake, $ctx) {
+ my $m = $fake->{m};
+ my $nqn = $ctx->{nqn};
+ my @bad;
+ if (my $subsys = $m->{subsystems}->{$nqn}) {
+ my $owned = owned_volumes($m, $nqn, $ctx->{pool});
+ my %identity = map { ("/dev/zvol/$_" => $owned->{$_}) } keys $owned->%*;
+ my %seen;
+ for my $id (sort keys $subsys->{ns}->%*) {
+ my $ns = $subsys->{ns}->{$id};
+ push @bad, "(ii) UUID $ns->{device_uuid} used by two namespaces"
+ if $seen{ lc($ns->{device_uuid}) }++;
+ next if $ns->{enable} ne '1';
+ my $want = $identity{ $ns->{device_path} };
+ push @bad, "(i) namespace $id exports $ns->{device_path} under another identity"
+ if !$want || $want->[0] ne $id || lc($want->[1]) ne lc($ns->{device_uuid});
+ }
+ if (grep { $_->{links}->{$nqn} } values $m->{ports}->%*) {
+ push @bad, '(vi) published while any host may connect'
+ if $subsys->{attr}->{attr_allow_any_host} ne '0';
+ for my $hostnqn ($ctx->{teardown} ? () : $ctx->{hosts}->@*) {
+ push @bad, "(vi) published without the ACL of $hostnqn"
+ if !$subsys->{acl}->{$hostnqn};
+ push @bad, "(vi) published without the key of $hostnqn"
+ if (($m->{hosts}->{$hostnqn} // {})->{key} // '') ne $ctx->{key};
+ }
+ }
+ }
+ push @bad,
+ '(vii) objects of other storages changed'
+ if !same(
+ foreign_view($m, $ctx->{port_ids}, $nqn, $ctx->{hosts}, $ctx->{pool}),
+ $ctx->{foreign},
+ );
+ return @bad;
+}
+
+# (iii): owned identities stay valid, unique and attached to their volume.
+sub identity_violations($m, $ctx) {
+ my @bad;
+ my $owned = owned_volumes($m, $ctx->{nqn}, $ctx->{pool});
+ my (%nsids, %uuids);
+ for my $name (sort keys $owned->%*) {
+ my ($nsid, $uuid) = $owned->{$name}->@*;
+ push @bad, "(iii) invalid identity on $name"
+ if ($nsid // '') !~ /\A[1-9][0-9]*\z/ || ($uuid // '') !~ $UUID_RE;
+ push @bad, "(iii) NSID $nsid on two volumes" if $nsids{ $nsid // '' }++;
+ push @bad, "(iii) UUID $uuid on two volumes" if $uuids{ lc($uuid // '') }++;
+ }
+ for my $name (sort keys $ctx->{initial_owned}->%*) {
+ my $identity = join('/', $ctx->{initial_owned}->{$name}->@*);
+ my @holders = grep { join('/', $owned->{$_}->@*) eq $identity } keys $owned->%*;
+ next if !@holders && ($ctx->{may_vanish} // '') eq $name;
+ my %allowed = map { $_ => 1 } $name, ($ctx->{renames}->{$name} // ());
+ push @bad, "(iii) identity of $name lost" if @holders != 1 || !$allowed{ $holders[0] };
+ }
+ return @bad;
+}
+
+# A creation cut right after its mkdir leaves an unused subsystem with the
+# kernel's default model, which another tool could have created as well.
+# Activation refuses it until an administrator removes it.
+sub default_subsystem($m, $nqn) {
+ my $subsys = $m->{subsystems}->{$nqn} // return 0;
+ return
+ $subsys->{attr}->{attr_model} eq 'Linux'
+ && !$subsys->{ns}->%*
+ && !$subsys->{acl}->%*
+ && !grep { $_->{links}->{$nqn} } values $m->{ports}->%*;
+}
+
+# (v): one more activation exports every owned volume exactly once.
+sub convergence_violations($fake, $ctx) {
+ my $res = flow($fake, sub { activate($ctx->{scfg}, $ctx->{key}, $ctx->{hosts}) });
+ if (my $error = $res->{error}) {
+ return ()
+ if $error =~ /not created by Proxmox VE/ && default_subsystem($fake->{m}, $ctx->{nqn});
+ return "(v) activation afterwards failed: $error";
+ }
+ my $m = $fake->{m};
+ my $owned = owned_volumes($m, $ctx->{nqn}, $ctx->{pool});
+ my $subsys = $m->{subsystems}->{ $ctx->{nqn} } // return '(v) subsystem missing';
+ my %want = map { $owned->{$_}->[0] => ["/dev/zvol/$_", lc($owned->{$_}->[1]), '1'] }
+ keys $owned->%*;
+ my %have = map {
+ my $ns = $subsys->{ns}->{$_};
+ ($_ => [$ns->{device_path}, lc($ns->{device_uuid}), $ns->{enable}])
+ } keys $subsys->{ns}->%*;
+ my @bad;
+ push @bad, '(v) exported namespaces differ from the owned volumes' if !same(\%want, \%have);
+ # An interrupted port creation leaves an incomplete port that is ignored;
+ # the portal is then served by a new port with the same address.
+ for my $portal ($PORTALS->@*) {
+ my ($address, $service) = $portal->@{qw(address port)};
+ push @bad, "(v) not published on $address"
+ if !grep {
+ $_->{links}->{ $ctx->{nqn} }
+ && $_->{attr}->{addr_traddr} eq $address
+ && $_->{attr}->{addr_trsvcid} eq "$service"
+ } values $m->{ports}->%*;
+ }
+ return @bad;
+}
+
+sub run_case($initial, $ctx, $faults, $read_fault = undef) {
+ my $fake = FakeTarget->from_state($initial);
+ $fake->{faults} = $faults;
+ $fake->{read_fault} = $read_fault;
+ my @online;
+ $fake->{after_call} = sub($f, $call) {
+ push @online,
+ map { "call $call->{index} ($call->{op}): $_" } online_violations($f, $ctx);
+ };
+ my $res = flow($fake, $ctx->{action});
+ $res->{fake} = $fake;
+ $res->{online} = \@online;
+ $res->{faults} = { $faults->%*, ($read_fault ? (read => $read_fault->{mode}) : ()) };
+ return $res;
+}
+
+sub case_violations($ctx, $res) {
+ my $fake = $res->{fake};
+ my @bad;
+ push @bad, 'domain lock taken ' . scalar($res->{locks}->@*) . ' times'
+ if $res->{locks}->@* > 1;
+ push @bad, 'nested domain lock request' if $res->{nested}->@*;
+ push @bad, 'key in the error text' if grep { index($res->{error}, $_) >= 0 } $KEY, $KEY_B;
+
+ # The result must match the outcome: success only when the goal is
+ # reached, and a flow whose only faults are lost replies (create and
+ # clone, which cannot settle one with a read) may fail after reaching its
+ # goal only with an unknown target state.
+ my @missing = $ctx->{done}->($fake->{m}, $res);
+ push @bad, map { "success without reaching the goal: $_" } @missing if $res->{error} eq '';
+ my @modes = values $res->{faults}->%*;
+ push @bad, "failure after reaching the goal: $res->{error}"
+ if $ctx->{lost_replies}
+ && $res->{error} ne ''
+ && $res->{error} !~ /the target state is unknown\n\z/
+ && !@missing
+ && @modes
+ && !grep { $_ ne 'after' } @modes;
+
+ # A call that ssh gave up on completes on the target afterwards. When the
+ # rest of it changes the target, the outcome was unknown.
+ if ($fake->{late}->@*) {
+ if (
+ grep {
+ grep { mutating_step($_) } $_->{steps}->@*
+ } $fake->{late}->@*
+ ) {
+ push @bad, "late call after a success" if $res->{error} eq '';
+ push @bad, "late call reported without an unknown state: $res->{error}"
+ if $res->{error} !~ /the target state is unknown\n\z/;
+ }
+ $fake->land;
+ push @bad, map { "after the late call: $_" } online_violations($fake, $ctx);
+ }
+ push @bad, identity_violations($fake->{m}, $ctx);
+ $fake->{faults} = {};
+ $fake->{read_fault} = undef;
+ if ($ctx->{retry}) {
+ my $retry = flow($fake, $ctx->{retry});
+ push @bad, "retry failed: $retry->{error}" if $retry->{error};
+ push @bad, $ctx->{goal}->($fake->{m}) if !$retry->{error};
+ } else {
+ push @bad, convergence_violations($fake, $ctx);
+ }
+ push @bad, $res->{online}->@*;
+ push @bad, map { "protocol: $_" } $fake->{violations}->@*;
+ return @bad;
+}
+
+# Fault sweep: every mutating call of a flow fails, and then every later
+# mutating call of that run fails as well, or the first read after it. After
+# each run: (i) every enabled namespace of the subsystem exports the zvol of
+# its identity, (ii) no UUID is used twice, (iii) owned identities are kept
+# and unique, (v) one more activation (or a retried removal) converges, (vi)
+# the subsystem is never published before its ACLs, keys and
+# allow_any_host=0, (vii) nothing of other storages changes, every change ran
+# under one domain lock, and the result matches the outcome. (i), (ii), (vi)
+# and (vii) are checked after every call; refusals are covered by their own
+# tests.
+sub sweep($name, $ctx) {
+ my $initial = dclone($ctx->{setup}->()->{m});
+ $ctx->{nqn} //= $NQN;
+ $ctx->{pool} //= 'tank';
+ $ctx->{hosts} //= [@HOSTS];
+ $ctx->{key} //= $KEY;
+ $ctx->{scfg} //= scfg();
+ $ctx->{initial_owned} = owned_volumes($initial, $ctx->{nqn}, $ctx->{pool});
+ $ctx->{port_ids} = [sort keys $initial->{ports}->%*];
+ $ctx->{foreign} =
+ foreign_view($initial, $ctx->{port_ids}, $ctx->{nqn}, $ctx->{hosts}, $ctx->{pool});
+
+ my $baseline = run_case($initial, $ctx, {});
+ is($baseline->{error}, '', "$name: succeeds without faults");
+ is_deeply([$ctx->{done}->($baseline->{fake}->{m}, $baseline)], [], "$name: reaches its goal");
+ is_deeply([case_violations($ctx, $baseline)], [], "$name: invariants hold without faults");
+
+ # A call fails in each mode of FakeTarget::execute, at each step of its
+ # chain. A second failure, in the compensation or the next round, is one
+ # that fails before or after its call, or the first read after the first
+ # failure fails or finds the target unreachable.
+ my $modes = sub($call) {
+ my @steps = 1 .. $call->{steps}->$#*;
+ return ('before', 'after', 'late', (map { "cut:$_" } @steps), map { "lost:$_" } @steps);
+ };
+ my @first = grep { $_->{mutating} } $baseline->{calls}->@*;
+ ok(scalar(@first), "$name: changes the target");
+ my ($runs, @failures) = (0);
+ for my $first (@first) {
+ my $k = $first->{index};
+ for my $mode ($modes->($first)) {
+ my $res = run_case($initial, $ctx, { $k => $mode });
+ $runs++;
+ my @later = grep { $_->{mutating} && $_->{index} > $k } $res->{calls}->@*;
+ push @failures, map { "call $k $mode: $_" } case_violations($ctx, $res);
+ for my $second (@later) {
+ my $j = $second->{index};
+ for my $again_mode (qw(before after)) {
+ my $again = run_case($initial, $ctx, { $k => $mode, $j => $again_mode });
+ $runs++;
+ push @failures,
+ map { "call $k $mode, call $j $again_mode: $_" }
+ case_violations($ctx, $again);
+ }
+ }
+ for my $read_mode (qw(before unreachable)) {
+ my $read_fault = { after => $k, mode => $read_mode };
+ my $again = run_case($initial, $ctx, { $k => $mode }, $read_fault);
+ $runs++;
+ push @failures,
+ map { "call $k $mode, next read $read_mode: $_" } case_violations($ctx, $again);
+ }
+ }
+ }
+ ok(!@failures, "$name: the invariants hold in $runs fault injections")
+ or diag(join("\n", @failures));
+}
+
+# ---------------------------------------------------------------------------
+# Fixtures and the fake target formats
+# ---------------------------------------------------------------------------
+
+my $FIXTURE_ZFS = slurp("$FIXTURES/zfs_inventory.txt");
+my $FIXTURE_CFS = slurp("$FIXTURES/configfs_snapshot.txt");
+my $FIXTURE_DIGESTS = join('', map { "$KEY_SHA $ROOT/hosts/$_/dhchap_key\n" } @FIXTURE_HOSTS);
+
+sub fixture_fake() {
+ my $fake = FakeTarget->new(pools => ['tank']);
+ $fake->add_host($_) for @FIXTURE_HOSTS;
+ $fake->add_subsystem($FIXTURE_NQN, acl => [@FIXTURE_HOSTS]);
+ $fake->add_port($_, "192.0.2.2$_", 4420, links => [$FIXTURE_NQN]) for 1, 2;
+ $fake->add_zvol('tank/vm-100-disk-0', identity => [$FIXTURE_NQN, 5, $FIXTURE_UUID5]);
+ $fake->add_zvol('tank/vm-100-cloudinit', identity => [$FIXTURE_NQN, 6, $FIXTURE_UUID6]);
+ $fake->add_namespace($FIXTURE_NQN, 5, $FIXTURE_UUID5, '/dev/zvol/tank/vm-100-disk-0');
+ $fake->add_namespace($FIXTURE_NQN, 6, $FIXTURE_UUID6, '/dev/zvol/tank/vm-100-cloudinit');
+ return $fake;
+}
+
+subtest 'the fake target answers in the formats of the fixture target' => sub {
+ my $fake = fixture_fake();
+ my @zfs = $fake->zfs_get(expected_inventory_read('tank')->@[1 .. 10]);
+ is(join('', map { "$_\n" } @zfs), $FIXTURE_ZFS, 'the ZFS inventory equals the fixture');
+ my $cfs = join('', map { "$_\n" } $fake->configfs_lines([]));
+ is_deeply(
+ nv('_nvmet_parse_configfs', $cfs),
+ nv('_nvmet_parse_configfs', $FIXTURE_CFS),
+ 'the configfs read parses to the fixture',
+ );
+};
+
+# ---------------------------------------------------------------------------
+# Parsers
+# ---------------------------------------------------------------------------
+
+sub zfs_rows($name, $type, $nqn = '-', $nsid = '-', $uuid = '-', $source = 'local', $last = '-') {
+ my $source_of = sub($value) { $value eq '-' ? '-' : $source };
+ return (
+ "$name\ttype\t$type\t-",
+ "$name\tproxmox:nvme-subsys\t$nqn\t" . $source_of->($nqn),
+ "$name\tproxmox:nvme-nsid\t$nsid\t" . $source_of->($nsid),
+ "$name\tproxmox:nvme-uuid\t$uuid\t" . $source_of->($uuid),
+ "$name\tproxmox:nvme-last-nsid\t$last\t" . $source_of->($last),
+ );
+}
+
+sub inventory(@rows) {
+ return nv('_nvmet_parse_zfs_inventory', join('', map { "$_\n" } @rows), 'tank');
+}
+
+my @POOL = zfs_rows('tank', 'filesystem');
+my @ONE = zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $U{1});
+my @TWO = zfs_rows('tank/vm-101-disk-0', 'volume', $NQN, 2, $U{2});
+
+subtest 'ZFS inventory parser' => sub {
+ is_deeply(
+ inventory(@POOL, @ONE, @TWO),
+ {
+ types => {
+ tank => 'filesystem',
+ 'tank/vm-100-disk-0' => 'volume',
+ 'tank/vm-101-disk-0' => 'volume',
+ },
+ volumes => {
+ 'tank/vm-100-disk-0' => { nqn => $NQN, nsid => 1, uuid => $U{1} },
+ 'tank/vm-101-disk-0' => { nqn => $NQN, nsid => 2, uuid => $U{2} },
+ },
+ last_nsid => 0,
+ },
+ 'complete inventory',
+ );
+ my $pool_last = sub($last, $source = 'local') {
+ return inventory(
+ zfs_rows('tank', 'filesystem', '-', '-', '-', $source, $last),
+ zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $U{1}, 'local', '-'),
+ )->{last_nsid};
+ };
+ is($pool_last->(7), 7, 'the last NSID handed out is read from the pool');
+ is($pool_last->(7, 'inherited from tank'), 0, 'but not inherited');
+ is($pool_last->('x'), 0, 'and only when it is a valid NSID');
+ is(
+ inventory(@POOL, zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $U{1}, 'local', 9))
+ ->{last_nsid},
+ 0,
+ 'a volume does not carry it',
+ );
+ my $short_owner = "tank/vm-101-disk-0\tproxmox:nvme-subsys\t$NQN";
+ my $foreign_source = "tank/vm-101-disk-0\tproxmox:nvme-uuid\t$U{2}\tgarbage";
+ for my $case (
+ [
+ 'missing owner row',
+ [@POOL, @ONE, @TWO[0, 2, 3]],
+ qr/incomplete ZFS properties on 'tank\/vm-101-disk-0'/,
+ ],
+ [
+ 'short row',
+ [@POOL, @ONE, $TWO[0], $short_owner, @TWO[2, 3]],
+ qr/malformed ZFS inventory row/,
+ ],
+ [
+ 'duplicate property',
+ [@POOL, @ONE, @TWO, $TWO[1]],
+ qr/duplicate ZFS property 'proxmox:nvme-subsys'/,
+ ],
+ [
+ 'unknown property',
+ [@POOL, @ONE, @TWO, "tank/vm-101-disk-0\tcompression\ton\tlocal"],
+ qr/unexpected ZFS property on 'tank\/vm-101-disk-0'/,
+ ],
+ [
+ 'extra field',
+ [@POOL, @ONE, @TWO[0 .. 2], "$TWO[3]\textra"],
+ qr/malformed ZFS inventory row/,
+ ],
+ [
+ 'unknown source',
+ [@POOL, @ONE, @TWO[0 .. 2], $foreign_source],
+ qr/invalid ZFS property source/,
+ ],
+ ['missing pool rows', [@ONE, @TWO], qr/missing the configured pool/],
+ [
+ 'missing pool type row',
+ [@POOL[1 .. 4], @ONE],
+ qr/incomplete ZFS properties on 'tank'/,
+ ],
+ [
+ 'pool name prefix',
+ [@POOL, zfs_rows('tank2/vm-1-disk-0', 'volume')],
+ qr/outside configured pool/,
+ ],
+ [
+ 'depth 2 dataset',
+ [@POOL, zfs_rows('tank/sub/vm-1-disk-0', 'volume')],
+ qr/outside configured pool/,
+ ],
+ [
+ 'invalid type',
+ [@POOL, "tank/vm-100-disk-0\ttype\tsnapshot\t-", @ONE[1 .. 3]],
+ qr/invalid ZFS dataset type/,
+ ],
+ [
+ 'value with CR',
+ [@POOL, @ONE[0 .. 2], "tank/vm-100-disk-0\tproxmox:nvme-uuid\t$U{1}\r\tlocal"],
+ qr/malformed ZFS inventory row/,
+ ],
+ [
+ 'inherited from a malformed pool',
+ [
+ @POOL,
+ @ONE[0 .. 2],
+ "tank/vm-100-disk-0\tproxmox:nvme-uuid\t$U{1}\tinherited from bad pool",
+ ],
+ qr/invalid ZFS pool name/,
+ ],
+ ) {
+ my ($name, $rows, $error) = $case->@*;
+ eval { inventory($rows->@*) };
+ like($@, $error, "rejects $name");
+ }
+
+ is_deeply(
+ inventory(@POOL),
+ { types => { tank => 'filesystem' }, volumes => {}, last_nsid => 0 },
+ 'a verified empty pool',
+ );
+ my $received = inventory(
+ @POOL, zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $U{1}, 'received'),
+ );
+ is($received->{volumes}->{'tank/vm-100-disk-0'}->{nqn}, $NQN, 'received properties count');
+ my $inherited = inventory(
+ zfs_rows('tank', 'filesystem', $NQN, '-', '-'),
+ zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $U{1}, 'inherited from tank'),
+ );
+ is_deeply(
+ $inherited->{volumes}->{'tank/vm-100-disk-0'},
+ { nqn => '-', nsid => '-', uuid => '-' },
+ 'inherited properties never own a volume',
+ );
+ my $subvol = inventory(@POOL, zfs_rows('tank/subvol-100-disk-0', 'filesystem'));
+ ok(
+ $subvol->{types}->{'tank/subvol-100-disk-0'} eq 'filesystem'
+ && !$subvol->{volumes}->{'tank/subvol-100-disk-0'},
+ 'a filesystem child is a dataset but not a volume',
+ );
+
+ # a direct child with a name PVE never uses is carried as not owned
+ my $spaced = inventory(@POOL, zfs_rows('tank/my data', 'volume', $NQN, 9, $U{9}));
+ is($spaced->{types}->{'tank/my data'}, 'volume',
+ 'a name with a space is kept as a dataset');
+ is($spaced->{volumes}->{'tank/my data'}->{nqn}, '-', 'a name with a space is never owned');
+ is_deeply(nv('_nvmet_desired_namespaces', $spaced, $NQN), {}, 'and never exported');
+ eval { inventory(@POOL, "tank/my data\tcompression\ton\tlocal") };
+ is($@, "unexpected ZFS property\n", 'errors do not echo names PVE never uses');
+
+ my $fixture = nv('_nvmet_parse_zfs_inventory', $FIXTURE_ZFS, 'tank');
+ is_deeply(
+ $fixture->{volumes},
+ {
+ 'tank/vm-100-disk-0' => { nqn => $FIXTURE_NQN, nsid => 5, uuid => $FIXTURE_UUID5 },
+ 'tank/vm-100-cloudinit' =>
+ { nqn => $FIXTURE_NQN, nsid => 6, uuid => $FIXTURE_UUID6 },
+ },
+ 'the fixture inventory',
+ );
+ eval { nv('_nvmet_parse_zfs_inventory', $FIXTURE_ZFS, 'bad pool') };
+ like($@, qr/invalid ZFS pool name/, 'the configured pool is validated');
+};
+
+subtest 'ZFS property values and sources' => sub {
+ for my $case (
+ [undef, 'local'],
+ ['', 'local'],
+ ["a\nb", 'local'],
+ ["a\tb", 'local'],
+ ['x', ''],
+ ['x', undef],
+ ['x', "local\r"],
+ ) {
+ eval { nv('_nvmet_property_value', $case->@*) };
+ like($@, qr/malformed ZFS property value/, 'rejects a malformed value or source');
+ }
+ is(nv('_nvmet_property_value', $NQN, 'local'), $NQN, 'local values count');
+ is(nv('_nvmet_property_value', $NQN, 'received'), $NQN, 'received values count');
+ is(nv('_nvmet_property_value', '-', '-'), '-', 'unset values');
+ is(
+ nv('_nvmet_property_value', $NQN, 'inherited from tank/a'),
+ '-',
+ 'inherited values do not',
+ );
+ eval { nv('_nvmet_property_value', $NQN, 'default') };
+ like($@, qr/invalid ZFS property source/, 'rejects other sources');
+ eval { nv('_nvmet_property_value', $NQN, '-') };
+ like($@, qr/invalid ZFS property source/, 'rejects a value without a source');
+ eval { nv('_nvmet_property_value', $NQN, 'inherited from tank;x') };
+ like($@, qr/invalid ZFS pool name/, 'rejects an inherited source outside pool names');
+};
+
+subtest 'configfs parser' => sub {
+ my $with_digests = nv('_nvmet_parse_configfs', $FIXTURE_CFS . $FIXTURE_DIGESTS);
+ is($with_digests->{hosts}->{$_}->{key_sha256}, $KEY_SHA, 'key digest of a fixture host')
+ for @FIXTURE_HOSTS;
+ my $binary = nv(
+ '_nvmet_parse_configfs',
+ $FIXTURE_CFS . join('', map { "$KEY_SHA *$ROOT/hosts/$_/dhchap_key\n" } @FIXTURE_HOSTS),
+ );
+ is($binary->{hosts}->{ $FIXTURE_HOSTS[0] }->{key_sha256}, $KEY_SHA, 'binary-mode digests');
+
+ my $without = sub($re) { return $FIXTURE_CFS =~ s/^$re\n//mr };
+ for my $case (
+ [
+ 'missing root',
+ $without->(qr{D \Q$ROOT\E}),
+ qr/^NVMe target configfs is unavailable$/,
+ ],
+ [
+ 'a namespace without enable',
+ $without->(qr{.*namespaces/5/enable:1}),
+ qr/incomplete NVMe namespace '5' of subsystem '\Q$FIXTURE_NQN\E'/,
+ ],
+ [
+ 'a port without address',
+ $without->(qr{.*ports/2/addr_traddr.*}),
+ qr/incomplete NVMe port '2'/,
+ ],
+ [
+ 'a duplicate attribute',
+ "$FIXTURE_CFS$ROOT/subsystems/$FIXTURE_NQN/attr_serial:X\n",
+ qr/duplicate NVMe target configfs attribute/,
+ ],
+ [
+ 'an attribute without its directory',
+ "$FIXTURE_CFS$ROOT/subsystems/nqn.x:y/attr_model:Z\n",
+ qr/incomplete NVMe subsystem 'nqn.x:y'/,
+ ],
+ [
+ 'a link to a missing port',
+ "${FIXTURE_CFS}L $ROOT/ports/9/subsystems/$FIXTURE_NQN\n",
+ qr/incomplete NVMe port '9'/,
+ ],
+ [
+ 'a digest for a missing host',
+ "$FIXTURE_CFS$KEY_SHA $ROOT/hosts/nqn.x:gone/dhchap_key\n",
+ qr/key digest for unknown NVMe host 'nqn.x:gone'/,
+ ],
+ [
+ 'a malformed digest',
+ $FIXTURE_CFS . substr($KEY_SHA, 1) . " $ROOT/hosts/$FIXTURE_HOSTS[0]/dhchap_key\n",
+ qr/unexpected NVMe target configfs line/,
+ ],
+ [
+ 'a duplicate digest',
+ "$FIXTURE_CFS$FIXTURE_DIGESTS$FIXTURE_DIGESTS",
+ qr/duplicate DH-HMAC-CHAP key digest/,
+ ],
+ ) {
+ my ($name, $text, $error) = $case->@*;
+ eval { nv('_nvmet_parse_configfs', $text) };
+ like($@, $error, "rejects $name");
+ }
+ eval { nv('_nvmet_parse_configfs', "$FIXTURE_CFS$KEY and more\n") };
+ is($@, "unexpected NVMe target configfs line\n", 'a garbage line is rejected without echo');
+ eval { nv('_nvmet_parse_configfs', "${FIXTURE_CFS}D $ROOT/subsystems/bad name\n") };
+ is($@, "incomplete NVMe subsystem\n", 'malformed names are not echoed');
+
+ my $tolerated = nv(
+ '_nvmet_parse_configfs',
+ join(
+ '',
+ $FIXTURE_CFS,
+ "D $ROOT/ports/1/referrals/x\n",
+ "L $ROOT/subsystems/$FIXTURE_NQN/passthru/x\n",
+ "D $ROOT/subsystems/$FIXTURE_NQN/namespaces/5/ana\n",
+ "$ROOT/ports/1/param_inline_data_size:16384\n",
+ "$ROOT/subsystems/$FIXTURE_NQN/attr_version:1.3\n",
+ ),
+ );
+ is_deeply(
+ $tolerated,
+ nv('_nvmet_parse_configfs', $FIXTURE_CFS),
+ 'other objects are ignored',
+ );
+ my $unset = $FIXTURE_CFS =~ s{(namespaces/6/device_path:).*}{$1(null)}r;
+ $unset =~ s{(namespaces/5/device_path:).*}{$1};
+ my $null = nv('_nvmet_parse_configfs', $unset);
+ is(
+ $null->{subsystems}->{$FIXTURE_NQN}->{namespaces}->{6}->{device_path},
+ '(null)',
+ 'a (null) device',
+ );
+ is(
+ $null->{subsystems}->{$FIXTURE_NQN}->{namespaces}->{5}->{device_path},
+ '',
+ 'an empty device',
+ );
+};
+
+subtest 'state marker and mount table' => sub {
+ my ($zfs, $cfs) = nv('_nvmet_split_state', "a\tb\n$MARKER\nD x\n");
+ is_deeply([$zfs, $cfs], ["a\tb\n", "D x\n"], 'splits at the marker');
+ for my $text ("a\n", "a\n$MARKER\nb\n$MARKER\n", "a\n $MARKER\n", "a\n${MARKER}x\n") {
+ eval { nv('_nvmet_split_state', $text) };
+ like(
+ $@,
+ qr/malformed NVMe target state/,
+ 'rejects a missing, repeated or altered marker',
+ );
+ }
+ my $mounts = "sysfs /sys sysfs rw 0 0\nnone /sys/kernel/config configfs rw 0 0\n";
+ ok(nv('_nvmet_parse_mounts', $mounts), 'configfs is mounted');
+ ok(!nv('_nvmet_parse_mounts', "sysfs /sys sysfs rw 0 0\n"), 'configfs is not mounted');
+ ok(!nv('_nvmet_parse_mounts', "none /mnt configfs rw 0 0\n"), 'configfs elsewhere');
+ ok(!nv('_nvmet_parse_mounts', "x /sys/kernel/config tmpfs rw 0 0\n"),
+ 'another file system');
+ ok(!nv('_nvmet_parse_mounts', ''), 'empty mount table');
+};
+
+# ---------------------------------------------------------------------------
+# Planners
+# ---------------------------------------------------------------------------
+
+my $S = "$ROOT/subsystems/$NQN";
+sub write_step($path, $value) { return { write => $path, value => $value } }
+
+# The step that makes the key attributes of hosts readable by root only.
+sub chmod_step(@hosts) {
+ return ['chmod', '0600', map { ("$_/dhchap_key", "$_/dhchap_ctrl_key") } @hosts];
+}
+
+sub cfs_model(%o) {
+ my $model = { subsystems => {}, ports => {}, hosts => {} };
+ if (!$o{no_subsystem}) {
+ $model->{subsystems}->{$NQN} = {
+ attr_model => $MODEL,
+ attr_serial => nv('_nvmet_serial', $NQN),
+ attr_allow_any_host => '0',
+ acl => {},
+ namespaces => {},
+ ($o{subsystem} // {})->%*,
+ };
+ }
+ return $model;
+}
+
+sub ns_model($uuid, $dev, $enable = '1') {
+ return { enable => $enable, device_path => $dev, device_uuid => $uuid, buffered_io => '0' };
+}
+
+sub build_steps($nsid, $uuid, $dev) {
+ my $ns = "$S/namespaces/$nsid";
+ return [
+ ['test', '-b', $dev],
+ ['mkdir', $ns],
+ write_step("$ns/device_path", $dev),
+ write_step("$ns/device_uuid", $uuid),
+ write_step("$ns/buffered_io", 0),
+ write_step("$ns/enable", 1),
+ ];
+}
+
+subtest 'owned identity' => sub {
+ my $inv = inventory(
+ @POOL,
+ @ONE,
+ @TWO,
+ zfs_rows('tank/vm-102-disk-0', 'filesystem'),
+ zfs_rows('tank/vm-103-disk-0', 'volume'),
+ zfs_rows('tank/vm-104-disk-0', 'volume', $FOREIGN_NQN, 1, $U{1}),
+ );
+ is_deeply(
+ [nv('_nvmet_owned_identity', $inv, $NQN, 'tank/vm-100-disk-0')],
+ [1, $U{1}],
+ 'identity of an owned volume; the same identity under another NQN is allowed',
+ );
+ for my $case (
+ ['tank/vm-999-disk-0', qr/does not exist/],
+ ['tank/vm-102-disk-0', qr/is not a ZFS volume/],
+ ['tank/vm-103-disk-0', qr/is not owned by NVMe subsystem/],
+ ['tank/vm-104-disk-0', qr/is not owned by NVMe subsystem/],
+ ) {
+ eval { nv('_nvmet_owned_identity', $inv, $NQN, $case->[0]) };
+ like($@, $case->[1], "refuses $case->[0]");
+ }
+ for my $nsid (0, 0xffffffff, 'x', '01') {
+ my $bad =
+ inventory(@POOL, zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, $nsid, $U{1}));
+ eval { nv('_nvmet_owned_identity', $bad, $NQN, 'tank/vm-100-disk-0') };
+ like($@, qr/invalid NSID/, "refuses NSID '$nsid'");
+ }
+ my $bad = inventory(@POOL, zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, 'nope'));
+ eval { nv('_nvmet_owned_identity', $bad, $NQN, 'tank/vm-100-disk-0') };
+ like($@, qr/invalid namespace UUID/, 'refuses a malformed UUID');
+ for my $other ([1, $U{2}], [2, $U{1}]) {
+ my $dup = inventory(
+ @POOL, @ONE, zfs_rows('tank/vm-101-disk-0', 'volume', $NQN, $other->@*),
+ );
+ eval { nv('_nvmet_owned_identity', $dup, $NQN, 'tank/vm-100-disk-0') };
+ like($@, qr/duplicate NVMe identity on 'tank\/vm-100-disk-0'/, 'refuses a duplicate');
+ }
+ is(nv('_nvmet_serial', $NQN), 'PVEZFS' . substr(sha256_hex($NQN), 0, 14), 'serial formula');
+ is(
+ nv('_nvmet_parse_configfs', $FIXTURE_CFS)->{subsystems}->{$FIXTURE_NQN}->{attr_serial},
+ nv('_nvmet_serial', $FIXTURE_NQN),
+ 'the fixture subsystem carries the serial of its NQN',
+ );
+};
+
+subtest 'namespace ID allocation' => sub {
+ my $rows = sub(@ids) {
+ my $uuid = sub($id) { sprintf('%08d-0000-4000-8000-000000000000', $id) };
+ return inventory(
+ @POOL,
+ map { zfs_rows("tank/vm-$_-disk-0", 'volume', $NQN, $_, $uuid->($_)) } @ids,
+ );
+ };
+ my $cfs = cfs_model();
+ is(nv('_nvmet_allocate_nsid', inventory(@POOL), $cfs, $NQN), 1, 'first NSID');
+ is(nv('_nvmet_allocate_nsid', $rows->(1, 3), $cfs, $NQN), 4, 'a gap is not reused');
+ my $dirs =
+ cfs_model(subsystem => { namespaces => { 7 => ns_model($U{7}, '(null)', '0') } });
+ is(nv('_nvmet_allocate_nsid', $rows->(1), $dirs, $NQN), 8,
+ 'configfs-only namespaces count');
+ my $foreign =
+ inventory(@POOL, zfs_rows('tank/vm-9-disk-0', 'volume', $FOREIGN_NQN, 9, $U{9}));
+ my $foreign_cfs = cfs_model();
+ $foreign_cfs->{subsystems}->{$FOREIGN_NQN} = { namespaces => { 12 => {} } };
+ is(nv('_nvmet_allocate_nsid', $foreign, $foreign_cfs, $NQN), 1, 'others do not count');
+ my $invalid = inventory(@POOL, zfs_rows('tank/vm-1-disk-0', 'volume', $NQN, 'x', $U{1}));
+ is(nv('_nvmet_allocate_nsid', $invalid, $cfs, $NQN), 1, 'invalid NSIDs do not count');
+ my $top = inventory(
+ @POOL,
+ zfs_rows('tank/vm-1-disk-0', 'volume', $NQN, 1, $U{1}),
+ zfs_rows('tank/vm-2-disk-0', 'volume', $NQN, 0xfffffffe, $U{2}),
+ );
+ is(
+ nv('_nvmet_allocate_nsid', $top, $cfs, $NQN),
+ 2,
+ 'after the last NSID the lowest free one',
+ );
+ my $reserved = $rows->(1);
+ $reserved->{last_nsid} = 9;
+ is(nv('_nvmet_allocate_nsid', $reserved, $cfs, $NQN), 10, 'a reserved NSID is not reused');
+ $reserved->{last_nsid} = 0xfffffffe;
+ is(
+ nv('_nvmet_allocate_nsid', $reserved, $cfs, $NQN),
+ 2,
+ 'until the last NSID was handed out',
+ );
+};
+
+subtest 'export planner' => sub {
+ my $dev = '/dev/zvol/tank/vm-100-disk-0';
+ my $case = sub($namespaces) {
+ return cfs_model(subsystem => { namespaces => $namespaces });
+ };
+ my $build = build_steps(1, $U{1}, $dev);
+ my @plans = (
+ ['absent', {}, [$build], 'absent'],
+ ['present', { 1 => ns_model($U{1}, $dev) }, [], 'present'],
+ [
+ 'disabled',
+ { 1 => ns_model($U{1}, $dev, '0') },
+ [[['test', '-b', $dev], write_step("$S/namespaces/1/enable", 1)]],
+ 'disabled',
+ ],
+ [
+ 'disabled and incomplete',
+ { 1 => ns_model($U{9}, '(null)', '0') },
+ [[['rmdir', "$S/namespaces/1"], $build->@*]],
+ 'reclaim',
+ ],
+ [
+ 'enabled on the template name of the zvol',
+ { 1 => ns_model($U{1}, '/dev/zvol/tank/base-100-disk-0') },
+ [[
+ write_step("$S/namespaces/1/enable", 0),
+ ['rmdir', "$S/namespaces/1"],
+ $build->@*,
+ ]],
+ 'stale',
+ ],
+ );
+ for my $plan (@plans) {
+ my ($name, $namespaces, $units, $state) = $plan->@*;
+ is_deeply(
+ [nv('_nvmet_plan_export', $case->($namespaces), $NQN, 1, $U{1}, $dev)],
+ [$units, $state],
+ "plan for a namespace that is $name",
+ );
+ }
+ my $incomplete = $case->({ 1 => ns_model($U{9}, '(null)', '0') });
+ is(
+ nv('_nvmet_export_state', $incomplete, $NQN, 1, $U{1}, $dev),
+ 'incomplete',
+ 'a disabled namespace with another identity is incomplete',
+ );
+ is(
+ nv('_nvmet_export_state', cfs_model(no_subsystem => 1), $NQN, 1, $U{1}, $dev),
+ 'nosubsys',
+ 'no subsystem',
+ );
+ for my $refusal (
+ [
+ 'enabled with another UUID',
+ { 1 => ns_model($U{9}, $dev) },
+ qr/NVMe namespace ID '1' has a different identity/,
+ ],
+ [
+ 'enabled with another device',
+ { 1 => ns_model($U{1}, '/dev/zvol/tank/other') },
+ qr/different identity/,
+ ],
+ [
+ 'enabled with another UUID on the template name',
+ { 1 => ns_model($U{9}, '/dev/zvol/tank/base-100-disk-0') },
+ qr/different identity/,
+ ],
+ [
+ 'UUID in use at another NSID',
+ { 2 => ns_model($U{1}, '/dev/zvol/tank/other') },
+ qr/namespace UUID '\Q@{[ $U{1} ]}\E' is already in use/,
+ ],
+ ) {
+ my ($name, $namespaces, $error) = $refusal->@*;
+ eval { nv('_nvmet_plan_export', $case->($namespaces), $NQN, 1, $U{1}, $dev) };
+ like($@, $error, "refuses a namespace $name");
+ }
+ eval { nv('_nvmet_plan_export', cfs_model(no_subsystem => 1), $NQN, 1, $U{1}, $dev) };
+ like($@, qr/NVMe subsystem does not exist/, 'refuses without subsystem');
+};
+
+subtest 'unexport planner' => sub {
+ my $dev = '/dev/zvol/tank/vm-100-disk-0';
+ is_deeply(
+ [nv('_nvmet_plan_unexport', cfs_model(), $NQN, $U{1})],
+ [[], undef],
+ 'nothing to remove',
+ );
+ is_deeply(
+ [nv('_nvmet_plan_unexport', cfs_model(no_subsystem => 1), $NQN, $U{1})],
+ [[], undef],
+ 'no subsystem',
+ );
+ my ($units, $pre) = nv(
+ '_nvmet_plan_unexport',
+ cfs_model(subsystem => { namespaces => { 4 => ns_model($U{1}, $dev) } }),
+ $NQN,
+ $U{1},
+ );
+ is_deeply(
+ $units,
+ [[write_step("$S/namespaces/4/enable", 0), ['rmdir', "$S/namespaces/4"]]],
+ 'an enabled namespace is disabled, then removed',
+ );
+ ok($pre->{nsid} == 4 && $pre->{enabled}, 'and was enabled');
+ ($units, $pre) = nv(
+ '_nvmet_plan_unexport',
+ cfs_model(subsystem => { namespaces => { 4 => ns_model($U{1}, $dev, '0') } }),
+ $NQN,
+ $U{1},
+ );
+ is_deeply($units, [[['rmdir', "$S/namespaces/4"]]], 'a disabled namespace is only removed');
+ ok($pre->{nsid} == 4 && !$pre->{enabled}, 'and was disabled');
+ eval {
+ nv(
+ '_nvmet_plan_unexport',
+ cfs_model(
+ subsystem => {
+ namespaces =>
+ { 4 => ns_model($U{1}, $dev), 5 => ns_model($U{1}, $dev, '0') },
+ },
+ ),
+ $NQN,
+ $U{1},
+ );
+ };
+ like($@, qr/duplicate namespace UUID/, 'refuses a duplicate UUID');
+};
+
+subtest 'port lookup' => sub {
+ my $port = sub($family, $address, $service, %links) {
+ return {
+ addr_trtype => 'tcp',
+ addr_adrfam => $family,
+ addr_traddr => $address,
+ addr_trsvcid => $service,
+ links => {%links},
+ };
+ };
+ my $cfs = {
+ ports => {
+ 3 => $port->('ipv4', '192.0.2.21', '4420'),
+ 5 => $port->('ipv4', '192.0.2.21', '4420', $NQN => 1),
+ 6 => $port->('ipv4', '192.0.2.22', '4420'),
+ 7 => $port->('ipv6', '2001:db8:0:0::11', '4420'),
+ 8 => { $port->('ipv4', '192.0.2.23', '4420')->%*, addr_trtype => '' },
+ 9 => { $port->('ipv4', '', '4420')->%* },
+ },
+ };
+ my $find =
+ sub($nqn, @portal) { return nv('_nvmet_find_port', $cfs, $nqn, portal(@portal)) };
+ is($find->($NQN, 'ipv4', '192.0.2.21', 4420), 5, 'the linked port wins');
+ is($find->($FOREIGN_NQN, 'ipv4', '192.0.2.21', 4420), 3, 'else the lowest');
+ is($find->($NQN, 'ipv4', '192.0.2.22', 4420), 6, 'exact match');
+ ok(!defined($find->($NQN, 'ipv6', '192.0.2.22', 4420)), 'wrong family');
+ ok(!defined($find->($NQN, 'ipv4', '192.0.2.22', 4421)), 'wrong service');
+ ok(!defined($find->($NQN, 'ipv4', '192.0.2.23', 4420)), 'incomplete port');
+ is($find->($NQN, 'ipv6', '2001:db8::11', 4420), 7, 'IPv6 spellings match');
+};
+
+subtest 'activation planner' => sub {
+ my $conf = { nqn => $NQN, portals => $PORTALS, hostnqns => [@HOSTS], keysha => $KEY_SHA };
+ my $inv = inventory(@POOL, @ONE);
+ my $empty = { subsystems => {}, ports => {}, hosts => {} };
+ my $dev = '/dev/zvol/tank/vm-100-disk-0';
+ my $serial = nv('_nvmet_serial', $NQN);
+
+ # A creation interrupted after the model write is completed while the
+ # subsystem is still unused ('an interrupted subsystem creation' below).
+ my $unfinished = sub(%attr) {
+ return {
+ $empty->%*,
+ subsystems => {
+ $NQN => {
+ attr_serial => '0f1e2d3c4b5a6978',
+ attr_allow_any_host => '0',
+ acl => {},
+ namespaces => {},
+ %attr,
+ },
+ },
+ };
+ };
+ is_deeply(
+ nv(
+ '_nvmet_plan_activation',
+ $inv,
+ $unfinished->(attr_model => $MODEL, attr_allow_any_host => '1'),
+ $conf,
+ )->{prepublish}->[0],
+ [write_step("$S/attr_serial", $serial), write_step("$S/attr_allow_any_host", 0)],
+ 'an unused subsystem with our model is completed and denies unknown hosts',
+ );
+ for my $case (
+ [
+ 'a namespace',
+ sub($c) { $c->{subsystems}->{$NQN}->{namespaces}->{1} = ns_model($U{1}, $dev) },
+ ],
+ ['an ACL', sub($c) { $c->{subsystems}->{$NQN}->{acl}->{ $HOSTS[0] } = 1 }],
+ [
+ 'a port link',
+ sub($c) {
+ $c->{ports}->{7} = {
+ addr_trtype => 'tcp',
+ addr_adrfam => 'ipv4',
+ addr_traddr => '192.0.2.99',
+ addr_trsvcid => '4420',
+ links => { $NQN => 1 },
+ };
+ },
+ ],
+ ) {
+ my ($name, $change) = $case->@*;
+ my $used = $unfinished->(attr_model => $MODEL);
+ $change->($used);
+ eval { nv('_nvmet_plan_activation', $inv, $used, $conf) };
+ like(
+ $@,
+ qr/refusing to take over existing NVMe subsystem/,
+ "a subsystem with our model and another serial and $name is refused",
+ );
+ }
+ # the identities of every owned volume are validated before any unit
+ for my $case (
+ ['a duplicate NSID', [$NQN, 1, $U{2}], qr/duplicate NSID '1'/],
+ ['a duplicate UUID', [$NQN, 2, $U{1}], qr/duplicate namespace UUID/],
+ ['an invalid UUID', [$NQN, 2, 'bad'], qr/invalid namespace UUID/],
+ ) {
+ my ($name, $identity, $error) = $case->@*;
+ my $bad =
+ inventory(@POOL, @ONE, zfs_rows('tank/vm-101-disk-0', 'volume', $identity->@*));
+ eval { nv('_nvmet_desired_namespaces', $bad, $NQN) };
+ like($@, $error, "$name is refused");
+ }
+
+ my $state = sub(%o) {
+ my $cfs = cfs_model(
+ subsystem => {
+ acl => { map { $_ => 1 } @HOSTS },
+ namespaces => { 1 => ns_model($U{1}, $dev) },
+ ($o{subsystem} // {})->%*,
+ },
+ );
+ $cfs->{ports} = {
+ map {
+ $_ => {
+ addr_trtype => 'tcp',
+ addr_adrfam => 'ipv4',
+ addr_traddr => "192.0.2.2$_",
+ addr_trsvcid => '4420',
+ links => { $NQN => 1 },
+ }
+ } 1,
+ 2,
+ };
+ $cfs->{hosts} = { map { $_ => { key_sha256 => $KEY_SHA } } @HOSTS };
+ $o{change}->($cfs) if $o{change};
+ return $cfs;
+ };
+ is_deeply(
+ nv('_nvmet_plan_activation', $inv, $state->(), $conf),
+ { prepublish => [], port_ids => [1, 2] },
+ 'a converged target plans nothing',
+ );
+ is_deeply(
+ nv('_nvmet_plan_publish', $state->(), $NQN, [1, 2], [@HOSTS]),
+ [],
+ 'nor publishes',
+ );
+ is_deeply(
+ nv(
+ '_nvmet_plan_activation',
+ $inv,
+ $state->(subsystem => { attr_allow_any_host => '1' }),
+ $conf,
+ )->{prepublish},
+ [[write_step("$S/attr_allow_any_host", 0)]],
+ 'allow_any_host is reset',
+ );
+
+ # the key matrix for the second host
+ my $h = $HOSTS[1];
+ my $H = "$ROOT/hosts/$h";
+ my $other = sha256_hex("$KEY_B\n");
+ my $foreign_acl = sub($cfs) {
+ $cfs->{subsystems}->{$FOREIGN_NQN} = { acl => { $h => 1 }, namespaces => {} };
+ };
+ for my $case (
+ [
+ 'absent',
+ sub($c) { delete $c->{hosts}->{$h} },
+ [[['mkdir', $H]], [chmod_step($H), { key => ["$H/dhchap_key"] }]],
+ ],
+ ['matching', sub($c) { }, []],
+ [
+ 'different and unlinked',
+ sub($c) {
+ $c->{hosts}->{$h}->{key_sha256} = $other;
+ delete $c->{subsystems}->{$NQN}->{acl}->{$h};
+ },
+ [
+ [chmod_step($H), { key => ["$H/dhchap_key"] }],
+ [['ln', '-s', $H, "$S/allowed_hosts/$h"]],
+ ],
+ ],
+ [
+ 'empty and unlinked',
+ sub($c) {
+ $c->{hosts}->{$h}->{key_sha256} = $EMPTY_KEY_SHA;
+ delete $c->{subsystems}->{$NQN}->{acl}->{$h};
+ },
+ [
+ [chmod_step($H), { key => ["$H/dhchap_key"] }],
+ [['ln', '-s', $H, "$S/allowed_hosts/$h"]],
+ ],
+ ],
+ ) {
+ my ($name, $change, $units) = $case->@*;
+ my $plan = nv('_nvmet_plan_activation', $inv, $state->(change => $change), $conf);
+ is_deeply($plan->{prepublish}, $units, "key of a host that is $name");
+ }
+ for my $case (
+ [
+ 'different and linked by us',
+ sub($c) { $c->{hosts}->{$h}->{key_sha256} = $other },
+ qr/refusing to replace an in-use DH-HMAC-CHAP key/,
+ ],
+ [
+ 'different and linked elsewhere',
+ sub($c) {
+ $c->{hosts}->{$h}->{key_sha256} = $other;
+ delete $c->{subsystems}->{$NQN}->{acl}->{$h};
+ $foreign_acl->($c);
+ },
+ qr/in-use DH-HMAC-CHAP key/,
+ ],
+ [
+ 'empty and linked',
+ sub($c) { $c->{hosts}->{$h}->{key_sha256} = $EMPTY_KEY_SHA },
+ qr/in-use DH-HMAC-CHAP key/,
+ ],
+ [
+ 'without a digest',
+ sub($c) { $c->{hosts}->{$h}->{key_sha256} = undef },
+ qr/does not expose a DH-HMAC-CHAP key for host/,
+ ],
+ ) {
+ my ($name, $change, $error) = $case->@*;
+ eval { nv('_nvmet_plan_activation', $inv, $state->(change => $change), $conf) };
+ like($@, $error, "refuses the key of a host that is $name");
+ }
+
+ my $undesired = $state->(
+ change => sub($c) {
+ $c->{subsystems}->{$NQN}->{namespaces}->{7} =
+ ns_model($U{7}, '/dev/zvol/tank/gone');
+ $c->{subsystems}->{$NQN}->{namespaces}->{8} = ns_model($U{8}, '(null)', '0');
+ },
+ );
+ is_deeply(
+ nv('_nvmet_plan_activation', $inv, $undesired, $conf)->{prepublish},
+ [
+ [write_step("$S/namespaces/7/enable", 0), ['rmdir', "$S/namespaces/7"]],
+ [['rmdir', "$S/namespaces/8"]],
+ ],
+ 'undesired namespaces are disabled and removed',
+ );
+};
+
+subtest 'publish planner' => sub {
+ my $cfs = cfs_model(subsystem => { acl => { map { $_ => 1 } @HOSTS } });
+ $cfs->{ports} = {
+ 1 => { links => { $NQN => 1 } },
+ 2 => { links => {} },
+ 5 => { links => { $NQN => 1, $FOREIGN_NQN => 1 } },
+ 6 => { links => { $FOREIGN_NQN => 1 } },
+ };
+ my @guards = (
+ ['grep', '-qx', '0', "$S/attr_allow_any_host"],
+ map { ['test', '-L', "$S/allowed_hosts/$_"] } @HOSTS,
+ );
+ is_deeply(
+ nv('_nvmet_plan_publish', $cfs, $NQN, [1, 2], [@HOSTS]),
+ [
+ {
+ port => 2,
+ action => 'link',
+ steps => [@guards, ['ln', '-s', $S, "$ROOT/ports/2/subsystems/$NQN"]],
+ },
+ {
+ port => 5,
+ action => 'unlink',
+ steps => [['rm', "$ROOT/ports/5/subsystems/$NQN"]],
+ },
+ ],
+ 'links a missing port behind the guards and unlinks a stray port',
+ );
+ $cfs->{ports}->{2}->{links}->{$NQN} = 1;
+ delete $cfs->{ports}->{5};
+ is_deeply(nv('_nvmet_plan_publish', $cfs, $NQN, [1, 2], [@HOSTS]), [], 'converged');
+};
+
+subtest 'target removal planners' => sub {
+ my $conf_hosts =
+ [@HOSTS, 'nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-000000000003'];
+ my $cfs = cfs_model(subsystem => { acl => { map { $_ => 1 } @HOSTS } });
+ $cfs->{ports} = {
+ 2 => { links => { $NQN => 1 } },
+ 1 => { links => { $NQN => 1 } },
+ 3 => { links => {} },
+ };
+ is_deeply(
+ [nv('_nvmet_plan_delete_target', inventory(@POOL), $cfs, $NQN, $conf_hosts)],
+ [
+ [[
+ [
+ 'rm',
+ "$ROOT/ports/1/subsystems/$NQN",
+ "$ROOT/ports/2/subsystems/$NQN",
+ map { "$S/allowed_hosts/$_" } @HOSTS,
+ ],
+ ['rmdir', $S],
+ ]],
+ [sort $conf_hosts->@*],
+ ],
+ 'teardown removes port links first, then ACLs, then the subsystem',
+ );
+ # The candidates are the ACL and the configured hosts: a retry after a
+ # teardown that removed the ACLs but not the subsystem still finds them.
+ my $stale_host = 'nqn.2026-01.com.example:stale';
+ my $stale = cfs_model(subsystem => { acl => { $stale_host => 1 } });
+ is_deeply(
+ (nv('_nvmet_plan_delete_target', inventory(@POOL), $stale, $NQN, [@HOSTS]))[1],
+ [sort $stale_host, @HOSTS],
+ 'a host that is only in the ACL is a candidate too',
+ );
+ is_deeply(
+ [nv('_nvmet_plan_delete_target', inventory(@POOL), cfs_model(), $NQN, [@HOSTS])],
+ [[[['rmdir', $S]]], [sort @HOSTS]],
+ 'without ACLs the configured hosts are candidates',
+ );
+
+ my $orphans = {
+ subsystems => { $FOREIGN_NQN => { acl => { $HOSTS[0] => 1 } } },
+ hosts => { map { $_ => {} } @HOSTS },
+ };
+ is_deeply(
+ nv('_nvmet_plan_orphan_hosts', $orphans, [@HOSTS, @HOSTS, 'nqn.x:gone']),
+ [[['rmdir', "$ROOT/hosts/$HOSTS[1]"]]],
+ 'orphans exclude linked and missing hosts',
+ );
+ is_deeply(nv('_nvmet_plan_orphan_hosts', $orphans, [$HOSTS[0]]), [], 'no orphans');
+ $orphans->{hosts}->{'..'} = {};
+ is_deeply(
+ nv('_nvmet_plan_orphan_hosts', $orphans, ['..']),
+ [],
+ 'a host name read from the target is only used when it is an NQN',
+ );
+ my $odd = cfs_model(subsystem => { acl => { $HOSTS[0] => 1, '..' => 1 } });
+ eval { nv('_nvmet_plan_delete_target', inventory(@POOL), $odd, $NQN, [@HOSTS]) };
+ like($@, qr/it allows a malformed host name/, 'an ACL with a malformed name is refused');
+};
+
+subtest 'template name and chunking' => sub {
+ is(nv('_nvmet_template_name', 'tank/vm-100-disk-0'), 'tank/base-100-disk-0', 'vm to base');
+ is(nv('_nvmet_template_name', 'tank/sub/vm-1-disk-2'), 'tank/sub/base-1-disk-2', 'nested');
+ for my $name ('tank/base-100-disk-0', 'tank/subvol-100-disk-0', 'tank/xvm-1-disk-0') {
+ eval { nv('_nvmet_template_name', $name) };
+ like($@, qr/only VM zvols can become templates/, "refuses $name");
+ }
+
+ my @units = map { build_steps($_, $U{1}, "/dev/zvol/tank/vm-$_-disk-0") } 1 .. 400;
+ my $chunks = nv('_nvmet_chunk', \@units);
+ ok($chunks->@* > 1, 'large plans are split');
+ ok(
+ !grep({ length(nv('_nvmet_render', $_)) > 65536 } $chunks->@*),
+ 'no call exceeds 64 KiB',
+ );
+ ok(!grep({ $_->[0]->[0] ne 'test' || $_->@* % 6 } $chunks->@*), 'chunks hold whole units');
+ is_deeply([map { $_->@* } $chunks->@*], [map { $_->@* } @units], 'order is preserved');
+ is_deeply(nv('_nvmet_chunk', []), [], 'no units, no calls');
+ my $small = nv(
+ '_nvmet_chunk',
+ [@units[0 .. 3]],
+ length(nv('_nvmet_render', [map { $_->@* } @units[0 .. 1]])),
+ );
+ is(scalar($small->@*), 2, 'the limit is configurable');
+ eval { nv('_nvmet_chunk', [$units[0]], 10) };
+ like($@, qr/NVMe target command too long/, 'a unit over the limit is refused');
+ my $key_unit = [{ key => ["$ROOT/hosts/$HOSTS[0]/dhchap_key"] }];
+ my $mixed = nv('_nvmet_chunk', [@units[0 .. 199], $key_unit, @units[200 .. 399]]);
+ is(
+ scalar(
+ grep {
+ grep { ref($_) eq 'HASH' && $_->{key} } $_->@*
+ } $mixed->@*
+ ),
+ 1,
+ 'the key step is in exactly one call',
+ );
+};
+
+# ---------------------------------------------------------------------------
+# Flows against the fake target
+# ---------------------------------------------------------------------------
+
+sub secret_file($storeid) {
+ return "/etc/pve/priv/storage/$storeid.nvme-dhchap";
+}
+
+# The target part of a storage activation, with the key stored for the storage.
+sub activate($scfg = scfg(), $key = $KEY, $hostnqns = [@HOSTS], $portals = $PORTALS) {
+ local $FILES{ secret_file('st') } = $key;
+ return nv('_nvmet_activate_target', 'st', $scfg, $portals, $hostnqns, $key);
+}
+
+# The public operations of the flows, on lifecycle_fake().
+my %ACT = (
+ activate => sub { activate() },
+ alloc => sub { $PLUGIN->alloc_image('st', scfg(), 200, 'raw', undef, 1024) },
+ clone => sub { $PLUGIN->clone_image(scfg(), 'st', 'base-102-disk-0', 201) },
+ free => sub { $PLUGIN->free_image('st', scfg(), 'vm-101-disk-0') },
+ rollback =>
+ sub { $PLUGIN->volume_snapshot_rollback(scfg(), 'st', 'vm-100-disk-0', 'snap1') },
+ template => sub { $PLUGIN->create_base('st', scfg(), 'vm-101-disk-0') },
+ resize => sub { $PLUGIN->volume_resize(scfg(), 'st', 'vm-100-disk-0', 2 * 1024**3) },
+ remove => sub { nv('_nvmet_delete_target', scfg()) },
+);
+
+# Runs an action against a new lifecycle_fake(%opts).
+sub run_on($action, %opts) {
+ my $fake = lifecycle_fake(%opts);
+ return ($fake, flow($fake, $ACT{$action}));
+}
+
+# An action (a name of %ACT or a sub) that is refused without changing the target.
+sub refused($fake, $action, $error, $name) {
+ my $before = dclone($fake->{m});
+ my $res = flow($fake, ref($action) ? $action : $ACT{$action});
+ like($res->{error}, $error, "refuses $name");
+ ok(!changes($res) && same($before, $fake->{m}), "without changes for $name");
+ return $res;
+}
+
+sub ns_of($fake, $nsid, $nqn = $NQN) {
+ return ($fake->{m}->{subsystems}->{$nqn} // {})->{ns}->{$nsid};
+}
+
+sub exported($fake, $nsid, $uuid, $dev) {
+ my $ns = ns_of($fake, $nsid) // return 0;
+ return $ns->{enable} eq '1' && $ns->{device_uuid} eq $uuid && $ns->{device_path} eq $dev;
+}
+
+sub step_is($step, @prefix) {
+ return
+ ref($step) eq 'ARRAY'
+ && $step->@* >= @prefix
+ && join("\0", $step->@[0 .. $#prefix]) eq join("\0", @prefix);
+}
+
+sub write_to($step, $suffix, $value = undef) {
+ return
+ ref($step) eq 'HASH'
+ && exists($step->{write})
+ && $step->{write} =~ m{\Q$suffix\E\z}
+ && (!defined($value) || $step->{value} eq $value);
+}
+
+subtest 'path() is one read-only ZFS query' => sub {
+ my $fake = lifecycle_fake();
+ my $res = flow($fake, sub { [$PLUGIN->path(scfg(), 'vm-100-disk-0', 'st')] });
+ is($res->{error}, '', 'lookup succeeds');
+ is_deeply(
+ $res->{result},
+ ["/dev/disk/by-id/nvme-uuid.$U{1}", 100, 'images'],
+ 'the UUID link',
+ );
+ is_deeply(ops($res), ['read ZFS inventory'], 'one call');
+ is(scalar($res->{locks}->@*), 0, 'without the lock');
+ $res = flow($fake, sub { [$PLUGIN->path(scfg(), 'vm-100-disk-0', 'st', 'snap1')] });
+ like($res->{error}, qr/direct access to snapshots not implemented/, 'snapshots');
+ is(scalar($res->{calls}->@*), 0, 'are refused locally');
+};
+
+subtest 'a failed read is retried' => sub {
+ local $SLEPT = 0;
+ my $fake = lifecycle_fake(read_fault => { after => -1, mode => 'before' });
+ my $res = flow($fake, sub { [$PLUGIN->path(scfg(), 'vm-100-disk-0', 'st')] });
+ is($res->{error}, '', 'a read that fails once');
+ is_deeply(ops($res), ['read ZFS inventory', 'read ZFS inventory'], 'is repeated');
+ is($SLEPT, 0.2, 'after 200 ms');
+
+ $fake = lifecycle_fake(fail => [['zfs', 'get'], 'I/O error']);
+ $res = flow($fake, sub { [$PLUGIN->path(scfg(), 'vm-100-disk-0', 'st')] });
+ is($res->{error}, "cannot read NVMe target state: I/O error\n",
+ 'a read that keeps failing');
+ is(scalar($res->{calls}->@*), 3, 'is tried three times');
+
+ $fake = lifecycle_fake(read_fault => { after => -1, mode => 'unreachable' });
+ $res = flow($fake, sub { [$PLUGIN->path(scfg(), 'vm-100-disk-0', 'st')] });
+ like($res->{error}, qr/NVMe target '192\.0\.2\.10' is unreachable/,
+ 'an unreachable target');
+ is(scalar($res->{calls}->@*), 1, 'is not tried again');
+};
+
+subtest 'activate_volume' => sub {
+ my %forced;
+ my $activations = 0;
+ my $activation = sub($class, $storeid, $scfg, $cache = undef) {
+ $activations++;
+ $forced{$storeid} = $cache->{'zfsnvme-force-reconcile'}->{$storeid};
+ die "activation attempted\n";
+ };
+ $plugin_mock->redefine(activate_storage => $activation);
+
+ my $fake = lifecycle_fake();
+ delete $fake->{m}->{subsystems}->{$NQN}->{ns}->{2};
+ my $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-101-disk-0') });
+ like($res->{error}, qr/activation attempted/, 'a missing namespace forces an activation');
+ is($forced{st}, 1, 'the activation is forced past its fast path');
+ is_deeply(ops($res), ['read target state'], 'after one read');
+ ok(!$res->{calls}->[0]->{mutating} && !$res->{locks}->@*, 'without a change or the lock');
+
+ $fake = lifecycle_fake();
+ ns_of($fake, 2)->{device_uuid} = $U{9};
+ $activations = 0;
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-101-disk-0') });
+ like(
+ $res->{error},
+ qr/NVMe namespace ID '2' has a different identity/,
+ 'refuses an enabled namespace with another identity',
+ );
+ is($activations, 0, 'without activating');
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-101-disk-0', 'snap') });
+ like($res->{error}, qr/unable to activate snapshot/, 'snapshots are refused');
+
+ # The local device: its udev link and whether it is the namespace.
+ my $ok = 1;
+ $plugin_mock->redefine(_nvmet_local_namespace_ok => sub(@args) { return $ok });
+ my $link = "/dev/disk/by-id/nvme-uuid.$U{1}";
+ local %BLOCK = ($link => 1);
+ $fake = target_fake();
+ add_volume($fake, 'vm-100-disk-0', 1, $U{1});
+ $activations = 0;
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-100-disk-0') });
+ is($res->{error}, '', 'a converged volume activates');
+ is_deeply(ops($res), ['read target state'], 'with one read');
+ is(scalar($res->{locks}->@*) + $activations, 0, 'without lock or activation');
+
+ for my $case (
+ ['disabled', sub($f) { ns_of($f, 1)->{enable} = '0' }],
+ ['absent', sub($f) { delete $f->{m}->{subsystems}->{$NQN}->{ns}->{1} }],
+ ['without subsystem', sub($f) { delete $f->{m}->{subsystems}->{$NQN} }],
+ [
+ 'on the template name of the zvol',
+ sub($f) { ns_of($f, 1)->{device_path} = '/dev/zvol/tank/base-100-disk-0' },
+ ],
+ [
+ 'being built',
+ sub($f) {
+ $f->{m}->{subsystems}->{$NQN}->{ns}->{1} = ns_model($U{9}, '(null)', '0');
+ },
+ ],
+ ) {
+ my ($name, $change) = $case->@*;
+ $fake = target_fake();
+ add_volume($fake, 'vm-100-disk-0', 1, $U{1});
+ $change->($fake);
+ %forced = ();
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-100-disk-0') });
+ is($forced{st}, 1, "a namespace that is $name on the target is exported again");
+ }
+
+ $fake = target_fake();
+ add_volume($fake, 'vm-100-disk-0', 1, $U{1});
+ $ok = 0;
+ $activations = 0;
+ $plugin_mock->redefine(activate_storage => sub(@args) { $activations++; return 1 });
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-100-disk-0') });
+ like(
+ $res->{error},
+ qr/NVMe namespace for 'vm-100-disk-0' has an unexpected identity/,
+ 'a local device of another namespace is never used',
+ );
+ is($activations, 1, 'after one forced activation');
+ is_deeply(ops($res), ['read target state'], 'and one read');
+ $plugin_mock->unmock('_nvmet_local_namespace_ok');
+
+ # While the device is missing, live controllers are rescanned.
+ my $file_mock = Test::MockModule->new('PVE::File');
+ $file_mock->redefine(
+ dir_glob_foreach => sub($dir, $regex, $func) {
+ return if $dir ne '/sys/class/nvme';
+ $func->($_) for qw(nvme7 nvme8 nvme9);
+ },
+ );
+ local %FILES = (
+ '/sys/class/nvme/nvme7/subsysnqn' => $NQN,
+ '/sys/class/nvme/nvme7/state' => 'live',
+ '/sys/class/nvme/nvme8/subsysnqn' => $NQN,
+ '/sys/class/nvme/nvme8/state' => 'connecting',
+ '/sys/class/nvme/nvme9/subsysnqn' => $FOREIGN_NQN,
+ '/sys/class/nvme/nvme9/state' => 'live',
+ );
+ $plugin_mock->redefine(activate_storage => sub(@args) { return 1 });
+ $fake = lifecycle_fake();
+ delete $fake->{m}->{subsystems}->{$NQN}->{ns}->{2};
+ my $start = $NOW;
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-101-disk-0') });
+ like(
+ $res->{error},
+ qr/NVMe namespace for 'vm-101-disk-0' did not appear/,
+ 'a missing device',
+ );
+ is($NOW - $start, 10, 'after 10 seconds');
+ my $rescan = ['/sys/class/nvme/nvme7/rescan_controller', "1\n"];
+ is_deeply(
+ $res->{sysfs_writes},
+ [($rescan) x 5],
+ 'the live controller is rescanned every 2 s',
+ );
+ is_deeply($res->{warnings}, [], 'without a warning');
+
+ # The stock file_write on files that fail: a rescan that fails is a task
+ # warning with the errno text, and a controller that went away since it
+ # was listed needs none.
+ my $file_write = $sysfs_mock->original('file_write');
+ my $dir = tempdir(CLEANUP => 1);
+ my @perl_warnings;
+ local $SIG{__WARN__} = sub($warning) { push @perl_warnings, $warning };
+ for my $case (
+ ['/dev/full', 'No space left on device'],
+ [$dir, 'Is a directory'],
+ ["$dir/gone/rescan_controller", undef],
+ ) {
+ my ($path, $reason) = $case->@*;
+ $sysfs_mock->redefine(file_write => sub($file, @args) { $file_write->($path, @args) });
+ $res = flow($fake, sub { $PLUGIN->activate_volume('st', scfg(), 'vm-101-disk-0') });
+ is_deeply(
+ $res->{warnings},
+ defined($reason) ? [("cannot rescan NVMe controller 'nvme7': $reason") x 5] : [],
+ defined($reason)
+ ? "a failed rescan is a task warning with the errno text ($reason)"
+ : 'a controller that went away is not',
+ );
+ }
+ is_deeply(\@perl_warnings, [], 'and file_write warns about none');
+ $sysfs_mock->redefine(file_write => $sysfs_write);
+ $file_mock->unmock_all();
+ $plugin_mock->unmock('activate_storage');
+};
+
+subtest 'activation of an empty target' => sub {
+ my $fake = FakeTarget->new(pools => ['tank']);
+ $fake->add_zvol('tank/vm-100-disk-0', identity => [$NQN, 1, $U{1}]);
+ my $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'the target is configured');
+ is_deeply(
+ [map { $_->{locked} } $res->{calls}->@*],
+ [0, 1, 1, 1, 1, 1],
+ 'read, then lock, read, apply, verify and publish',
+ );
+ is_deeply(
+ [map { [$_->@{qw(name timeout)}] } $res->{locks}->@*],
+ [['zfsnvme-192.0.2.10', 30]],
+ 'under the domain lock of the target, waiting up to 30 seconds for it',
+ );
+ my $m = $fake->{m};
+ my $subsys = $m->{subsystems}->{$NQN};
+ is_deeply(
+ $subsys->{attr},
+ {
+ attr_model => $MODEL,
+ attr_serial => nv('_nvmet_serial', $NQN),
+ attr_allow_any_host => '0',
+ },
+ 'the subsystem carries our marker',
+ );
+ is_deeply($subsys->{acl}, { map { $_ => 1 } @HOSTS }, 'every host has an ACL');
+ is_deeply([map { $m->{hosts}->{$_}->{key} } @HOSTS], [$KEY, $KEY], 'and the key');
+ ok(exported($fake, 1, $U{1}, '/dev/zvol/tank/vm-100-disk-0'), 'the volume is exported');
+
+ $res = flow($fake, $ACT{activate});
+ is_deeply(ops($res), ['read target state'], 'a converged target costs one read');
+ ok(!$res->{locks}->@* && !$res->{calls}->[0]->{mutating}, 'without lock or change');
+};
+
+subtest 'activation refusals change nothing' => sub {
+ for my $case (
+ [
+ 'a foreign subsystem at our NQN',
+ sub($f) { $f->add_subsystem($NQN, model => 'Other Storage', serial => 'abc') },
+ qr/refusing to take over existing NVMe subsystem/,
+ ],
+ [
+ 'a used subsystem with the default model',
+ sub($f) {
+ $f->add_host($HOSTS[0], $KEY);
+ $f->add_subsystem($NQN, model => 'Linux', serial => 'abc', acl => [$HOSTS[0]]);
+ },
+ qr/refusing to take over existing NVMe subsystem/,
+ ],
+ [
+ 'a duplicate UUID on a published target',
+ sub($f) {
+ $f->{m} = lifecycle_fake()->{m};
+ $f->add_zvol('tank/vm-150-disk-0', identity => [$NQN, 7, $U{5}]);
+ $f->add_zvol('tank/vm-151-disk-0', identity => [$NQN, 8, $U{5}]);
+ },
+ qr/duplicate namespace UUID '\Q@{[ $U{5} ]}\E'/,
+ ],
+ [
+ 'an enabled namespace of another volume',
+ sub($f) {
+ $f->add_zvol('tank/vm-100-disk-0', identity => [$NQN, 1, $U{1}]);
+ $f->add_subsystem($NQN);
+ $f->add_namespace($NQN, 1, $U{9}, '/dev/zvol/tank/other');
+ },
+ qr/NVMe namespace ID '1' has a different identity/,
+ ],
+ ) {
+ my ($name, $setup, $error) = $case->@*;
+ my $fake = FakeTarget->new(pools => ['tank']);
+ $setup->($fake);
+ my $res = refused($fake, 'activate', $error, $name);
+ unlike($res->{error}, qr/DHHC-1/, 'without key material in the message');
+ }
+};
+
+subtest 'nothing is published unless a fresh read converges' => sub {
+ my $acl = "$S/allowed_hosts/$HOSTS[1]";
+ my $failing = sub($limit) {
+ my $failures = 0;
+ return sub($step, $call) {
+ return undef
+ if !step_is($step, 'ln', '-s') || $step->[3] ne $acl || $failures >= $limit;
+ $failures++;
+ return "ln: failed to create symbolic link '$acl': Invalid argument";
+ };
+ };
+ my $fake = FakeTarget->new(pools => ['tank'], step_fault => $failing->(99));
+ my $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/NVMe target did not converge: ln: failed to create symbolic link/,
+ 'a plan that fails twice stops',
+ );
+ is(
+ scalar(grep { $_->{op} eq 'configure NVMe target' } $res->{calls}->@*),
+ 2,
+ 'in two rounds',
+ );
+ is(scalar(grep { $_->{op} =~ /\Alink / } $res->{calls}->@*), 0, 'and publishes nothing');
+ ok(!grep({ $_->{links}->%* } values $fake->{m}->{ports}->%*),
+ 'no port links the subsystem');
+
+ $fake = FakeTarget->new(pools => ['tank'], step_fault => $failing->(1));
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'a transient failure is repaired by the second round');
+ is(scalar(grep { $_->{op} =~ /\Alink / } $res->{calls}->@*), 2, 'which then publishes');
+
+ $fake = FakeTarget->new(pools => ['tank']);
+ $fake->{m}->{mounted} = 0;
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'configfs is mounted when it is missing');
+ my @seen = grep { /mount/ } ops($res)->@*;
+ is_deeply(\@seen, ['read mounts', 'mount configfs'], 'after reading the mount table');
+ is(
+ $res->{calls}->[-1]->{op},
+ 'link NVMe subsystem on port 2',
+ 'and the target is published',
+ );
+
+ $fake = FakeTarget->new(pools => ['tank']);
+ $fake->{step_fault} = sub($step, $call) {
+ return step_is($step, 'env') ? "find: '$ROOT/hosts': Permission denied" : undef;
+ };
+ $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/cannot read NVMe target state: find: '\Q$ROOT\E\/hosts': Permission denied/,
+ 'an unreadable, mounted configfs stops the activation',
+ );
+ ok(!grep({ $_ eq 'mount configfs' } ops($res)->@*), 'without mounting');
+ is(changes($res), 0, 'and without changes');
+};
+
+subtest 'an interrupted subsystem creation' => sub {
+ # The chain that creates the subsystem stops after its mkdir or after the
+ # model write, for example when the remote shell is killed. Call 2 is the
+ # configuration. Its outcome is unknown; the next activation repairs it.
+ my $marker = {
+ attr_model => $MODEL,
+ attr_serial => nv('_nvmet_serial', $NQN),
+ attr_allow_any_host => '0',
+ };
+ my $completed = sub($fake, $what) {
+ is_deeply($fake->{m}->{subsystems}->{$NQN}->{attr}, $marker, "$what: our marker");
+ ok(
+ exported($fake, 1, $U{1}, '/dev/zvol/tank/vm-100-disk-0'),
+ "$what: exports the volume",
+ );
+ ok(
+ $fake->{m}->{ports}->{1}->{links}->{$NQN}
+ && $fake->{m}->{ports}->{2}->{links}->{$NQN},
+ "$what: and publishes it",
+ );
+ is_deeply($fake->{violations}, [], "$what: no protocol violation");
+ };
+ my $new = sub(%faults) {
+ my $fake = FakeTarget->new(pools => ['tank'], faults => {%faults});
+ $fake->add_zvol('tank/vm-100-disk-0', identity => [$NQN, 1, $U{1}]);
+ return $fake;
+ };
+
+ # After the model write, only this plugin can have created the subsystem.
+ my $fake = $new->(2 => 'cut:2');
+ my $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/'configure NVMe target' did not complete .* state is unknown\n\z/,
+ 'a cut after the model write',
+ );
+ ok(!grep({ $_->{links}->%* } values $fake->{m}->{ports}->%*), 'is unpublished');
+ $fake->{faults} = {};
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'the next activation completes the subsystem');
+ my ($configure) = grep { $_->{op} eq 'configure NVMe target' } $res->{calls}->@*;
+ is_deeply(
+ $configure->{steps}->[0],
+ write_step("$S/attr_serial", $marker->{attr_serial}),
+ 'by writing the serial',
+ );
+ $completed->($fake, 'next activation after the model write');
+
+ # Right after the mkdir, the subsystem looks like one of another tool.
+ $fake = $new->(2 => 'cut:1');
+ $res = flow($fake, $ACT{activate});
+ like($res->{error}, qr/state is unknown\n\z/, 'a cut right after the mkdir');
+ $fake->{faults} = {};
+ $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/refusing to take over existing NVMe subsystem .* not created by Proxmox VE/,
+ 'is refused by the next activation',
+ );
+ is($fake->{m}->{subsystems}->{$NQN}->{attr}->{attr_model}, 'Linux',
+ 'the subsystem is kept');
+ ok(!grep({ $_->{links}->%* } values $fake->{m}->{ports}->%*), 'and unpublished');
+ delete $fake->{m}->{subsystems}->{$NQN};
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'once an administrator removed it, the next activation creates it');
+ $completed->($fake, 'activation after the removal');
+};
+
+subtest 'quorum and reachability' => sub {
+ my $fake = FakeTarget->new(pools => ['tank']);
+ local $QUORATE = 0;
+ local @QUORUM = ();
+ my $res = flow($fake, $ACT{activate});
+ like($res->{error}, qr/cluster not quorate - refusing NVMe target changes/, 'no quorum');
+ is_deeply(\@QUORUM, [1], 'with a quorum check that does not wait');
+ is(scalar($res->{locks}->@*), 0, 'before requesting the lock');
+ is_deeply(ops($res), ['read target state'], 'after the lockless read');
+ $QUORATE = 1;
+
+ $fake->{unreachable} = 1;
+ $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/NVMe target '192\.0\.2\.10' is unreachable: ssh: connect/,
+ 'an unreachable target fails in the lockless read',
+ );
+ ok(!$res->{locks}->@* && $res->{calls}->@* == 1, 'without lock or retry');
+
+ $fake = lifecycle_fake();
+ local $QUORATE = 0;
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'a converged target needs no quorum');
+};
+
+subtest 'create' => sub {
+ my ($fake, $res) = run_on('alloc');
+ is($res->{error}, '', 'allocation succeeds');
+ is($res->{result}, 'vm-200-disk-0', 'with the next free name');
+ is_deeply(
+ [map { $_->{locked} } $res->{calls}->@*],
+ [0, 1, 1, 1, 1, 1, 1],
+ 'the name lookup without the lock, the rest under it',
+ );
+ my $uuid = $fake->{m}->{ds}->{'tank/vm-200-disk-0'}->{props}->{'proxmox:nvme-uuid'};
+ like($uuid, $UUID_RE, 'a new UUID');
+ ok(exported($fake, 4, $uuid, '/dev/zvol/tank/vm-200-disk-0'), 'exported at the next NSID');
+
+ my $thick = scfg();
+ delete $thick->@{qw(sparse blocksize)};
+ $fake = lifecycle_fake();
+ $res =
+ flow($fake, sub { $PLUGIN->alloc_image('st', $thick, 200, 'raw', 'vm-200-disk-1', 4000) });
+ is($res->{result}, 'vm-200-disk-1', 'an explicit name is used');
+ my ($create) = grep { $_->{op} eq 'create zvol' } $res->{calls}->@*;
+ is_deeply(
+ [grep { /\A(?:-[sb]|[0-9]+k)\z/ } $create->{steps}->[0]->@*],
+ ['4096k'],
+ 'the size is rounded up to KiB; without sparse and blocksize the zvol is thick',
+ );
+
+ ($fake, $res) = run_on('alloc', udev_delay => 3);
+ is($res->{error}, '', 'a slow udev');
+ is(scalar(grep { $_ eq 'wait for zvol' } ops($res)->@*), 4, 'adds polls');
+
+ local @UUIDS = ($U{1}, $U{9});
+ ($fake, $res) = run_on('alloc');
+ is(
+ $fake->{m}->{ds}->{'tank/vm-200-disk-0'}->{props}->{'proxmox:nvme-uuid'},
+ $U{9},
+ 'a UUID that is in use is drawn again',
+ );
+ local @UUIDS = ($U{1}) x 10;
+ ($fake, $res) = run_on('alloc');
+ is($res->{error}, "cannot generate an unused namespace UUID\n", 'but not forever');
+ is(changes($res), 0, 'and nothing changes');
+ @UUIDS = ();
+
+ ($fake, $res) = run_on('alloc', faults => { 3 => 'after' });
+ like(
+ $res->{error},
+ qr/'create zvol' did not complete .*state is unknown\n\z/,
+ 'a lost reply aborts even when the create reached the target',
+ );
+ is($res->{calls}->[-1]->{op}, 'create zvol', 'no read tries to settle it');
+ ok(!ns_of($fake, 4), 'the abandoned operation does not export it');
+ $fake->{faults} = {};
+ $res = flow($fake, $ACT{activate});
+ ok(!$res->{error} && ns_of($fake, 4), 'the next activation exports the owned zvol');
+
+ ($fake, $res) = run_on('alloc', faults => { 5 => 'before' });
+ like($res->{error}, qr/'export namespace' failed: injected failure/, 'a failed export');
+ ok(!$fake->{m}->{ds}->{'tank/vm-200-disk-0'} && !ns_of($fake, 4), 'destroys the new zvol');
+ is($res->{locks}->@* + $res->{nested}->@*, 1, 'the compensation reuses the lock');
+
+ # The export may still run on the target after its connection broke.
+ ($fake, $res) = run_on('alloc', faults => { 5 => 'after' });
+ like(
+ $res->{error},
+ qr/\A.*'export namespace' did not complete \(Connection.*state is unknown\n\z/,
+ 'a lost reply of the export leaves the state unknown',
+ );
+ ok(!grep({ /destroy|remove/ } ops($res)->@*), 'and is not compensated');
+ ok($fake->{m}->{ds}->{'tank/vm-200-disk-0'} && ns_of($fake, 4), 'so the disk is kept');
+
+ ($fake, $res) = run_on(
+ 'alloc',
+ before_call => sub($f, $call) {
+ $f->add_zvol('tank/vm-777-disk-0', identity => [$NQN, 4, $U{9}])
+ if $call->{index} == 6 && !$f->{m}->{ds}->{'tank/vm-777-disk-0'};
+ },
+ );
+ like(
+ $res->{error},
+ qr/duplicate NVMe identity on 'tank\/vm-200-disk-0'/,
+ 'a duplicate that appears before the verify read is refused',
+ );
+ ok(
+ !$fake->{m}->{ds}->{'tank/vm-200-disk-0'} && !ns_of($fake, 4),
+ 'the new zvol is destroyed',
+ );
+ ok($fake->{m}->{ds}->{'tank/vm-777-disk-0'}, 'the other zvol is kept');
+
+ ($fake, $res) = run_on(
+ 'alloc',
+ before_call => sub($f, $call) {
+ my $ds = $f->{m}->{ds}->{'tank/vm-200-disk-0'};
+ $ds->{props}->{'proxmox:nvme-uuid'} = $U{9} if $ds && $call->{index} == 6;
+ },
+ );
+ is(
+ $res->{error},
+ "NVMe identity of zvol 'tank/vm-200-disk-0' changed\n",
+ 'an identity that changes before the verify read is refused',
+ );
+ ok(!ns_of($fake, 4), 'the new namespace is removed');
+ ok($fake->{m}->{ds}->{'tank/vm-200-disk-0'}, 'but a zvol without our identity is kept');
+
+ # A namespace of another zvol that was renamed away while exported uses
+ # the device path of the new zvol: only what this call built is removed.
+ ($fake, $res) = run_on(
+ 'alloc',
+ faults => { 5 => 'before' },
+ before_call => sub($f, $call) {
+ $f->add_namespace($NQN, 9, $U{9}, '/dev/zvol/tank/vm-200-disk-0')
+ if $call->{index} == 0;
+ },
+ );
+ like($res->{error}, qr/'export namespace' failed: injected failure/, 'a failed export');
+ ok(
+ exported($fake, 9, $U{9}, '/dev/zvol/tank/vm-200-disk-0'),
+ 'keeps an enabled namespace of another UUID with the same device path',
+ );
+ ok(
+ !grep({ grep { write_to($_, '/namespaces/9/enable') } $_->{steps}->@* }
+ $res->{calls}->@*),
+ 'without touching it',
+ );
+
+ ($fake, $res) = run_on(
+ 'alloc',
+ faults => { 5 => 'before' },
+ fail => [['zfs', 'destroy'], "cannot destroy 'tank/vm-200-disk-0': dataset is busy"],
+ );
+ like(
+ $res->{error},
+ qr/'export namespace' failed: injected failure/,
+ 'keeps the first error',
+ );
+ my $cleanup = qr/\Afailed to clean up zvol 'vm-200-disk-0': NVMe target operation/;
+ is(
+ scalar(grep { /$cleanup 'destroy zvol' failed: cannot destroy/ } $res->{warnings}->@*),
+ 1,
+ 'a failed cleanup is reported',
+ );
+
+ # ssh gives up on a call, which the target still runs to its end later
+ ($fake, $res) = run_on('alloc', faults => { 3 => 'late' });
+ is(
+ $res->{error},
+ "NVMe target operation 'create zvol' did not complete (timeout);"
+ . " the target state is unknown\n",
+ 'a create that did not complete in time',
+ );
+ is($res->{calls}->[-1]->{op}, 'create zvol', 'is not followed by anything');
+ my $second =
+ flow($fake, sub { $PLUGIN->alloc_image('st', scfg(), 201, 'raw', undef, 1024) });
+ is($second->{error}, '', 'another allocation meanwhile');
+ $fake->land;
+ is_deeply(
+ [map { owned_volumes($fake->{m})->{"tank/vm-$_-disk-0"}->[0] } 200, 201],
+ [4, 5],
+ 'never gets the NSID of the create that completes afterwards',
+ );
+ $res = flow($fake, $ACT{activate});
+ ok(
+ !$res->{error} && ns_of($fake, 4) && ns_of($fake, 5),
+ 'and the next activation exports both',
+ );
+
+ ($fake, $res) = run_on(
+ 'alloc',
+ before_call =>
+ sub($f, $call) { die "received interrupt\n" if $call->{op} eq 'export namespace' },
+ );
+ is($res->{error}, "received interrupt\n", 'a stopped task');
+ ok(!grep({ /destroy|remove/ } ops($res)->@*), 'is not compensated either');
+
+ ($fake, $res) = run_on('alloc', read_fault => { after => 5, mode => 'failed' });
+ is(
+ $res->{error},
+ "NVMe target operation 'read target state' did not complete (ssh failed);"
+ . " the target state is unknown\n",
+ 'a read under the lock that ssh did not run to its end',
+ );
+ ok(!grep({ /destroy|remove/ } ops($res)->@*), 'is not followed by a compensation');
+
+ $fake = lifecycle_fake();
+ $fake->add_zvol('tank/vm-200-disk-0');
+ my $named = sub { $PLUGIN->alloc_image('st', scfg(), 200, 'raw', 'vm-200-disk-0', 1024) };
+ refused($fake, $named, qr/zvol 'tank\/vm-200-disk-0' already exists/, 'an existing zvol');
+ $fake = lifecycle_fake();
+ delete $fake->{m}->{subsystems}->{$NQN};
+ refused(
+ $fake,
+ 'alloc',
+ qr/NVMe subsystem does not exist; activate the storage first/,
+ 'an allocation without the subsystem',
+ );
+ $res =
+ flow($fake, sub { $PLUGIN->alloc_image('st', scfg(), 200, 'raw', 'vm-201-disk-0', 1024) });
+ like($res->{error}, qr/illegal name 'vm-201-disk-0'/, 'names must belong to the VM');
+};
+
+subtest 'clone' => sub {
+ my ($fake, $res) = run_on('clone');
+ is($res->{error}, '', 'a linked clone');
+ is($res->{result}, 'base-102-disk-0/vm-201-disk-0', 'is named after its base');
+ my $clone = $fake->{m}->{ds}->{'tank/vm-201-disk-0'};
+ is($clone->{origin}, 'tank/base-102-disk-0@__base__', 'from the base snapshot');
+ my $uuid = $clone->{props}->{'proxmox:nvme-uuid'};
+ ok(exported($fake, 4, $uuid, '/dev/zvol/tank/vm-201-disk-0'), 'with its own identity');
+
+ $fake = lifecycle_fake();
+ $res =
+ flow($fake, sub { $PLUGIN->clone_image(scfg(), 'st', 'base-102-disk-0', 201, 'missing') });
+ like(
+ $res->{error},
+ qr/'create zvol' failed: cannot open 'tank\/base-102-disk-0\@missing'/,
+ 'a missing snapshot',
+ );
+ ok(!$fake->{m}->{ds}->{'tank/vm-201-disk-0'}, 'leaves nothing behind');
+ $res = flow($fake, sub { $PLUGIN->clone_image(scfg(), 'st', 'base-102-disk-0', 201, '') });
+ is($res->{error}, '', 'an empty snapshot name');
+ is(
+ $fake->{m}->{ds}->{'tank/vm-201-disk-0'}->{origin},
+ 'tank/base-102-disk-0@__base__',
+ 'means the base snapshot',
+ );
+
+ for my $case (
+ ['basevol-102-disk-0', qr/clone_image requires a ZFS volume/],
+ ['vm-100-disk-0', qr/clone_image only works on base images/],
+ ) {
+ $res = flow($fake, sub { $PLUGIN->clone_image(scfg(), 'st', $case->[0], 201) });
+ like($res->{error}, $case->[1], "refuses $case->[0]");
+ is(scalar($res->{calls}->@*), 0, 'locally');
+ }
+ $res =
+ flow($fake, sub { $PLUGIN->clone_image(scfg(), 'st', 'base-102-disk-0', 201, 'a b') });
+ like($res->{error}, qr/invalid snapshot name/, 'refuses a malformed snapshot name');
+};
+
+subtest 'destroy' => sub {
+ my $dev = '/dev/zvol/tank/vm-101-disk-0';
+ my ($fake, $res) = run_on('free');
+ is($res->{error}, '', 'a volume is freed');
+ is_deeply(ops($res), ['read target state', 'destroy zvol'], 'in two calls');
+ ok(!$fake->{m}->{ds}->{'tank/vm-101-disk-0'} && !ns_of($fake, 2), 'both are gone');
+ $res = flow($fake, $ACT{free});
+ is($res->{error}, '', 'a missing volume is already freed');
+ is(changes($res), 0, 'without changes');
+
+ for my $case (
+ [
+ 'a volume that is not owned',
+ 'vm-900-disk-0',
+ undef,
+ qr/is not owned by NVMe subsystem/,
+ ],
+ [
+ 'a UUID exported at another NSID',
+ 'vm-101-disk-0',
+ sub($f) { $f->add_namespace($NQN, 9, $U{2}, '/dev/zvol/tank/x', 0) },
+ qr/namespace UUID '\Q@{[ $U{2} ]}\E' is already in use/,
+ ],
+ [
+ 'another identity at its NSID',
+ 'vm-101-disk-0',
+ sub($f) { ns_of($f, 2)->{device_uuid} = $U{9} },
+ qr/NVMe namespace ID '2' has a different identity/,
+ ],
+ [
+ 'a duplicate identity',
+ 'vm-101-disk-0',
+ sub($f) { $f->add_zvol('tank/vm-104-disk-0', identity => [$NQN, 2, $U{9}]) },
+ qr/duplicate NVMe identity/,
+ ],
+ ) {
+ my ($name, $volume, $change, $error) = $case->@*;
+ $fake = lifecycle_fake();
+ $change->($fake) if $change;
+ refused($fake, sub { $PLUGIN->free_image('st', scfg(), $volume) }, $error, $name);
+ }
+ $res = flow($fake, sub { $PLUGIN->free_image('st', scfg(), 'subvol-100-disk-0') });
+ like($res->{error}, qr/free_image requires a ZFS volume/, 'subvolumes are refused');
+
+ my $busy = 'Device or resource busy';
+ ($fake, $res) =
+ run_on('free', fail => [{ write => '/namespaces/2/enable', value => 0 }, $busy]);
+ like($res->{error}, qr/cannot disable namespace '2': .*busy/, 'a failed disable');
+ ok(exported($fake, 2, $U{2}, $dev), 'keeps the namespace');
+
+ ($fake, $res) = run_on('free', fail => [['rmdir', "$S/namespaces/2"], $busy]);
+ like(
+ $res->{error},
+ qr/cannot remove namespace '2': Device or resource busy\n\z/,
+ 'a failed rmdir',
+ );
+ ok(exported($fake, 2, $U{2}, $dev), 're-enables the namespace');
+ is($res->{calls}->[-1]->{op}, 'enable namespace', 'from a fresh read');
+
+ my $rmdir_and_enable = sub($s) {
+ step_is($s, 'rmdir', "$S/namespaces/2") || write_to($s, '/namespaces/2/enable', 1);
+ };
+ ($fake, $res) = run_on('free', fail => [$rmdir_and_enable, $busy]);
+ like(
+ $res->{error},
+ qr/cannot remove namespace '2': .*; cannot restore namespace: /,
+ 'a failed restore is reported too',
+ );
+
+ my $destroy = ['zfs', 'destroy', '-r', 'tank/vm-101-disk-0'];
+ local $SLEPT = 0;
+ ($fake, $res) = run_on('free', fail => [$destroy, $busy]);
+ like(
+ $res->{error},
+ qr/\Afailed to destroy 'tank\/vm-101-disk-0': Device or resource busy; namespace restored/,
+ 'a zvol that stays busy',
+ );
+ is(scalar(grep { $_ eq 'destroy zvol' } ops($res)->@*), 6, 'is destroyed six times');
+ is($SLEPT, 5, 'a second apart');
+ ok(exported($fake, 2, $U{2}, $dev), 'and its namespace is restored');
+
+ ($fake, $res) = run_on('free', fail => [$destroy, $busy, 3]);
+ is($res->{error}, '', 'a zvol that becomes free is destroyed by a retry');
+ ok(!$fake->{m}->{ds}->{'tank/vm-101-disk-0'}, 'and is gone');
+
+ ($fake, $res) = run_on('free', fail => [$destroy, $busy, 1], faults => { 3 => 'after' });
+ like(
+ $res->{error},
+ qr/'destroy zvol' did not complete .* state is unknown\n\z/,
+ 'a retry whose reply is lost',
+ );
+ is($res->{calls}->[-1]->{op}, 'destroy zvol', 'is not followed by anything');
+
+ ($fake, $res) = run_on(
+ 'free',
+ fail => [$destroy, $busy],
+ read_fault => { after => 1, mode => 'unreachable' },
+ );
+ is(
+ $res->{error},
+ "failed to destroy 'tank/vm-101-disk-0': Device or resource busy\n",
+ 'a failed destroy whose outcome cannot be read is a failure',
+ );
+ ok($fake->{m}->{ds}->{'tank/vm-101-disk-0'}, 'and the zvol is still there');
+
+ ($fake, $res) = run_on(
+ 'free',
+ step_fault => sub($step, $call) {
+ return 'dataset is busy' if step_is($step, $destroy->@*);
+ return 'File exists' if step_is($step, 'mkdir', "$S/namespaces/2");
+ return undef;
+ },
+ );
+ like(
+ $res->{error},
+ qr/failed to destroy 'tank\/vm-101-disk-0': dataset is busy; namespace restoration also/,
+ 'a failed restore after the retries',
+ );
+
+ $fake = lifecycle_fake(fail => [$destroy, $busy]);
+ delete $fake->{m}->{subsystems}->{$NQN}->{ns}->{2};
+ $res = flow($fake, $ACT{free});
+ like(
+ $res->{error},
+ qr/\Afailed to destroy 'tank\/vm-101-disk-0': Device or resource busy\n\z/,
+ 'an unexported busy zvol is reported without restore',
+ );
+ ok(!ns_of($fake, 2), 'and stays unexported');
+
+ $fake = lifecycle_fake();
+ delete $fake->{m}->{subsystems}->{$NQN};
+ delete $_->{links}->{$NQN} for values $fake->{m}->{ports}->%*;
+ $res = flow($fake, $ACT{free});
+ is($res->{error}, '', 'a volume is freed without subsystem');
+ ok(!$fake->{m}->{ds}->{'tank/vm-101-disk-0'}, 'nothing exports it');
+};
+
+subtest 'rollback' => sub {
+ my $dev = '/dev/zvol/tank/vm-100-disk-0';
+ my ($fake, $res) = run_on('rollback');
+ is($res->{error}, '', 'a rollback');
+ is_deeply(
+ ops($res),
+ ['read target state', 'rollback zvol', 'wait for zvol', 'restore namespace'],
+ 'unexports, rolls back and exports again',
+ );
+ ok(exported($fake, 1, $U{1}, $dev), 'with the same identity');
+
+ # A snapshot without the identity properties: the rollback loses them.
+ my $bare = sub(%opts) {
+ my $f = lifecycle_fake(%opts);
+ $f->add_snapshot('tank/vm-100-disk-0@bare', props => {});
+ return $f;
+ };
+ for my $case (
+ ['the disable', { write => '/namespaces/1/enable', value => 0 }],
+ ['the rmdir', ['rmdir', "$S/namespaces/1"]],
+ ['the rollback', ['zfs', 'rollback']],
+ ['the identity', ['zfs', 'set']],
+ ) {
+ my ($name, $match) = $case->@*;
+ $fake = $bare->(fail => [$match, "injected $name failure", 1]);
+ $res = flow(
+ $fake,
+ sub { $PLUGIN->volume_snapshot_rollback(scfg(), 'st', 'vm-100-disk-0', 'bare') },
+ );
+ like(
+ $res->{error},
+ qr/\ANVMe target operation 'rollback zvol' failed: injected \Q$name\E failure\n\z/,
+ "a failure at $name is reported",
+ );
+ ok(exported($fake, 1, $U{1}, $dev), "the namespace is restored after $name");
+ is_deeply(
+ owned_volumes($fake->{m})->{'tank/vm-100-disk-0'},
+ [1, $U{1}],
+ 'with identity',
+ );
+ }
+ my @restore = grep { $_->{op} eq 'restore namespace' } $res->{calls}->@*;
+ ok(step_is($restore[0]->{steps}->[0], 'zfs', 'set'), 'a lost identity is written first');
+
+ ($fake, $res) = run_on(
+ 'rollback',
+ step_fault => sub($step, $call) {
+ return 'rollback failed' if step_is($step, 'zfs', 'rollback');
+ return 'mkdir failed' if step_is($step, 'mkdir');
+ return undef;
+ },
+ );
+ like(
+ $res->{error},
+ qr/
+ \Arollback\ failed:\ NVMe\ target\ operation\ 'rollback\ zvol'\ failed:
+ \ rollback\ failed;
+ \ namespace\ restoration\ failed:\ NVMe\ target\ operation\ 'restore\ namespace'
+ \ failed:\ mkdir\ failed\n\z
+ /x,
+ 'both failures are reported',
+ );
+ ($fake, $res) = run_on('rollback', fail => [['mkdir'], 'mkdir failed']);
+ like(
+ $res->{error},
+ qr/\Anamespace restoration failed after rollback: .*mkdir failed\n\z/,
+ 'a failed export after a good rollback is reported',
+ );
+
+ $fake = lifecycle_fake();
+ for my $case (
+ ['a malformed snapshot name', 'vm-100-disk-0', 'a b', qr/invalid snapshot name/],
+ [
+ 'a subvolume',
+ 'subvol-100-disk-0',
+ 'snap1',
+ qr/snapshot rollback requires a ZFS volume/,
+ ],
+ ['a volume that is not owned', 'vm-900-disk-0', 'snap1', qr/is not owned/],
+ ) {
+ my ($name, $volume, $snap, $error) = $case->@*;
+ my $rollback = sub { $PLUGIN->volume_snapshot_rollback(scfg(), 'st', $volume, $snap) };
+ refused($fake, $rollback, $error, $name);
+ }
+ my $taken = lifecycle_fake();
+ $taken->add_namespace($NQN, 1, $U{9}, '/dev/zvol/tank/vm-900-disk-0');
+ refused(
+ $taken,
+ 'rollback',
+ qr/NVMe namespace ID '1' has a different identity/,
+ 'a namespace ID that holds another identity',
+ );
+ delete $fake->{m}->{subsystems}->{$NQN};
+ refused(
+ $fake,
+ 'rollback',
+ qr/NVMe subsystem does not exist/,
+ 'a rollback without the subsystem',
+ );
+};
+
+subtest 'template' => sub {
+ my $old = '/dev/zvol/tank/vm-101-disk-0';
+ my $new = '/dev/zvol/tank/base-101-disk-0';
+ my ($fake, $res) = run_on('template');
+ is($res->{result}, 'base-101-disk-0', 'a template');
+ is_deeply(
+ ops($res),
+ ['read target state', 'rename zvol', 'wait for zvol', 'create template'],
+ 'is renamed, exported and snapshotted',
+ );
+ ok(exported($fake, 2, $U{2}, $new), 'under the same identity');
+ ok($fake->{m}->{snaps}->{'tank/base-101-disk-0@__base__'}, 'with its base snapshot');
+
+ my $restored = sub($f) {
+ return
+ exported($f, 2, $U{2}, $old)
+ && $f->{m}->{ds}->{'tank/vm-101-disk-0'}
+ && !$f->{m}->{ds}->{'tank/base-101-disk-0'};
+ };
+ for my $case (
+ ['the unexport', sub($s, $c) { write_to($s, '/namespaces/2/enable', 0) }],
+ ['the rename', sub($s, $c) { step_is($s, 'zfs', 'rename', 'tank/vm-101-disk-0') }],
+ ['the export', sub($s, $c) { step_is($s, 'mkdir') && $c->{op} eq 'create template' }],
+ ['the snapshot', sub($s, $c) { step_is($s, 'zfs', 'snapshot') }],
+ ) {
+ my ($name, $match) = $case->@*;
+ my $fault =
+ sub($step, $call) { $match->($step, $call) ? "injected $name failure" : undef };
+ ($fake, $res) = run_on('template', step_fault => $fault);
+ my $prefix = qr/\Atemplate conversion failed: NVMe target operation '[a-z ]+' failed:/;
+ like(
+ $res->{error},
+ qr/$prefix injected \Q$name\E failure\n\z/,
+ "a failure at $name is reported",
+ );
+ ok($restored->($fake), "the old name is exported again after $name");
+ }
+
+ ($fake, $res) = run_on(
+ 'template',
+ after_call => sub($f, $call) {
+ $f->{m}->{ds}->{'tank/base-101-disk-0'}->{devwait} = 1000
+ if $call->{op} eq 'rename zvol';
+ },
+ );
+ like(
+ $res->{error},
+ qr/\Atemplate conversion failed: zvol '\Q$new\E' is not a block device after 10 seconds\n/,
+ 'a device that never appears',
+ );
+ ok($restored->($fake), 'is renamed back');
+
+ ($fake, $res) = run_on('template', faults => { 1 => 'late' });
+ like(
+ $res->{error},
+ qr/\ANVMe target operation 'rename zvol' did not complete .* state is unknown\n\z/,
+ 'a rename that does not complete in time is passed on as it is',
+ );
+ ok(!grep({ $_->{op} =~ /back|restore/ } $res->{calls}->@*), 'is not compensated');
+ $fake->land;
+ $res = flow($fake, $ACT{activate});
+ ok(!$res->{error} && exported($fake, 2, $U{2}, $new), 'the next activation exports it');
+
+ my $taken = lifecycle_fake();
+ $taken->add_namespace($NQN, 2, $U{9}, '/dev/zvol/tank/vm-900-disk-0');
+ refused(
+ $taken,
+ 'template',
+ qr/NVMe namespace ID '2' has a different identity/,
+ 'a namespace ID that holds another identity',
+ );
+ $fake = lifecycle_fake();
+ $fake->add_zvol('tank/base-101-disk-0');
+ refused(
+ $fake,
+ 'template',
+ qr/template zvol 'tank\/base-101-disk-0' already exists/,
+ 'a taken name',
+ );
+ $res = flow($fake, sub { $PLUGIN->create_base('st', scfg(), 'base-102-disk-0') });
+ like($res->{error}, qr/create_base not possible with base image/, 'refuses a base');
+ refused(
+ $fake,
+ sub { $PLUGIN->create_base('st', scfg(), 'vm-900-disk-0') },
+ qr/is not owned/,
+ 'a volume that is not owned',
+ );
+};
+
+subtest 'resize' => sub {
+ my ($fake, $res) = run_on('resize');
+ is($res->{error}, '', 'an exported volume grows');
+ is($res->{result}, 2 * 1024**2, 'the new size in KiB is returned');
+ is_deeply(ops($res), ['read target state', 'resize zvol'], 'read, then resize');
+ ok(!grep({ !$_->{locked} } $res->{calls}->@*), 'under the lock');
+ is($fake->{revalidated}->{"$NQN/1"}, 1,
+ 'the namespace reads the new size in the same call');
+
+ $fake = lifecycle_fake();
+ delete $fake->{m}->{subsystems}->{$NQN}->{ns}->{1};
+ $res = flow($fake, $ACT{resize});
+ is($res->{error}, '', 'an unexported volume grows');
+ is(scalar($res->{calls}->[1]->{steps}->@*), 1, 'without revalidation');
+ $fake = lifecycle_fake();
+ ns_of($fake, 1)->{enable} = '0';
+ $res = flow($fake, $ACT{resize});
+ is(scalar($res->{calls}->[1]->{steps}->@*), 1, 'a disabled namespace is not revalidated');
+ $fake = lifecycle_fake()->reboot(0);
+ $res = flow($fake, $ACT{resize});
+ is($res->{error}, '', 'a volume grows after a target restart');
+ is($fake->{m}->{ds}->{'tank/vm-100-disk-0'}->{volsize}, 2 * 1024**3,
+ 'once nvmet is loaded');
+
+ for my $case (
+ [
+ 'a duplicate UUID',
+ 'vm-100-disk-0',
+ sub($f) { $f->add_namespace($NQN, 9, $U{1}, '/dev/zvol/tank/x', 0) },
+ qr/duplicate namespace UUID/,
+ ],
+ ['a volume that is not owned', 'vm-900-disk-0', sub($f) { }, qr/is not owned/],
+ ['a volume of another subsystem', 'vm-901-disk-0', sub($f) { }, qr/is not owned/],
+ ) {
+ my ($name, $volname, $change, $error) = $case->@*;
+ $fake = lifecycle_fake();
+ $change->($fake);
+ my $resize = sub { $PLUGIN->volume_resize(scfg(), 'st', $volname, 2 * 1024**3) };
+ refused($fake, $resize, $error, $name);
+ }
+ ($fake, $res) = run_on('resize', fail => [['zfs', 'set'], 'out of space']);
+ like($res->{error}, qr/'resize zvol' failed: out of space/, 'a failed resize');
+ is_deeply(ops($res), ['read target state', 'resize zvol'], 'stops there');
+ ($fake, $res) =
+ run_on('resize', fail => [{ write => 'revalidate_size' }, 'Invalid argument']);
+ like($res->{error}, qr/'resize zvol' failed: Invalid argument/, 'a failed revalidation');
+ is($fake->{m}->{ds}->{'tank/vm-100-disk-0'}->{volsize}, 2 * 1024**3, 'keeps the new size');
+};
+
+sub removal_fake(%opts) {
+ my $fake = target_fake(%opts);
+ $fake->add_zvol('other/foreign-disk', identity => [$FOREIGN_NQN, 1, $U{8}]);
+ $fake->add_subsystem(
+ $FOREIGN_NQN,
+ model => 'Linux',
+ serial => '0123456789abcdef',
+ acl => [$HOSTS[0]],
+ );
+ $fake->add_namespace($FOREIGN_NQN, 1, $U{8}, '/dev/zvol/other/foreign-disk');
+ $fake->{m}->{ports}->{1}->{links}->{$FOREIGN_NQN} = 1;
+ return $fake;
+}
+
+subtest 'target removal' => sub {
+ my $fake = removal_fake();
+ my $res = flow($fake, $ACT{remove});
+ is($res->{error}, '', 'the subsystem is removed');
+ is_deeply(
+ ops($res),
+ [
+ 'read target state',
+ 'remove NVMe subsystem',
+ 'read target state',
+ 'remove orphan NVMe hosts',
+ ],
+ 'teardown, then orphans from a fresh read',
+ );
+ my $m = $fake->{m};
+ ok(
+ !$m->{subsystems}->{$NQN} && $m->{hosts}->{ $HOSTS[0] } && !$m->{hosts}->{ $HOSTS[1] },
+ 'subsystem and orphan host are gone, the host linked by another subsystem stays',
+ );
+ ok($m->{ports}->{1} && $m->{ports}->{2}, 'ports are never removed');
+ $res = flow($fake, $ACT{remove});
+ is($res->{error}, '', 'a removed subsystem is already removed');
+ is_deeply(ops($res), ['read target state'], 'with nothing left to do');
+
+ $fake = removal_fake();
+ delete $fake->{m}->{subsystems}->{$NQN};
+ delete $_->{links}->{$NQN} for values $fake->{m}->{ports}->%*;
+ $res = flow($fake, $ACT{remove});
+ is_deeply(
+ ops($res),
+ ['read target state', 'remove orphan NVMe hosts'],
+ 'without subsystem the configured hosts are candidates',
+ );
+ ok(
+ $fake->{m}->{hosts}->{ $HOSTS[0] } && !$fake->{m}->{hosts}->{ $HOSTS[1] },
+ 'only unlinked hosts go',
+ );
+
+ for my $case (
+ [
+ 'an owned volume',
+ sub($f) { add_volume($f, 'vm-100-disk-0', 1, $U{1}, unexported => 1) },
+ qr/owned ZFS volume 'tank\/vm-100-disk-0' exists/,
+ ],
+ [
+ 'a foreign subsystem',
+ sub($f) { $f->{m}->{subsystems}->{$NQN}->{attr}->{attr_model} = 'Linux' },
+ qr/refusing to delete foreign NVMe subsystem/,
+ ],
+ [
+ 'remaining namespaces',
+ sub($f) { $f->add_namespace($NQN, 3, $U{3}, '(null)', 0) },
+ qr/namespaces remain/,
+ ],
+ ) {
+ my ($name, $change, $error) = $case->@*;
+ $fake = removal_fake();
+ $change->($fake);
+ refused($fake, 'remove', $error, $name);
+ }
+
+ $fake =
+ removal_fake(fail => [['rmdir', "$ROOT/hosts/$HOSTS[1]"], 'Device or resource busy']);
+ $res = flow($fake, $ACT{remove});
+ is($res->{error}, '', 'a failed orphan removal');
+ like(
+ $res->{warnings}->[0],
+ qr/could not remove orphan NVMe host\(s\): Device or resource busy/,
+ 'only warns',
+ );
+
+ $fake = removal_fake(fail => [['rmdir', $S], 'Directory not empty']);
+ $res = flow($fake, $ACT{remove});
+ like(
+ $res->{error},
+ qr/cannot remove NVMe subsystem '\Q$NQN\E': Directory not empty/,
+ 'a failed teardown is reported',
+ );
+ ok($fake->{m}->{subsystems}->{$NQN}, 'and the subsystem stays');
+};
+
+subtest 'snapshots and the generic ZFS calls' => sub {
+ my $fake = lifecycle_fake();
+ my $res =
+ flow($fake, sub { $PLUGIN->volume_snapshot(scfg(), 'st', 'vm-100-disk-0', 'snap_2.0') });
+ is($res->{error}, '', 'a snapshot');
+ ok($fake->{m}->{snaps}->{'tank/vm-100-disk-0@snap_2.0'}, 'is taken');
+ is(scalar($res->{locks}->@*), 1, 'under the target lock');
+ $res = flow(
+ $fake,
+ sub { $PLUGIN->volume_rollback_is_possible(scfg(), 'st', 'vm-100-disk-0', 'snap1') },
+ );
+ like(
+ $res->{error},
+ qr/'snap1' is not most recent snapshot/,
+ 'only from the latest snapshot',
+ );
+ $res = flow(
+ $fake,
+ sub { $PLUGIN->volume_snapshot_delete(scfg(), 'st', 'vm-100-disk-0', 'snap_2.0') },
+ );
+ ok(
+ !$res->{error} && !$fake->{m}->{snaps}->{'tank/vm-100-disk-0@snap_2.0'},
+ 'deleted by name',
+ );
+ is(scalar($res->{locks}->@*), 1, 'under the target lock too');
+ $res = flow(
+ $fake,
+ sub { $PLUGIN->volume_rollback_is_possible(scfg(), 'st', 'vm-100-disk-0', 'snap1') },
+ );
+ is($res->{result}, 1, 'now the rollback is possible');
+ $res = flow(
+ $fake,
+ sub { $PLUGIN->volume_rollback_is_possible(scfg(), 'st', 'vm-100-disk-0', 'gone') },
+ );
+ like($res->{error}, qr/snapshot 'gone' does not exist/, 'and needs an existing snapshot');
+ $res = flow($fake, sub { $PLUGIN->volume_snapshot_info(scfg(), 'st', 'vm-100-disk-0') });
+ is_deeply([sort keys $res->{result}->%*], ['snap1'], 'snapshot information');
+
+ $res = flow($fake, sub { [$PLUGIN->volume_size_info(scfg(), 'st', 'vm-100-disk-0', 7)] });
+ is_deeply($res->{result}, [1073741824, 'raw', 4096, undef], 'size information');
+ is($res->{calls}->[0]->{timeout}, 7, 'with the timeout of the caller');
+ $res =
+ flow($fake, sub { scalar($PLUGIN->volume_size_info(scfg(), 'st', 'vm-100-disk-0')) });
+ is($res->{result}, 1073741824, 'the size alone in scalar context');
+
+ # Snapshot names are validated before zfs reads '%' and ',' in them as a
+ # range and a list, and the pool name wherever a call uses it.
+ for my $case (
+ ['volume_snapshot', 'a b'],
+ ['volume_snapshot_delete', 'snap1%'],
+ ['volume_rollback_is_possible', 'snap1,snap2'],
+ ) {
+ my ($method, $snap) = $case->@*;
+ $res = refused(
+ $fake,
+ sub { $PLUGIN->$method(scfg(), 'st', 'vm-100-disk-0', $snap) },
+ qr/\Ainvalid snapshot name\n\z/,
+ "$method with the snapshot name '$snap'",
+ );
+ is(scalar($res->{calls}->@*), 0, 'locally');
+ }
+ my $bad_pool = scfg(pool => '-rH');
+ for my $case (
+ ['list_images', sub { $PLUGIN->list_images('st', $bad_pool) }],
+ [
+ 'volume_size_info',
+ sub { $PLUGIN->volume_size_info($bad_pool, 'st', 'vm-100-disk-0') },
+ ],
+ [
+ 'volume_snapshot_info',
+ sub { $PLUGIN->volume_snapshot_info($bad_pool, 'st', 'vm-100-disk-0') },
+ ],
+ [
+ 'volume_snapshot',
+ sub { $PLUGIN->volume_snapshot($bad_pool, 'st', 'vm-100-disk-0', 'snap2') },
+ ],
+ ) {
+ $res = refused(
+ $fake,
+ $case->[1],
+ qr/\Ainvalid ZFS pool name\n\z/,
+ "$case->[0] with an invalid pool name",
+ );
+ is(scalar($res->{calls}->@*), 0, 'locally');
+ }
+ my @warnings;
+ {
+ local $SIG{__WARN__} = sub($warning) { push @warnings, $warning };
+ $res = flow($fake, sub { [$PLUGIN->status('st', $bad_pool)] });
+ }
+ is_deeply(
+ [$res->{result}, $res->{calls}, \@warnings],
+ [[0, 0, 0, 0], [], ["storage 'st': invalid ZFS pool name\n"]],
+ 'status with an invalid pool name is inactive, locally',
+ );
+
+ # The plugin creates zvols only.
+ for my $case (
+ [
+ 'volume_snapshot',
+ sub { $PLUGIN->volume_snapshot(scfg(), 'st', 'subvol-100-disk-0', 's') },
+ ],
+ [
+ 'volume_size_info',
+ sub { $PLUGIN->volume_size_info(scfg(), 'st', 'subvol-100-disk-0') },
+ ],
+ ) {
+ $res = refused(
+ $fake,
+ $case->[1],
+ qr/\A$case->[0] requires a ZFS volume\n\z/,
+ "$case->[0] of a subvolume",
+ );
+ is(scalar($res->{calls}->@*), 0, 'locally');
+ }
+};
+
+subtest 'lock ownership' => sub {
+ my $fake = lifecycle_fake();
+ pipe(my $reader, my $writer) or die "pipe: $!\n";
+ my $child;
+ $fake->{before_call} = sub($f, $call) {
+ return if $child || $call->{op} ne 'create zvol';
+ $child = fork() // die "fork: $!\n";
+ return if $child;
+ close($reader);
+ my $res = flow(
+ $f, sub { $PLUGIN->volume_resize(scfg(), 'st', 'vm-100-disk-0', 2 * 1024**3, 0) },
+ );
+ my $locked = grep { $_->{pid} == $$ } $res->{locks}->@*;
+ my @counts = ($locked, scalar($res->{nested}->@*), scalar($res->{calls}->@*));
+ print {$writer} "@counts\n";
+ close($writer);
+ POSIX::_exit(0);
+ };
+ my $res = flow($fake, sub { $PLUGIN->alloc_image('st', scfg(), 200, 'raw', undef, 1024) });
+ close($writer);
+ my $line = <$reader>;
+ waitpid($child, 0);
+ is($res->{error}, '', 'the parent keeps its lock');
+ is($line, "1 1 0\n", 'a child requests the lock itself, waits and changes nothing');
+
+ local $QUORATE = 0;
+ $fake = lifecycle_fake();
+ for my $action (qw(alloc free rollback template resize remove)) {
+ $res = flow($fake, $ACT{$action});
+ like($res->{error}, qr/cluster not quorate/, "$action needs quorum");
+ ok(!$res->{locks}->@* && !changes($res), 'and fails before the lock');
+ }
+};
+
+subtest 'target call timeouts' => sub {
+ my ($fake, $res) = run_on('alloc', after_call => sub($f, $call) { $NOW += 100 });
+ is($res->{error}, '', 'slow target calls do not end an operation');
+ is_deeply(
+ [map { $_->{timeout} } grep { $_->{locked} } $res->{calls}->@*],
+ [15, 60, 60, 15, 30, 15],
+ 'reads take up to 15 s, zfs changes 60 s and configfs changes 30 s',
+ );
+
+ # in a worker, a zfs change may take an hour, like in the other ZFS plugins
+ my $rpcenv_mock = Test::MockModule->new('PVE::RPCEnvironment');
+ $rpcenv_mock->redefine(is_worker => sub(@args) { return 1 });
+ my %timeouts;
+ for my $action (qw(alloc template resize rollback free)) {
+ (undef, $res) = run_on($action);
+ $timeouts{ $_->{op} } = $_->{timeout} for $res->{calls}->@*;
+ }
+ is_deeply(
+ [
+ @timeouts{
+ 'reserve namespace ID',
+ 'create zvol',
+ 'rename zvol',
+ 'create template',
+ 'resize zvol',
+ 'rollback zvol',
+ 'restore namespace',
+ 'destroy zvol',
+ },
+ ],
+ [(3600) x 8],
+ 'every zfs change',
+ );
+ $rpcenv_mock->unmock_all();
+};
+
+# ---------------------------------------------------------------------------
+# Fault sweep
+# ---------------------------------------------------------------------------
+
+# The goals of the swept flows. They only read the state: a lookup that
+# created a missing namespace would change the outcome of the next check.
+sub exported_ns($m, $nsid) {
+ my $subsys = $m->{subsystems}->{$NQN} // return undef;
+ return $subsys->{ns}->{$nsid};
+}
+
+sub template_done($m, $res) {
+ my $ds = owned_volumes($m)->{'tank/base-101-disk-0'} // return 'template missing';
+ my $ns = exported_ns($m, 2) // return 'template not exported';
+ return 'template not exported' if $ns->{device_path} ne '/dev/zvol/tank/base-101-disk-0';
+ return $m->{snaps}->{'tank/base-101-disk-0@__base__'} ? () : 'base snapshot missing';
+}
+
+sub new_volume_done($name) {
+ return sub($m, $res) {
+ my $owned = owned_volumes($m)->{"tank/$name"} // return "$name missing";
+ my $ns = exported_ns($m, $owned->[0]) // return "$name not exported";
+ return () if $ns->{enable} eq '1' && $ns->{device_uuid} eq $owned->[1];
+ return "$name not exported";
+ };
+}
+
+sub removal_goal($m) {
+ my @bad;
+ push @bad, 'subsystem left' if $m->{subsystems}->{$NQN};
+ push @bad, 'orphan host left' if $m->{hosts}->{ $HOSTS[1] };
+ push @bad, 'shared host removed' if !$m->{hosts}->{ $HOSTS[0] };
+ push @bad, 'port removed' if !$m->{ports}->{1} || !$m->{ports}->{2};
+ return @bad;
+}
+
+subtest 'fault sweep' => sub {
+ sweep(
+ 'create',
+ {
+ setup => \&lifecycle_fake,
+ lost_replies => 1,
+ action => $ACT{alloc},
+ done => new_volume_done('vm-200-disk-0'),
+ },
+ );
+ sweep(
+ 'clone',
+ {
+ setup => \&lifecycle_fake,
+ lost_replies => 1,
+ action => $ACT{clone},
+ done => new_volume_done('vm-201-disk-0'),
+ },
+ );
+ sweep(
+ 'destroy',
+ {
+ setup => \&lifecycle_fake,
+ action => $ACT{free},
+ may_vanish => 'tank/vm-101-disk-0',
+ done => sub($m, $res) { $m->{ds}->{'tank/vm-101-disk-0'} ? 'volume left' : () },
+ },
+ );
+ sweep(
+ 'rollback',
+ {
+ setup => \&lifecycle_fake,
+ action => $ACT{rollback},
+ done => sub($m, $res) {
+ my $ns = exported_ns($m, 1) // return 'not exported';
+ return $ns->{enable} eq '1' ? () : 'not exported';
+ },
+ },
+ );
+ sweep(
+ 'template',
+ {
+ setup => \&lifecycle_fake,
+ action => $ACT{template},
+ renames => { 'tank/vm-101-disk-0' => 'tank/base-101-disk-0' },
+ done => \&template_done,
+ },
+ );
+ sweep(
+ 'resize',
+ {
+ setup => \&lifecycle_fake,
+ action => $ACT{resize},
+ done => sub($m, $res) {
+ my $ds = $m->{ds}->{'tank/vm-100-disk-0'} // return 'volume missing';
+ return 'not resized' if $ds->{volsize} != 2 * 1024**3;
+ return $res->{fake}->{revalidated}->{"$NQN/1"} ? () : 'not revalidated';
+ },
+ },
+ );
+ sweep(
+ 'activation after a configfs loss',
+ {
+ setup => sub () { lifecycle_fake()->reboot(1) },
+ action => $ACT{activate},
+ done => sub($m, $res) { () },
+ },
+ );
+ sweep(
+ 'target removal',
+ {
+ setup => \&removal_fake,
+ action => $ACT{remove},
+ retry => $ACT{remove},
+ goal => \&removal_goal,
+ teardown => 1,
+ done => sub($m, $res) { $m->{subsystems}->{$NQN} ? 'subsystem left' : () },
+ },
+ );
+};
+
+# ---------------------------------------------------------------------------
+# Two storages on one target, serialized by the lock mock
+# ---------------------------------------------------------------------------
+
+subtest 'two storages on one target' => sub {
+ my $nqn_b = 'nqn.2026-01.com.example:zfsnvme-b';
+ my $scfg_a = scfg(pool => 'tank/a');
+ my $scfg_b = scfg(pool => 'tank/b', subsysnqn => $nqn_b);
+ my $new = sub () { FakeTarget->new(pools => [qw(tank tank/a tank/b)]) };
+
+ # B, with another key, takes the lock first, between A's lockless read
+ # and A's locked read
+ my $fake = $new->();
+ my ($res, $b_res);
+ {
+ local $LOCK_HOOK = sub($name) {
+ $b_res = flow($fake, sub { activate($scfg_b, $KEY_B) });
+ };
+ $res = flow($fake, sub { activate($scfg_a, $KEY) });
+ }
+ is($b_res->{error}, '', 'B configures the target first');
+ like($res->{error}, qr/in-use DH-HMAC-CHAP key/, 'A is refused by its locked read');
+ ok(!$fake->{m}->{subsystems}->{$NQN}, 'without creating anything');
+ is_deeply(
+ [map { $fake->{m}->{hosts}->{$_}->{key} } @HOSTS],
+ [$KEY_B, $KEY_B],
+ "B's key is kept",
+ );
+
+ # (a) a target restart between the verify read and the publish
+ for my $loaded (0, 1) {
+ $fake = FakeTarget->new(pools => ['tank']);
+ my $restarted = 0;
+ my @online;
+ my $ctx =
+ { nqn => $NQN, pool => 'tank', hosts => [@HOSTS], key => $KEY, port_ids => [] };
+ $ctx->{foreign} = foreign_view($fake->{m}, [], $NQN, [@HOSTS]);
+ $fake->{before_call} = sub($f, $call) {
+ $f->reboot($loaded) if $call->{op} =~ /\Alink / && !$restarted++;
+ };
+ $fake->{after_call} = sub($f, $call) { push @online, online_violations($f, $ctx) };
+ $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/cannot publish NVMe subsystem '\Q$NQN\E'/,
+ 'a restart before the publish fails it',
+ );
+ ok(!grep({ $_->{links}->%* } values $fake->{m}->{ports}->%*), 'nothing is linked');
+ $fake->{before_call} = undef;
+ $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'the next activation restores the target');
+ is_deeply(\@online, [], 'never published before the ACLs and keys');
+ ok(
+ $fake->{m}->{ports}->{1}->{links}->{$NQN}
+ && $fake->{m}->{ports}->{2}->{links}->{$NQN},
+ 'and publishes it',
+ );
+ }
+
+ # ports are shared, whoever creates them
+ $fake = $new->();
+ {
+ local $LOCK_HOOK = sub($name) {
+ $b_res = flow($fake, sub { activate($scfg_b, $KEY) });
+ };
+ $res = flow($fake, sub { activate($scfg_a, $KEY) });
+ }
+ ok(!$res->{error} && !$b_res->{error}, 'two storages with one key on the same portals');
+ is_deeply(
+ [sort map { $_->{attr}->{addr_traddr} } values $fake->{m}->{ports}->%*],
+ ['192.0.2.21', '192.0.2.22'],
+ 'share one port per address',
+ );
+ ok(
+ !grep({ !($_->{links}->{$NQN} && $_->{links}->{$nqn_b}) }
+ values $fake->{m}->{ports}->%*),
+ 'both subsystems are published on both',
+ );
+
+ # lifecycle operations of both storages
+ my ($a_vol, $b_vol);
+ {
+ local $LOCK_HOOK = sub($name) {
+ $b_vol = flow(
+ $fake, sub { $PLUGIN->alloc_image('b', $scfg_b, 300, 'raw', undef, 1024) },
+ );
+ };
+ $a_vol =
+ flow($fake, sub { $PLUGIN->alloc_image('a', $scfg_a, 300, 'raw', undef, 1024) });
+ }
+ is_deeply(
+ [$a_vol->{result}, $b_vol->{result}],
+ ['vm-300-disk-0', 'vm-300-disk-0'],
+ 'volumes in both pools',
+ );
+ is_deeply(
+ [
+ sort keys $fake->{m}->{subsystems}->{$NQN}->{ns}->%*,
+ sort keys $fake->{m}->{subsystems}->{$nqn_b}->{ns}->%*,
+ ],
+ [1, 1],
+ 'with independent namespace IDs',
+ );
+};
+
+# ---------------------------------------------------------------------------
+# Target details
+# ---------------------------------------------------------------------------
+
+subtest 'each port is published on its own' => sub {
+ my $fake = FakeTarget->new(pools => ['tank'], unbindable => { 2 => 1 });
+ my $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'a port that cannot bind its address');
+ like(
+ join("\n", $res->{warnings}->@*),
+ qr/cannot publish NVMe subsystem '\Q$NQN\E' on port 2: .*Cannot assign requested address/,
+ 'only warns',
+ );
+ ok(
+ $fake->{m}->{ports}->{1}->{links}->{$NQN} && !$fake->{m}->{ports}->{2}->{links}->{$NQN},
+ 'while the other port serves the subsystem',
+ );
+ $res = flow($fake, $ACT{activate});
+ ok(grep({ $_ eq 'link NVMe subsystem on port 2' } ops($res)->@*), 'and is retried');
+
+ $fake = FakeTarget->new(pools => ['tank'], unbindable => { 1 => 1, 2 => 1 });
+ $res = flow($fake, $ACT{activate});
+ like(
+ $res->{error},
+ qr/\Acannot publish NVMe subsystem '\Q$NQN\E': ln: .*Cannot assign requested address\n\z/,
+ 'without any port the activation fails',
+ );
+};
+
+subtest 'UUIDs are compared in lower case' => sub {
+ my $upper = uc($U{9} =~ tr/9/a/r);
+ my $lower = lc($upper);
+ is(
+ inventory(@POOL, zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $upper))->{volumes}
+ ->{'tank/vm-100-disk-0'}->{uuid},
+ $lower,
+ 'the inventory normalizes the UUID',
+ );
+ my $dup = inventory(
+ @POOL,
+ zfs_rows('tank/vm-100-disk-0', 'volume', $NQN, 1, $upper),
+ zfs_rows('tank/vm-101-disk-0', 'volume', $NQN, 2, $lower),
+ );
+ eval { nv('_nvmet_owned_identity', $dup, $NQN, 'tank/vm-100-disk-0') };
+ like($@, qr/duplicate NVMe identity/, 'a UUID that differs only in case is a duplicate');
+
+ my $fake = target_fake();
+ $fake->add_zvol('tank/vm-100-disk-0', identity => [$NQN, 1, $upper]);
+ my $res = flow($fake, $ACT{activate});
+ is($res->{error}, '', 'the volume is exported');
+ ok(exported($fake, 1, $lower, '/dev/zvol/tank/vm-100-disk-0'), 'under its UUID');
+ $res = flow($fake, $ACT{activate});
+ is_deeply(ops($res), ['read target state'], 'which then is converged');
+};
+
+subtest 'removing the storage is best effort' => sub {
+ my $file_mock = Test::MockModule->new('PVE::File');
+ $file_mock->redefine(
+ dir_glob_foreach => sub($dir, $regex, $func) {
+ $func->('nvme7') if $dir eq '/sys/class/nvme';
+ },
+ );
+ local %FILES = (%FILES, '/sys/class/nvme/nvme7/subsysnqn' => $NQN);
+ my $remove = sub($fake) {
+ local @UNLINKED = ();
+ my $res = flow($fake, sub { $PLUGIN->on_delete_hook('st', scfg()) });
+ $res->{unlinked} = [sort @UNLINKED];
+ $res->{disconnects} =
+ scalar(grep { $_->[0] eq '/sys/class/nvme/nvme7/delete_controller' }
+ $res->{sysfs_writes}->@*);
+ return $res;
+ };
+
+ my $fake = lifecycle_fake();
+ my $before = dclone($fake->{m});
+ my $res = $remove->($fake);
+ is($res->{error}, '', 'a storage that owns volumes can be removed');
+ my $owned = "refusing to delete NVMe subsystem '$NQN': owned ZFS volume"
+ . " 'tank/base-102-disk-0' exists";
+ like(
+ join("\n", $res->{warnings}->@*),
+ qr/keeping the NVMe target configuration of storage 'st': \Q$owned\E/,
+ 'its volumes and subsystem are kept',
+ );
+ ok(!changes($res) && same($before, $fake->{m}), 'the target is untouched');
+ is_deeply($res->{unlinked}, [secret_file('st')], 'the key file is removed');
+ is($res->{disconnects}, 1, 'the local paths are disconnected');
+
+ $fake = removal_fake();
+ local $FILES{ secret_file('st') } = $KEY;
+ my $key_at_first_call;
+ $fake->{before_call} =
+ sub($f, $call) { $key_at_first_call //= $FILES{ secret_file('st') } };
+ $res = $remove->($fake);
+ is($res->{error}, '', 'an empty storage is removed');
+ ok(
+ !$fake->{m}->{subsystems}->{$NQN} && !$fake->{m}->{hosts}->{ $HOSTS[1] },
+ 'with its subsystem and orphan hosts',
+ );
+ ok(scalar($res->{calls}->@*) && !defined($key_at_first_call), 'after its key is removed');
+ $fake->{before_call} = undef;
+
+ # Another node read the key before the removal, and its activation waited
+ # for the target lock until the removal released it.
+ $res =
+ flow($fake, sub { nv('_nvmet_activate_target', 'st', scfg(), $PORTALS, [@HOSTS], $KEY) });
+ is($res->{error}, "storage 'st' is being removed or its key changed\n",
+ 'a late activation');
+ ok(
+ !changes($res)
+ && !$fake->{m}->{subsystems}->{$NQN}
+ && !$fake->{m}->{hosts}->{ $HOSTS[1] },
+ 'does not restore the target',
+ );
+
+ $plugin_mock->redefine(
+ _namespace_openers => sub($nqn) { return ['qemu (PID 7, /dev/nvme0n1p1)'] });
+ $fake = removal_fake();
+ $res = $remove->($fake);
+ is($res->{error}, '', 'a storage with open namespaces can be removed');
+ like(
+ $res->{warnings}->[0],
+ qr/not disconnecting NVMe storage 'st': .*in use by qemu \(PID 7, \/dev\/nvme0n1p1/,
+ 'without disconnecting',
+ );
+ is($res->{disconnects}, 0, 'the paths stay connected');
+ is_deeply($res->{unlinked}, [secret_file('st')], 'and its key file is removed');
+ $plugin_mock->redefine(_namespace_openers => sub($nqn) { return [] });
+};
+
+# ---------------------------------------------------------------------------
+# Renderer
+# ---------------------------------------------------------------------------
+
+my $INVENTORY_TEXT = 'zfs get -H -p -d 1 -t filesystem,volume -o name,property,value,source'
+ . ' type,proxmox:nvme-subsys,proxmox:nvme-nsid,proxmox:nvme-uuid,proxmox:nvme-last-nsid tank';
+my $MARKER_TEXT = q{printf '%s\n' ZFSNVME-CONFIGFS};
+my $CONFIGFS_TEXT =
+ q{env 'LC_ALL=C' find /sys/kernel/config/nvmet}
+ . q{ '(' -path '/sys/kernel/config/nvmet/subsystems/*/passthru'}
+ . q{ -o -path '/sys/kernel/config/nvmet/ports/*/ana_groups'}
+ . q{ -o -path '/sys/kernel/config/nvmet/ports/*/referrals' ')' -type d -prune}
+ . q{ -o -type d -exec printf 'D %s\n' '{}' +}
+ . q{ -o -type l -exec printf 'L %s\n' '{}' +}
+ . q{ -o -type f '(' -name enable -o -name device_path -o -name device_uuid -o -name buffered_io}
+ . q{ -o -name attr_model -o -name attr_serial -o -name attr_allow_any_host}
+ . q{ -o -name addr_trtype -o -name addr_adrfam -o -name addr_traddr -o -name addr_trsvcid ')'}
+ . q{ -exec grep '' /dev/null '{}' +};
+my $CONFIGFS_KEYS_TEXT =
+ $CONFIGFS_TEXT
+ . q{ -o -type f '('}
+ . join(' -o', map { " -path $ROOT/hosts/$_/dhchap_key" } @HOSTS)
+ . q{ ')' -exec sha256sum '{}' +};
+my $STATE_TEXT = "$INVENTORY_TEXT && $MARKER_TEXT && $CONFIGFS_TEXT";
+my $LOCKED_STATE_TEXT = "modprobe nvmet_tcp && $STATE_TEXT";
+
+sub w_text($path, $value) {
+ return "printf '%s\\n' $value > $path";
+}
+
+sub build_text($nsid, $uuid, $dev) {
+ my $ns = "$S/namespaces/$nsid";
+ return join(
+ ' && ',
+ "test -b $dev", "mkdir $ns",
+ w_text("$ns/device_path", $dev),
+ w_text("$ns/device_uuid", $uuid),
+ w_text("$ns/buffered_io", 0),
+ w_text("$ns/enable", 1),
+ );
+}
+
+sub rendered($res) {
+ return [map { $_->{rendered} } $res->{calls}->@*];
+}
+
+subtest 'golden commands' => sub {
+ my @H = map { "$ROOT/hosts/$_" } @HOSTS;
+ my $serial = 'PVEZFS' . substr(sha256_hex($NQN), 0, 14);
+ my $port = sub($id, $address) {
+ my $P = "$ROOT/ports/$id";
+ return join(
+ ' && ',
+ "mkdir $P",
+ w_text("$P/addr_trtype", 'tcp'),
+ w_text("$P/addr_adrfam", 'ipv4'),
+ w_text("$P/addr_traddr", $address),
+ w_text("$P/addr_trsvcid", 4420),
+ );
+ };
+ my $link = sub($id) {
+ return join(
+ ' && ',
+ "grep -qx 0 $S/attr_allow_any_host",
+ (map { "test -L $S/allowed_hosts/$_" } @HOSTS),
+ "ln -s $S $ROOT/ports/$id/subsystems/$NQN",
+ );
+ };
+
+ my $fake = FakeTarget->new(pools => ['tank']);
+ $fake->add_zvol('tank/vm-100-disk-0', identity => [$NQN, 1, $U{1}]);
+ is_deeply(
+ rendered(flow($fake, $ACT{activate})),
+ [
+ "$INVENTORY_TEXT && $MARKER_TEXT && $CONFIGFS_KEYS_TEXT",
+ "modprobe nvmet_tcp && $INVENTORY_TEXT && $MARKER_TEXT && $CONFIGFS_KEYS_TEXT",
+ join(
+ ' && ',
+ "mkdir $S",
+ w_text("$S/attr_model", q{'Proxmox ZFS NVMe'}),
+ w_text("$S/attr_serial", $serial),
+ w_text("$S/attr_allow_any_host", 0),
+ $port->(1, '192.0.2.21'),
+ $port->(2, '192.0.2.22'),
+ "mkdir $H[0]",
+ "mkdir $H[1]",
+ "chmod 0600 $H[0]/dhchap_key $H[0]/dhchap_ctrl_key $H[1]/dhchap_key"
+ . " $H[1]/dhchap_ctrl_key",
+ "dd ibs=4096 obs=8192 2>/dev/null | tee $H[0]/dhchap_key $H[1]/dhchap_key"
+ . ' >/dev/null',
+ "ln -s $H[0] $S/allowed_hosts/$HOSTS[0]",
+ "ln -s $H[1] $S/allowed_hosts/$HOSTS[1]",
+ build_text(1, $U{1}, '/dev/zvol/tank/vm-100-disk-0'),
+ ),
+ "$INVENTORY_TEXT && $MARKER_TEXT && $CONFIGFS_KEYS_TEXT",
+ $link->(1),
+ $link->(2),
+ ],
+ 'read, configure with a key from stdin, verify and publish',
+ );
+ $fake = lifecycle_fake();
+ is_deeply(
+ rendered(flow($fake, sub { $PLUGIN->path(scfg(), 'vm-100-disk-0', 'st') })),
+ [$INVENTORY_TEXT],
+ 'the ZFS inventory',
+ );
+ my $res = flow($fake, sub { $PLUGIN->alloc_image('st', scfg(), 200, 'raw', undef, 1024) });
+ my $uuid = $fake->{m}->{ds}->{'tank/vm-200-disk-0'}->{props}->{'proxmox:nvme-uuid'};
+ is_deeply(
+ rendered($res),
+ [
+ 'zfs list -o name,volsize,origin,type -t volume,filesystem -d1 -Hp tank',
+ $LOCKED_STATE_TEXT,
+ "zfs set 'proxmox:nvme-last-nsid=4' tank",
+ "zfs create -s -b 16k -o 'proxmox:nvme-subsys=$NQN' -o 'proxmox:nvme-nsid=4'"
+ . " -o 'proxmox:nvme-uuid=$uuid' -V 1024k tank/vm-200-disk-0",
+ 'test -b /dev/zvol/tank/vm-200-disk-0',
+ build_text(4, $uuid, '/dev/zvol/tank/vm-200-disk-0'),
+ $STATE_TEXT,
+ ],
+ 'read, reserve, create, wait, export and verify',
+ );
+ is_deeply(
+ rendered(flow(
+ $fake,
+ sub { $PLUGIN->volume_resize(scfg(), 'st', 'vm-200-disk-0', 2 * 1024**3, 0) },
+ )),
+ [
+ $LOCKED_STATE_TEXT,
+ "zfs set 'volsize=2097152k' tank/vm-200-disk-0 && "
+ . w_text("$S/namespaces/4/revalidate_size", 1),
+ ],
+ 'read, then resize and revalidate',
+ );
+ is_deeply(
+ rendered(flow($fake, sub { $PLUGIN->free_image('st', scfg(), 'vm-101-disk-0') })),
+ [
+ $LOCKED_STATE_TEXT,
+ join(
+ ' && ',
+ w_text("$S/namespaces/2/enable", 0),
+ "rmdir $S/namespaces/2",
+ 'zfs destroy -r tank/vm-101-disk-0',
+ ),
+ ],
+ 'unexport and destroy',
+ );
+ is_deeply(
+ rendered(flow(
+ $fake,
+ sub { $PLUGIN->volume_snapshot_rollback(scfg(), 'st', 'vm-100-disk-0', 'snap1') },
+ )),
+ [
+ $LOCKED_STATE_TEXT,
+ join(' && ',
+ w_text("$S/namespaces/1/enable", 0),
+ "rmdir $S/namespaces/1",
+ 'zfs rollback tank/vm-100-disk-0@snap1',
+ "zfs set 'proxmox:nvme-subsys=$NQN' 'proxmox:nvme-nsid=1'"
+ . " 'proxmox:nvme-uuid=$U{1}' tank/vm-100-disk-0"),
+ 'test -b /dev/zvol/tank/vm-100-disk-0',
+ build_text(1, $U{1}, '/dev/zvol/tank/vm-100-disk-0'),
+ ],
+ 'unexport, roll back, write the identity and export again',
+ );
+ is_deeply(
+ rendered(flow($fake, sub { $PLUGIN->create_base('st', scfg(), 'vm-100-disk-0') })),
+ [
+ $LOCKED_STATE_TEXT,
+ join(' && ',
+ w_text("$S/namespaces/1/enable", 0),
+ "rmdir $S/namespaces/1",
+ 'zfs rename tank/vm-100-disk-0 tank/base-100-disk-0'),
+ 'test -b /dev/zvol/tank/base-100-disk-0',
+ build_text(1, $U{1}, '/dev/zvol/tank/base-100-disk-0')
+ . ' && zfs snapshot tank/base-100-disk-0@__base__',
+ ],
+ 'unexport, rename, export and snapshot',
+ );
+ $res = flow($fake, sub { $PLUGIN->clone_image(scfg(), 'st', 'base-102-disk-0', 201) });
+ $uuid = $fake->{m}->{ds}->{'tank/vm-201-disk-0'}->{props}->{'proxmox:nvme-uuid'};
+ is_deeply(
+ [map { $_->{rendered} } $res->{calls}->@[2, 3]],
+ [
+ "zfs set 'proxmox:nvme-last-nsid=5' tank",
+ "zfs clone -o 'proxmox:nvme-subsys=$NQN' -o 'proxmox:nvme-nsid=5'"
+ . " -o 'proxmox:nvme-uuid=$uuid' tank/base-102-disk-0\@__base__ tank/vm-201-disk-0",
+ ],
+ 'reserve and clone',
+ );
+
+ $fake = lifecycle_fake(
+ step_fault => sub($s, $c) { step_is($s, 'rmdir', "$S/namespaces/2") ? 'busy' : undef },
+ );
+ $res = flow($fake, sub { $PLUGIN->free_image('st', scfg(), 'vm-101-disk-0') });
+ is(
+ $res->{calls}->[-1]->{rendered},
+ 'test -b /dev/zvol/tank/vm-101-disk-0 && ' . w_text("$S/namespaces/2/enable", 1),
+ 'enable again',
+ );
+
+ $fake = removal_fake();
+ is_deeply(
+ rendered(flow($fake, sub { nv('_nvmet_delete_target', scfg()) })),
+ [
+ $LOCKED_STATE_TEXT,
+ "rm $ROOT/ports/1/subsystems/$NQN $ROOT/ports/2/subsystems/$NQN"
+ . " $S/allowed_hosts/$HOSTS[0] $S/allowed_hosts/$HOSTS[1] && rmdir $S",
+ $STATE_TEXT,
+ "rmdir $H[1]",
+ ],
+ 'teardown and orphan hosts',
+ );
+
+ $fake = FakeTarget->new(pools => ['tank']);
+ $fake->{m}->{mounted} = 0;
+ $res = flow($fake, $ACT{activate});
+ ok(grep({ $_ eq 'cat /proc/mounts' } rendered($res)->@*), 'the mount table');
+ ok(
+ grep({ $_ eq 'mount -t configfs none /sys/kernel/config' } rendered($res)->@*),
+ 'the mount',
+ );
+
+ $fake = target_fake();
+ $fake->add_port(5, '192.0.2.99', 4420, links => [$NQN]);
+ is(
+ rendered(flow($fake, $ACT{activate}))->[-1],
+ "rm $ROOT/ports/5/subsystems/$NQN",
+ 'unlink',
+ );
+
+ $fake = lifecycle_fake();
+ $fake->{m}->{subsystems}->{$NQN}->{attr}->{attr_allow_any_host} = '1';
+ $fake->{m}->{subsystems}->{$NQN}->{acl} = {};
+ is(
+ rendered(flow($fake, $ACT{activate}))->[2],
+ join(
+ ' && ',
+ w_text("$S/attr_allow_any_host", 0),
+ map { "ln -s $ROOT/hosts/$_ $S/allowed_hosts/$_" } @HOSTS,
+ ),
+ 'allow_any_host before the ACLs',
+ );
+
+ $fake = lifecycle_fake();
+ my @generic = (
+ [sub { [$PLUGIN->status('st', scfg())] }, 'zfs get -o value -Hp available,used tank'],
+ [
+ sub { $PLUGIN->volume_snapshot_info(scfg(), 'st', 'vm-100-disk-0') },
+ 'zfs list -Hp -r -t snapshot -o name,guid,creation tank/vm-100-disk-0',
+ ],
+ [
+ sub { $PLUGIN->volume_rollback_is_possible(scfg(), 'st', 'vm-100-disk-0', 'snap1') }
+ ,
+ 'zfs list -H -r -t snapshot -o name -s creation tank/vm-100-disk-0',
+ ],
+ [
+ sub { [$PLUGIN->volume_size_info(scfg(), 'st', 'vm-100-disk-0')] },
+ 'zfs get -o value -Hp volsize,usedbydataset tank/vm-100-disk-0',
+ ],
+ [
+ sub { $PLUGIN->volume_snapshot(scfg(), 'st', 'vm-100-disk-0', 'snap2') },
+ 'zfs snapshot tank/vm-100-disk-0@snap2',
+ ],
+ [
+ sub { $PLUGIN->volume_snapshot_delete(scfg(), 'st', 'vm-100-disk-0', 'snap2') },
+ 'zfs destroy tank/vm-100-disk-0@snap2',
+ ],
+ );
+
+ for my $case (@generic) {
+ is(rendered(flow($fake, $case->[0]))->[0], $case->[1], $case->[1]);
+ }
+ is(
+ rendered(flow($fake, sub { $PLUGIN->list_images('st', scfg()) }))->[1],
+ 'zfs get -H -d 1 -o name,value,source proxmox:nvme-subsys tank',
+ 'the ownership query',
+ );
+};
+
+sub sh($command, $input = undef) {
+ my $err = gensym;
+ my $pid = open3(my $in, my $out, $err, '/bin/sh', '-c', $command);
+ print {$in} $input if defined($input);
+ close($in);
+ my $stdout = do { local $/; <$out> }
+ // '';
+ my $stderr = do { local $/; <$err> }
+ // '';
+ waitpid($pid, 0) == $pid or die "wait for test shell: $!\n";
+ my $status = $?;
+ return ($status & 127 ? 128 + ($status & 127) : $status >> 8, $stdout, $stderr);
+}
+
+# The subtests that run rendered commands need these tools. A Debian build
+# has them, so a missing one is an error, never a silently skipped test.
+{
+ my @tools = qw(find grep sha256sum dd tee);
+ BAIL_OUT('needs /bin/sh with ' . join(', ', @tools))
+ if !-x '/bin/sh' || (sh('command -v ' . join(' ', @tools)))[0] != 0;
+}
+
+subtest 'quoting' => sub {
+ my $dir = tempdir(CLEANUP => 1);
+ for my $value (
+ 'a b',
+ q{it's},
+ "\$(touch $dir/pwned)",
+ "`touch $dir/pwned`",
+ '-n',
+ '*',
+ '%s\n',
+ q{"x"},
+ 'x;y',
+ 'a && b',
+ '|',
+ '~root',
+ "tab\there",
+ '',
+ '$HOME',
+ 'a\\b',
+ '#c',
+ ) {
+ my $file = "$dir/value";
+ my ($rc) = sh(nv('_nvmet_render', [write_step($file, $value)]));
+ is($rc . slurp($file), "0$value\n", 'a write keeps ' . ($value =~ s/\t/\\t/r));
+ my (undef, $out) = sh(nv('_nvmet_render', [['printf', '%s\n', $value]]));
+ is($out, "$value\n", 'an argument keeps ' . ($value =~ s/\t/\\t/r));
+ }
+ ok(!-e "$dir/pwned", 'no value is executed');
+};
+
+subtest 'renderer rejections' => sub {
+ for my $case (
+ ['a newline in a value', [write_step("$S/x", "a\nb")]],
+ ['a NUL in a value', [write_step("$S/x", "a\0b")]],
+ ['a CR in an argument', [['zfs', 'get', "a\rb"]]],
+ ['an undefined argument', [['zfs', 'get', undef]]],
+ ['an undefined value', [write_step("$S/x", undef)]],
+ ['a reference as argument', [['zfs', 'get', []]]],
+ ['a relative write', [write_step('x', 1)]],
+ ['a relative key path', [{ key => ['x'] }]],
+ ['an empty key step', [{ key => [] }]],
+ ['two key steps', [{ key => ['/a'] }, { key => ['/b'] }]],
+ ['bash', [['bash', '-c', 'true']]],
+ ['sh', [['sh', '-c', 'true']]],
+ ['perl', [['perl', '-e', '1']]],
+ ['an absolute command', [['/bin/rm', '/a']]],
+ ['an empty command', [[]]],
+ ['a write with extra keys', [{ write => '/a', value => 1, mode => 1 }]],
+ ['an unknown step', [{ run => '/a' }]],
+ ['a string step', ['mkdir /a']],
+ ['no steps', []],
+ ['a string', 'mkdir /a'],
+ ) {
+ eval { nv('_nvmet_render', $case->[1]) };
+ is($@, "internal error: invalid NVMe target step\n", "rejects $case->[0]");
+ }
+};
+
+subtest 'the rendered chains on a configfs stand-in' => sub {
+ my $dir = tempdir(CLEANUP => 1);
+ my $root = "$dir/nvmet";
+ mkdir($_) or die "mkdir $_: $!\n" for $root, map { "$root/$_" } qw(hosts ports subsystems);
+ my $device = "$dir/device";
+ symlink('/dev/null', $device) or die "symlink: $!\n";
+ my $box = sub($path) { $path =~ s{\A\Q$ROOT\E(?=/|\z)}{$root}r };
+ my $kind = sub($path) {
+ my $rel = substr($path, length($root) + 1);
+ return 'subsystem' if $rel =~ m{\Asubsystems/[^/]+\z};
+ return 'namespace' if $rel =~ m{\Asubsystems/[^/]+/namespaces/[0-9]+\z};
+ return 'port' if $rel =~ m{\Aports/[0-9]+\z};
+ return 'host' if $rel =~ m{\Ahosts/[^/]+\z};
+ die "unexpected directory $path\n";
+ };
+ my %attrs = (
+ subsystem => {
+ attr_model => 'Linux',
+ attr_serial => '0123456789abcdef',
+ attr_allow_any_host => 0,
+ },
+ namespace => {
+ enable => 0,
+ device_path => '(null)',
+ device_uuid => $U{9},
+ buffered_io => 0,
+ },
+ port => { addr_trtype => '', addr_adrfam => '', addr_traddr => '', addr_trsvcid => '' },
+ host => { dhchap_key => '', dhchap_ctrl_key => '' },
+ );
+ my %groups = (subsystem => ['namespaces', 'allowed_hosts'], port => ['subsystems']);
+ # configfs creates attributes and default groups with each directory and
+ # removes them with it; the stand-in does both with the same commands.
+ my $shim = sub($steps) {
+ my @out;
+ for my $step ($steps->@*) {
+ if (ref($step) eq 'HASH' && exists($step->{write})) {
+ push @out, write_step($box->($step->{write}), $step->{value});
+ } elsif (ref($step) eq 'HASH') {
+ push @out, { key => [map { $box->($_) } $step->{key}->@*] };
+ } elsif ($step->[0] eq 'test' && $step->[1] eq '-b') {
+ push @out, ['test', '-L', $device];
+ } elsif ($step->[0] eq 'mkdir') {
+ for my $path (map { $box->($_) } $step->@[1 .. $step->$#*]) {
+ my $type = $kind->($path);
+ push @out, ['mkdir', $path, map { "$path/$_" } ($groups{$type} // [])->@*];
+ push @out, map { write_step("$path/$_", $attrs{$type}->{$_}) }
+ sort keys $attrs{$type}->%*;
+ }
+ } elsif ($step->[0] eq 'rmdir') {
+ for my $path (map { $box->($_) } $step->@[1 .. $step->$#*]) {
+ my $type = $kind->($path);
+ push @out, ['rm', map { "$path/$_" } sort keys $attrs{$type}->%*];
+ push @out, ['rmdir', map { "$path/$_" } $groups{$type}->@*]
+ if $groups{$type};
+ push @out, ['rmdir', $path];
+ }
+ } else {
+ push @out, [map { $box->($_) } $step->@*];
+ }
+ }
+ return \@out;
+ };
+ my $run = sub($steps, $input = undef) {
+ my ($rc, undef, $err) = sh(nv('_nvmet_render', $shim->($steps)), $input);
+ return ($rc, $err);
+ };
+
+ my $capture = FakeTarget->new(pools => ['tank']);
+ flow($capture, $ACT{activate});
+ my ($find) = grep { step_is($_, 'env') } $capture->{calls}->[0]->{steps}->@*;
+ my $read = sub () {
+ my ($rc, $out, $err) = sh(nv('_nvmet_render', $shim->([$find])));
+ die "configfs read failed: $err\n" if $rc;
+ $out =~ s{\Q$root\E}{$ROOT}g;
+ return nv('_nvmet_parse_configfs', $out);
+ };
+ my $inv = inventory(@POOL, @ONE);
+ my $conf = { nqn => $NQN, portals => $PORTALS, hostnqns => [@HOSTS], keysha => $KEY_SHA };
+ my $plan = nv('_nvmet_plan_activation', $inv, $read->(), $conf);
+ my ($rc, $err) = $run->([map { $_->@* } $plan->{prepublish}->@*], "$KEY\n");
+ is($rc, 0, 'the whole configuration runs as one chain') or diag($err);
+ my $cfs = $read->();
+ is_deeply(
+ nv('_nvmet_plan_activation', $inv, $cfs, $conf)->{prepublish},
+ [],
+ 'and converges',
+ );
+ is_deeply(
+ [map { $cfs->{hosts}->{$_}->{key_sha256} } @HOSTS],
+ [$KEY_SHA, $KEY_SHA],
+ 'the digests read back equal sha256 of the key and a newline',
+ );
+ is_deeply(
+ [map { (stat("$root/hosts/$_/dhchap_key"))[2] & 07777 } @HOSTS],
+ [0600, 0600],
+ 'and the key attributes are readable by their owner only',
+ );
+
+ # Only the kernel's own groups are skipped: the ACL of another tool's
+ # subsystem named like one of them is read and protects the key.
+ my $other = "$root/subsystems/referrals";
+ mkdir($_) or die "mkdir $_: $!\n" for $other, "$other/allowed_hosts", "$other/namespaces";
+ for my $attr (keys $attrs{subsystem}->%*) {
+ open(my $fh, '>', "$other/$attr") or die "open: $!\n";
+ print {$fh} "$attrs{subsystem}->{$attr}\n";
+ close($fh);
+ }
+ symlink("$root/hosts/$HOSTS[0]", "$other/allowed_hosts/$HOSTS[0]") or die "symlink: $!\n";
+ mkdir($_)
+ or die "mkdir $_: $!\n"
+ for "$root/ports/1/referrals", "$root/ports/1/referrals/r";
+ my (undef, $raw) = sh(nv('_nvmet_render', $shim->([$find])));
+ ok(
+ $read->()->{subsystems}->{referrals}->{acl}->{ $HOSTS[0] },
+ 'a subsystem named referrals',
+ );
+ unlike($raw, qr{/ports/1/referrals}, 'while the referrals of a port are skipped');
+ unlink("$other/allowed_hosts/$HOSTS[0]", map { "$other/$_" } keys $attrs{subsystem}->%*);
+ rmdir($_)
+ or die "rmdir $_: $!\n"
+ for "$other/allowed_hosts", "$other/namespaces", $other, "$root/ports/1/referrals/r",
+ "$root/ports/1/referrals";
+
+ for my $unit (nv('_nvmet_plan_publish', $cfs, $NQN, $plan->{port_ids}, [@HOSTS])->@*) {
+ ($rc) = $run->($unit->{steps});
+ is($rc, 0, "the guarded link to port $unit->{port}");
+ }
+ is_deeply(nv('_nvmet_plan_publish', $read->(), $NQN, [1, 2], [@HOSTS]), [], 'publishes');
+
+ unlink("$root/ports/1/subsystems/$NQN", "$root/subsystems/$NQN/allowed_hosts/$HOSTS[1]");
+ my ($unit) = nv('_nvmet_plan_publish', $read->(), $NQN, [1], [@HOSTS])->@*;
+ ($rc) = $run->($unit->{steps});
+ ok($rc && !-l "$root/ports/1/subsystems/$NQN", 'the guard blocks a link without every ACL');
+ symlink("$root/hosts/$HOSTS[1]", "$root/subsystems/$NQN/allowed_hosts/$HOSTS[1]") or die;
+ ($rc) = $run->([write_step("$ROOT/subsystems/$NQN/attr_allow_any_host", 1)]);
+ ($rc) = $run->($unit->{steps});
+ ok($rc && !-l "$root/ports/1/subsystems/$NQN", 'and while any host may connect');
+ $run->([write_step("$ROOT/subsystems/$NQN/attr_allow_any_host", 0)]);
+ ($rc) = $run->($unit->{steps});
+ ok(!$rc && -l "$root/ports/1/subsystems/$NQN", 'but links once both hold');
+
+ my ($units) = nv('_nvmet_plan_unexport', $read->(), $NQN, $U{1});
+ ($rc) = $run->([map { $_->@* } $units->@*]);
+ ok(!$rc && !-e "$root/subsystems/$NQN/namespaces/1", 'the unexport chain removes it');
+ my ($teardown, $candidates) = nv(
+ '_nvmet_plan_delete_target', inventory(@POOL), $read->(), $NQN, [@HOSTS],
+ );
+ ($rc) = $run->([map { $_->@* } $teardown->@*]);
+ ok(!$rc && !-e "$root/subsystems/$NQN", 'the teardown chain removes the subsystem');
+ ($rc) = $run->([map { $_->@* } nv('_nvmet_plan_orphan_hosts', $read->(), $candidates)->@*]);
+ ok(!$rc && !-e "$root/hosts/$HOSTS[0]" && !-e "$root/hosts/$HOSTS[1]", 'and the hosts');
+ ok(-d "$root/ports/1" && -d "$root/ports/2", 'ports stay');
+};
+
+# Last: every command rendered during this test run, and every protocol
+# violation recorded (by a fake target: a key on a command line or in output,
+# a change without the lock; or a local command).
+subtest 'every rendered command is a POSIX command chain' => sub {
+ is_deeply(\@VIOLATIONS, [], 'no flow violated the target protocol');
+ my %shapes;
+ for my $command (keys %CORPUS) {
+ (my $shape = $command) =~ s/[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}/U/gi;
+ $shape =~ s/[0-9]+/N/g;
+ $shapes{$shape} //= $command;
+ }
+ ok(keys(%shapes) >= 40, 'the corpus covers ' . keys(%shapes) . ' command shapes');
+ my $meta = qr/[;&|<>`\$(){}\\*?\[\]~#!"]/;
+ my @bad;
+ for my $command (sort values %shapes) {
+ push @bad, "sh -n: $command" if system('/bin/sh', '-n', '-c', $command) != 0;
+ push @bad, "bash-ism: $command"
+ if $command =~ /\[\[|\$\(\(|<<<|\$'|pipefail|(?:\A|&& )(?:function|source|\.) /;
+ (my $bare = $command) =~ s/'[^']*'|\\'//g;
+ for my $segment (split / && /, $bare) {
+ next
+ if $segment =~ m{\Add\ ibs=4096\ obs=8192\ 2>/dev/null\ \|\ tee(?:\ /[^\s;&|<>]+)+
+ \ >/dev/null\z}x;
+ if ($segment =~ /\Aprintf +(\S*) +> (\S+)\z/) {
+ push @bad, "write: $segment" if "$1$2" =~ $meta;
+ next;
+ }
+ push @bad, "shell syntax: $segment" if $segment =~ $meta;
+ }
+ }
+ is_deeply(\@bad, [], 'every command is a plain POSIX command chain');
+};
+
+done_testing();
diff --git a/src/test/zfsnvme_test.pm b/src/test/zfsnvme_test.pm
new file mode 100644
index 00000000..3b311bfa
--- /dev/null
+++ b/src/test/zfsnvme_test.pm
@@ -0,0 +1,2353 @@
+package PVE::Storage::TestZFSNVMe;
+
+# Storage API and local NVMe host side of the zfsnvme plugin. The target side
+# is covered by zfsnvme_target_test.pm. Nothing here opens a connection or
+# changes the host outside a temporary directory: every probe of the local
+# host is mocked, and the children that the plugin forks for its connect and
+# delete steps run inline, except where the child itself is tested.
+
+use v5.36;
+
+use lib qw(..);
+
+use Compress::Zlib qw(crc32);
+use Config;
+use Errno qw(EACCES ECONNREFUSED EINVAL ENOENT);
+use Fcntl qw(O_NOCTTY O_RDWR);
+use File::Temp qw(tempdir);
+use MIME::Base64 qw(encode_base64);
+use POSIX qw(
+ ECHO ECHONL ICANON OPOST SIG_BLOCK SIGALRM SIGHUP SIGINT SIGQUIT SIGTERM TCSANOW WNOHANG
+ sigprocmask
+);
+use Test::MockModule;
+use Test::More;
+
+use PVE::Storage;
+use PVE::Storage::ZFSNVMePlugin;
+
+my $PLUGIN = 'PVE::Storage::ZFSNVMePlugin';
+my $NQN = 'nqn.2026-01.com.example:test';
+my $OTHER_NQN = 'nqn.2026-01.com.example:other';
+my $UUID = '12345678-1234-1234-1234-123456789abc';
+my $ROOT = '/sys/kernel/config/nvmet';
+my $hostnqn_a = 'nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-000000000001';
+my $hostnqn_b = 'nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-000000000002';
+
+sub make_key($hash, $length, $seed = 1) {
+ my $raw = join('', map { chr(($_ * 131 + $seed * 17) % 256) } 1 .. $length);
+ return "DHHC-1:$hash:" . encode_base64($raw . pack('V', crc32($raw)), '') . ':';
+}
+my $KEY = make_key('00', 32);
+my $KEY_B = make_key('00', 32, 2);
+
+sub scfg(%override) {
+ return {
+ type => 'zfsnvme',
+ server => '192.0.2.10',
+ pool => 'tank',
+ subsysnqn => $NQN,
+ 'nvme-portals' => '192.0.2.21,192.0.2.22',
+ 'nvme-host-ifaces' => 'ens19,ens20',
+ 'nvme-host-nqns' => "$hostnqn_a,$hostnqn_b",
+ %override,
+ };
+}
+
+# Calls a plugin package sub by name, so a mocked sub is the one called.
+sub nv($name, @args) {
+ my $code = $PLUGIN->can($name) // die "plugin has no sub '$name'\n";
+ return $code->(@args);
+}
+
+# ---------------------------------------------------------------------------
+# Mocks. Target calls are answered by $TARGET, the ssh command of the runner
+# by $COMMAND; sysfs and key files are read from %FILES and listed from %DIRS,
+# block devices are %BLOCK and symbolic links %LINKS. The children of the
+# bounded host steps run inline and record their bound in @FORKS. Waits
+# advance $NOW. The host side runs no external command: only the runner
+# tests, which set $COMMAND, may reach run_command.
+# ---------------------------------------------------------------------------
+
+our ($TARGET, $COMMAND, $NOW);
+our (%FILES, %DIRS, %CONFIG, %BLOCK, %LINKS, @TARGET_CALLS, @WARNINGS, @WRITES);
+our (@SYSFS_WRITES, @GLOBS, @CONNECTS, @ACTIVATIONS, @UNLINKED, @RESTRICTED, @DELETED, @FORKS);
+our (%REFUSE, %CONNECT_STATE); # address => the connect is refused, the new controller's state
+$NOW = 1_000_000;
+
+my $nvme_mock = Test::MockModule->new($PLUGIN);
+my $file_mock = Test::MockModule->new('PVE::File');
+my $sysfs_mock = Test::MockModule->new('PVE::SysFSTools');
+my $tools_mock = Test::MockModule->new('PVE::Tools');
+my $network_mock = Test::MockModule->new('PVE::Network');
+my $storage_mock = Test::MockModule->new('PVE::Storage');
+my $parent_mock = Test::MockModule->new('PVE::Storage::Plugin');
+
+$nvme_mock->redefine(
+ _nvmet_run => sub($scfg, $steps, %opts) {
+ push @TARGET_CALLS, { steps => $steps, %opts };
+ die "unexpected NVMe target call '$opts{op}'\n" if !$TARGET;
+ return $TARGET->($steps, %opts);
+ },
+);
+# A command is also recorded, in case the caller handles the error; the last
+# test checks that there was none.
+our @COMMANDS;
+my $no_command = sub($cmd, %opts) {
+ push @COMMANDS, $cmd;
+ die "test error: unexpected command '" . (ref($cmd) ? $cmd->[0] : $cmd) . "'\n";
+};
+$nvme_mock->redefine(
+ run_command => sub($cmd, %opts) {
+ return $COMMAND ? $COMMAND->($cmd, %opts) : $no_command->($cmd, %opts);
+ },
+);
+$tools_mock->redefine(run_command => $no_command);
+$nvme_mock->redefine(_now => sub () { return $NOW });
+$nvme_mock->redefine(_sleep => sub($seconds) { $NOW += $seconds; return });
+$nvme_mock->redefine(_block_device => sub($path) { return $BLOCK{$path} });
+$nvme_mock->redefine(_link_target => sub($path) { return $LINKS{$path} });
+$nvme_mock->redefine(_restrict_attr => sub($path) { push @RESTRICTED, $path; return });
+$nvme_mock->redefine(_unlink_file => sub($path) { push @UNLINKED, $path; return 1 });
+$nvme_mock->redefine(log_warn => sub($message) { push @WARNINGS, $message; return });
+$nvme_mock->redefine(file_read_firstline => sub($path) { return $FILES{$path} });
+$nvme_mock->redefine(
+ file_set_contents => sub($path, $data, $perm = undef, @rest) {
+ push @WRITES, { path => $path, data => $data, perm => $perm };
+ return;
+ },
+);
+$nvme_mock->redefine(make_path => sub(@args) { push @WRITES, { make_path => [@args] }; return });
+$nvme_mock->redefine(_local_iface_exists => sub($iface) { return 1 });
+
+# The connect child: creates a controller of the portal named in the options.
+my $connect_mock = sub($options, $names) {
+ my %option = map { split(/=/, $_, 2) } split(/,/, $options);
+ push @CONNECTS, "$option{traddr}\@$option{host_iface}";
+ die "connect failed: Connection refused\n" if $REFUSE{ $option{traddr} };
+ my $instance = 70 + scalar(@CONNECTS);
+ add_controller(
+ "nvme$instance",
+ $option{traddr},
+ $option{trsvcid},
+ $option{host_iface},
+ $CONNECT_STATE{ $option{traddr} } // 'live',
+ );
+ return $instance;
+};
+$nvme_mock->redefine(_fabrics_connect => $connect_mock);
+# The deletion child: the controllers disappear from sysfs.
+my $delete_mock = sub($controllers) {
+ for my $controller ($controllers->@*) {
+ push @DELETED, $controller->{name};
+ $DIRS{'/sys/class/nvme'} =
+ [grep { $_ ne $controller->{name} } ($DIRS{'/sys/class/nvme'} // [])->@*];
+ }
+ return scalar($controllers->@*);
+};
+$nvme_mock->redefine(_delete_controllers => $delete_mock);
+my $fork_inline = sub($timeout, $code, $opts = undef) {
+ push @FORKS, $timeout;
+ my $res = $code->();
+ return wantarray ? ($res, 0) : $res;
+};
+$tools_mock->redefine(run_fork_with_timeout => $fork_inline);
+$file_mock->redefine(
+ dir_glob_foreach => sub($dir, $regex, $func) {
+ push @GLOBS, [$dir, $regex];
+ for my $entry (($DIRS{$dir} // [])->@*) {
+ if (my @res = $entry =~ m/^($regex)$/) {
+ $func->(@res);
+ }
+ }
+ },
+);
+my $sysfs_write = sub($path, $data, $allow_existing = undef) {
+ push @SYSFS_WRITES, [$path, $data];
+ $FILES{$path} = $data =~ s/\n\z//r;
+ return 1;
+};
+$sysfs_mock->redefine(file_write => $sysfs_write);
+$network_mock->redefine(tcp_ping => sub($host, $port, $timeout = undef) { return 1 });
+$storage_mock->redefine(config => sub () { return { ids => {%CONFIG} } });
+
+# A local controller of $NQN, as nvme-core shows it in sysfs.
+sub add_controller($name, $address, $port, $iface, $state = 'live', $nqn = $NQN) {
+ my $base = "/sys/class/nvme/$name";
+ push $DIRS{'/sys/class/nvme'}->@*, $name;
+ $FILES{"$base/subsysnqn"} = $nqn;
+ $FILES{"$base/state"} = $state;
+ $FILES{"$base/address"} = "traddr=$address,trsvcid=$port,host_iface=$iface";
+ $FILES{"$base/reconnect_delay"} //= '2';
+ $FILES{"$base/ctrl_loss_tmo"} //= '600';
+ $FILES{"$base/fast_io_fail_tmo"} //= 'off';
+ return;
+}
+
+sub reset_host() {
+ %DIRS = ('/sys/class/nvme-subsystem' => ['nvme-subsys7']);
+ %FILES = (
+ '/sys/module/nvme_core/parameters/multipath' => 'Y',
+ '/etc/nvme/hostnqn' => $hostnqn_a,
+ '/etc/nvme/hostid' => '12345678-1234-1234-1234-123456789abc',
+ '/sys/class/nvme-subsystem/nvme-subsys7/subsysnqn' => $NQN,
+ );
+ %BLOCK = %LINKS = %CONNECT_STATE = ();
+ @CONNECTS = @ACTIVATIONS = @SYSFS_WRITES = @WRITES = @WARNINGS = @GLOBS = ();
+ @RESTRICTED = @UNLINKED = @DELETED = @FORKS = ();
+ $NOW += 61; # a new minute: no storage is in its slow-path backoff
+ return;
+}
+
+# ---------------------------------------------------------------------------
+# Volume names and features (same formats as the other ZFS plugins)
+# ---------------------------------------------------------------------------
+
+# [feature, volume, snapshot, running, expected]
+for my $case (
+ ['copy', 'base-100-disk-0', undef, 0, 1],
+ ['copy', 'vm-100-disk-0', undef, 0, 1],
+ ['copy', 'vm-100-disk-0', undef, 1, 1],
+ ['copy', 'base-100-disk-0/vm-101-disk-0', undef, 0, 1],
+ ['copy', 'vm-100-disk-0', 'snap', 0, undef],
+ ['clone', 'base-100-disk-0', undef, undef, 1],
+ ['clone', 'vm-100-disk-0', undef, undef, undef],
+ ['template', 'vm-100-disk-0', undef, undef, 1],
+ ['snapshot', 'vm-100-disk-0', undef, 0, 1],
+ ['snapshot', 'vm-100-disk-0', 'snap', 0, 1],
+) {
+ my ($feature, $volname, $snap, $running, $expected) = $case->@*;
+ my @args = defined($running) ? ($snap, $running, {}) : ();
+ is(
+ $PLUGIN->volume_has_feature({}, $feature, 'nvmetest', $volname, @args),
+ $expected,
+ "$feature of $volname"
+ . ($snap ? '@' . $snap : '')
+ . ($running ? ' while running' : ''),
+ );
+}
+ok(!$PLUGIN->storage_can_replicate({}, 'nvmetest'), 'no storage replication');
+
+for my $case (
+ ['vm-100-disk-0', ['images', 'vm-100-disk-0', 100, undef, undef, '', 'raw']],
+ ['base-100-disk-0', ['images', 'base-100-disk-0', 100, undef, undef, 1, 'raw']],
+ ['subvol-100-disk-0', ['images', 'subvol-100-disk-0', 100, undef, undef, '', 'subvol']],
+ ['basevol-100-disk-0', ['images', 'basevol-100-disk-0', 100, undef, undef, 1, 'subvol']],
+ [
+ 'base-100-disk-0/vm-200-disk-0',
+ ['images', 'vm-200-disk-0', 200, 'base-100-disk-0', 100, '', 'raw'],
+ ],
+ [
+ 'basevol-100-disk-0/subvol-200-disk-0',
+ ['images', 'subvol-200-disk-0', 200, 'basevol-100-disk-0', 100, '', 'subvol'],
+ ],
+) {
+ my ($name, $expected) = $case->@*;
+ is_deeply([$PLUGIN->parse_volname($name)], $expected, "parse_volname keeps the tuple of $name");
+}
+for my $name ('not-a-volume', 'vm-no-id-disk-0', 'vm-100-disk 0', 'vm-100-') {
+ eval { $PLUGIN->parse_volname($name) };
+ like($@, qr/unable to parse zfs volume name/, "rejects invalid volume name $name");
+}
+
+is_deeply(
+ \@PVE::Storage::ZFSNVMePlugin::ISA,
+ ['PVE::Storage::Plugin'],
+ 'the backend inherits directly from the storage plugin base',
+);
+for my $method (qw(
+ zfs_request zfs_get_lu_name zfs_create_lu zfs_resize_lu zfs_list_zvol zfs_parse_zvol_list
+ zfs_get_properties zfs_get_pool_stats zfs_create_zvol zfs_delete_zvol zfs_get_base
+))
+{
+ ok(!$PLUGIN->can($method), "the ZFS helper $method is private");
+}
+
+# ---------------------------------------------------------------------------
+# Configuration
+# ---------------------------------------------------------------------------
+
+is(nv('verify_nvme_nqn', $hostnqn_a), $hostnqn_a, 'accepts a standard NQN');
+ok(!nv('verify_nvme_nqn', 'not-an-nqn', 1), 'rejects an invalid NQN');
+ok(!nv('verify_nvme_nqn', 'nqn.x:' . ('a' x 220), 1), 'rejects a long NQN');
+is_deeply(
+ nv('parse_nvme_host_nqns', "$hostnqn_a, $hostnqn_b"),
+ [$hostnqn_a, $hostnqn_b],
+ 'parses the host allow-list',
+);
+ok(!nv('parse_nvme_host_nqns', "$hostnqn_a,$hostnqn_a", 1), 'rejects duplicate host NQNs');
+ok(!nv('parse_nvme_host_nqns', '', 1), 'requires a host NQN');
+
+is_deeply(
+ nv('parse_nvme_portals', '198.51.100.11:4420,[2001:db8::11]:4421,198.51.100.12'),
+ [
+ { address => '198.51.100.11', port => 4420, family => 'ipv4' },
+ { address => '2001:db8::11', port => 4421, family => 'ipv6' },
+ { address => '198.51.100.12', port => 4420, family => 'ipv4' },
+ ],
+ 'parses IPv4 and IPv6 portals',
+);
+is_deeply(
+ [
+ map { $_->{address} }
+ nv('parse_nvme_portals', '[2001:DB8:0:0::11],[2001:db8::0:12]:4421')->@*
+ ],
+ ['2001:db8::11', '2001:db8::12'],
+ 'IPv6 addresses are canonical',
+);
+eval { nv('parse_nvme_portals', '[2001:db8::11]:4420,[2001:db8:0::11]:4420'); };
+like($@, qr/duplicate NVMe\/TCP portal/, 'two spellings of one IPv6 portal are a duplicate');
+eval { nv('parse_nvme_portals', '198.51.100.11:4420,198.51.100.11:4420') };
+like($@, qr/duplicate NVMe\/TCP portal/, 'duplicate portal is rejected');
+eval { nv('parse_nvme_portals', '198.51.100.11,198.51.100.11:04420') };
+is($@, "duplicate NVMe/TCP portal '198.51.100.11:04420'\n", 'a port with a leading zero');
+eval { nv('parse_nvme_portals', '198.51.100.11:4421,[::ffff:198.51.100.11]:4421') };
+is(
+ $@,
+ "duplicate NVMe/TCP portal '[::ffff:198.51.100.11]:4421'\n",
+ 'an IPv4-mapped IPv6 address of an IPv4 portal',
+);
+ok(!nv('parse_nvme_portals', '198.51.100.11:70000', 1), 'rejects an invalid port');
+ok(!nv('parse_nvme_portals', '[198.51.100.11]', 1), 'rejects IPv4 in brackets');
+my $too_many = join(',', map { "198.51.100.$_:4420" } 1 .. 17);
+ok(!nv('parse_nvme_portals', $too_many, 1), 'limits the number of paths');
+
+is_deeply(
+ nv(
+ '_configured_portals',
+ {
+ 'nvme-portals' => '198.51.100.11:4420,198.51.100.12:4421',
+ 'nvme-host-ifaces' => 'ens20,ens21',
+ },
+ ),
+ [
+ { address => '198.51.100.11', port => 4420, family => 'ipv4', host_iface => 'ens20' },
+ { address => '198.51.100.12', port => 4421, family => 'ipv4', host_iface => 'ens21' },
+ ],
+ 'binds each portal to its local interface',
+);
+eval {
+ nv(
+ '_configured_portals',
+ {
+ 'nvme-portals' => '198.51.100.11,198.51.100.12',
+ 'nvme-host-ifaces' => 'ens20',
+ },
+ );
+};
+like($@, qr/one interface for each/, 'portal and interface counts must match');
+ok(!nv('parse_nvme_host_ifaces', 'ens20,not/an/interface', 1), 'rejects an invalid interface name');
+for my $name ('.', '..') {
+ ok(!nv('parse_nvme_host_ifaces', "ens20,$name", 1), "rejects the directory name '$name'");
+}
+for my $option ('nvme-host-ifaces', 'nvme-host-nqns') {
+ my $opts = $PLUGIN->options()->{$option};
+ ok(!$opts->{optional} && !$opts->{fixed}, "$option is required, and can be changed");
+ my $config = scfg(type => 'zfsnvme', content => 'images', blocksize => '16k');
+ delete $config->{$option};
+ eval { $PLUGIN->check_config('st', $config, 1, 1) };
+ like($@, qr/missing value for required option '\Q$option\E'/, "a new storage needs $option");
+}
+$nvme_mock->redefine(_local_iface_exists => sub($iface) { return $iface eq 'ens20' });
+eval { nv('_validate_local_ifaces', [{ host_iface => 'ens20' }, { host_iface => 'ens21' }]); };
+like($@, qr/host interface 'ens21' does not exist/, 'a missing local interface fails preflight');
+$nvme_mock->redefine(_local_iface_exists => sub($iface) { return 1 });
+
+my $listener_used = sub($address, $port, $storeid = 'existing') {
+ return "NVMe/TCP portal '$address' port $port is already used by storage '$storeid';"
+ . " give each storage its own address or port\n";
+};
+# [name, the existing storage, our overrides, error]
+for my $case (
+ [
+ 'the NQN of another storage',
+ { server => '192.0.2.11', pool => 'tank/existing', subsysnqn => $NQN },
+ {},
+ qr/NQN is already used by storage 'existing'/,
+ ],
+ [
+ 'the NQN and the listener of another storage',
+ { pool => 'data', subsysnqn => $NQN, 'nvme-portals' => '192.0.2.21' },
+ {},
+ "NVMe subsystem NQN is already used by storage 'existing'\n",
+ ],
+ [
+ 'the pool of another storage',
+ { pool => 'tank' },
+ {},
+ qr/ZFS pool 'tank' on '192\.0\.2\.10' is already used/,
+ ],
+ [
+ 'a pool inside ours',
+ { pool => 'tank/sub' },
+ {},
+ qr/ZFS pool 'tank' on '192\.0\.2\.10' overlaps the pool of storage 'existing'/,
+ ],
+ [
+ 'a pool above ours',
+ { pool => 'tank' },
+ { pool => 'tank/sub' },
+ qr/overlaps the pool of storage 'existing'/,
+ ],
+ [
+ 'a pool above ours on another server',
+ { pool => 'tank', server => '192.0.2.99' },
+ { pool => 'tank/sub' },
+ undef,
+ ],
+ ['a pool around ours', { pool => 'ta' }, {}, undef],
+ ['a parent pool', { pool => 'tan' }, {}, undef],
+ # one NVMe/TCP listener (family, address, port) per storage, on any server
+ [
+ 'the listener of another storage',
+ { pool => 'data', 'nvme-portals' => '192.0.2.31,192.0.2.22' },
+ {},
+ $listener_used->('192.0.2.22', 4420),
+ ],
+ [
+ 'the listener of another storage that names the default port',
+ { pool => 'data', 'nvme-portals' => '192.0.2.21:4420' },
+ {},
+ $listener_used->('192.0.2.21', 4420),
+ ],
+ [
+ 'the listener of a disabled storage',
+ { pool => 'data', disable => 1, 'nvme-portals' => '192.0.2.21' },
+ {},
+ $listener_used->('192.0.2.21', 4420),
+ ],
+ [
+ 'the listener of a storage that spells the server another way',
+ { server => 'nas.example', pool => 'data', 'nvme-portals' => '192.0.2.21' },
+ {},
+ $listener_used->('192.0.2.21', 4420),
+ ],
+ [
+ 'the listener of a storage on another server',
+ { server => '192.0.2.99', pool => 'tank', 'nvme-portals' => '192.0.2.21' },
+ {},
+ $listener_used->('192.0.2.21', 4420),
+ ],
+ [
+ 'an IPv6 listener written another way',
+ { pool => 'data', 'nvme-portals' => '[2001:db8:0::21]' },
+ { 'nvme-portals' => '[2001:DB8::0021]:4420' },
+ $listener_used->('2001:db8::21', 4420),
+ ],
+ [
+ 'an IPv4 listener reached through an IPv4-mapped IPv6 address',
+ { pool => 'data', 'nvme-portals' => '[::ffff:192.0.2.21]' },
+ {},
+ $listener_used->('::ffff:192.0.2.21', 4420),
+ ],
+ [
+ 'an IPv4-mapped IPv6 address of an IPv4 listener',
+ { pool => 'data', 'nvme-portals' => '192.0.2.21' },
+ { 'nvme-portals' => '[::FFFF:c000:215]' },
+ $listener_used->('192.0.2.21', 4420),
+ ],
+ [
+ 'the same addresses on other ports',
+ { pool => 'data', 'nvme-portals' => '192.0.2.21:4421,192.0.2.22:4421' },
+ {},
+ undef,
+ ],
+ [
+ 'the same address on another port through another spelling of the server',
+ {
+ server => 'nas.example',
+ pool => 'data',
+ 'nvme-portals' => '192.0.2.31,192.0.2.22:4421',
+ },
+ {},
+ "storage 'existing' reaches target address '192.0.2.22' through server 'nas.example';"
+ . " use the same server value for both storages\n",
+ ],
+ [
+ 'an IPv6 address written another way through another spelling of the server',
+ { server => 'nas.example', pool => 'data', 'nvme-portals' => '[2001:db8::0:21]:4421' },
+ { 'nvme-portals' => '[2001:db8::21]' },
+ qr/\Astorage 'existing' reaches target address '2001:db8::21' through server/,
+ ],
+ [
+ 'other addresses on another server',
+ { server => 'nas.example', pool => 'tank', 'nvme-portals' => '192.0.2.31,192.0.2.32' },
+ {},
+ undef,
+ ],
+ [
+ 'other addresses on the same server',
+ { pool => 'data', 'nvme-portals' => '192.0.2.31,[2001:db8::22]' },
+ {},
+ undef,
+ ],
+ ['a storage without portals', { pool => 'data' }, {}, undef],
+ # it cannot be activated, and is checked when its portals are fixed
+ [
+ 'a storage whose portals do not parse',
+ { pool => 'data', 'nvme-portals' => '192.0.2.21,192.0.2.300' },
+ {},
+ undef,
+ ],
+ [
+ 'the listener of another storage type',
+ { type => 'zfs', pool => 'tank', 'nvme-portals' => '192.0.2.21' },
+ {},
+ undef,
+ ],
+) {
+ my ($name, $other, $ours, $error) = $case->@*;
+ my $existing = {
+ type => 'zfsnvme',
+ server => '192.0.2.10',
+ subsysnqn => $OTHER_NQN,
+ pool => 'x',
+ $other->%*,
+ };
+ my $cfg = { ids => { existing => $existing } };
+ eval { nv('_assert_unique_target', 'new', scfg($ours->%*), $cfg) };
+ if (ref($error)) {
+ like($@, $error, "refuses $name");
+ } elsif (defined($error)) {
+ is($@, $error, "refuses $name");
+ } else {
+ is($@, '', "accepts $name");
+ }
+}
+
+{
+ my $states = {
+ '198.51.100.11:4420' => [{ state => 'live', host_iface => 'eth0' }],
+ '198.51.100.12:4420' => [
+ { state => 'connecting', host_iface => 'eth1' },
+ { state => 'live', host_iface => 'ens21' },
+ ],
+ '198.51.100.13:4420' => [{ state => 'dead', host_iface => 'ens22' }],
+ };
+ my $portals = [
+ { address => '198.51.100.11', port => 4420, host_iface => 'ens20' },
+ { address => '198.51.100.12', port => 4420, host_iface => 'ens21' },
+ { address => '198.51.100.13', port => 4420, host_iface => 'ens22' },
+ ];
+ is(nv('_live_portal_count', $states, $portals), 1, 'the wrong interface is not healthy');
+ is(nv('_live_portal_count', $states, $portals, 1), 2, 'but it carries I/O');
+}
+
+# DHHC-1:<hash>:<base64 of the key and its little-endian CRC-32>:
+for my $case (['00', 32], ['00', 48], ['00', 64], ['01', 32], ['02', 48], ['03', 64]) {
+ my $key = make_key($case->@*);
+ is(nv('_validate_secret', $key), $key, "accepts a $case->[1] byte key with hash $case->[0]");
+}
+my $bad_crc = do {
+ my $raw = 'k' x 32;
+ 'DHHC-1:00:' . encode_base64($raw . pack('V', crc32($raw) ^ 1), '') . ':';
+};
+for my $case (
+ ['a plaintext secret', 'plaintext'],
+ ['a key without the final colon', substr($KEY, 0, -1)],
+ ['an unknown hash', make_key('04', 32)],
+ ['a 48 byte key for SHA-256', make_key('01', 48)],
+ ['a 32 byte key for SHA-384', make_key('02', 32)],
+ ['a 32 byte key for SHA-512', make_key('03', 32)],
+ ['a 16 byte key', make_key('00', 16)],
+ ['a wrong CRC', $bad_crc],
+ ['truncated base64', substr($KEY, 0, -2) . ':'],
+ ['a key with a trailing newline', "$KEY\n"],
+) {
+ my ($name, $key) = $case->@*;
+ eval { nv('_validate_secret', $key) };
+ is($@, "invalid NVMe DH-HMAC-CHAP key representation\n", "rejects $name without echoing it");
+}
+eval { nv('_validate_secret', undef) };
+is($@, "missing NVMe DH-HMAC-CHAP key\n", 'a missing key');
+# check_config only validates. Defaults are applied where they are used.
+{
+ $parent_mock->redefine(
+ check_config => sub($class, $id, $config, $create, $skip = undef) { return $config },
+ );
+ my $config = scfg();
+ my $copy = { $config->%* };
+ is_deeply($PLUGIN->check_config('st', $config, 1, 1), $copy, 'no defaults on create');
+ is_deeply(
+ $PLUGIN->check_config('st', { 'nvme-host-ifaces' => 'ens20,ens21' }, 0),
+ { 'nvme-host-ifaces' => 'ens20,ens21' },
+ 'a partial update is accepted',
+ );
+ eval { $PLUGIN->check_config('st', scfg('nvme-fast-io-fail-tmo' => 700), 1, 1) };
+ like($@, qr/must not exceed nvme-ctrl-loss-tmo/, 'validated against the default');
+ eval { $PLUGIN->check_config('st', scfg('nvme-host-ifaces' => 'ens19'), 1, 1) };
+ like($@, qr/one interface for each/, 'cross-field validation stays');
+ # the pool is validated like every later use of it validates it
+ is(
+ $PLUGIN->check_config('st', scfg(pool => 'tank/nvme.1'), 1, 1)->{pool},
+ 'tank/nvme.1',
+ 'a nested pool is accepted',
+ );
+ for my $pool ('-rH', 'tank/vm 1', '') {
+ eval { $PLUGIN->check_config('st', scfg(pool => $pool), 1, 1) };
+ is($@, "invalid ZFS pool name\n", "the pool name '$pool' is refused on create");
+ eval { $PLUGIN->check_config('st', { pool => $pool }, 0) };
+ is($@, "invalid ZFS pool name\n", 'and on update');
+ }
+ $parent_mock->unmock('check_config');
+}
+eval {
+ nv(
+ '_validate_fail_fast_timeout',
+ { 'nvme-ctrl-loss-tmo' => 30, 'nvme-fast-io-fail-tmo' => 31 },
+ 600,
+ );
+};
+like($@, qr/must not exceed nvme-ctrl-loss-tmo/, 'fast I/O fail cannot outlive the loss timeout');
+eval {
+ nv(
+ '_validate_fail_fast_timeout',
+ { 'nvme-ctrl-loss-tmo' => -1, 'nvme-fast-io-fail-tmo' => 30 },
+ 600,
+ );
+};
+is($@, '', 'fast I/O fail stays valid with infinite reconnects');
+
+# ---------------------------------------------------------------------------
+# Storage hooks and secrets
+# ---------------------------------------------------------------------------
+
+{
+ my $secret = '/etc/pve/priv/storage/nvmetest.nvme-dhchap';
+ local %CONFIG = ();
+ local %FILES = ();
+ local @WRITES = ();
+ eval { $PLUGIN->on_add_hook('nvmetest', scfg()) };
+ is($@, "missing NVMe DH-HMAC-CHAP key\n", 'a new storage needs a key');
+ eval { $PLUGIN->on_add_hook('nvmetest', scfg(), 'dhchap-key' => $KEY) };
+ is($@, '', 'a new storage stores its key');
+ is_deeply(
+ [grep { $_->{path} } @WRITES],
+ [{ path => $secret, data => "$KEY\n", perm => 0600 }],
+ 'readable by root only',
+ );
+ is_deeply(
+ [map { $_->{make_path} } grep { $_->{make_path} } @WRITES],
+ [['/etc/pve/priv/storage', { mode => 0700 }]],
+ 'in a private directory',
+ );
+
+ local $FILES{$secret} = $KEY;
+ @WRITES = ();
+ eval {
+ $PLUGIN->on_update_hook_full(
+ 'nvmetest',
+ scfg(),
+ { 'nvme-host-ifaces' => 'ens21,ens22' },
+ );
+ };
+ is($@, '', 'a partial update validates against the current configuration');
+ eval { $PLUGIN->on_update_hook_full('nvmetest', scfg(), {}, undef, { 'dhchap-key' => $KEY }) };
+ is($@, '', 'the same key is accepted');
+ eval {
+ $PLUGIN->on_update_hook_full('nvmetest', scfg(), {}, undef, { 'dhchap-key' => $KEY_B });
+ };
+ like($@, qr/NVMe DH-HMAC-CHAP key rotation is not supported/, 'a different key is refused');
+ eval { $PLUGIN->on_update_hook_full('nvmetest', scfg(), { 'nvme-host-nqns' => $hostnqn_a }) };
+ like($@, qr/removing NVMe host NQN '\Q$hostnqn_b\E' is not supported/, 'no host removal');
+ eval { $PLUGIN->on_update_hook_full('nvmetest', scfg(), {}, ['nvme-host-nqns']) };
+ like($@, qr/at least one NVMe host NQN is required/, 'deleting the host list is refused');
+ @WRITES = ();
+ eval {
+ my $update = { 'nvme-host-nqns' => "$hostnqn_a,$hostnqn_b,nqn.x:c" };
+ $PLUGIN->on_update_hook_full('nvmetest', scfg(), $update);
+ };
+ is($@, '', 'adding a host is accepted');
+ is_deeply(\@WRITES, [], 'without storing the key again');
+
+ delete local $FILES{$secret};
+ @WRITES = ();
+ eval { $PLUGIN->on_update_hook_full('nvmetest', scfg(), {}, undef, { 'dhchap-key' => $KEY }) };
+ is($@, '', 'a storage without key file');
+ is_deeply(
+ [grep { $_->{path} } @WRITES],
+ [{ path => $secret, data => "$KEY\n", perm => 0600 }],
+ 'stores the new key',
+ );
+}
+
+# nvmet keeps one key per host NQN: storages on one target that share a host
+# must share the key.
+{
+ my $other = {
+ type => 'zfsnvme',
+ server => '192.0.2.10',
+ pool => 'data',
+ subsysnqn => $OTHER_NQN,
+ 'nvme-host-nqns' => $hostnqn_b,
+ };
+ my $cfg = { ids => { other => $other, nvmetest => scfg() } };
+ local %FILES = ('/etc/pve/priv/storage/other.nvme-dhchap' => $KEY_B);
+ eval { nv('_assert_shared_host_key', 'nvmetest', scfg(), $KEY, $cfg) };
+ is(
+ $@,
+ "storage 'other' uses a different DH-HMAC-CHAP key for the same NVMe host NQNs"
+ . " on '192.0.2.10'\n",
+ 'a different key for a shared host is refused',
+ );
+ for my $case (
+ ['the same key', $KEY_B, {}],
+ ['another server', $KEY, { server => '192.0.2.11' }],
+ ['no shared host', $KEY, { 'nvme-host-nqns' => 'nqn.2026-01.com.example:other-host' }],
+ ['another storage type', $KEY, { type => 'zfs' }],
+ ) {
+ my ($name, $key, $change) = $case->@*;
+ my $ids = { ids => { other => { $other->%*, $change->%* } } };
+ eval { nv('_assert_shared_host_key', 'nvmetest', scfg(), $key, $ids); };
+ is($@, '', "accepts $name");
+ }
+ {
+ local %FILES = ();
+ eval { nv('_assert_shared_host_key', 'nvmetest', scfg(), $KEY, $cfg); };
+ is($@, '', 'a storage without a key file is skipped');
+ }
+
+ local %CONFIG = (other => $other);
+ local @WRITES = ();
+ eval { $PLUGIN->on_add_hook('nvmetest', scfg(), 'dhchap-key' => $KEY) };
+ like($@, qr/storage 'other' uses a different DH-HMAC-CHAP key/, 'on_add_hook refuses it');
+ unlike($@, qr/DHHC-1/, 'without naming a key');
+ is_deeply([grep { $_->{path} } @WRITES], [], 'and stores no key');
+ eval { $PLUGIN->on_add_hook('nvmetest', scfg(), 'dhchap-key' => $KEY_B) };
+ is($@, '', 'the same key is accepted');
+
+ local $FILES{'/etc/pve/priv/storage/nvmetest.nvme-dhchap'} = $KEY;
+ my $current = scfg('nvme-host-nqns' => $hostnqn_a);
+ eval {
+ $PLUGIN->on_update_hook_full(
+ 'nvmetest',
+ $current,
+ { 'nvme-host-nqns' => "$hostnqn_a,$hostnqn_b" },
+ );
+ };
+ like($@, qr/storage 'other' uses a different DH-HMAC-CHAP key/, 'nor shared by an update');
+ eval { $PLUGIN->on_update_hook_full('nvmetest', $current, { 'nvme-iopolicy' => 'numa' }) };
+ is($@, '', 'other updates are accepted');
+}
+
+# Adding and updating a storage check its listeners; an update checks the new
+# portals, and the current definition of the storage itself does not count.
+{
+ my $secret = '/etc/pve/priv/storage/nvmetest.nvme-dhchap';
+ my $other = scfg(
+ subsysnqn => $OTHER_NQN,
+ pool => 'data',
+ 'nvme-portals' => '192.0.2.23',
+ 'nvme-host-ifaces' => 'ens21',
+ );
+ local %CONFIG = (other => $other);
+ local %FILES = ();
+ local @WRITES = ();
+ my $shared = scfg('nvme-portals' => '192.0.2.21,192.0.2.23');
+ eval { $PLUGIN->on_add_hook('nvmetest', $shared, 'dhchap-key' => $KEY) };
+ is($@, $listener_used->('192.0.2.23', 4420, 'other'), 'on_add_hook refuses a used listener');
+ is_deeply([grep { $_->{path} } @WRITES], [], 'and stores no key');
+
+ %CONFIG = (other => $other, nvmetest => scfg());
+ $FILES{$secret} = $KEY;
+ my $update = sub($current, $portals) {
+ eval {
+ $PLUGIN->on_update_hook_full('nvmetest', $current, { 'nvme-portals' => $portals });
+ };
+ return $@;
+ };
+ is($update->(scfg(), '192.0.2.22,192.0.2.21'), '', 'an update keeps its own listeners');
+ is(
+ $update->(scfg(), '192.0.2.21,192.0.2.23:4420'),
+ $listener_used->('192.0.2.23', 4420, 'other'),
+ 'on_update_hook_full refuses a used listener',
+ );
+ is($update->(scfg(), '192.0.2.21,192.0.2.23:4421'), '', 'but not its address on another port');
+ is($update->($shared, '192.0.2.21,192.0.2.22'), '', 'an update can leave a used listener');
+ is_deeply([grep { $_->{path} } @WRITES], [], 'without storing the key again');
+}
+
+# pvesm reads the key from a file, so it never appears on a command line.
+{
+ require PVE::CLI::pvesm;
+
+ my $dir = tempdir(CLEANUP => 1);
+ my $file = sub($name, $content) {
+ open(my $fh, '>', "$dir/$name") or die "open: $!\n";
+ print {$fh} $content;
+ close($fh);
+ return "$dir/$name";
+ };
+ my $padded = $file->('padded', " $KEY \nsecond line\n");
+ my $empty = $file->('empty', "\n");
+ my %map = map {
+ my $command = $_;
+ (
+ $command => (
+ grep { $_->{name} eq 'dhchap-key' }
+ PVE::CLI::pvesm::param_mapping($command)->@*
+ )[0],
+ )
+ } qw(create update);
+ ok($map{create} && $map{update}, 'pvesm create and update map dhchap-key');
+ my $read = $map{create}->{func};
+ eval { $read->($KEY) };
+ is(
+ $@,
+ "dhchap-key expects the path of a file containing the key\n",
+ 'a key in place of the file name is refused without repeating it',
+ );
+ is($read->($padded), $KEY, 'the first line of the file, trimmed');
+ eval { $read->($empty) };
+ like($@, qr/key file '.*' is empty/, 'an empty file is refused');
+ eval { $read->('/dev/null') };
+ like($@, qr/is empty/, 'a file that is not regular, like /dev/stdin, is read');
+ eval { $read->($dir) };
+ like($@, qr/expects the path of a file/, 'a directory is refused');
+}
+
+# ---------------------------------------------------------------------------
+# Volumes through the target runner
+# ---------------------------------------------------------------------------
+
+sub target(%answers) {
+ return sub($steps, %opts) {
+ my $answer = $answers{ $opts{op} } // die "unexpected NVMe target call '$opts{op}'\n";
+ return $answer->($steps, %opts) if ref($answer) eq 'CODE';
+ return { rc => 0, out => $answer, err => '' };
+ };
+}
+
+my @volume_list = (
+ "tank\t-\t-\tfilesystem",
+ "tank/vm-100-disk-0\t1048576\t-\tvolume",
+ "tank/vm-200-disk-0\t1048576\t-\tvolume",
+ "tank/vm-300-disk-0\t1048576\t-\tvolume",
+ "tank/vm-400-disk-0\t1048576\t-\tvolume",
+ "tank/base-90-disk-0\t1048576\t-\tvolume",
+ "tank/vm-101-disk-0\t1048576\ttank/base-90-disk-0\@__base__\tvolume",
+ "tank/vm-102-disk-0\t1048576\ttankXprod/base-90-disk-0\@__base__\tvolume",
+ "tank/subvol-103-disk-0\t-\t-\tfilesystem",
+ "tank/vm-104-disk-0\t-\t-\tfilesystem",
+);
+my @ownership = (
+ "tank\t-\t-",
+ "tank/vm-100-disk-0\t$NQN\tlocal",
+ "tank/vm-200-disk-0\t-\t-",
+ "tank/vm-300-disk-0\t$NQN\tinherited from tank",
+ "tank/vm-400-disk-0\t$NQN\treceived",
+ "tank/base-90-disk-0\t$NQN\tlocal",
+ "tank/vm-101-disk-0\t$NQN\tlocal",
+ "tank/vm-102-disk-0\t$NQN\tlocal",
+ "tank/subvol-103-disk-0\t$OTHER_NQN\tlocal",
+ "tank/vm-104-disk-0\t$NQN\tlocal",
+);
+{
+ local $TARGET = target('zfs list' => \@volume_list, 'zfs get' => \@ownership);
+ my $images = $PLUGIN->list_images('nvmetest', scfg());
+ my %by_volid = map { $_->{volid} => $_ } $images->@*;
+ my %by_name = map { $_->{name} => $_ } $images->@*;
+ is_deeply(
+ [sort keys %by_name],
+ ['base-90-disk-0', 'vm-100-disk-0', 'vm-101-disk-0', 'vm-102-disk-0', 'vm-400-disk-0'],
+ 'listing requires a zvol with a local or received owner',
+ );
+ is(
+ $by_name{'vm-102-disk-0'}->{parent},
+ 'tankXprod/base-90-disk-0@__base__',
+ 'a foreign pool origin is not mistaken for a local parent',
+ );
+ is_deeply(
+ $by_volid{'nvmetest:base-90-disk-0/vm-101-disk-0'},
+ {
+ name => 'vm-101-disk-0',
+ size => 1048576,
+ parent => 'base-90-disk-0@__base__',
+ format => 'raw',
+ vmid => 101,
+ volid => 'nvmetest:base-90-disk-0/vm-101-disk-0',
+ },
+ 'a linked clone keeps the volume ID format of the ZFS plugins',
+ );
+ is_deeply(
+ [map { $_->{volid} } $PLUGIN->list_images('nvmetest', scfg(), 400)->@*],
+ ['nvmetest:vm-400-disk-0'],
+ 'filtered by VM',
+ );
+ my $listed = $PLUGIN->list_images('nvmetest', scfg(), undef, ['nvmetest:vm-100-disk-0']);
+ is_deeply([map { $_->{volid} } $listed->@*], ['nvmetest:vm-100-disk-0'], 'by volume list');
+ is(
+ $PLUGIN->find_free_diskname('nvmetest', scfg(), 200),
+ 'vm-200-disk-1',
+ 'a name held by an unowned zvol is not handed out',
+ );
+ is($PLUGIN->find_free_diskname('nvmetest', scfg(), 90), 'vm-90-disk-1', 'nor one of a base');
+ is(
+ $PLUGIN->find_free_diskname('nvmetest', scfg(), 104),
+ 'vm-104-disk-1',
+ 'nor one held by a filesystem',
+ );
+ my @steps = map { $_->{steps}->@* } @TARGET_CALLS;
+ ok(!grep({ $_->[0] ne 'zfs' || $_->[1] !~ /\A(?:get|list)\z/ } @steps), 'listing only reads');
+}
+
+{
+ my @warnings;
+ local $SIG{__WARN__} = sub($warning) { push @warnings, $warning };
+ for my $case (
+ [['100', '200'], [300, 100, 200, 1]],
+ [['-', '200'], [0, 0, 0, 0]],
+ [['1'], [0, 0, 0, 0]],
+ ) {
+ local $TARGET = target('zfs get' => $case->[0]);
+ is_deeply([$PLUGIN->status('nvmetest', scfg())], $case->[1], "status for '$case->[0]->@*'");
+ }
+ local $TARGET =
+ target('zfs get' => sub(@args) { return { rc => 255, out => [], err => 'no route' } });
+ is_deeply([$PLUGIN->status('nvmetest', scfg())], [0, 0, 0, 0], 'unreachable is inactive');
+ is(scalar(@warnings), 3, 'and every inactive status warns');
+ like($warnings[0], qr/unexpected ZFS pool usage for storage 'nvmetest'/, 'about the usage');
+ is(
+ $warnings[-1],
+ "storage 'nvmetest': zfs error on '192.0.2.10': no route\n",
+ 'or naming the storage and the target',
+ );
+}
+
+{
+ local @TARGET_CALLS = ();
+ local $TARGET = target();
+ eval { $PLUGIN->volume_resize(scfg(), 'nvmetest', 'vm-100-disk-0', 2 * 1024**3, 1) };
+ like(
+ $@,
+ qr/online resize is not supported for NVMe\/TCP block devices; stop the VM first/,
+ 'online resize is refused',
+ );
+ eval { $PLUGIN->volume_resize(scfg(), 'nvmetest', 'vm-100-disk-0', 2 * 1024**3, 0, 'snap') };
+ like($@, qr/resizing a snapshot is not supported/, 'as is resizing a snapshot');
+ is_deeply(\@TARGET_CALLS, [], 'before any target call');
+ for my $method (qw(volume_export volume_import rename_volume rename_snapshot)) {
+ eval { $PLUGIN->$method(scfg(), 'nvmetest', 'vm-100-disk-0') };
+ like($@, qr/not supported/, "$method is not supported");
+ }
+ for my $direction (qw(export import)) {
+ my $formats = "volume_${direction}_formats";
+ is_deeply(
+ [$PLUGIN->$formats(scfg(), 'nvmetest', 'vm-100-disk-0')],
+ [],
+ "no $direction formats",
+ );
+ }
+ is($PLUGIN->deactivate_volume('nvmetest', scfg(), 'vm-100-disk-0'), 1, 'no-op deactivation');
+ eval { $PLUGIN->deactivate_volume('nvmetest', scfg(), 'vm-100-disk-0', 'snap') };
+ like($@, qr/unable to deactivate snapshot/, 'snapshots cannot be deactivated');
+}
+
+{
+ $nvme_mock->redefine(_nvmet_volume_uuid => sub($scfg, $name) { return $UUID });
+ is_deeply(
+ $PLUGIN->qemu_blockdev_options(scfg(), 'nvmetest', 'vm-100-disk-0'),
+ { driver => 'host_device', filename => "/dev/disk/by-id/nvme-uuid.$UUID" },
+ 'the QEMU host_device driver on the stable namespace UUID link',
+ );
+ eval {
+ my $options = { 'snapshot-name' => 's' };
+ $PLUGIN->qemu_blockdev_options(scfg(), 'nvmetest', 'vm-100-disk-0', undef, $options);
+ };
+ like($@, qr/direct access to snapshots not implemented/, 'snapshots have no block device');
+ $nvme_mock->unmock('_nvmet_volume_uuid');
+}
+
+# ---------------------------------------------------------------------------
+# The runner: one ssh argv, no Perl, no shell on the local side
+# ---------------------------------------------------------------------------
+
+{
+ my $run = $nvme_mock->original('_nvmet_run');
+ my @runs;
+ local $COMMAND = sub($cmd, %opts) {
+ push @runs, { cmd => $cmd, %opts };
+ return 0;
+ };
+ my $steps = [['zfs', 'get', '-H', 'type', 'tank'], ['printf', '%s\n', 'x y']];
+ my $res = $run->(scfg(), $steps, op => 'test');
+ is_deeply(
+ $runs[0]->{cmd},
+ [
+ '/usr/bin/ssh',
+ '-o',
+ 'BatchMode=yes',
+ '-o',
+ 'ConnectTimeout=10',
+ '-o',
+ 'LogLevel=ERROR',
+ '-i',
+ '/etc/pve/priv/zfs/192.0.2.10_id_rsa',
+ 'root@192.0.2.10',
+ q{zfs get -H type tank && printf '%s\n' 'x y'},
+ ],
+ 'one ssh argv with the rendered command, and no login banner in errors',
+ );
+ unlike(join(' ', $runs[0]->{cmd}->@*), qr/perl/, 'no Perl runs on the target');
+ is($runs[0]->{timeout}, 15, 'reads time out after 15 seconds by default');
+ ok(!$runs[0]->{noerr} && !exists($runs[0]->{input}),
+ 'failures are exceptions, stdin is closed');
+ is_deeply($res, { rc => 0, out => [], err => '' }, 'a successful call');
+
+ @runs = ();
+ $run->(
+ scfg(),
+ [{ key => ["$ROOT/hosts/$hostnqn_a/dhchap_key"] }],
+ op => 'key',
+ input => "$KEY\n",
+ timeout => 7,
+ );
+ is($runs[0]->{input}, "$KEY\n", 'the key goes to stdin');
+ unlike(join(' ', $runs[0]->{cmd}->@*), qr/DHHC-1/, 'and never into the command line');
+ is($runs[0]->{timeout}, 7, 'with the timeout of the caller');
+
+ for my $case (
+ [
+ 'output and the first error line',
+ sub(%o) {
+ $o{outfunc}->('a');
+ $o{outfunc}->('b');
+ $o{errfunc}->('');
+ $o{errfunc}->('first');
+ $o{errfunc}->('second');
+ die "command 'ssh' failed: exit code 3\n";
+ },
+ { rc => 3, out => ['a', 'b'], err => 'first' },
+ ],
+ [
+ 'an exit code without message',
+ sub(%o) { die "command 'ssh' failed: exit code 2\n" },
+ { rc => 2, out => [], err => 'exit code 2' },
+ ],
+ [
+ 'a timeout',
+ sub(%o) { die "command 'ssh' failed: got timeout\n" },
+ { rc => -1, out => [], err => 'timeout' },
+ ],
+ [
+ 'an ssh that could not connect',
+ sub(%o) { die "command 'ssh' failed: exit code 255\n" },
+ { rc => 255, out => [], err => 'exit code 255' },
+ ],
+ [
+ 'an ssh killed by a signal',
+ sub(%o) { die "command 'ssh' failed: got signal 9\n" },
+ { rc => -1, out => [], err => 'ssh failed' },
+ ],
+ ) {
+ my ($name, $behaviour, $expected) = $case->@*;
+ local $COMMAND = sub($cmd, %opts) { return $behaviour->(%opts) };
+ is_deeply(
+ $run->(scfg(), [['cat', '/proc/mounts']], op => 'test'),
+ $expected,
+ "returns $name",
+ );
+ }
+ for my $exception ("received interrupt\n", "command 'ssh $KEY' failed: received interrupt\n") {
+ local $COMMAND = sub($cmd, %opts) { $opts{outfunc}->('partial'); die $exception };
+ eval { $run->(scfg(), [['cat', '/proc/mounts']], op => 'test') };
+ is($@, "received interrupt\n", 'a stopped task dies with the task marker and nothing else');
+ }
+ @runs = ();
+ eval { $run->(scfg(), [['cat', '/proc/mounts']]) };
+ like($@, qr/NVMe target call without label/, 'every call has a label');
+ eval { $run->(scfg(), [['printf', '%s', 'x' x 65536]], op => 'test') };
+ like($@, qr/NVMe target command too long/, 'oversized calls are refused');
+ is_deeply(\@runs, [], 'before running anything');
+}
+
+# ---------------------------------------------------------------------------
+# The shared storage lock
+# ---------------------------------------------------------------------------
+
+subtest 'vdisk_alloc dispatches under the shared storage lock' => sub {
+ my $cluster_mock = Test::MockModule->new('PVE::Cluster');
+ my (@locks, @allocations);
+ $cluster_mock->redefine(
+ cfs_lock_storage => sub($storeid, $timeout, $func, @param) {
+ push @locks, [$storeid, $timeout];
+ return $func->(@param);
+ },
+ );
+ $parent_mock->redefine(lookup => sub($class, $type) { return $PLUGIN });
+ $storage_mock->redefine(activate_storage => sub($cfg, $storeid) { return 1 });
+ $nvme_mock->redefine(
+ alloc_image => sub($class, $storeid, $scfg, $vmid, $fmt, $name, $size) {
+ push @allocations, [$storeid, $vmid, $fmt, $name, $size];
+ return $name;
+ },
+ );
+ my $cfg = { ids => { nvmetest => scfg(shared => 1) } };
+ is(
+ PVE::Storage::vdisk_alloc($cfg, 'nvmetest', 105, 'raw', 'vm-105-disk-0', 131_072),
+ 'nvmetest:vm-105-disk-0',
+ 'core allocation returns the allocated volume ID',
+ );
+ is_deeply(\@locks, [['nvmetest', undef]], 'cfs_lock_storage gets the timeout of the core');
+ is_deeply(
+ \@allocations,
+ [['nvmetest', 105, 'raw', 'vm-105-disk-0', 131_072]],
+ 'alloc_image runs once inside the lock',
+ );
+ $nvme_mock->unmock('alloc_image');
+ $storage_mock->unmock('activate_storage');
+ $parent_mock->unmock('lookup');
+};
+
+# ---------------------------------------------------------------------------
+# Local NVMe host side
+# ---------------------------------------------------------------------------
+
+# The connect options of a controller: one write to the fabrics device.
+{
+ my $portal = { address => '192.0.2.22', port => 4420, host_iface => 'ens21' };
+ my $options = sub($scfg) {
+ return nv(
+ '_fabrics_options', $scfg, $portal, $hostnqn_a, $UUID, $KEY,
+ );
+ };
+ my ($string, $names) = $options->({ subsysnqn => $NQN });
+ is(
+ $string,
+ "transport=tcp,traddr=192.0.2.22,trsvcid=4420,host_iface=ens21,nqn=$NQN,"
+ . "hostnqn=$hostnqn_a,hostid=$UUID,dhchap_secret=$KEY,keep_alive_tmo=5,"
+ . 'reconnect_delay=2,ctrl_loss_tmo=600',
+ 'the portal, the identities, the key and the default timeouts',
+ );
+ is_deeply(
+ $names,
+ [
+ qw(transport traddr trsvcid host_iface nqn hostnqn hostid dhchap_secret),
+ qw(keep_alive_tmo reconnect_delay ctrl_loss_tmo),
+ ],
+ 'with the names of the options',
+ );
+ my $tuned = {
+ subsysnqn => $NQN,
+ 'nvme-keep-alive-tmo' => 10,
+ 'nvme-reconnect-delay' => 5,
+ 'nvme-ctrl-loss-tmo' => 60,
+ 'nvme-fast-io-fail-tmo' => 15,
+ 'nvme-nr-io-queues' => 4,
+ };
+ ($string, $names) = $options->($tuned);
+ my $tail = 'keep_alive_tmo=10,reconnect_delay=5,ctrl_loss_tmo=60,'
+ . 'fast_io_fail_tmo=15,nr_io_queues=4';
+ like($string, qr/,\Q$tail\E\z/, 'the configured timeouts and queues');
+ is_deeply([$names->@[-2, -1]], ['fast_io_fail_tmo', 'nr_io_queues'], 'are named too');
+ ($string) = $options->({ $tuned->%*, 'nvme-fast-io-fail-tmo' => 0 });
+ like($string, qr/,fast_io_fail_tmo=0,/, 'a fast I/O fail timeout of 0 is a real value');
+ ($string) = $options->({ $tuned->%*, 'nvme-ctrl-loss-tmo' => -1 });
+ like($string, qr/,ctrl_loss_tmo=-1,/, 'an infinite loss timeout');
+
+ for my $value ('6 0', '60,duplicate_connect', "60\x00", '') {
+ eval { $options->({ $tuned->%*, 'nvme-ctrl-loss-tmo' => $value }) };
+ is(
+ $@,
+ "internal error: invalid NVMe connect option 'ctrl_loss_tmo'\n",
+ 'a value that would end the option early is refused without echoing it',
+ );
+ }
+}
+
+# The connect runs in a child bounded by 10 seconds. Its result and errors
+# pass through; a stopped task and a timeout have their own message.
+{
+ my $portal = { address => '192.0.2.21', port => 4420, host_iface => 'ens19' };
+ my $connect = sub($child) {
+ local @FORKS = ();
+ $nvme_mock->redefine(_fabrics_connect => $child);
+ my $res = eval { nv('_connect_portal', scfg(), $portal, $hostnqn_a, $UUID, $KEY); };
+ my $error = $@;
+ $nvme_mock->redefine(_fabrics_connect => $connect_mock);
+ return ($res, $error, [@FORKS]);
+ };
+ my ($res, $error, $forks) = $connect->(sub($options, $names) { return 7 });
+ is($res, 7, 'returns the instance of the new controller');
+ is_deeply($forks, [10], 'from a child bounded by 10 seconds');
+ for my $exception ("received interrupt\n", "interrupted by unexpected signal\n") {
+ (undef, $error) = $connect->(sub(@args) {
+ die $exception;
+ });
+ is($error, "received interrupt\n", 'a stopped task dies with the task marker');
+ }
+ my @warnings;
+ local $SIG{__WARN__} = sub($warning) { push @warnings, $warning };
+ $tools_mock->redefine(
+ run_fork_with_timeout => sub($timeout, $code, $opts = undef) { return (undef, 1) },
+ );
+ (undef, $error) = $connect->(sub(@args) {
+ return 7;
+ });
+ is($error, "NVMe/TCP connect did not complete within 10 seconds\n", 'a timeout');
+ $tools_mock->redefine(
+ run_fork_with_timeout => sub($timeout, $code, $opts = undef) {
+ warn "malformed JSON string\n"; # what a child that died without a result leaves
+ return (undef, 0);
+ },
+ );
+ (undef, $error) = $connect->(sub(@args) {
+ return 7;
+ });
+ is($error, "NVMe/TCP connect left no result\n", 'a child without a result');
+ is_deeply(\@warnings, [], 'without the warning of run_fork_with_timeout in the task log');
+ $tools_mock->redefine(run_fork_with_timeout => $fork_inline);
+}
+
+# The signals a stopped task can bring that are blocked while a child runs.
+my %signal = (HUP => SIGHUP, INT => SIGINT, QUIT => SIGQUIT, TERM => SIGTERM, ALRM => SIGALRM);
+
+sub blocked_signals() {
+ my $mask = POSIX::SigSet->new();
+ sigprocmask(SIG_BLOCK, POSIX::SigSet->new(), $mask) or die "sigprocmask: $!\n";
+ return join(',', grep { $mask->ismember($signal{$_}) } sort keys %signal);
+}
+
+# The child itself: a real fork under the real run_fork_with_timeout.
+subtest 'the connect child' => sub {
+ $tools_mock->unmock('run_fork_with_timeout');
+ my $portal = { address => '192.0.2.21', port => 4420, host_iface => 'ens19' };
+ my $connect = sub() {
+ my $res = eval { nv('_connect_portal', scfg(), $portal, $hostnqn_a, $UUID, $KEY); };
+ return ($res, $@);
+ };
+ $nvme_mock->redefine(_fabrics_connect => sub(@args) { return 7 });
+ my ($res, $error) = $connect->();
+ is($res, 7, 'the result of the child reaches the parent');
+ is(blocked_signals(), '', 'and the signal mask is restored');
+ $nvme_mock->redefine(
+ _fabrics_connect => sub(@args) { die "connect failed: Connection refused\n" },
+ );
+ (undef, $error) = $connect->();
+ is($error, "connect failed: Connection refused\n", 'and so does its error');
+ is(blocked_signals(), '', 'with the signal mask restored');
+
+ # The warnings of the child go to the standard error of the task.
+ my $dir = tempdir(CLEANUP => 1);
+ $nvme_mock->redefine(_fabrics_connect => sub(@args) { warn "child warning\n"; return 7 });
+ open(my $saved, '>&', \*STDERR) or die "cannot save STDERR: $!\n";
+ open(STDERR, '>', "$dir/stderr") or die "cannot redirect STDERR: $!\n";
+ ($res, $error) = $connect->();
+ open(STDERR, '>&', $saved) or die "cannot restore STDERR: $!\n";
+ is($res, 7, 'a child that warns');
+ open(my $fh, '<', "$dir/stderr") or die "cannot read '$dir/stderr': $!\n";
+ is(join('', <$fh>), "child warning\n", 'is heard');
+ close($fh);
+
+ # A stopped task: the signal stays pending while the child runs and, once
+ # the child is reaped, the handler of the worker dies with the task marker.
+ for my $name (qw(TERM QUIT INT)) {
+ pipe(my $from_child, my $to_parent) or die "pipe: $!\n";
+ $nvme_mock->redefine(
+ _fabrics_connect => sub(@args) {
+ kill($name, getppid()) or die "kill: $!\n";
+ print {$to_parent} "$$ " . blocked_signals() . "\n";
+ close($to_parent);
+ return 7;
+ },
+ );
+ my $seen;
+ local $SIG{$name} = sub { # as in a PVE worker
+ chomp(my $report = <$from_child> // '');
+ my ($pid, $blocked) = split(/ /, $report, 2);
+ $seen = { blocked => $blocked, reaped => $pid && waitpid($pid, WNOHANG) == -1 };
+ die "received interrupt\n";
+ };
+ ($res, $error) = $connect->();
+ close($to_parent);
+ close($from_child);
+ is($error, "received interrupt\n", "$name stops the task with the task marker");
+ is(
+ $seen->{blocked},
+ 'HUP,INT,QUIT,TERM',
+ 'once the child, with the inherited mask, is done',
+ );
+ ok($seen->{reaped}, 'and reaped');
+ is(blocked_signals(), '', 'and the signal mask is restored');
+ }
+ $nvme_mock->redefine(_fabrics_connect => $connect_mock);
+ $tools_mock->redefine(run_fork_with_timeout => $fork_inline);
+};
+
+# The body of the connect child, inline. A pseudoterminal stands in for the
+# fabrics device: a character device that answers reads with the lines the
+# test writes to the master, and passes writes to the master.
+subtest 'the connect child body' => sub {
+ my $body = $nvme_mock->original('_fabrics_connect');
+ my $portal = { address => '192.0.2.21', port => 4420, host_iface => 'ens19' };
+ my ($options, $names) = nv('_fabrics_options', scfg(), $portal, $hostnqn_a, $UUID, $KEY);
+ my $tokens = sub(@names) {
+ return join(',', map { "$_=%s" } @names) . "\n";
+ };
+ # a body that waits for a line the test did not write must not hang the suite
+ local $SIG{ALRM} = sub { die "the child body did not finish\n" };
+ my $run = sub() {
+ @RESTRICTED = ();
+ alarm(30);
+ my $res = eval { $body->($options, $names) };
+ alarm(0);
+ return ($res, $@);
+ };
+ my $dir = tempdir(CLEANUP => 1);
+ my $file = "$dir/file";
+ open(my $fh, '>', $file) or die "cannot create '$file': $!\n";
+ close($fh);
+ for my $case (
+ [
+ 'a missing device',
+ "$dir/missing",
+ qr/\A'\Q$dir\E\/missing' does not exist; load the nvme-tcp kernel module\n\z/,
+ ],
+ ['a regular file', $file, qr/\A'\Q$file\E' is not a character device\n\z/],
+ [
+ 'a device without the option list',
+ '/dev/null',
+ qr/\Athe running kernel does not support the NVMe connect option 'transport'\n\z/,
+ ],
+ ) {
+ my ($name, $device, $expected) = $case->@*;
+ $nvme_mock->redefine(_fabrics_device => sub () { return $device });
+ my (undef, $error) = $run->();
+ like($error, $expected, "$name is refused");
+ unlike($error, qr/\Q$KEY\E/, 'without the key in the error');
+ }
+ is(-s $file, 0, 'and nothing is written before the device is checked');
+
+ my $pty = sub() {
+ # TIOCSPTLCK and TIOCGPTN are only known for these architectures
+ return if $Config{archname} !~ m/\A(?:x86_64|i[3-6]86|aarch64|arm|riscv64)-linux/;
+ sysopen(my $master, '/dev/ptmx', O_RDWR | O_NOCTTY) or return;
+ my $zero = pack('i', 0);
+ ioctl($master, 0x40045431, $zero) or return;
+ my $number = pack('i', 0);
+ ioctl($master, 0x80045430, $number) or return;
+ my $slave = '/dev/pts/' . unpack('i', $number);
+ # keeps the line discipline: canonical reads, no echo, no translation
+ sysopen(my $keep, $slave, O_RDWR | O_NOCTTY) or return;
+ my $termios = POSIX::Termios->new();
+ $termios->getattr(fileno($keep)) or die "tcgetattr: $!\n";
+ $termios->setlflag(($termios->getlflag() & ~(ECHO | ECHONL)) | ICANON);
+ $termios->setoflag($termios->getoflag() & ~OPOST);
+ $termios->setattr(fileno($keep), TCSANOW) or die "tcsetattr: $!\n";
+ return ($master, $slave, $keep);
+ };
+ my $received = sub($master) {
+ my $readable = '';
+ vec($readable, fileno($master), 1) = 1;
+ return '' if !select($readable, undef, undef, 0.1);
+ sysread($master, my $buffer, 65536) // die "cannot read the pseudoterminal: $!\n";
+ return $buffer;
+ };
+ my ($master, $slave, $keep) = $pty->();
+ SKIP: {
+ skip 'no pseudoterminal available', 25 if !$master;
+ $nvme_mock->redefine(_fabrics_device => sub () { return $slave });
+
+ my ($res, $error);
+ for my $instance (3, 12) {
+ syswrite($master, $tokens->($names->@*, 'tls'));
+ syswrite($master, "instance=$instance,cntlid=1\n");
+ ($res, $error) = $run->();
+ is($error, '', "a connect (instance $instance)");
+ is($res, $instance, 'returns the instance of the controller');
+ is($received->($master), $options, 'after exactly one write of the options');
+ my $attrs = "/sys/class/nvme/nvme$instance";
+ is_deeply(
+ \@RESTRICTED,
+ ["$attrs/dhchap_secret", "$attrs/dhchap_ctrl_secret"],
+ 'and restricts the secret attributes of that controller',
+ );
+ }
+
+ for my $missing (qw(host_iface nqn)) {
+ syswrite($master,
+ $tokens->(map { $_ eq $missing ? "${_}_suffix" : $_ } $names->@*));
+ ($res, $error) = $run->();
+ is(
+ $error,
+ "the running kernel does not support the NVMe connect option '$missing'\n",
+ "an option the kernel does not list ($missing)",
+ );
+ is($received->($master), '', 'is found before anything is written');
+ }
+
+ for my $result ("garbage\n", "instance=3,cntlid=1,garbage\n") {
+ syswrite($master, $tokens->($names->@*));
+ syswrite($master, $result);
+ ($res, $error) = $run->();
+ is($error, "unexpected connect result\n", 'an unexpected result is refused');
+ is($received->($master), $options, 'after the one write');
+ is_deeply(\@RESTRICTED, [], 'and nothing is restricted');
+ }
+
+ symlink($slave, "$dir/link") or die "symlink: $!\n";
+ $nvme_mock->redefine(_fabrics_device => sub () { return "$dir/link" });
+ ($res, $error) = $run->();
+ is(
+ $error,
+ "cannot open '$dir/link': Too many levels of symbolic links\n",
+ 'a symbolic link to the device is not followed',
+ );
+ $nvme_mock->redefine(_fabrics_device => sub () { return $slave });
+
+ # A failed write: the error has the errno text and never the options.
+ my $writes = 0;
+ $nvme_mock->redefine(
+ _fabrics_write => sub($fh, $data) { $writes++; $! = ECONNREFUSED; return undef },
+ );
+ syswrite($master, $tokens->($names->@*));
+ ($res, $error) = $run->();
+ is(
+ $error,
+ "connect failed: Connection refused\n",
+ 'a refused connect has the errno text',
+ );
+ is($writes, 1, 'after one write, which is not repeated');
+ is_deeply(\@RESTRICTED, [], 'and nothing is restricted');
+ $writes = 0;
+ $nvme_mock->redefine(
+ _fabrics_write => sub($fh, $data) { $writes++; return length($data) - 1 });
+ syswrite($master, $tokens->($names->@*));
+ syswrite($master, "instance=3,cntlid=1\n"); # never read: a short write ends the connect
+ ($res, $error) = $run->();
+ is($error, "connect failed: short write\n", 'a short write');
+ is($writes, 1, 'is not completed');
+ is_deeply(\@RESTRICTED, [], 'and restricts nothing');
+ $nvme_mock->unmock('_fabrics_write');
+ }
+ $nvme_mock->unmock('_fabrics_device');
+};
+
+# Deactivation deletes the controllers of the subsystem in one bounded child.
+{
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ add_controller('nvme3', '192.0.2.23', 4420, 'ens21', 'live', $OTHER_NQN);
+ $nvme_mock->redefine(
+ _namespace_openers =>
+ sub($nqn) { return ['qemu-system-x86_64 (PID 123, /dev/nvme0n1)'] },
+ );
+ eval { $PLUGIN->deactivate_storage('nvmetest', scfg()) };
+ like(
+ $@,
+ qr/refusing to disconnect.*namespace in use by qemu-system-x86_64/s,
+ 'deactivation refuses a namespace opened by a VM',
+ );
+ is_deeply(\@DELETED, [], 'and deletes nothing');
+ $nvme_mock->redefine(_namespace_openers => sub($nqn) { return [] });
+ is($PLUGIN->deactivate_storage('nvmetest', scfg()), 1, 'an unused storage is deactivated');
+ is_deeply(\@DELETED, ['nvme1', 'nvme2'], 'by deleting the controllers of its subsystem');
+ is_deeply(\@FORKS, [15], 'in one child bounded by 15 seconds');
+ is_deeply($DIRS{'/sys/class/nvme'}, ['nvme3'], 'the controllers of other subsystems stay');
+ is($PLUGIN->deactivate_storage('nvmetest', scfg()), 1, 'a storage without controllers');
+ is_deeply(\@FORKS, [15], 'needs no child');
+ $nvme_mock->unmock('_namespace_openers');
+
+ my $delete = $nvme_mock->original('_delete_controllers');
+ @SYSFS_WRITES = ();
+ is($delete->([{ name => 'nvme1' }, { name => 'nvme2' }]), 2, 'the child deletes each one');
+ is_deeply(
+ \@SYSFS_WRITES,
+ [map { ["/sys/class/nvme/$_/delete_controller", '1'] } qw(nvme1 nvme2)],
+ 'through sysfs',
+ );
+ # The stock file_write on files that fail: a failed write warns, with the
+ # errno text, and a failed open does not.
+ my $file_write = $sysfs_mock->original('file_write');
+ my $dir = tempdir(CLEANUP => 1);
+ my @warnings;
+ local $SIG{__WARN__} = sub($warning) { push @warnings, $warning };
+ for my $case (['/dev/full', 'No space left on device'], [$dir, 'Is a directory']) {
+ my ($path, $reason) = $case->@*;
+ $sysfs_mock->redefine(file_write => sub($file, @args) { $file_write->($path, @args) });
+ eval { $delete->([{ name => 'nvme1' }, { name => 'nvme2' }]) };
+ is(
+ $@,
+ "cannot delete NVMe controller 'nvme1': $reason\n",
+ "a failed delete names the controller and the errno text ($reason)",
+ );
+ }
+ $sysfs_mock->redefine(file_write => sub($file, @args) { $file_write->("$dir/a/b", @args) });
+ is(
+ $delete->([{ name => 'nvme1' }, { name => 'nvme2' }]),
+ 2,
+ 'a controller that went away since it was listed counts as deleted',
+ );
+ $sysfs_mock->redefine(
+ file_write => sub($path, $data, @rest) {
+ warn "error writing '$data' to '$path': Device or resource busy\n";
+ $! = 0;
+ return 0;
+ },
+ );
+ eval { $delete->([{ name => 'nvme1' }]) };
+ is(
+ $@,
+ "cannot delete NVMe controller 'nvme1': Device or resource busy\n",
+ 'the errno text of a failed write is the one of the warning',
+ );
+ is_deeply(\@warnings, [], 'which is not passed on');
+ $sysfs_mock->redefine(file_write => $sysfs_write);
+}
+
+# The users of the namespaces: file descriptors on a namespace or one of its
+# partitions, and kernel holders such as device-mapper.
+{
+ reset_host();
+ $DIRS{'/sys/class/nvme-subsystem/nvme-subsys7'} = ['nvme7n1', 'nvme7c1n1'];
+ $DIRS{'/sys/class/block/nvme7n1'} = ['nvme7n1p1', 'nvme7n1p2', 'holders'];
+ is_deeply(nv('_namespace_openers', $NQN), [], 'no device, no users');
+ my ($scan) = grep { $_->[0] eq '/sys/class/block/nvme7n1' } @GLOBS;
+ ok($scan, 'the partitions of the namespace are listed');
+ ok($scan && 'nvme7n1p1' =~ /^($scan->[1])$/ && 'nvme7n1' !~ /^($scan->[1])$/, 'only those');
+ ok(!grep({ $_->[0] eq '/sys/class/block/nvme7c1n1' } @GLOBS), 'not the path devices');
+
+ %BLOCK = map { ("/dev/$_" => 1) } qw(nvme7n1 nvme7n1p1 nvme7n1p2);
+ $DIRS{'/proc'} = [qw(100 200 300 self)];
+ $DIRS{'/proc/100/fd'} = [0, 1, 7];
+ $DIRS{'/proc/200/fd'} = [3];
+ $DIRS{'/proc/300/fd'} = [4];
+ %LINKS = (
+ '/proc/100/fd/0' => '/dev/null',
+ '/proc/100/fd/7' => '/dev/nvme7n1',
+ '/proc/200/fd/3' => '/dev/nvme7n1p1',
+ '/proc/300/fd/4' => '/dev/nvme8n1',
+ );
+ $FILES{'/proc/100/comm'} = 'qemu-system-x86';
+ $FILES{'/proc/200/comm'} = 'mkfs.ext4';
+ $DIRS{'/sys/class/block/nvme7n1p2/holders'} = ['dm-0'];
+ is_deeply(
+ nv('_namespace_openers', $NQN),
+ [
+ '/dev/nvme7n1p2 held by dm-0',
+ 'mkfs.ext4 (PID 200, /dev/nvme7n1p1)',
+ 'qemu-system-x86 (PID 100, /dev/nvme7n1)',
+ ],
+ 'open namespaces and partitions and their holders are users, other devices are not',
+ );
+}
+
+# The local device behind a namespace link must be that namespace.
+{
+ my $link = "/dev/disk/by-id/nvme-uuid.$UUID";
+ my $check = sub($target, %files) {
+ local %LINKS = (defined($target) ? ($link => $target) : ());
+ local %FILES = (
+ '/sys/block/nvme3n1/uuid' => uc($UUID),
+ '/sys/block/nvme3n1/nsid' => '7',
+ '/sys/block/nvme3n1/device/subsysnqn' => $NQN,
+ %files,
+ );
+ my $ok = nv('_nvmet_local_namespace_ok', $link, $NQN, 7, $UUID);
+ return $ok ? 1 : 0;
+ };
+ my $device = '../../nvme3n1';
+ is($check->($device), 1, 'accepts the namespace of the subsystem, whatever the UUID case');
+ is($check->($device, '/sys/block/nvme3n1/nsid' => '8'), 0, 'refuses another NSID');
+ is(
+ $check->($device, '/sys/block/nvme3n1/device/subsysnqn' => $OTHER_NQN),
+ 0,
+ 'another subsystem',
+ );
+ is(
+ $check->($device, '/sys/block/nvme3n1/uuid' => '22345678-1234-1234-1234-123456789abc'),
+ 0,
+ 'another UUID',
+ );
+ is($check->($device, '/sys/block/nvme3n1/uuid' => undef), 0, 'an unreadable device');
+ is($check->(undef), 0, 'a missing link');
+
+ for my $target ('../../nvme3c1n1', '../../nvme3n1p1', '../../sda') {
+ is($check->($target), 0, "a link to $target");
+ }
+
+ my $dir = tempdir(CLEANUP => 1);
+ symlink('../../nvme3n1', "$dir/link") or die "symlink: $!\n";
+ is($nvme_mock->original('_link_target')->("$dir/link"), '../../nvme3n1', 'read, not followed');
+}
+
+# Storage activation. Everything on the local host is mocked.
+my $activate = sub($storeid, $scfg, $cache = undef) {
+ local $FILES{"/etc/pve/priv/storage/$storeid.nvme-dhchap"} = $KEY;
+ local %CONFIG = (%CONFIG, $storeid => $scfg);
+ @ACTIVATIONS = ();
+ $nvme_mock->redefine(
+ _nvmet_activate_target => sub(@args) { push @ACTIVATIONS, [@args]; return },
+ );
+ my $res = eval { $PLUGIN->activate_storage($storeid, $scfg, $cache) };
+ my $error = $@;
+ $nvme_mock->unmock('_nvmet_activate_target');
+ return ($res, $error);
+};
+my $forced = sub($storeid) { return { 'zfsnvme-force-reconcile' => { $storeid => 1 } } };
+
+subtest 'activation preconditions' => sub {
+ my $file = sub($path, $value) {
+ return sub() { $FILES{$path} = $value }
+ };
+ my $missing_iface = sub() {
+ $nvme_mock->redefine(_local_iface_exists => sub($iface) { return $iface ne 'ens20' });
+ };
+ my $other_storage = sub(%override) {
+ return sub() { $CONFIG{other} = scfg(subsysnqn => $OTHER_NQN, %override) };
+ };
+ my $fabrics_injection = 'traddr=198.51.100.9';
+ for my $case (
+ [
+ 'NVMe kernel modules that are not loaded',
+ $file->('/sys/module/nvme_core/parameters/multipath', undef),
+ qr/the NVMe kernel modules are not loaded; load the nvme-tcp kernel module/,
+ ],
+ [
+ 'multipath disabled',
+ $file->('/sys/module/nvme_core/parameters/multipath', 'N'),
+ qr/native NVMe multipath is disabled/,
+ ],
+ [
+ 'no host NQN',
+ $file->('/etc/nvme/hostnqn', undef),
+ qr/missing \/etc\/nvme\/hostnqn; the nvme-cli package generates it/,
+ ],
+ [
+ 'a host that is not allowed',
+ $file->(
+ '/etc/nvme/hostnqn',
+ 'nqn.2014-08.org.nvmexpress:uuid:00000000-0000-4000-8000-00000000dead',
+ ),
+ qr/local NVMe host NQN .* is missing from nvme-host-nqns/,
+ ],
+ [
+ 'no host ID',
+ $file->('/etc/nvme/hostid', undef),
+ qr/missing \/etc\/nvme\/hostid; the nvme-cli package generates it/,
+ ],
+ [
+ 'the zero host ID',
+ $file->('/etc/nvme/hostid', '00000000-0000-0000-0000-000000000000'),
+ qr/invalid NVMe host ID/,
+ ],
+ [
+ 'a host ID with a connect option',
+ $file->('/etc/nvme/hostid', "$UUID,$fabrics_injection"),
+ qr/invalid NVMe host ID/,
+ ],
+ [
+ 'a host ID with a newline',
+ $file->('/etc/nvme/hostid', "$UUID\n"),
+ qr/invalid NVMe host ID/,
+ ],
+ [
+ 'a connection parameter that is not a number',
+ sub() { },
+ qr/invalid NVMe connection parameter 'reconnect-delay'/,
+ 'nvme-reconnect-delay' => "2,$fabrics_injection",
+ ],
+ [
+ 'a missing local interface',
+ $missing_iface,
+ qr/interface 'ens20' does not exist on this node/,
+ ],
+ [
+ 'a pool shared with another storage',
+ $other_storage->('nvme-portals' => '192.0.2.31,192.0.2.32'),
+ qr/ZFS pool 'tank' on '192\.0\.2\.10' is already used by storage 'other'/,
+ ],
+ [
+ 'a listener shared with another storage',
+ $other_storage->(pool => 'data', 'nvme-portals' => '192.0.2.31,192.0.2.22'),
+ qr/portal '192\.0\.2\.22' port 4420 is already used by storage 'other'/,
+ ],
+ ) {
+ my ($name, $setup, $error, %override) = $case->@*;
+ reset_host();
+ local %CONFIG = ();
+ $nvme_mock->redefine(_local_iface_exists => sub($iface) { return 1 });
+ $setup->();
+ my (undef, $got) = $activate->('zfsnvme-unit-s', scfg(%override));
+ like($got, $error, "refuses $name");
+ is_deeply([@ACTIVATIONS, @CONNECTS, @SYSFS_WRITES], [], 'and does nothing');
+ }
+ $nvme_mock->redefine(_local_iface_exists => sub($iface) { return 1 });
+};
+
+subtest 'a stopped task ends the activation at the connect' => sub {
+ for my $case (
+ ["received interrupt\n", 1],
+ ["interrupted by unexpected signal\n", 1],
+ ["connect failed: Connection refused\n", 0],
+ ["NVMe/TCP connect did not complete within 10 seconds\n", 0],
+ ) {
+ my ($exception, $cancelled) = $case->@*;
+ reset_host();
+ my $calls = 0;
+ $nvme_mock->redefine(
+ _fabrics_connect => sub($options, $names) {
+ my %option = map { split(/=/, $_, 2) } split(/,/, $options);
+ ++$calls;
+ # cancellation must escape even if the write created a live path
+ add_controller("nvme$calls", $option{traddr}, 4420, $option{host_iface})
+ if $cancelled || $calls > 1;
+ die $exception if $calls == 1;
+ return $calls;
+ },
+ );
+ my ($res, $err) = $activate->('zfsnvme-cancel', scfg());
+ is($err, $cancelled ? "received interrupt\n" : '', 'only a stopped task escapes');
+ is($res, $cancelled ? undef : 1, 'an ordinary failure still permits a degraded path');
+ is($calls, $cancelled ? 1 : 2, 'a stopped task never attempts the next portal');
+ unlike($err . join('', @WARNINGS), qr/\Q$KEY\E/, 'errors and warnings hide the key');
+ }
+ $nvme_mock->redefine(_fabrics_connect => $connect_mock);
+
+ reset_host();
+ no warnings 'redefine';
+ local *PVE::Storage::ZFSNVMePlugin::_restrict_attr =
+ sub($path) { die "received interrupt\n" };
+ my ($res, $err) = $activate->('zfsnvme-cancel-scan', scfg());
+ is($err, "received interrupt\n",
+ 'cancellation in the post-connect permission scan escapes');
+ is(scalar(@CONNECTS), 1, 'no next portal after a cancelled permission scan');
+};
+
+# Deleting a controller runs in a bounded child: 5 seconds for a dead path
+# during an activation, which goes on, and 15 seconds for a deactivation,
+# which fails. Only a stopped task escapes an activation.
+subtest 'deleting a controller is bounded and keeps a stopped task' => sub {
+ my $interrupt = "received interrupt\n";
+ my $failed = "cannot delete NVMe controller 'nvme1'\n";
+ # [label, what the child does, activation error, deactivation error]
+ for my $case (
+ ['a stopped task', $interrupt, $interrupt, $interrupt],
+ ['a signalled child', "interrupted by unexpected signal\n", $interrupt, $interrupt],
+ ['a failed deletion', $failed, '', $failed],
+ [
+ 'a timeout',
+ undef,
+ '',
+ "disconnecting NVMe subsystem '$NQN' did not complete within 15 seconds\n",
+ ],
+ ) {
+ my ($label, $exception, $activation_error, $deactivation_error) = $case->@*;
+ $tools_mock->redefine(
+ run_fork_with_timeout => sub($seconds, $code, $opts = undef) {
+ push @FORKS, $seconds;
+ # without an exception, the deletion times out
+ return (undef, 1) if !defined($exception) && $seconds != 10;
+ return ($code->(), 0);
+ },
+ );
+ $nvme_mock->redefine(_delete_controllers => sub($controllers) { die $exception })
+ if defined($exception);
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19', 'dead');
+ my ($res, $err) = $activate->('zfsnvme-delete', scfg());
+ is($err, $activation_error, "$label while a dead path is deleted");
+ is($FORKS[0], 5, 'in a child bounded by 5 seconds');
+ is(scalar(@CONNECTS), $activation_error ? 0 : 2, 'only a failure goes on to connect');
+ my $warned = $exception
+ // "deleting NVMe controller 'nvme1' did not complete within 5 seconds\n";
+ like(join("\n", @WARNINGS), qr/\Q$warned\E/, 'and is warned about')
+ if !$activation_error;
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ eval { $PLUGIN->deactivate_storage('zfsnvme-delete', scfg()) };
+ is($@, $deactivation_error, "$label fails the deactivation");
+ is_deeply(\@FORKS, [15], 'in a child bounded by 15 seconds');
+ $nvme_mock->redefine(_delete_controllers => $delete_mock);
+ }
+ $tools_mock->redefine(run_fork_with_timeout => $fork_inline);
+};
+
+subtest 'storage deletion preserves task cancellation at each catch boundary' => sub {
+ for my $case (
+ ['deactivate', "received interrupt\n", 1, 0],
+ ['deactivate', "ordinary disconnect failure\n", 0, 1],
+ [
+ 'deactivate',
+ "disconnecting NVMe subsystem '$NQN' did not complete within 15 seconds\n",
+ 0,
+ 1,
+ ],
+ ['target', "received interrupt\n", 1, 1],
+ ['target', "ordinary target failure\n", 0, 1],
+ ) {
+ my ($phase, $exception, $cancelled, $expected_target_calls) = $case->@*;
+ reset_host();
+ my $target_calls = 0;
+ no warnings 'redefine';
+ local *PVE::Storage::ZFSNVMePlugin::deactivate_storage = sub(@args) {
+ die $exception if $phase eq 'deactivate';
+ return 1;
+ };
+ local *PVE::Storage::ZFSNVMePlugin::_nvmet_delete_target = sub(@args) {
+ ++$target_calls;
+ die $exception if $phase eq 'target';
+ return;
+ };
+ eval { $PLUGIN->on_delete_hook('zfsnvme-delete-cancel', scfg()) };
+ is(
+ $@,
+ $cancelled ? "received interrupt\n" : '',
+ "$phase preserves only task cancellation",
+ );
+ is($target_calls, $expected_target_calls, 'no target deletion after a cancellation');
+ is_deeply(
+ \@UNLINKED,
+ ['/etc/pve/priv/storage/zfsnvme-delete-cancel.nvme-dhchap'],
+ 'the key file is removed first',
+ );
+ chomp(my $message = $exception);
+ like(
+ join("\n", @WARNINGS),
+ qr/not disconnecting NVMe storage 'zfsnvme-delete-cancel': \Q$message\E/,
+ 'an ordinary deactivation failure only warns',
+ ) if $phase eq 'deactivate' && !$cancelled;
+ }
+};
+
+subtest 'slow-path backoff' => sub {
+ reset_host();
+ local %REFUSE = ('192.0.2.22' => 1);
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ my $scfg = scfg();
+ my ($res, $error) = $activate->('zfsnvme-unit-a', $scfg);
+ is($error, '', 'a degraded storage activates');
+ is(scalar(@ACTIVATIONS), 1, 'through the target');
+ is_deeply(
+ [$ACTIVATIONS[0]->@[0, 2 .. 4]],
+ [
+ 'zfsnvme-unit-a',
+ [
+ map { {
+ family => 'ipv4',
+ address => "192.0.2.2$_",
+ port => 4420,
+ host_iface => 'ens' . (18 + $_),
+ } } 1,
+ 2,
+ ],
+ [$hostnqn_a, $hostnqn_b],
+ $KEY,
+ ],
+ 'with the storage, its parsed portals, hosts and key',
+ );
+ like(
+ join("\n", @WARNINGS),
+ qr/storage 'zfsnvme-unit-a' is degraded: 1\/2 paths live/,
+ 'and warns',
+ );
+ ($res, $error) = $activate->('zfsnvme-unit-a', $scfg);
+ is($error, '', 'within a minute, a live path takes the fast path');
+ is(scalar(@ACTIVATIONS), 0, 'without a target call');
+ {
+ local $NOW = $NOW + 61;
+ ($res, $error) = $activate->('zfsnvme-unit-a', $scfg);
+ is(scalar(@ACTIVATIONS), 1, 'a minute later the target is tried again');
+ }
+ my $cache = $forced->('zfsnvme-unit-a');
+ ($res, $error) = $activate->('zfsnvme-unit-a', $scfg, $cache);
+ is(scalar(@ACTIVATIONS), 1, 'a forced activation always reaches the target');
+ ok(!$cache->{'zfsnvme-force-reconcile'}->{'zfsnvme-unit-a'}, 'and consumes the request');
+ $FILES{'/sys/class/nvme/nvme1/state'} = 'dead';
+ ($res, $error) = $activate->('zfsnvme-unit-a', $scfg);
+ is(
+ $error,
+ "no live NVMe/TCP path for storage 'zfsnvme-unit-a'\n",
+ 'within a minute, without a live path the activation fails at once',
+ );
+ is_deeply([@ACTIVATIONS, @DELETED], [], 'without a target call or a reconnect');
+ {
+ local $NOW = $NOW + 61;
+ ($res, $error) = $activate->('zfsnvme-unit-a', $scfg);
+ is(scalar(@ACTIVATIONS), 1, 'a minute later the target is tried');
+ is_deeply(\@DELETED, ['nvme1'], 'and the dead path is reconnected');
+ }
+
+ reset_host();
+ local %REFUSE = ('192.0.2.22' => 1);
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ local $NOW = 30; # the monotonic clock shortly after boot
+ ($res, $error) = $activate->('zfsnvme-unit-b', $scfg);
+ is(scalar(@ACTIVATIONS), 1, 'a storage that was never activated is not in its backoff');
+
+ # A storage without any path waits for one once per backoff period, not
+ # on every status update.
+ reset_host();
+ local %REFUSE = ('192.0.2.21' => 1, '192.0.2.22' => 1);
+ ($res, $error) = $activate->('zfsnvme-unit-d', $scfg);
+ is($error, "no live NVMe/TCP path for storage 'zfsnvme-unit-d'\n", 'no path, no storage');
+ is_deeply(
+ [scalar(@ACTIVATIONS), scalar(@CONNECTS)],
+ [1, 2],
+ 'after the target and each portal',
+ );
+ @CONNECTS = ();
+ ($res, $error) = $activate->('zfsnvme-unit-d', $scfg);
+ is(
+ $error,
+ "no live NVMe/TCP path for storage 'zfsnvme-unit-d'\n",
+ 'within a minute it fails',
+ );
+ is_deeply([@ACTIVATIONS, @CONNECTS], [], 'at once, without a target call or a connect');
+ ($res, $error) = $activate->('zfsnvme-unit-d', $scfg, $forced->('zfsnvme-unit-d'));
+ is_deeply(
+ [scalar(@ACTIVATIONS), scalar(@CONNECTS)],
+ [1, 2],
+ 'a forced activation tries again',
+ );
+ {
+ local $NOW = $NOW + 61;
+ @CONNECTS = ();
+ ($res, $error) = $activate->('zfsnvme-unit-d', $scfg);
+ is_deeply(
+ [scalar(@ACTIVATIONS), scalar(@CONNECTS)],
+ [1, 2],
+ 'so does one a minute later',
+ );
+ }
+
+ # A slow path whose target call fails starts the backoff as well.
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ local $FILES{'/etc/pve/priv/storage/zfsnvme-unit-t.nvme-dhchap'} = $KEY;
+ local %CONFIG = ('zfsnvme-unit-t' => $scfg);
+ my $calls = 0;
+ $nvme_mock->redefine(
+ _nvmet_activate_target => sub(@args) {
+ $calls++;
+ die "NVMe target '192.0.2.10' is unreachable: timeout\n";
+ },
+ );
+ eval { $PLUGIN->activate_storage('zfsnvme-unit-t', $scfg) };
+ like($@, qr/is unreachable/, 'an unreachable target fails the slow path');
+ eval { $PLUGIN->activate_storage('zfsnvme-unit-t', $scfg) };
+ is($@, '', 'within a minute the live path takes the fast path');
+ is($calls, 1, 'without another target call');
+ $nvme_mock->unmock('_nvmet_activate_target');
+};
+
+subtest 'connecting the paths' => sub {
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ $network_mock->redefine(
+ tcp_ping => sub($host, $port, $timeout = undef) { return $host ne '192.0.2.22' },
+ );
+ my (undef, $error) = $activate->('zfsnvme-unit-v', scfg(), $forced->('zfsnvme-unit-v'));
+ is_deeply(\@CONNECTS, [], 'an unreachable portal is not connected');
+ like(join("\n", @WARNINGS), qr/'192\.0\.2\.22:4420' is unreachable/, 'but warned about');
+ $network_mock->redefine(tcp_ping => sub($host, $port, $timeout = undef) { return 1 });
+
+ # After a target restart, the kernel reconnects its controllers itself.
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19', 'connecting');
+ local %REFUSE = ('192.0.2.22' => 1);
+ my $sleeps = 0;
+ $nvme_mock->redefine(
+ _sleep => sub($seconds) {
+ $NOW += $seconds;
+ $FILES{'/sys/class/nvme/nvme1/state'} = 'live' if ++$sleeps == 3;
+ return;
+ },
+ );
+ (undef, $error) = $activate->('zfsnvme-unit-w', scfg());
+ is($error, '', 'an activation waits for a controller that reconnects');
+ is($sleeps, 3, 'until it is live');
+ $nvme_mock->redefine(_sleep => sub($seconds) { $NOW += $seconds; return });
+};
+
+subtest 'moving a path to its configured interface' => sub {
+ my $single = scfg('nvme-portals' => '192.0.2.21', 'nvme-host-ifaces' => 'ens19');
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'eth9');
+ my ($res, $error) = $activate->('zfsnvme-unit-c', scfg());
+ is($error, '', 'a storage whose only live path uses another interface');
+ is_deeply(\@CONNECTS, ['192.0.2.21@ens19', '192.0.2.22@ens20'], 'connects both paths');
+ is_deeply(\@DELETED, ['nvme1'], 'and then disconnects the old one');
+ is_deeply(\@WARNINGS, [], 'the storage is healthy');
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'eth9');
+ local %CONNECT_STATE = ('192.0.2.21' => 'connecting');
+ my $start = $NOW;
+ ($res, $error) = $activate->('zfsnvme-unit-g', $single);
+ is($error, '', 'a new path that does not become live');
+ is_deeply(\@DELETED, [], 'keeps the old path');
+ like(
+ join("\n", @WARNINGS),
+ qr/'192\.0\.2\.21:4420' is not live on 'ens19' yet; keeping its path on another interface/,
+ 'and warns',
+ );
+ like(join("\n", @WARNINGS), qr/is degraded: 0\/1 paths live/, 'the storage is degraded');
+ is($NOW - $start, 10, 'after waiting 10 seconds for the new path');
+ ($res, $error) = $activate->('zfsnvme-unit-g', $single);
+ ok(!$error && !@ACTIVATIONS, 'within a minute, the old path takes the fast path');
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'eth9');
+ local %REFUSE = ('192.0.2.21' => 1);
+ ($res, $error) = $activate->('zfsnvme-unit-h', $single);
+ is($error, '', 'a new path that cannot connect');
+ is_deeply(\@DELETED, [], 'keeps the old path too');
+ like(
+ join("\n", @WARNINGS),
+ qr/192\.0\.2\.21:4420: connect failed: Connection refused/,
+ 'and warns',
+ );
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'eth9');
+ my $connect = $PLUGIN->can('_connect_portal');
+ {
+ no warnings 'redefine';
+ local %REFUSE = ();
+ local *PVE::Storage::ZFSNVMePlugin::_connect_portal = sub($scfg, $portal, @ids) {
+ $connect->($scfg, $portal, @ids);
+ die "connection failed after creation\n";
+ };
+ ($res, $error) = $activate->('zfsnvme-unit-h-partial', $single);
+ }
+ ok(!$error && $res == 1, 'a partially successful connect leaves the storage usable');
+ is($FILES{'/sys/class/nvme/nvme71/state'}, 'live', 'a live replacement was created');
+ is_deeply(\@DELETED, [], 'the connect error still preserves the old interface path');
+ like(
+ join("\n", @WARNINGS),
+ qr/192\.0\.2\.21:4420: connection failed after creation/,
+ 'and preserves the original connect warning',
+ );
+};
+
+subtest 'reconnect timeouts reach connected controllers' => sub {
+ my $sysfs = sub () {
+ return [
+ map { [$_->[0] =~ s{\A/sys/class/nvme/}{}r, $_->[1] =~ s/\n\z//r] }
+ grep { $_->[0] =~ m{\A/sys/class/nvme/} } @SYSFS_WRITES
+ ];
+ };
+ my $case = sub($storeid, $scfg, %current) {
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ for my $name (qw(nvme1 nvme2)) {
+ $FILES{"/sys/class/nvme/$name/$_"} = $current{$_} for keys %current;
+ }
+ my ($res, $error) = $activate->($storeid, $scfg, $forced->($storeid));
+ is($error, '', "activation of $storeid");
+ return $sysfs->();
+ };
+ my $timeouts = scfg(
+ 'nvme-reconnect-delay' => 5,
+ 'nvme-ctrl-loss-tmo' => 60,
+ 'nvme-fast-io-fail-tmo' => 10,
+ );
+ my $written = [
+ map { (
+ ["$_/reconnect_delay", 5], ["$_/ctrl_loss_tmo", 60], ["$_/fast_io_fail_tmo", 10],
+ ) } qw(nvme1 nvme2)
+ ];
+ is_deeply($case->('zfsnvme-unit-i', $timeouts), $written, 'changes reach every controller');
+ is_deeply(
+ [grep { $_->[0] =~ /iopolicy/ } @SYSFS_WRITES],
+ [['/sys/class/nvme-subsystem/nvme-subsys7/iopolicy', "round-robin\n"]],
+ 'with the multipath policy',
+ );
+ my %current = (reconnect_delay => '5', ctrl_loss_tmo => '60', fast_io_fail_tmo => '10');
+ is_deeply(
+ $case->('zfsnvme-unit-j', $timeouts, %current),
+ [],
+ 'unchanged ones are not written',
+ );
+ is_deeply(
+ $case->('zfsnvme-unit-k', scfg('nvme-ctrl-loss-tmo' => -1)),
+ [map { ["$_/ctrl_loss_tmo", -1] } qw(nvme1 nvme2)],
+ 'an infinite loss timeout',
+ );
+ is_deeply(
+ $case->('zfsnvme-unit-l', scfg('nvme-ctrl-loss-tmo' => 7), ctrl_loss_tmo => '8'),
+ [],
+ 'the loss timeout the kernel rounds up to the reconnect delay',
+ );
+ is_deeply(
+ $case->('zfsnvme-unit-m', scfg(), fast_io_fail_tmo => '5'),
+ [map { ["$_/fast_io_fail_tmo", -1] } qw(nvme1 nvme2)],
+ 'an unset fast I/O fail timeout is turned off',
+ );
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ my ($res, $error) = $activate->('zfsnvme-unit-n', $timeouts);
+ ok(!$error && $res == 1, 'a healthy storage activates on the fast path');
+ is(scalar(@ACTIVATIONS) + scalar(@CONNECTS), 0, 'without a target call or connect');
+ is_deeply($sysfs->(), $written, 'which also writes changed timeouts');
+ is_deeply(
+ [grep { $_->[0] =~ /iopolicy/ } @SYSFS_WRITES],
+ [['/sys/class/nvme-subsystem/nvme-subsys7/iopolicy', "round-robin\n"]],
+ 'and the multipath policy',
+ );
+ delete $FILES{'/etc/pve/priv/storage/zfsnvme-unit-n.nvme-dhchap'};
+ $res = eval {
+ local %CONFIG = ('zfsnvme-unit-n' => $timeouts);
+ $PLUGIN->activate_storage('zfsnvme-unit-n', $timeouts);
+ };
+ is($@, "missing NVMe DH-HMAC-CHAP key\n", 'the fast path needs the key file');
+
+ # file_write warns about a failed write, and is silent about a failed open,
+ # leaving its errno in $!
+ my @perl_warnings;
+ local $SIG{__WARN__} = sub($warning) { push @perl_warnings, $warning };
+ my $failing = sub($pattern, $errno, $warn) {
+ $sysfs_mock->redefine(
+ file_write => sub($path, $data, @rest) {
+ return $sysfs_write->($path, $data) if $path !~ $pattern;
+ $! = $errno;
+ return undef if !$warn;
+ warn "error writing '$data' to '$path': $!\n";
+ $! = ENOENT; # not the reason: file_write closed the file since
+ return 0;
+ },
+ );
+ };
+ my $reason = sub($errno) { local $! = $errno; return "$!" };
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $failing->(qr{/nvme1/ctrl_loss_tmo\z}, EINVAL, 1);
+ ($res, $error) = $activate->('zfsnvme-unit-e', $timeouts);
+ is($error, '', 'a timeout that cannot be written does not fail the activation');
+ my $einval = $reason->(EINVAL);
+ is_deeply(
+ [@WARNINGS, @perl_warnings],
+ ["cannot set ctrl_loss_tmo of NVMe controller 'nvme1': $einval"],
+ 'and is reported once, as a task warning with the reason',
+ );
+ reset_host();
+ @perl_warnings = ();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $failing->(qr{/nvme2/ctrl_loss_tmo\z}, EACCES, 0);
+ ($res, $error) = $activate->('zfsnvme-unit-tmo-open', $timeouts);
+ is($error, '', 'a timeout that cannot be opened does not fail the activation');
+ is_deeply(
+ [@WARNINGS, @perl_warnings],
+ ["cannot set ctrl_loss_tmo of NVMe controller 'nvme2': " . $reason->(EACCES)],
+ 'and is reported once, with the reason',
+ );
+ reset_host();
+ @perl_warnings = ();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $failing->(qr{/iopolicy\z}, EACCES, 0);
+ ($res, $error) = $activate->('zfsnvme-unit-f', $timeouts);
+ is(
+ $error,
+ 'unable to set NVMe multipath policy: ' . $reason->(EACCES) . "\n",
+ 'a policy that cannot be set fails the activation',
+ );
+ is_deeply([@WARNINGS, @perl_warnings], [], 'with the reason in the error only');
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $failing->(qr{/iopolicy\z}, EINVAL, 1);
+ ($res, $error) = $activate->('zfsnvme-unit-policy-write', $timeouts);
+ is(
+ $error,
+ "unable to set NVMe multipath policy: $einval\n",
+ 'a policy that cannot be written fails the activation',
+ );
+ is_deeply([@WARNINGS, @perl_warnings], [],
+ 'and the warning of file_write is not passed on');
+
+ # The stock file_write on an attribute that does not exist: unlike a
+ # deletion or a rescan, setting a timeout or the policy treats the failed
+ # open (ENOENT) as a failure like any other.
+ my $file_write = $sysfs_mock->original('file_write');
+ my $missing = tempdir(CLEANUP => 1) . '/a/b';
+ my $absent = sub($pattern) {
+ $sysfs_mock->redefine(
+ file_write => sub($path, $data, @rest) {
+ return $sysfs_write->($path, $data) if $path !~ $pattern;
+ return $file_write->($missing, $data, @rest);
+ },
+ );
+ };
+ my $enoent = $reason->(ENOENT);
+ reset_host();
+ @perl_warnings = ();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $absent->(qr{/nvme1/reconnect_delay\z});
+ ($res, $error) = $activate->('zfsnvme-unit-tmo-missing', $timeouts);
+ is($error, '', 'a missing timeout attribute does not fail the activation');
+ is_deeply(
+ [@WARNINGS, @perl_warnings],
+ ["cannot set reconnect_delay of NVMe controller 'nvme1': $enoent"],
+ 'but is reported once, as a task warning with the reason',
+ );
+ reset_host();
+ @perl_warnings = ();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ $absent->(qr{/iopolicy\z});
+ ($res, $error) = $activate->('zfsnvme-unit-policy-missing', $timeouts);
+ is(
+ $error,
+ "unable to set NVMe multipath policy: $enoent\n",
+ 'a missing policy attribute fails the activation',
+ );
+ is_deeply([@WARNINGS, @perl_warnings], [], 'with the reason in the error only');
+ $sysfs_mock->redefine(file_write => $sysfs_write);
+};
+
+subtest 'the key attributes of the controllers are private' => sub {
+ my $attrs = sub(@names) {
+ my @attrs = qw(dhchap_secret dhchap_ctrl_secret);
+ return [
+ map {
+ my $name = $_;
+ map { "/sys/class/nvme/$name/$_" } @attrs
+ } @names
+ ];
+ };
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ add_controller('nvme2', '192.0.2.22', 4420, 'ens20');
+ add_controller('nvme3', '192.0.2.23', 4420, 'ens21', 'live', $OTHER_NQN);
+ my ($res, $error) = $activate->('zfsnvme-unit-o', scfg());
+ is($error, '', 'a healthy storage');
+ is_deeply(
+ \@RESTRICTED,
+ $attrs->('nvme1', 'nvme2'),
+ 'restricts its controllers on the fast path',
+ );
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19');
+ ($res, $error) = $activate->('zfsnvme-unit-p', scfg(), $forced->('zfsnvme-unit-p'));
+ is($error, '', 'a forced activation');
+ is_deeply(
+ \@RESTRICTED,
+ $attrs->('nvme1', 'nvme1', 'nvme71'),
+ 'restricts those of the existing controllers first, then those of new ones',
+ );
+
+ reset_host();
+ local %CONNECT_STATE = map { ($_ => 'connecting') } '192.0.2.21', '192.0.2.22';
+ ($res, $error) = $activate->('zfsnvme-unit-q', scfg());
+ is(
+ $error,
+ "no live NVMe/TCP path for storage 'zfsnvme-unit-q'\n",
+ 'an activation that fails',
+ );
+ is_deeply(
+ \@RESTRICTED,
+ $attrs->('nvme71', 'nvme71', 'nvme72'),
+ 'each one after its connect',
+ );
+
+ reset_host();
+ add_controller('nvme1', '192.0.2.21', 4420, 'ens19', 'connecting');
+ $nvme_mock->redefine(_nvmet_activate_target => sub(@args) { die "unreachable\n" });
+ $res = eval {
+ local $FILES{'/etc/pve/priv/storage/zfsnvme-unit-x.nvme-dhchap'} = $KEY;
+ local %CONFIG = ('zfsnvme-unit-x' => scfg());
+ $PLUGIN->activate_storage('zfsnvme-unit-x', scfg());
+ };
+ is($@, "unreachable\n", 'a slow path that fails at the target');
+ is_deeply(\@RESTRICTED, $attrs->('nvme1'), 'still restricts the existing controllers');
+ $nvme_mock->unmock('_nvmet_activate_target');
+
+ # The first connect fails after it created a live controller; the scan of
+ # the permissions after it fails too in the second case.
+ my $connect = $PLUGIN->can('_connect_portal');
+ my $connect_error = "connection failed after creation\n";
+ for my $scan_error (0, 1) {
+ reset_host();
+ my $scan_failed = 0;
+ {
+ no warnings 'redefine';
+ local *PVE::Storage::ZFSNVMePlugin::_connect_portal = sub($scfg, $portal, @ids) {
+ is_deeply(
+ \@RESTRICTED,
+ $attrs->('nvme71'),
+ 'a failed connect restricts both attributes before the next connect',
+ ) if @CONNECTS == 1 && !$scan_error;
+ $connect->($scfg, $portal, @ids);
+ die $connect_error if @CONNECTS == 1;
+ };
+ local *PVE::Storage::ZFSNVMePlugin::_restrict_attr = sub($path) {
+ push @RESTRICTED, $path;
+ eval { 1 }; # A permission helper must not erase the saved connect error.
+ die "permission scan failed\n" if $scan_error && !$scan_failed++;
+ return;
+ };
+ ($res, $error) = $activate->("zfsnvme-unit-secret-failure-$scan_error", scfg());
+ }
+ is($error, '', 'a connect error does not discard its live controller');
+ ok(
+ scalar(grep { $_ eq "192.0.2.21:4420: $connect_error" } @WARNINGS),
+ 'and warns about it',
+ );
+ if ($scan_error) {
+ ok(
+ scalar(
+ grep { $_ eq "cannot restrict NVMe controller attributes for '$NQN'" }
+ @WARNINGS
+ ),
+ 'the scan failure has its own warning',
+ );
+ unlike(join("\n", @WARNINGS), qr/permission scan failed/, 'without its exception');
+ } else {
+ is_deeply(
+ \@RESTRICTED,
+ $attrs->('nvme71', 'nvme71', 'nvme72'),
+ 'each attempt restricts the controllers, including a failed last connect',
+ );
+ }
+ }
+
+ my $dir = tempdir(CLEANUP => 1);
+ my $restrict = $nvme_mock->original('_restrict_attr');
+ for my $mode (0644, 0640, 0600, 0400) {
+ my $file = sprintf('%s/attr-%o', $dir, $mode);
+ open(my $fh, '>', $file) or die "open: $!\n";
+ close($fh);
+ chmod($mode, $file);
+ $restrict->($file);
+ is(
+ (stat($file))[2] & 07777,
+ $mode & 077 ? 0600 : $mode,
+ sprintf('an attribute with mode %o', $mode),
+ );
+ }
+ @WARNINGS = ();
+ $restrict->("$dir/missing");
+ is_deeply(\@WARNINGS, [], 'a missing attribute is skipped');
+ my $not_directory = "$dir/attr-600/child";
+ $restrict->($not_directory);
+ like(join("\n", @WARNINGS), qr/\Q$not_directory\E/, 'another stat failure warns');
+};
+
+subtest 'controllers match IPv6 portals in any spelling' => sub {
+ reset_host();
+ add_controller('nvme1', '2001:db8:0:0::11', 4420, 'ens19');
+ add_controller('nvme2', '2001:DB8::12', 4420, 'ens20');
+ my $scfg = scfg('nvme-portals' => '[2001:db8::11],[2001:db8::0:12]');
+ my ($res, $error) = $activate->('zfsnvme-unit-r', $scfg);
+ is($error, '', 'both paths are live');
+ is(scalar(@ACTIVATIONS) + scalar(@CONNECTS), 0, 'without a target call or connect');
+};
+
+is_deeply(\@COMMANDS, [], 'the host side ran no command');
+
+done_testing();
+
+1;
next prev parent reply other threads:[~2026-10-06 8:54 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-05 0:26 [PATCH storage v3 0/4] add ZFS over NVMe/TCP storage plugin Joaquin Varela
2026-10-05 0:26 ` [PATCH storage v3 1/4] zfsnvme: " Joaquin Varela
2026-10-05 0:26 ` Joaquin Varela [this message]
2026-10-05 0:26 ` [PATCH storage v3 3/4] zfsnvme: fence target commands of abandoned transactions Joaquin Varela
2026-10-05 0:26 ` [PATCH storage v3 4/4] zfsnvme: wait up to 30 seconds for the shared storage lock Joaquin Varela
2026-10-05 0:26 ` [PATCH docs v3] storage: document ZFS over NVMe/TCP Joaquin Varela
2026-10-05 0:26 ` [PATCH manager v3] ui: storage: add ZFS over NVMe/TCP editor Joaquin Varela
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261005002609.571-3-joaquinvarela@neatech.ar \
--to=joaquinvarela@neatech.ar \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox