[PATCH docs v2 0/5] document ZFS over NVMe/TCP storage

[PATCH docs v2 0/5] document ZFS over NVMe/TCP storage

[Joaquin Varela]

Hi,

this is the documentation companion series for the `zfsnvme` backend. It
covers target and initiator setup, native multipath, DH-HMAC-CHAP, lifecycle
semantics, failure policy and the storage schema.

The follow-up patches document target reboot prerequisites, the complete Host
NQN ACL list required for atomic reconstruction, and the cloud-init datasource
hazard on a dedicated target that can see guest `cidata` zvols.

The documentation packages build successfully. The exact branch is:

https://github.com/joaquinv98/pve-docs/tree/submission/zfs-nvme-tcp-rfc-v2

Joaquin Varela (5):
  storage: document ZFS over NVMe/TCP backend
  storage: harden ZFS over NVMe/TCP guidance
  storage: document NVMe all-path loss policy
  zfsnvme: document target reboot prerequisites
  zfsnvme: document host ACL and cloud-init prerequisites

 pve-storage-zfsnvme.adoc | 200 +++++++++++++++++++++++++++++++++++++++
 pvesm.adoc               |   4 +
 2 files changed, 204 insertions(+)
 create mode 100644 pve-storage-zfsnvme.adoc

-- 
2.54.0.windows.1

[PATCH docs v2 1/5] storage: document ZFS over NVMe/TCP backend

[Joaquin Varela]

Document the zfsnvme backend, target and initiator setup, native
NVMe multipath, DH-HMAC-CHAP, lifecycle behavior and schema.

Include the storage type in the shared-storage overview. Explain
the stable namespace UUID path used by guests.

Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
 pve-storage-zfsnvme.adoc | 141 +++++++++++++++++++++++++++++++++++++++
 pvesm.adoc               |   4 ++
 2 files changed, 145 insertions(+)
 create mode 100644 pve-storage-zfsnvme.adoc

diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc
new file mode 100644
index 0000000..46d16ad
--- /dev/null
+++ b/pve-storage-zfsnvme.adoc
@@ -0,0 +1,141 @@
+[[storage_zfsnvme]]
+ZFS over NVMe/TCP Backend
+-------------------------
+ifdef::wiki[]
+:pve-toplevel:
+:title: Storage: ZFS over NVMe/TCP
+endif::wiki[]
+
+Storage pool type: `zfsnvme`
+
+This backend accesses a remote Linux machine with a ZFS pool and the kernel
+NVMe target through `ssh`. For each guest disk it creates a ZVOL, exports it as
+an NVMe namespace, and connects the {pve} nodes through native Linux NVMe/TCP
+multipath.
+
+The backend supports thin provisioning, snapshots, rollback, templates, linked
+clones, offline resize, and shared-storage live migration. Namespace UUIDs are
+stored as ZFS user properties, and the stable `nvme-uuid` device link is used
+for guest disks.
+
+Configuration
+~~~~~~~~~~~~~
+
+The target needs OpenZFS, configfs, and the `nvmet` and `nvmet-tcp` kernel
+modules. Configure root SSH access like for the ZFS over iSCSI backend. The key
+for the server address is stored at
+`/etc/pve/priv/zfs/<server>_id_rsa`.
+
+Each {pve} node needs `nvme-cli`, the `nvme-tcp` module, native NVMe multipath,
+a unique `/etc/nvme/hostnqn`, and `/etc/nvme/hostid`. Interface names listed in
+the storage configuration must exist on every node where the storage is
+enabled.
+
+The following properties are specific to ZFS over NVMe/TCP:
+
+server::
+
+IP address or DNS name used for the SSH control connection.
+
+pool::
+
+ZFS pool or child dataset used exclusively by this storage definition.
+
+subsysnqn::
+
+NVMe qualified name of the target subsystem. It must be unique across the
+cluster's storage definitions.
+
+nvme-portals::
+
+Comma-separated NVMe/TCP target addresses. The default service is `4420`.
+Specify IPv6 addresses in brackets, for example `[2001:db8::10]:4420`.
+
+nvme-host-ifaces::
+
+Comma-separated local interfaces, matched to `nvme-portals` by position. An
+explicit interface prevents a failed data path from reconnecting over the
+management network.
+
+dhchap-key::
+
+NVMe DH-HMAC-CHAP secret in `DHHC-1` representation. The value is handled as a
+sensitive property and stored below `/etc/pve/priv/storage/` with mode `0600`.
+In-place key rotation is not supported; create a new storage and subsystem for
+a coordinated rotation.
+
+nvme-iopolicy::
+
+Native multipath policy: `round-robin`, `queue-depth`, or `numa`.
+
+nvme-keep-alive-tmo::
+
+Keep-alive timeout in seconds.
+
+nvme-reconnect-delay::
+
+Delay between controller reconnect attempts in seconds.
+
+nvme-ctrl-loss-tmo::
+
+How long a controller remains reconnectable after path loss. `-1` retries
+indefinitely.
+
+nvme-nr-io-queues::
+
+Optional number of I/O queues per controller.
+
+blocksize::
+
+ZFS volume block size.
+
+sparse::
+
+Use ZFS thin provisioning instead of reserving the full virtual size.
+
+.Configuration Example (`/etc/pve/storage.cfg`)
+----
+zfsnvme: nvme-shared
+        server 192.0.2.10
+        pool tank/pve-nvme
+        subsysnqn nqn.2026-07.example:pve-nvme
+        nvme-portals 10.10.1.10:4420,10.10.2.10:4420
+        nvme-host-ifaces ens20,ens21
+        blocksize 16k
+        sparse 1
+        nvme-iopolicy round-robin
+        nvme-keep-alive-tmo 5
+        nvme-reconnect-delay 2
+        nvme-ctrl-loss-tmo 600
+        content images
+        shared 1
+----
+
+The DH-HMAC-CHAP key is intentionally not shown in `storage.cfg`. Set it with
+the storage creation API or web interface.
+
+Security and Availability
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The target uses an ACL for the unique Host NQN of every node and never enables
+`allow_any_host`. DH-HMAC-CHAP authenticates the endpoints, but it does not
+encrypt data. This backend does not configure NVMe/TCP TLS, so use isolated
+storage networks or equivalent protection.
+
+Shared access relies on {pve} cluster locking and fencing. Test quorum and
+fencing before placing production guests on the storage. Use independent
+failure domains for the configured paths and make sure the ZFS target itself
+is not a single point of failure.
+
+Online resize of a running guest disk is not supported. Stop the VM before
+resizing; the new size is detected when the block device is reopened.
+
+Storage Features
+~~~~~~~~~~~~~~~~
+
+.Storage features for backend `zfsnvme`
+[width="100%",cols="m,m,3*d",options="header"]
+|==============================================================================
+|Content types  |Image formats  |Shared |Snapshots |Clones
+|images         |raw            |yes    |yes       |yes
+|==============================================================================
diff --git a/pvesm.adoc b/pvesm.adoc
index 5bd24b2..d503f6f 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -439,6 +439,8 @@ See Also
 
 * link:/wiki/Storage:_ZFS_over_ISCSI[Storage: ZFS over ISCSI]
 
+* link:/wiki/Storage:_ZFS_over_NVMe_TCP[Storage: ZFS over NVMe/TCP]
+
 endif::wiki[]
 
 ifndef::wiki[]
@@ -471,6 +473,8 @@ include::pve-storage-btrfs.adoc[]
 
 include::pve-storage-zfs.adoc[]
 
+include::pve-storage-zfsnvme.adoc[]
+
 
 ifdef::manvolnum[]
 include::pve-copyright.adoc[]
-- 
2.54.0.windows.1

[PATCH docs v2 2/5] storage: harden ZFS over NVMe/TCP guidance

[Joaquin Varela]

Clarify interface consistency, control-plane availability, safe
storage removal, offline resize and the DH-HMAC-CHAP security
boundary.

Require qualification of fencing and independent failure domains.
Do not present nested-lab validation as hardware sizing.

Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
 pve-storage-zfsnvme.adoc | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc
index 46d16ad..c9ae180 100644
--- a/pve-storage-zfsnvme.adoc
+++ b/pve-storage-zfsnvme.adoc
@@ -35,7 +35,10 @@ The following properties are specific to ZFS over NVMe/TCP:
 
 server::
 
-IP address or DNS name used for the SSH control connection.
+IP address or DNS name used for the SSH control connection. Connected guest I/O
+uses the independent NVMe/TCP data paths, but capacity reporting and lifecycle
+operations require this endpoint. Use a redundant management DNS name or VIP
+where the storage appliance provides one.
 
 pool::
 
@@ -55,7 +58,8 @@ nvme-host-ifaces::
 
 Comma-separated local interfaces, matched to `nvme-portals` by position. An
 explicit interface prevents a failed data path from reconnecting over the
-management network.
+management network. Activation fails before changing target state if any
+configured interface is missing on the local node.
 
 dhchap-key::
 
@@ -130,6 +134,10 @@ is not a single point of failure.
 Online resize of a running guest disk is not supported. Stop the VM before
 resizing; the new size is detected when the block device is reopened.
 
+Removing the storage disconnects the complete subsystem. The backend requires
+its owned ZFS dataset to be empty and also refuses the operation while a local
+process or kernel block holder is using one of its namespaces.
+
 Storage Features
 ~~~~~~~~~~~~~~~~
 
-- 
2.54.0.windows.1

[PATCH docs v2 3/5] storage: document NVMe all-path loss policy

[Joaquin Varela]

Describe queue-until-controller-loss and optional fast-I/O-fail
behavior. Make the availability-versus-prompt-error trade-off
explicit and show an example without prescribing one policy for
every workload.

Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
 pve-storage-zfsnvme.adoc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc
index c9ae180..35f09f8 100644
--- a/pve-storage-zfsnvme.adoc
+++ b/pve-storage-zfsnvme.adoc
@@ -85,6 +85,13 @@ nvme-ctrl-loss-tmo::
 How long a controller remains reconnectable after path loss. `-1` retries
 indefinitely.
 
+nvme-fast-io-fail-tmo::
+
+Optional time in seconds before outstanding I/O fails while a controller is
+reconnecting. If this property is absent, the kernel queues I/O until
+`nvme-ctrl-loss-tmo` expires. A configured value must not exceed a finite
+controller-loss timeout. A value of `0` selects immediate fail-fast behavior.
+
 nvme-nr-io-queues::
 
 Optional number of I/O queues per controller.
@@ -111,6 +118,7 @@ zfsnvme: nvme-shared
         nvme-keep-alive-tmo 5
         nvme-reconnect-delay 2
         nvme-ctrl-loss-tmo 600
+        nvme-fast-io-fail-tmo 30
         content images
         shared 1
 ----
@@ -131,6 +139,13 @@ fencing before placing production guests on the storage. Use independent
 failure domains for the configured paths and make sure the ZFS target itself
 is not a single point of failure.
 
+Choose the all-path outage policy for the workload. The default configuration
+leaves `nvme-fast-io-fail-tmo` unset: transient outages can recover without
+guest block errors, but an I/O request can remain queued for as long as
+`nvme-ctrl-loss-tmo` (600 seconds in the example). Set a shorter fast-I/O-fail
+timeout when the application prefers a prompt block error over a long stall.
+This is a service-level decision, not a universally safer default.
+
 Online resize of a running guest disk is not supported. Stop the VM before
 resizing; the new size is detected when the block device is reopened.
 
-- 
2.54.0.windows.1

[PATCH docs v2 4/5] zfsnvme: document target reboot prerequisites

[Joaquin Varela]

Document module loading and configfs availability on the Linux nvmet
target. Clarify that target configuration is derived state reconstructed
from ZFS properties during activation.

Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
 pve-storage-zfsnvme.adoc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc
index 35f09f8..edfb938 100644
--- a/pve-storage-zfsnvme.adoc
+++ b/pve-storage-zfsnvme.adoc
@@ -26,6 +26,21 @@ modules. Configure root SSH access like for the ZFS over iSCSI backend. The key
 for the server address is stored at
 `/etc/pve/priv/zfs/<server>_id_rsa`.
 
+The NVMe target modules and configfs must be available after a target reboot,
+before a {pve} node attempts to reconcile the exported namespaces. On a Linux
+target using systemd, load the transport at boot and verify the configfs mount:
+
+----
+# echo nvmet_tcp >/etc/modules-load.d/pve-nvmet.conf
+# modprobe nvmet_tcp
+# mountpoint /sys/kernel/config
+----
+
+The target configuration below configfs is derived state. It does not need a
+separate persistence service: after the module and ZFS pool are available, the
+backend reconstructs subsystems, namespaces, UUIDs, ACLs, and ports from the
+ZFS user properties during activation.
+
 Each {pve} node needs `nvme-cli`, the `nvme-tcp` module, native NVMe multipath,
 a unique `/etc/nvme/hostnqn`, and `/etc/nvme/hostid`. Interface names listed in
 the storage configuration must exist on every node where the storage is
-- 
2.54.0.windows.1

[PATCH docs v2 5/5] zfsnvme: document host ACL and cloud-init prerequisites

[Joaquin Varela]

Document the complete Host NQN allow-list required for atomic target
reconstruction. Warn dedicated targets to disable guest cloud-init
datasource discovery so a cidata ZVOL cannot be mistaken for host
provisioning media.

Signed-off-by: Joaquin Varela <joaquinvarela@neatech.ar>
---
 pve-storage-zfsnvme.adoc | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc
index edfb938..9d2c2f0 100644
--- a/pve-storage-zfsnvme.adoc
+++ b/pve-storage-zfsnvme.adoc
@@ -38,8 +38,20 @@ target using systemd, load the transport at boot and verify the configfs mount:
 
 The target configuration below configfs is derived state. It does not need a
 separate persistence service: after the module and ZFS pool are available, the
-backend reconstructs subsystems, namespaces, UUIDs, ACLs, and ports from the
-ZFS user properties during activation.
+backend reconstructs namespaces and UUIDs from ZFS user properties, and
+reconstructs the subsystem, host ACLs, and ports from the shared storage
+configuration during activation.
+
+Do not leave cloud-init device discovery enabled on a dedicated Linux target.
+A guest cloud-init ZVOL contains a `cidata` filesystem and can otherwise be
+mistaken for the target host's own NoCloud datasource during boot. After the
+target host is provisioned, remove cloud-init or disable it according to the
+distribution's documentation. For distributions supporting the standard
+disable marker, use:
+
+----
+# touch /etc/cloud/cloud-init.disabled
+----
 
 Each {pve} node needs `nvme-cli`, the `nvme-tcp` module, native NVMe multipath,
 a unique `/etc/nvme/hostnqn`, and `/etc/nvme/hostid`. Interface names listed in
@@ -76,6 +88,14 @@ explicit interface prevents a failed data path from reconnecting over the
 management network. Activation fails before changing target state if any
 configured interface is missing on the local node.
 
+nvme-host-nqns::
+
+Comma-separated contents of `/etc/nvme/hostnqn` from every cluster node that
+may activate the storage. The complete allow-list lets any one node restore all
+host ACLs before publishing the subsystem after a target reboot. Activation
+fails if the local node's Host NQN is absent. Update this property before
+enabling the storage on a newly added cluster node.
+
 dhchap-key::
 
 NVMe DH-HMAC-CHAP secret in `DHHC-1` representation. The value is handled as a
@@ -127,6 +147,7 @@ zfsnvme: nvme-shared
         subsysnqn nqn.2026-07.example:pve-nvme
         nvme-portals 10.10.1.10:4420,10.10.2.10:4420
         nvme-host-ifaces ens20,ens21
+        nvme-host-nqns nqn.2014-08.org.nvmexpress:uuid:11111111-1111-1111-1111-111111111111,nqn.2014-08.org.nvmexpress:uuid:22222222-2222-2222-2222-222222222222
         blocksize 16k
         sparse 1
         nvme-iopolicy round-robin
-- 
2.54.0.windows.1