From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id DAAB31FF0EF for ; Sun, 02 Aug 2026 05:36:55 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DE5A121569; Sun, 02 Aug 2026 05:36:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=neatech-ar.20251104.gappssmtp.com; s=20251104; t=1785641795; x=1786246595; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=RGwE6YQgga7xaSp4PMGWbpfKkAhDOS+W44gDVb0fJGU=; b=IpoQaLDbpwhyvXaE0Lb0l0y0EB8xoMk5e2AZtqLG3HjRAKU0c0VmRirNw3KHUo6JAm Wzq6sysF0y7OuzmF8zFkWLS41DRqHb/4roOcv51dpL4XhKIchhPQCEfyscx9QtT41NV8 2E3cgeCIMNtYwRP1L94W3v2PXe94+wh10J8z0tOb4p2WN2UdmPqGHakzL++qbDe32GJl CFsD4uJ8Hr93oEnoykDonFtKk7MLA5y0gZk1MsJaFHmyKwiZMA4k99HDG8bDXayM2gJe ixpWp+ROR6OiwWqINJgd1zO6Ljt2I5xQnKn2ZFAl14OgDnHJH0pRuNVndyWkAmKGwJ+8 FMBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785641795; x=1786246595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=RGwE6YQgga7xaSp4PMGWbpfKkAhDOS+W44gDVb0fJGU=; b=OH2SpZ3nHeoeuScE/SmRyakwyQysIRRhgSfQJAfwL6LFgaboPzDPzu3qIg5NIpK86Q uPDs5KiqiFmEzMu4X3sMzdk2P0YVSXfI/fZGBXnq5geFeTKUt86mzCAkGfLHkmJXLOnt oNaBhi1s5BDUnp7MIgPVzG3CMSTqCvrHQd5GAr3ZqUCPbNUA9qHsafOGfvDk24D9H0qP TzpKoYIDETOPocLKy32OpAVYMtQW+suJIkyhJg/4kALvYxcgj1HtPQ2j4lLireteExpt lRx8mOXLxXiZMaqAxsQFmTCwztU8kdlCxIxZcHLcOcHySN3DVmKG+MSnU/AZe3+fPnPv 6ZBw== X-Gm-Message-State: AOJu0YyC5o3DUd7843TwyWb9beAGZk6Xwfs15MbJA5sROyRYWJuz11yR jBwDY0S1f1lDVw00Pqb+PyvWkXNPGZ7fFmMwnWbufKgZ41rLBrRHcr74aaL8zl4hEejK/ijNF1g LFGYLnb4= X-Gm-Gg: AR+sD12x6X+2Q7Zjn7N+dCkg3JSZmJdixRs8rMgDGNuk56mm0AcvxkJv8EPk8zyjwqc 4seD5CrgHAbKX9fXkJkkjV/9V8RnRm10KXH91LX8C4WcrdrdjX1mdTFiU3PIlqRdYCsGPUJzgcZ DrseccPsI4nX+D4Oh4xGanRPMhJVTcqnDh7I7oiwV3BaXadRRLYgYIZcK7AGdCDrVNaujHgeN6d CEiRW+jvJNUOMq3qJfH/DStJnH3o+DC7bVTJXIX3c4Sfweot99qH8M5nrhH+BVodUjwLBkTTYu+ /GcUOaByckN6ejOgbxVjvuAhorkAfJdAoCtEzWYSxSchMHDfWQuFcd7pr+AOj4RPG/uVXDTRtkX oyTjpjVLJVupFgFQnO4uOtROlMRXAl3AekY64VTS7yLQ2Le+RsJ8R7pvS/6GccjYKvTGOCPi4Lj d4UnteJl4LvbG+AdRehHq4iwGX/jx0v30xC3NaC5nDPorDLm5b5gVYNjRIXHOHTd5/Hn5+ATel5 0Fy8ehyXJwVNjIjTzRC+OguoANIKhDrWMUASqk0Go6a X-Received: by 2002:a17:90b:4b90:b0:381:28e0:6248 with SMTP id 98e67ed59e1d1-38fbc3f0737mr7015592a91.1.1785641795393; Sat, 01 Aug 2026 20:36:35 -0700 (PDT) From: Joaquin Varela To: pve-devel@lists.proxmox.com Subject: [PATCH docs v2 1/5] storage: document ZFS over NVMe/TCP backend Date: Sun, 2 Aug 2026 00:36:26 -0300 Message-ID: <294c23abc82179ff7fb9759f4fe4bcbfe79d557d.1785636981.git.joaquinvarela@neatech.ar> X-Mailer: git-send-email 2.54.0.windows.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.400 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_PASS -0.1 DMARC pass policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TX6NVHDBCD7TAZ7OBJYQJNRGK5DBENQ7 X-Message-ID-Hash: TX6NVHDBCD7TAZ7OBJYQJNRGK5DBENQ7 X-MailFrom: joaquinvarela@neatech.ar X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Joaquin Varela X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Document the zfsnvme backend, target and initiator setup, native NVMe multipath, DH-HMAC-CHAP, lifecycle behavior and schema. Include the storage type in the shared-storage overview. Explain the stable namespace UUID path used by guests. Signed-off-by: Joaquin Varela --- pve-storage-zfsnvme.adoc | 141 +++++++++++++++++++++++++++++++++++++++ pvesm.adoc | 4 ++ 2 files changed, 145 insertions(+) create mode 100644 pve-storage-zfsnvme.adoc diff --git a/pve-storage-zfsnvme.adoc b/pve-storage-zfsnvme.adoc new file mode 100644 index 0000000..46d16ad --- /dev/null +++ b/pve-storage-zfsnvme.adoc @@ -0,0 +1,141 @@ +[[storage_zfsnvme]] +ZFS over NVMe/TCP Backend +------------------------- +ifdef::wiki[] +:pve-toplevel: +:title: Storage: ZFS over NVMe/TCP +endif::wiki[] + +Storage pool type: `zfsnvme` + +This backend accesses a remote Linux machine with a ZFS pool and the kernel +NVMe target through `ssh`. For each guest disk it creates a ZVOL, exports it as +an NVMe namespace, and connects the {pve} nodes through native Linux NVMe/TCP +multipath. + +The backend supports thin provisioning, snapshots, rollback, templates, linked +clones, offline resize, and shared-storage live migration. Namespace UUIDs are +stored as ZFS user properties, and the stable `nvme-uuid` device link is used +for guest disks. + +Configuration +~~~~~~~~~~~~~ + +The target needs OpenZFS, configfs, and the `nvmet` and `nvmet-tcp` kernel +modules. Configure root SSH access like for the ZFS over iSCSI backend. The key +for the server address is stored at +`/etc/pve/priv/zfs/_id_rsa`. + +Each {pve} node needs `nvme-cli`, the `nvme-tcp` module, native NVMe multipath, +a unique `/etc/nvme/hostnqn`, and `/etc/nvme/hostid`. Interface names listed in +the storage configuration must exist on every node where the storage is +enabled. + +The following properties are specific to ZFS over NVMe/TCP: + +server:: + +IP address or DNS name used for the SSH control connection. + +pool:: + +ZFS pool or child dataset used exclusively by this storage definition. + +subsysnqn:: + +NVMe qualified name of the target subsystem. It must be unique across the +cluster's storage definitions. + +nvme-portals:: + +Comma-separated NVMe/TCP target addresses. The default service is `4420`. +Specify IPv6 addresses in brackets, for example `[2001:db8::10]:4420`. + +nvme-host-ifaces:: + +Comma-separated local interfaces, matched to `nvme-portals` by position. An +explicit interface prevents a failed data path from reconnecting over the +management network. + +dhchap-key:: + +NVMe DH-HMAC-CHAP secret in `DHHC-1` representation. The value is handled as a +sensitive property and stored below `/etc/pve/priv/storage/` with mode `0600`. +In-place key rotation is not supported; create a new storage and subsystem for +a coordinated rotation. + +nvme-iopolicy:: + +Native multipath policy: `round-robin`, `queue-depth`, or `numa`. + +nvme-keep-alive-tmo:: + +Keep-alive timeout in seconds. + +nvme-reconnect-delay:: + +Delay between controller reconnect attempts in seconds. + +nvme-ctrl-loss-tmo:: + +How long a controller remains reconnectable after path loss. `-1` retries +indefinitely. + +nvme-nr-io-queues:: + +Optional number of I/O queues per controller. + +blocksize:: + +ZFS volume block size. + +sparse:: + +Use ZFS thin provisioning instead of reserving the full virtual size. + +.Configuration Example (`/etc/pve/storage.cfg`) +---- +zfsnvme: nvme-shared + server 192.0.2.10 + pool tank/pve-nvme + subsysnqn nqn.2026-07.example:pve-nvme + nvme-portals 10.10.1.10:4420,10.10.2.10:4420 + nvme-host-ifaces ens20,ens21 + blocksize 16k + sparse 1 + nvme-iopolicy round-robin + nvme-keep-alive-tmo 5 + nvme-reconnect-delay 2 + nvme-ctrl-loss-tmo 600 + content images + shared 1 +---- + +The DH-HMAC-CHAP key is intentionally not shown in `storage.cfg`. Set it with +the storage creation API or web interface. + +Security and Availability +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The target uses an ACL for the unique Host NQN of every node and never enables +`allow_any_host`. DH-HMAC-CHAP authenticates the endpoints, but it does not +encrypt data. This backend does not configure NVMe/TCP TLS, so use isolated +storage networks or equivalent protection. + +Shared access relies on {pve} cluster locking and fencing. Test quorum and +fencing before placing production guests on the storage. Use independent +failure domains for the configured paths and make sure the ZFS target itself +is not a single point of failure. + +Online resize of a running guest disk is not supported. Stop the VM before +resizing; the new size is detected when the block device is reopened. + +Storage Features +~~~~~~~~~~~~~~~~ + +.Storage features for backend `zfsnvme` +[width="100%",cols="m,m,3*d",options="header"] +|============================================================================== +|Content types |Image formats |Shared |Snapshots |Clones +|images |raw |yes |yes |yes +|============================================================================== diff --git a/pvesm.adoc b/pvesm.adoc index 5bd24b2..d503f6f 100644 --- a/pvesm.adoc +++ b/pvesm.adoc @@ -439,6 +439,8 @@ See Also * link:/wiki/Storage:_ZFS_over_ISCSI[Storage: ZFS over ISCSI] +* link:/wiki/Storage:_ZFS_over_NVMe_TCP[Storage: ZFS over NVMe/TCP] + endif::wiki[] ifndef::wiki[] @@ -471,6 +473,8 @@ include::pve-storage-btrfs.adoc[] include::pve-storage-zfs.adoc[] +include::pve-storage-zfsnvme.adoc[] + ifdef::manvolnum[] include::pve-copyright.adoc[] -- 2.54.0.windows.1