From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 655BF1FF138 for ; Tue, 21 Jul 2026 13:46:03 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3D085214B1; Tue, 21 Jul 2026 13:46:02 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Tue, 21 Jul 2026 13:45:57 +0200 Message-Id: To: "Joaquin Varela" , From: "Max R. Carrara" Subject: Re: [RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE X-Mailer: aerc 0.18.2-0-ge037c095a049 References: In-Reply-To: X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1784634331133 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.032 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6CEIXAZAIZAY6P7PXC6QF5GZ7YFTBDSW X-Message-ID-Hash: 6CEIXAZAIZAY6P7PXC6QF5GZ7YFTBDSW X-MailFrom: m.carrara@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed Jul 15, 2026 at 6:23 PM CEST, Joaquin Varela wrote: > Hello, > > I have implemented an in-tree Proxmox VE storage backend named `zfsnvme`. > It follows the remote ZFS lifecycle model used by ZFS over iSCSI, but use= s > Linux nvmet, NVMe/TCP and native kernel multipath. Hello! This sounds like a great contribution! Using the existing flow of the ZFS-over-iSCSI plugin is also a good idea. > > The ZFS dataset is the durable source of truth. Each owned zvol stores it= s > subsystem NQN, NSID and namespace UUID as ZFS user properties. Configfs i= s > reconciled from those properties, and the namespace UUID provides the sta= ble > by-id path used by QEMU. > > The implementation includes: > > - one NVMe/TCP connection per configured portal and host interface; > - native multipath and ANA state handling; > - DH-HMAC-CHAP without exposing the key in process arguments; > - thin provisioning, discard, snapshots, rollback and linked clones; > - safe teardown and foreign-zvol identity checks; > - an explicit queue-versus-fast-fail policy for complete path loss; > - pve-manager and pve-docs integration. > > Validation covered the complete pve-storage test suite (593 assertions), > two-node live migration under guest I/O, migration to a degraded > destination, > real ANA transitions, simultaneous allocations from both cluster nodes, > single and dual path loss, configfs reconstruction, watchdog HA and a liv= e > Corosync/QDevice partition. No source/destination QEMU overlap was observ= ed > in > the live partition test. > > The implementation and detailed validation report are available here: > > https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp > https://github.com/joaquinv98/pve-manager/tree/feature/zfs-nvme-tcp > https://github.com/joaquinv98/pve-docs/tree/feature/zfs-nvme-tcp > > The lab results are not presented as hardware certification. TLS, online > resize and target control-plane HA remain explicit limitations. > > I have prepared signed patch series for pve-storage, pve-manager and > pve-docs. > Before posting the full series, I would appreciate feedback on whether th= is > in-tree backend and its architecture are acceptable for upstream review. I have skimmed through the respective branches on your repositories above, and from what I can tell, the commits look pretty decent. I do have some remarks up front that you may want to consider before sending this in as a patch series -- mostly to save you some time: - If you haven't already, please sign our CLA. You can find the respective section here: https://proxmox.com/en/about/open-source/develop= ers - From what I can tell, you're already adhering quite well to our style guide, which is great! A few remarks: * I suggest using `use v5.36;` for new Perl modules instead of `use strict; use warnings;`, as that enables subroutine signatures by default. Helps modernizing the code base overall. See our style guide for more details: https://pve.proxmox.com/wiki/Perl= _Style_Guide * While not in the style guide, I can personally recommend using the `n` and `x`/`xx` modifiers for your regexes, and declaring any inline regexes as constants (=3D> `my $RE_NVME_FOO =3D qr/.../nxx`) in a little section at the top of your module. These flags help future readers to understand your regexes a little easier, and since `n` forces the use of named capture groups, they'll also more likely to be "self-documenting" in a sense. * For subroutines that are not part of the storage plugin API, i.e. helpers, utilts, etc. specific to your module, prefer declaring them using `my sub` instead of a plain `sub`, as that keeps them visible inside the module only (=3D> "private" in other languages). That prevents other places from using these helpers. You can then also drop the `_` prefix in that case (if you prefer). * I also suggest using postfix dereferencing instead of the "traditional" dereferencing syntax, that is, `$hashref->%*` instead of `%$hashref`, for example. It's a little more legible and makes nested hash access a little easier to follow as well. Note that this isn't explicitly stated in our style guide either (we only have it there for subref calls, hash and array accessing), so no hard feelings. - When you submit your series, please do not any entries in `debian/changelog` -- though it's nice to see that you've been thorough there! That's all I can think of / find at the moment without giving it a proper in-depth review. I'll happily review your series once you send it in. I'd also be interested in what kind of storage backend you were using for your tests, so that I can replicate your setup myself. I have skimmed through your docs and validation reports, but haven't found anything specific -- maybe I just missed it. > > Kind regards, Thanks a lot for your interest in contributing and your work already, it's much appreciated! Kind regards, Max