From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id E08201FF0E6 for ; Fri, 07 Aug 2026 11:46:36 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8720321517; Fri, 07 Aug 2026 11:46:36 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Fri, 07 Aug 2026 11:46:32 +0200 Message-Id: Subject: Re: [RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE To: "Joaquin Varela" From: "Max R. Carrara" X-Mailer: aerc 0.18.2-0-ge037c095a049 References: In-Reply-To: X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1786095976251 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.773 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 5FQ7CYE5PGDH2HZGRKR2JOZP6SREV6EY X-Message-ID-Hash: 5FQ7CYE5PGDH2HZGRKR2JOZP6SREV6EY X-MailFrom: m.carrara@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Sun Aug 2, 2026 at 5:20 AM CEST, Joaquin Varela wrote: > Hi Max, > > sorry for the delayed reply. Gmail had grouped your response into the > original RFC conversation and I missed it at the time. > > Thank you very much for taking the time to review the repositories and > for the encouraging feedback. > > I signed and submitted the Proxmox Individual CLA on July 15. Hi Joaquin! All good! You're very welcome. Thanks for all of your work and for submitting your CLA -- I will have a look at the patch series you have sent in once I find some time. It's definitely in my backlog, so I will see to it eventually. Thanks again! - Max > > I have now applied the style changes you suggested: > > - the new Perl modules use `use v5.36`; > - the relevant regular expressions are declared as constants and use > the `n` and `x`/`xx` modifiers with named capture groups; > - internal helpers use lexical `my sub` declarations where appropriate; > - the new code uses postfix dereferencing; > - the submission branches do not contain debian/changelog entries. > > Regarding the storage backend used for validation, the target was a > dedicated Ubuntu 24.04 virtual machine running: > > - kernel 6.8.0-136-generic; > - ZFS 2.2.2; > - Linux nvmet; > - nvme-cli 2.8; > - a ZFS pool named `tank`; > - two independent NVMe/TCP portals. > > The initiators were a two-node nested Proxmox VE 9.2 cluster running > kernel 7.0.14-8-pve and nvme-cli 2.13. Each node had two independent > NVMe/TCP paths and native kernel multipath enabled. > > The tests covered thin provisioning and discard, snapshots, clones, > concurrent namespace allocation, path failures, all-path loss policies, > target reboot reconstruction, bidirectional live migration under guest > I/O, migration to a degraded destination, HA fencing, a live corosync > partition and explicit dual-writer detection. > > The target-reboot testing also uncovered an ACL publication race. The > backend now restores every configured Host NQN, DH-HMAC-CHAP key and > namespace before linking the subsystem to its NVMe/TCP ports. > > The complete test report, machine-readable results and raw logs are > available here: > > https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp > > The clean branches corresponding to the patch series are: > > https://github.com/joaquinv98/pve-storage/tree/submission/zfs-nvme-tcp-rf= c-v2 > https://github.com/joaquinv98/pve-manager/tree/submission/zfs-nvme-tcp-rf= c-v2 > https://github.com/joaquinv98/pve-docs/tree/submission/zfs-nvme-tcp-rfc-v= 2 > > I have prepared the formal pve-storage, pve-manager and pve-docs patch > series and will send them to pve-devel as separate threads shortly. > > Thank you again for your review. > > Kind regards, > Joaquin Varela > > On Tue, Jul 21, 2026 at 8:46=E2=80=AFAM Max R. Carrara > wrote: > > > On Wed Jul 15, 2026 at 6:23 PM CEST, Joaquin Varela wrote: > > > Hello, > > > > > > I have implemented an in-tree Proxmox VE storage backend named `zfsnv= me`. > > > It follows the remote ZFS lifecycle model used by ZFS over iSCSI, but > > uses > > > Linux nvmet, NVMe/TCP and native kernel multipath. > > > > Hello! > > > > This sounds like a great contribution! Using the existing flow of the > > ZFS-over-iSCSI plugin is also a good idea. > > > > > > > > The ZFS dataset is the durable source of truth. Each owned zvol store= s > > its > > > subsystem NQN, NSID and namespace UUID as ZFS user properties. Config= fs > > is > > > reconciled from those properties, and the namespace UUID provides the > > stable > > > by-id path used by QEMU. > > > > > > The implementation includes: > > > > > > - one NVMe/TCP connection per configured portal and host interface; > > > - native multipath and ANA state handling; > > > - DH-HMAC-CHAP without exposing the key in process arguments; > > > - thin provisioning, discard, snapshots, rollback and linked clones; > > > - safe teardown and foreign-zvol identity checks; > > > - an explicit queue-versus-fast-fail policy for complete path loss; > > > - pve-manager and pve-docs integration. > > > > > > Validation covered the complete pve-storage test suite (593 assertion= s), > > > two-node live migration under guest I/O, migration to a degraded > > > destination, > > > real ANA transitions, simultaneous allocations from both cluster node= s, > > > single and dual path loss, configfs reconstruction, watchdog HA and a > > live > > > Corosync/QDevice partition. No source/destination QEMU overlap was > > observed > > > in > > > the live partition test. > > > > > > The implementation and detailed validation report are available here: > > > > > > https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp > > > https://github.com/joaquinv98/pve-manager/tree/feature/zfs-nvme-tcp > > > https://github.com/joaquinv98/pve-docs/tree/feature/zfs-nvme-tcp > > > > > > The lab results are not presented as hardware certification. TLS, onl= ine > > > resize and target control-plane HA remain explicit limitations. > > > > > > I have prepared signed patch series for pve-storage, pve-manager and > > > pve-docs. > > > Before posting the full series, I would appreciate feedback on whethe= r > > this > > > in-tree backend and its architecture are acceptable for upstream revi= ew. > > > > I have skimmed through the respective branches on your repositories > > above, and from what I can tell, the commits look pretty decent. > > > > I do have some remarks up front that you may want to consider before > > sending this in as a patch series -- mostly to save you some time: > > > > - If you haven't already, please sign our CLA. You can find the > > respective section here: > > https://proxmox.com/en/about/open-source/developers > > > > - From what I can tell, you're already adhering quite well to our style > > guide, which is great! A few remarks: > > > > * I suggest using `use v5.36;` for new Perl modules instead of > > `use strict; use warnings;`, as that enables subroutine signatures > > by default. Helps modernizing the code base overall. > > See our style guide for more details: > > https://pve.proxmox.com/wiki/Perl_Style_Guide > > > > * While not in the style guide, I can personally recommend using the > > `n` and `x`/`xx` modifiers for your regexes, and declaring any > > inline regexes as constants (=3D> `my $RE_NVME_FOO =3D qr/.../nxx`) > > in a little section at the top of your module. These flags help > > future readers to understand your regexes a little easier, and sinc= e > > `n` forces the use of named capture groups, they'll also more likel= y > > to be "self-documenting" in a sense. > > > > * For subroutines that are not part of the storage plugin API, i.e. > > helpers, utilts, etc. specific to your module, prefer declaring the= m > > using `my sub` instead of a plain `sub`, as that keeps them visible > > inside the module only (=3D> "private" in other languages). That > > prevents other places from using these helpers. > > > > You can then also drop the `_` prefix in that case (if you prefer). > > > > * I also suggest using postfix dereferencing instead of the > > "traditional" dereferencing syntax, that is, `$hashref->%*` instead > > of `%$hashref`, for example. It's a little more legible and makes > > nested hash access a little easier to follow as well. Note that thi= s > > isn't explicitly stated in our style guide either (we only have it > > there for subref calls, hash and array accessing), so no hard > > feelings. > > > > - When you submit your series, please do not any entries in > > `debian/changelog` -- though it's nice to see that you've been > > thorough there! > > > > That's all I can think of / find at the moment without giving it a > > proper in-depth review. I'll happily review your series once you send i= t > > in. > > > > I'd also be interested in what kind of storage backend you were using > > for your tests, so that I can replicate your setup myself. I have > > skimmed through your docs and validation reports, but haven't found > > anything specific -- maybe I just missed it. > > > > > > > > Kind regards, > > > > Thanks a lot for your interest in contributing and your work already, > > it's much appreciated! > > > > Kind regards, > > Max > > > > > >