Re: [RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE

[Joaquin Varela <joaquinvarela@neatech.ar>]

Hi Max,

sorry for the delayed reply. Gmail had grouped your response into the
original RFC conversation and I missed it at the time.

Thank you very much for taking the time to review the repositories and
for the encouraging feedback.

I signed and submitted the Proxmox Individual CLA on July 15.

I have now applied the style changes you suggested:

- the new Perl modules use `use v5.36`;
- the relevant regular expressions are declared as constants and use
  the `n` and `x`/`xx` modifiers with named capture groups;
- internal helpers use lexical `my sub` declarations where appropriate;
- the new code uses postfix dereferencing;
- the submission branches do not contain debian/changelog entries.

Regarding the storage backend used for validation, the target was a
dedicated Ubuntu 24.04 virtual machine running:

- kernel 6.8.0-136-generic;
- ZFS 2.2.2;
- Linux nvmet;
- nvme-cli 2.8;
- a ZFS pool named `tank`;
- two independent NVMe/TCP portals.

The initiators were a two-node nested Proxmox VE 9.2 cluster running
kernel 7.0.14-8-pve and nvme-cli 2.13. Each node had two independent
NVMe/TCP paths and native kernel multipath enabled.

The tests covered thin provisioning and discard, snapshots, clones,
concurrent namespace allocation, path failures, all-path loss policies,
target reboot reconstruction, bidirectional live migration under guest
I/O, migration to a degraded destination, HA fencing, a live corosync
partition and explicit dual-writer detection.

The target-reboot testing also uncovered an ACL publication race. The
backend now restores every configured Host NQN, DH-HMAC-CHAP key and
namespace before linking the subsystem to its NVMe/TCP ports.

The complete test report, machine-readable results and raw logs are
available here:

https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp

The clean branches corresponding to the patch series are:

https://github.com/joaquinv98/pve-storage/tree/submission/zfs-nvme-tcp-rfc-v2
https://github.com/joaquinv98/pve-manager/tree/submission/zfs-nvme-tcp-rfc-v2
https://github.com/joaquinv98/pve-docs/tree/submission/zfs-nvme-tcp-rfc-v2

I have prepared the formal pve-storage, pve-manager and pve-docs patch
series and will send them to pve-devel as separate threads shortly.

Thank you again for your review.

Kind regards,
Joaquin Varela

On Tue, Jul 21, 2026 at 8:46 AM Max R. Carrara <m.carrara@proxmox.com>
wrote:

> On Wed Jul 15, 2026 at 6:23 PM CEST, Joaquin Varela wrote:
> > Hello,
> >
> > I have implemented an in-tree Proxmox VE storage backend named `zfsnvme`.
> > It follows the remote ZFS lifecycle model used by ZFS over iSCSI, but
> uses
> > Linux nvmet, NVMe/TCP and native kernel multipath.
>
> Hello!
>
> This sounds like a great contribution! Using the existing flow of the
> ZFS-over-iSCSI plugin is also a good idea.
>
> >
> > The ZFS dataset is the durable source of truth. Each owned zvol stores
> its
> > subsystem NQN, NSID and namespace UUID as ZFS user properties. Configfs
> is
> > reconciled from those properties, and the namespace UUID provides the
> stable
> > by-id path used by QEMU.
> >
> > The implementation includes:
> >
> > - one NVMe/TCP connection per configured portal and host interface;
> > - native multipath and ANA state handling;
> > - DH-HMAC-CHAP without exposing the key in process arguments;
> > - thin provisioning, discard, snapshots, rollback and linked clones;
> > - safe teardown and foreign-zvol identity checks;
> > - an explicit queue-versus-fast-fail policy for complete path loss;
> > - pve-manager and pve-docs integration.
> >
> > Validation covered the complete pve-storage test suite (593 assertions),
> > two-node live migration under guest I/O, migration to a degraded
> > destination,
> > real ANA transitions, simultaneous allocations from both cluster nodes,
> > single and dual path loss, configfs reconstruction, watchdog HA and a
> live
> > Corosync/QDevice partition. No source/destination QEMU overlap was
> observed
> > in
> > the live partition test.
> >
> > The implementation and detailed validation report are available here:
> >
> > https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp
> > https://github.com/joaquinv98/pve-manager/tree/feature/zfs-nvme-tcp
> > https://github.com/joaquinv98/pve-docs/tree/feature/zfs-nvme-tcp
> >
> > The lab results are not presented as hardware certification. TLS, online
> > resize and target control-plane HA remain explicit limitations.
> >
> > I have prepared signed patch series for pve-storage, pve-manager and
> > pve-docs.
> > Before posting the full series, I would appreciate feedback on whether
> this
> > in-tree backend and its architecture are acceptable for upstream review.
>
> I have skimmed through the respective branches on your repositories
> above, and from what I can tell, the commits look pretty decent.
>
> I do have some remarks up front that you may want to consider before
> sending this in as a patch series -- mostly to save you some time:
>
> - If you haven't already, please sign our CLA. You can find the
>   respective section here:
> https://proxmox.com/en/about/open-source/developers
>
> - From what I can tell, you're already adhering quite well to our style
>   guide, which is great! A few remarks:
>
>   * I suggest using `use v5.36;` for new Perl modules instead of
>     `use strict; use warnings;`, as that enables subroutine signatures
>     by default. Helps modernizing the code base overall.
>     See our style guide for more details:
> https://pve.proxmox.com/wiki/Perl_Style_Guide
>
>   * While not in the style guide, I can personally recommend using the
>     `n` and `x`/`xx` modifiers for your regexes, and declaring any
>     inline regexes as constants (=> `my $RE_NVME_FOO = qr/.../nxx`)
>     in a little section at the top of your module. These flags help
>     future readers to understand your regexes a little easier, and since
>     `n` forces the use of named capture groups, they'll also more likely
>     to be "self-documenting" in a sense.
>
>   * For subroutines that are not part of the storage plugin API, i.e.
>     helpers, utilts, etc. specific to your module, prefer declaring them
>     using `my sub` instead of a plain `sub`, as that keeps them visible
>     inside the module only (=> "private" in other languages). That
>     prevents other places from using these helpers.
>
>     You can then also drop the `_` prefix in that case (if you prefer).
>
>   * I also suggest using postfix dereferencing instead of the
>     "traditional" dereferencing syntax, that is, `$hashref->%*` instead
>     of `%$hashref`, for example. It's a little more legible and makes
>     nested hash access a little easier to follow as well. Note that this
>     isn't explicitly stated in our style guide either (we only have it
>     there for subref calls, hash and array accessing), so no hard
>     feelings.
>
> - When you submit your series, please do not any entries in
>   `debian/changelog` -- though it's nice to see that you've been
>   thorough there!
>
> That's all I can think of / find at the moment without giving it a
> proper in-depth review. I'll happily review your series once you send it
> in.
>
> I'd also be interested in what kind of storage backend you were using
> for your tests, so that I can replicate your setup myself. I have
> skimmed through your docs and validation reports, but haven't found
> anything specific -- maybe I just missed it.
>
> >
> > Kind regards,
>
> Thanks a lot for your interest in contributing and your work already,
> it's much appreciated!
>
> Kind regards,
> Max
>
>
>

-- 
Joaquin Varela