[RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE

[RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE

[Joaquin Varela]

Hello,

I have implemented an in-tree Proxmox VE storage backend named `zfsnvme`.
It follows the remote ZFS lifecycle model used by ZFS over iSCSI, but uses
Linux nvmet, NVMe/TCP and native kernel multipath.

The ZFS dataset is the durable source of truth. Each owned zvol stores its
subsystem NQN, NSID and namespace UUID as ZFS user properties. Configfs is
reconciled from those properties, and the namespace UUID provides the stable
by-id path used by QEMU.

The implementation includes:

- one NVMe/TCP connection per configured portal and host interface;
- native multipath and ANA state handling;
- DH-HMAC-CHAP without exposing the key in process arguments;
- thin provisioning, discard, snapshots, rollback and linked clones;
- safe teardown and foreign-zvol identity checks;
- an explicit queue-versus-fast-fail policy for complete path loss;
- pve-manager and pve-docs integration.

Validation covered the complete pve-storage test suite (593 assertions),
two-node live migration under guest I/O, migration to a degraded
destination,
real ANA transitions, simultaneous allocations from both cluster nodes,
single and dual path loss, configfs reconstruction, watchdog HA and a live
Corosync/QDevice partition. No source/destination QEMU overlap was observed
in
the live partition test.

The implementation and detailed validation report are available here:

https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp
https://github.com/joaquinv98/pve-manager/tree/feature/zfs-nvme-tcp
https://github.com/joaquinv98/pve-docs/tree/feature/zfs-nvme-tcp

The lab results are not presented as hardware certification. TLS, online
resize and target control-plane HA remain explicit limitations.

I have prepared signed patch series for pve-storage, pve-manager and
pve-docs.
Before posting the full series, I would appreciate feedback on whether this
in-tree backend and its architecture are acceptable for upstream review.

Kind regards,

-- 
Joaquin Varela

Re: [RFC] Native ZFS over NVMe/TCP storage backend for Proxmox VE

[Max R. Carrara]

On Wed Jul 15, 2026 at 6:23 PM CEST, Joaquin Varela wrote:
> Hello,
>
> I have implemented an in-tree Proxmox VE storage backend named `zfsnvme`.
> It follows the remote ZFS lifecycle model used by ZFS over iSCSI, but uses
> Linux nvmet, NVMe/TCP and native kernel multipath.

Hello!

This sounds like a great contribution! Using the existing flow of the
ZFS-over-iSCSI plugin is also a good idea.

>
> The ZFS dataset is the durable source of truth. Each owned zvol stores its
> subsystem NQN, NSID and namespace UUID as ZFS user properties. Configfs is
> reconciled from those properties, and the namespace UUID provides the stable
> by-id path used by QEMU.
>
> The implementation includes:
>
> - one NVMe/TCP connection per configured portal and host interface;
> - native multipath and ANA state handling;
> - DH-HMAC-CHAP without exposing the key in process arguments;
> - thin provisioning, discard, snapshots, rollback and linked clones;
> - safe teardown and foreign-zvol identity checks;
> - an explicit queue-versus-fast-fail policy for complete path loss;
> - pve-manager and pve-docs integration.
>
> Validation covered the complete pve-storage test suite (593 assertions),
> two-node live migration under guest I/O, migration to a degraded
> destination,
> real ANA transitions, simultaneous allocations from both cluster nodes,
> single and dual path loss, configfs reconstruction, watchdog HA and a live
> Corosync/QDevice partition. No source/destination QEMU overlap was observed
> in
> the live partition test.
>
> The implementation and detailed validation report are available here:
>
> https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp
> https://github.com/joaquinv98/pve-manager/tree/feature/zfs-nvme-tcp
> https://github.com/joaquinv98/pve-docs/tree/feature/zfs-nvme-tcp
>
> The lab results are not presented as hardware certification. TLS, online
> resize and target control-plane HA remain explicit limitations.
>
> I have prepared signed patch series for pve-storage, pve-manager and
> pve-docs.
> Before posting the full series, I would appreciate feedback on whether this
> in-tree backend and its architecture are acceptable for upstream review.

I have skimmed through the respective branches on your repositories
above, and from what I can tell, the commits look pretty decent.

I do have some remarks up front that you may want to consider before
sending this in as a patch series -- mostly to save you some time:

- If you haven't already, please sign our CLA. You can find the
  respective section here: https://proxmox.com/en/about/open-source/developers

- From what I can tell, you're already adhering quite well to our style
  guide, which is great! A few remarks:

  * I suggest using `use v5.36;` for new Perl modules instead of
    `use strict; use warnings;`, as that enables subroutine signatures
    by default. Helps modernizing the code base overall.
    See our style guide for more details: https://pve.proxmox.com/wiki/Perl_Style_Guide

  * While not in the style guide, I can personally recommend using the
    `n` and `x`/`xx` modifiers for your regexes, and declaring any
    inline regexes as constants (=> `my $RE_NVME_FOO = qr/.../nxx`)
    in a little section at the top of your module. These flags help
    future readers to understand your regexes a little easier, and since
    `n` forces the use of named capture groups, they'll also more likely
    to be "self-documenting" in a sense.

  * For subroutines that are not part of the storage plugin API, i.e.
    helpers, utilts, etc. specific to your module, prefer declaring them
    using `my sub` instead of a plain `sub`, as that keeps them visible
    inside the module only (=> "private" in other languages). That
    prevents other places from using these helpers.

    You can then also drop the `_` prefix in that case (if you prefer).

  * I also suggest using postfix dereferencing instead of the
    "traditional" dereferencing syntax, that is, `$hashref->%*` instead
    of `%$hashref`, for example. It's a little more legible and makes
    nested hash access a little easier to follow as well. Note that this
    isn't explicitly stated in our style guide either (we only have it
    there for subref calls, hash and array accessing), so no hard
    feelings.

- When you submit your series, please do not any entries in
  `debian/changelog` -- though it's nice to see that you've been
  thorough there!

That's all I can think of / find at the moment without giving it a
proper in-depth review. I'll happily review your series once you send it
in.

I'd also be interested in what kind of storage backend you were using
for your tests, so that I can replicate your setup myself. I have
skimmed through your docs and validation reports, but haven't found
anything specific -- maybe I just missed it.

>
> Kind regards,

Thanks a lot for your interest in contributing and your work already,
it's much appreciated!

Kind regards,
Max