From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id E661D1FF129 for ; Sun, 02 Aug 2026 05:32:48 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id AE01D207DA; Sun, 02 Aug 2026 05:32:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=neatech-ar.20251104.gappssmtp.com; s=20251104; t=1785641557; x=1786246357; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to:content-type; bh=EUdVd4sYG3/yxYb39zufXHJgyKn3ZlvUkqD4GdyjTK4=; b=aZQMOcKZBUNE75G8D1+wgHmoWH8NSgw2fShY6uWZLZXWVsBBlnv3YT5pOHlPG9hXpZ bjd8jTdWsw7YjpO5BZiu42gTuDxYhxtIIAE4UMdSHzU367GYh/Ob6fnakICZQvUSkmmT t9eb1YRFcVpfBhTW/m6x/hiL8WeGK7LFsVQEQF/mpdGlkMIsxpFV09NajMui2ZALQWc8 rQgcYlyyh/J91oX6tVARZOa54K7ROSUo9N+LER9luUsmvfb4wj0kmVQNye10wGKvYMQZ 2JF3X4ay6pSuOOvgaMwruWDCQkVBn6LPYqJFRUu/2L2fOXhPGy0pe7uGkTQHHrgsIFjM VMKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785641557; x=1786246357; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EUdVd4sYG3/yxYb39zufXHJgyKn3ZlvUkqD4GdyjTK4=; b=sxcOC+scG2Rp2qkfYfc4VjRccMdAkiFwzmH1m/iMsgE+t6lu7Isy+xd5rQeD3FcmKJ cEipp5g6TlkOvl0UaslrjCcmD+E7fujQMMT20gJTmv033NiCtJYqvKz5WCbS3UTmMT9p IVtk21ehdLE7+jXB4IjjUVgRU+2OtPc609OMyO0a5/TZFRyxh8dqDKAXFIkaBORRDHl6 1ppNJiVayqiyTi/LS4g3BXzM1XRnptW3f+oEavvbBii1lbocakMLFT5LgMHfI6vV9W/W C5gu96LBkqJ2xZVpLWeELf7P/7cWyqT2NVmKrXCdEec9JjKEH/Nv0KKrOsSoI44ta2ZT jfFg== X-Gm-Message-State: AOJu0Yz2gIJd66P1NjRL17imjwr68+ybDy3EyPdL97iOnHgVD2MJNTDx dWKhyi5ZS7z5Oi45QSIA1dq5+9CenmmMkNrRFD2ODNE1q7rU8zpbnesuT+Vx3ITJu9MKGZU6Olo eshPYneg= X-Gm-Gg: AR+sD13qeD+zPBarbteQYdmYL2XBo9rPg0Vn9btRbNsgFIVWbGgDcmKtpHNjddln3bI M1UWpt5fQxO1VjX68NINFWN8mSuU8dy7z+0f/4QgtoDkS92KxzoRy3ZhFYIFbtqw52QcsLGeI3H H9cYiKM80Tp9UaeQFgjW8lsHM+/tJ/XCjjIXEfDumV+S/goAS5Y4LITX4q5ItQBOowLYERLH0SL jhR3/bB2iqbV2nyHy4pxgLzPP+h/rHLxEEZBVlpu+vknov9bGtQGExBUKXdroF3SRsds1j4DzeJ LPm0tTJkK4Luf4XjoCTO+wi7eNyK5iVOgHSbhA8s5p7JVcfk3KlbFLfPf96drR61X0bGMlJrfFV 2gRzNp3s5PwOFGjGGiF7ROu0vd12ipCKd2mVus8NSHoean/59X+QCgIbjOwWUqsVliOoq4lEK0y 1EX8n1T0dtDqpOp1f2vbz25xonKuv+yTkOwNt+dzs+2XkVCrVmJAzIR5ul6fMsSr1sIT2Yr1QPW +HXK+LupbOml4fFWui1qnfn6NrCqe7mzJGnSGf+lFY4 X-Received: by 2002:a05:6a21:6906:b0:3c3:8f7a:c0a2 with SMTP id adf61e73a8af0-3c92a8d1775mr7063132637.4.1785641556492; Sat, 01 Aug 2026 20:32:36 -0700 (PDT) From: Joaquin Varela To: pve-devel@lists.proxmox.com Subject: [PATCH storage v2 0/7] add native ZFS over NVMe/TCP backend Date: Sun, 2 Aug 2026 00:31:34 -0300 Message-ID: X-Mailer: git-send-email 2.54.0.windows.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RXS7G3PAMAVSY7M4OVYKCTN72ZOYX6UK X-Message-ID-Hash: RXS7G3PAMAVSY7M4OVYKCTN72ZOYX6UK X-MailFrom: joaquinvarela@neatech.ar X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Joaquin Varela X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, this series adds an in-tree `zfsnvme` shared storage backend. It reuses the remote ZFS lifecycle model of ZFS over iSCSI, publishes zvols through Linux nvmet, and uses native NVMe multipath on PVE initiators. The ZFS dataset remains the source of truth for namespace identity. The backend stores NQN, NSID and UUID as ZFS user properties and reconstructs derived configfs state after a target restart. Target publication is atomic: all namespaces, Host NQN ACLs and authentication keys are restored before the subsystem is linked to its NVMe/TCP ports. This revision follows the feedback on the RFC: * new Perl modules use v5.36; * regular expressions are named constants using n/x and named captures; * module-private helpers use lexical subroutines where appropriate; * postfix dereferencing is used throughout; * no debian/changelog entries are included. It also makes the all-path-loss policy explicit, checks active users before teardown, and handles both the current activation-hints API and the short activation form used by cloud-init. Test target: * Ubuntu 24.04, kernel 6.8.0-136-generic; * ZFS 2.2.2, Linux nvmet, nvme-cli 2.8; * one ZFS pool with two independent 100 Mbit/s NVMe/TCP paths. Initiators were a two-node nested PVE 9.2 cluster running kernel 7.0.14-8-pve and nvme-cli 2.13. The test matrix covered thin provisioning and discard, snapshots and clones, concurrent allocation, ANA multipath, single/all-path faults, bidirectional live migration under guest I/O, watchdog fencing, live split-brain isolation, target reboot reconstruction, package upgrade protection, and a verified 45.7 GB mixed-I/O soak. No dual-writer interval was observed. The complete report, machine-readable results and raw logs are available at: https://github.com/joaquinv98/pve-storage/tree/feature/zfs-nvme-tcp The branch corresponding exactly to this series is: https://github.com/joaquinv98/pve-storage/tree/submission/zfs-nvme-tcp-rfc-v2 This remains a lab-qualified release candidate; hardware qualification and a long-duration production-hardware soak are intentionally not claimed. Joaquin Varela (7): zfs: make LUN provider dispatch overridable zfs: add native NVMe/TCP storage backend zfsnvme: harden node preflight and storage teardown zfsnvme: make all-path loss policy explicit zfsnvme: accept activation hints from storage API zfsnvme: accept short volume activation calls zfsnvme: restore ACLs before publishing target debian/control | 1 + src/PVE/Storage.pm | 2 + src/PVE/Storage/LunCmd/Makefile | 2 +- src/PVE/Storage/LunCmd/NVMET.pm | 722 +++++++++++++++++++++++++ src/PVE/Storage/Makefile | 1 + src/PVE/Storage/Plugin.pm | 2 +- src/PVE/Storage/ZFSNVMePlugin.pm | 893 +++++++++++++++++++++++++++++++ src/PVE/Storage/ZFSPlugin.pm | 45 +- src/test/run_plugin_tests.pl | 1 + src/test/zfsnvme_test.pm | 513 ++++++++++++++++++ 10 files changed, 2158 insertions(+), 24 deletions(-) create mode 100644 src/PVE/Storage/LunCmd/NVMET.pm create mode 100644 src/PVE/Storage/ZFSNVMePlugin.pm create mode 100644 src/test/zfsnvme_test.pm -- 2.54.0.windows.1