From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id EEE3B1FF165 for ; Thu, 17 Jul 2025 16:16:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id F169A3C96E; Thu, 17 Jul 2025 16:16:52 +0200 (CEST) Mime-Version: 1.0 Date: Thu, 17 Jul 2025 16:16:49 +0200 Message-Id: From: "Christoph Heiss" To: "Proxmox VE development discussion" X-Mailer: aerc 0.20.1 References: <20250703115621.883244-1-c.heiss@proxmox.com> In-Reply-To: <20250703115621.883244-1-c.heiss@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1752761808285 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH ve-rs/firewall/qemu-server/manager/docs v3 00/14] fix #5180: migrate conntrack state on live migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" v4 posted: https://lore.proxmox.com/pve-devel/20250717141530.1471199-1-c.heiss@proxmox.com/ On Thu Jul 3, 2025 at 1:54 PM CEST, Christoph Heiss wrote: > Fixes #5180 [0]. > > This implements migration of per-VM conntrack state on live-migration. > > The core of the implementation are in patch #7 & #8. See there for more > details. > > Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique > VMID. This is needed later on to filter conntrack entries for the > migration. These three patches can be applied independently, > CONNMARK'ing traffic does not have any visible impact. > > Regarding the use of connlabels instead, as was discussed in v2: > Connlabels are (unfortunately) pure bitmasks and cannot be used for > storing arbitrary values such as VMIDs. Thus we are basically stuck with > using connmarks. > > Currently, remote/inter-cluster migration is not supported and indicated > to the user with a warning. See also patch #8 for a bit more in-depth > explanation. > > [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180 > > Dependencies > ============ > > proxmox-firewall depends on the proxmox-ve-rs changes. > qemu-server depends on the pve-firewall/proxmox-firewall changes. > > pve-manager only soft-depends on the other, as it will detect whether > conntrack migration is supported. > > Testing > ======= > > I've primarily tested intra-cluster live-migrations, with both the > iptables-based and nftables-based firewall), using the reproducer as > described in #5180. I further verified that the D-Bus services get > started as expected and are _always_ stopped, even in the case of some > migration error. > > Finally, I also checked using `conntrack -L -m ` tool that the > conntrack entries are > a) added/updated on the target node and > b) removed from the source node afterwards > > Also tested was the migration from/to an "old" (unpatched) node, which > results in the issue as per #5180 & appropriate warnings in the UI. > > For remote migrations, tested that the warning is logged as expected. > > History > ======= > > v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/ > v2: https://lore.proxmox.com/pve-devel/20250424111941.730528-1-c.heiss@proxmox.com/ > > Changes v1 -> v2: > * rebased as necessary > * "un-rfc'd" firewall conntrack flushing patches > * use an instanced systemd service instead of fork+exec for the > pve-dbus-vmstate helper > > Changes v2 -> v3: > * rebased on trixie/latest masters > * added documentation patch > * moved node capability module to > PVE::API2::NodeCapabilities::Qemu::Migration, based on Fiona's > suggestion > > Diffstat > ======== > > proxmox-ve-rs: > > Christoph Heiss (1): > config: guest: allow access to raw Vmid value > > proxmox-ve-config/src/guest/types.rs | 4 ++++ > 1 file changed, 4 insertions(+) > > proxmox-firewall: > > Christoph Heiss (1): > firewall: add connmark rule with VMID to all guest chains > > proxmox-firewall/src/firewall.rs | 14 +++- > .../integration_tests__firewall.snap | 84 +++++++++++++++++++ > proxmox-nftables/src/expression.rs | 9 ++ > proxmox-nftables/src/statement.rs | 10 ++- > 4 files changed, 114 insertions(+), 3 deletions(-) > > pve-firewall: > > Christoph Heiss (2): > firewall: add connmark rule with VMID to all guest chains > firewall: helpers: add sub for flushing conntrack entries by mark > > debian/control | 3 ++- > src/PVE/Firewall.pm | 14 ++++++++++++-- > src/PVE/Firewall/Helpers.pm | 12 ++++++++++++ > 3 files changed, 26 insertions(+), 3 deletions(-) > > qemu-server: > > Christoph Heiss (5): > qmp helpers: allow passing structured args via qemu_objectadd() > api2: qemu: add module exposing node migration capabilities > fix #5180: dbus-vmstate: add daemon for QEMUs dbus-vmstate interface > fix #5180: migrate: integrate helper for live-migrating conntrack info > migrate: flush old VM conntrack entries after successful migration > > Makefile | 4 +- > debian/control | 7 +- > src/Makefile | 1 + > src/PVE/API2/Makefile | 1 + > src/PVE/API2/NodeCapabilities/Makefile | 9 + > .../API2/NodeCapabilities/Qemu/Migration.pm | 48 +++++ > src/PVE/API2/Qemu.pm | 75 ++++++++ > src/PVE/CLI/qm.pm | 5 + > src/PVE/QemuMigrate.pm | 78 ++++++++ > src/PVE/QemuServer.pm | 6 + > src/PVE/QemuServer/DBusVMState.pm | 125 +++++++++++++ > src/PVE/QemuServer/Makefile | 1 + > src/PVE/QemuServer/QMPHelpers.pm | 4 +- > src/dbus-vmstate/Makefile | 11 ++ > src/dbus-vmstate/dbus-vmstate | 168 ++++++++++++++++++ > src/dbus-vmstate/org.qemu.VMState1.conf | 11 ++ > src/dbus-vmstate/pve-dbus-vmstate@.service | 10 ++ > 17 files changed, 560 insertions(+), 4 deletions(-) > create mode 100644 src/PVE/API2/NodeCapabilities/Makefile > create mode 100644 src/PVE/API2/NodeCapabilities/Qemu/Migration.pm > create mode 100644 src/PVE/QemuServer/DBusVMState.pm > create mode 100644 src/dbus-vmstate/Makefile > create mode 100755 src/dbus-vmstate/dbus-vmstate > create mode 100644 src/dbus-vmstate/org.qemu.VMState1.conf > create mode 100644 src/dbus-vmstate/pve-dbus-vmstate@.service > > pve-manager: > > Christoph Heiss (4): > api2: capabilities: explicitly import CPU capabilities module > api2: capabilities: proxy index endpoints to respective nodes > api2: capabilities: expose new qemu/migration endpoint > ui: window: Migrate: add checkbox for migrating VM conntrack state > > PVE/API2/Capabilities.pm | 11 ++++- > www/manager6/window/Migrate.js | 82 ++++++++++++++++++++++++++++++++-- > 2 files changed, 89 insertions(+), 4 deletions(-) > > pve-docs: > > Christoph Heiss (1): > qm: document conntrack state migration for live migrations > > qm.adoc | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel