From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 6B2FE1FF183 for <inbox@lore.proxmox.com>; Wed, 4 Jun 2025 10:59:58 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D27F237A15; Wed, 4 Jun 2025 11:00:17 +0200 (CEST) Mime-Version: 1.0 Date: Wed, 04 Jun 2025 11:00:12 +0200 Message-Id: <DADMJKSON9DT.3HR5ISL6Q6EZZ@proxmox.com> From: "Christoph Heiss" <c.heiss@proxmox.com> To: "Stefan Hanreich" <s.hanreich@proxmox.com> X-Mailer: aerc 0.20.1 References: <20250424111941.730528-1-c.heiss@proxmox.com> <902b7cc1-8962-4bf6-be1e-a996f24798e3@proxmox.com> In-Reply-To: <902b7cc1-8962-4bf6-be1e-a996f24798e3@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.030 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [dbus-vmstate.com, expression.rs, statement.rs, capabilities.pm, dbusvmstate.pm, helpers.pm, qm.pm, proxmox.com, migration.pm, firewall.rs, types.rs, firewall.pm, qemu.pm, qemumigrate.pm, qemuserver.pm, qmphelpers.pm] Subject: Re: [pve-devel] [PATCH ve-rs/firewall/qemu-server/manager v2 00/13] fix #5180: migrate conntrack state on live migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Cc: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> On Tue Jun 3, 2025 at 11:35 AM CEST, Stefan Hanreich wrote: > Gave this a spin on my test cluster, and it worked as advertised. As > already mentioned and talked off-list we should imo add documentation > for this setting and mention the current restrictions (no migration of > NAT entries). We might also want to mention that only CT entries created > *after* the firewall is enabled for a VM get migrated, although that is > probably rather uncommon. Thanks for the (pretty extensive, as also talked off-list) testing! I'll definitely add documentation in the next revision, mentioning especially all the limitations. > > Other than that consider this: > > Tested-by: Stefan Hanreich <s.hanreich@proxmox.com> > > I've also mainly looked at the firewall + pve-manager code, maybe > someone with more knowledge about QEMU can chime in on the DBus server > parts. Consider the firewall and ui patches: > > Reviewed-by: Stefan Hanreich <s.hanreich@proxmox.com> > > > On 4/24/25 13:19, Christoph Heiss wrote: >> Fixes #5180 [0]. >> >> This implements migration of per-VM conntrack state on live-migration. >> >> The core of the implementation are in patch #7 & #8. See there for more >> details. >> >> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique >> VMID. This is needed later on to filter conntrack entries for the >> migration. These three patches can be applied independently, >> CONNMARK'ing traffic does not have any visible impact. >> >> Currently, remote/inter-cluster migration is not supported and indicated >> to the user with a warning. See also patch #8 for a bit more in-depth >> explanation. >> >> Needed dependency bumps between packages are indicated in the notes >> appropriately. >> >> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180 >> >> Testing >> ======= >> >> I've primarily tested intra-cluster live-migrations, with both the >> iptables-based and nftables-based firewall), using the reproducer as >> described in #5180. I further verified that the D-Bus servers get >> started as expected and are _always_ stopped, even in the case of some >> migration error. >> >> Finally, I also checked using `conntrack -L -m <vmid>` tool that the >> conntrack entries are >> a) added/updated on the target node and >> b) removed from the source node afterwards >> >> Also tested was the migration from/to an "old" (unpatched) node, which >> results in the issue as per #5180 & appropriate warnings in the UI. >> >> For remote migrations, only tested that the warning is logged as >> expected. >> >> History >> ======= >> >> v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/ >> >> Changes v1 -> v2: >> * rebased as necessary >> * "un-rfc'd" firewall conntrack flushing patches >> * use an instanced systemd service instead of fork+exec for the >> pve-dbus-vmstate helper >> >> Diffstat >> ======== >> >> pve-firewall: >> >> Christoph Heiss (2): >> firewall: add connmark rule with VMID to all guest chains >> firewall: helpers: add sub for flushing conntrack entries by mark >> >> debian/control | 3 ++- >> src/PVE/Firewall.pm | 7 +++++-- >> src/PVE/Firewall/Helpers.pm | 11 +++++++++++ >> 3 files changed, 18 insertions(+), 3 deletions(-) >> >> proxmox-firewall: >> >> Christoph Heiss (1): >> firewall: add connmark rule with VMID to all guest chains >> >> proxmox-firewall/src/firewall.rs | 14 +++- >> .../integration_tests__firewall.snap | 84 +++++++++++++++++++ >> proxmox-nftables/src/expression.rs | 9 ++ >> proxmox-nftables/src/statement.rs | 10 ++- >> 4 files changed, 114 insertions(+), 3 deletions(-) >> >> proxmox-ve-rs: >> >> Christoph Heiss (1): >> config: guest: allow access to raw Vmid value >> >> proxmox-ve-config/src/guest/types.rs | 4 ++++ >> 1 file changed, 4 insertions(+) >> >> qemu-server: >> >> Christoph Heiss (5): >> qmp helpers: allow passing structured args via qemu_objectadd() >> api2: qemu: add module exposing node migration capabilities >> fix #5180: libexec: add QEMU dbus-vmstate daemon for migrating >> conntrack >> fix #5180: migrate: integrate helper for live-migrating conntrack info >> migrate: flush old VM conntrack entries after successful migration >> >> Makefile | 7 +- >> PVE/API2/Qemu.pm | 72 +++++++++++ >> PVE/API2/Qemu/Makefile | 2 +- >> PVE/API2/Qemu/Migration.pm | 46 +++++++ >> PVE/CLI/qm.pm | 5 + >> PVE/QemuMigrate.pm | 69 ++++++++++ >> PVE/QemuServer.pm | 6 + >> PVE/QemuServer/DBusVMState.pm | 120 ++++++++++++++++++ >> PVE/QemuServer/Makefile | 1 + >> PVE/QemuServer/QMPHelpers.pm | 4 +- >> dbus-vmstate/Makefile | 7 ++ >> dbus-vmstate/dbus-vmstate | 168 +++++++++++++++++++++++++ >> dbus-vmstate/org.qemu.VMState1.conf | 11 ++ >> dbus-vmstate/pve-dbus-vmstate@.service | 10 ++ >> debian/control | 7 +- >> 15 files changed, 530 insertions(+), 5 deletions(-) >> create mode 100644 PVE/API2/Qemu/Migration.pm >> create mode 100644 PVE/QemuServer/DBusVMState.pm >> create mode 100644 dbus-vmstate/Makefile >> create mode 100755 dbus-vmstate/dbus-vmstate >> create mode 100644 dbus-vmstate/org.qemu.VMState1.conf >> create mode 100644 dbus-vmstate/pve-dbus-vmstate@.service >> >> pve-manager: >> >> Christoph Heiss (4): >> api2: capabilities: explicitly import CPU capabilities module >> api2: capabilities: proxy index endpoints to respective nodes >> api2: capabilities: expose new qemu/migration endpoint >> ui: window: Migrate: add checkbox for migrating VM conntrack state >> >> PVE/API2/Capabilities.pm | 9 +++++ >> www/manager6/window/Migrate.js | 73 ++++++++++++++++++++++++++++++++-- >> 2 files changed, 78 insertions(+), 4 deletions(-) >> _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel