From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 9799C1FF15E for ; Tue, 3 Jun 2025 11:35:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 679AA14E3C; Tue, 3 Jun 2025 11:35:56 +0200 (CEST) Message-ID: <902b7cc1-8962-4bf6-be1e-a996f24798e3@proxmox.com> Date: Tue, 3 Jun 2025 11:35:53 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Proxmox VE development discussion , Christoph Heiss References: <20250424111941.730528-1-c.heiss@proxmox.com> Content-Language: en-US From: Stefan Hanreich In-Reply-To: <20250424111941.730528-1-c.heiss@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.683 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemumigrate.pm, dbusvmstate.pm, qemu.pm, dbus-vmstate.com, qmphelpers.pm, capabilities.pm, expression.rs, statement.rs, qm.pm] Subject: Re: [pve-devel] [PATCH ve-rs/firewall/qemu-server/manager v2 00/13] fix #5180: migrate conntrack state on live migration X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Gave this a spin on my test cluster, and it worked as advertised. As already mentioned and talked off-list we should imo add documentation for this setting and mention the current restrictions (no migration of NAT entries). We might also want to mention that only CT entries created *after* the firewall is enabled for a VM get migrated, although that is probably rather uncommon. Other than that consider this: Tested-by: Stefan Hanreich I've also mainly looked at the firewall + pve-manager code, maybe someone with more knowledge about QEMU can chime in on the DBus server parts. Consider the firewall and ui patches: Reviewed-by: Stefan Hanreich On 4/24/25 13:19, Christoph Heiss wrote: > Fixes #5180 [0]. > > This implements migration of per-VM conntrack state on live-migration. > > The core of the implementation are in patch #7 & #8. See there for more > details. > > Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique > VMID. This is needed later on to filter conntrack entries for the > migration. These three patches can be applied independently, > CONNMARK'ing traffic does not have any visible impact. > > Currently, remote/inter-cluster migration is not supported and indicated > to the user with a warning. See also patch #8 for a bit more in-depth > explanation. > > Needed dependency bumps between packages are indicated in the notes > appropriately. > > [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180 > > Testing > ======= > > I've primarily tested intra-cluster live-migrations, with both the > iptables-based and nftables-based firewall), using the reproducer as > described in #5180. I further verified that the D-Bus servers get > started as expected and are _always_ stopped, even in the case of some > migration error. > > Finally, I also checked using `conntrack -L -m ` tool that the > conntrack entries are > a) added/updated on the target node and > b) removed from the source node afterwards > > Also tested was the migration from/to an "old" (unpatched) node, which > results in the issue as per #5180 & appropriate warnings in the UI. > > For remote migrations, only tested that the warning is logged as > expected. > > History > ======= > > v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/ > > Changes v1 -> v2: > * rebased as necessary > * "un-rfc'd" firewall conntrack flushing patches > * use an instanced systemd service instead of fork+exec for the > pve-dbus-vmstate helper > > Diffstat > ======== > > pve-firewall: > > Christoph Heiss (2): > firewall: add connmark rule with VMID to all guest chains > firewall: helpers: add sub for flushing conntrack entries by mark > > debian/control | 3 ++- > src/PVE/Firewall.pm | 7 +++++-- > src/PVE/Firewall/Helpers.pm | 11 +++++++++++ > 3 files changed, 18 insertions(+), 3 deletions(-) > > proxmox-firewall: > > Christoph Heiss (1): > firewall: add connmark rule with VMID to all guest chains > > proxmox-firewall/src/firewall.rs | 14 +++- > .../integration_tests__firewall.snap | 84 +++++++++++++++++++ > proxmox-nftables/src/expression.rs | 9 ++ > proxmox-nftables/src/statement.rs | 10 ++- > 4 files changed, 114 insertions(+), 3 deletions(-) > > proxmox-ve-rs: > > Christoph Heiss (1): > config: guest: allow access to raw Vmid value > > proxmox-ve-config/src/guest/types.rs | 4 ++++ > 1 file changed, 4 insertions(+) > > qemu-server: > > Christoph Heiss (5): > qmp helpers: allow passing structured args via qemu_objectadd() > api2: qemu: add module exposing node migration capabilities > fix #5180: libexec: add QEMU dbus-vmstate daemon for migrating > conntrack > fix #5180: migrate: integrate helper for live-migrating conntrack info > migrate: flush old VM conntrack entries after successful migration > > Makefile | 7 +- > PVE/API2/Qemu.pm | 72 +++++++++++ > PVE/API2/Qemu/Makefile | 2 +- > PVE/API2/Qemu/Migration.pm | 46 +++++++ > PVE/CLI/qm.pm | 5 + > PVE/QemuMigrate.pm | 69 ++++++++++ > PVE/QemuServer.pm | 6 + > PVE/QemuServer/DBusVMState.pm | 120 ++++++++++++++++++ > PVE/QemuServer/Makefile | 1 + > PVE/QemuServer/QMPHelpers.pm | 4 +- > dbus-vmstate/Makefile | 7 ++ > dbus-vmstate/dbus-vmstate | 168 +++++++++++++++++++++++++ > dbus-vmstate/org.qemu.VMState1.conf | 11 ++ > dbus-vmstate/pve-dbus-vmstate@.service | 10 ++ > debian/control | 7 +- > 15 files changed, 530 insertions(+), 5 deletions(-) > create mode 100644 PVE/API2/Qemu/Migration.pm > create mode 100644 PVE/QemuServer/DBusVMState.pm > create mode 100644 dbus-vmstate/Makefile > create mode 100755 dbus-vmstate/dbus-vmstate > create mode 100644 dbus-vmstate/org.qemu.VMState1.conf > create mode 100644 dbus-vmstate/pve-dbus-vmstate@.service > > pve-manager: > > Christoph Heiss (4): > api2: capabilities: explicitly import CPU capabilities module > api2: capabilities: proxy index endpoints to respective nodes > api2: capabilities: expose new qemu/migration endpoint > ui: window: Migrate: add checkbox for migrating VM conntrack state > > PVE/API2/Capabilities.pm | 9 +++++ > www/manager6/window/Migrate.js | 73 ++++++++++++++++++++++++++++++++-- > 2 files changed, 78 insertions(+), 4 deletions(-) > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel