Re: [pve-devel] [PATCH ve-rs/firewall/qemu-server/manager/docs v3 00/14] fix #5180: migrate conntrack state on live migration

From: "Christoph Heiss" <c.heiss@proxmox.com>
To: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH ve-rs/firewall/qemu-server/manager/docs v3 00/14] fix #5180: migrate conntrack state on live migration
Date: Thu, 17 Jul 2025 16:16:49 +0200	[thread overview]
Message-ID: <DBEE7FEV8YVD.1F5ZRQDBJ2HIK@proxmox.com> (raw)
In-Reply-To: <20250703115621.883244-1-c.heiss@proxmox.com>

v4 posted: https://lore.proxmox.com/pve-devel/20250717141530.1471199-1-c.heiss@proxmox.com/

On Thu Jul 3, 2025 at 1:54 PM CEST, Christoph Heiss wrote:
> Fixes #5180 [0].
>
> This implements migration of per-VM conntrack state on live-migration.
>
> The core of the implementation are in patch #7 & #8. See there for more
> details.
>
> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique
> VMID. This is needed later on to filter conntrack entries for the
> migration. These three patches can be applied independently,
> CONNMARK'ing traffic does not have any visible impact.
>
> Regarding the use of connlabels instead, as was discussed in v2:
> Connlabels are (unfortunately) pure bitmasks and cannot be used for
> storing arbitrary values such as VMIDs. Thus we are basically stuck with
> using connmarks.
>
> Currently, remote/inter-cluster migration is not supported and indicated
> to the user with a warning. See also patch #8 for a bit more in-depth
> explanation.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180
>
> Dependencies
> ============
>
> proxmox-firewall depends on the proxmox-ve-rs changes.
> qemu-server depends on the pve-firewall/proxmox-firewall changes.
>
> pve-manager only soft-depends on the other, as it will detect whether
> conntrack migration is supported.
>
> Testing
> =======
>
> I've primarily tested intra-cluster live-migrations, with both the
> iptables-based and nftables-based firewall), using the reproducer as
> described in #5180. I further verified that the D-Bus services get
> started as expected and are _always_ stopped, even in the case of some
> migration error.
>
> Finally, I also checked using `conntrack -L -m <vmid>` tool that the
> conntrack entries are
> a) added/updated on the target node and
> b) removed from the source node afterwards
>
> Also tested was the migration from/to an "old" (unpatched) node, which
> results in the issue as per #5180 & appropriate warnings in the UI.
>
> For remote migrations, tested that the warning is logged as expected.
>
> History
> =======
>
> v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/
> v2: https://lore.proxmox.com/pve-devel/20250424111941.730528-1-c.heiss@proxmox.com/
>
> Changes v1 -> v2:
>   * rebased as necessary
>   * "un-rfc'd" firewall conntrack flushing patches
>   * use an instanced systemd service instead of fork+exec for the
>     pve-dbus-vmstate helper
>
> Changes v2 -> v3:
>   * rebased on trixie/latest masters
>   * added documentation patch
>   * moved node capability module to
>     PVE::API2::NodeCapabilities::Qemu::Migration, based on Fiona's
>     suggestion
>
> Diffstat
> ========
>
> proxmox-ve-rs:
>
> Christoph Heiss (1):
>   config: guest: allow access to raw Vmid value
>
>  proxmox-ve-config/src/guest/types.rs | 4 ++++
>  1 file changed, 4 insertions(+)
>
> proxmox-firewall:
>
> Christoph Heiss (1):
>   firewall: add connmark rule with VMID to all guest chains
>
>  proxmox-firewall/src/firewall.rs              | 14 +++-
>  .../integration_tests__firewall.snap          | 84 +++++++++++++++++++
>  proxmox-nftables/src/expression.rs            |  9 ++
>  proxmox-nftables/src/statement.rs             | 10 ++-
>  4 files changed, 114 insertions(+), 3 deletions(-)
>
> pve-firewall:
>
> Christoph Heiss (2):
>   firewall: add connmark rule with VMID to all guest chains
>   firewall: helpers: add sub for flushing conntrack entries by mark
>
>  debian/control              |  3 ++-
>  src/PVE/Firewall.pm         | 14 ++++++++++++--
>  src/PVE/Firewall/Helpers.pm | 12 ++++++++++++
>  3 files changed, 26 insertions(+), 3 deletions(-)
>
> qemu-server:
>
> Christoph Heiss (5):
>   qmp helpers: allow passing structured args via qemu_objectadd()
>   api2: qemu: add module exposing node migration capabilities
>   fix #5180: dbus-vmstate: add daemon for QEMUs dbus-vmstate interface
>   fix #5180: migrate: integrate helper for live-migrating conntrack info
>   migrate: flush old VM conntrack entries after successful migration
>
>  Makefile                                      |   4 +-
>  debian/control                                |   7 +-
>  src/Makefile                                  |   1 +
>  src/PVE/API2/Makefile                         |   1 +
>  src/PVE/API2/NodeCapabilities/Makefile        |   9 +
>  .../API2/NodeCapabilities/Qemu/Migration.pm   |  48 +++++
>  src/PVE/API2/Qemu.pm                          |  75 ++++++++
>  src/PVE/CLI/qm.pm                             |   5 +
>  src/PVE/QemuMigrate.pm                        |  78 ++++++++
>  src/PVE/QemuServer.pm                         |   6 +
>  src/PVE/QemuServer/DBusVMState.pm             | 125 +++++++++++++
>  src/PVE/QemuServer/Makefile                   |   1 +
>  src/PVE/QemuServer/QMPHelpers.pm              |   4 +-
>  src/dbus-vmstate/Makefile                     |  11 ++
>  src/dbus-vmstate/dbus-vmstate                 | 168 ++++++++++++++++++
>  src/dbus-vmstate/org.qemu.VMState1.conf       |  11 ++
>  src/dbus-vmstate/pve-dbus-vmstate@.service    |  10 ++
>  17 files changed, 560 insertions(+), 4 deletions(-)
>  create mode 100644 src/PVE/API2/NodeCapabilities/Makefile
>  create mode 100644 src/PVE/API2/NodeCapabilities/Qemu/Migration.pm
>  create mode 100644 src/PVE/QemuServer/DBusVMState.pm
>  create mode 100644 src/dbus-vmstate/Makefile
>  create mode 100755 src/dbus-vmstate/dbus-vmstate
>  create mode 100644 src/dbus-vmstate/org.qemu.VMState1.conf
>  create mode 100644 src/dbus-vmstate/pve-dbus-vmstate@.service
>
> pve-manager:
>
> Christoph Heiss (4):
>   api2: capabilities: explicitly import CPU capabilities module
>   api2: capabilities: proxy index endpoints to respective nodes
>   api2: capabilities: expose new qemu/migration endpoint
>   ui: window: Migrate: add checkbox for migrating VM conntrack state
>
>  PVE/API2/Capabilities.pm       | 11 ++++-
>  www/manager6/window/Migrate.js | 82 ++++++++++++++++++++++++++++++++--
>  2 files changed, 89 insertions(+), 4 deletions(-)
>
> pve-docs:
>
> Christoph Heiss (1):
>   qm: document conntrack state migration for live migrations
>
>  qm.adoc | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel