From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A47931FF0AE for ; Tue, 13 Oct 2026 14:22:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5F1AF21569; Tue, 13 Oct 2026 14:22:37 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 smtp.infra.comaite.com 1EF9162C11 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comaite.com; s=377A38C6-3A8D-11E5-A0EC-E97757B27CB3; t=1791893674; bh=jbWJ4phEEocgrOBkpJbZrxpGmkZjv7JPbmPO6Fjnogo=; h=From:To:Date:Message-ID:MIME-Version; b=twZmfIDoQja3pbecavCO+B9bkE3relq4zzdMY2t5tW/HyRlQtvRTHfFPIObofQ7kd pB5LNyDbAM+cRrwlGOaOHPlwp4jIiNRt0z2Hgzv8kSpuSq3uC9duV+mBdU5AYcTpEh 6ADF9m8cTo9kQZap4C4ywf8Fxk4gL0jgG2oV27HZAMr9Y67hj2QO/m/yS0qN9ZAy2f aGsze0JPQ27RO0Oiomvj25X8VhQo/3IelQEVyLeijldJGjytCd0X7jFE7XMMmZRl5v 14nGAoIQWBx6ghihVJvZUbXKI1PI4/1glIm9llVbe4hheecIpYPUSuQs4Ok7msvvGO GjxJvDbXGGoSg== From: =?UTF-8?q?R=C3=A9mi=20PAETA?= To: pve-devel@lists.proxmox.com Subject: [RFC PATCH network 10/16] sdn: ovn: bind guest ports to the chassis of their node Date: Tue, 13 Oct 2026 14:14:26 +0200 Message-ID: <20261013121432.2642979-11-remi.paeta@comaite.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261013121432.2642979-1-remi.paeta@comaite.com> References: <1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com> <20261013121432.2642979-1-remi.paeta@comaite.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: G4YRHM6CXGYUGSIORFW3JI7Q3B2Q6JAC X-Message-ID-Hash: G4YRHM6CXGYUGSIORFW3JI7Q3B2Q6JAC X-MailFrom: remi.paeta@comaite.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: During a live migration, QEMU already runs on the destination node while the memory gets copied, so both ovn-controller instances see an interface for the logical switch port and keep claiming it from each other, every 500 ms, until the source releases it. Traffic goes to the wrong node in the meantime: 2 to 3 s of loss were measured, while QEMU reported a downtime of about 20 ms. Set the requested-chassis option of guest ports to the local node when plugging them. On the destination of a live migration, set it to both nodes with activation-strategy=rarp instead, as recommended by OVN for this case: the destination becomes an additional chassis and only gets the traffic once the guest announces itself there. qemu-server calls add_bridge_fdb on the destination when resuming the guest, which binds the port to the destination only. requested-chassis refers to the chassis hostname, which Open vSwitch sets to the FQDN, so set it to the node name when configuring the chassis. With this, the loss went down to 0.4 to 0.8 s, close to the resolution of the probe used. Signed-off-by: Rémi PAETA --- src/PVE/Network/SDN/Ovn.pm | 61 ++++++++++++++++++++++---- src/PVE/Network/SDN/Zones/OvnPlugin.pm | 16 +++++-- src/test/run_test_ovn.pl | 52 +++++++++++++++++++++- 3 files changed, 116 insertions(+), 13 deletions(-) diff --git a/src/PVE/Network/SDN/Ovn.pm b/src/PVE/Network/SDN/Ovn.pm index ad9dda3..646f9b6 100644 --- a/src/PVE/Network/SDN/Ovn.pm +++ b/src/PVE/Network/SDN/Ovn.pm @@ -43,7 +43,7 @@ my $EXT_VMID = 'pve-sdn-vmid'; my $EXT_MAC = 'pve-sdn-mac'; # chassis settings written to the local Open_vSwitch table -my @CHASSIS_KEYS = ('ovn-remote', 'ovn-encap-type', 'ovn-encap-ip', $EXT_CONTROLLER); +my @CHASSIS_KEYS = ('ovn-remote', 'ovn-encap-type', 'ovn-encap-ip', 'hostname', $EXT_CONTROLLER); =head3 ovn_controllers(\%running_cfg) @@ -618,16 +618,52 @@ sub port_vnet { return $output eq '' ? undef : $output; } -=head3 update_port($controller, $iface, $level, \%running_cfg, $vnetid, $mac) +=head3 chassis_commands($iface, $nodename, $migrated_from) + +Returns the C commands binding the logical switch port C<$iface> to +the chassis of the node C<$nodename>, so that no other chassis claims it. + +During a live migration from the node C<$migrated_from>, the port is bound to +both chassis, the source one staying the main one. The destination chassis only +gets the traffic of the port once the guest announces itself there (QEMU sends a +RARP after the switchover). Otherwise, both chassis keep claiming the port while +the memory gets copied, and traffic goes to the wrong node for seconds. + +=cut + +sub chassis_commands { + my ($iface, $nodename, $migrated_from) = @_; + + if (defined($migrated_from)) { + return [[ + 'set', + 'Logical_Switch_Port', + $iface, + "options:requested-chassis=$migrated_from,$nodename", + 'options:activation-strategy=rarp', + ]]; + } + + return [ + ['set', 'Logical_Switch_Port', $iface, "options:requested-chassis=$nodename"], + ['remove', 'Logical_Switch_Port', $iface, 'options', 'activation-strategy'], + ]; +} + +=head3 update_port($controller, $iface, $level, \%running_cfg, $vnetid, $mac, $nodename) Sets the guest MAC C<$mac> and its IPAM IPs on the logical switch port C<$iface> according to the port security C<$level>, and records the MAC so that later syncs can update the port. +Also binds the port to the chassis of C<$nodename> only. qemu-server calls this +on the destination node when resuming a guest after a live migration, which +ends the binding to both chassis set up by C. + =cut sub update_port { - my ($controller, $iface, $level, $running_cfg, $vnetid, $mac) = @_; + my ($controller, $iface, $level, $running_cfg, $vnetid, $mac, $nodename) = @_; # Without an answer from the IPAM, only the MAC can be enforced: the guest # keeps its connectivity, and the next sync applies the IPs. @@ -640,19 +676,21 @@ sub update_port { port_addresses_commands($iface, $addresses, $port_security)->@*, # as written in the guest config, which is what the IPAM is keyed by ['set', 'Logical_Switch_Port', $iface, "external_ids:$EXT_MAC=$mac"], + chassis_commands($iface, $nodename)->@*, ], ); } -=head3 ensure_port($controllerid, $controller, $zoneid, $vnetid, $iface) +=head3 ensure_port($controllerid, $controller, $zoneid, $vnetid, $iface, $nodename, $migrated_from) Creates or updates the logical switch port of the guest interface C<$iface> on -the logical switch of C<$vnetid>, moving it over from another switch if needed. +the logical switch of C<$vnetid>, moving it over from another switch if needed, +and binds it to the chassis of C<$nodename>, see C. =cut sub ensure_port { - my ($controllerid, $controller, $zoneid, $vnetid, $iface) = @_; + my ($controllerid, $controller, $zoneid, $vnetid, $iface, $nodename, $migrated_from) = @_; eval { my $current_switch; @@ -672,6 +710,7 @@ sub ensure_port { my $external_ids = port_external_ids($controllerid, $zoneid, $vnetid, $iface); my $commands = port_commands($vnetid, $iface, $external_ids, $current_switch); + push @$commands, chassis_commands($iface, $nodename, $migrated_from)->@*; nbctl_transaction($controller, $commands); }; die "unable to create OVN logical switch port '$iface' on '$vnetid' - $@" if $@; @@ -765,20 +804,24 @@ sub parse_remote_ip { die "unable to parse OVN database connection '$first'\n"; } -=head3 chassis_settings($controllerid, $controller, $encap_ip) +=head3 chassis_settings($controllerid, $controller, $encap_ip, $nodename) Returns the external_ids to set in the local Open_vSwitch table so that ovn-controller joins the OVN deployment of C<$controller>. +The chassis hostname is set to the node name, which logical switch ports refer +to in their requested-chassis option. Open vSwitch would use the FQDN instead. + =cut sub chassis_settings { - my ($controllerid, $controller, $encap_ip) = @_; + my ($controllerid, $controller, $encap_ip, $nodename) = @_; return { 'ovn-remote' => $controller->{'sb-connection'}, 'ovn-encap-type' => $controller->{'encap-type'} // 'geneve', 'ovn-encap-ip' => $encap_ip, + hostname => $nodename, $EXT_CONTROLLER => $controllerid, }; } @@ -844,7 +887,7 @@ sub configure_chassis { uses_ssl($controller->{'sb-connection'}) ? $id : undef, $nodename, ); - my $settings = chassis_settings($id, $controller, $encap_ip); + my $settings = chassis_settings($id, $controller, $encap_ip, $nodename); run_command([ $VSCTL, 'set', diff --git a/src/PVE/Network/SDN/Zones/OvnPlugin.pm b/src/PVE/Network/SDN/Zones/OvnPlugin.pm index a474dce..861059c 100644 --- a/src/PVE/Network/SDN/Zones/OvnPlugin.pm +++ b/src/PVE/Network/SDN/Zones/OvnPlugin.pm @@ -3,6 +3,7 @@ package PVE::Network::SDN::Zones::OvnPlugin; use v5.36; use PVE::Exception qw(raise_param_exc); +use PVE::INotify; use PVE::Network; use PVE::Network::SDN::Ovn; @@ -99,8 +100,15 @@ sub tap_plug { die "OVN controller '$controllerid' of vnet '$vnetid' is not applied yet\n" if !$controller; + # set by qemu-server on the destination node of a live migration PVE::Network::SDN::Ovn::ensure_port( - $controllerid, $controller, $vnet->{zone}, $vnetid, $iface, + $controllerid, + $controller, + $vnet->{zone}, + $vnetid, + $iface, + PVE::INotify::nodename(), + $ENV{PVE_MIGRATED_FROM} || undef, ); # detach from any previous bridge @@ -112,8 +120,9 @@ sub tap_plug { die "VLAN tags and trunks are not supported on OVN vnet '$vnetid'\n" if $tag || $trunks; } -# Called by qemu-server on guest start and by pve-container on plug, with the -# guest MAC, which tap_plug does not get. +# Called by qemu-server on guest start and resume, including after a live +# migration, and by pve-container on plug, with the guest MAC, which tap_plug +# does not get. sub add_bridge_fdb { my ($class, $plugin_config, $iface, $macaddr) = @_; @@ -138,6 +147,7 @@ sub add_bridge_fdb { $running_config, $vnetid, $macaddr, + PVE::INotify::nodename(), ); }; warn "unable to set the addresses of OVN logical switch port '$iface' - $@" if $@; diff --git a/src/test/run_test_ovn.pl b/src/test/run_test_ovn.pl index 83bfd80..0daaed0 100755 --- a/src/test/run_test_ovn.pl +++ b/src/test/run_test_ovn.pl @@ -270,6 +270,53 @@ is_deeply( 'port_commands: port moved to another vnet', ); +is_deeply( + PVE::Network::SDN::Ovn::chassis_commands('tap100i0', 'node1'), + [ + ['set', 'Logical_Switch_Port', 'tap100i0', 'options:requested-chassis=node1'], + ['remove', 'Logical_Switch_Port', 'tap100i0', 'options', 'activation-strategy'], + ], + 'chassis_commands: bound to the local chassis', +); +is_deeply( + PVE::Network::SDN::Ovn::chassis_commands('tap100i0', 'node2', 'node1'), + [[ + 'set', + 'Logical_Switch_Port', + 'tap100i0', + 'options:requested-chassis=node1,node2', + 'options:activation-strategy=rarp', + ]], + 'chassis_commands: live migration, bound to both chassis until the guest announces itself', +); + +{ + my $transactions = []; + no warnings 'redefine'; + local *PVE::Network::SDN::Ovn::nbctl = sub { return '' }; + local *PVE::Network::SDN::Ovn::nbctl_transaction = sub { push @$transactions, $_[1] }; + + PVE::Network::SDN::Ovn::ensure_port( + 'ovnctl', $controller, 'zone1', 'vnet1', 'tap100i0', 'node2', 'node1', + ); + is_deeply( + $transactions->[0]->[-1], + PVE::Network::SDN::Ovn::chassis_commands('tap100i0', 'node2', 'node1')->[0], + 'ensure_port: binds the port to both chassis on an incoming migration', + ); + + $transactions = []; + local *PVE::Network::SDN::Ovn::guest_ips = sub { return [] }; + PVE::Network::SDN::Ovn::update_port( + $controller, 'tap100i0', 'off', {}, 'vnet1', 'BC:24:11:00:00:01', 'node2', + ); + is_deeply( + [$transactions->[0]->@[-2, -1]], + PVE::Network::SDN::Ovn::chassis_commands('tap100i0', 'node2'), + 'update_port: binds the port to the local chassis only', + ); +} + { no warnings 'redefine'; local *PVE::Network::SDN::Ovn::run_command = sub { @@ -548,11 +595,12 @@ is_deeply( ); is_deeply( - PVE::Network::SDN::Ovn::chassis_settings('ovnctl', $controller, '192.168.0.10'), + PVE::Network::SDN::Ovn::chassis_settings('ovnctl', $controller, '192.168.0.10', 'node1'), { 'ovn-remote' => $controller->{'sb-connection'}, 'ovn-encap-type' => 'geneve', 'ovn-encap-ip' => '192.168.0.10', + hostname => 'node1', 'pve-sdn-controller' => 'ovnctl', }, 'chassis_settings', @@ -584,6 +632,7 @@ is_deeply( 'set', 'Open_vSwitch', '.', + 'external_ids:hostname=node1', 'external_ids:ovn-encap-ip=192.168.0.10', 'external_ids:ovn-encap-type=geneve', "external_ids:ovn-remote=$controller->{'sb-connection'}", @@ -608,6 +657,7 @@ is_deeply( 'ovn-remote', 'ovn-encap-type', 'ovn-encap-ip', + 'hostname', 'pve-sdn-controller', ]], 'configure_chassis: detach chassis', -- 2.47.3