From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 8B2051FF0AE for ; Tue, 13 Oct 2026 14:23:05 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B5CBD21589; Tue, 13 Oct 2026 14:22:37 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 smtp.infra.comaite.com 9530162BEA DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comaite.com; s=377A38C6-3A8D-11E5-A0EC-E97757B27CB3; t=1791893672; bh=/twVE0tuGdk+Iu89L01G9udo4JgUnN4bFhOAV6j8aiY=; h=From:To:Date:Message-ID:MIME-Version; b=SN0eWdAaD1hmor1vkx2V8TUhGDdPJNciRKXG0RUJdlbX5vHh9++okV5ILXH3NxwQa NDl9j0jSyRYdlyXg77HLmSZdOxGmILUeXHoTZp24P25IpdqL2Sygec5mCU9Fy9eSwX GUFY4FL9PwXVqQEaYmUMDMSQl4Avvc8hDo72hfkkiHDzqpPjjkg3ZeUWZ/balshpLe 9R12bSX/WNf/tTcMaUgUCEACv5DNMkMNJDA0neiw8UIFUozEtugPoi4VjvAHpzYGXS UYsYxUiS51VmcbNXlZvV8lkF3bQksb8URUSWWOiSucr9g8uSlrzV3KFYYO6MwJZ1Lb PlL5NecLHC2rA== From: =?UTF-8?q?R=C3=A9mi=20PAETA?= To: pve-devel@lists.proxmox.com Subject: [RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone and controller plugins Date: Tue, 13 Oct 2026 14:14:16 +0200 Message-ID: <20261013121432.2642979-1-remi.paeta@comaite.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com> References: <1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 2 AWL -2.500 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_DBL_SPAM 5 Contains a spam URL listed in the Spamhaus DBL blocklist [sdn.pm] Message-ID-Hash: XJDWFW7XXGJCW4UBHGXXQCQ3J5TV6G5F X-Message-ID-Hash: XJDWFW7XXGJCW4UBHGXXQCQ3J5TV6G5F X-MailFrom: remi.paeta@comaite.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series adds OVN (Open Virtual Network) support to the SDN stack, as first discussed in: https://lore.proxmox.com/pve-devel/1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com/ It covers the first step outlined there: an OVN controller, OVN zones and vnets, providing a distributed L2 overlay with optional per-port anti-spoofing. Routing (subnet gateways, SNAT), native DHCP and OVN ACLs are left for later steps. Sent as RFC, as some design choices below would benefit from your feedback before going further. I am contributing on behalf of my employer, Comaite, whose entity CLA has been accepted by Proxmox. Overview ======== - The OVN central components (northbound/southbound databases, ovn-northd) are deployed by the administrator, ideally as a RAFT cluster, on the PVE nodes or elsewhere. An "ovn" SDN controller holds their connection strings, plain TCP or SSL. - Each vnet of an "ovn" zone is an OVN logical switch. On reload, the northbound database is synced once for the whole cluster: switches get created, updated and deleted, ports of removed guests and guest interfaces get cleaned up. Only objects tagged with pve-sdn-* external_ids are touched, so the deployment can be shared. - On every node of an OVN zone, ovn-controller gets configured (remote, encapsulation, tunnel IP from the route towards the southbound database). pvenetcommit.service removes the Open vSwitch database on every boot, so a oneshot pve-sdn-ovn-chassis.service sets it again before the guests start. - Guest interfaces are plugged into br-int and bound to a logical switch port named after them. Each vnet still gets an empty OVS bridge, which lets qemu-server and pve-container read its MTU and set up the firewall bridge like for any OVS bridge, so the guest firewall works unchanged. pve-manager needs no backend change. - Port security, per zone with a per-vnet override: off (default), mac, or mac-ip with the IPAM addresses (IPv4 and IPv6). The guest MAC is set from add_bridge_fdb. A changed level applies on reload, without restarting guests. - With SSL, the administrator provides the CA and one certificate per node in /etc/pve/priv/sdn/ovn//. ovn-controller only takes certificates on its command line, so a runtime drop-in for its unit adds them; it is only restarted when that setup changes. - Guest ports are bound to the chassis of their node (requested-chassis option). On live migration, they are bound to both nodes with activation-strategy=rarp, and back to the destination only when qemu-server calls add_bridge_fdb on resume, see below. Changes since the discussion mail ================================= - Port security can be overridden per vnet, which adds a 'port-security' option to the common vnet schema (refused on other zone types, see the sixth network patch). - The web UI is now part of the series (pve-manager patches). - The proxmox-ve-rs patch is needed for both firewalls: without it, proxmox-firewall fails on the whole SDN configuration, and pve-firewall, through libpve-rs-perl, generates no SDN IPSet at all, for any zone, as soon as an OVN zone exists. My first mail only mentioned the nftables firewall. - SSL connections to the OVN databases. - Live migration: while QEMU runs on both nodes during the memory copy, both ovn-controller instances kept claiming the port from each other, every 500 ms, until the source released it. Measured with a probe every ~55 ms, this lost 2 to 3 s of traffic, while QEMU reported a downtime of ~20 ms. Binding the port to both chassis with activation-strategy=rarp during the migration, as OVN recommends for this case, and to the destination only on resume (qemu-server already calls add_bridge_fdb there) brings it down to 0.4 to 0.8 s, close to the resolution of the probe. This needs no change outside pve-network; the chassis hostname is set to the node name, as requested-chassis refers to it and Open vSwitch uses the FQDN. Dependencies ============ - proxmox-ve-rs (first patch): libpve-rs-perl and proxmox-firewall need to be rebuilt with it before OVN zones can be used, so pve-network will likely need Breaks on older versions of both. How would you prefer to handle this? - pve-manager depends on a libpve-network-perl knowing the vnet 'port-security' option. - Hotplugged NICs of VMs only get their MAC and port security on the next start, as qemu-server does not call add_bridge_fdb on hotplug. This is the existing bug #6490 for bridges with MAC learning disabled; I sent a separate fix for it: https://lore.proxmox.com/pve-devel/20261013095852.2586609-1-remi.paeta@comaite.com/ Known limitations ================= - No routing, SNAT or DHCP yet: subnet gateways, SNAT and DHCP ranges are refused on OVN zones, as are VLAN tags, VLAN-aware vnets and isolated ports. - With mac-ip, IPv6 link-local traffic is only allowed from the address derived from the MAC, which breaks guests using stable privacy link-local addresses for link-local protocols. - If the IPAM cannot be queried, a reload keeps the current port addresses, while a starting guest only gets its MAC enforced. - If the northbound database cannot be reached, running guests keep their connectivity, but guests on OVN zones cannot be started. Testing ======= Unit tests cover the new code (332 tests in pve-network). On a three node cluster with the OVN databases clustered on the nodes, over SSL: - VM and container connectivity across nodes, live migration (0.4 to 0.8 s of loss, QEMU downtime ~20 ms), container migration - guest firewall with iptables and nftables, ipfilter, macfilter, firewall toggling on running guests - port security levels with IPv4 and IPv6, spoofed MAC and IPs blocked - northbound/southbound leader failover and full outage of all database members (no loss for running guests) - node crash, HA recovery and port claim on another node with a stale binding of the dead node - Geneve to VXLAN encapsulation switch and back - zone restricted to nodes, removal of zones and of the controller - orphaned ports after removing or moving guest interfaces Design notes ============ - The OVN central components are deployed and operated separately, by the administrator. Deploying and managing them directly on the Proxmox VE cluster (for example as a RAFT cluster on some of the nodes, set up from the API) could be a later evolution. - The guest start is refused while the northbound database cannot be reached. Starting the guest without its port would not help: the southbound database usually runs on the same hosts and is down too, so no chassis can claim a port, and a guest started elsewhere by HA could not be claimed anyway, as requested-chassis still names the previous node and only the northbound database can change it. - With mac-ip and an unreachable IPAM, a starting guest only gets its MAC enforced (fail-open). The IPs are looked up through Ipams::get_ips_from_mac, so the MAC cache answers for known guests, and this only happens for a MAC the IPAM never returned before. Zones with DHCP fail the guest start in that case instead (through add_dhcp_mapping); I kept fail-open here so that an IPAM outage does not prevent starting guests, but can change it if you prefer. - Certificates are left to the PKI of the OVN deployment (renewal, expiry). An API/GUI import could come later, ideally with the key generated on each node and only the signed certificate imported. Open questions ============== 1. Each vnet currently gets an empty OVS bridge, while the guest interfaces are plugged into br-int. It only exists because qemu-server reads the vnet MTU from /sys/class/net//mtu (read_bridge_mtu, which also checks the bridges on migration targets), and to keep the firewall bridge setup of OVS bridges. This needs no change outside pve-network. Is this acceptable, or would you prefer one of these alternatives? a) An SDN dispatcher, for example PVE::Network::SDN::Zones::get_bridge_mtu(), returning the zone MTU for OVN vnets and reading sysfs otherwise, called by qemu-server and pve-container instead of read_bridge_mtu(). No placeholder interface anymore, but it touches both repositories (pve-common cannot depend on pve-network), and every place assuming that a vnet is a local interface would need the same treatment. b) A lighter placeholder: a dummy interface named after the vnet instead of an OVS bridge. Still no change outside pve-network, and nothing in the Open vSwitch database, but the firewall bridge setup would need to be checked again. 2. Is extending the common vnet schema with 'port-security' fine? As vnets have a single section type, I saw no other place for a per-vnet option. 3. VLAN-aware vnets are out of scope for this series. Guest-tagged trunks could later use the vlan-passthru option of logical switches, while per-NIC VLAN tags would need one logical switch per VLAN. Is either of interest? 4. Not specific to OVN, but noticed while testing it: deleting a vnet without subnets is not refused while guests still use it, on any zone type. Running guests end up unplugged and fail on their next start. Would you take a separate patch refusing the deletion with the list of guests using the vnet? ve-rs: proxmox-ve-config/src/sdn/config.rs | 3 +++ proxmox-ve-config/tests/sdn/main.rs | 28 ++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) network: debian/control | 1 + debian/libpve-network-perl.install | 1 + debian/libpve-network-perl.postinst | 2 + src/PVE/API2/Network/SDN.pm | 8 + src/PVE/API2/Network/SDN/Vnets.pm | 8 + src/PVE/Network/SDN.pm | 39 +- src/PVE/Network/SDN/Controllers.pm | 6 + src/PVE/Network/SDN/Controllers/Makefile | 2 +- src/PVE/Network/SDN/Controllers/OvnPlugin.pm | 90 ++ src/PVE/Network/SDN/Makefile | 3 +- src/PVE/Network/SDN/Ovn.pm | 1024 ++++++++++++++++++ src/PVE/Network/SDN/SubnetPlugin.pm | 9 + src/PVE/Network/SDN/VnetPlugin.pm | 8 + src/PVE/Network/SDN/Zones.pm | 2 + src/PVE/Network/SDN/Zones/Makefile | 2 +- src/PVE/Network/SDN/Zones/OvnPlugin.pm | 184 ++++ src/services/Makefile | 1 + src/services/pve-sdn-ovn-chassis.service | 16 + src/test/Makefile | 5 +- src/test/run_test_ovn.pl | 835 ++++++++++++++ src/test/run_test_vnets_blackbox.pl | 21 + .../ovn/basic/expected_controller_config | 8 + .../zones/ovn/basic/expected_sdn_interfaces | 12 + src/test/zones/ovn/basic/interfaces | 5 + src/test/zones/ovn/basic/sdn_config | 24 + .../ovn/with_evpn/expected_controller_config | 44 + .../ovn/with_evpn/expected_sdn_interfaces | 47 + src/test/zones/ovn/with_evpn/interfaces | 7 + src/test/zones/ovn/with_evpn/sdn_config | 35 + 29 files changed, 2444 insertions(+), 5 deletions(-) manager: www/manager6/Makefile | 2 + www/manager6/Utils.js | 10 +++++ www/manager6/form/SDNControllerSelector.js | 9 +++- www/manager6/sdn/VnetEdit.js | 37 +++++++++++++++- www/manager6/sdn/controllers/OvnEdit.js | 50 ++++++++++++++++++++++ www/manager6/sdn/zones/OvnEdit.js | 34 +++++++++++++++ 6 files changed, 139 insertions(+), 3 deletions(-) docs: pvesdn.adoc | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 293 insertions(+), 2 deletions(-) Summary over all repositories: 38 files changed, 2907 insertions(+), 10 deletions(-)