From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 4973D1FF0AA for ; Tue, 20 Oct 2026 19:07:54 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B9FDF21607; Tue, 20 Oct 2026 19:07:50 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 zmta1.comaite.com AB1BB1559F07 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comaite.com; s=377A38C6-3A8D-11E5-A0EC-E97757B27CB3; t=1792515600; bh=htqyZTiTwiS2T0XU3lvCfK00xF6FYZ5xHIIBHMl2sJY=; h=Date:From:To:Message-ID:MIME-Version; b=pcbj/rMpsgFgr0VSeCjknxQCQPTU1Y/JPonRRhqIitqa14tB50AG2CJ+f/txSF3S/ I9t6wV/ImLztFNT3wrmCCdK+CWBDAWEOusXxHMqTiijzfTw7NklHaw46/eqRZPCCEI tIX3ZRU39DxgIHaeGNoiDhL6fIMSamgmQdnK6o9FsBnXkUae56vOgD4ZDSSAwNJ4IN JHTdU9figywx6TSanXCNI6UVzrpt7xTx2eHYk+7ZtQLGkyiKElWKiW43tvm4NEG42I vGzKRYdGmytYRE5XzMxCAqnCot2l0c1HVxpRmlj6iep6H7dDKWzog8aR0caIzmmWRd VHffmha6sfvCw== X-Virus-Scanned: amavisd-new at zmta1.comaite.com Date: Tue, 20 Oct 2026 18:57:34 +0200 (CEST) From: =?utf-8?B?UsOpbWk=?= PAETA To: Stefan Hanreich Message-ID: <1130824684.192837.1792515454020.JavaMail.zimbra@comaite.com> In-Reply-To: <0dda60a5-0bed-47de-8bf4-73e8837cf98a@proxmox.com> References: <1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com> <20261013121432.2642979-1-remi.paeta@comaite.com> <0dda60a5-0bed-47de-8bf4-73e8837cf98a@proxmox.com> Subject: Re: [RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone and controller plugins MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [45.80.27.88] X-Mailer: Zimbra 8.7.11_GA_3800 (ZimbraWebClient - GC155 (Linux)/8.7.11_GA_3800) Thread-Topic: add OVN zone and controller plugins Thread-Index: GE5y+nSgGurMja/dHtmbc1u87nrrtA== X-SPAM-LEVEL: Spam detection results: 2 AWL -2.500 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy MXG_THREAD_INDEX_BAD 0.001 Invalid Thread-Index header SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_DBL_SPAM 5 Contains a spam URL listed in the Spamhaus DBL blocklist [sdn.pm] Message-ID-Hash: BVZLASE4ESSC7DZTLD7BMKECI5QSHDZW X-Message-ID-Hash: BVZLASE4ESSC7DZTLD7BMKECI5QSHDZW X-MailFrom: remi.paeta@comaite.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Stefan, Thanks a lot for the feedback, and no worries about the delay! While waiting, I did another review round of the failure paths and found a few issues in this version, for example that a pmxcfs error at the start of an SDN reload could make the sync delete the guest ports, or that the chassis setup could block at boot if ovs-vswitchd does not run. They do not affect the design or the open questions, and normal operation is not affected, so feedback on this version is very welcome; I will send a v2 with the fixes once they are fully tested. For the setup, the docs patch (16/16) describes the OVN central deployment I used (three node RAFT cluster, plain TCP or SSL); I'm happy to help if anything is unclear. Regards,=20 ------------------------------------------------------ ----- Mail original ----- De: "Stefan Hanreich" =C3=80: "pve-devel" Envoy=C3=A9: Mardi 20 Octobre 2026 18:11:43 Objet: Re: [RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone a= nd controller plugins Hi, sorry for the late response and thanks for your contribution! I've looked at your initial proposal and the concept looks solid at first glance. I'll need to familiarize myself more with OVN and how it works and setup a test cluster here on my machine so I can give more in-depth feedback than that. I'll try to take a closer look at your series this or next week, I just wanted to let you know that this has not gone unnoticed! On 10/13/26 2:23 PM, R=C3=A9mi PAETA wrote: > This series adds OVN (Open Virtual Network) support to the SDN stack, > as first discussed in: >=20 > https://lore.proxmox.com/pve-devel/1116090294.1714263.1791537779868.Jav= aMail.zimbra@comaite.com/ >=20 > It covers the first step outlined there: an OVN controller, OVN zones > and vnets, providing a distributed L2 overlay with optional per-port > anti-spoofing. Routing (subnet gateways, SNAT), native DHCP and OVN > ACLs are left for later steps. >=20 > Sent as RFC, as some design choices below would benefit from your > feedback before going further. >=20 > I am contributing on behalf of my employer, Comaite, whose entity CLA > has been accepted by Proxmox. >=20 >=20 > Overview > =3D=3D=3D=3D=3D=3D=3D=3D >=20 > - The OVN central components (northbound/southbound databases, > ovn-northd) are deployed by the administrator, ideally as a RAFT > cluster, on the PVE nodes or elsewhere. An "ovn" SDN controller holds > their connection strings, plain TCP or SSL. >=20 > - Each vnet of an "ovn" zone is an OVN logical switch. On reload, the > northbound database is synced once for the whole cluster: switches > get created, updated and deleted, ports of removed guests and guest > interfaces get cleaned up. Only objects tagged with pve-sdn-* > external_ids are touched, so the deployment can be shared. >=20 > - On every node of an OVN zone, ovn-controller gets configured (remote, > encapsulation, tunnel IP from the route towards the southbound > database). pvenetcommit.service removes the Open vSwitch database on > every boot, so a oneshot pve-sdn-ovn-chassis.service sets it again > before the guests start. >=20 > - Guest interfaces are plugged into br-int and bound to a logical > switch port named after them. Each vnet still gets an empty OVS > bridge, which lets qemu-server and pve-container read its MTU and set > up the firewall bridge like for any OVS bridge, so the guest firewall > works unchanged. pve-manager needs no backend change. >=20 > - Port security, per zone with a per-vnet override: off (default), > mac, or mac-ip with the IPAM addresses (IPv4 and IPv6). The guest MAC > is set from add_bridge_fdb. A changed level applies on reload, > without restarting guests. >=20 > - With SSL, the administrator provides the CA and one certificate per > node in /etc/pve/priv/sdn/ovn//. ovn-controller only > takes certificates on its command line, so a runtime drop-in for its > unit adds them; it is only restarted when that setup changes. >=20 > - Guest ports are bound to the chassis of their node (requested-chassis > option). On live migration, they are bound to both nodes with > activation-strategy=3Drarp, and back to the destination only when > qemu-server calls add_bridge_fdb on resume, see below. >=20 >=20 > Changes since the discussion mail > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > - Port security can be overridden per vnet, which adds a > 'port-security' option to the common vnet schema (refused on other > zone types, see the sixth network patch). >=20 > - The web UI is now part of the series (pve-manager patches). >=20 > - The proxmox-ve-rs patch is needed for both firewalls: without it, > proxmox-firewall fails on the whole SDN configuration, and > pve-firewall, through libpve-rs-perl, generates no SDN IPSet at all, > for any zone, as soon as an OVN zone exists. My first mail only > mentioned the nftables firewall. >=20 > - SSL connections to the OVN databases. >=20 > - Live migration: while QEMU runs on both nodes during the memory copy, > both ovn-controller instances kept claiming the port from each other, > every 500 ms, until the source released it. Measured with a probe > every ~55 ms, this lost 2 to 3 s of traffic, while QEMU reported a > downtime of ~20 ms. Binding the port to both chassis with > activation-strategy=3Drarp during the migration, as OVN recommends for > this case, and to the destination only on resume (qemu-server already > calls add_bridge_fdb there) brings it down to 0.4 to 0.8 s, close to > the resolution of the probe. This needs no change outside > pve-network; the chassis hostname is set to the node name, as > requested-chassis refers to it and Open vSwitch uses the FQDN. >=20 >=20 > Dependencies > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > - proxmox-ve-rs (first patch): libpve-rs-perl and proxmox-firewall > need to be rebuilt with it before OVN zones can be used, so > pve-network will likely need Breaks on older versions of both. How > would you prefer to handle this? >=20 > - pve-manager depends on a libpve-network-perl knowing the vnet > 'port-security' option. >=20 > - Hotplugged NICs of VMs only get their MAC and port security on the > next start, as qemu-server does not call add_bridge_fdb on hotplug. > This is the existing bug #6490 for bridges with MAC learning > disabled; I sent a separate fix for it: >=20 > https://lore.proxmox.com/pve-devel/20261013095852.2586609-1-remi.paet= a@comaite.com/ >=20 >=20 > Known limitations > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > - No routing, SNAT or DHCP yet: subnet gateways, SNAT and DHCP ranges > are refused on OVN zones, as are VLAN tags, VLAN-aware vnets and > isolated ports. >=20 > - With mac-ip, IPv6 link-local traffic is only allowed from the > address derived from the MAC, which breaks guests using stable > privacy link-local addresses for link-local protocols. >=20 > - If the IPAM cannot be queried, a reload keeps the current port > addresses, while a starting guest only gets its MAC enforced. >=20 > - If the northbound database cannot be reached, running guests keep > their connectivity, but guests on OVN zones cannot be started. >=20 >=20 > Testing > =3D=3D=3D=3D=3D=3D=3D >=20 > Unit tests cover the new code (332 tests in pve-network). On a three > node cluster with the OVN databases clustered on the nodes, over SSL: >=20 > - VM and container connectivity across nodes, live migration > (0.4 to 0.8 s of loss, QEMU downtime ~20 ms), container migration > - guest firewall with iptables and nftables, ipfilter, macfilter, > firewall toggling on running guests > - port security levels with IPv4 and IPv6, spoofed MAC and IPs blocked > - northbound/southbound leader failover and full outage of all > database members (no loss for running guests) > - node crash, HA recovery and port claim on another node with a stale > binding of the dead node > - Geneve to VXLAN encapsulation switch and back > - zone restricted to nodes, removal of zones and of the controller > - orphaned ports after removing or moving guest interfaces >=20 >=20 > Design notes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > - The OVN central components are deployed and operated separately, by > the administrator. Deploying and managing them directly on the > Proxmox VE cluster (for example as a RAFT cluster on some of the > nodes, set up from the API) could be a later evolution. >=20 > - The guest start is refused while the northbound database cannot be > reached. Starting the guest without its port would not help: the > southbound database usually runs on the same hosts and is down too, > so no chassis can claim a port, and a guest started elsewhere by HA > could not be claimed anyway, as requested-chassis still names the > previous node and only the northbound database can change it. >=20 > - With mac-ip and an unreachable IPAM, a starting guest only gets its > MAC enforced (fail-open). The IPs are looked up through > Ipams::get_ips_from_mac, so the MAC cache answers for known guests, > and this only happens for a MAC the IPAM never returned before. Zones > with DHCP fail the guest start in that case instead (through > add_dhcp_mapping); I kept fail-open here so that an IPAM outage does > not prevent starting guests, but can change it if you prefer. >=20 > - Certificates are left to the PKI of the OVN deployment (renewal, > expiry). An API/GUI import could come later, ideally with the key > generated on each node and only the signed certificate imported. >=20 >=20 > Open questions > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > 1. Each vnet currently gets an empty OVS bridge, while the guest > interfaces are plugged into br-int. It only exists because > qemu-server reads the vnet MTU from /sys/class/net//mtu > (read_bridge_mtu, which also checks the bridges on migration > targets), and to keep the firewall bridge setup of OVS bridges. This > needs no change outside pve-network. Is this acceptable, or would > you prefer one of these alternatives? >=20 > a) An SDN dispatcher, for example > PVE::Network::SDN::Zones::get_bridge_mtu(), returning the zone > MTU for OVN vnets and reading sysfs otherwise, called by > qemu-server and pve-container instead of read_bridge_mtu(). No > placeholder interface anymore, but it touches both repositories > (pve-common cannot depend on pve-network), and every place > assuming that a vnet is a local interface would need the same > treatment. >=20 > b) A lighter placeholder: a dummy interface named after the vnet > instead of an OVS bridge. Still no change outside pve-network, > and nothing in the Open vSwitch database, but the firewall bridge > setup would need to be checked again. >=20 > 2. Is extending the common vnet schema with 'port-security' fine? As > vnets have a single section type, I saw no other place for a per-vnet > option. >=20 > 3. VLAN-aware vnets are out of scope for this series. Guest-tagged > trunks could later use the vlan-passthru option of logical switches, > while per-NIC VLAN tags would need one logical switch per VLAN. Is > either of interest? >=20 > 4. Not specific to OVN, but noticed while testing it: deleting a vnet > without subnets is not refused while guests still use it, on any > zone type. Running guests end up unplugged and fail on their next > start. Would you take a separate patch refusing the deletion with the > list of guests using the vnet? >=20 >=20 > ve-rs: >=20 > proxmox-ve-config/src/sdn/config.rs | 3 +++ > proxmox-ve-config/tests/sdn/main.rs | 28 ++++++++++++++++++++++++++++ > 2 files changed, 31 insertions(+) >=20 > network: >=20 > debian/control | 1 + > debian/libpve-network-perl.install | 1 + > debian/libpve-network-perl.postinst | 2 + > src/PVE/API2/Network/SDN.pm | 8 + > src/PVE/API2/Network/SDN/Vnets.pm | 8 + > src/PVE/Network/SDN.pm | 39 +- > src/PVE/Network/SDN/Controllers.pm | 6 + > src/PVE/Network/SDN/Controllers/Makefile | 2 +- > src/PVE/Network/SDN/Controllers/OvnPlugin.pm | 90 ++ > src/PVE/Network/SDN/Makefile | 3 +- > src/PVE/Network/SDN/Ovn.pm | 1024 ++++++++++++++++++ > src/PVE/Network/SDN/SubnetPlugin.pm | 9 + > src/PVE/Network/SDN/VnetPlugin.pm | 8 + > src/PVE/Network/SDN/Zones.pm | 2 + > src/PVE/Network/SDN/Zones/Makefile | 2 +- > src/PVE/Network/SDN/Zones/OvnPlugin.pm | 184 ++++ > src/services/Makefile | 1 + > src/services/pve-sdn-ovn-chassis.service | 16 + > src/test/Makefile | 5 +- > src/test/run_test_ovn.pl | 835 ++++++++++++++ > src/test/run_test_vnets_blackbox.pl | 21 + > .../ovn/basic/expected_controller_config | 8 + > .../zones/ovn/basic/expected_sdn_interfaces | 12 + > src/test/zones/ovn/basic/interfaces | 5 + > src/test/zones/ovn/basic/sdn_config | 24 + > .../ovn/with_evpn/expected_controller_config | 44 + > .../ovn/with_evpn/expected_sdn_interfaces | 47 + > src/test/zones/ovn/with_evpn/interfaces | 7 + > src/test/zones/ovn/with_evpn/sdn_config | 35 + > 29 files changed, 2444 insertions(+), 5 deletions(-) >=20 > manager: >=20 > www/manager6/Makefile | 2 + > www/manager6/Utils.js | 10 +++++ > www/manager6/form/SDNControllerSelector.js | 9 +++- > www/manager6/sdn/VnetEdit.js | 37 +++++++++++++++- > www/manager6/sdn/controllers/OvnEdit.js | 50 ++++++++++++++++++++++ > www/manager6/sdn/zones/OvnEdit.js | 34 +++++++++++++++ > 6 files changed, 139 insertions(+), 3 deletions(-) >=20 > docs: >=20 > pvesdn.adoc | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 293 insertions(+), 2 deletions(-) >=20 > Summary over all repositories: > 38 files changed, 2907 insertions(+), 10 deletions(-) >=20 >=20 >=20 >