Re: [RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone and controller plugins

["Rémi PAETA" <remi.paeta@comaite.com>]

Hi Stefan,

Thanks a lot for the feedback, and no worries about the delay!

While waiting, I did another review round of the failure paths and found
a few issues in this version, for example that a pmxcfs error at the
start of an SDN reload could make the sync delete the guest ports, or
that the chassis setup could block at boot if ovs-vswitchd does not run.
They do not affect the design or the open questions, and normal
operation is not affected, so feedback on this version is very welcome;
I will send a v2 with the fixes once they are fully tested.

For the setup, the docs patch (16/16) describes the OVN central
deployment I used (three node RAFT cluster, plain TCP or SSL); I'm happy
to help if anything is unclear.

Regards, 

------------------------------------------------------

----- Mail original -----
De: "Stefan Hanreich" <s.hanreich@proxmox.com>
À: "pve-devel" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 20 Octobre 2026 18:11:43
Objet: Re: [RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone and controller plugins

Hi, sorry for the late response and thanks for your contribution!

I've looked at your initial proposal and the concept looks solid at
first glance. I'll need to familiarize myself more with OVN and how
it works and setup a test cluster here on my machine so I can give
more in-depth feedback than that.

I'll try to take a closer look at your series this or next week, I
just wanted to let you know that this has not gone unnoticed!


On 10/13/26 2:23 PM, Rémi PAETA wrote:
> This series adds OVN (Open Virtual Network) support to the SDN stack,
> as first discussed in:
> 
>   https://lore.proxmox.com/pve-devel/1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com/
> 
> It covers the first step outlined there: an OVN controller, OVN zones
> and vnets, providing a distributed L2 overlay with optional per-port
> anti-spoofing. Routing (subnet gateways, SNAT), native DHCP and OVN
> ACLs are left for later steps.
> 
> Sent as RFC, as some design choices below would benefit from your
> feedback before going further.
> 
> I am contributing on behalf of my employer, Comaite, whose entity CLA
> has been accepted by Proxmox.
> 
> 
> Overview
> ========
> 
> - The OVN central components (northbound/southbound databases,
>   ovn-northd) are deployed by the administrator, ideally as a RAFT
>   cluster, on the PVE nodes or elsewhere. An "ovn" SDN controller holds
>   their connection strings, plain TCP or SSL.
> 
> - Each vnet of an "ovn" zone is an OVN logical switch. On reload, the
>   northbound database is synced once for the whole cluster: switches
>   get created, updated and deleted, ports of removed guests and guest
>   interfaces get cleaned up. Only objects tagged with pve-sdn-*
>   external_ids are touched, so the deployment can be shared.
> 
> - On every node of an OVN zone, ovn-controller gets configured (remote,
>   encapsulation, tunnel IP from the route towards the southbound
>   database). pvenetcommit.service removes the Open vSwitch database on
>   every boot, so a oneshot pve-sdn-ovn-chassis.service sets it again
>   before the guests start.
> 
> - Guest interfaces are plugged into br-int and bound to a logical
>   switch port named after them. Each vnet still gets an empty OVS
>   bridge, which lets qemu-server and pve-container read its MTU and set
>   up the firewall bridge like for any OVS bridge, so the guest firewall
>   works unchanged. pve-manager needs no backend change.
> 
> - Port security, per zone with a per-vnet override: off (default),
>   mac, or mac-ip with the IPAM addresses (IPv4 and IPv6). The guest MAC
>   is set from add_bridge_fdb. A changed level applies on reload,
>   without restarting guests.
> 
> - With SSL, the administrator provides the CA and one certificate per
>   node in /etc/pve/priv/sdn/ovn/<controller>/. ovn-controller only
>   takes certificates on its command line, so a runtime drop-in for its
>   unit adds them; it is only restarted when that setup changes.
> 
> - Guest ports are bound to the chassis of their node (requested-chassis
>   option). On live migration, they are bound to both nodes with
>   activation-strategy=rarp, and back to the destination only when
>   qemu-server calls add_bridge_fdb on resume, see below.
> 
> 
> Changes since the discussion mail
> =================================
> 
> - Port security can be overridden per vnet, which adds a
>   'port-security' option to the common vnet schema (refused on other
>   zone types, see the sixth network patch).
> 
> - The web UI is now part of the series (pve-manager patches).
> 
> - The proxmox-ve-rs patch is needed for both firewalls: without it,
>   proxmox-firewall fails on the whole SDN configuration, and
>   pve-firewall, through libpve-rs-perl, generates no SDN IPSet at all,
>   for any zone, as soon as an OVN zone exists. My first mail only
>   mentioned the nftables firewall.
> 
> - SSL connections to the OVN databases.
> 
> - Live migration: while QEMU runs on both nodes during the memory copy,
>   both ovn-controller instances kept claiming the port from each other,
>   every 500 ms, until the source released it. Measured with a probe
>   every ~55 ms, this lost 2 to 3 s of traffic, while QEMU reported a
>   downtime of ~20 ms. Binding the port to both chassis with
>   activation-strategy=rarp during the migration, as OVN recommends for
>   this case, and to the destination only on resume (qemu-server already
>   calls add_bridge_fdb there) brings it down to 0.4 to 0.8 s, close to
>   the resolution of the probe. This needs no change outside
>   pve-network; the chassis hostname is set to the node name, as
>   requested-chassis refers to it and Open vSwitch uses the FQDN.
> 
> 
> Dependencies
> ============
> 
> - proxmox-ve-rs (first patch): libpve-rs-perl and proxmox-firewall
>   need to be rebuilt with it before OVN zones can be used, so
>   pve-network will likely need Breaks on older versions of both. How
>   would you prefer to handle this?
> 
> - pve-manager depends on a libpve-network-perl knowing the vnet
>   'port-security' option.
> 
> - Hotplugged NICs of VMs only get their MAC and port security on the
>   next start, as qemu-server does not call add_bridge_fdb on hotplug.
>   This is the existing bug #6490 for bridges with MAC learning
>   disabled; I sent a separate fix for it:
> 
>     https://lore.proxmox.com/pve-devel/20261013095852.2586609-1-remi.paeta@comaite.com/
> 
> 
> Known limitations
> =================
> 
> - No routing, SNAT or DHCP yet: subnet gateways, SNAT and DHCP ranges
>   are refused on OVN zones, as are VLAN tags, VLAN-aware vnets and
>   isolated ports.
> 
> - With mac-ip, IPv6 link-local traffic is only allowed from the
>   address derived from the MAC, which breaks guests using stable
>   privacy link-local addresses for link-local protocols.
> 
> - If the IPAM cannot be queried, a reload keeps the current port
>   addresses, while a starting guest only gets its MAC enforced.
> 
> - If the northbound database cannot be reached, running guests keep
>   their connectivity, but guests on OVN zones cannot be started.
> 
> 
> Testing
> =======
> 
> Unit tests cover the new code (332 tests in pve-network). On a three
> node cluster with the OVN databases clustered on the nodes, over SSL:
> 
> - VM and container connectivity across nodes, live migration
>   (0.4 to 0.8 s of loss, QEMU downtime ~20 ms), container migration
> - guest firewall with iptables and nftables, ipfilter, macfilter,
>   firewall toggling on running guests
> - port security levels with IPv4 and IPv6, spoofed MAC and IPs blocked
> - northbound/southbound leader failover and full outage of all
>   database members (no loss for running guests)
> - node crash, HA recovery and port claim on another node with a stale
>   binding of the dead node
> - Geneve to VXLAN encapsulation switch and back
> - zone restricted to nodes, removal of zones and of the controller
> - orphaned ports after removing or moving guest interfaces
> 
> 
> Design notes
> ============
> 
> - The OVN central components are deployed and operated separately, by
>   the administrator. Deploying and managing them directly on the
>   Proxmox VE cluster (for example as a RAFT cluster on some of the
>   nodes, set up from the API) could be a later evolution.
> 
> - The guest start is refused while the northbound database cannot be
>   reached. Starting the guest without its port would not help: the
>   southbound database usually runs on the same hosts and is down too,
>   so no chassis can claim a port, and a guest started elsewhere by HA
>   could not be claimed anyway, as requested-chassis still names the
>   previous node and only the northbound database can change it.
> 
> - With mac-ip and an unreachable IPAM, a starting guest only gets its
>   MAC enforced (fail-open). The IPs are looked up through
>   Ipams::get_ips_from_mac, so the MAC cache answers for known guests,
>   and this only happens for a MAC the IPAM never returned before. Zones
>   with DHCP fail the guest start in that case instead (through
>   add_dhcp_mapping); I kept fail-open here so that an IPAM outage does
>   not prevent starting guests, but can change it if you prefer.
> 
> - Certificates are left to the PKI of the OVN deployment (renewal,
>   expiry). An API/GUI import could come later, ideally with the key
>   generated on each node and only the signed certificate imported.
> 
> 
> Open questions
> ==============
> 
> 1. Each vnet currently gets an empty OVS bridge, while the guest
>    interfaces are plugged into br-int. It only exists because
>    qemu-server reads the vnet MTU from /sys/class/net/<vnet>/mtu
>    (read_bridge_mtu, which also checks the bridges on migration
>    targets), and to keep the firewall bridge setup of OVS bridges. This
>    needs no change outside pve-network. Is this acceptable, or would
>    you prefer one of these alternatives?
> 
>    a) An SDN dispatcher, for example
>       PVE::Network::SDN::Zones::get_bridge_mtu(), returning the zone
>       MTU for OVN vnets and reading sysfs otherwise, called by
>       qemu-server and pve-container instead of read_bridge_mtu(). No
>       placeholder interface anymore, but it touches both repositories
>       (pve-common cannot depend on pve-network), and every place
>       assuming that a vnet is a local interface would need the same
>       treatment.
> 
>    b) A lighter placeholder: a dummy interface named after the vnet
>       instead of an OVS bridge. Still no change outside pve-network,
>       and nothing in the Open vSwitch database, but the firewall bridge
>       setup would need to be checked again.
> 
> 2. Is extending the common vnet schema with 'port-security' fine? As
>    vnets have a single section type, I saw no other place for a per-vnet
>    option.
> 
> 3. VLAN-aware vnets are out of scope for this series. Guest-tagged
>    trunks could later use the vlan-passthru option of logical switches,
>    while per-NIC VLAN tags would need one logical switch per VLAN. Is
>    either of interest?
> 
> 4. Not specific to OVN, but noticed while testing it: deleting a vnet
>    without subnets is not refused while guests still use it, on any
>    zone type. Running guests end up unplugged and fail on their next
>    start. Would you take a separate patch refusing the deletion with the
>    list of guests using the vnet?
> 
> 
> ve-rs:
> 
>  proxmox-ve-config/src/sdn/config.rs |  3 +++
>  proxmox-ve-config/tests/sdn/main.rs | 28 ++++++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
> 
> network:
> 
>  debian/control                                |   1 +
>  debian/libpve-network-perl.install            |   1 +
>  debian/libpve-network-perl.postinst           |   2 +
>  src/PVE/API2/Network/SDN.pm                   |   8 +
>  src/PVE/API2/Network/SDN/Vnets.pm             |   8 +
>  src/PVE/Network/SDN.pm                        |  39 +-
>  src/PVE/Network/SDN/Controllers.pm            |   6 +
>  src/PVE/Network/SDN/Controllers/Makefile      |   2 +-
>  src/PVE/Network/SDN/Controllers/OvnPlugin.pm  |  90 ++
>  src/PVE/Network/SDN/Makefile                  |   3 +-
>  src/PVE/Network/SDN/Ovn.pm                    | 1024 ++++++++++++++++++
>  src/PVE/Network/SDN/SubnetPlugin.pm           |   9 +
>  src/PVE/Network/SDN/VnetPlugin.pm             |   8 +
>  src/PVE/Network/SDN/Zones.pm                  |   2 +
>  src/PVE/Network/SDN/Zones/Makefile            |   2 +-
>  src/PVE/Network/SDN/Zones/OvnPlugin.pm        | 184 ++++
>  src/services/Makefile                         |   1 +
>  src/services/pve-sdn-ovn-chassis.service      |  16 +
>  src/test/Makefile                             |   5 +-
>  src/test/run_test_ovn.pl                      | 835 ++++++++++++++
>  src/test/run_test_vnets_blackbox.pl           |  21 +
>  .../ovn/basic/expected_controller_config      |   8 +
>  .../zones/ovn/basic/expected_sdn_interfaces   |  12 +
>  src/test/zones/ovn/basic/interfaces           |   5 +
>  src/test/zones/ovn/basic/sdn_config           |  24 +
>  .../ovn/with_evpn/expected_controller_config  |  44 +
>  .../ovn/with_evpn/expected_sdn_interfaces     |  47 +
>  src/test/zones/ovn/with_evpn/interfaces       |   7 +
>  src/test/zones/ovn/with_evpn/sdn_config       |  35 +
>  29 files changed, 2444 insertions(+), 5 deletions(-)
> 
> manager:
> 
>  www/manager6/Makefile                      |  2 +
>  www/manager6/Utils.js                      | 10 +++++
>  www/manager6/form/SDNControllerSelector.js |  9 +++-
>  www/manager6/sdn/VnetEdit.js               | 37 +++++++++++++++-
>  www/manager6/sdn/controllers/OvnEdit.js    | 50 ++++++++++++++++++++++
>  www/manager6/sdn/zones/OvnEdit.js          | 34 +++++++++++++++
>  6 files changed, 139 insertions(+), 3 deletions(-)
> 
> docs:
> 
>  pvesdn.adoc | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 293 insertions(+), 2 deletions(-)
> 
> Summary over all repositories:
>   38 files changed, 2907 insertions(+), 10 deletions(-)
> 
> 
> 
>