From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id D521D1FF0AE for ; Tue, 13 Oct 2026 14:34:09 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D92E7216FA; Tue, 13 Oct 2026 14:33:00 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 smtp.infra.comaite.com 03A9D62C16 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comaite.com; s=377A38C6-3A8D-11E5-A0EC-E97757B27CB3; t=1791893674; bh=1g8RxMCFEOgDE6NTokNPbvDBMM95R6gSv5HEvLRb6DA=; h=From:To:Date:Message-ID:MIME-Version; b=p4aJn9aCL2NXkTF+Mdi4tnJQmkYnaSF4qRK/qFUr402okQEDvIhdduydKJ8BeKHfK bAhnrVBprXD9iWz2MYDGWkF+ppD50neyYmgnqGpKh6YOqI5vhBnf5dK8br2KVjEQaC yKYTxTsxoskDk2OAv14H1XOxHv+rAPjLj6ftaDktKEi9h4GuU9pJourLm23+IXRCmj 0fcASB83sblSoYC91LGVCDDBLBFhJI7sZe/h+Gz6GiVEKL+/1iPyIbThHVWp8hjjSo BNRlSDa4cm1cWI15uqFccTvu0jyN33MZIybuK8Zi8VOzeBuyBeM84OjrbB9Fg42dlc HkZ7uYu0qfo1A== From: =?UTF-8?q?R=C3=A9mi=20PAETA?= To: pve-devel@lists.proxmox.com Subject: [RFC PATCH docs 16/16] sdn: document ovn zones and controllers Date: Tue, 13 Oct 2026 14:14:32 +0200 Message-ID: <20261013121432.2642979-17-remi.paeta@comaite.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261013121432.2642979-1-remi.paeta@comaite.com> References: <1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com> <20261013121432.2642979-1-remi.paeta@comaite.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.625 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy KAM_SHORT 0.001 Use of a URL Shortener for very short URL SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 727G6UMY4BHPZLBXD3ZS324GTF55INVE X-Message-ID-Hash: 727G6UMY4BHPZLBXD3ZS324GTF55INVE X-MailFrom: remi.paeta@comaite.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Describe the installation of OVN, the OVN zone with its port security levels and limitations, the port security override on VNets, and the OVN controller with its encapsulation types, the detection of the tunnel IP and SSL connections to the databases. Add a setup example with the OVN databases clustered on the nodes, and notes on securing the parts of an OVN deployment that are not managed by Proxmox VE. Signed-off-by: Rémi PAETA --- pvesdn.adoc | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 293 insertions(+), 2 deletions(-) diff --git a/pvesdn.adoc b/pvesdn.adoc index 3fd3533..dce6dd4 100644 --- a/pvesdn.adoc +++ b/pvesdn.adoc @@ -131,6 +131,26 @@ Then enable the frr service on all nodes: systemctl enable frr.service ---- +[[pvesdn_install_ovn]] +Open Virtual Network +~~~~~~~~~~~~~~~~~~~~ + +xref:pvesdn_zone_plugin_ovn[OVN zones] use +https://www.ovn.org/[Open Virtual Network (OVN)]. This is currently opt-in. + +You need to install the `ovn-host` package on every node that is part of an OVN +zone. It provides `ovn-controller` and pulls in Open vSwitch: + +---- +apt update +apt install ovn-host +---- + +The OVN central components, the northbound and southbound databases and +`ovn-northd`, are not managed by {pve}. You need to deploy them yourself, either +on the {pve} nodes or on separate hosts, see the +xref:pvesdn_setup_example_ovn[OVN setup example]. + [[pvesdn_main_configuration]] Configuration Overview ---------------------- @@ -209,6 +229,8 @@ Different technologies can be used for separation: * EVPN (BGP EVPN): VXLAN with BGP to establish Layer 3 routing +* OVN: Layer 2 overlay network managed by Open Virtual Network (OVN) + [[pvesdn_config_common_options]] Common Options @@ -372,6 +394,56 @@ MTU:: Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes defaults to 1450. +[[pvesdn_zone_plugin_ovn]] +OVN Zones +~~~~~~~~~ + +The OVN zone creates a Layer 2 overlay network managed by Open Virtual Network +(OVN). Each VNet is an OVN logical switch spanning all nodes of the zone through +Geneve (or VXLAN) tunnels. As OVN knows on which node every guest interface is +located, it can filter the traffic of each guest port. + +The OVN zone requires an xref:pvesdn_controller_plugin_ovn[OVN controller] and +the `ovn-host` package on every node of the zone, see +xref:pvesdn_install_ovn[install Open Virtual Network]. + +OVN zone configuration options: + +Controller:: The OVN controller of this zone. All OVN zones of a node must use + the same controller, as a node can only be part of one OVN deployment. + +Port Security:: Filtering of the traffic of the guests, which VNets can + override. With `MAC`, guests can only send traffic with the MAC address of + their interface. With `MAC + IP`, they can additionally only use the IP + addresses of their interface in the IPAM, and only receive traffic for them. + Optional, defaults to `Off`. + +MTU:: Because Geneve encapsulation uses up to 58 bytes, the MTU needs to be 58 + bytes lower than the outgoing physical interface. Optional, defaults to 1442. + +With `Off` and `MAC`, guests still receive traffic for other MAC addresses, as +required by virtual MAC addresses, for example with VRRP or macvlan interfaces. +Only use `MAC + IP` for guests without additional MAC or IP addresses, which +excludes for example Kubernetes nodes, VRRP or Keepalived setups. A changed port +security level is applied with the SDN configuration, without restarting the +guests. + +With `MAC + IP`, the IPv6 link-local address of a guest is only allowed if it is +derived from its MAC address. Guests that use stable privacy link-local +addresses, for example by default with NetworkManager, keep their IPv4 and +global IPv6 connectivity, but cannot send any traffic from their link-local +address, as required for example by router advertisements or OSPFv3. + +If the IPAM cannot be queried when a guest starts, for example because a NetBox +server is unreachable, only the MAC address of the guest is enforced until the +SDN configuration gets applied again while the IPAM is reachable. When applying +the SDN configuration, ports whose IP addresses cannot be looked up keep their +current port security. + +NOTE: OVN zones do not support VLAN tags, VLAN-aware VNets and isolated ports. +Their subnets cannot have a gateway, SNAT or DHCP ranges, as OVN zones do not +route traffic or serve DHCP yet. + [[pvesdn_config_vnet]] VNets ----- @@ -407,7 +479,13 @@ to take effect, you need to restart the affected guest. NOTE: Port isolation is local to each host. Use the xref:pvesdn_firewall_integration[VNET Firewall] to further isolate traffic in the VNET across nodes. For example, DROP by default and only allow traffic from -the IP subnet to the gateway and vice versa. +the IP subnet to the gateway and vice versa. OVN zones do not support port +isolation. + +Port Security:: Only for xref:pvesdn_zone_plugin_ovn[OVN zones]. Overrides the + port security level of the zone for the guests on this VNet, for example to + enforce the IP addresses of regular guests, while another VNet of the same + zone carries a Kubernetes cluster without filtering. [[pvesdn_config_subnet]] Subnets @@ -439,6 +517,9 @@ SNAT:: Enable Source NAT which allows VMs from inside a outgoing interface. On EVPN zones, forwarding is done on EVPN gateway-nodes. Optional. +NOTE: Subnets of xref:pvesdn_zone_plugin_ovn[OVN zones] support neither a +gateway, nor SNAT, nor DHCP ranges. + DNS Zone Prefix:: Add a prefix to the domain registration, like .prefix. Optional. @@ -450,7 +531,7 @@ Controllers Some zones implement a separated control and data plane that require an external controller to manage the VNet's control plane. -Currently, only the `EVPN` zone requires an external controller. +Currently, the `EVPN` and `OVN` zones require an external controller. [[pvesdn_controller_plugin_evpn]] @@ -596,6 +677,111 @@ Loopback:: Use a loopback or dummy interface as the source of the EVPN network (for multipath). +[[pvesdn_controller_plugin_ovn]] +OVN Controller +~~~~~~~~~~~~~~ + +The `OVN` zone requires an OVN controller, which describes an existing OVN +deployment. The OVN controller plugin manages the logical switches and ports in +the northbound database and configures `ovn-controller` on every node of the OVN +zones. + +{pve} does not deploy the OVN databases and `ovn-northd`, see the +xref:pvesdn_setup_example_ovn[OVN setup example]. + +OVN controller configuration options: + +Northbound DB:: A comma separated list of the northbound database servers, for + example `tcp:10.0.0.1:6641,tcp:10.0.0.2:6641`. Use `ssl:` instead of `tcp:` + for xref:pvesdn_ovn_ssl[SSL connections]. + +Southbound DB:: A comma separated list of the southbound database servers, for + example `tcp:10.0.0.1:6642,tcp:10.0.0.2:6642`. Use `ssl:` instead of `tcp:` + for xref:pvesdn_ovn_ssl[SSL connections]. + +Encapsulation:: The tunnel encapsulation between the nodes, `Geneve` or `VXLAN`. + Optional, defaults to `Geneve`. + +Northbound Database +^^^^^^^^^^^^^^^^^^^ +When applying the SDN configuration, the controller creates, updates and +deletes the logical switches of the VNets and the logical switch ports of the +guests. It only changes objects it created, which are tagged with `pve-sdn-*` +external IDs, so other consumers can share the OVN deployment. A logical switch +with the name of a VNet that was not created by {pve} is refused. When you +remove an OVN controller, all objects it created get removed from the northbound +database. + +Database Outages +^^^^^^^^^^^^^^^^ +If the OVN databases cannot be reached, guests that are already running keep +their connectivity, as `ovn-controller` keeps the flows it installed. Guests on +OVN zones cannot be started though, as their logical switch port cannot be +created, and applying the SDN configuration only warns that the northbound +database could not be synchronized. Run the OVN databases as a cluster of at +least three members to tolerate the failure of one of them. + +Live Migration +^^^^^^^^^^^^^^ +The logical switch port of a guest interface is bound to the chassis of the node +the guest runs on, through its `requested-chassis` option. During a live +migration, the port is bound to both nodes, and the destination node only gets +the traffic of the guest once the guest announces itself there, after the +switchover. This avoids both nodes claiming the port while the memory of the +guest gets copied. Once the guest resumes on the destination node, the port is +bound to that node only. + +Tunnel IP auto-detection +^^^^^^^^^^^^^^^^^^^^^^^^ +The tunnel IP of a node is the source IP address of the route towards the first +southbound database server in the default routing table. {pve} sets it, along +with the southbound database, the encapsulation and the chassis hostname, which +is set to the node name, in the Open vSwitch configuration of the node, which gets recreated on every boot. The +`pve-sdn-ovn-chassis` service sets them again on boot, before the guests start. +When a node is not part of any OVN zone anymore, {pve} removes these settings +again. + +Encapsulation +^^^^^^^^^^^^^ +Prefer Geneve, which uses UDP port `6081`. VXLAN, which uses UDP port `4789`, +lacks room for the information OVN needs in the tunnel header: it limits a +deployment to about 4000 logical networks and about 2000 ports per network, and +some OVN features do not work with it. Use VXLAN only if the network between the +nodes does not support Geneve. + +Changing the encapsulation interrupts the traffic between the nodes until all of +them use the new encapsulation, which usually takes a few seconds when applying +the SDN configuration. + +[[pvesdn_ovn_ssl]] +SSL Connections +^^^^^^^^^^^^^^^ +SSL connections authenticate and encrypt the traffic with the OVN databases. +They are configured separately for the northbound database, which {pve} uses to +manage the logical switches and ports, and for the southbound database, which +`ovn-controller` uses on every node. Use SSL for both: without it, any host that +can reach the southbound database can read the whole network topology, register +as a chassis and claim the ports of guests. + +The certificates are provided by the public key infrastructure of the OVN +deployment, in `/etc/pve/priv/sdn/ovn//`: + +`ca.pem`:: + The CA certificate that signed the certificates of the database servers. +`.pem` and `.key`:: + The certificate and private key of each node, used for both databases. For + the role-based access control of the southbound database, use the Open + vSwitch `system-id` of the node as common name. + +{pve} copies the certificate of the node to `/etc/ovn/pve-sdn/` for +`ovn-controller` and restarts it when the certificates or the connection +settings change. While `ovn-controller` reconnects, the guests on the node can +lose connectivity for a few seconds. + +To renew the certificates, replace the files and apply the SDN configuration. +An expired or invalid certificate shows up as `ovn-controller` not being +connected in the status of the OVN zones. + [[pvesdn_config_fabrics]] Fabrics ------- @@ -1445,6 +1631,12 @@ SDN integrates with the Proxmox VE firewall by automatically generating IPSets which can then be referenced in the source / destination fields of firewall rules. This happens automatically for VNets and IPAM entries. +NOTE: With xref:pvesdn_zone_plugin_ovn[OVN zones], the firewall only generates +the SDN IPSets if the `libpve-rs-perl` package, and with the nftables-based +firewall also the `proxmox-firewall` package, support OVN zones. Otherwise, the +firewall cannot parse the SDN configuration and no SDN IPSet gets generated, for +any zone. + VNets and Subnets ~~~~~~~~~~~~~~~~~ @@ -1867,6 +2059,89 @@ If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 and 10.0.2.0/24 in this example), will be announced dynamically. +[[pvesdn_setup_example_ovn]] +OVN Setup Example +~~~~~~~~~~~~~~~~~ + +The example assumes a cluster with three nodes, with the node IP addresses +192.168.0.1, 192.168.0.2 and 192.168.0.3, which also run the OVN central +components as a three member cluster. + +Install the `ovn-central` and `ovn-host` packages on all nodes: + +---- +apt install ovn-central ovn-host +---- + +On the first node, configure the databases to create the cluster, in +`/etc/default/ovn-central`: + +---- +OVN_CTL_OPTS="--db-nb-addr=192.168.0.1 --db-sb-addr=192.168.0.1 \ + --db-nb-cluster-local-addr=192.168.0.1 --db-sb-cluster-local-addr=192.168.0.1 \ + --db-nb-create-insecure-remote=yes --db-sb-create-insecure-remote=yes \ + --ovn-northd-nb-db=tcp:192.168.0.1:6641,tcp:192.168.0.2:6641,tcp:192.168.0.3:6641 \ + --ovn-northd-sb-db=tcp:192.168.0.1:6642,tcp:192.168.0.2:6642,tcp:192.168.0.3:6642" +---- + +On the other nodes, use their own address and join the cluster of the first +node, for example on the second node: + +---- +OVN_CTL_OPTS="--db-nb-addr=192.168.0.2 --db-sb-addr=192.168.0.2 \ + --db-nb-cluster-local-addr=192.168.0.2 --db-sb-cluster-local-addr=192.168.0.2 \ + --db-nb-cluster-remote-addr=192.168.0.1 --db-sb-cluster-remote-addr=192.168.0.1 \ + --db-nb-create-insecure-remote=yes --db-sb-create-insecure-remote=yes \ + --ovn-northd-nb-db=tcp:192.168.0.1:6641,tcp:192.168.0.2:6641,tcp:192.168.0.3:6641 \ + --ovn-northd-sb-db=tcp:192.168.0.1:6642,tcp:192.168.0.2:6642,tcp:192.168.0.3:6642" +---- + +Restart `ovn-central` on all nodes, starting with the first one: + +---- +systemctl restart ovn-central +---- + +This setup uses unencrypted TCP connections. For production setups, use +xref:pvesdn_ovn_ssl[SSL connections] instead. + +Create an OVN controller named `ovnctl`: + +---- +ID: ovnctl +Northbound DB: tcp:192.168.0.1:6641,tcp:192.168.0.2:6641,tcp:192.168.0.3:6641 +Southbound DB: tcp:192.168.0.1:6642,tcp:192.168.0.2:6642,tcp:192.168.0.3:6642 +---- + +Create an OVN zone named `myovnzone` using the controller: + +---- +ID: myovnzone +Controller: ovnctl +Port Security: mac +---- + +Create a VNet named `ovnvnet1` using the OVN zone `myovnzone`: + +---- +ID: ovnvnet1 +Zone: myovnzone +---- + +Apply the configuration on the main SDN web interface panel. Then create two +virtual machines on different nodes, each with a vNIC on `ovnvnet1`, and use the +following network configuration for them (note the lower MTU): + +---- +auto eth0 +iface eth0 inet static + address 10.0.4.100/24 + mtu 1442 +---- + +Use `10.0.4.101/24` for the second virtual machine. Then, you should be able to +ping between the virtual machines. + [[pvesdn_setup_example_wireguard]] WireGuard Setup Example ~~~~~~~~~~~~~~~~~~~~~~~ @@ -2067,3 +2342,19 @@ and add the key to `/etc/ipsec.secrets`, so that the file contents looks like: ---- Copy the PSK and the configuration to all nodes participating in the VXLAN network. + +OVN Security +~~~~~~~~~~~~ + +Besides using xref:pvesdn_ovn_ssl[SSL connections] to the OVN databases, +consider securing the parts of an OVN deployment that {pve} does not manage: + +* The replication between the members of a clustered OVN database uses + separate connections, which can also use SSL. +* The role-based access control of the southbound database restricts each + chassis to its own data. It requires SSL connections and certificates whose + common name is the name of the chassis. +* The traffic of the guests in the Geneve or VXLAN tunnels is not encrypted. + OVN can encrypt it with IPsec. + +Refer to the OVN documentation for these settings. -- 2.47.3