[RFC PATCH ve-rs/network/manager/docs 00/16] sdn: add OVN zone and controller plugins

["Rémi PAETA" <remi.paeta@comaite.com>]

This series adds OVN (Open Virtual Network) support to the SDN stack,
as first discussed in:

  https://lore.proxmox.com/pve-devel/1116090294.1714263.1791537779868.JavaMail.zimbra@comaite.com/

It covers the first step outlined there: an OVN controller, OVN zones
and vnets, providing a distributed L2 overlay with optional per-port
anti-spoofing. Routing (subnet gateways, SNAT), native DHCP and OVN
ACLs are left for later steps.

Sent as RFC, as some design choices below would benefit from your
feedback before going further.

I am contributing on behalf of my employer, Comaite, whose entity CLA
has been accepted by Proxmox.


Overview
========

- The OVN central components (northbound/southbound databases,
  ovn-northd) are deployed by the administrator, ideally as a RAFT
  cluster, on the PVE nodes or elsewhere. An "ovn" SDN controller holds
  their connection strings, plain TCP or SSL.

- Each vnet of an "ovn" zone is an OVN logical switch. On reload, the
  northbound database is synced once for the whole cluster: switches
  get created, updated and deleted, ports of removed guests and guest
  interfaces get cleaned up. Only objects tagged with pve-sdn-*
  external_ids are touched, so the deployment can be shared.

- On every node of an OVN zone, ovn-controller gets configured (remote,
  encapsulation, tunnel IP from the route towards the southbound
  database). pvenetcommit.service removes the Open vSwitch database on
  every boot, so a oneshot pve-sdn-ovn-chassis.service sets it again
  before the guests start.

- Guest interfaces are plugged into br-int and bound to a logical
  switch port named after them. Each vnet still gets an empty OVS
  bridge, which lets qemu-server and pve-container read its MTU and set
  up the firewall bridge like for any OVS bridge, so the guest firewall
  works unchanged. pve-manager needs no backend change.

- Port security, per zone with a per-vnet override: off (default),
  mac, or mac-ip with the IPAM addresses (IPv4 and IPv6). The guest MAC
  is set from add_bridge_fdb. A changed level applies on reload,
  without restarting guests.

- With SSL, the administrator provides the CA and one certificate per
  node in /etc/pve/priv/sdn/ovn/<controller>/. ovn-controller only
  takes certificates on its command line, so a runtime drop-in for its
  unit adds them; it is only restarted when that setup changes.

- Guest ports are bound to the chassis of their node (requested-chassis
  option). On live migration, they are bound to both nodes with
  activation-strategy=rarp, and back to the destination only when
  qemu-server calls add_bridge_fdb on resume, see below.


Changes since the discussion mail
=================================

- Port security can be overridden per vnet, which adds a
  'port-security' option to the common vnet schema (refused on other
  zone types, see the sixth network patch).

- The web UI is now part of the series (pve-manager patches).

- The proxmox-ve-rs patch is needed for both firewalls: without it,
  proxmox-firewall fails on the whole SDN configuration, and
  pve-firewall, through libpve-rs-perl, generates no SDN IPSet at all,
  for any zone, as soon as an OVN zone exists. My first mail only
  mentioned the nftables firewall.

- SSL connections to the OVN databases.

- Live migration: while QEMU runs on both nodes during the memory copy,
  both ovn-controller instances kept claiming the port from each other,
  every 500 ms, until the source released it. Measured with a probe
  every ~55 ms, this lost 2 to 3 s of traffic, while QEMU reported a
  downtime of ~20 ms. Binding the port to both chassis with
  activation-strategy=rarp during the migration, as OVN recommends for
  this case, and to the destination only on resume (qemu-server already
  calls add_bridge_fdb there) brings it down to 0.4 to 0.8 s, close to
  the resolution of the probe. This needs no change outside
  pve-network; the chassis hostname is set to the node name, as
  requested-chassis refers to it and Open vSwitch uses the FQDN.


Dependencies
============

- proxmox-ve-rs (first patch): libpve-rs-perl and proxmox-firewall
  need to be rebuilt with it before OVN zones can be used, so
  pve-network will likely need Breaks on older versions of both. How
  would you prefer to handle this?

- pve-manager depends on a libpve-network-perl knowing the vnet
  'port-security' option.

- Hotplugged NICs of VMs only get their MAC and port security on the
  next start, as qemu-server does not call add_bridge_fdb on hotplug.
  This is the existing bug #6490 for bridges with MAC learning
  disabled; I sent a separate fix for it:

    https://lore.proxmox.com/pve-devel/20261013095852.2586609-1-remi.paeta@comaite.com/


Known limitations
=================

- No routing, SNAT or DHCP yet: subnet gateways, SNAT and DHCP ranges
  are refused on OVN zones, as are VLAN tags, VLAN-aware vnets and
  isolated ports.

- With mac-ip, IPv6 link-local traffic is only allowed from the
  address derived from the MAC, which breaks guests using stable
  privacy link-local addresses for link-local protocols.

- If the IPAM cannot be queried, a reload keeps the current port
  addresses, while a starting guest only gets its MAC enforced.

- If the northbound database cannot be reached, running guests keep
  their connectivity, but guests on OVN zones cannot be started.


Testing
=======

Unit tests cover the new code (332 tests in pve-network). On a three
node cluster with the OVN databases clustered on the nodes, over SSL:

- VM and container connectivity across nodes, live migration
  (0.4 to 0.8 s of loss, QEMU downtime ~20 ms), container migration
- guest firewall with iptables and nftables, ipfilter, macfilter,
  firewall toggling on running guests
- port security levels with IPv4 and IPv6, spoofed MAC and IPs blocked
- northbound/southbound leader failover and full outage of all
  database members (no loss for running guests)
- node crash, HA recovery and port claim on another node with a stale
  binding of the dead node
- Geneve to VXLAN encapsulation switch and back
- zone restricted to nodes, removal of zones and of the controller
- orphaned ports after removing or moving guest interfaces


Design notes
============

- The OVN central components are deployed and operated separately, by
  the administrator. Deploying and managing them directly on the
  Proxmox VE cluster (for example as a RAFT cluster on some of the
  nodes, set up from the API) could be a later evolution.

- The guest start is refused while the northbound database cannot be
  reached. Starting the guest without its port would not help: the
  southbound database usually runs on the same hosts and is down too,
  so no chassis can claim a port, and a guest started elsewhere by HA
  could not be claimed anyway, as requested-chassis still names the
  previous node and only the northbound database can change it.

- With mac-ip and an unreachable IPAM, a starting guest only gets its
  MAC enforced (fail-open). The IPs are looked up through
  Ipams::get_ips_from_mac, so the MAC cache answers for known guests,
  and this only happens for a MAC the IPAM never returned before. Zones
  with DHCP fail the guest start in that case instead (through
  add_dhcp_mapping); I kept fail-open here so that an IPAM outage does
  not prevent starting guests, but can change it if you prefer.

- Certificates are left to the PKI of the OVN deployment (renewal,
  expiry). An API/GUI import could come later, ideally with the key
  generated on each node and only the signed certificate imported.


Open questions
==============

1. Each vnet currently gets an empty OVS bridge, while the guest
   interfaces are plugged into br-int. It only exists because
   qemu-server reads the vnet MTU from /sys/class/net/<vnet>/mtu
   (read_bridge_mtu, which also checks the bridges on migration
   targets), and to keep the firewall bridge setup of OVS bridges. This
   needs no change outside pve-network. Is this acceptable, or would
   you prefer one of these alternatives?

   a) An SDN dispatcher, for example
      PVE::Network::SDN::Zones::get_bridge_mtu(), returning the zone
      MTU for OVN vnets and reading sysfs otherwise, called by
      qemu-server and pve-container instead of read_bridge_mtu(). No
      placeholder interface anymore, but it touches both repositories
      (pve-common cannot depend on pve-network), and every place
      assuming that a vnet is a local interface would need the same
      treatment.

   b) A lighter placeholder: a dummy interface named after the vnet
      instead of an OVS bridge. Still no change outside pve-network,
      and nothing in the Open vSwitch database, but the firewall bridge
      setup would need to be checked again.

2. Is extending the common vnet schema with 'port-security' fine? As
   vnets have a single section type, I saw no other place for a per-vnet
   option.

3. VLAN-aware vnets are out of scope for this series. Guest-tagged
   trunks could later use the vlan-passthru option of logical switches,
   while per-NIC VLAN tags would need one logical switch per VLAN. Is
   either of interest?

4. Not specific to OVN, but noticed while testing it: deleting a vnet
   without subnets is not refused while guests still use it, on any
   zone type. Running guests end up unplugged and fail on their next
   start. Would you take a separate patch refusing the deletion with the
   list of guests using the vnet?


ve-rs:

 proxmox-ve-config/src/sdn/config.rs |  3 +++
 proxmox-ve-config/tests/sdn/main.rs | 28 ++++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)

network:

 debian/control                                |   1 +
 debian/libpve-network-perl.install            |   1 +
 debian/libpve-network-perl.postinst           |   2 +
 src/PVE/API2/Network/SDN.pm                   |   8 +
 src/PVE/API2/Network/SDN/Vnets.pm             |   8 +
 src/PVE/Network/SDN.pm                        |  39 +-
 src/PVE/Network/SDN/Controllers.pm            |   6 +
 src/PVE/Network/SDN/Controllers/Makefile      |   2 +-
 src/PVE/Network/SDN/Controllers/OvnPlugin.pm  |  90 ++
 src/PVE/Network/SDN/Makefile                  |   3 +-
 src/PVE/Network/SDN/Ovn.pm                    | 1024 ++++++++++++++++++
 src/PVE/Network/SDN/SubnetPlugin.pm           |   9 +
 src/PVE/Network/SDN/VnetPlugin.pm             |   8 +
 src/PVE/Network/SDN/Zones.pm                  |   2 +
 src/PVE/Network/SDN/Zones/Makefile            |   2 +-
 src/PVE/Network/SDN/Zones/OvnPlugin.pm        | 184 ++++
 src/services/Makefile                         |   1 +
 src/services/pve-sdn-ovn-chassis.service      |  16 +
 src/test/Makefile                             |   5 +-
 src/test/run_test_ovn.pl                      | 835 ++++++++++++++
 src/test/run_test_vnets_blackbox.pl           |  21 +
 .../ovn/basic/expected_controller_config      |   8 +
 .../zones/ovn/basic/expected_sdn_interfaces   |  12 +
 src/test/zones/ovn/basic/interfaces           |   5 +
 src/test/zones/ovn/basic/sdn_config           |  24 +
 .../ovn/with_evpn/expected_controller_config  |  44 +
 .../ovn/with_evpn/expected_sdn_interfaces     |  47 +
 src/test/zones/ovn/with_evpn/interfaces       |   7 +
 src/test/zones/ovn/with_evpn/sdn_config       |  35 +
 29 files changed, 2444 insertions(+), 5 deletions(-)

manager:

 www/manager6/Makefile                      |  2 +
 www/manager6/Utils.js                      | 10 +++++
 www/manager6/form/SDNControllerSelector.js |  9 +++-
 www/manager6/sdn/VnetEdit.js               | 37 +++++++++++++++-
 www/manager6/sdn/controllers/OvnEdit.js    | 50 ++++++++++++++++++++++
 www/manager6/sdn/zones/OvnEdit.js          | 34 +++++++++++++++
 6 files changed, 139 insertions(+), 3 deletions(-)

docs:

 pvesdn.adoc | 295 +++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 293 insertions(+), 2 deletions(-)

Summary over all repositories:
  38 files changed, 2907 insertions(+), 10 deletions(-)