[PATCH container/docs/manager/network/proxmox{-ebpf,-perl-rs}/qemu-server v2 00/16] sdn: implement DHCP for all zones using eBPF

[Hannes Laimer <h.laimer@proxmox.com>]

Adds a second DHCP backend, `ebpf`, next to dnsmasq, selectable per
zone. It aims to replace dnsmasq eventually and keeping both for now
makes a migration simple. Every zone type can enable DHCP and the UI
offers the selector on the zone types it edits. `dnsmasq` stays limited
to simple zones.

The responder lives in proxmox-ebpf [1]. The two patches for it apply
on top of the v3 of that series, which follows separately. pve-rs gains
a build dependency on the librust-proxmox-ebpf-dev it provides.
pve-network needs this series' pve-rs at build time and at run time.
qemu-server needs this series' pve-network as a Depends. pve-container
works without the package and needs a Breaks on the older versions.
Both pve-manager patches need it as well.

# On the tap

An eBPF program on the ingress of every guest tap on a vnet of an ebpf
zone parses DHCP requests. It looks the interface up in an
interface -> ip+options map and turns the request into the reply in
place, back out of the tap. The exchange never reaches the bridge.
Everything else passes untouched to the bridge and through the firewall
as before. The answer is the interface's and only a request carrying
its NIC's MAC gets it. Another MAC is passed on unanswered. So no guest
can read another one's record by asking with its MAC, and a nested
client is not handed the outer guest's address. Equal subnets in two
zones do not mix either, each interface is answered from its own vnet's
subnet.

# The records and who writes them

IPAM is the source and the map holds the answer of every guest
interface on the node. A record change writes the records of the MAC's
interfaces here. A tap plug attaches its interface with its record and
an unplug drops it. All of that runs under a shared lock. The full pass
builds the desired state from the configs, every guest NIC in the
cluster with the vnet it sits on and the answer its MAC gets there from
the MAC cache. The responder gives every listed interface present here
its record, and a link where it is served. An interface the configs
serve without a mapping gets a marker, so a plug that read the cache
before the mapping went loses to it. It leaves a linked interface the
configs do not name yet alone, since a hotplug plugs before it writes
the config. It drops what a crashed guest left. A link on an interface
without a record is harmless, the program passes everything there. So
nothing is lost by attaching first and answering once the record is in.
A pass failing midway warns in its task, and what was served keeps
being served meanwhile. A change drawn before it is dropped, the next
apply carries its input.

 - a guest start, NIC hotplug or migration allocates the record before
   the interface is plugged. Then a new tap plug hook of the dhcp
   plugins attaches the program, and a stop or unplug detaches it
 - a mapping created, updated or deleted through the API is written by
   the editing node. That node pokes the node running the guest through
   a new node endpoint (POST /nodes/{node}/sdn/dhcp-mapping) with a
   detached pvesh call, proxied over ssh like pvesh does. An
   unreachable node catches up on its next apply
 - an SDN apply runs the full pass and refreshes the programs, so a
   schema rebuild is refilled and a zone switching its backend takes
   effect for its running guests that hold a mapping as well
 - after a boot nothing is pinned, so the SDN commit brings the
   responder back before the guests start. A change that still finds
   nothing loaded runs the full pass itself

# Ordering

Nothing forces the collection of that state into one order, so every
change carries a generation, a per-node counter the responder hands
out. There is one rule. A change draws after its input is written and
a reader before it reads. So two changes are ordered by the input they
saw and not by when they reach the kernel.

Who draws when:
 - record change  after the IPAM record and the cache are written
 - full pass      before it reads the configs and the cache
 - tap plug       before it reads the cache. Its number orders the
                  cache read and says nothing about the guest config,
                  which is not written yet

What the number decides:
 - record         keeps the generation that wrote it, an older change
                  never overwrites it. A removal by a record change
                  leaves a marker with its generation for the same
                  reason
 - full pass      records its generation before it runs, so an older
                  pass is dropped. It leaves alone every record a
                  newer change wrote since it read its input

Links carry no generation. The configs decide about the interfaces they
name, a linked interface they do not name is left alone. So a hotplug's
plug survives a pass that read the config before the write.

A generation cannot order a pass against a hotplug that moves a tap
between bridges. The plug runs before the guest config names the new
bridge, so a pass in between reads the old one. That is why every
record also names the vnet its writer saw the tap on.

What the vnet decides:
 - pass, record change   leave an entry alone that names another vnet
                         than their config does, their config is stale
                         for that tap. The pass lets the link follow
                         the entry
 - tap plug              replaces such an entry. Plugs of one tap are
                         serialized by the guest lock, so the plug is
                         the latest word on where it sits. On its own
                         vnet it compares by generation like any other
                         change. A plug finding no entry writes, a pass
                         that stamped meanwhile never saw its interface
 - plug elsewhere        a tap moved onto a plain bridge or a zone of
                         another backend trades its entry for a marker
                         naming that place. So a pass still reading
                         the old vnet leaves it alone, and one reading
                         the new place keeps it as it is. A tap without
                         an entry gets none
 - tap unplug            drops the entry with the link, the tap is
                         gone or about to be. A tap with neither a
                         link nor an interface left leaves its entry
                         to the next pass

The counter lives under /run and restarts with the node like
everything it orders.

# What a guest sees

A changed mapping reaches a running guest at its next renewal, so the
lease time bounds how long an old answer survives. A deleted one is no
longer answered, the guest keeps its address until the lease runs out.
Subnets get a `dhcp-lease-time` property, used by both backends.
Without it dnsmasq keeps handing out infinite leases and the responder
defaults to ten minutes.

Since the request is answered on the tap, neither the firewall nor the
rate limit of the interface sees it. So neither the guest's rules nor
its DHCP firewall option nor the rate limit apply to an answered
request. The reply is put on the tap's egress, so the rate limit shapes
it and the firewall never sees it.

The responder identifies itself with the subnet gateway. A subnet
without one is served under a link-local identifier and without a
default route, its guests renew by broadcast at rebinding time. It
hands out an IPv4 DNS server only, an IPv6 one on a v4 subnet is left
out. A guest tagging its own requests is not answered either, so a
trunk port keeps working as before. Only DHCPv4 for now, but adding v6
is very possible once we are happy with the design.

Changes made directly on an external IPAM service are not detectable
and the answers are cached per MAC. So an apply does not pick them up
either, exactly like with dnsmasq today.

note on the earlier versions: The main problem was that a tap plug ran
on information not in the config yet, only pending. This allowed a race
between the tap plug hook updating the kernel state and a fast enough
SDN apply undoing it again, and a few other subtleties. This version
should address all of these properly. Sorry for the noise with the other
two versions, both the result of me spotting the problem too late and
being too fast in assuming I had solved it.


[1] https://lore.proxmox.com/pve-devel/20260909103952.1108084-1-h.laimer@proxmox.com/T/#t


proxmox-ebpf:

Hannes Laimer (2):
  dhcp: add per-tap responder BPF program
  dhcp: add responder subsystem

 Cargo.toml                     |    5 +
 debian/control                 |    6 +-
 src/bpf-shim/bpf/bpf_helpers.h |    3 +
 src/bpf-shim/vmlinux.h         |    1 +
 src/dhcp/bpf/dhcp.bpf.c        |  405 ++++++++++++
 src/dhcp/bpf/types.h           |   25 +
 src/dhcp/mod.rs                | 1113 ++++++++++++++++++++++++++++++++
 src/dhcp/types.rs              |   80 +++
 src/lib.rs                     |    3 +
 tests/common/mod.rs            |   63 +-
 tests/dhcp.rs                  |  827 ++++++++++++++++++++++++
 11 files changed, 2526 insertions(+), 5 deletions(-)
 create mode 100644 src/dhcp/bpf/dhcp.bpf.c
 create mode 100644 src/dhcp/bpf/types.h
 create mode 100644 src/dhcp/mod.rs
 create mode 100644 src/dhcp/types.rs
 create mode 100644 tests/dhcp.rs


proxmox-perl-rs:

Hannes Laimer (1):
  pve-rs: sdn: add dhcp responder bindings

 pve-rs/Cargo.toml               |   2 +
 pve-rs/Makefile                 |   1 +
 pve-rs/debian/control           |   2 +
 pve-rs/src/bindings/sdn/dhcp.rs | 162 ++++++++++++++++++++++++++++++++
 pve-rs/src/bindings/sdn/mod.rs  |   1 +
 5 files changed, 168 insertions(+)
 create mode 100644 pve-rs/src/bindings/sdn/dhcp.rs


pve-network:

Hannes Laimer (8):
  sdn: push mapping changes from the ipam API to the dhcp backend
  sdn: ipam: do not cache negative per-MAC answers, lock the write
  sdn: subnets: add dhcp-lease-time property
  sdn: dhcp: only assert a backend's availability for zones using it
  sdn: dhcp: add ebpf plugin
  sdn: zones: attach the dhcp responder on tap plug, detach on unplug
  sdn: dhcp: apply mapping edits on the node serving the guest
  sdn: zones: offer dhcp on all zone types, keep dnsmasq simple-only

 debian/libpve-network-perl.prerm          |   13 +
 src/PVE/API2/Network/SDN/Ips.pm           |   10 +
 src/PVE/API2/Network/SDN/Nodes/Status.pm  |   37 +
 src/PVE/API2/Network/SDN/Zones.pm         |   11 +-
 src/PVE/Network/SDN/Dhcp.pm               |  202 ++-
 src/PVE/Network/SDN/Dhcp/Dnsmasq.pm       |  104 +-
 src/PVE/Network/SDN/Dhcp/Ebpf.pm          |  355 ++++++
 src/PVE/Network/SDN/Dhcp/Makefile         |    2 +-
 src/PVE/Network/SDN/Dhcp/Plugin.pm        |   25 +-
 src/PVE/Network/SDN/Ipams.pm              |   61 +-
 src/PVE/Network/SDN/SubnetPlugin.pm       |   11 +
 src/PVE/Network/SDN/Vnets.pm              |    5 +-
 src/PVE/Network/SDN/Zones.pm              |   31 +-
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm   |   32 +
 src/PVE/Network/SDN/Zones/FaucetPlugin.pm |    1 +
 src/PVE/Network/SDN/Zones/QinQPlugin.pm   |    7 +
 src/PVE/Network/SDN/Zones/SimplePlugin.pm |    2 +-
 src/PVE/Network/SDN/Zones/VlanPlugin.pm   |    7 +
 src/PVE/Network/SDN/Zones/VxlanPlugin.pm  |   25 +
 src/test/run_test_vnets_blackbox.pl       | 1361 ++++++++++++++++++++-
 20 files changed, 2223 insertions(+), 79 deletions(-)
 create mode 100755 debian/libpve-network-perl.prerm
 create mode 100644 src/PVE/Network/SDN/Dhcp/Ebpf.pm


qemu-server:

Hannes Laimer (1):
  network: report NIC plug and unplug to SDN with the MAC

 src/PVE/CLI/qm.pm             | 4 ++--
 src/PVE/QemuServer.pm         | 4 ++++
 src/PVE/QemuServer/Network.pm | 4 ++--
 src/usr/pve-bridge            | 8 +++++++-
 src/usr/pve-bridgedown        | 4 ++--
 5 files changed, 17 insertions(+), 7 deletions(-)


pve-container:

Hannes Laimer (1):
  net: report veth plug and unplug to SDN with the hwaddr

 src/PVE/LXC.pm            | 24 ++++++++++++++++++++++--
 src/PVE/LXC/Config.pm     |  2 +-
 src/lxc-pve-poststop-hook |  5 ++---
 3 files changed, 25 insertions(+), 6 deletions(-)


pve-manager:

Hannes Laimer (2):
  ui: sdn: dhcp backend selector on all zones, expose dhcp options
  sdn: bring the dhcp backends up at boot before the guests start

 bin/pve-sdn-commit                   |  3 +++
 services/pve-sdn-commit.service      |  3 ++-
 www/manager6/sdn/SubnetEdit.js       | 23 +++++++++++++++++++++++
 www/manager6/sdn/zones/Base.js       | 17 +++++++++++++++++
 www/manager6/sdn/zones/SimpleEdit.js | 11 -----------
 5 files changed, 45 insertions(+), 12 deletions(-)


pve-docs:

Hannes Laimer (1):
  sdn: dhcp: document the ebpf backend

 pvesdn.adoc | 113 +++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 95 insertions(+), 18 deletions(-)


Summary over all repositories:
  50 files changed, 5099 insertions(+), 127 deletions(-)

-- 
Generated by murpp 0.12.0