superseded: [PATCH docs/manager/network/qemu-server v5 00/13] sdn: improve MAC cache, DHCP reservations and scope both to the vnet

[Hannes Laimer <h.laimer@proxmox.com>]

superseded-by:
https://lore.proxmox.com/pve-devel/20261105093529.209052-1-h.laimer@proxmox.com/

On 2026-11-04 10:55, Hannes Laimer wrote:
> This series addresses some problems in the IPAM code, mostly around
> caching. It also solves the problem that the MAC cache and the lookup
> behind it held a single address per MAC. This is a problem since MACs
> only have to be unique within a vnet and can repeat across vnets, and
> it led to a guest getting the IP of a guest with the same MAC on
> another vnet.
> 
> Patches 1-5 address general, mostly cache related problems. Patch 1
> leaves entries without an address in the file and treats them as a
> miss, patch 2 skips the fill when the file changed while the IPAM was
> asked, and patch 3 leaves the cache keys as written, since nodes not
> yet upgraded still use them. Patch 4 replaces a stale reservation
> line instead of dropping it, and patch 5 lets the built-in IPAM
> release an address it no longer holds, so its cache entry is cleared.
> 
> Patch 6 adds updating of the dnsmasq reservations when a mapping
> changes, so dnsmasq picks the change up immediately and a guest gets
> its new address without a restart. The edit runs on the guest's node
> under its migration lock, so pve-network now depends on
> pve-guest-common. The lock could move to a package below both if we
> want that.
> 
> Patches 7-8 scope the lookup and the cache to the vnet, so guests
> don't get addresses of another vnet. The cache file now contains a
> `vnets` map next to `macs`. `macs` stays, since nodes not yet
> upgraded read and write only it, and keeps holding one address per
> MAC as before. Every write goes into both maps. A lookup answers
> from `vnets` as long as `macs` still names the same address.
> Otherwise it asks the IPAM within the vnet's subnets, since a write
> for the MAC on another vnet or a node not yet upgraded changed or
> released the address in between. An entry only `macs` has is one
> from before, it answers for a vnet only with an address in the
> vnet's subnets. A MAC with an address of the same family on two
> vnets can match `macs` on one of them at most, so a start on the
> other asks the IPAM. So an old entry answers only for the vnet it
> was written for. The one exception is the same or an overlapping range
> on two vnets, where an old entry answers for the other vnet too, as
> it did before, until one of the two has an entry of its own or the
> address is released.
> 
> Patch 9 adds separation by zone for the two external IPAM providers
> we support, since two zones can use the same range. It is behind a
> `zone-namespaces` option of the IPAM entry, off by default, to be
> turned on once every node runs this version.
>  - NetBox: each zone gets its own `pve-<zone>` VRF, and the prefixes
>    of its subnets go in there
>  - phpIPAM: each zone gets a sub-section `pve-<zone>` under the
>    configured one, and its subnets go in there(only if the configured
>    section is not a sub-section already, phpIPAM only allows 1 deep
>    nesting)
> The names start with a `zone-namespace-prefix` of the IPAM entry,
> `pve` by default, so clusters sharing an IPAM stay apart. Patch 9
> also gives phpIPAM the lookup by MAC, which failed before.
> 
> Patch 10 is qemu-server's side. An edit and a migration of the guest
> exclude each other since patch 6. An edit through a node not yet
> upgraded still goes through during one, and it writes the IPAM and the
> cache but no reservation. A resume now pushes the current mapping of
> each NIC, and a migration resumes the target after the config moved,
> so the target serves the current addresses. The patch needs the
> pve-network of this series.
> 
> Patches 11-12 are pve-manager's side, the DHCP tree keeping the
> subnets of two zones with one range apart, it merged them under the
> first zone before, and the two options in the NetBox and phpIPAM
> dialogs. Patch 12 needs the pve-network of this series.
> 
> Patch 13 just adds docs explaining this for the two external IPAMs.
> 
> This is not a breaking change, existing assignments stay as they are
> and keep answering for their vnet. Cache entries from before keep
> their addresses, since nodes not yet upgraded still use them, and
> answer for a vnet with the addresses in its subnets, so a running
> guest keeps its reservation. Records in an external IPAM stay where
> they are, in NetBox's global table or in phpIPAM's configured section.
> Like the cache, an IPAM lookup with the option on first checks the
> zone's VRF or section for the subnet and falls back to the global
> table or the configured section. Unlike with the cache, migration can
> be done manually, by moving a prefix together with its addresses and
> IP ranges into the zone's VRF in NetBox, or a subnet into the zone's
> sub-section in phpIPAM, since the lookup checks those first. Migration
> is not necessary though. With the option on, creating a subnet creates
> the VRF or sub-section if it doesn't exist yet, nothing else does. A
> subnet whose range is still in the global table or the configured
> section is then refused, since a copy would hide the records in it.
> The error says to move that prefix or subnet into the VRF or
> sub-section of its zone, or to remove it. We don't move anything and
> we don't have to. With the option off nothing changes in the external
> IPAM, so a rolling upgrade is safe, and a node not yet upgraded never
> meets a subnet in a VRF or sub-section.
> 
> v5:
>  - also check the zone-namespace flag on external ipam lookup, so
>    manually moved entries are not picked up without the flag set
>  - for a dhcp update push on migration resume lock non-blocking,
>    whatever write would be holding the lock writes the same line the
>    push would have written, and ask the ipam if the cache can't vouch
>    for the entry
>  - phpipam: dont allow setting the zone-namespace flag if the configured
>    section is a sub-section, phpipam only allows 1 deep nesting, and of
>    the configured section is already 1 deep, we can't create the zone
>    sections
>  - refresh vmlist under lock
>  - migrate resume asks ipam if cache has disagreeing entries
> 
> v4:
>  - add IPAM config flag to use zone scoped layout, tried avoiding this,
>    but that added much more problems than it solved
>  - a mapping edit and a migration of the guest exclude each other
>  - the resume of a guest pushes its reservations from the cache only
>  - the same address on two vnets of a MAC is checked with the IPAM at
>    every start
>  - a prefix option of the IPAM entry names the zone namespaces, pve
>    by default, so clusters sharing an IPAM stay apart
>  - pve-manager: the two options in the dialogs, and the DHCP tree
>    keeps the subnets of two zones with one range apart
> 
> v3:
>  - a start asks the IPAM when the MAC sits on two vnets and the cache
>    can't vouch for it
>  - ask the IPAM for the changed MAC before walking the reservations,
>    so its stale line is replaced
>  - phpIPAM: fail subnet creation on api error
>  - an API mapping edit fails when the reservation update does, the
>    records are written by then
> 
> v2:
>  - the cache and the lookup are kept per vnet, the old map stays
>    for nodes not yet upgraded
>  - entries from before are left as written, a fill is skipped when
>    the file changed meanwhile
>  - a release leaves a record that belongs to another MAC alone
>  - each zone gets its own VRF or section in the external IPAMs, and
>    phpIPAM gets the lookup by MAC
>  - an edit for a guest whose node is offline runs here, and the
>    reservation rewrite reads the cache once
>  - the target of a live migration catches up when the guest resumes
> 
> 
> pve-network:
> 
> Hannes Laimer (9):
>   sdn: ipam: stop caching IPAM misses
>   sdn: ipam: cache a MAC lookup's answer under the cluster lock
>   sdn: ipam: read the MAC cache case-insensitively
>   sdn: dhcp: replace a stale dnsmasq reservation instead of dropping it
>   sdn: ipam: release an address the built-in IPAM no longer holds
>   sdn: dhcp: update reservations when IP mappings change
>   sdn: ipam: look a MAC up within the vnet's subnets
>   sdn: ipam: keep the MAC cache per vnet
>   sdn: ipam: give each zone a VRF in NetBox and a section in phpIPAM
> 
>  debian/control                             |   2 +
>  src/PVE/API2/Network/SDN/Ips.pm            | 120 +++-
>  src/PVE/Network/SDN/Dhcp.pm                |  50 +-
>  src/PVE/Network/SDN/Dhcp/Dnsmasq.pm        | 179 +++--
>  src/PVE/Network/SDN/Dhcp/Plugin.pm         |   8 +-
>  src/PVE/Network/SDN/Ipams.pm               | 188 +++++-
>  src/PVE/Network/SDN/Ipams/NetboxPlugin.pm  | 241 +++++--
>  src/PVE/Network/SDN/Ipams/PVEPlugin.pm     |  27 +-
>  src/PVE/Network/SDN/Ipams/PhpIpamPlugin.pm | 244 ++++++-
>  src/PVE/Network/SDN/Ipams/Plugin.pm        |  24 +-
>  src/PVE/Network/SDN/Subnets.pm             |  14 +-
>  src/PVE/Network/SDN/Vnets.pm               |  21 +-
>  src/test/run_test_ipams.pl                 | 612 +++++++++++++++++
>  src/test/run_test_vnets_blackbox.pl        | 733 ++++++++++++++++++++-
>  14 files changed, 2233 insertions(+), 230 deletions(-)
> 
> 
> qemu-server:
> 
> Hannes Laimer (1):
>   network: sdn: push the DHCP mappings of a guest when it resumes
> 
>  src/PVE/QemuServer/Network.pm  | 17 +++++++++++++++++
>  src/PVE/QemuServer/RunState.pm |  2 ++
>  2 files changed, 19 insertions(+)
> 
> 
> pve-manager:
> 
> Hannes Laimer (2):
>   ui: sdn: dhcp tree: do not merge the same subnet in different zones
>   ui: sdn: ipam: add the zone namespace options
> 
>  www/manager6/sdn/ipams/Base.js        | 25 +++++++++++++++++++++++++
>  www/manager6/sdn/ipams/NetboxEdit.js  |  2 ++
>  www/manager6/sdn/ipams/PhpIpamEdit.js |  2 ++
>  www/manager6/tree/DhcpTree.js         |  8 +++++---
>  4 files changed, 34 insertions(+), 3 deletions(-)
> 
> 
> pve-docs:
> 
> Hannes Laimer (1):
>   sdn: ipam: document zone namespaces in NetBox and phpIPAM
> 
>  pvesdn.adoc | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
> 
> 
> Summary over all repositories:
>   21 files changed, 2321 insertions(+), 233 deletions(-)
>