Re: [PATCH docs/network/qemu-server v2 00/11] sdn: improve MAC cache, DHCP reservations and scope both to the vnet

[Hannes Laimer <h.laimer@proxmox.com>]

should be v3 in the mail prefix, not v2, sorry

On 2026-11-03 16:16, Hannes Laimer wrote:
> This series addresses some problems in the IPAM code, mostly around
> caching. It also solves the problem that the MAC cache and the lookup
> behind it held a single address per MAC. This is a problem since MACs
> only have to be unique within a vnet and can repeat across vnets, and
> it led to a guest getting the IP of a guest with the same MAC on
> another vnet.
> 
> Patches 1-5 address general, mostly cache related problems. Patches
> 1-4 were already included in [1], 1-3 changed since then. Patch 1
> leaves entries without an address in the file and treats them as a
> miss, patch 2 skips the fill when the file changed while the IPAM was
> asked, and patch 3 leaves the cache keys as written, since nodes not
> yet upgraded still use them. Patch 5 is new.
> 
> Patch 6 adds updating of the dnsmasq reservations when a mapping
> changes, so dnsmasq picks the change up immediately and a guest gets
> its new address without a restart.
> 
> Patches 7-8 scope the lookup and the cache to the vnet, so guests
> don't get addresses of another vnet. The cache file now contains a
> `vnets` map next to `macs`. `macs` stays, since nodes not yet
> upgraded read and write only it, and keeps holding one address per
> MAC as before. Every write goes into both maps. A lookup answers
> from `vnets` as long as `macs` still names the same address.
> Otherwise it asks the IPAM within the vnet's subnets, since a node
> not yet upgraded changed or released the address in between. An
> entry only `macs` has is one from before, it answers for a vnet
> only with an address in the vnet's subnets. A MAC on two vnets can
> match `macs` on one of them only, so the other one asks the IPAM at
> every start. So an old entry answers only for the vnet it was
> written for. The one exception is the same or an overlapping range
> on two vnets, where an old entry answers for the other vnet too, as
> it did before, until that vnet has an entry of its own or the
> address is released.
> 
> Patch 9 adds separation by zone for the two external IPAM providers
> we support, since two zones can use the same range.
>  - NetBox: each zone gets its own `pve-<zone>` VRF, and the prefixes
>    of its subnets go in there
>  - phpIPAM: each zone gets a sub-section `pve-<zone>` under the
>    configured one, and its subnets go in there(only if the configured
>    section is not a sub-section already, phpIPAM only allows 1 deep
>    nesting). Section names are global in phpIPAM, so zone ids must
>    be unique across the clusters sharing one
> 
> Patch 10 is qemu-server's side. The vmlist names the source of a live
> migration until the config moves at its end, so an edit in that
> window updates the source. Every resume now pushes the current
> mapping of each NIC, and the migration's last step resumes the target
> after the move, so it catches up there. The patch needs the
> pve-network of this series. Taken from [1] as well.
> 
> Patch 11 just adds docs explaining this for the two external IPAMs.
> 
> This is not a breaking change, existing assignments are returned
> exactly like they are already. Cache entries from before keep their
> addresses, since nodes not yet upgraded still use them, and answer for
> a vnet with the addresses in its subnets, so a running guest keeps its
> reservation. Records in an external IPAM stay where they are, in
> NetBox's global table or in phpIPAM's configured section. Like the
> cache, an IPAM lookup first checks the zone's VRF or section for the
> subnet and falls back to the global table or the configured section.
> Unlike with the cache, migration can be done manually, by moving a
> prefix together with its addresses and IP ranges into the zone's VRF
> in NetBox, or a subnet into the zone's sub-section in phpIPAM, since
> the lookup checks those first. Migration is not necessary though.
> Creating a subnet creates the VRF or sub-section if it doesn't exist
> yet, nothing else does. A subnet whose range is still in the global
> table or the configured section is refused, since a copy would hide
> the records in it. The error says to move that prefix or subnet into
> the zone's VRF or sub-section, or to remove it. We don't move
> anything and we don't have to.
> If users are bothered by guests getting IPs not assigned to them, they
> can move the global entries into the `pve-<zone>` VRF or sub-section,
> where they'll be picked up correctly. While a rolling upgrade is
> under way, a node not yet upgraded puts an explicit mapping into
> NetBox's global table, also for a subnet created on an upgraded
> node, such addresses get moved like the ones from before.
> 
> note: I'll rebase [1] onto this, it'll lose some patches that moved
> here. With [1] things addressed here become more apparent/visible
> eventhough they are pre-existing, that is why [1] will be based on this.
> 
> [1] https://lore.proxmox.com/pve-devel/20261021084445.462195-1-h.laimer@proxmox.com/
> 
> v3:
>  - a start asks the IPAM when the MAC sits on two vnets and the cache
>    can't vouch for it
>  - ask the IPAM for the changed MAC before walking the reservations,
>    so its stale line is replaced
>  - phpIPAM: fail subnet creation on api error
>  - an API mapping edit fails when the reservation update does, the
>    records are written by then
> 
> v2:
>  - the cache and the lookup are kept per vnet, the old map stays
>    for nodes not yet upgraded
>  - entries from before are left as written, a fill is skipped when
>    the file changed meanwhile
>  - a release leaves a record that belongs to another MAC alone
>  - each zone gets its own VRF or section in the external IPAMs, and
>    phpIPAM gets the lookup by MAC
>  - an edit for a guest whose node is offline runs here, and the
>    reservation rewrite reads the cache once
>  - the target of a live migration catches up when the guest resumes
> 
> 
> pve-network:
> 
> Hannes Laimer (9):
>   sdn: ipam: stop caching IPAM misses
>   sdn: ipam: write the lookup's cache entry under the cluster lock
>   sdn: ipam: read the MAC cache case-insensitively
>   sdn: dhcp: replace a stale dnsmasq reservation instead of dropping it
>   sdn: ipam: release an address the built-in IPAM no longer holds
>   sdn: dhcp: update reservations when ip mappings change
>   sdn: ipam: look a MAC up within the vnet's subnets
>   sdn: ipam: keep the MAC cache per vnet
>   sdn: ipam: give each zone a VRF in NetBox and a section in phpIPAM
> 
>  src/PVE/API2/Network/SDN/Ips.pm               |  38 +-
>  src/PVE/Network/SDN/Dhcp.pm                   |  56 +-
>  src/PVE/Network/SDN/Dhcp/Dnsmasq.pm           | 153 +++--
>  src/PVE/Network/SDN/Dhcp/Plugin.pm            |   7 +-
>  src/PVE/Network/SDN/Ipams.pm                  | 180 ++++-
>  src/PVE/Network/SDN/Ipams/NetboxPlugin.pm     | 223 +++++--
>  src/PVE/Network/SDN/Ipams/PVEPlugin.pm        |  27 +-
>  src/PVE/Network/SDN/Ipams/PhpIpamPlugin.pm    | 237 ++++++-
>  src/PVE/Network/SDN/Ipams/Plugin.pm           |   7 +-
>  src/PVE/Network/SDN/Subnets.pm                |  14 +-
>  src/PVE/Network/SDN/Vnets.pm                  |  21 +-
>  src/test/ipams/netbox/expected.add_ip         |   2 +-
>  .../ipams/netbox/expected.add_ip_notgateway   |   2 +-
>  src/test/ipams/netbox/expected.add_subnet     |   2 +-
>  src/test/ipams/netbox/expected.update_ip      |   2 +-
>  src/test/run_test_ipams.pl                    | 468 +++++++++++++
>  src/test/run_test_vnets_blackbox.pl           | 618 +++++++++++++++++-
>  17 files changed, 1836 insertions(+), 221 deletions(-)
> 
> 
> qemu-server:
> 
> Hannes Laimer (1):
>   network: sdn: push the DHCP mappings of a guest when it resumes
> 
>  src/PVE/QemuServer/Network.pm  | 19 +++++++++++++++++++
>  src/PVE/QemuServer/RunState.pm |  2 ++
>  2 files changed, 21 insertions(+)
> 
> 
> pve-docs:
> 
> Hannes Laimer (1):
>   sdn: ipam: document the per-zone VRF and section in NetBox and phpIPAM
> 
>  pvesdn.adoc | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> 
> Summary over all repositories:
>   20 files changed, 1888 insertions(+), 221 deletions(-)
>