[PATCH docs/manager/network/qemu-server v6 00/13] sdn: improve MAC cache, DHCP reservations and scope both to the vnet

[Hannes Laimer <h.laimer@proxmox.com>]

This series addresses some problems in the IPAM code, mostly around
caching. It also solves the problem that the MAC cache and the lookup
behind it held a single address per MAC. This is a problem since MACs
only have to be unique within a vnet and can repeat across vnets, and
it led to a guest getting the IP of a guest with the same MAC on
another vnet.

Patches 1-5 address general, mostly cache related problems. Patch 1
leaves entries without an address in the file and treats them as a
miss, patch 2 skips the fill when the file changed while the IPAM was
asked, and patch 3 leaves the cache keys as written, since nodes not
yet upgraded still use them. Patch 4 replaces a stale reservation
line instead of dropping it, and patch 5 lets the built-in IPAM
release an address it no longer holds, so its cache entry is cleared.

Patch 6 adds updating of the dnsmasq reservations when a mapping
changes, so dnsmasq picks the change up immediately and a guest gets
its new address without a restart. The edit runs on the guest's node
under its migration lock, so pve-network now depends on
pve-guest-common. The lock could move to a package below both if we
want that.

Patches 7-8 scope the lookup and the cache to the vnet, so guests
don't get addresses of another vnet. The cache file now contains a
`vnets` map next to `macs`. `macs` stays, since nodes not yet
upgraded read and write only it, and keeps holding one address per
MAC as before. Every write goes into both maps. A lookup answers
from `vnets` as long as `macs` still names the same address.
Otherwise it asks the IPAM within the vnet's subnets, since a write
for the MAC on another vnet or a node not yet upgraded changed or
released the address in between. An entry only `macs` has is one
from before, it answers for a vnet only with an address in the
vnet's subnets. A MAC with an address of the same family on two
vnets can match `macs` on one of them at most, so a start on the
other asks the IPAM. So an old entry answers only for the vnet it
was written for. The one exception is the same or an overlapping range
on two vnets, where an old entry answers for the other vnet too, as
it did before, until one of the two has an entry of its own or the
address is released.

Patch 9 adds separation by zone for the two external IPAM providers
we support, since two zones can use the same range. It is behind a
`zone-namespaces` option of the IPAM entry, off by default, to be
turned on once every node runs this version.
 - NetBox: each zone gets its own `pve-<zone>` VRF, and the prefixes
   of its subnets go in there
 - phpIPAM: each zone gets a sub-section `pve-<zone>` under the
   configured one, and its subnets go in there (the option is refused
   if the configured section is a sub-section already, phpIPAM only
   allows 1 deep nesting)
The names start with a `zone-namespace-prefix` of the IPAM entry,
`pve` by default, so clusters sharing an IPAM stay apart. Patch 9
also gives phpIPAM the lookup by MAC, which failed before.

Patch 10 is qemu-server's side. An edit and a migration of the guest
exclude each other since patch 6. An edit through a node not yet
upgraded still goes through during one, and it writes the IPAM and the
cache but no reservation. A resume now pushes the current mapping of
each NIC, and a migration resumes the target after the config moved,
so the target serves the current addresses. The patch needs the
pve-network of this series.

Patches 11-12 are pve-manager's side, the DHCP tree keeping the
subnets of two zones with one range apart, it merged them under the
first zone before, and the two options in the NetBox and phpIPAM
dialogs. Patch 12 needs the pve-network of this series.

Patch 13 just adds docs explaining this for the two external IPAMs.

This is not a breaking change, existing assignments stay as they are
and keep answering for their vnet. Cache entries from before keep
their addresses, since nodes not yet upgraded still use them, and
answer for a vnet with the addresses in its subnets, so a running
guest keeps its reservation. Records in an external IPAM stay where
they are, in NetBox's global table or in phpIPAM's configured section.
Like the cache, an IPAM lookup with the option on first checks the
zone's VRF or section for the subnet and falls back to the global
table or the configured section. Unlike with the cache, migration can
be done manually, by moving a prefix together with its addresses and
IP ranges into the zone's VRF in NetBox, or a subnet into the zone's
sub-section in phpIPAM, since the lookup checks those first. Migration
is not necessary though. With the option on, creating a subnet creates
the VRF or sub-section if it doesn't exist yet, nothing else does. A
subnet whose range is still in the global table or the configured
section is then refused, since a copy would hide the records in it.
The error says to move that prefix or subnet into the VRF or
sub-section of its zone, or to remove it. We don't move anything and
we don't have to. With the option off nothing changes in the external
IPAM, so a rolling upgrade is safe, and a node not yet upgraded never
meets a subnet in a VRF or sub-section.

v6:
 - never write entries lc'd into the old `macs`, this trips up
   not-yet-updated nodes
 - only ask ipam that errored out once in a pass
 - only have a shorter timeout for resume-pushes, the whole resume
   fast-path could leave a guest without a reservation. a too slow ipam
   now leads to an error in the migrations resume task, the guest runs
   fine though
 - with the zone-namespace flag off the external ipam lookups are as
   before again, first hit in any VRF or section, so manually moved
   entries are picked up without the flag
 - a resume-push writes the ipam's answer if the cache has nothing for
   the MAC
 - a resume-push failing on an ipam error fails the task too, after the
   guest continued
 - a failed update adds the old address back only if it was released
 - the pve-guest-common dependency is declared by the api package

v5:
 - also check the zone-namespace flag on external ipam lookup, so
   manually moved entries are not picked up without the flag set
 - for a dhcp update push on migration resume lock non-blocking,
   whatever write would be holding the lock writes the same line the
   push would have written, and ask the ipam if the cache can't vouch
   for the entry
 - phpipam: dont allow setting the zone-namespace flag if the configured
   section is a sub-section, phpipam only allows 1 deep nesting, and of
   the configured section is already 1 deep, we can't create the zone
   sections
 - refresh vmlist under lock
 - migrate resume asks ipam if cache has disagreeing entries

v4:
 - add IPAM config flag to use zone scoped layout, tried avoiding this,
   but that added much more problems than it solved
 - a mapping edit and a migration of the guest exclude each other
 - the resume of a guest pushes its reservations from the cache only
 - the same address on two vnets of a MAC is checked with the IPAM at
   every start
 - a prefix option of the IPAM entry names the zone namespaces, pve
   by default, so clusters sharing an IPAM stay apart
 - pve-manager: the two options in the dialogs, and the DHCP tree
   keeps the subnets of two zones with one range apart

v3:
 - a start asks the IPAM when the MAC sits on two vnets and the cache
   can't vouch for it
 - ask the IPAM for the changed MAC before walking the reservations,
   so its stale line is replaced
 - phpIPAM: fail subnet creation on api error
 - an API mapping edit fails when the reservation update does, the
   records are written by then

v2:
 - the cache and the lookup are kept per vnet, the old map stays
   for nodes not yet upgraded
 - entries from before are left as written, a fill is skipped when
   the file changed meanwhile
 - a release leaves a record that belongs to another MAC alone
 - each zone gets its own VRF or section in the external IPAMs, and
   phpIPAM gets the lookup by MAC
 - an edit for a guest whose node is offline runs here, and the
   reservation rewrite reads the cache once
 - the target of a live migration catches up when the guest resumes


pve-network:

Hannes Laimer (9):
  sdn: ipam: stop caching IPAM misses
  sdn: ipam: cache a MAC lookup's answer under the cluster lock
  sdn: ipam: read the MAC cache case-insensitively
  sdn: dhcp: replace a stale dnsmasq reservation instead of dropping it
  sdn: ipam: release an address the built-in IPAM no longer holds
  sdn: dhcp: update reservations when IP mappings change
  sdn: ipam: look a MAC up within the vnet's subnets
  sdn: ipam: keep the MAC cache per vnet
  sdn: ipam: give each zone a VRF in NetBox and a section in phpIPAM

 debian/control                             |   2 +
 src/PVE/API2/Network/SDN/Ips.pm            | 123 ++-
 src/PVE/Network/SDN/Dhcp.pm                |  52 +-
 src/PVE/Network/SDN/Dhcp/Dnsmasq.pm        | 191 +++--
 src/PVE/Network/SDN/Dhcp/Plugin.pm         |   8 +-
 src/PVE/Network/SDN/Ipams.pm               | 188 ++++-
 src/PVE/Network/SDN/Ipams/NetboxPlugin.pm  | 244 ++++--
 src/PVE/Network/SDN/Ipams/PVEPlugin.pm     |  27 +-
 src/PVE/Network/SDN/Ipams/PhpIpamPlugin.pm | 250 +++++-
 src/PVE/Network/SDN/Ipams/Plugin.pm        |  24 +-
 src/PVE/Network/SDN/Subnets.pm             |  14 +-
 src/PVE/Network/SDN/Vnets.pm               |  21 +-
 src/test/run_test_ipams.pl                 | 620 +++++++++++++++
 src/test/run_test_vnets_blackbox.pl        | 870 ++++++++++++++++++++-
 14 files changed, 2405 insertions(+), 229 deletions(-)


qemu-server:

Hannes Laimer (1):
  network: sdn: push the DHCP mappings of a guest when it resumes

 src/PVE/QemuServer/Network.pm  | 18 ++++++++++++++++++
 src/PVE/QemuServer/RunState.pm |  8 ++++++++
 2 files changed, 26 insertions(+)


pve-manager:

Hannes Laimer (2):
  ui: sdn: dhcp tree: do not merge the same subnet in different zones
  ui: sdn: ipam: add the zone namespace options

 www/manager6/sdn/ipams/Base.js        | 37 +++++++++++++++++++++++++++
 www/manager6/sdn/ipams/NetboxEdit.js  |  4 +++
 www/manager6/sdn/ipams/PhpIpamEdit.js |  4 +++
 www/manager6/tree/DhcpTree.js         |  8 +++---
 4 files changed, 50 insertions(+), 3 deletions(-)


pve-docs:

Hannes Laimer (1):
  sdn: ipam: document zone namespaces and the mapping edit

 pvesdn.adoc | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)


Summary over all repositories:
  21 files changed, 2524 insertions(+), 233 deletions(-)

-- 
Generated by murpp 0.12.0