From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4B35A9C18E for ; Mon, 23 Oct 2023 14:52:52 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2352F13DB6 for ; Mon, 23 Oct 2023 14:52:52 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 23 Oct 2023 14:52:51 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E21B34445D for ; Mon, 23 Oct 2023 14:52:50 +0200 (CEST) From: Stefan Lendl To: pve-devel@lists.proxmox.com In-Reply-To: <875y2xab2d.fsf@gmail.com> References: <875y2xab2d.fsf@gmail.com> Date: Mon, 23 Oct 2023 14:52:49 +0200 Message-ID: <87sf61bivy.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-SPAM-LEVEL: Spam detection results: 0 AWL -0.463 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemuserver.pm, vnets.pm, subnetplugin.pm, dhcp.pm, plugin.pm, subnets.pm, network.pm, pveplugin.pm, cluster.pm, lxc.pm, sdn.pm, proxmox.com, dnsmasq.pm] Subject: Re: [pve-devel] [WIP v2 cluster/network/manager/qemu-server/container 00/10] Add support for DHCP servers to SDN X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Oct 2023 12:52:52 -0000 PS: Sorry for double posting. this mail was sent with invalid In-Reply-To and References Headers. I sent it again with the correct Headers after I managed to correctly setup my mail client. Stefan Lendl writes: > I am currently working on the SDN feature. This is an initial review of > the patch series and I am trying to make a strong case against ephemeral > DHCP IP reservation. > > The current state of the patch series invokes the IPAM on every VM/CT > start/stop to add or remove the IP from the IPAM. > This triggers the dnsmasq config generation on the specific host with > only the MAC/IP mapping of that particular host. > > From reading the discussion of the v1 patch series I understand this > approach tries to implement the ephemeral IP reservation strategy. From > off-list conversations with Stefan Hanreich, I agree that having > ephemeral IP reservation coordinated by the IPAM requires us to > re-implement DHCP functionality in the IPAM and heavily rely on syncing > between the different services. > > To maintain reliable sync we need to hook into many different places > where the IPAM need to be queried. Any issues with the implementation > may lead to IPAM and DHCP local config state running out of sync causing > network issues duplicate multiple IPs. > > Furthermore, every interaction with the IPAM requires a cluster-wide > lock on the IPAM. Having a central cluster-wide lock on every VM > start/stop/migrate will significantly limit parallel operations. Event > starting two VMs in parallel will be limited by this central lock. At > boot trying to start many VMs (ideally as much in parallel as possible) > is limited by the central IPAM lock even further. > > I argue that we shall not support ephemeral IPs altogether. > The alternative is to make all IPAM reservations persistent. > > Using persistent IPs only reduces the interactions of VM/CTs with the > IPAM to a minimum of NIC joining a subnet and NIC leaving a subnet. I am > deliberately not referring to VMs because a VM may be part of multiple > VNets or even multiple times in the same VNet (regardless if that is > sensible). > > Cases the IPAM needs to be involved: > > - NIC with DHCP enabled VNet is added to VM config > - NIC with DHCP enabled VNet is removed from VM config > - NIC is assigned to another Bridge > can be treated as individual leave + join events > > Cases that are explicitly not covered but may be added if desired: > > - Manually assign an IP address on a NIC > will not be automatically visible in the IPAM > - Manually change the MAC on a NIC > don't do that > you are on your own. > Not handled > change in IPAM manually > > Once an IP is reserved via IPAM, the dnsmasq config can be generated > stateless and idempotent from the pve IPAM and is identical on all nodes > regardless if a VM/CT actually resides on that node or is running or > stopped. This is especially useful for VM migration because the IP > stays consistent without spacial considering. > > Snapshot/revert, backup/restore, suspend/hibernate/resume cases are > automatically covered because the IP will already be reserved for that > MAC. > > If the admin wants to change, the IP of a VM this can be done via the > IPAM API/UI which will have to be implemented separately. > > A limitation of this approach vs dynamic IP reservation is that the IP > range on the subnet needs to be large enough to hold all IPs of all, > even stopped, VMs in that subnet. This is in contrast to default DHCP > functionality where only the number of actively running VMs is limited. > It should be enough to mention this in the docs. > > I will further review the code an try to implement the aforementioned > approach. > > Best regards, > Stefan Lendl > > Stefan Hanreich writes: > >> This is a WIP patch series, since I will be gone for 3 weeks and wanted to >> share my current progress with the DHCP support for SDN. >> >> This patch series adds support for automatically deploying dnsmasq as a DHCP >> server to a simple SDN Zone. >> >> While certainly not 100% polished on some ends (looking at restarting systemd >> services in particular), the general idea behind the mechanism shows. I wanted >> to gather some feedback on how I approached designing the plugins and the >> config regeneration process before comitting to this design by creating an API >> and UI around it. >> >> You need to install dnsmasq (and disable it afterwards): >> >> apt install dnsmasq && systemctl disable --now dnsmasq >> >> >> You can use the following example configuration for deploying a DHCP server in >> a SDN subnet: >> >> /etc/pve/sdn/dhcp.cfg: >> >> dnsmasq: nat >> >> >> /etc/pve/sdn/zones.cfg: >> >> simple: DHCPNAT >> ipam pve >> >> >> /etc/pve/sdn/vnets.cfg: >> >> vnet: dhcpnat >> zone DHCPNAT >> >> >> /etc/pve/sdn/subnets.cfg: >> >> subnet: DHCPNAT-10.1.0.0-16 >> vnet dhcpnat >> dhcp-dns-server 10.1.0.1 >> dhcp-range server=nat,start-address=10.1.0.100,end-address=10.1.0.200 >> gateway 10.1.0.1 >> snat 1 >> >> >> Then apply the SDN configuration: >> >> pvesh set /cluster/sdn >> >> You need to apply the SDN configuration once after adding the dhcp-range lines >> to the configuration, since the running configuration is used for managing >> DHCP. It will not work otherwise! >> >> For testing it can be helpful to monitor the following files (e.g. with watch) >> to find out what is happening >> * /etc/dnsmasq.d//ethers (on each node) >> * /etc/pve/priv/ipam.db >> >> Changes from v1 -> v2: >> * added hooks for handling DHCP when starting / stopping / .. VMs and CTs >> * Get an IP from IPAM and register that IP in the DHCP server >> (pve only for now) >> * remove lease-time, since it is now infinite and managed by the VM lifecycle >> * add hooks for setting & deleting DHCP mappings to DHCP plugins >> * modified interface of the abstract class to reflect new requirements >> * added helpers in existing SDN classes >> * simplified DHCP configuration settings >> >> >> >> pve-cluster: >> >> Stefan Hanreich (1): >> cluster files: add dhcp.cfg >> >> src/PVE/Cluster.pm | 1 + >> src/pmxcfs/status.c | 1 + >> 2 files changed, 2 insertions(+) >> >> >> pve-network: >> >> Stefan Hanreich (6): >> subnets: vnets: preparations for DHCP plugins >> dhcp: add abstract class for DHCP plugins >> dhcp: subnet: add DHCP options to subnet configuration >> dhcp: add DHCP plugin for dnsmasq >> ipam: Add helper methods for DHCP to PVE IPAM >> dhcp: regenerate config for DHCP servers on reload >> >> debian/control | 1 + >> src/PVE/Network/SDN.pm | 11 +- >> src/PVE/Network/SDN/Dhcp.pm | 192 +++++++++++++++++++++++++ >> src/PVE/Network/SDN/Dhcp/Dnsmasq.pm | 186 ++++++++++++++++++++++++ >> src/PVE/Network/SDN/Dhcp/Makefile | 8 ++ >> src/PVE/Network/SDN/Dhcp/Plugin.pm | 83 +++++++++++ >> src/PVE/Network/SDN/Ipams/PVEPlugin.pm | 64 +++++++++ >> src/PVE/Network/SDN/Makefile | 3 +- >> src/PVE/Network/SDN/SubnetPlugin.pm | 32 +++++ >> src/PVE/Network/SDN/Subnets.pm | 43 ++++-- >> src/PVE/Network/SDN/Vnets.pm | 27 ++-- >> 11 files changed, 622 insertions(+), 28 deletions(-) >> create mode 100644 src/PVE/Network/SDN/Dhcp.pm >> create mode 100644 src/PVE/Network/SDN/Dhcp/Dnsmasq.pm >> create mode 100644 src/PVE/Network/SDN/Dhcp/Makefile >> create mode 100644 src/PVE/Network/SDN/Dhcp/Plugin.pm >> >> >> pve-manager: >> >> Stefan Hanreich (1): >> sdn: regenerate DHCP config on reload >> >> PVE/API2/Network.pm | 1 + >> 1 file changed, 1 insertion(+) >> >> >> qemu-server: >> >> Stefan Hanreich (1): >> sdn: dhcp: add DHCP setup to vm-network-scripts >> >> PVE/QemuServer.pm | 14 ++++++++++++++ >> vm-network-scripts/pve-bridge | 3 +++ >> vm-network-scripts/pve-bridgedown | 19 +++++++++++++++++++ >> 3 files changed, 36 insertions(+) >> >> >> pve-container: >> >> Stefan Hanreich (1): >> sdn: dhcp: setup DHCP mappings in LXC hooks >> >> src/PVE/LXC.pm | 10 ++++++++++ >> src/lxc-pve-poststop-hook | 1 + >> src/lxc-pve-prestart-hook | 9 +++++++++ >> 3 files changed, 20 insertions(+) >> >> >> Summary over all repositories: >> 20 files changed, 681 insertions(+), 28 deletions(-) >> >> -- >> murpp v0.4.0 >> >> >> _______________________________________________ >> pve-devel mailing list >> pve-devel@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel