From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 406CA1FF0DE for ; Thu, 19 Nov 2026 13:33:27 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2AF4C216BB; Thu, 19 Nov 2026 13:33:26 +0100 (CET) Message-ID: <80c9476c-75fb-4539-b397-e7ff165f87ce@proxmox.com> Date: Thu, 19 Nov 2026 13:33:22 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Subject: Re: [PATCH manager/qemu-server v5 00/13] pci-handling rewrite To: Elias Huhsovitz , pve-devel@lists.proxmox.com References: <20261118152803.3843758-1-d.csapak@proxmox.com> Content-Language: en-US From: Dominik Csapak In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795091602228 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.275 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: HP3ZJYGJQJWE4WSK3YN45IJU2RZAP7DQ X-Message-ID-Hash: HP3ZJYGJQJWE4WSK3YN45IJU2RZAP7DQ X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Thanks for the review! On 11/19/26 1:18 PM, Elias Huhsovitz wrote: > Some comments inline. > > On Wed Nov 18, 2026 at 4:27 PM CET, Dominik Csapak wrote: >> # Motivation >> [snip] >> ## Phase 4 - Host and NUMA layouts (the next part) > > Isn't this Phase 4 (this part) Or is there more coming? > yes, this was a wrong copy/paste from an older cover letter >> >> With all the pieces in place, we can then easily extend the new layout by PCI >> switches that are needed for assigning to NUMA nodes and replicating the host >> topology for passed through devices. >> >> This is a separate setting on top of the grouped layout instead of more >> layouts: >> >> machine: q35,pci-layout=grouped,pci-topology= >> >> * flat (default): every hostpci device with pcie=1 gets its own root port on >> the root bus >> * numa: adds a pxb-pcie expander bridge per guest NUMA node and puts the root >> ports of that node's devices behind it >> * host: additionally replicates the host PCIe switches that more than one >> passed through device is behind >> >> The expander bridges go into slots of the grouped layout that were unused >> before, so enabling a topology does not move any of the other devices. >> qemu-server reads the NUMA node and the bridges of each device from sysfs, so >> nothing has to be configured per device. >> >> Since the placement depends on the hardware of the host, live migration is >> prevented for all topologies except 'flat'. > > Wouldnt it theoretically be possible support live migration for nearly > identical host hardware, e.g. for datacenter use cases. > > Or are there potential pitfalls in comparing topology between hosts? > The topology provided by tools like lostop (hwloc) seem consistent > enough. > yes it would be possible, but in reality, live migration with pci passthrough only works with a very select number of hardware (i only know of nvidia vgpu and some nics) while the intended way of using this would imply the user passing through multiple devices (disks, whole gpus, etc.) and that would prevent this anyway my though process was that allowing later is easy, so let's wait until actual users want this >> >> The pve-manager patch only makes sure that the two new options don't get lost >> when editing the machine in the UI, they are not exposed there (yet). >> >> # Notes >> >> * Neither 'pci-layout' nor 'pci-topology' is bound to a (new) machine >> version, both are opt-in via the machine property and work with every >> machine version. That means that all nodes in a cluster must know the >> options before they are used, since older nodes can't parse the machine >> property with them. >> * The options are only checked when starting the guest, not when setting >> them, e.g. a non-flat topology with the legacy layout or without q35 only >> fails on start. Could be added later still though. > > IMO this is the right approach. Otherwise you would have to apply > settings in the same "time" step to avoid errors, i.e. when do you know > that configuration has finished and it is safe to validate the config? > I guess it would be possible, but implementing this cleanly would be a > maintenance from my understanding. > >> * For the 'numa' and 'host' topologies, the guest to host NUMA node mapping >> is taken from the 'numaN' entries. If there are none, we assume that guest >> node N is on host node N. Not sure if we should require an explicit >> binding here, opinions welcome. (Similar for devices where the host does >> not report a NUMA node, these currently count as node 0.) > > I think the fix here is to make our numa handling more robust in case > no/incomplete numa[N] entires are provided. IMO if numa is enabled but > the config is incomplete we should complete the numa[N] automatically > internally. > > As you mentioned we can make reasonable assumptions on what these values > should be: > * cpus > * hostnodes > * memory > > Generally idea on vm start: > If no numa[N] is provided: > generate a sensible perl hash based on the sockets/cores and host > nodes. > If numa[N] is povided but incomplete: > fill in the gaps (missing cpus,hostnodes,memory) > > Make the perl hash available to other callers (e.g. pci), so everyone operates > on a consistent understanding of the numa config is. > > (AFAIK this is already somewhat implemented but the generated config is > directly passed to the qemu command and no "completed" layout is > provided to other callers) > > I was thinking of doing something like this for the cpu pinning serires. ok but this is somewhat orthogonal to the series here and can always improved later on too > >> * The bus numbers behind the expander bridges can shift when hostpci devices >> are added or removed. Since hostpci devices can't be hotplugged and the >> topologies prevent live migration, this only happens on a fresh start. >> * Hostpci devices with pcie=1 now always get a root port on function 0 of >> their slot, also for existing machine versions, see the commit message of >> 'pci: legacy: generate q35 chipset devices and hostpcie root ports'. >> >> changes from v4 (most are packaging related, not code): >> * fixed the hotplug allowed check for usb >> * move the crate into the qemu-server repo as 'pve-qemu-rs' workspace >> * rename the crate from 'pve-qemu-server-pci' to 'pve-qemu-pci' >> * move the perl bindings from proxmox-perl-rs into a qemu-server perlmod library >> * rename the bindings from 'PVE::RS::*' to 'PVE::QemuServer::RS::*' >> * use an own 'pve-qemu' lib tag for the library >> * add the bindings together with the crate features they expose >> * tests use the built library directly, no install needed >> * new build-dependencies: cargo, rustc, dh-cargo, perlmod-bin, librust-*-dev > > Side Note: I gave this a quick test by building+installing the package > and doing some basic PCI operations. So this seems fine so far. Thanks!