From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 3CF561FF0C1 for ; Wed, 26 Aug 2026 11:30:30 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 827A321640; Wed, 26 Aug 2026 11:29:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787667280; x=1788272080; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/gD+M69TRjLtpRIJlP4thzMOCb2aeknXa3mdvHJTw+0=; b=oxUGT7IX/RgoTKtO3aclb1tbxUczUgMj5ji97Fm+UwGmKArz3/f9Zq10C/wKExn5ap qgG3rMNxL2kcgl0VpUKS/CkvbP5hwsVROu5MCz0BKJHcYwunDpdo1DS53Sh96QTUCwGA h1GGR0lucf+LLvzXitseij1SCE/MO9w1im3XzZzYxWEH/t9PX0T3MUBnOE37CQkAUckZ PAOb5qUwl1xg9eqiTrDZd4/jxVY+rtYuiJaJrbkz2OHehl3gQx+pRE4AsHNQUoF5igec pxsIVNP2BwNXiZC1XYhST2B1VBArXEE3JjKCzZJEuw6YU6t9JvNsHOgF49cEUSjtXfDO tknQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787667280; x=1788272080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/gD+M69TRjLtpRIJlP4thzMOCb2aeknXa3mdvHJTw+0=; b=q5v/PSzrhUS6GEm4jlUAPUCHuQR722/VRxDAQSlOfcZ92NdVtBaAZqrgjve5wVdZt9 OR21SNiA+2IW+leedf7XDFmoNtKDd4f7hYCFfiJ5A6iiQ+MEXxWZFiWMuxUGcPaZO4BS AUU5y2BV5Kb/cjiTf6vN5cXO1AkSdxSkVqHpazdZERb7R94WBBkpYgXRDA94PkRi8cD3 JTgh+2bptSkpDprm67uz47YN1q4RQSuQ3vnzJqngODLc7+M8//tQjqJLfzdiV4pCMLz4 nNIQdyhNuafh0v+86SE/UPsna8Vloi0xe5FHXlaRx+T1/3FdUNn3vE5erbhTnN7mfgfU nE8w== X-Gm-Message-State: AFuF++mkNwXEdFa5duFmIYrK1R94sSQqt9LaRZMVcUli0J7lk+OtuI3H Qj86oS2KXh2U6EADMD+9hqD4AFXscdzmgWnUIEI5n1HUMRxF5+khrVTshXIDSopMYHA= X-Gm-Gg: AR+sD13OYXpdtFFcBn9fyRZXmA5A6ihrf0A2ZLsBijHYkh3rj+GZDllzsNEF0FwgblU 2LR88pgAJ4cnh2NWtAWCMvshrqul2/7SLS+k2emkIkDsXuNWnpwOL3bIMoN5rULgY+Wki23nStP 1wjEhQMEqzaOsGi5MoWTfDpZl/2EC+eZKmzMBjVWcjqETWDRFOPwLIyYHHsaG/fEpk10+SsdFAR ewWF9jc+dHwbRbkbvrSqYvi5bIw3cAuNvOo3TH+KlQSA3++CSdFtBVps5ZIcOvEeM29UwnW82Qw A0ScXjHL3qC79pPDxk64xWAVyN/1hDtoSBEWYTV/RyymbERDJK8UnyBIYR09xFUHeiIDkMcSkek JScZUbsyBTl1w9cbCgqM60HL75LBKN8+JYuWWZ9+LIagAByiigNvhP11yWGcXo5te2gEGlyqeAH W2mNm46cTbmg4+Cp8M0QHGnhfaJtDO2ucrIriyxRcCn1HxrXH6mjr85z3We2q84WVvYkJsHa+1Z S1ZDy+HYkEpA8THgBE7XFX/jFZ3aEf/ZI021Do288BrRN8K1ISn4SiRgnrcPUYcArKUI3gRMBOG rcacxvR6WlGbXzRvolE18CihEUoRHHiqt8g8JSQz X-Received: by 2002:a05:6402:3044:b0:6a5:d907:7703 with SMTP id 4fb4d7f45d1cf-6a5d907781bmr1810529a12.14.1787667279471; Tue, 25 Aug 2026 07:14:39 -0700 (PDT) From: Yao Xu X-Google-Original-From: Yao Xu To: pve-devel@lists.proxmox.com Subject: [PATCH pve-docs v2] qm: pci passthrough: note the AtomicOps caveat for multi-GPU guests Date: Tue, 25 Aug 2026 16:12:56 +0200 Message-ID: <20260825141257.80190-1-Xy2462381442@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260824170828.42821-1-Xy2462381442@gmail.com> References: <20260824170828.42821-1-Xy2462381442@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 1 AWL -0.000 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_ENVFROM_END_DIGIT 1 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider GB_FREEMAIL_NUM 0.75 Freemail spammy address RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: xy2462381442@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: CC3CA7LPTR5BHCNDDXTNHJ27L2KGMEC4 X-Message-ID-Hash: CC3CA7LPTR5BHCNDDXTNHJ27L2KGMEC4 X-Mailman-Approved-At: Wed, 26 Aug 2026 11:28:46 +0200 CC: Yao Xu X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The shortened ``00:02`' syntax is documented as a convenience. It also has a side effect that stays invisible until a guest workload needs PCIe AtomicOps, which in practice means multi-GPU collectives. QEMU adds AtomicOp completer support to the emulated root port only for a single-function device sitting below a root port that supports DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c). Passing every function of a card therefore leaves that port advertising AtomicOpsCap: 32bit- 64bit-, with no warning anywhere. The failure then surfaces several layers away from its cause. On AMD cards amdgpu logs "PCIE atomic ops is not supported" and RCCL collectives abort with "the operation cannot be performed in the present state", so it is easy to conclude that the hardware or the ROCm installation is at fault. The difference is an omitted .0 suffix. Bare metal is unaffected, and the function count is not what decides it there: pci_enable_atomic_ops_to_root() in drivers/pci/pci.c checks that the device is a PCIe endpoint, that the root port's DEVCAP2 advertises the requested completion widths, and that every bridge on the path routes AtomicOps without blocking egress. It never reads the device's function number. Extending QEMU's automatic path to multifunction devices was proposed in February 2026 and declined. QEMU can compose a guest multifunction package out of devices that are unrelated on the host, so it cannot infer device-to-device AtomicOps support, and the vfio interface reports capability relative to the root bus only; the maintainer's conclusion was that the burden belongs with VM builders and management tools rather than QEMU. The rationale is worth having to hand if this is ever revisited, so it is linked from the note as well as here: https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/ The guard is unchanged from v8.1.0, where the automatic path landed, through v11.1.0. Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT (gfx1100) passed to one q35 guest. With hostpci0: 0000:0b:00,pcie=1 hostpci1: 0000:44:00,pcie=1 both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs the message above for both cards, and a two-rank RCCL all_reduce fails. Appending .0 to both entries and changing nothing else gives 32bit+ 64bit+, no driver message, and the same collective completes. Reverting reproduces the failure. Signed-off-by: Yao Xu --- v2, after Dominik Csapak's review: - scope the note to multi-GPU workloads and name the QEMU version, so it does not read as something every passthrough user needs - drop the second patch, which appended .0 to the GPU example. Passing the card as it is on the host is the better default, and the note covers the case that needs otherwise - link the upstream discussion from the note itself as well as from the commit message, so the rationale is to hand if this is revisited - say that passing a single function is a trade-off: some guest drivers expect the card's other functions to be present. That caveat is Dominik's, from the review; I had not hit it qm-pci-passthrough.adoc | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index 00d9478..ed64661 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -338,6 +338,27 @@ you can pass them through all together with the shortened syntax ``00:02`'. This is equivalent with checking the ``All Functions`' checkbox in the web interface. +.Multi-GPU passthrough and PCIe AtomicOps +[NOTE] +==== +Workloads that use PCIe AtomicOps, in practice multi-GPU collectives such as +ROCm's RCCL, need the guest's virtual root port to advertise completer support. +QEMU up to 11.1 advertises it only for a *single-function* device, so passing +every function of a card leaves it unadvertised and the AMD driver logs +`PCIE atomic ops is not supported`. + +If you need it, pass the function explicitly, for example +``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The *host* root port above the +device must also support AtomicOp completion, which `lspci -vv` shows as +`AtomicOpsCap: 32bit+ 64bit+`. This is a trade-off rather than a better default: +some guest drivers expect the card's other functions to be present. + +Extending the automatic path to multifunction devices was +https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/[proposed upstream and declined] +in February 2026, because QEMU cannot see the host's PCIe routing and so cannot +decide which capability to advertise when a slot's functions disagree. +==== + There are some options to which may be necessary, depending on the device and guest OS: -- 2.50.1 (Apple Git-155)