From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 606071FF0A7 for ; Wed, 02 Sep 2026 10:32:50 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 34E052157C; Wed, 02 Sep 2026 10:32:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788181678; x=1788786478; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YwtX8BUQGU20ILQ3Isx/342nvDlQ49ZbarPRzDpB6+4=; b=Iaxw0v1owP484CUTtEq+Kz8dstLK8psyNS6Nx4WJkaLC5XsbRbDabUxlA4n0AfaGlm S+HGkjUmwWY8R65hbkxOwm/oQMzy59fjVggY92j1tP43yqF5ncmNVRxi6vGaH9RkZuMh od/+r2TmpgaRKpqL2X+Vd3w93IaxjimUKdLO+ljuqPVpq5Bz5Y20wkrrx3isVKr8u+wG 5lJ4J5Ziof10lakUhyV3nmJ73qIrbQRPtOR5JrgVfKdAosWbG43MzqPwDIE9dzvrnfHf ZXA7ZRl9A6/TmZdgCXlMgWX0+HZ+hkbOGchgaYr92CgeJl1XMTqUcbcX3PWs36j0kcsk nrDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788181678; x=1788786478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YwtX8BUQGU20ILQ3Isx/342nvDlQ49ZbarPRzDpB6+4=; b=dX8EqSOAGA5jgb3kVr9Bi2+qBZegws5MXHgK5Gn5I7p6sLWD3Ycr5YVfGd1YDRei2V VyumAToQ3gzBzmuuPP17Tc8BisnNf+X85JhdpExpLQO2Q85T7qx997zoH43jQll6m93v pu7smG/e4bW0Nc8M6t0V+g92g8suTjZ69Wcy4Uqu7XrzUoXLRmq89ksiZBX0WRclUj0C DO1ZqGY0mgiTkUKS3lJCPUZ/2GhDKT1gwjERvYPJXdDPltxCs8Oxg19ouk7yHKSM3mx7 Gr7yCWSxbHF2yHk6EjQBdLA7z9epd/2DF+Fgv2n8dWT3Lm06dc3TJjRIXndiQ5m7lY04 /wwA== X-Gm-Message-State: AFuF++liZvcFFAzBkhIkf826j84IDdXrtDD/Njqe00cRc4K0sitRbRzN odYxdy41BnFG6STQ17+C8yBHjS3HTWqUCegRu/lFffypfShbhd33ZiKg9fDM2G0SBws= X-Gm-Gg: AR+sD125Bgg4jZc8I8pki67X6QdsrtrWzhPuTgjzoB25g5BXoEVyVfrQ+jSDWYoBb7o vs5m/Zr/v1OLAGm19OwP5h9P7chRyI+1OIUqPJVy2kNNHvGayQlkSLFw8veZXSh3bmcTSrw5FT5 QhO7qtz/fe45OSsTMuDCP2fLGlrHgr24ltUq7yp42xTBAmIRMr7Ybk2M//VHxEIdnp9AyLap06l vq+jeuVCTtLfieOcySfJ7fpOD7QhojxpZ/GdyHmDoXyJIqIiR4PggILG9aX+kMa/aEqpwFB9jJO YMMH6uE3pJ9TsRbd2cHF1o5Q737E6+gRTXqkLu722N+HC46HJ1Cj+P+HnwF1/4e4xAa+xK36cnH U3NHFGdDB2Y3bDxpORXtcKCbiYpibSmRjDo1b3GDrD9+hI5VVRhjtkY9qfmqYkH2e+Aj5oqQCn3 UquQ7Y7cFw2b3abEwv3eNJIASxEhEMu8vsRTdDeOiXLa33IY506UuOV0taq/v0mPcIqz1D4pPfb dD2WFrILSAJlaN4XRaAXd9C047Aa0puPfmD+XTk9CokPOhNYp+ZKamLUbh+gDRKBzpTTJOKNNtq 9Ni3WwAmomrcJgl5gTB1oCZGr2inp/Usms8gbdMp X-Received: by 2002:a17:907:94c1:b0:c25:6230:dfcd with SMTP id a640c23a62f3a-c256230e4d2mr1291663966b.15.1788181677458; Mon, 31 Aug 2026 06:07:57 -0700 (PDT) From: Yao Xu X-Google-Original-From: Yao Xu To: pve-devel@lists.proxmox.com Subject: [PATCH pve-docs v3] qm: pci passthrough: note the AtomicOps caveat for multi-GPU guests Date: Mon, 31 Aug 2026 15:07:52 +0200 Message-ID: <20260831130752.37364-1-Xy2462381442@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 1 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_ENVFROM_END_DIGIT 1 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider GB_FREEMAIL_NUM 0.75 Freemail spammy address RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: xy2462381442@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: SUJ4BCZG4RQCYLTGQYJXEY63WCPM5G6L X-Message-ID-Hash: SUJ4BCZG4RQCYLTGQYJXEY63WCPM5G6L X-Mailman-Approved-At: Wed, 02 Sep 2026 10:32:21 +0200 CC: Dominik Csapak , Yao Xu , Yao Xu X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Yao Xu The shortened ``00:02`' syntax is documented as a convenience. It also has a side effect that stays invisible until a guest workload needs PCIe AtomicOps, which in practice means multi-GPU collectives. QEMU adds AtomicOp completer support to the emulated root port only for a single-function device sitting below a root port that supports DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c). Passing every function of a card therefore leaves that port advertising AtomicOpsCap: 32bit- 64bit-, with no warning anywhere. The failure then surfaces several layers away from its cause. On AMD cards amdgpu logs "PCIE atomic ops is not supported" and RCCL collectives abort with "the operation cannot be performed in the present state", so it is easy to conclude that the hardware or the ROCm installation is at fault. The difference is an omitted .0 suffix. Bare metal is unaffected, and the function count is not what decides it there: pci_enable_atomic_ops_to_root() in drivers/pci/pci.c checks that the device is a PCIe endpoint, that the root port's DEVCAP2 advertises the requested completion widths, and that every bridge on the path routes AtomicOps without blocking egress. It never reads the device's function number. Extending QEMU's automatic path to multifunction devices was proposed in February 2026 and declined. QEMU can compose a guest multifunction package out of devices that are unrelated on the host, so it cannot infer device-to-device AtomicOps support, and the vfio interface reports capability relative to the root bus only; the maintainer's conclusion was that the burden belongs with VM builders and management tools rather than QEMU. The rationale is worth having to hand if this is ever revisited, so it is linked from the note as well as here: https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/ The guard is unchanged from v8.1.0, where the automatic path landed, through v11.1.0. Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT (gfx1100) passed to one q35 guest. With hostpci0: 0000:0b:00,pcie=1 hostpci1: 0000:44:00,pcie=1 both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs the message above for both cards, and a two-rank RCCL all_reduce fails. Appending .0 to both entries and changing nothing else gives 32bit+ 64bit+, no driver message, and the same collective completes. Reverting reproduces the failure. Signed-off-by: Yao Xu --- qm-pci-passthrough.adoc | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index 00d9478..f42ba24 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -338,6 +338,27 @@ you can pass them through all together with the shortened syntax ``00:02`'. This is equivalent with checking the ``All Functions`' checkbox in the web interface. +.Multi-GPU passthrough and PCIe AtomicOps +[NOTE] +==== +Workloads that use PCIe AtomicOps, in practice multi-GPU collectives such as +ROCm's RCCL, need the guest's virtual root port to advertise completer support. +QEMU advertises it only for a *single-function* device, so passing every +function of a card leaves it unadvertised and the AMD driver logs +`PCIE atomic ops is not supported`. + +If you need it, pass the function explicitly, for example +``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The *host* root port above the +device must also support AtomicOp completion, which `lspci -vv` shows as +`AtomicOpsCap: 32bit+ 64bit+`. This is a trade-off rather than a better default: +some guest drivers expect the card's other functions to be present. + +Extending the automatic path to multifunction devices was +https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/[proposed upstream and declined] +in February 2026, because QEMU cannot see the host's PCIe routing and so cannot +decide which capability to advertise when a slot's functions disagree. +==== + There are some options to which may be necessary, depending on the device and guest OS: -- 2.50.1 (Apple Git-155)