[PATCH pve-docs v2] qm: pci passthrough: note the AtomicOps caveat for multi-GPU guests

[Yao Xu <xy2462381442@gmail.com>]

The shortened ``00:02`' syntax is documented as a convenience. It also
has a side effect that stays invisible until a guest workload needs
PCIe AtomicOps, which in practice means multi-GPU collectives.

QEMU adds AtomicOp completer support to the emulated root port only
for a single-function device sitting below a root port that supports
DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c). Passing every
function of a card therefore leaves that port advertising
AtomicOpsCap: 32bit- 64bit-, with no warning anywhere.

The failure then surfaces several layers away from its cause. On AMD
cards amdgpu logs "PCIE atomic ops is not supported" and RCCL
collectives abort with "the operation cannot be performed in the
present state", so it is easy to conclude that the hardware or the
ROCm installation is at fault. The difference is an omitted .0 suffix.

Bare metal is unaffected, and the function count is not what decides
it there: pci_enable_atomic_ops_to_root() in drivers/pci/pci.c checks
that the device is a PCIe endpoint, that the root port's DEVCAP2
advertises the requested completion widths, and that every bridge on
the path routes AtomicOps without blocking egress. It never reads the
device's function number.

Extending QEMU's automatic path to multifunction devices was proposed
in February 2026 and declined. QEMU can compose a guest multifunction
package out of devices that are unrelated on the host, so it cannot
infer device-to-device AtomicOps support, and the vfio interface
reports capability relative to the root bus only; the maintainer's
conclusion was that the burden belongs with VM builders and management
tools rather than QEMU. The rationale is worth having to hand if this
is ever revisited, so it is linked from the note as well as here:

https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/

The guard is unchanged from v8.1.0, where the automatic path landed,
through v11.1.0.

Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT
(gfx1100) passed to one q35 guest. With

    hostpci0: 0000:0b:00,pcie=1
    hostpci1: 0000:44:00,pcie=1

both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs
the message above for both cards, and a two-rank RCCL all_reduce
fails. Appending .0 to both entries and changing nothing else gives
32bit+ 64bit+, no driver message, and the same collective completes.
Reverting reproduces the failure.

Signed-off-by: Yao Xu <Xy2462381442@gmail.com>
---
v2, after Dominik Csapak's review:
 - scope the note to multi-GPU workloads and name the QEMU version, so it
   does not read as something every passthrough user needs
 - drop the second patch, which appended .0 to the GPU example. Passing the
   card as it is on the host is the better default, and the note covers the
   case that needs otherwise
 - link the upstream discussion from the note itself as well as from the
   commit message, so the rationale is to hand if this is revisited
 - say that passing a single function is a trade-off: some guest drivers
   expect the card's other functions to be present. That caveat is Dominik's,
   from the review; I had not hit it

 qm-pci-passthrough.adoc | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index 00d9478..ed64661 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -338,6 +338,27 @@ you can pass them through all together with the shortened syntax ``00:02`'.
 This is equivalent with checking the ``All Functions`' checkbox in the
 web interface.
 
+.Multi-GPU passthrough and PCIe AtomicOps
+[NOTE]
+====
+Workloads that use PCIe AtomicOps, in practice multi-GPU collectives such as
+ROCm's RCCL, need the guest's virtual root port to advertise completer support.
+QEMU up to 11.1 advertises it only for a *single-function* device, so passing
+every function of a card leaves it unadvertised and the AMD driver logs
+`PCIE atomic ops is not supported`.
+
+If you need it, pass the function explicitly, for example
+``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The *host* root port above the
+device must also support AtomicOp completion, which `lspci -vv` shows as
+`AtomicOpsCap: 32bit+ 64bit+`. This is a trade-off rather than a better default:
+some guest drivers expect the card's other functions to be present.
+
+Extending the automatic path to multifunction devices was
+https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/[proposed upstream and declined]
+in February 2026, because QEMU cannot see the host's PCIe routing and so cannot
+decide which capability to advertise when a slot's functions disagree.
+====
+
 There are some options to which may be necessary, depending on the device
 and guest OS:
 
-- 
2.50.1 (Apple Git-155)