Re: [PATCH pve-docs v2] qm: pci passthrough: note the AtomicOps caveat for multi-GPU guests

[Dominik Csapak <d.csapak@proxmox.com>]

thanks for the v2,

I'm still a bit torn if that should go into the docs,
or if it should just live in the pci passthrough troubleshooting
wiki[0] instead. I'll leave that questions for the pve-docs
maintainers ;)

aside from that, one comment inline

0: https://pve.proxmox.com/wiki/PCI_Passthrough

On 8/25/26 4:14 PM, Yao Xu wrote:
> The shortened ``00:02`' syntax is documented as a convenience. It also
> has a side effect that stays invisible until a guest workload needs
> PCIe AtomicOps, which in practice means multi-GPU collectives.
> 
> QEMU adds AtomicOp completer support to the emulated root port only
> for a single-function device sitting below a root port that supports
> DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c). Passing every
> function of a card therefore leaves that port advertising
> AtomicOpsCap: 32bit- 64bit-, with no warning anywhere.
> 
> The failure then surfaces several layers away from its cause. On AMD
> cards amdgpu logs "PCIE atomic ops is not supported" and RCCL
> collectives abort with "the operation cannot be performed in the
> present state", so it is easy to conclude that the hardware or the
> ROCm installation is at fault. The difference is an omitted .0 suffix.
> 
> Bare metal is unaffected, and the function count is not what decides
> it there: pci_enable_atomic_ops_to_root() in drivers/pci/pci.c checks
> that the device is a PCIe endpoint, that the root port's DEVCAP2
> advertises the requested completion widths, and that every bridge on
> the path routes AtomicOps without blocking egress. It never reads the
> device's function number.
> 
> Extending QEMU's automatic path to multifunction devices was proposed
> in February 2026 and declined. QEMU can compose a guest multifunction
> package out of devices that are unrelated on the host, so it cannot
> infer device-to-device AtomicOps support, and the vfio interface
> reports capability relative to the root bus only; the maintainer's
> conclusion was that the burden belongs with VM builders and management
> tools rather than QEMU. The rationale is worth having to hand if this
> is ever revisited, so it is linked from the note as well as here:
> 
> https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/
> 
> The guard is unchanged from v8.1.0, where the automatic path landed,
> through v11.1.0.
> 
> Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT
> (gfx1100) passed to one q35 guest. With
> 
>      hostpci0: 0000:0b:00,pcie=1
>      hostpci1: 0000:44:00,pcie=1
> 
> both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs
> the message above for both cards, and a two-rank RCCL all_reduce
> fails. Appending .0 to both entries and changing nothing else gives
> 32bit+ 64bit+, no driver message, and the same collective completes.
> Reverting reproduces the failure.
> 
> Signed-off-by: Yao Xu <Xy2462381442@gmail.com>
> ---
> v2, after Dominik Csapak's review:
>   - scope the note to multi-GPU workloads and name the QEMU version, so it
>     does not read as something every passthrough user needs
>   - drop the second patch, which appended .0 to the GPU example. Passing the
>     card as it is on the host is the better default, and the note covers the
>     case that needs otherwise
>   - link the upstream discussion from the note itself as well as from the
>     commit message, so the rationale is to hand if this is revisited
>   - say that passing a single function is a trade-off: some guest drivers
>     expect the card's other functions to be present. That caveat is Dominik's,
>     from the review; I had not hit it
> 
>   qm-pci-passthrough.adoc | 21 +++++++++++++++++++++
>   1 file changed, 21 insertions(+)
> 
> diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
> index 00d9478..ed64661 100644
> --- a/qm-pci-passthrough.adoc
> +++ b/qm-pci-passthrough.adoc
> @@ -338,6 +338,27 @@ you can pass them through all together with the shortened syntax ``00:02`'.
>   This is equivalent with checking the ``All Functions`' checkbox in the
>   web interface.
>   
> +.Multi-GPU passthrough and PCIe AtomicOps
> +[NOTE]
> +====
> +Workloads that use PCIe AtomicOps, in practice multi-GPU collectives such as
> +ROCm's RCCL, need the guest's virtual root port to advertise completer support.
> +QEMU up to 11.1 advertises it only for a *single-function* device, so passing

reading this now, I'd leave the QEMU version off, since it's soon
outdated and could be interpreted that this is "fixed" with later
QEMU versions.

Just saying 'QEMU advertises ...' should be enough.

(i know I suggested such a wording, but revisiting that a few days later
and reading it with fresh mind leads me to believe we should rather
not mention the current version here)

> +every function of a card leaves it unadvertised and the AMD driver logs
> +`PCIE atomic ops is not supported`.
> +
> +If you need it, pass the function explicitly, for example
> +``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The *host* root port above the
> +device must also support AtomicOp completion, which `lspci -vv` shows as
> +`AtomicOpsCap: 32bit+ 64bit+`. This is a trade-off rather than a better default:
> +some guest drivers expect the card's other functions to be present.
> +
> +Extending the automatic path to multifunction devices was
> +https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/[proposed upstream and declined]
> +in February 2026, because QEMU cannot see the host's PCIe routing and so cannot
> +decide which capability to advertise when a slot's functions disagree.
> +====
> +
>   There are some options to which may be necessary, depending on the device
>   and guest OS:
>