Re: [PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[Yao Xu <xy2462381442@gmail.com>]

Hi Dominik,

Thanks for looking at it.

The CLA is signed and on record — sent to office@proxmox.com and
acknowledged on 2026-08-24. I left it off the cover letter on the
assumption that pve-devel is the technical list; happy to be corrected.

> I have a question: how is this handled on real hardware, since I
> guess that works just fine there? (The cards in question
> do have multiple functions...)

It does work on real hardware, and the function count is not what
decides it. The kernel side is pci_enable_atomic_ops_to_root() in
drivers/pci/pci.c: it requires the device to be a PCIe endpoint, the
root port's DEVCAP2 to advertise the requested completion widths, and
every bridge on the path to route AtomicOps without blocking egress. It
never reads the device's function number and never asks whether the slot
is multifunction. A card with a GPU and an HDMI audio function gets
atomics on bare metal exactly as a single-function card would.

The restriction is QEMU's, and it is deliberate. It also has already been
challenged upstream, for exactly the reason you give.

In February 2026 AMD posted "vfio/pci: Add multifunction atomic ops
support", which removed the multifunction guard and computed the
intersection of the functions' capabilities instead. Their motivation
was yours: "we have come up on more than one occasion where the topology
of the bare metal was mimicked by VM's configuration ... from UX
standpoint, the correct way is that user shouldn't think about it".

Alex Williamson declined it:

    Back to the point, atomic ops routing is complicated, QEMU currently
    kicks anything beyond the trivial case back to the VM administrator.
    If the VM administrator doesn't want to think about it, analyze the
    host topology, create a compatible VM topology, and manually set
    appropriate atomic ops bits, then the burden probably needs to go in
    the direction of VM builders and management tools rather than pushed
    down into QEMU.  QEMU doesn't have the visibility to determine host
    routing and is forced to work with the topology that's been
    specified.

and closed with "I'm not convinced it's QEMU's job, or that QEMU is even
capable of serving the intended goal here."

  https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/

His reasoning is worth reading in full, because it is not the reason the
code comment gives — he says so himself: the restriction is less about
picking a common capability set than about device-to-device AtomicOps,
which QEMU cannot reason about because it cannot see the host's routing,
and because a guest multifunction package need not correspond to one on
the host at all.

> If yes, I'd rather have this reported as a bug on the QEMU side,
> rather have a behavior documented that might change with any release.

It has been reported, by the vendor, and declined six months ago. I can
reopen it if you want, but I do not think re-litigating it is a good use
of anyone's time, and on the release-stability point the guard is
unchanged from v8.1.0 through v11.1.0.

That is also why I sent this to pve-devel rather than only upstream. The
decision QEMU is declining to make is which capability to advertise when
a slot's functions disagree, and Proxmox is what composes the slot.

> Also, the way the note is phrased makes it sound like this is something
> everyone needs, while it's probably only relevant for some use cases.

You are right, and your framing is better than mine. It is a narrow
case: single-GPU inference is unaffected, and what breaks is multi-GPU
collectives, which is where AtomicOps get used. I will send a v2 with
the note scoped that way and the version stated, along the lines of:

    .Multi-GPU passthrough and PCIe AtomicOps
    [NOTE]
    ====
    Workloads that use PCIe AtomicOps — in practice multi-GPU
    collectives such as ROCm's RCCL — need the guest's virtual root port
    to advertise completer support. QEMU (as of 11.1) adds it only for a
    single-function device, so passing every function of a card leaves it
    unadvertised, and the AMD driver logs `PCIE atomic ops is not
    supported`. If you need it, pass the function explicitly, for example
    ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The host root port
    above the device must also support AtomicOp completion, which
    `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`.
    ====

> From my experience, in most situations, you want to pass the card
> through as it is on the host, e.g. with functions.

Then patch 2 should go. It changes the GPU example to a single function,
and on your reading of what most users want that makes the default
worse for them to fix a case the note already covers. I will drop it
from v2 rather than argue for it.

Verified on PVE 9.2.4 with QEMU 11.0.2, two RX 7900 XT in one q35
guest: with both cards passed as 0000:0b:00 and 0000:44:00 the guest
root ports report AtomicOpsCap: 32bit- 64bit- and a two-rank RCCL
all_reduce fails; appending .0 to both and changing nothing else gives
32bit+ 64bit+ and the same collective completes. Reverting reproduces
the failure.

Best regards
Yao