Re: [PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[Dominik Csapak <d.csapak@proxmox.com>]

On 8/25/26 3:51 PM, Yao Xu wrote:
> Hi Dominik,
> 
> Thanks for looking at it.
> 
> The CLA is signed and on record — sent to office@proxmox.com and
> acknowledged on 2026-08-24. I left it off the cover letter on the
> assumption that pve-devel is the technical list; happy to be corrected.

It is, but I personally don't have a direct insight on
who already sent a CLA, so for first contributors, I always ask ;)

> 
>> I have a question: how is this handled on real hardware, since I
>> guess that works just fine there? (The cards in question
>> do have multiple functions...)
> 
> It does work on real hardware, and the function count is not what
> decides it. The kernel side is pci_enable_atomic_ops_to_root() in
> drivers/pci/pci.c: it requires the device to be a PCIe endpoint, the
> root port's DEVCAP2 to advertise the requested completion widths, and
> every bridge on the path to route AtomicOps without blocking egress. It
> never reads the device's function number and never asks whether the slot
> is multifunction. A card with a GPU and an HDMI audio function gets
> atomics on bare metal exactly as a single-function card would.
> 
> The restriction is QEMU's, and it is deliberate. It also has already been
> challenged upstream, for exactly the reason you give.
> 
> In February 2026 AMD posted "vfio/pci: Add multifunction atomic ops
> support", which removed the multifunction guard and computed the
> intersection of the functions' capabilities instead. Their motivation
> was yours: "we have come up on more than one occasion where the topology
> of the bare metal was mimicked by VM's configuration ... from UX
> standpoint, the correct way is that user shouldn't think about it".
> 
> Alex Williamson declined it:
> 
>      Back to the point, atomic ops routing is complicated, QEMU currently
>      kicks anything beyond the trivial case back to the VM administrator.
>      If the VM administrator doesn't want to think about it, analyze the
>      host topology, create a compatible VM topology, and manually set
>      appropriate atomic ops bits, then the burden probably needs to go in
>      the direction of VM builders and management tools rather than pushed
>      down into QEMU.  QEMU doesn't have the visibility to determine host
>      routing and is forced to work with the topology that's been
>      specified.
> 
> and closed with "I'm not convinced it's QEMU's job, or that QEMU is even
> capable of serving the intended goal here."
> 
>    https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/
> 
> His reasoning is worth reading in full, because it is not the reason the
> code comment gives — he says so himself: the restriction is less about
> picking a common capability set than about device-to-device AtomicOps,
> which QEMU cannot reason about because it cannot see the host's routing,
> and because a guest multifunction package need not correspond to one on
> the host at all.

After reading the full answer, I get why they don't want to implicitly
enable this based on the hosts capabilities.

Without being an expert in the respective QEMU code it's impossible
for me to say if there could be another solution though, e.g.
making it configurable explicitly, that way one could maybe use
multi-function devices and still enable it manually.

Reason is that passing through 'part of the card' works most of
the time, but not always. I've seen cases where the guest
driver refused to work properly when e.g. the audio device is missing.
(that said, this was on windows guests..)

I doubt the intersection between people running into such driver
issues and needing multi-GPU AtomicOps are big, so this shouldn't
come up often.

> 
>> If yes, I'd rather have this reported as a bug on the QEMU side,
>> rather have a behavior documented that might change with any release.
> 
> It has been reported, by the vendor, and declined six months ago. I can
> reopen it if you want, but I do not think re-litigating it is a good use
> of anyone's time, and on the release-stability point the guard is
> unchanged from v8.1.0 through v11.1.0.
> 
> That is also why I sent this to pve-devel rather than only upstream. The
> decision QEMU is declining to make is which capability to advertise when
> a slot's functions disagree, and Proxmox is what composes the slot.
> 

thanks for taking the time to explain it. If QEMU does not want to,
or cannot handle this themselves it's fine do document on our side
of course.

That said, including a link to the discsussion/rationale in the docs,
or at least in the commit message would be appreciated, so we have
a complete picture if we want to revisit this later on.

>> Also, the way the note is phrased makes it sound like this is something
>> everyone needs, while it's probably only relevant for some use cases.
> 
> You are right, and your framing is better than mine. It is a narrow
> case: single-GPU inference is unaffected, and what breaks is multi-GPU
> collectives, which is where AtomicOps get used. I will send a v2 with
> the note scoped that way and the version stated, along the lines of:
> 
>      .Multi-GPU passthrough and PCIe AtomicOps
>      [NOTE]
>      ====
>      Workloads that use PCIe AtomicOps — in practice multi-GPU
>      collectives such as ROCm's RCCL — need the guest's virtual root port
>      to advertise completer support. QEMU (as of 11.1) adds it only for a
>      single-function device, so passing every function of a card leaves it
>      unadvertised, and the AMD driver logs `PCIE atomic ops is not
>      supported`. If you need it, pass the function explicitly, for example
>      ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The host root port
>      above the device must also support AtomicOp completion, which
>      `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`.
>      ====
> 
>>  From my experience, in most situations, you want to pass the card
>> through as it is on the host, e.g. with functions.
> 
> Then patch 2 should go. It changes the GPU example to a single function,
> and on your reading of what most users want that makes the default
> worse for them to fix a case the note already covers. I will drop it
> from v2 rather than argue for it.

great, so I'm looking forward to the v2 :)

> 
> Verified on PVE 9.2.4 with QEMU 11.0.2, two RX 7900 XT in one q35
> guest: with both cards passed as 0000:0b:00 and 0000:44:00 the guest
> root ports report AtomicOpsCap: 32bit- 64bit- and a two-rank RCCL
> all_reduce fails; appending .0 to both and changing nothing else gives
> 32bit+ 64bit+ and the same collective completes. Reverting reproduces
> the failure.
> 
> Best regards
> Yao