[PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[Yao Xu]

Passing a card with the shortened ``00:02`' syntax quietly costs the
guest PCIe AtomicOps. QEMU adds AtomicOp completer support to the
emulated root port only for a single-function device, so the port ends
up advertising none, and nothing in the config or the logs points at
the cause. The failure surfaces as a driver or ROCm problem several
layers away.

Patch 1 documents this next to the paragraph that introduces the
syntax. Patch 2 appends the function to the GPU example below it, which
currently pairs pcie=on with the all-functions form and would hit the
caveat as written.

Verified on Proxmox VE 9.2.4 with QEMU 11.0.2 and two RX 7900 XT
(gfx1100) in one q35 guest; the reproduction is in patch 1.

Yao Xu (2):
  qm: pci passthrough: note the AtomicOps caveat of all-functions form
  qm: pci passthrough: name the function in the GPU example

 qm-pci-passthrough.adoc | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

-- 
2.50.1 (Apple Git-155)

[PATCH pve-docs 1/2] qm: pci passthrough: note the AtomicOps caveat of all-functions form

[Yao Xu]

The shortened ``00:02`' syntax is documented as a convenience. It also
has a side effect that stays invisible until a guest driver needs PCIe
AtomicOps.

QEMU currently adds AtomicOp completer support to the emulated root
port only for a single-function device sitting below a root port that
supports DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c).
Passing every function of a card therefore leaves that port
advertising AtomicOpsCap: 32bit- 64bit-, with no warning anywhere.

The failure then surfaces several layers away from its cause. On AMD
cards amdgpu logs "PCIE atomic ops is not supported" and RCCL
collectives abort with "the operation cannot be performed in the
present state", so it is easy to conclude that the hardware or the
ROCm installation is at fault. The difference is an omitted .0 suffix.

Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT
(gfx1100) passed to one q35 guest. With

    hostpci0: 0000:0b:00,pcie=1
    hostpci1: 0000:44:00,pcie=1

both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs
the message above for both cards, and a two-rank RCCL all_reduce
fails. Appending .0 to both entries and changing nothing else gives
32bit+ 64bit+, no driver message, and the same collective completes.
Reverting reproduces the failure.

Signed-off-by: Yao Xu <Xy2462381442@gmail.com>
---
 qm-pci-passthrough.adoc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index 00d9478..850ba40 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -338,6 +338,21 @@ you can pass them through all together with the shortened syntax ``00:02`'.
 This is equivalent with checking the ``All Functions`' checkbox in the
 web interface.
 
+.All functions and PCIe AtomicOps
+[NOTE]
+====
+QEMU currently advertises PCIe AtomicOp completer support on the virtual root
+port only for a *single-function* device. Passing every function of a card
+therefore leaves that port advertising no AtomicOp support at all. Workloads
+that need AtomicOps can then fail: the AMD GPU driver, for example, logs
+`PCIE atomic ops is not supported`, and multi-GPU ROCm collectives can abort.
+
+If the guest needs AtomicOps, pass the function explicitly and use a PCIe port,
+for example ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. This also
+requires the *host* root port above the device to support AtomicOp completion,
+which `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`.
+====
+
 There are some options to which may be necessary, depending on the device
 and guest OS:
 
-- 
2.50.1 (Apple Git-155)

[PATCH pve-docs 2/2] qm: pci passthrough: name the function in the GPU example

[Yao Xu]

The example pairs a PCIe port with the shortened all-functions form,
which is the case the note above warns about: QEMU adds AtomicOp
completer support to the virtual root port only for a single-function
device, so a guest driver that asks for AtomicOps finds none. A GPU is
the most likely device to ask.

What the example demonstrates is x-vga and pcie, not the shortened
syntax, so naming the function changes nothing else about it.

Signed-off-by: Yao Xu <Xy2462381442@gmail.com>
---
 qm-pci-passthrough.adoc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index 850ba40..b95de06 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -374,7 +374,7 @@ This is a relative path under */usr/share/kvm/*.
 An example of PCIe passthrough with a GPU set to primary:
 
 ----
-# qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
+# qm set VMID -hostpci0 02:00.0,pcie=on,x-vga=on
 ----
 
 .PCI ID overrides
-- 
2.50.1 (Apple Git-155)

Re: [PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[Dominik Csapak]

On 8/25/26 10:10 AM, Yao Xu wrote:
> Passing a card with the shortened ``00:02`' syntax quietly costs the
> guest PCIe AtomicOps. QEMU adds AtomicOp completer support to the
> emulated root port only for a single-function device, so the port ends
> up advertising none, and nothing in the config or the logs points at
> the cause. The failure surfaces as a driver or ROCm problem several
> layers away.
> 
> Patch 1 documents this next to the paragraph that introduces the
> syntax. Patch 2 appends the function to the GPU example below it, which
> currently pairs pcie=on with the all-functions form and would hit the
> caveat as written.
> 
> Verified on Proxmox VE 9.2.4 with QEMU 11.0.2 and two RX 7900 XT
> (gfx1100) in one q35 guest; the reproduction is in patch 1.
> 
> Yao Xu (2):
>    qm: pci passthrough: note the AtomicOps caveat of all-functions form
>    qm: pci passthrough: name the function in the GPU example
> 
>   qm-pci-passthrough.adoc | 17 ++++++++++++++++-
>   1 file changed, 16 insertions(+), 1 deletion(-)
> 
Hi,

thanks for the contribution, did you already see our developer docs:
https://pve.proxmox.com/wiki/Developer_Documentation
?

especially the part about the CLA.

I have a question: how is this handled on real hardware, since I
guess that works just fine there? (The cards in question
do have multiple functions...)

If yes, I'd rather have this reported as a bug on the QEMU side,
rather have a behavior documented that might change with any release.
 From my experience, in most situations, you want to pass the card
through as it is on the host, e.g. with functions.

Also, the way the note is phrased makes it sound like this is something
everyone needs, while it's probably only relevant for some use cases.

In that case we could rephrase the note to something like:

Note for multi-GPU ROCm users:

QEMU (up to version x.y as of 2026) only advertises...


or something similar.

Best regards
Dominik

Re: [PATCH pve-docs 0/2] qm: pci passthrough: the AtomicOps caveat of the all-functions form

[Dominik Csapak]

On 8/25/26 3:51 PM, Yao Xu wrote:
> Hi Dominik,
> 
> Thanks for looking at it.
> 
> The CLA is signed and on record — sent to office@proxmox.com and
> acknowledged on 2026-08-24. I left it off the cover letter on the
> assumption that pve-devel is the technical list; happy to be corrected.

It is, but I personally don't have a direct insight on
who already sent a CLA, so for first contributors, I always ask ;)

> 
>> I have a question: how is this handled on real hardware, since I
>> guess that works just fine there? (The cards in question
>> do have multiple functions...)
> 
> It does work on real hardware, and the function count is not what
> decides it. The kernel side is pci_enable_atomic_ops_to_root() in
> drivers/pci/pci.c: it requires the device to be a PCIe endpoint, the
> root port's DEVCAP2 to advertise the requested completion widths, and
> every bridge on the path to route AtomicOps without blocking egress. It
> never reads the device's function number and never asks whether the slot
> is multifunction. A card with a GPU and an HDMI audio function gets
> atomics on bare metal exactly as a single-function card would.
> 
> The restriction is QEMU's, and it is deliberate. It also has already been
> challenged upstream, for exactly the reason you give.
> 
> In February 2026 AMD posted "vfio/pci: Add multifunction atomic ops
> support", which removed the multifunction guard and computed the
> intersection of the functions' capabilities instead. Their motivation
> was yours: "we have come up on more than one occasion where the topology
> of the bare metal was mimicked by VM's configuration ... from UX
> standpoint, the correct way is that user shouldn't think about it".
> 
> Alex Williamson declined it:
> 
>      Back to the point, atomic ops routing is complicated, QEMU currently
>      kicks anything beyond the trivial case back to the VM administrator.
>      If the VM administrator doesn't want to think about it, analyze the
>      host topology, create a compatible VM topology, and manually set
>      appropriate atomic ops bits, then the burden probably needs to go in
>      the direction of VM builders and management tools rather than pushed
>      down into QEMU.  QEMU doesn't have the visibility to determine host
>      routing and is forced to work with the topology that's been
>      specified.
> 
> and closed with "I'm not convinced it's QEMU's job, or that QEMU is even
> capable of serving the intended goal here."
> 
>    https://lore.kernel.org/qemu-devel/8b3e30e6-3c3e-49ab-b9db-8296aaf819d1@app.fastmail.com/
> 
> His reasoning is worth reading in full, because it is not the reason the
> code comment gives — he says so himself: the restriction is less about
> picking a common capability set than about device-to-device AtomicOps,
> which QEMU cannot reason about because it cannot see the host's routing,
> and because a guest multifunction package need not correspond to one on
> the host at all.

After reading the full answer, I get why they don't want to implicitly
enable this based on the hosts capabilities.

Without being an expert in the respective QEMU code it's impossible
for me to say if there could be another solution though, e.g.
making it configurable explicitly, that way one could maybe use
multi-function devices and still enable it manually.

Reason is that passing through 'part of the card' works most of
the time, but not always. I've seen cases where the guest
driver refused to work properly when e.g. the audio device is missing.
(that said, this was on windows guests..)

I doubt the intersection between people running into such driver
issues and needing multi-GPU AtomicOps are big, so this shouldn't
come up often.

> 
>> If yes, I'd rather have this reported as a bug on the QEMU side,
>> rather have a behavior documented that might change with any release.
> 
> It has been reported, by the vendor, and declined six months ago. I can
> reopen it if you want, but I do not think re-litigating it is a good use
> of anyone's time, and on the release-stability point the guard is
> unchanged from v8.1.0 through v11.1.0.
> 
> That is also why I sent this to pve-devel rather than only upstream. The
> decision QEMU is declining to make is which capability to advertise when
> a slot's functions disagree, and Proxmox is what composes the slot.
> 

thanks for taking the time to explain it. If QEMU does not want to,
or cannot handle this themselves it's fine do document on our side
of course.

That said, including a link to the discsussion/rationale in the docs,
or at least in the commit message would be appreciated, so we have
a complete picture if we want to revisit this later on.

>> Also, the way the note is phrased makes it sound like this is something
>> everyone needs, while it's probably only relevant for some use cases.
> 
> You are right, and your framing is better than mine. It is a narrow
> case: single-GPU inference is unaffected, and what breaks is multi-GPU
> collectives, which is where AtomicOps get used. I will send a v2 with
> the note scoped that way and the version stated, along the lines of:
> 
>      .Multi-GPU passthrough and PCIe AtomicOps
>      [NOTE]
>      ====
>      Workloads that use PCIe AtomicOps — in practice multi-GPU
>      collectives such as ROCm's RCCL — need the guest's virtual root port
>      to advertise completer support. QEMU (as of 11.1) adds it only for a
>      single-function device, so passing every function of a card leaves it
>      unadvertised, and the AMD driver logs `PCIE atomic ops is not
>      supported`. If you need it, pass the function explicitly, for example
>      ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. The host root port
>      above the device must also support AtomicOp completion, which
>      `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`.
>      ====
> 
>>  From my experience, in most situations, you want to pass the card
>> through as it is on the host, e.g. with functions.
> 
> Then patch 2 should go. It changes the GPU example to a single function,
> and on your reading of what most users want that makes the default
> worse for them to fix a case the note already covers. I will drop it
> from v2 rather than argue for it.

great, so I'm looking forward to the v2 :)

> 
> Verified on PVE 9.2.4 with QEMU 11.0.2, two RX 7900 XT in one q35
> guest: with both cards passed as 0000:0b:00 and 0000:44:00 the guest
> root ports report AtomicOpsCap: 32bit- 64bit- and a two-rank RCCL
> all_reduce fails; appending .0 to both and changing nothing else gives
> 32bit+ 64bit+ and the same collective completes. Reverting reproduces
> the failure.
> 
> Best regards
> Yao