[PATCH pve-docs 1/2] qm: pci passthrough: note the AtomicOps caveat of all-functions form

[Yao Xu <xy2462381442@gmail.com>]

The shortened ``00:02`' syntax is documented as a convenience. It also
has a side effect that stays invisible until a guest driver needs PCIe
AtomicOps.

QEMU currently adds AtomicOp completer support to the emulated root
port only for a single-function device sitting below a root port that
supports DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c).
Passing every function of a card therefore leaves that port
advertising AtomicOpsCap: 32bit- 64bit-, with no warning anywhere.

The failure then surfaces several layers away from its cause. On AMD
cards amdgpu logs "PCIE atomic ops is not supported" and RCCL
collectives abort with "the operation cannot be performed in the
present state", so it is easy to conclude that the hardware or the
ROCm installation is at fault. The difference is an omitted .0 suffix.

Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT
(gfx1100) passed to one q35 guest. With

    hostpci0: 0000:0b:00,pcie=1
    hostpci1: 0000:44:00,pcie=1

both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs
the message above for both cards, and a two-rank RCCL all_reduce
fails. Appending .0 to both entries and changing nothing else gives
32bit+ 64bit+, no driver message, and the same collective completes.
Reverting reproduces the failure.

Signed-off-by: Yao Xu <Xy2462381442@gmail.com>
---
 qm-pci-passthrough.adoc | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index 00d9478..850ba40 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -338,6 +338,21 @@ you can pass them through all together with the shortened syntax ``00:02`'.
 This is equivalent with checking the ``All Functions`' checkbox in the
 web interface.
 
+.All functions and PCIe AtomicOps
+[NOTE]
+====
+QEMU currently advertises PCIe AtomicOp completer support on the virtual root
+port only for a *single-function* device. Passing every function of a card
+therefore leaves that port advertising no AtomicOp support at all. Workloads
+that need AtomicOps can then fail: the AMD GPU driver, for example, logs
+`PCIE atomic ops is not supported`, and multi-GPU ROCm collectives can abort.
+
+If the guest needs AtomicOps, pass the function explicitly and use a PCIe port,
+for example ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. This also
+requires the *host* root port above the device to support AtomicOp completion,
+which `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`.
+====
+
 There are some options to which may be necessary, depending on the device
 and guest OS:
 
-- 
2.50.1 (Apple Git-155)