From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id DFF8C1FF0C1 for ; Tue, 25 Aug 2026 10:11:21 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E749E2189A; Tue, 25 Aug 2026 10:09:46 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1787591500; x=1788196300; darn=lists.proxmox.com; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=YWM0d1QS3UkebMUKUQZ1bzdPHEc6tmgoy3EDo86AEU8=; b=hXPlDVLPzJYqDg8Uozi0LTUBkiJ9jR5dbpF1rv3/FhVHs8z7PEcN0Zw1cnnzlrsuZR 8lNcGIx194GHI5uAuJd3rMll9CXl/bvJvCWzO1/vOSUFZrsFKBBzRjLF1Kw/gx2f0hsh xMeVqt/eJnbUDCskMwU0tD62e1SVCQUTD8SrGs68YBn2qfAEghKsBaetHqwsDh6yShuT TcboYECcvmwwxsa7WM2mWBwPco5/xUDEL43uNsvILRgO8jYPYBzPJnSxyUXkVE3VANmL 67LZ0/KxxGJA7BmVR6iizBriKOuyiITgbrAj1z56myTOUkthOEgi0MwVsZEQGlww48m4 Y2ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787591500; x=1788196300; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=YWM0d1QS3UkebMUKUQZ1bzdPHEc6tmgoy3EDo86AEU8=; b=jnUkzygPAXjGZyt27kQPOjAcILEPTMLziAUVXrVca7rKMqSjwlYLclVxjMhxppT3eL 5ACixAkEfXxB1rM1aM83E2CAHZPLZb+sX6S0SZuZuY7UjaOGhV0f+gtZpV1sPhw2XkRf cz6Ij8nNsOA9ufZWIIg7iPRDEpzgSDYz5RSjR007YhwuTsdQPcBU7ESxeCyMYKQiP/9B hXtL9hzAHdDHlNcLjENEErseUGrT02kBgJA3YkibiG/TpQTgh9lsV5zNTXMz/zSJYIBA OkSUkfdhZNwuFealE6fjVQLrhq1mrC8vgfeh0agtNP/uN4Uyc5Z1mcK4scw8H9QsJQQZ O6tg== X-Gm-Message-State: AFuF++k7kf0wcw6yNiWieU98g6GVuQur08UiNN5pMRIWJpQoBtHYW5Ft bi6RQbP2JblxNfWOVeOh2DQ0ecH+UMT5sfAyE+Ug4hWavXR3bwC0rtncoyih1Yom X-Gm-Gg: AR+sD11y8VmZx7AmofL/fCzR5I9vXLz4qDWhfHZoEmwOdABK67QMHe6TO9u67o4xdXz wf4ZvhYtRLGugSdtWccuw2Izs4tdkoi6qdoOH4TMTGd7pQzxUkuJc24VRXA7rijL3VPHE8zg/pa kvUuE/U2J3fPjRRzL5boEBizTiU1EXpVuDb8F/tj0ePIGHNL4OZMsEFSsFkfK1Xt/+dOVdzo2dQ 4xQj1QRRaXnvc+AsPkcnsju+Vpbe7gTd5K87PqCtvfFlbHhoVE80AWyMl9Mg90USah898RFahUn QuE36CoI/zse9Xeg2Rb2Kx1N/hwfzTdq9RKtiU76ON1De6cefO1Ednoy0zu+zceupHFw1LIopW+ OEWuufBREnsMB3pvEx6Uqzde4J12jj4ix8pby/U0rtds5GB9fBmyfc5niZWf3AOrYd3zFB9wynR S2ZAA49p0ormB9WuAvfYoZUq2GBvQPF5JP2b32Y+uKvMg4aVZUQ0OUnq9w64ZSe7e2WcaZPQQ02 t7p9HVfHpp+dDfUS2+aGnbf8NVW2yA8KgMBGY3Pk77iaVnez8aTIWstcR5xw/YdpctzXRw5PjHj a/BsTsAXTSF2IgMCxqtjuPE1Tz3bzGfrQXb3RG86 X-Received: by 2002:a17:907:9d1a:b0:c20:367d:f8b8 with SMTP id a640c23a62f3a-c24e2ae5e0emr47919066b.4.1787591499917; Mon, 24 Aug 2026 10:11:39 -0700 (PDT) From: Yao Xu X-Google-Original-From: Yao Xu To: pve-devel@lists.proxmox.com Subject: [PATCH pve-docs 1/2] qm: pci passthrough: note the AtomicOps caveat of all-functions form Date: Mon, 24 Aug 2026 19:08:27 +0200 Message-ID: <20260824170828.42821-2-Xy2462381442@gmail.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20260824170828.42821-1-Xy2462381442@gmail.com> References: <20260824170828.42821-1-Xy2462381442@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 1 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_ENVFROM_END_DIGIT 1 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider GB_FREEMAIL_NUM 0.75 Freemail spammy address RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: xy2462381442@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: AZH77I77JER5YGQUUUGYM6D7MSLT7GW5 X-Message-ID-Hash: AZH77I77JER5YGQUUUGYM6D7MSLT7GW5 X-Mailman-Approved-At: Tue, 25 Aug 2026 10:09:15 +0200 CC: Yao Xu X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The shortened ``00:02`' syntax is documented as a convenience. It also has a side effect that stays invisible until a guest driver needs PCIe AtomicOps. QEMU currently adds AtomicOp completer support to the emulated root port only for a single-function device sitting below a root port that supports DEVCAP2 (vfio_pci_enable_rp_atomics() in hw/vfio/pci.c). Passing every function of a card therefore leaves that port advertising AtomicOpsCap: 32bit- 64bit-, with no warning anywhere. The failure then surfaces several layers away from its cause. On AMD cards amdgpu logs "PCIE atomic ops is not supported" and RCCL collectives abort with "the operation cannot be performed in the present state", so it is easy to conclude that the hardware or the ROCm installation is at fault. The difference is an omitted .0 suffix. Verified on Proxmox VE 9.2.4 with QEMU 11.0.2, two RX 7900 XT (gfx1100) passed to one q35 guest. With hostpci0: 0000:0b:00,pcie=1 hostpci1: 0000:44:00,pcie=1 both guest root ports report AtomicOpsCap: 32bit- 64bit-, amdgpu logs the message above for both cards, and a two-rank RCCL all_reduce fails. Appending .0 to both entries and changing nothing else gives 32bit+ 64bit+, no driver message, and the same collective completes. Reverting reproduces the failure. Signed-off-by: Yao Xu --- qm-pci-passthrough.adoc | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index 00d9478..850ba40 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -338,6 +338,21 @@ you can pass them through all together with the shortened syntax ``00:02`'. This is equivalent with checking the ``All Functions`' checkbox in the web interface. +.All functions and PCIe AtomicOps +[NOTE] +==== +QEMU currently advertises PCIe AtomicOp completer support on the virtual root +port only for a *single-function* device. Passing every function of a card +therefore leaves that port advertising no AtomicOp support at all. Workloads +that need AtomicOps can then fail: the AMD GPU driver, for example, logs +`PCIE atomic ops is not supported`, and multi-GPU ROCm collectives can abort. + +If the guest needs AtomicOps, pass the function explicitly and use a PCIe port, +for example ``hostpci0: 00:02.0,pcie=on`' on a `q35` machine. This also +requires the *host* root port above the device to support AtomicOp completion, +which `lspci -vv` shows as `AtomicOpsCap: 32bit+ 64bit+`. +==== + There are some options to which may be necessary, depending on the device and guest OS: -- 2.50.1 (Apple Git-155)