From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 399731FF142 for ; Fri, 03 Jul 2026 10:40:57 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5B053213A0; Fri, 03 Jul 2026 10:40:56 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [PATCH docs] qm: pci passthrough: add note for passing through GPUs with large VRAM Date: Fri, 3 Jul 2026 10:40:12 +0200 Message-ID: <20260703084020.1091413-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.071 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6J6OSYBTDYX5FCEGX7RQDJ67N64XZMFR X-Message-ID-Hash: 6J6OSYBTDYX5FCEGX7RQDJ67N64XZMFR X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add it in a new section for common issues. We can reuse that when we encounter more setups that need more fine-tuning or configuration to work properly. This is mostly copied from the wiki[0] but with slight reordering and a short introductory sentence. Signed-off-by: Dominik Csapak --- qm-pci-passthrough.adoc | 52 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index 8a44a13..ca0ce64 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -510,6 +510,58 @@ properly used with HA and hardware changes are detected and non root users can configure them. See xref:resource_mapping[Resource Mapping] for details on that. +Common Issues +~~~~~~~~~~~~~ + +Passing through GPUs with large amount of VRAM with OVMF +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +By default, OVMF reserves only 32 GB of address space for the 64-bit PCI MMIO +window (see the https://github.com/tianocore/edk2/blob/master/OvmfPkg/RUNTIME_CONFIG.md#platform-optovmfx-pcimmio64mb[edk2 docs]). +When you pass through a GPU with more than 16 GB of VRAM, that window +is too small to map the device's memory, and the guest fails to initialize it. + +This can surface in different ways depending on the guest OS. Two common examples: + +In the guest's kernel journal (Linux): + +---- +kernel: nvidia 0000:01:00.0: enabling device (0000 -> 0002) +kernel: NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: + NVRM: BAR0 is 0M @ 0x0 (PCI:0000:01:00.0) +kernel: nvidia: probe of 0000:01:00.0 failed with error -1 +kernel: NVRM: The NVIDIA probe routine failed for 1 device(s). +---- + +In Windows Device Manager: + +---- +This device cannot find enough free resources that it can use. (Code 12) +---- + +There are multiple possible solutions: + +* Use `host` CPU type. This changes the MMIO size calculation for OVMF with +info from the host CPU. +* If you can't or don't want to use `host` type CPU, you can set `phys-bits` to +`host` (or a sufficiently large number). If the necessary bits are greater +than `40` the flag `pdpe1gb` is also necessary. That flag is included in some +CPU models but it can be added with (using x86-64-v2-AES as an example): + +---- +qm set VMID --cpu x86-64-v2-AES,phys-bits=host,flags=+pdpe1gb +---- + +* Should neither of that work, you can manually set the MMIO window size, e.g. +to 64 GB: + +---- +qm set VMID --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536' +---- +Set it to larger values if necessary, e.g. 131072 for 128GB or 262144 for 256GB. + +* Alternatively, SeaBIOS can be used instead of OVMF, which should not cause such issues. + [[qm_pci_viommu]] vIOMMU (emulated IOMMU) ~~~~~~~~~~~~~~~~~~~~~~~ -- 2.47.3