From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id ADE16906EA for ; Thu, 16 Mar 2023 13:05:58 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 868723ECD for ; Thu, 16 Mar 2023 13:05:28 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 16 Mar 2023 13:05:27 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0C6D343656 for ; Thu, 16 Mar 2023 13:05:27 +0100 (CET) From: Noel Ullreich To: pve-devel@lists.proxmox.com Date: Thu, 16 Mar 2023 13:05:13 +0100 Message-Id: <20230316120513.33349-1-n.ullreich@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.055 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH pve-docs v2] update the PCI(e) docs X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Mar 2023 12:05:58 -0000 A little update to the PCI(e) docs with the plan of reworking the PCI wiki as well. Along some minor grammar fixes added: * how to check if kernelmodules are being loaded * how to check which drivers to blacklist * how to add softdeps for module loading * where to find kernel params Signed-off-by: Noel Ullreich --- changes from v1: * fixed spelling mistakes * reduced code snippets of how to check iommu groupings to one * moved where to find kernel params to kernel cmdline section * removed wrong info on display output. will add correct info to Examples-Wiki * changed module names to variable-names, so that people can't blindly copy-paste. * restructured commit message ;) qm-pci-passthrough.adoc | 75 ++++++++++++++++++++++++++++++++++------- system-booting.adoc | 9 +++++ 2 files changed, 71 insertions(+), 13 deletions(-) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index df6cf21..3f75ed0 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -16,16 +16,17 @@ device anymore on the host or in any other VM. General Requirements ~~~~~~~~~~~~~~~~~~~~ -Since passthrough is a feature which also needs hardware support, there are -some requirements to check and preparations to be done to make it work. - +Since passthrough is performed on real hardware, it needs to fulfill some +requirements. A brief overview of these requirements is given below, for more +information on specific devices, see +https://pve.proxmox.com/wiki/PCI_Passthrough[PCI Passthrough Examples]. Hardware ^^^^^^^^ Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement **U**nit) interrupt remapping, this includes the CPU and the mainboard. -Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this. +Generally, Intel systems with VT-d and AMD systems with AMD-Vi support this. But it is not guaranteed that everything will work out of the box, due to bad hardware implementation and missing or low quality drivers. @@ -44,8 +45,8 @@ some configuration to enable PCI(e) passthrough. .IOMMU -First, you have to enable IOMMU support in your BIOS/UEFI. Usually the -corresponding setting is called `IOMMU` or `VT-d`,but you should find the exact +First, you will have to enable IOMMU support in your BIOS/UEFI. Usually the +corresponding setting is called `IOMMU` or `VT-d`, but you should find the exact option name in the manual of your motherboard. For Intel CPUs, you may also need to enable the IOMMU on the @@ -72,6 +73,9 @@ hardware IOMMU. To enable these options, add: to the xref:sysboot_edit_kernel_cmdline[kernel commandline]. +For a complete list of kernel commandline options (of kernel 5.15), see +https://www.kernel.org/doc/html/v5.15/admin-guide/kernel-parameters.html[kernel.org]. + .Kernel Modules You have to make sure the following modules are loaded. This can be achieved by @@ -92,6 +96,14 @@ After changing anything modules related, you need to refresh your # update-initramfs -u -k all ---- +To check if the modules are being loaded, the output of + +---- +# lsmod | grep vfio +---- + +should include the four modules from above. + .Finish Configuration Finally reboot to bring the changes into effect and check that it is indeed @@ -105,10 +117,11 @@ should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is enabled, depending on hardware and kernel the exact message can vary. It is also important that the device(s) you want to pass through -are in a *separate* `IOMMU` group. This can be checked with: +are in a *separate* `IOMMU` group. This can be checked with a call to the {pve} +API: ---- -# find /sys/kernel/iommu_groups/ -type l +# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist "" ---- It is okay if the device is in an `IOMMU` group together with its functions @@ -159,8 +172,8 @@ PCI(e) card, for example a GPU or a network card. Host Configuration ^^^^^^^^^^^^^^^^^^ -In this case, the host must not use the card. There are two methods to achieve -this: +{pve} tries to automatically make the PCI(e) device unavailable for the host. +However, if this doesn't work, there are two things that can be done: * pass the device IDs to the options of the 'vfio-pci' modules by adding + @@ -175,7 +188,7 @@ the vendor and device IDs obtained by: # lspci -nn ---- -* blacklist the driver completely on the host, ensuring that it is free to bind +* blacklist the driver on the host completely, ensuring that it is free to bind for passthrough, with + ---- @@ -183,11 +196,49 @@ for passthrough, with ---- + in a .conf file in */etc/modprobe.d/*. ++ +To find the drivername, execute ++ +---- +# lspci -k +---- ++ +for example: ++ +---- +# lspci -k | grep -A 3 "VGA" +---- ++ +will output something similar to ++ +---- +01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1) + Subsystem: Micro-Star International Co., Ltd. [MSI] GP108 [GeForce GT 1030] + Kernel driver in use: + Kernel modules: +---- ++ +Now we can blacklist the drivers by writing them into a .conf file: ++ +---- +echo "blacklist " >> /etc/modprobe.d/blacklist.conf +---- For both methods you need to xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and reboot after that. +Should this not work, you might need to set a soft dependency to load the gpu +modules before loading 'vfio-pci'. This can be done with the 'softdep' flag, see +also the manpages on 'modprobe.d' for more information. + +For example, if you are using drivers named : + +---- +# echo "softdep pre: vfio-pci" >> /etc/modprobe.d/.conf +---- + + .Verify Configuration To check if your changes were successful, you can use @@ -262,7 +313,6 @@ For example: # qm set VMID -hostpci0 02:00,device-id=0x10f6,sub-vendor-id=0x0000 ---- - Other considerations ^^^^^^^^^^^^^^^^^^^^ @@ -288,7 +338,6 @@ Currently, the most common use case for this are NICs (**N**etwork physical port. This allows using features such as checksum offloading, etc. to be used inside a VM, reducing the (host) CPU overhead. - Host Configuration ^^^^^^^^^^^^^^^^^^ diff --git a/system-booting.adoc b/system-booting.adoc index 30621a6..d0f847e 100644 --- a/system-booting.adoc +++ b/system-booting.adoc @@ -272,6 +272,15 @@ initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve Editing the Kernel Commandline ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +A complete list of kernel parameters can be found at +'https://www.kernel.org/doc/html//admin-guide/kernel-parameters.html'. +replace with the major.minor version (e.g. 5.15). You can +find your kernel version by running + +---- +# uname -r +---- + You can modify the kernel commandline in the following places, depending on the bootloader used: -- 2.30.2