From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <m.frank@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 653CF9A196 for <pve-devel@lists.proxmox.com>; Thu, 12 Oct 2023 12:02:19 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4905C11F68 for <pve-devel@lists.proxmox.com>; Thu, 12 Oct 2023 12:02:19 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-devel@lists.proxmox.com>; Thu, 12 Oct 2023 12:02:17 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 515BC40EE1 for <pve-devel@lists.proxmox.com>; Thu, 12 Oct 2023 12:02:17 +0200 (CEST) From: Markus Frank <m.frank@proxmox.com> To: pve-devel@lists.proxmox.com Date: Thu, 12 Oct 2023 12:02:06 +0200 Message-Id: <20231012100207.83441-4-m.frank@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231012100207.83441-1-m.frank@proxmox.com> References: <20231012100207.83441-1-m.frank@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.045 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemu.org, proxmox.com, wikipedia.org, archive.org, intel.com, medium.com] Subject: [pve-devel] [PATCH docs v7 3/4] added vIOMMU documentation X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 12 Oct 2023 10:02:19 -0000 Signed-off-by: Markus Frank <m.frank@proxmox.com> --- qm-pci-passthrough.adoc | 53 +++++++++++++++++++++++++++++++++++++++++ qm.adoc | 1 + 2 files changed, 54 insertions(+) diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index b90a0b9..7f7d4a7 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -408,6 +408,59 @@ properly used with HA and hardware changes are detected and non root users can configure them. See xref:resource_mapping[Resource Mapping] for details on that. +[[qm_pci_viommu]] +vIOMMU (emulated IOMMU) +~~~~~~~~~~~~~~~~~~~~~~~ + +Using the vIOMMU option allows you to to passthrough PCI devices to layer-2 +VMs in layer-1 VMs via +https://pve.proxmox.com/wiki/Nested_Virtualization[nested virtualization]. +There are currently two vIOMMU implementations available: Intel and VirtIO. + +Host requirement: + +* Add `intel_iommu=on` or `amd_iommu=on` depending on your CPU to your kernel +command line. + +Nested virtualization VM requirement: + +* Enable *kvm* and use the *host* cpu type to use nested virtualization. + +Intel vIOMMU +^^^^^^^^^^^^ + +Intel vIOMMU specific VM requirements: + +* Whether you are using an Intel or AMD CPU on your host, it is important to set +`intel_iommu=on` in the VMs kernel parameters. + +* To use Intel vIOMMU you need to set *q35* as the machine type. + +If all requirements are met, you can add `viommu=intel` to the machine parameter +in the configuration of the VM that should be able to passthrough PCI devices. + +---- +# qm set VMID -machine q35,viommu=intel +---- + +https://wiki.qemu.org/Features/VT-d[QEMU documentation for VT-d] + +VirtIO vIOMMU +^^^^^^^^^^^^^ + +This vIOMMU implementation is more recent and does not have as many limitations +as Intel vIOMMU but is currently less used in production and less documentated. + +With VirtIO vIOMMU there is *no* need to set any kernel parameters. +It is also *not* necessary to use q35 as the machine type, but it is advisable +if you want to use PCIe. + +---- +# qm set VMID -machine q35,viommu=virtio +---- + +https://web.archive.org/web/20230804075844/https://michael2012z.medium.com/virtio-iommu-789369049443[Blog-Post by Michael Zhao explaining virtio-iommu] + ifdef::wiki[] See Also diff --git a/qm.adoc b/qm.adoc index b3c3034..2e5b109 100644 --- a/qm.adoc +++ b/qm.adoc @@ -145,6 +145,7 @@ default https://en.wikipedia.org/wiki/Intel_440FX[Intel 440FX] or the https://ark.intel.com/content/www/us/en/ark/products/31918/intel-82q35-graphics-and-memory-controller.html[Q35] chipset, which also provides a virtual PCIe bus, and thus may be desired if one wants to pass through PCIe hardware. +Additionally, you can select a xref:qm_pci_viommu[vIOMMU] implementation. [[qm_hard_disk]] Hard Disk -- 2.39.2