From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 3114594D38 for ; Mon, 16 Jan 2023 11:00:36 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0F90524C01 for ; Mon, 16 Jan 2023 11:00:36 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 16 Jan 2023 11:00:35 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 17D0B44391 for ; Mon, 16 Jan 2023 11:00:35 +0100 (CET) Date: Mon, 16 Jan 2023 11:00:33 +0100 From: Wolfgang Bumiller To: Markus Frank Cc: Proxmox VE development discussion Message-ID: <20230116100033.drlnujc5kpnhmltw@casey.proxmox.com> References: <20221125140857.121622-1-m.frank@proxmox.com> <20221125140857.121622-5-m.frank@proxmox.com> <20230113100929.ug2oivbml5tuizst@fwblub> <2585bfbd-2d44-5bef-1a0f-ca9e7e84b653@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2585bfbd-2d44-5bef-1a0f-ca9e7e84b653@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.209 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH docs v4 4/5] added vIOMMU documentation X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jan 2023 10:00:36 -0000 On Fri, Jan 13, 2023 at 02:31:36PM +0100, Markus Frank wrote: > > > On 1/13/23 11:09, Wolfgang Bumiller wrote: > > On Fri, Nov 25, 2022 at 03:08:56PM +0100, Markus Frank wrote: > > > Signed-off-by: Markus Frank > > > --- > > > qm-pci-passthrough.adoc | 25 +++++++++++++++++++++++++ > > > 1 file changed, 25 insertions(+) > > > > > > diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc > > > index fa6ba35..7ed4d49 100644 > > > --- a/qm-pci-passthrough.adoc > > > +++ b/qm-pci-passthrough.adoc > > > @@ -389,6 +389,31 @@ Example configuration with an `Intel GVT-g vGPU` (`Intel Skylake 6700k`): > > > With this set, {pve} automatically creates such a device on VM start, and > > > cleans it up again when the VM stops. > > > +[[qm_pci_viommu]] > > > +vIOMMU > > > +~~~~~~ > > > + > > > +vIOMMU enables the option to passthrough pci devices to Level-2 VMs > > > +in Level-1 VMs via Nested Virtualisation. > > > + > > > +Host-Requirement: Set `intel_iommu=on` or `amd_iommu=on` depending on your > > > +CPU. > > > > And by "CPU" you mean kernel command line? ;-) > > Host-Requirement: Add `intel_iommu=on` or `amd_iommu=on` > depending on your CPU to your kernel command line. > > like this? > > > > > + > > > +VM-Requirement: For both Intel and AMD CPUs you will have to set > > > +`intel_iommu=on` as a Linux boot parameter in the vIOMMU-enabled-VM, because > > > +Qemu implements the Intel variant. > > > > ^ As mentioned, there does appear to be an amd_iommu device in the qemu > > code, so would the amd variant work? > > > > In my reply to the code patch I mentioned checking the host arch. But if > > you say we can use intel_iommu on AMD as well, I'd say, if both work, > > give the user a choice, otherwise we can of course just stick to the one > > that works ;-) > > intel_iommu works better on my AMD CPU than amd_iommu ;) Can you define "better"? My main concern is that if we don't give users the option to choose, the only data point we have is yours ;-) If we explicitly mention that you can use one on the other in the docs, people can try it themselves and maybe we'll see some feedback on the forums etc. However, I'm fine with a patch for only the intel version for now as we can always add an option later. > Moreover it adds an extra AMDVI-PCI device that is using the first pci address. > `kvm: -device VGA,id=vga,bus=pcie.0,addr=0x1: PCI: slot 1 function 0 not available for VGA, in use by AMDVI-PCI,id=(null)` For that I'd say, try to add the AMDVI-PCI device manually to an explicitly chosen slot. We need to avoid automatically added devices like the plague, because moving them later can break live snapshots (and windows). > > I cannot find any good documentation for amd_iommu but it also seems like > it has less features. Less, or just not configurable? ;-) I mean, if it works it works ;-) > > $ qemu-system-x86_64 -device 'amd-iommu,help' > amd-iommu options: > device-iotlb= - (default: false) > intremap= - on/off/auto (default: "auto") > pt= - (default: true) > $ qemu-system-x86_64 -device 'intel-iommu,help' > intel-iommu options: > aw-bits= - (default: 39) > caching-mode= - (default: false) > device-iotlb= - (default: false) > dma-drain= - (default: true) > dma-translation= - (default: true) > eim= - on/off/auto (default: "auto") > intremap= - on/off/auto (default: "auto") > pt= - (default: true) > snoop-control= - (default: false) > version= - (default: 0) > x-buggy-eim= - (default: false) > x-pasid-mode= - (default: false) > x-scalable-mode= - (default: false)