From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <w.bumiller@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 3114594D38
for <pve-devel@lists.proxmox.com>; Mon, 16 Jan 2023 11:00:36 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 0F90524C01
for <pve-devel@lists.proxmox.com>; Mon, 16 Jan 2023 11:00:36 +0100 (CET)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Mon, 16 Jan 2023 11:00:35 +0100 (CET)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 17D0B44391
for <pve-devel@lists.proxmox.com>; Mon, 16 Jan 2023 11:00:35 +0100 (CET)
Date: Mon, 16 Jan 2023 11:00:33 +0100
From: Wolfgang Bumiller <w.bumiller@proxmox.com>
To: Markus Frank <m.frank@proxmox.com>
Cc: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Message-ID: <20230116100033.drlnujc5kpnhmltw@casey.proxmox.com>
References: <20221125140857.121622-1-m.frank@proxmox.com>
<20221125140857.121622-5-m.frank@proxmox.com>
<20230113100929.ug2oivbml5tuizst@fwblub>
<2585bfbd-2d44-5bef-1a0f-ca9e7e84b653@proxmox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2585bfbd-2d44-5bef-1a0f-ca9e7e84b653@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.209 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: Re: [pve-devel] [PATCH docs v4 4/5] added vIOMMU documentation
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Mon, 16 Jan 2023 10:00:36 -0000
On Fri, Jan 13, 2023 at 02:31:36PM +0100, Markus Frank wrote:
>
>
> On 1/13/23 11:09, Wolfgang Bumiller wrote:
> > On Fri, Nov 25, 2022 at 03:08:56PM +0100, Markus Frank wrote:
> > > Signed-off-by: Markus Frank <m.frank@proxmox.com>
> > > ---
> > > qm-pci-passthrough.adoc | 25 +++++++++++++++++++++++++
> > > 1 file changed, 25 insertions(+)
> > >
> > > diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
> > > index fa6ba35..7ed4d49 100644
> > > --- a/qm-pci-passthrough.adoc
> > > +++ b/qm-pci-passthrough.adoc
> > > @@ -389,6 +389,31 @@ Example configuration with an `Intel GVT-g vGPU` (`Intel Skylake 6700k`):
> > > With this set, {pve} automatically creates such a device on VM start, and
> > > cleans it up again when the VM stops.
> > > +[[qm_pci_viommu]]
> > > +vIOMMU
> > > +~~~~~~
> > > +
> > > +vIOMMU enables the option to passthrough pci devices to Level-2 VMs
> > > +in Level-1 VMs via Nested Virtualisation.
> > > +
> > > +Host-Requirement: Set `intel_iommu=on` or `amd_iommu=on` depending on your
> > > +CPU.
> >
> > And by "CPU" you mean kernel command line? ;-)
>
> Host-Requirement: Add `intel_iommu=on` or `amd_iommu=on`
> depending on your CPU to your kernel command line.
>
> like this?
> >
> > > +
> > > +VM-Requirement: For both Intel and AMD CPUs you will have to set
> > > +`intel_iommu=on` as a Linux boot parameter in the vIOMMU-enabled-VM, because
> > > +Qemu implements the Intel variant.
> >
> > ^ As mentioned, there does appear to be an amd_iommu device in the qemu
> > code, so would the amd variant work?
> >
> > In my reply to the code patch I mentioned checking the host arch. But if
> > you say we can use intel_iommu on AMD as well, I'd say, if both work,
> > give the user a choice, otherwise we can of course just stick to the one
> > that works ;-)
>
> intel_iommu works better on my AMD CPU than amd_iommu ;)
Can you define "better"?
My main concern is that if we don't give users the option to choose, the
only data point we have is yours ;-)
If we explicitly mention that you can use one on the other in the docs,
people can try it themselves and maybe we'll see some feedback on the
forums etc.
However, I'm fine with a patch for only the intel version for now as we
can always add an option later.
> Moreover it adds an extra AMDVI-PCI device that is using the first pci address.
> `kvm: -device VGA,id=vga,bus=pcie.0,addr=0x1: PCI: slot 1 function 0 not available for VGA, in use by AMDVI-PCI,id=(null)`
For that I'd say, try to add the AMDVI-PCI device manually to an
explicitly chosen slot. We need to avoid automatically added devices
like the plague, because moving them later can break live snapshots (and
windows).
>
> I cannot find any good documentation for amd_iommu but it also seems like
> it has less features.
Less, or just not configurable? ;-)
I mean, if it works it works ;-)
>
> $ qemu-system-x86_64 -device 'amd-iommu,help'
> amd-iommu options:
> device-iotlb=<bool> - (default: false)
> intremap=<OnOffAuto> - on/off/auto (default: "auto")
> pt=<bool> - (default: true)
> $ qemu-system-x86_64 -device 'intel-iommu,help'
> intel-iommu options:
> aw-bits=<uint8> - (default: 39)
> caching-mode=<bool> - (default: false)
> device-iotlb=<bool> - (default: false)
> dma-drain=<bool> - (default: true)
> dma-translation=<bool> - (default: true)
> eim=<OnOffAuto> - on/off/auto (default: "auto")
> intremap=<OnOffAuto> - on/off/auto (default: "auto")
> pt=<bool> - (default: true)
> snoop-control=<bool> - (default: false)
> version=<uint32> - (default: 0)
> x-buggy-eim=<bool> - (default: false)
> x-pasid-mode=<bool> - (default: false)
> x-scalable-mode=<bool> - (default: false)