From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A703294854 for ; Fri, 13 Jan 2023 10:51:52 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 883A035F1 for ; Fri, 13 Jan 2023 10:51:52 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 13 Jan 2023 10:51:50 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1F1F844874 for ; Fri, 13 Jan 2023 10:51:50 +0100 (CET) Date: Fri, 13 Jan 2023 10:51:48 +0100 From: Wolfgang Bumiller To: Markus Frank Cc: pve-devel@lists.proxmox.com Message-ID: <20230113095148.27nhzxmqdr4nmr6h@fwblub> References: <20221125140857.121622-1-m.frank@proxmox.com> <20221125140857.121622-3-m.frank@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20221125140857.121622-3-m.frank@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.211 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemu.pm, qemuconfig.pm, machine.pm, qemu.org, qemuserver.pm] Subject: Re: [pve-devel] [PATCH qemu-server v4 2/5] fix #3784: Parameter for guest vIOMMU & machine as property-string X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2023 09:51:52 -0000 On Fri, Nov 25, 2022 at 03:08:54PM +0100, Markus Frank wrote: > vIOMMU enables the option to passthrough pci devices to L2 VMs > in L1 VMs via Nested Virtualisation. > > QEMU-Parameters: > https://www.qemu.org/docs/master/system/qemu-manpage.html > https://wiki.qemu.org/Features/VT-d > > -machine ...,kernel-irqchip=split: > > "split" because of intremap see below. > > > -device intel-iommu: AFAICT qemu also has an amd-iommu - so shouldn't we check the host arch for which variant we need to use? > > * caching-mode=on: > > "It is required for -device vfio-pci to work with the VT-d device, because host > assigned devices requires to setup the DMA mapping on the host before guest DMA > starts." > > * intremap=on: > > "This enables interrupt remapping feature. It's required to enable complete > x2apic. Currently it only supports kvm kernel-irqchip modes off or split, while > full kernel-irqchip is not yet supported." > > > Signed-off-by: Markus Frank > --- > > for dmar on virtio-devices: > > * device-iotlb > > "This enables device-iotlb capability for the emulated VT-d device. So far > virtio/vhost should be the only real user for this parameter, paired with > ats=on configured for the device." > > * disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on: > > I did not find any good documentation. > Maybe someone can explain these parameters and how to use them right. > As I tried them with virtio-net-pci I got about 4-9 times less transfer-speed > when sending then without them. I mean, the viommu adds overhead, so I'd expect some downsides. - iommu_platform=on: Now, normally virtio devices can just directly access the guest memory since the hypervisor has full access. `iommu_platform=on` disables this, and it'll go through some generic DMA process that is supposed to deal with things such as AMD-SEV where the hypervisor doesn't actually have access to the full guest memory. I'd expect a large performance hit from that. I don't expect the others to make much of a difference, in fact, AFAICT disable-legacy shouldn't do much at all on modern guests I think. - 'disable-legacy=on': Virtio has evolved quite a bit and this option AFAICT disables support for "legacy" (pre-virtio-1.0) parts, but I don't know the details, you can probably read them in the virtio spec, it mentions things such as pci configuration space having been in native-endian rather than little-endian as is defined by PCI (apparently). There are apparently 3 "flavors" of virtio devices: legacy, transitional (supporting "IO" and "MMIO" modes (according to qemu's docs/pcie.txt)), and modern. Qemu seems to decide the defaults there depending on whether the device is on a pci or pcie port. disable-legacy and disable-modern override this explicitly. > > However these Parameters seem not to be necessary for passthroughing > Assigned Devices, so I would say "dmar for virtio" would be its own > separate feature. > > v4: > * added kvm/q35 checks in API > * reused pve-qemu-machine > > v3: > * replaced old machine type with property-string with viommu-parameter > > v2: > * moved viommu-parameter inside of machine_fmt and added it the new > parameter machine_properties > new Config -> machine_properties: viommu=1,etc > * check if kvm and q35 are set > > > PVE/API2/Qemu.pm | 21 ++++++++++++--- > PVE/QemuConfig.pm | 3 ++- > PVE/QemuServer.pm | 55 ++++++++++++++++++++++++++++++++++++--- > PVE/QemuServer/Machine.pm | 6 +++-- > 4 files changed, 75 insertions(+), 10 deletions(-) > > diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm > index badfc37..5268e56 100644 > --- a/PVE/API2/Qemu.pm > +++ b/PVE/API2/Qemu.pm > @@ -979,13 +979,19 @@ __PACKAGE__->register_method({ > $conf->{vmgenid} = PVE::QemuServer::generate_uuid(); > } > > - my $machine = $conf->{machine}; > + my $machine_conf = PVE::QemuServer::parse_machine($conf->{machine}); > + my $machine = $machine_conf->{type}; > if (!$machine || $machine =~ m/^(?:pc|q35|virt)$/) { > # always pin Windows' machine version on create, they get to easily confused > - if (PVE::QemuServer::Helpers::windows_version($conf->{ostype})) { > - $conf->{machine} = PVE::QemuServer::windows_get_pinned_machine_version($machine); > + if (PVE::QemuServer::windows_version($conf->{ostype})) { You dropped the Helpers::' part here, is this intentional? AFAICT "windows_version still lives in Helpers.pm? > + $machine_conf->{type} = PVE::QemuServer::windows_get_pinned_machine_version($machine); > + $conf->{machine} = PVE::QemuServer::print_machine($machine_conf); > } > } > + my $q35 = $machine_conf->{type} && ($machine_conf->{type} =~ m/q35/) ? 1 : 0; > + if ((!$conf->{kvm} || !$q35) && $machine_conf->{viommu}) { > + die "to use vIOMMU please enable kvm and set the machine type to q35\n" > + } > > PVE::QemuConfig->write_config($vmid, $conf); >