From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 2E3641FF186 for ; Fri, 29 Aug 2025 11:46:19 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BC7BE2A26D; Fri, 29 Aug 2025 11:46:28 +0200 (CEST) Mime-Version: 1.0 Date: Fri, 29 Aug 2025 11:46:24 +0200 Message-Id: Cc: "pve-devel" From: "Daniel Kral" To: "Proxmox VE development discussion" X-Mailer: aerc 0.20.0 References: <20250827150419.275285-1-d.kral@proxmox.com> In-Reply-To: <20250827150419.275285-1-d.kral@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1756460777066 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.014 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC qemu-server] fix #6608: expose viommu driver aw-bits option X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" On Wed Aug 27, 2025 at 5:03 PM CEST, Daniel Kral wrote: > Since QEMU 9.2 [0], the default I/O address space bit width was raised > from 39 bits to 48 bits for the Intel vIOMMU driver, which makes the > aw-bits check introduced in [1] to trip for host CPUs with less than 48 > bits physical address width from QEMU 9.2 onwards: > > vfio 0000:XX:YY.Z: Failed to set vIOMMU: aw-bits 48 > host aw-bits 39 > > For VFIO devices where a vIOMMU is in-use, QEMU fetches the IOVA ranges > with the iommufd ioctl IOMMU_IOAS_IOVA_RANGES or the vfio_iommu_type1's > VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE info, so 'phys-bits' doesn't change > the behavior of the check. > > Therefore, expose the 'aw-bits' option of the intel-iommu and > virtio-iommu QEMU drivers to allow users to set the value. > > [0] https://lore.kernel.org/qemu-devel/20241212083757.605022-17-zhenzhong.duan@intel.com/ > [1] https://lore.kernel.org/qemu-devel/20240605083043.317831-18-zhenzhong.duan@intel.com/ > > Signed-off-by: Daniel Kral > --- > There were quite a few changes in the way in qemu upstream since 9.0 for > the vIOMMU drivers to utilize the Intel VT-d's dual-stage vIOMMU > translation better, but I'm not entirely sure why the default value was > changed for legacy mode too, i.e. when scalable mode (x-scalable-mode) > and first level translation support (x-flts) is off, as I haven't looked > into it too much whether there are any strict requirements for this in > the future when 5-level paging is supported. > > My CPU itself reports 39 bits physical address size according to > /proc/cpuinfo and setting aw-bits=39 made the check mentioned above > happy and the VM startable again. I haven't tested this yet with any CPU > that has 46 or 48 bit physical address width. A user reported [0] that both errors vanished (the one mentioned above in the commit message and the vfio_container_dma_map(...) = -22 one) by setting the combination of cpu.guest-phys-bits and intel-iommu.aw-bits so that these are equal on systems where these differ. It seems like mostly Intel consumer-grade CPUs are the ones where these mismatch or are below the default 48 bits - it seems the physical address width ranges from anywhere between 39 and 48 bits on Intel CPUs; the other 2 AMD CPUs I checked were both 48 bits physical address width - even though these were quite beefy enthusiast 7900X / 9900X ones. There was a patch that wasn't applied in qemu upstream [1] that should warn users about the mismatch but wasn't perfect as one can see in the replies. I'll follow up on this patch with a possible check that compares the cpu's physical bits (or the phys-bits / guest-phys-bits) to the IOMMU's address width size, which can be found through the iommu's sysfs. [0] https://forum.proxmox.com/threads/169586/post-795813 [1] https://lore.kernel.org/qemu-devel/20250130134346.1754143-9-clg@redhat.com/ _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel