From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id 2300B1FF15C
for <inbox@lore.proxmox.com>; Fri, 2 May 2025 13:52:30 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id A33EC1F8BC;
Fri, 2 May 2025 13:52:41 +0200 (CEST)
Message-ID: <01768d6c-227f-4d31-93de-cc752b8ebf07@proxmox.com>
Date: Fri, 2 May 2025 13:52:37 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
Fiona Ebner <f.ebner@proxmox.com>
References: <20250428101732.27679-1-f.ebner@proxmox.com>
Content-Language: en-US
From: Markus Frank <m.frank@proxmox.com>
In-Reply-To: <20250428101732.27679-1-f.ebner@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.001 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [virtiofs.pm, proxmox.com]
Subject: Re: [pve-devel] [PATCH qemu-server] virtiofs: prevent issue with
Windows OS and too many files
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
Hello,
On 2025-04-28 12:17, Fiona Ebner wrote:
> As reported in the community forum [0] and the virtio-win project [1],
> virtiofsd will run into its open file limit when used with a Windows
> guest that reads too many files. It's also reported that the issue
> does not occur with Linux guests and a workaround is using
> '--inode-file-handles=mandatory' on virtiofsd command line.
I was able to reproduce the issue with a directory containing a million files.
The virtiofs share became unusable when I tried to list the files with 'ls'.
This patch allows to list all the files without problems.
>
> The option is described as follows in the vritiofsd help:
typo: virtiofsd
>
>> When to use file handles to reference inodes instead of O_PATH file
>> descriptors (never, prefer, mandatory)
>
> and the default is 'never'.
>
> Fix the above issue by using 'prefer' rather than 'mandatory', because
> that should not break other edge cases:
>
>> prefer: Attempt to generate file handles, but fall back to O_PATH
>> file descriptors where the underlying filesystem does not support
>> file handles. Useful when there are various different filesystems
>> under the shared directory and some of them do not support file
>> handles.
>
> [0]: https://forum.proxmox.com/threads/165565/
> [1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
>
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Markus Frank <m.frank@proxmox.com>
> ---
>
> Didn't get around to measure the performance impact yet, so feel free
> to check that if you test this patch. While not being broken is more
> important than good performance, it would still be good to know for
> completeness.
I made a few read/write tests with fio in a Windows 11 guest.
without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --syslog):
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=11.0MiB/s (11.5MB/s), 11.0MiB/s-11.0MiB/s (11.5MB/s-11.5MB/s), io=665MiB (698MB), run=60667-60667msec
WRITE: bw=6054KiB/s (6200kB/s), 6054KiB/s-6054KiB/s (6200kB/s-6200kB/s), io=359MiB (376MB), run=60667-60667msec
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=11.1MiB/s (11.6MB/s), 11.1MiB/s-11.1MiB/s (11.6MB/s-11.6MB/s), io=665MiB (698MB), run=59933-59933msec
WRITE: bw=6129KiB/s (6276kB/s), 6129KiB/s-6129KiB/s (6276kB/s-6276kB/s), io=359MiB (376MB), run=59933-59933msec
with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --inode-file-handles=prefer --syslog):
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s), io=665MiB (698MB), run=68094-68094msec
WRITE: bw=5394KiB/s (5524kB/s), 5394KiB/s-5394KiB/s (5524kB/s-5524kB/s), io=359MiB (376MB), run=68094-68094msec
test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
READ: bw=10.8MiB/s (11.3MB/s), 10.8MiB/s-10.8MiB/s (11.3MB/s-11.3MB/s), io=665MiB (698MB), run=61461-61461msec
WRITE: bw=5976KiB/s (6120kB/s), 5976KiB/s-5976KiB/s (6120kB/s-6120kB/s), io=359MiB (376MB), run=61461-61461msec
(/share is a directory on an ext4 filesystem on the host)
>
> PVE/QemuServer/Virtiofs.pm | 8 ++++++--
> 1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/PVE/QemuServer/Virtiofs.pm b/PVE/QemuServer/Virtiofs.pm
> index cfde92c9..5a91b23a 100644
> --- a/PVE/QemuServer/Virtiofs.pm
> +++ b/PVE/QemuServer/Virtiofs.pm
> @@ -130,14 +130,17 @@ sub start_all_virtiofsd {
> next if !$conf->{$opt};
> my $virtiofs = parse_property_string('pve-qm-virtiofs', $conf->{$opt});
>
> - my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs);
> + # See https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
> + my $prefer_inode_fh = PVE::QemuServer::Helpers::windows_version($conf->{ostype}) ? 1 : 0;
> +
> + my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs, $prefer_inode_fh);
> push @$virtiofs_sockets, $virtiofs_socket;
> }
> return $virtiofs_sockets;
> }
>
> sub start_virtiofsd {
> - my ($vmid, $fsid, $virtiofs) = @_;
> + my ($vmid, $fsid, $virtiofs, $prefer_inode_fh) = @_;
>
> mkdir $socket_path_root;
> my $socket_path = "$socket_path_root/vm$vmid-fs$fsid";
> @@ -175,6 +178,7 @@ sub start_virtiofsd {
> push @$cmd, '--announce-submounts';
> push @$cmd, '--allow-direct-io' if $virtiofs->{'direct-io'};
> push @$cmd, '--cache='.$virtiofs->{cache} if $virtiofs->{cache};
> + push @$cmd, '--inode-file-handles=prefer' if $prefer_inode_fh;
> push @$cmd, '--syslog';
> exec(@$cmd);
> } elsif (!defined($pid2)) {
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel