[pve-devel] [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[pve-devel] [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[Fiona Ebner]

As reported in the community forum [0] and the virtio-win project [1],
virtiofsd will run into its open file limit when used with a Windows
guest that reads too many files. It's also reported that the issue
does not occur with Linux guests and a workaround is using
'--inode-file-handles=mandatory' on virtiofsd command line.

The option is described as follows in the vritiofsd help:

> When to use file handles to reference inodes instead of O_PATH file
> descriptors (never, prefer, mandatory)

and the default is 'never'.

Fix the above issue by using 'prefer' rather than 'mandatory', because
that should not break other edge cases:

> prefer: Attempt to generate file handles, but fall back to O_PATH
> file descriptors where the underlying filesystem does not support
> file handles. Useful when there are various different filesystems
> under the shared directory and some of them do not support file
> handles.

[0]: https://forum.proxmox.com/threads/165565/
[1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Didn't get around to measure the performance impact yet, so feel free
to check that if you test this patch. While not being broken is more
important than good performance, it would still be good to know for
completeness.

 PVE/QemuServer/Virtiofs.pm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/PVE/QemuServer/Virtiofs.pm b/PVE/QemuServer/Virtiofs.pm
index cfde92c9..5a91b23a 100644
--- a/PVE/QemuServer/Virtiofs.pm
+++ b/PVE/QemuServer/Virtiofs.pm
@@ -130,14 +130,17 @@ sub start_all_virtiofsd {
 	next if !$conf->{$opt};
 	my $virtiofs = parse_property_string('pve-qm-virtiofs', $conf->{$opt});
 
-	my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs);
+	# See https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
+	my $prefer_inode_fh = PVE::QemuServer::Helpers::windows_version($conf->{ostype}) ? 1 : 0;
+
+	my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs, $prefer_inode_fh);
 	push @$virtiofs_sockets, $virtiofs_socket;
     }
     return $virtiofs_sockets;
 }
 
 sub start_virtiofsd {
-    my ($vmid, $fsid, $virtiofs) = @_;
+    my ($vmid, $fsid, $virtiofs, $prefer_inode_fh) = @_;
 
     mkdir $socket_path_root;
     my $socket_path = "$socket_path_root/vm$vmid-fs$fsid";
@@ -175,6 +178,7 @@ sub start_virtiofsd {
 	    push @$cmd, '--announce-submounts';
 	    push @$cmd, '--allow-direct-io' if $virtiofs->{'direct-io'};
 	    push @$cmd, '--cache='.$virtiofs->{cache} if $virtiofs->{cache};
+	    push @$cmd, '--inode-file-handles=prefer' if $prefer_inode_fh;
 	    push @$cmd, '--syslog';
 	    exec(@$cmd);
 	} elsif (!defined($pid2)) {
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[Markus Frank]

Hello,

On 2025-04-28 12:17, Fiona Ebner wrote:
> As reported in the community forum [0] and the virtio-win project [1],
> virtiofsd will run into its open file limit when used with a Windows
> guest that reads too many files. It's also reported that the issue
> does not occur with Linux guests and a workaround is using
> '--inode-file-handles=mandatory' on virtiofsd command line.
I was able to reproduce the issue with a directory containing a million files.
The virtiofs share became unusable when I tried to list the files with 'ls'.
This patch allows to list all the files without problems.
> 
> The option is described as follows in the vritiofsd help:
typo: virtiofsd
> 
>> When to use file handles to reference inodes instead of O_PATH file
>> descriptors (never, prefer, mandatory)
> 
> and the default is 'never'.
> 
> Fix the above issue by using 'prefer' rather than 'mandatory', because
> that should not break other edge cases:
> 
>> prefer: Attempt to generate file handles, but fall back to O_PATH
>> file descriptors where the underlying filesystem does not support
>> file handles. Useful when there are various different filesystems
>> under the shared directory and some of them do not support file
>> handles.
> 
> [0]: https://forum.proxmox.com/threads/165565/
> [1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
> 
> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Markus Frank <m.frank@proxmox.com>
> ---
> 
> Didn't get around to measure the performance impact yet, so feel free
> to check that if you test this patch. While not being broken is more
> important than good performance, it would still be good to know for
> completeness.
I made a few read/write tests with fio in a Windows 11 guest.


without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --syslog):

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
    READ: bw=11.0MiB/s (11.5MB/s), 11.0MiB/s-11.0MiB/s (11.5MB/s-11.5MB/s), io=665MiB (698MB), run=60667-60667msec
   WRITE: bw=6054KiB/s (6200kB/s), 6054KiB/s-6054KiB/s (6200kB/s-6200kB/s), io=359MiB (376MB), run=60667-60667msec

test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
    READ: bw=11.1MiB/s (11.6MB/s), 11.1MiB/s-11.1MiB/s (11.6MB/s-11.6MB/s), io=665MiB (698MB), run=59933-59933msec
   WRITE: bw=6129KiB/s (6276kB/s), 6129KiB/s-6129KiB/s (6276kB/s-6276kB/s), io=359MiB (376MB), run=59933-59933msec


with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --inode-file-handles=prefer --syslog):

test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
    READ: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s), io=665MiB (698MB), run=68094-68094msec
   WRITE: bw=5394KiB/s (5524kB/s), 5394KiB/s-5394KiB/s (5524kB/s-5524kB/s), io=359MiB (376MB), run=68094-68094msec

test: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32
Run status group 0 (all jobs):
    READ: bw=10.8MiB/s (11.3MB/s), 10.8MiB/s-10.8MiB/s (11.3MB/s-11.3MB/s), io=665MiB (698MB), run=61461-61461msec
   WRITE: bw=5976KiB/s (6120kB/s), 5976KiB/s-5976KiB/s (6120kB/s-6120kB/s), io=359MiB (376MB), run=61461-61461msec


(/share is a directory on an ext4 filesystem on the host)

> 
>   PVE/QemuServer/Virtiofs.pm | 8 ++++++--
>   1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/PVE/QemuServer/Virtiofs.pm b/PVE/QemuServer/Virtiofs.pm
> index cfde92c9..5a91b23a 100644
> --- a/PVE/QemuServer/Virtiofs.pm
> +++ b/PVE/QemuServer/Virtiofs.pm
> @@ -130,14 +130,17 @@ sub start_all_virtiofsd {
>   	next if !$conf->{$opt};
>   	my $virtiofs = parse_property_string('pve-qm-virtiofs', $conf->{$opt});
>   
> -	my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs);
> +	# See https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
> +	my $prefer_inode_fh = PVE::QemuServer::Helpers::windows_version($conf->{ostype}) ? 1 : 0;
> +
> +	my $virtiofs_socket = start_virtiofsd($vmid, $i, $virtiofs, $prefer_inode_fh);
>   	push @$virtiofs_sockets, $virtiofs_socket;
>       }
>       return $virtiofs_sockets;
>   }
>   
>   sub start_virtiofsd {
> -    my ($vmid, $fsid, $virtiofs) = @_;
> +    my ($vmid, $fsid, $virtiofs, $prefer_inode_fh) = @_;
>   
>       mkdir $socket_path_root;
>       my $socket_path = "$socket_path_root/vm$vmid-fs$fsid";
> @@ -175,6 +178,7 @@ sub start_virtiofsd {
>   	    push @$cmd, '--announce-submounts';
>   	    push @$cmd, '--allow-direct-io' if $virtiofs->{'direct-io'};
>   	    push @$cmd, '--cache='.$virtiofs->{cache} if $virtiofs->{cache};
> +	    push @$cmd, '--inode-file-handles=prefer' if $prefer_inode_fh;
>   	    push @$cmd, '--syslog';
>   	    exec(@$cmd);
>   	} elsif (!defined($pid2)) {



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[Fiona Ebner]

Am 02.05.25 um 13:52 schrieb Markus Frank:
> On 2025-04-28 12:17, Fiona Ebner wrote:
>> As reported in the community forum [0] and the virtio-win project [1],
>> virtiofsd will run into its open file limit when used with a Windows
>> guest that reads too many files. It's also reported that the issue
>> does not occur with Linux guests and a workaround is using
>> '--inode-file-handles=mandatory' on virtiofsd command line.
> I was able to reproduce the issue with a directory containing a million
> files.
> The virtiofs share became unusable when I tried to list the files with
> 'ls'.
> This patch allows to list all the files without problems.
>>
>> The option is described as follows in the vritiofsd help:
> typo: virtiofsd

Will fix!

>>
>>> When to use file handles to reference inodes instead of O_PATH file
>>> descriptors (never, prefer, mandatory)
>>
>> and the default is 'never'.
>>
>> Fix the above issue by using 'prefer' rather than 'mandatory', because
>> that should not break other edge cases:
>>
>>> prefer: Attempt to generate file handles, but fall back to O_PATH
>>> file descriptors where the underlying filesystem does not support
>>> file handles. Useful when there are various different filesystems
>>> under the shared directory and some of them do not support file
>>> handles.
>>
>> [0]: https://forum.proxmox.com/threads/165565/
>> [1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
>>
>> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Markus Frank <m.frank@proxmox.com>
>> ---
>>
>> Didn't get around to measure the performance impact yet, so feel free
>> to check that if you test this patch. While not being broken is more
>> important than good performance, it would still be good to know for
>> completeness.
> I made a few read/write tests with fio in a Windows 11 guest.

Since the option affects file handles, it would be more interesting to
test something that involves handling many files rather than just IO.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[Markus Frank]

On 2025-05-02 14:13, Fiona Ebner wrote:
> Am 02.05.25 um 13:52 schrieb Markus Frank:
>> On 2025-04-28 12:17, Fiona Ebner wrote:
>>> As reported in the community forum [0] and the virtio-win project [1],
>>> virtiofsd will run into its open file limit when used with a Windows
>>> guest that reads too many files. It's also reported that the issue
>>> does not occur with Linux guests and a workaround is using
>>> '--inode-file-handles=mandatory' on virtiofsd command line.
>> I was able to reproduce the issue with a directory containing a million
>> files.
>> The virtiofs share became unusable when I tried to list the files with
>> 'ls'.
>> This patch allows to list all the files without problems.
>>>
>>> The option is described as follows in the vritiofsd help:
>> typo: virtiofsd
> 
> Will fix!
> 
>>>
>>>> When to use file handles to reference inodes instead of O_PATH file
>>>> descriptors (never, prefer, mandatory)
>>>
>>> and the default is 'never'.
>>>
>>> Fix the above issue by using 'prefer' rather than 'mandatory', because
>>> that should not break other edge cases:
>>>
>>>> prefer: Attempt to generate file handles, but fall back to O_PATH
>>>> file descriptors where the underlying filesystem does not support
>>>> file handles. Useful when there are various different filesystems
>>>> under the shared directory and some of them do not support file
>>>> handles.
>>>
>>> [0]: https://forum.proxmox.com/threads/165565/
>>> [1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
>>>
>>> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
>> Tested-by: Markus Frank <m.frank@proxmox.com>
>>> ---
>>>
>>> Didn't get around to measure the performance impact yet, so feel free
>>> to check that if you test this patch. While not being broken is more
>>> important than good performance, it would still be good to know for
>>> completeness.
>> I made a few read/write tests with fio in a Windows 11 guest.
> 
> Since the option affects file handles, it would be more interesting to
> test something that involves handling many files rather than just IO.

Okay, here is a test that creates 10000 4kB files.

without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --syslog):

PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k --directory="testdir" --ioengine=windowsaio --create_on_open=1 --file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 4 (f=18): [_(3),f(1),_(2),f(1),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8000: Fri May 2 15:45:44 2025
   write: IOPS=967, BW=3870KiB/s (3962kB/s)(114MiB/30049msec); 0 zone resets
     slat (nsec): min=0, max=632300, avg=20663.88, stdev=13715.30
     clat (usec): min=494, max=12174, avg=2302.34, stdev=676.20
      lat (usec): min=512, max=12195, avg=2323.00, stdev=675.62
     clat percentiles (usec):
      |  1.00th=[ 1037],  5.00th=[ 1352], 10.00th=[ 1532], 20.00th=[ 1745],
      | 30.00th=[ 1926], 40.00th=[ 2073], 50.00th=[ 2212], 60.00th=[ 2376],
      | 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3195], 95.00th=[ 3523],
      | 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5211], 99.95th=[ 5669],
      | 99.99th=[ 8455]
    bw (  KiB/s): min= 1621, max= 4387, per=99.24%, avg=3840.90, stdev=41.34, samples=600
    iops        : min=  401, max= 1094, avg=957.38, stdev=10.37, samples=600
   lat (usec)   : 500=0.01%, 750=0.10%, 1000=0.70%
   lat (msec)   : 2=34.57%, 4=62.77%, 10=1.86%, 20=0.01%
   cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      issued rwts: total=0,29069,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   WRITE: bw=3870KiB/s (3962kB/s), 3870KiB/s-3870KiB/s (3962kB/s-3962kB/s), io=114MiB (119MB), run=30049-30049msec



with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --inode-file-handles=prefer --syslog):

PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k --directory="testdir" --ioengine=windowsaio --create_on_open=1 --file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 6 (f=318): [_(1),f(1),_(1),f(1),_(1),f(2),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8872: Fri May 2 15:55:39 2025
   write: IOPS=964, BW=3859KiB/s (3952kB/s)(113MiB/30028msec); 0 zone resets
     slat (nsec): min=0, max=1623.2k, avg=20705.88, stdev=16208.95
     clat (usec): min=462, max=25441, avg=2312.40, stdev=778.69
      lat (usec): min=489, max=25458, avg=2333.11, stdev=778.39
     clat percentiles (usec):
      |  1.00th=[  996],  5.00th=[ 1336], 10.00th=[ 1516], 20.00th=[ 1745],
      | 30.00th=[ 1926], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2409],
      | 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3163], 95.00th=[ 3523],
      | 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5473], 99.95th=[ 7832],
      | 99.99th=[25035]
    bw (  KiB/s): min= 1180, max= 4385, per=98.89%, avg=3816.80, stdev=44.92, samples=600
    iops        : min=  291, max= 1094, avg=952.10, stdev=11.25, samples=600
   lat (usec)   : 500=0.01%, 750=0.14%, 1000=0.86%
   lat (msec)   : 2=33.73%, 4=63.43%, 10=1.79%, 20=0.02%, 50=0.03%
   cpu          : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
      issued rwts: total=0,28969,0,0 short=0,0,0,0 dropped=0,0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   WRITE: bw=3859KiB/s (3952kB/s), 3859KiB/s-3859KiB/s (3952kB/s-3952kB/s), io=113MiB (119MB), run=30028-30028msec



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] superseded: [PATCH qemu-server] virtiofs: prevent issue with Windows OS and too many files

[Fiona Ebner]

v2 here:
https://lore.proxmox.com/pve-devel/20250502142133.59401-1-f.ebner@proxmox.com/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel