From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id 1AFF81FF15C
for <inbox@lore.proxmox.com>; Fri, 2 May 2025 16:15:04 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 3BFD831886;
Fri, 2 May 2025 16:15:15 +0200 (CEST)
Message-ID: <caa85f4b-77b6-4043-8513-754d0b6e85b0@proxmox.com>
Date: Fri, 2 May 2025 16:15:10 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
From: Markus Frank <m.frank@proxmox.com>
To: Fiona Ebner <f.ebner@proxmox.com>,
Proxmox VE development discussion <pve-devel@lists.proxmox.com>
References: <20250428101732.27679-1-f.ebner@proxmox.com>
<01768d6c-227f-4d31-93de-cc752b8ebf07@proxmox.com>
<31df6e00-1bc0-4983-8f7e-4101604cdd50@proxmox.com>
Content-Language: en-US
In-Reply-To: <31df6e00-1bc0-4983-8f7e-4101604cdd50@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.001 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [proxmox.com]
Subject: Re: [pve-devel] [PATCH qemu-server] virtiofs: prevent issue with
Windows OS and too many files
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
On 2025-05-02 14:13, Fiona Ebner wrote:
> Am 02.05.25 um 13:52 schrieb Markus Frank:
>> On 2025-04-28 12:17, Fiona Ebner wrote:
>>> As reported in the community forum [0] and the virtio-win project [1],
>>> virtiofsd will run into its open file limit when used with a Windows
>>> guest that reads too many files. It's also reported that the issue
>>> does not occur with Linux guests and a workaround is using
>>> '--inode-file-handles=mandatory' on virtiofsd command line.
>> I was able to reproduce the issue with a directory containing a million
>> files.
>> The virtiofs share became unusable when I tried to list the files with
>> 'ls'.
>> This patch allows to list all the files without problems.
>>>
>>> The option is described as follows in the vritiofsd help:
>> typo: virtiofsd
>
> Will fix!
>
>>>
>>>> When to use file handles to reference inodes instead of O_PATH file
>>>> descriptors (never, prefer, mandatory)
>>>
>>> and the default is 'never'.
>>>
>>> Fix the above issue by using 'prefer' rather than 'mandatory', because
>>> that should not break other edge cases:
>>>
>>>> prefer: Attempt to generate file handles, but fall back to O_PATH
>>>> file descriptors where the underlying filesystem does not support
>>>> file handles. Useful when there are various different filesystems
>>>> under the shared directory and some of them do not support file
>>>> handles.
>>>
>>> [0]: https://forum.proxmox.com/threads/165565/
>>> [1]: https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1136
>>>
>>> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
>> Tested-by: Markus Frank <m.frank@proxmox.com>
>>> ---
>>>
>>> Didn't get around to measure the performance impact yet, so feel free
>>> to check that if you test this patch. While not being broken is more
>>> important than good performance, it would still be good to know for
>>> completeness.
>> I made a few read/write tests with fio in a Windows 11 guest.
>
> Since the option affects file handles, it would be more interesting to
> test something that involves handling many files rather than just IO.
Okay, here is a test that creates 10000 4kB files.
without this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --syslog):
PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k --directory="testdir" --ioengine=windowsaio --create_on_open=1 --file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 4 (f=18): [_(3),f(1),_(2),f(1),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8000: Fri May 2 15:45:44 2025
write: IOPS=967, BW=3870KiB/s (3962kB/s)(114MiB/30049msec); 0 zone resets
slat (nsec): min=0, max=632300, avg=20663.88, stdev=13715.30
clat (usec): min=494, max=12174, avg=2302.34, stdev=676.20
lat (usec): min=512, max=12195, avg=2323.00, stdev=675.62
clat percentiles (usec):
| 1.00th=[ 1037], 5.00th=[ 1352], 10.00th=[ 1532], 20.00th=[ 1745],
| 30.00th=[ 1926], 40.00th=[ 2073], 50.00th=[ 2212], 60.00th=[ 2376],
| 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3195], 95.00th=[ 3523],
| 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5211], 99.95th=[ 5669],
| 99.99th=[ 8455]
bw ( KiB/s): min= 1621, max= 4387, per=99.24%, avg=3840.90, stdev=41.34, samples=600
iops : min= 401, max= 1094, avg=957.38, stdev=10.37, samples=600
lat (usec) : 500=0.01%, 750=0.10%, 1000=0.70%
lat (msec) : 2=34.57%, 4=62.77%, 10=1.86%, 20=0.01%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,29069,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=3870KiB/s (3962kB/s), 3870KiB/s-3870KiB/s (3962kB/s-3962kB/s), io=114MiB (119MB), run=30049-30049msec
with this patch (/usr/libexec/virtiofsd --fd=15 --shared-dir=/share --announce-submounts --inode-file-handles=prefer --syslog):
PS Z:\> fio --name=filetest --rw=write --bs=4k --size=4k --numjobs=10 --filesize=4k --directory="testdir" --ioengine=windowsaio --create_on_open=1 --file_service_type=random --nrfiles=10000 --runtime=30 --time_based --group_reporting
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
filetest: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=1
...
fio-3.39
Starting 10 threads
Jobs: 6 (f=318): [_(1),f(1),_(1),f(1),_(1),f(2),_(1),f(2)][100.0%][eta 00m:00s]
filetest: (groupid=0, jobs=10): err= 0: pid=8872: Fri May 2 15:55:39 2025
write: IOPS=964, BW=3859KiB/s (3952kB/s)(113MiB/30028msec); 0 zone resets
slat (nsec): min=0, max=1623.2k, avg=20705.88, stdev=16208.95
clat (usec): min=462, max=25441, avg=2312.40, stdev=778.69
lat (usec): min=489, max=25458, avg=2333.11, stdev=778.39
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1336], 10.00th=[ 1516], 20.00th=[ 1745],
| 30.00th=[ 1926], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2409],
| 70.00th=[ 2573], 80.00th=[ 2802], 90.00th=[ 3163], 95.00th=[ 3523],
| 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5473], 99.95th=[ 7832],
| 99.99th=[25035]
bw ( KiB/s): min= 1180, max= 4385, per=98.89%, avg=3816.80, stdev=44.92, samples=600
iops : min= 291, max= 1094, avg=952.10, stdev=11.25, samples=600
lat (usec) : 500=0.01%, 750=0.14%, 1000=0.86%
lat (msec) : 2=33.73%, 4=63.43%, 10=1.79%, 20=0.02%, 50=0.03%
cpu : usr=0.00%, sys=0.00%, ctx=0, majf=0, minf=0
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,28969,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=3859KiB/s (3952kB/s), 3859KiB/s-3859KiB/s (3952kB/s-3952kB/s), io=113MiB (119MB), run=30028-30028msec
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel