[pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[Alexandre Derumier via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 4454 bytes --]

From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-storage] qcow2 format: enable subcluster allocation by default
Date: Wed,  3 Jul 2024 16:24:47 +0200
Message-ID: <20240703142447.602210-1-alexandre.derumier@groupe-cyllene.com>

extended_l2 is an optimisation to reduce write amplification.
Currently,without it, when a vm write 4k, a full 64k cluster
need to be writen.

When enabled, the cluster is splitted in 32 subclusters.

We use a 128k cluster by default, to have 32 * 4k subclusters

https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/
https://static.sched.com/hosted_files/kvmforum2020/d9/qcow2-subcluster-allocation.pdf

some stats for 4k randwrite benchmark

Cluster size   Without subclusters     With subclusters
16 KB          5859 IOPS               8063 IOPS
32 KB          5674 IOPS               11107 IOPS
64 KB          2527 IOPS               12731 IOPS
128 KB         1576 IOPS               11808 IOPS
256 KB         976 IOPS                 9195 IOPS
512 KB         510 IOPS                 7079 IOPS
1 MB           448 IOPS                 3306 IOPS
2 MB           262 IOPS                 2269 IOPS

Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
---
 src/PVE/Storage/Plugin.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 6444390..31b20fe 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -561,7 +561,7 @@ sub preallocation_cmd_option {
 	die "preallocation mode '$prealloc' not supported by format '$fmt'\n"
 	    if !$QCOW2_PREALLOCATION->{$prealloc};
 
-	return "preallocation=$prealloc";
+	return "preallocation=$prealloc,extended_l2=on,cluster_size=128k";
     } elsif ($fmt eq 'raw') {
 	$prealloc = $prealloc // 'off';
 	$prealloc = 'off' if $prealloc eq 'metadata';
-- 
2.39.2



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[Fiona Ebner]

Am 03.07.24 um 16:24 schrieb Alexandre Derumier via pve-devel:
> 
> 
> extended_l2 is an optimisation to reduce write amplification.
> Currently,without it, when a vm write 4k, a full 64k cluster

s/write/writes/

> need to be writen.

needs to be written.

> 
> When enabled, the cluster is splitted in 32 subclusters.

s/splitted/split/

> 
> We use a 128k cluster by default, to have 32 * 4k subclusters
> 
> https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/
> https://static.sched.com/hosted_files/kvmforum2020/d9/qcow2-subcluster-allocation.pdf
> 
> some stats for 4k randwrite benchmark

Can you please share the exact command you used? What kind of underlying
disks do you have?

> 
> Cluster size   Without subclusters     With subclusters
> 16 KB          5859 IOPS               8063 IOPS
> 32 KB          5674 IOPS               11107 IOPS
> 64 KB          2527 IOPS               12731 IOPS
> 128 KB         1576 IOPS               11808 IOPS
> 256 KB         976 IOPS                 9195 IOPS
> 512 KB         510 IOPS                 7079 IOPS
> 1 MB           448 IOPS                 3306 IOPS
> 2 MB           262 IOPS                 2269 IOPS
> 

How does read performance compare for you (with 128 KiB cluster size)?

I don't see any noticeable difference in my testing with an ext4
directory storage on an SSD, attaching the qcow2 images as SCSI disks to
the VM, neither for reading nor writing. I only tested without your
change and with your change using 4k (rand)read and (rand)write.

I'm not sure we should enable this for everybody, there's always a risk
to break stuff with added complexity. Maybe it's better to have a
storage configuration option that people can opt-in to, e.g.

qcow2-create-opts extended_l2=on,cluster_size=128k

If we get enough positive feedback, we can still change the default in a
future (major) release.

> Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
> ---
>  src/PVE/Storage/Plugin.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 6444390..31b20fe 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -561,7 +561,7 @@ sub preallocation_cmd_option {
>  	die "preallocation mode '$prealloc' not supported by format '$fmt'\n"
>  	    if !$QCOW2_PREALLOCATION->{$prealloc};
>  
> -	return "preallocation=$prealloc";
> +	return "preallocation=$prealloc,extended_l2=on,cluster_size=128k";

Also, it doesn't really fit here in the preallocation helper as the
helper is specific to that setting.

>      } elsif ($fmt eq 'raw') {
>  	$prealloc = $prealloc // 'off';
>  	$prealloc = 'off' if $prealloc eq 'metadata';
> -- 
> 2.39.2
> 
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 16382 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "f.ebner@proxmox.com" <f.ebner@proxmox.com>
Subject: Re: [pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default
Date: Thu, 14 Nov 2024 08:31:06 +0000
Message-ID: <98cdc246d14fdfc5dcfedf09dd4bc596acb0814f.camel@groupe-cyllene.com>

Hi Fiona,

I'm really sorry, I didn't see your reponse, lost in the flood of email
:(


>>How does read performance compare for you (with 128 KiB cluster
size)?

>>I don't see any noticeable difference in my testing with an ext4
>>directory storage on an SSD, attaching the qcow2 images as SCSI disks
>>to
>>the VM, neither for reading nor writing. I only tested without your
>>change and with your change using 4k (rand)read and (rand)write.
>>
>>I'm not sure we should enable this for everybody, there's always a
>>risk
>>to break stuff with added complexity. Maybe it's better to have a
>>storage configuration option that people can opt-in to, e.g.
>>
>>qcow2-create-opts extended_l2=on,cluster_size=128k
>>
>>If we get enough positive feedback, we can still change the default
>>in a
>>future (major) release.

What disk size do you use for your bench ? 
It's really important, because generally, the more bigger, the slower
read in qcow2 will be , because l2 metadatas need to be cached handle
in memory. (and qemu have 1MB cache by default)

witout subcluster, for 1TB image, it's around 128MB l2 metadas in qcow2
file. 

for write, it's mostly for first block allocation, so it's depend
of the filesystem behind, if fallocate is supported.
(I have seen really big difference on shared ocfs2/gfs2 fs)

Also, for write, this is helping a lot with backing file. (so for
linked qcow2 clone, and maybe in the future for external snapshots).
Because currently, if you write 4k on a overlay , you need to read the
full cluster 64k on the base write, and rewrite it. (so 8x
overamplification)


They are good information from the developper :
https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/


I really don't think it's hurt, but maybe a default for 9.0 release to
be safe. (I'll try to have the external snapshot ready for this date
too)


Regards,

Alexandre

> Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-
> cyllene.com>
> ---
>  src/PVE/Storage/Plugin.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 6444390..31b20fe 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -561,7 +561,7 @@ sub preallocation_cmd_option {
>   die "preallocation mode '$prealloc' not supported by format
> '$fmt'\n"
>       if !$QCOW2_PREALLOCATION->{$prealloc};
>  
> - return "preallocation=$prealloc";
> + return "preallocation=$prealloc,extended_l2=on,cluster_size=128k";

Also, it doesn't really fit here in the preallocation helper as the
helper is specific to that setting.

>      } elsif ($fmt eq 'raw') {
>   $prealloc = $prealloc // 'off';
>   $prealloc = 'off' if $prealloc eq 'metadata';
> -- 
> 2.39.2
> 
> 



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[Fiona Ebner]

Am 14.11.24 um 09:31 schrieb DERUMIER, Alexandre:
> Hi Fiona,
> 
> I'm really sorry, I didn't see your reponse, lost in the flood of email
> :(
> 
> 
>>> How does read performance compare for you (with 128 KiB cluster
> size)?
> 
>>> I don't see any noticeable difference in my testing with an ext4
>>> directory storage on an SSD, attaching the qcow2 images as SCSI disks
>>> to
>>> the VM, neither for reading nor writing. I only tested without your
>>> change and with your change using 4k (rand)read and (rand)write.
>>>
>>> I'm not sure we should enable this for everybody, there's always a
>>> risk
>>> to break stuff with added complexity. Maybe it's better to have a
>>> storage configuration option that people can opt-in to, e.g.
>>>
>>> qcow2-create-opts extended_l2=on,cluster_size=128k
>>>
>>> If we get enough positive feedback, we can still change the default
>>> in a
>>> future (major) release.
> 
> What disk size do you use for your bench ? 
> It's really important, because generally, the more bigger, the slower
> read in qcow2 will be , because l2 metadatas need to be cached handle
> in memory. (and qemu have 1MB cache by default)
> 

I don't recall the exact size. Today I retested with a 900 GiB image.

> witout subcluster, for 1TB image, it's around 128MB l2 metadas in qcow2
> file. 
> 
> for write, it's mostly for first block allocation, so it's depend
> of the filesystem behind, if fallocate is supported.
> (I have seen really big difference on shared ocfs2/gfs2 fs)
> 

Okay, I see.

Mine are backed by an ext4 storage. With
'extended_l2=on,cluster_size=128k' I actually get much slower results
for the initial allocations (note that this is with
preallocation=metadata - without preallocation, I still see "without"
being about 1.5 times faster, i.e. 375MB written versus 255MB, however
the usage on the underlying storage is much(!) worse, i.e. 5.71 GiB
versus 373 MiB):

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randwrite --numjobs=1 --group_reporting --time_based --runtime 60

With options:

> WRITE: bw=5603KiB/s (5737kB/s), 5603KiB/s-5603KiB/s (5737kB/s-5737kB/s), io=328MiB (344MB), run=60001-60001msec

Without:

> WRITE: bw=27.1MiB/s (28.4MB/s), 27.1MiB/s-27.1MiB/s (28.4MB/s-28.4MB/s), io=1626MiB (1705MB), run=60001-60001msec

For a subsequent randrw:

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randrw --numjobs=1 --group_reporting --time_based --runtime 60

With options:

>    READ: bw=7071KiB/s (7241kB/s), 7071KiB/s-7071KiB/s (7241kB/s-7241kB/s), io=414MiB (434MB), run=60001-60001msec
>   WRITE: bw=7058KiB/s (7227kB/s), 7058KiB/s-7058KiB/s (7227kB/s-7227kB/s), io=414MiB (434MB), run=60001-60001msec

Without:

>    READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=708MiB (742MB), run=60001-60001msec
>   WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60001-60001msec

Took a snapshot and afterwards I get:

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randrw --numjobs=1 --group_reporting --time_based --runtime 60

With options:

>    READ: bw=1250KiB/s (1280kB/s), 1250KiB/s-1250KiB/s (1280kB/s-1280kB/s), io=73.3MiB (76.8MB), run=60003-60003msec
>   WRITE: bw=1251KiB/s (1281kB/s), 1251KiB/s-1251KiB/s (1281kB/s-1281kB/s), io=73.3MiB (76.9MB), run=60003-60003msec

Without:

>    READ: bw=1198KiB/s (1227kB/s), 1198KiB/s-1198KiB/s (1227kB/s-1227kB/s), io=70.2MiB (73.6MB), run=60002-60002msec
>   WRITE: bw=1201KiB/s (1230kB/s), 1201KiB/s-1201KiB/s (1230kB/s-1230kB/s), io=70.4MiB (73.8MB), run=60002-60002msec

But there is a big difference in the read performance, this time in
favor of "with options" (but there is less allocated on the image, as a
consequence of the previous tests):

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randread --numjobs=1 --group_reporting --time_based --runtime 60

With options:

> READ: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=1871MiB (1962MB), run=60001-60001msec

Without:

> READ: bw=19.2MiB/s (20.1MB/s), 19.2MiB/s-19.2MiB/s (20.1MB/s-20.1MB/s), io=1151MiB (1207MB), run=60001-60001msec


> Also, for write, this is helping a lot with backing file. (so for
> linked qcow2 clone, and maybe in the future for external snapshots).
> Because currently, if you write 4k on a overlay , you need to read the
> full cluster 64k on the base write, and rewrite it. (so 8x
> overamplification)
> 

For clone, we should add the options in clone_image() too ;)

> 
> They are good information from the developper :
> https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/
> 
> 
> I really don't think it's hurt, but maybe a default for 9.0 release to
> be safe. (I'll try to have the external snapshot ready for this date
> too)
> 

It seems to hurt at least in some cases (initial allocation speed can be
worse than without the option, in particular when using
preallocation=metadata). If we have it as opt-in already, users can give
it a try and then we can still think about making it the default for
9.0, should there be enough positive feedback.

We could also think about only using it for linked clones by default
initially.

Independently, you can make it the default for LvmQcow2Plugin of course,
since you see much better results for that use case :)


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel