[pve-devel] qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?

[pve-devel] qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14195 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?
Date: Fri, 30 Aug 2024 16:01:44 +0000
Message-ID: <fbddd72caac9d3474d63bdfcdb34bc515059ad70.camel@groupe-cyllene.com>

Hi,

I was doing tests with gfs2 && ocfs2,

and I have notice 4k randwrite iops going from 20000 iops to
100~200iops when a snapshot is present.

I thinked it was related to gfs2 && ocfs2 allocation, 
but I can reproduce too with a simple qcow2 file on
a local ssd drive.

is it expected ?

(I don't have use qcow2 snapshot since 10year, so I really don't 
 remember performance).


Using external snapshot file, I'm around 10000iops.

test inside the vm:

fio --filename=/dev/sdX --ioengine=libaio --direct 1 --bs=4k  --
rw=randwrite --numjobs=64



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?

[Fiona Ebner]

Hi,

Am 30.08.24 um 18:01 schrieb DERUMIER, Alexandre via pve-devel:
> Hi,
> 
> I was doing tests with gfs2 && ocfs2,
> 
> and I have notice 4k randwrite iops going from 20000 iops to
> 100~200iops when a snapshot is present.
> 
> I thinked it was related to gfs2 && ocfs2 allocation, 
> but I can reproduce too with a simple qcow2 file on
> a local ssd drive.
> 

Is the performance drop also this big with the local storage?

> is it expected ?
> 
> (I don't have use qcow2 snapshot since 10year, so I really don't 
>  remember performance).
> 

A performance drop is of course expected, because AFAIU it needs to do
COW for the sectors that the snapshot references, but the drop does
sound rather dramatic in your case. The performance should be better
again once the COW is done, e.g. when running another test afterwards. I
quickly tested in a nested VM with qcow2 disk on a directory storage and
--numjobs=4:

ran test, got ~33k IOPS
made a snapshot
ran test, got ~20k IOPS (it starts out very slow and then recovers)
ran test, got ~33k IOPS again

Best Regards,
Fiona

> 
> Using external snapshot file, I'm around 10000iops.
> 
> test inside the vm:
> 
> fio --filename=/dev/sdX --ioengine=libaio --direct 1 --bs=4k  --
> rw=randwrite --numjobs=64
> 
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 15671 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "f.ebner@proxmox.com" <f.ebner@proxmox.com>
Subject: Re: [pve-devel] qcow2 internal snapshot: 4k write speed reduce to 100~200 iops on ssd ?
Date: Thu, 19 Sep 2024 13:50:41 +0000
Message-ID: <a58e0f18bd41976deed2e7177a9d1d26956f06e6.camel@groupe-cyllene.com>

>>Hi,

Hi Fiona, 

sorry I didn't see your response.


>>Is the performance drop also this big with the local storage?

yes. (The result is even worst on nfs, or gfs/ocfs2)


>>A performance drop is of course expected, because AFAIU it needs to
>>do
>>COW for the sectors that the snapshot references, but the drop does
>>sound rather dramatic in your case. The performance should be better
>>again once the COW is done, e.g. when running another test
>>afterwards. I
>>quickly tested in a nested VM with qcow2 disk on a directory storage
and
>>--numjobs=4:²
>>
>>ran test, got ~33k IOPS
>>made a snapshot
>>ran test, got ~20k IOPS (it starts out very slow and then recovers)
>>ran test, got ~33k IOPS again
>>

 I have tested with another server, I around 60k write and going down
to 600~1000k iops max when a snapshot is token. 


I need to increase slowing after some time, maybe 2~3min to get 20k
iops
and 7~10min to get 50kiops again

I'm testing with a 100gb disk

Maybe is is related to metadatas preallocation ? (I'm not sure, but
when you have a new snapshot, the preallocated metadatas are not using
anymore ?)


Anyway, I just send patch for external qcow2 snapshot :)
If somebody have time to review it ^_^


Thanks again for your response !




> is it expected ?
> 
> (I don't have use qcow2 snapshot since 10year, so I really don't 
>  remember performance).
> 

Best Regards,
Fiona

> 
> Using external snapshot file, I'm around 10000iops.
> 
> test inside the vm:
> 
> fio --filename=/dev/sdX --ioengine=libaio --direct 1 --bs=4k  --
> rw=randwrite --numjobs=64
> 
> 



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel