[pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Alexandre Derumier via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 5384 bytes --]

From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Mon, 30 Sep 2024 13:31:50 +0200
Message-ID: <20240930113153.2896648-1-alexandre.derumier@groupe-cyllene.com>

This patch series implement qcow2 external snapshot support for files && lvm volumes

The current internal qcow2 snapshots have a lot of performance problems.

I have tested through nfs and also local filesystem
https://lore.proxmox.com/pve-devel/6a44716a-88bc-4523-b210-d67031917d8f@proxmox.com/t/

I see that Fiona don't have same result than me, but I got something like 200~300iops
vs 20000 iops with 4k randwrite when a snapshot exist.

The result is even worst on a shared filesystem like ocfs2 or gfs2. (around 80 iops)
I think (I'm not 100% sure) this is mostly because metadatas are not preallocated
anymore with qcow2 internal snap.

With external snapshot, I almost don't have any performance impact when a snapshot exist.

Also other bugs are freeze/lock reported by users since years on snapshots delete on nfs
https://forum.proxmox.com/threads/snapshot-removal-jams-the-vm.111648/
(The disk access seem to be frozen during all the delete duration)

External qcow2 snapshots also allow snapshot of raw devices ,so 0 performance impact without snapshots.

This also open doors for remote snapshot export-import for storage replication.

This V2 introduce support for qcow2 external snapshot for lvm, extra lvm
volume is created for each snapsphot and formated with qcow2.
This is a lot more performant than lvm (non-thin/nomedata) snapshot, and allow to use
it for shared lvm.  (I have another patch series for thick lvm dynamic extend, but if we could have at minimum
snapshot working, it could great :)

I have tested: snasphot, snap rollback, snap delete, clone, move disk, rename disk, create_base. (online && offline)

lxc is not yet supported, but I think we could look to implement the recent dm-qcow2 kernel block driver
https://lore.kernel.org/lkml/164846619932.251310.3668540533992131988.stgit@pro/T/


storage.cfg example:

dir: local2
        path /var/liv/vz
        content snippets,vztmpl,backup,images,iso,rootdir
        snapext 1

lvmqcow2:test
        vgname test
        snapext 1
        content images


changelog v2:
 implement lvm with external qcow2 snapshots

pve-storage:

Alexandre Derumier (2):
  add external snasphot support
  add lvmqcow2 plugin: (lvm with external qcow2 snapshot)

 src/PVE/Storage.pm                |   2 +
 src/PVE/Storage/DirPlugin.pm      |   1 +
 src/PVE/Storage/LvmQcow2Plugin.pm | 460 ++++++++++++++++++++++++++++++
 src/PVE/Storage/Makefile          |   3 +-
 src/PVE/Storage/Plugin.pm         | 225 +++++++++++++--
 5 files changed, 665 insertions(+), 26 deletions(-)
 create mode 100644 src/PVE/Storage/LvmQcow2Plugin.pm


qemu-server:


Alexandre Derumier (1):
  implement external snapshot

 PVE/QemuServer.pm | 108 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 95 insertions(+), 13 deletions(-)

-- 
2.39.2



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 19550 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 13:03:45 +0000
Message-ID: <6e37f6d8cc4d8bd65c72c58edc429407d64fafab.camel@groupe-cyllene.com>

Hi,

Any comment about this patch series ?

I really think that external snapshot could be a great feature (as I
still see report on the forum about freeze on snasphot deletion),
and support for lvm and shared san is really a feature than enterprise
users are waiting.  (To be honest, I have a lot of customers stuck on
vmware because of this)


About my previous patch serie, with lvm dynamic extent, I think I'll
give up, it seem to be too complex, with too many corner case.
(So keeping shared lvm + external snapshot without thin provisioning)


In Parallel, I have done more tests with gfs2/ocfs2, and I finally
found a 
way to have good performance on block allocation for thin non
preallocated qcow2 file.

Currently, I'm around 200 iops on gfs2 with 4k randwrite (instead
20000iops...)

(fio --rw=randwrite ---direct=1 -bs=4k --ioengine=libaio --iodepth=64 -
-filename=/dev/sdX)

qemu have "preallocate" filter feature
https://patchwork.kernel.org/project/qemu-devel/cover/20200814130348.20625-1-vsementsov@virtuozzo.com/

-drive driver=qcow2,file.driver=preallocate,file.prealloc-
size=1073741824,file.file.driver=file,file.file.filename=/mnt/pve/gfs2s
an/images/100/vm-100-disk-0.qcow2,id=drive-scsi2,if=none


which allow to prellocate on the fly more blocks than requested.
(for example, you need to write a 4k block on a unallocated block, I'll
reserve 128MB for example).
This reduce a lot the number of locks and round-trip network for fs
like ocfs2/gfs2  when you have a lot of write.

With qcow2 format allocating random blocks consecutively, that's
working very well.

I have done a small test, the fio result is around to 20~30k write.
I'll send a patch soon after doing more test.


Regards,

Alexandre

-------- Message initial --------
De: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
À: pve-devel@lists.proxmox.com
Objet: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2
snapshot support
Date: 30/09/2024 13:31:50

This patch series implement qcow2 external snapshot support for files
&& lvm volumes

The current internal qcow2 snapshots have a lot of performance
problems.

I have tested through nfs and also local filesystem


I see that Fiona don't have same result than me, but I got something
like 200~300iops
vs 20000 iops with 4k randwrite when a snapshot exist.

The result is even worst on a shared filesystem like ocfs2 or gfs2.
(around 80 iops)
I think (I'm not 100% sure) this is mostly because metadatas are not
preallocated
anymore with qcow2 internal snap.

With external snapshot, I almost don't have any performance impact when
a snapshot exist.

Also other bugs are freeze/lock reported by users since years on
snapshots delete on nfs
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=64b60d6fd396d266b432ee693cc8f61d2632a8524491fef07cef3c3
f51c98871&u=https%3A%2F%2Fforum.proxmox.com%2Fthreads%2Fsnapshot-
removal-jams-the-vm.111648%2F
(The disk access seem to be frozen during all the delete duration)

External qcow2 snapshots also allow snapshot of raw devices ,so 0
performance impact without snapshots.

This also open doors for remote snapshot export-import for storage
replication.

This V2 introduce support for qcow2 external snapshot for lvm, extra
lvm
volume is created for each snapsphot and formated with qcow2.
This is a lot more performant than lvm (non-thin/nomedata) snapshot,
and allow to use
it for shared lvm.  (I have another patch series for thick lvm dynamic
extend, but if we could have at minimum
snapshot working, it could great :)

I have tested: snasphot, snap rollback, snap delete, clone, move disk,
rename disk, create_base. (online && offline)

lxc is not yet supported, but I think we could look to implement the
recent dm-qcow2 kernel block driver
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=1865f514f95ac1d8e0088b598376751d4d98fa25de6a8b2868a74f9
2ac661cfa&u=https%3A%2F%2Flore.kernel.org%2Flkml%2F164846619932.251310.
3668540533992131988.stgit%40pro%2FT%2F


storage.cfg example:

dir: local2
        path /var/liv/vz
        content snippets,vztmpl,backup,images,iso,rootdir
        snapext 1

lvmqcow2:test
        vgname test
        snapext 1
        content images


changelog v2:
 implement lvm with external qcow2 snapshots

pve-storage:

Alexandre Derumier (2):
  add external snasphot support
  add lvmqcow2 plugin: (lvm with external qcow2 snapshot)

 src/PVE/Storage.pm                |   2 +
 src/PVE/Storage/DirPlugin.pm      |   1 +
 src/PVE/Storage/LvmQcow2Plugin.pm | 460 ++++++++++++++++++++++++++++++
 src/PVE/Storage/Makefile          |   3 +-
 src/PVE/Storage/Plugin.pm         | 225 +++++++++++++--
 5 files changed, 665 insertions(+), 26 deletions(-)
 create mode 100644 src/PVE/Storage/LvmQcow2Plugin.pm


qemu-server:


Alexandre Derumier (1):
  implement external snapshot

 PVE/QemuServer.pm | 108 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 95 insertions(+), 13 deletions(-)



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Roland privat via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 6300 bytes --]

From: Roland privat <devzero@web.de>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 19:34:05 +0200
Message-ID: <4A4D6F36-75C7-4346-BA43-EED2DD7B3F70@web.de>

i never understood,  why qemu has inline snapshot by default.  from an admin perspective it simply sucks.  for example  you won‘t know when something goes wrong on removal,  how much orphaned data is being left.  furthermore,  the freezes on removal are a pita for us and we often avoid live snapshot removal because of this.  this is one of the reasons why i think ,  proxmox isn‘t on the same enterprise level like vmware.

so,  i strongly vote for inclusion of this feature and will happily test it, when available

regards
roland

> Am 20.10.2024 um 15:04 schrieb DERUMIER, Alexandre via pve-devel <pve-devel@lists.proxmox.com>:
> 
> 
> <mime-attachment>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Esi Y via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 7131 bytes --]

From: Esi Y <esiy0676+proxmox@gmail.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 21:08:54 +0200
Message-ID: <CABtLnHqZVhDKnog6jaUBP4HcSwfanyEzWeLdUXnzJs2esJQQkA@mail.gmail.com>

QEMU was not designed to use internal snapshots "by default":
https://wiki.qemu.org/Features/Snapshots

Discussions on this are discouraged, to say the least:
https://forum.proxmox.com/threads/blockdev-snapshot-sync-to-snapshot-a-raw-file.38188/#post-712825

On Sun, Oct 20, 2024 at 7:39 PM Roland privat via pve-devel
<pve-devel@lists.proxmox.com> wrote:
>
>
>
>
> ---------- Forwarded message ----------
> From: Roland privat <devzero@web.de>
> To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
> Cc:
> Bcc:
> Date: Sun, 20 Oct 2024 19:34:05 +0200
> Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
> i never understood,  why qemu has inline snapshot by default.  from an admin perspective it simply sucks.  for example  you won‘t know when something goes wrong on removal,  how much orphaned data is being left.  furthermore,  the freezes on removal are a pita for us and we often avoid live snapshot removal because of this.  this is one of the reasons why i think ,  proxmox isn‘t on the same enterprise level like vmware.
>
> so,  i strongly vote for inclusion of this feature and will happily test it, when available
>
> regards
> roland
>
> > Am 20.10.2024 um 15:04 schrieb DERUMIER, Alexandre via pve-devel <pve-devel@lists.proxmox.com>:
> >
> > 
> > <mime-attachment>
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Roland privat via pve-devel <pve-devel@lists.proxmox.com>
> To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
> Cc: Roland privat <devzero@web.de>
> Bcc:
> Date: Sun, 20 Oct 2024 19:34:05 +0200
> Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Thomas Lamprecht]

Am 20/10/2024 um 21:08 schrieb Esi Y:
> QEMU was not designed to use internal snapshots "by default":
> https://wiki.qemu.org/Features/Snapshots

Just because you find an article that mentions qcow2 and snapshots it
doesn't have to be related...
The thing you linked has not only nothing to do with the snapshots of the
qcow2 format, which are just one of the targets, it also nowhere states
what you crudely try to imply.

Let's stop this dunning-kruger fueled spew of technical nonsense on the
devel lists, ideally all other channels too. It's not welcomed here and
wasting developers time that then, among other things, delays actual
review of all patches, including this series.

> Discussions on this are discouraged, to say the least:
> https://forum.proxmox.com/threads/blockdev-snapshot-sync-to-snapshot-a-raw-file.38188/#post-712825

The reason was clearly stated here [0], so why lie to make this seem like
some big conspiracy?

[0]: https://forum.proxmox.com/threads/blockdev-snapshot-sync-to-snapshot-a-raw-file.38188/#post-712902

You must stop your toxic bad-faith behavior immediately, it was pointed out
many times by Proxmox Staff and even other community users. If that won't
work, we have to exclude you from all our channels!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Esi Y via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 6045 bytes --]

From: Esi Y <esiy0676+proxmox@gmail.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Tue, 22 Oct 2024 11:51:31 +0200
Message-ID: <CABtLnHrgbiZcT0oMwpuhHhMiJh_nqoM7ZLNKMWYMBXJjkXa5VA@mail.gmail.com>

On Tue, Oct 22, 2024 at 8:39 AM Thomas Lamprecht
<t.lamprecht@proxmox.com> wrote:
> The thing you linked has not only nothing to do with the snapshots of the
> qcow2 format, which are just one of the targets, it also nowhere states
> what you crudely try to imply.

It's 2016 wiki:

"Internal snapshots to images which support internal snapshots (QCOW2
& QED) are not expected to be supported initially."

"By making the snapshot-file argument of the monitor and QMP command
optional, that could be used as a request to make the snapshot
internally instead of to an external file. However, without live block
migration of an internal snapshot, there is no way to make a backup of
an internal snapshot while still leaving the VM running, so this
feature is not planned at the present. For now, the snapshot-file
argument is required, and only external snapshots are implemented."

> wasting developers time that then, among other things, delays actual

I hoped this would bump it up for Alexandre to get a response.


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14854 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: "esiy0676+proxmox@gmail.com" <esiy0676+proxmox@gmail.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Tue, 22 Oct 2024 14:54:24 +0000
Message-ID: <2f07646b51c85ffe01089c2481dbb9680d75cfcb.camel@groupe-cyllene.com>

-------- Message initial --------
De: Esi Y via pve-devel <pve-devel@lists.proxmox.com>
Répondre à: Proxmox VE development discussion <pve-
devel@lists.proxmox.com>
À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Cc: Esi Y <esiy0676+proxmox@gmail.com>
Objet: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add
external qcow2 snapshot support
Date: 22/10/2024 11:51:31

> wasting developers time that then, among other things, delays actual

>>I hoped this would bump it up for Alexandre to get a response.
As far I remember, when we have implement snapshot for qcow2 (I think
in 2010~2011, I'm becoming old ^_^ ) , only internal snapshot was
possible, 
because they were no block-commit job. (to merge data in parent on
snapshot deletion).

Only block-stream job was available at this time (merge snapshot to
child snapshot)

I think that redhat have mostly worked on external snapshots these last
10years (mostly because they used them for backup, but also replication
where it's not possible with internal snasphot).

And the missing block job to merge data for internal snapshot is also
why the io need to be frozen during the merge.


So, that's why I never haved use qcow2 in production (mostly ceph,
or though custom netapp api for customer using nfs with raw files).

That mean that we don't have a clean snapshot solution currently for
shared san/nas without api.

I'm trying to fix/improve both nas (nfs) && san (iscsi,lvm) snapshots
implementation.

Mainly because I have a lot of onprem customers coming from vmware with
small san (iscsi/fiberchannel) needing snapshot feature.












[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support

[Esi Y via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 7259 bytes --]

From: Esi Y <esiy0676+proxmox@gmail.com>
Cc: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Thu, 24 Oct 2024 05:37:19 +0200
Message-ID: <CABtLnHrx=-DOvTYVbFcoJtYvQ=9PRA4g+vhV3Eix2LaqxRb4Kg@mail.gmail.com>

Thank you, Alexandre, for the background / historical information.

On Tue, Oct 22, 2024 at 4:54 PM DERUMIER, Alexandre
<alexandre.derumier@groupe-cyllene.com> wrote:
>
> -------- Message initial --------
> De: Esi Y via pve-devel <pve-devel@lists.proxmox.com>
> Répondre à: Proxmox VE development discussion <pve-
> devel@lists.proxmox.com>
> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
> Cc: Esi Y <esiy0676+proxmox@gmail.com>
> Objet: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add
> external qcow2 snapshot support
> Date: 22/10/2024 11:51:31
>
> > wasting developers time that then, among other things, delays actual
>
> >>I hoped this would bump it up for Alexandre to get a response.
> As far I remember, when we have implement snapshot for qcow2 (I think
> in 2010~2011, I'm becoming old ^_^ ) , only internal snapshot was
> possible,
> because they were no block-commit job. (to merge data in parent on
> snapshot deletion).

I just found it relevant to point out that historical Wiki on Live snapshots
were approaching this with the external-first in mind - when I look at
revisions [1], already in 2011:

"Internal snapshots to images which support internal snapshots (QCOW2 & QED)
are not expected to be supported initially."

> That mean that we don't have a clean snapshot solution currently for
> shared san/nas without api.

And that it is overdue at least since 2017 (the original forum attempt
at discussion,
not mine).

Anyhow, good that is now being reviewed. There's enough interested in the said
forum for this already.

NB I don't believe I should have been asked to take that supporting
input elsewhere.

[1] https://wiki.qemu.org/index.php?title=Features/Snapshots&diff=5964&oldid=1405


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel