all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "DERUMIER, Alexandre via pve-devel" <pve-devel@lists.proxmox.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 13:03:45 +0000	[thread overview]
Message-ID: <mailman.432.1729429465.332.pve-devel@lists.proxmox.com> (raw)
In-Reply-To: <20240930113153.2896648-1-alexandre.derumier@groupe-cyllene.com>

[-- Attachment #1: Type: message/rfc822, Size: 19550 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 13:03:45 +0000
Message-ID: <6e37f6d8cc4d8bd65c72c58edc429407d64fafab.camel@groupe-cyllene.com>

Hi,

Any comment about this patch series ?

I really think that external snapshot could be a great feature (as I
still see report on the forum about freeze on snasphot deletion),
and support for lvm and shared san is really a feature than enterprise
users are waiting.  (To be honest, I have a lot of customers stuck on
vmware because of this)


About my previous patch serie, with lvm dynamic extent, I think I'll
give up, it seem to be too complex, with too many corner case.
(So keeping shared lvm + external snapshot without thin provisioning)


In Parallel, I have done more tests with gfs2/ocfs2, and I finally
found a 
way to have good performance on block allocation for thin non
preallocated qcow2 file.

Currently, I'm around 200 iops on gfs2 with 4k randwrite (instead
20000iops...)

(fio --rw=randwrite ---direct=1 -bs=4k --ioengine=libaio --iodepth=64 -
-filename=/dev/sdX)

qemu have "preallocate" filter feature
https://patchwork.kernel.org/project/qemu-devel/cover/20200814130348.20625-1-vsementsov@virtuozzo.com/

-drive driver=qcow2,file.driver=preallocate,file.prealloc-
size=1073741824,file.file.driver=file,file.file.filename=/mnt/pve/gfs2s
an/images/100/vm-100-disk-0.qcow2,id=drive-scsi2,if=none


which allow to prellocate on the fly more blocks than requested.
(for example, you need to write a 4k block on a unallocated block, I'll
reserve 128MB for example).
This reduce a lot the number of locks and round-trip network for fs
like ocfs2/gfs2  when you have a lot of write.

With qcow2 format allocating random blocks consecutively, that's
working very well.

I have done a small test, the fio result is around to 20~30k write.
I'll send a patch soon after doing more test.


Regards,

Alexandre

-------- Message initial --------
De: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
À: pve-devel@lists.proxmox.com
Objet: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2
snapshot support
Date: 30/09/2024 13:31:50

This patch series implement qcow2 external snapshot support for files
&& lvm volumes

The current internal qcow2 snapshots have a lot of performance
problems.

I have tested through nfs and also local filesystem


I see that Fiona don't have same result than me, but I got something
like 200~300iops
vs 20000 iops with 4k randwrite when a snapshot exist.

The result is even worst on a shared filesystem like ocfs2 or gfs2.
(around 80 iops)
I think (I'm not 100% sure) this is mostly because metadatas are not
preallocated
anymore with qcow2 internal snap.

With external snapshot, I almost don't have any performance impact when
a snapshot exist.

Also other bugs are freeze/lock reported by users since years on
snapshots delete on nfs
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=64b60d6fd396d266b432ee693cc8f61d2632a8524491fef07cef3c3
f51c98871&u=https%3A%2F%2Fforum.proxmox.com%2Fthreads%2Fsnapshot-
removal-jams-the-vm.111648%2F
(The disk access seem to be frozen during all the delete duration)

External qcow2 snapshots also allow snapshot of raw devices ,so 0
performance impact without snapshots.

This also open doors for remote snapshot export-import for storage
replication.

This V2 introduce support for qcow2 external snapshot for lvm, extra
lvm
volume is created for each snapsphot and formated with qcow2.
This is a lot more performant than lvm (non-thin/nomedata) snapshot,
and allow to use
it for shared lvm.  (I have another patch series for thick lvm dynamic
extend, but if we could have at minimum
snapshot working, it could great :)

I have tested: snasphot, snap rollback, snap delete, clone, move disk,
rename disk, create_base. (online && offline)

lxc is not yet supported, but I think we could look to implement the
recent dm-qcow2 kernel block driver
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=1865f514f95ac1d8e0088b598376751d4d98fa25de6a8b2868a74f9
2ac661cfa&u=https%3A%2F%2Flore.kernel.org%2Flkml%2F164846619932.251310.
3668540533992131988.stgit%40pro%2FT%2F


storage.cfg example:

dir: local2
        path /var/liv/vz
        content snippets,vztmpl,backup,images,iso,rootdir
        snapext 1

lvmqcow2:test
        vgname test
        snapext 1
        content images


changelog v2:
 implement lvm with external qcow2 snapshots

pve-storage:

Alexandre Derumier (2):
  add external snasphot support
  add lvmqcow2 plugin: (lvm with external qcow2 snapshot)

 src/PVE/Storage.pm                |   2 +
 src/PVE/Storage/DirPlugin.pm      |   1 +
 src/PVE/Storage/LvmQcow2Plugin.pm | 460 ++++++++++++++++++++++++++++++
 src/PVE/Storage/Makefile          |   3 +-
 src/PVE/Storage/Plugin.pm         | 225 +++++++++++++--
 5 files changed, 665 insertions(+), 26 deletions(-)
 create mode 100644 src/PVE/Storage/LvmQcow2Plugin.pm


qemu-server:


Alexandre Derumier (1):
  implement external snapshot

 PVE/QemuServer.pm | 108 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 95 insertions(+), 13 deletions(-)



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

  parent reply	other threads:[~2024-10-20 13:03 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20240930113153.2896648-1-alexandre.derumier@groupe-cyllene.com>
2024-09-30 11:31 ` [pve-devel] [PATCH v2 pve-storage 1/2] add external snasphot support Alexandre Derumier via pve-devel
2024-10-23 10:12   ` Fabian Grünbichler
2024-10-23 12:59     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <f066c13a25b30e3107a9dec8091b456ce2852293.camel@groupe-cyllene.com>
2024-10-24  6:42       ` Fabian Grünbichler
2024-10-24  7:59         ` Giotta Simon RUAGH via pve-devel
2024-10-24  9:48           ` Fabian Grünbichler
2024-10-25 20:04             ` DERUMIER, Alexandre via pve-devel
     [not found]             ` <7974c74b2d3a85086e8eda76e52d7a2c58d1dcb9.camel@groupe-cyllene.com>
2024-10-28 11:12               ` Fabian Grünbichler
2024-10-25  5:52         ` DERUMIER, Alexandre via pve-devel
2024-10-24  7:50     ` Fabian Grünbichler
2024-09-30 11:31 ` [pve-devel] [PATCH v2 qemu-server 1/1] implement external snapshot Alexandre Derumier via pve-devel
2024-10-23 10:14   ` Fabian Grünbichler
2024-10-23 14:31     ` DERUMIER, Alexandre via pve-devel
2024-10-23 18:09     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <aeb9b8ea34826483eabe7fec5e2c12b1e22e132f.camel@groupe-cyllene.com>
2024-10-24  7:43       ` Fabian Grünbichler
2024-09-30 11:31 ` [pve-devel] [PATCH v2 pve-storage 2/2] add lvmqcow2 plugin: (lvm with external qcow2 snapshot) Alexandre Derumier via pve-devel
2024-10-23 10:13   ` Fabian Grünbichler
2024-10-23 13:45     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <e976104d8ed7c365d8a482fa320a0691456e69c1.camel@groupe-cyllene.com>
2024-10-24  7:42       ` Fabian Grünbichler
2024-10-24 11:01         ` DERUMIER, Alexandre via pve-devel
2024-10-20 13:03 ` DERUMIER, Alexandre via pve-devel [this message]
2024-10-20 17:34   ` [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support Roland privat via pve-devel
2024-10-20 19:08     ` Esi Y via pve-devel
     [not found]     ` <CABtLnHqZVhDKnog6jaUBP4HcSwfanyEzWeLdUXnzJs2esJQQkA@mail.gmail.com>
2024-10-22  6:39       ` Thomas Lamprecht
2024-10-22  9:51         ` Esi Y via pve-devel
2024-10-22 14:54           ` DERUMIER, Alexandre via pve-devel
     [not found]           ` <2f07646b51c85ffe01089c2481dbb9680d75cfcb.camel@groupe-cyllene.com>
2024-10-24  3:37             ` Esi Y via pve-devel
2024-09-30 11:31 Alexandre Derumier via pve-devel

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=mailman.432.1729429465.332.pve-devel@lists.proxmox.com \
    --to=pve-devel@lists.proxmox.com \
    --cc=alexandre.derumier@groupe-cyllene.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal