public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "DERUMIER, Alexandre via pve-devel" <pve-devel@lists.proxmox.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Cc: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
Subject: Re: [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 13:03:45 +0000	[thread overview]
Message-ID: <mailman.432.1729429465.332.pve-devel@lists.proxmox.com> (raw)
In-Reply-To: <20240930113153.2896648-1-alexandre.derumier@groupe-cyllene.com>

[-- Attachment #1: Type: message/rfc822, Size: 19550 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support
Date: Sun, 20 Oct 2024 13:03:45 +0000
Message-ID: <6e37f6d8cc4d8bd65c72c58edc429407d64fafab.camel@groupe-cyllene.com>

Hi,

Any comment about this patch series ?

I really think that external snapshot could be a great feature (as I
still see report on the forum about freeze on snasphot deletion),
and support for lvm and shared san is really a feature than enterprise
users are waiting.  (To be honest, I have a lot of customers stuck on
vmware because of this)


About my previous patch serie, with lvm dynamic extent, I think I'll
give up, it seem to be too complex, with too many corner case.
(So keeping shared lvm + external snapshot without thin provisioning)


In Parallel, I have done more tests with gfs2/ocfs2, and I finally
found a 
way to have good performance on block allocation for thin non
preallocated qcow2 file.

Currently, I'm around 200 iops on gfs2 with 4k randwrite (instead
20000iops...)

(fio --rw=randwrite ---direct=1 -bs=4k --ioengine=libaio --iodepth=64 -
-filename=/dev/sdX)

qemu have "preallocate" filter feature
https://patchwork.kernel.org/project/qemu-devel/cover/20200814130348.20625-1-vsementsov@virtuozzo.com/

-drive driver=qcow2,file.driver=preallocate,file.prealloc-
size=1073741824,file.file.driver=file,file.file.filename=/mnt/pve/gfs2s
an/images/100/vm-100-disk-0.qcow2,id=drive-scsi2,if=none


which allow to prellocate on the fly more blocks than requested.
(for example, you need to write a 4k block on a unallocated block, I'll
reserve 128MB for example).
This reduce a lot the number of locks and round-trip network for fs
like ocfs2/gfs2  when you have a lot of write.

With qcow2 format allocating random blocks consecutively, that's
working very well.

I have done a small test, the fio result is around to 20~30k write.
I'll send a patch soon after doing more test.


Regards,

Alexandre

-------- Message initial --------
De: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
À: pve-devel@lists.proxmox.com
Objet: [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2
snapshot support
Date: 30/09/2024 13:31:50

This patch series implement qcow2 external snapshot support for files
&& lvm volumes

The current internal qcow2 snapshots have a lot of performance
problems.

I have tested through nfs and also local filesystem


I see that Fiona don't have same result than me, but I got something
like 200~300iops
vs 20000 iops with 4k randwrite when a snapshot exist.

The result is even worst on a shared filesystem like ocfs2 or gfs2.
(around 80 iops)
I think (I'm not 100% sure) this is mostly because metadatas are not
preallocated
anymore with qcow2 internal snap.

With external snapshot, I almost don't have any performance impact when
a snapshot exist.

Also other bugs are freeze/lock reported by users since years on
snapshots delete on nfs
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=64b60d6fd396d266b432ee693cc8f61d2632a8524491fef07cef3c3
f51c98871&u=https%3A%2F%2Fforum.proxmox.com%2Fthreads%2Fsnapshot-
removal-jams-the-vm.111648%2F
(The disk access seem to be frozen during all the delete duration)

External qcow2 snapshots also allow snapshot of raw devices ,so 0
performance impact without snapshots.

This also open doors for remote snapshot export-import for storage
replication.

This V2 introduce support for qcow2 external snapshot for lvm, extra
lvm
volume is created for each snapsphot and formated with qcow2.
This is a lot more performant than lvm (non-thin/nomedata) snapshot,
and allow to use
it for shared lvm.  (I have another patch series for thick lvm dynamic
extend, but if we could have at minimum
snapshot working, it could great :)

I have tested: snasphot, snap rollback, snap delete, clone, move disk,
rename disk, create_base. (online && offline)

lxc is not yet supported, but I think we could look to implement the
recent dm-qcow2 kernel block driver
https://antiphishing.vadesecure.com/v4?f=S1Zkd042VWdrZG5qQUxxWk5ps4t67k
NuHsBZzdzhpquLKuXqTZLIq2K1DfKr9N61yBafm7AuAITd6bHtRU4zEQ&i=MlZSTzBhZFZ6
Nzl4c3EyN5T6buHjA4kKs6Oz9IPjCIg&k=F1is&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYx
z7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&s=1865f514f95ac1d8e0088b598376751d4d98fa25de6a8b2868a74f9
2ac661cfa&u=https%3A%2F%2Flore.kernel.org%2Flkml%2F164846619932.251310.
3668540533992131988.stgit%40pro%2FT%2F


storage.cfg example:

dir: local2
        path /var/liv/vz
        content snippets,vztmpl,backup,images,iso,rootdir
        snapext 1

lvmqcow2:test
        vgname test
        snapext 1
        content images


changelog v2:
 implement lvm with external qcow2 snapshots

pve-storage:

Alexandre Derumier (2):
  add external snasphot support
  add lvmqcow2 plugin: (lvm with external qcow2 snapshot)

 src/PVE/Storage.pm                |   2 +
 src/PVE/Storage/DirPlugin.pm      |   1 +
 src/PVE/Storage/LvmQcow2Plugin.pm | 460 ++++++++++++++++++++++++++++++
 src/PVE/Storage/Makefile          |   3 +-
 src/PVE/Storage/Plugin.pm         | 225 +++++++++++++--
 5 files changed, 665 insertions(+), 26 deletions(-)
 create mode 100644 src/PVE/Storage/LvmQcow2Plugin.pm


qemu-server:


Alexandre Derumier (1):
  implement external snapshot

 PVE/QemuServer.pm | 108 ++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 95 insertions(+), 13 deletions(-)



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

  parent reply	other threads:[~2024-10-20 13:03 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20240930113153.2896648-1-alexandre.derumier@groupe-cyllene.com>
2024-09-30 11:31 ` [pve-devel] [PATCH v2 pve-storage 1/2] add external snasphot support Alexandre Derumier via pve-devel
2024-10-23 10:12   ` Fabian Grünbichler
2024-10-23 12:59     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <f066c13a25b30e3107a9dec8091b456ce2852293.camel@groupe-cyllene.com>
2024-10-24  6:42       ` Fabian Grünbichler
2024-10-24  7:59         ` Giotta Simon RUAGH via pve-devel
2024-10-24  9:48           ` Fabian Grünbichler
2024-10-25 20:04             ` DERUMIER, Alexandre via pve-devel
     [not found]             ` <7974c74b2d3a85086e8eda76e52d7a2c58d1dcb9.camel@groupe-cyllene.com>
2024-10-28 11:12               ` Fabian Grünbichler
2024-10-25  5:52         ` DERUMIER, Alexandre via pve-devel
2024-10-24  7:50     ` Fabian Grünbichler
2024-09-30 11:31 ` [pve-devel] [PATCH v2 qemu-server 1/1] implement external snapshot Alexandre Derumier via pve-devel
2024-10-23 10:14   ` Fabian Grünbichler
2024-10-23 14:31     ` DERUMIER, Alexandre via pve-devel
2024-10-23 18:09     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <aeb9b8ea34826483eabe7fec5e2c12b1e22e132f.camel@groupe-cyllene.com>
2024-10-24  7:43       ` Fabian Grünbichler
2024-09-30 11:31 ` [pve-devel] [PATCH v2 pve-storage 2/2] add lvmqcow2 plugin: (lvm with external qcow2 snapshot) Alexandre Derumier via pve-devel
2024-10-23 10:13   ` Fabian Grünbichler
2024-10-23 13:45     ` DERUMIER, Alexandre via pve-devel
     [not found]     ` <e976104d8ed7c365d8a482fa320a0691456e69c1.camel@groupe-cyllene.com>
2024-10-24  7:42       ` Fabian Grünbichler
2024-10-24 11:01         ` DERUMIER, Alexandre via pve-devel
2024-10-20 13:03 ` DERUMIER, Alexandre via pve-devel [this message]
2024-10-20 17:34   ` [pve-devel] [PATCH SERIES v2 pve-storage/qemu-server] add external qcow2 snapshot support Roland privat via pve-devel
2024-10-20 19:08     ` Esi Y via pve-devel
     [not found]     ` <CABtLnHqZVhDKnog6jaUBP4HcSwfanyEzWeLdUXnzJs2esJQQkA@mail.gmail.com>
2024-10-22  6:39       ` Thomas Lamprecht
2024-10-22  9:51         ` Esi Y via pve-devel
2024-10-22 14:54           ` DERUMIER, Alexandre via pve-devel
     [not found]           ` <2f07646b51c85ffe01089c2481dbb9680d75cfcb.camel@groupe-cyllene.com>
2024-10-24  3:37             ` Esi Y via pve-devel
2024-09-30 11:31 Alexandre Derumier via pve-devel

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=mailman.432.1729429465.332.pve-devel@lists.proxmox.com \
    --to=pve-devel@lists.proxmox.com \
    --cc=alexandre.derumier@groupe-cyllene.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal