[pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Alexandre Derumier via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 13270 bytes --]

From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Tue,  3 Jun 2025 09:55:35 +0200
Message-ID: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>

This patch series implement qcow2 external snapshot support for files && lvm volumes

The current internal qcow2 snapshots have bad write performance because no metadatas can be preallocated.

This is particulary visible on a shared filesystem like ocfs2 or gfs2.

Also other bugs are freeze/lock reported by users since years on snapshots delete on nfs
(The disk access seem to be frozen during all the delete duration)

This also open doors for remote snapshot export-import for storage replication.

The pve-storage patches applied on top of Fiona qemu_blockdev_options patch series
https://lists.proxmox.com/pipermail/pve-devel/2025-May/071204.html

This new patch series enable blockdev for qemu machine > 10.0 to avoid breaking efidisk and maybe
potential migrations bug

Changelog v6:
     pve-storage:
      - use fiona qemu_blockdev_options patch
      - add helpers for qemu-img call
      - use qemu-img measure for lvm size (including l2_extended=on)
      - add a new helper volume_support_qemu_snapshot
      - fix Fabian requested changes

     qemu-server:
      - enable blockdev for machine version > 10.0
      - move blockdev code in QemuServer/Blockdev.pm
      - use Fiona qemu_blockdev_options patch for storage plugins
      - add raw size for efidisk
      - fix drivedel with backing chain
      - add tests for qemu-img convert, backing chain,....
      - fix Fabian requested changes
     pve-qemu:
      - add a fake 10.0 qemu bump for testing

MAIN TODO:
    - convert/implement all the backup code, dirty bitmap mirro, ...
    - add snapshots tests


pve-storage:

Alexandre Derumier (9):
  common: add qemu_img_create an preallocation_cmd_option
  common: add qemu_img_create_qcow2_backed
  common: add qemu_img_info helper
  common: add qemu_img_measure
  rename_volume: add source && target snap
  storage: volume_snapshot: add $running param + api bump
  qcow2: add external snapshot support
  lvmplugin: add qcow2 snapshot
  storage: add volume_support_qemu_snapshot

 src/PVE/Storage.pm                   |  28 +-
 src/PVE/Storage/BTRFSPlugin.pm       |   2 +-
 src/PVE/Storage/Common.pm            | 175 ++++++++++
 src/PVE/Storage/DirPlugin.pm         |  11 +
 src/PVE/Storage/ESXiPlugin.pm        |   2 +-
 src/PVE/Storage/GlusterfsPlugin.pm   |   2 +-
 src/PVE/Storage/ISCSIDirectPlugin.pm |   2 +-
 src/PVE/Storage/LVMPlugin.pm         | 467 +++++++++++++++++++++++----
 src/PVE/Storage/LvmThinPlugin.pm     |   2 +-
 src/PVE/Storage/PBSPlugin.pm         |   2 +-
 src/PVE/Storage/Plugin.pm            | 339 ++++++++++++++-----
 src/PVE/Storage/RBDPlugin.pm         |   8 +-
 src/PVE/Storage/ZFSPoolPlugin.pm     |   2 +-
 13 files changed, 875 insertions(+), 167 deletions(-)

qemu-server:

Alexandre Derumier (13):
  blockdev: cmdline: add blockdev syntax support
  blockdev: add support for ovmf && efidisk
  blockdev: vm_devices_list : fix block-query
  blockdev: add support to qemu_driveadd && qemu_drivedel
  blockdev: add support to qemu_block_set_io_throttle
  blockdev: add support for cdrom media eject/insert
  blockdev: add support for block_resize
  blockdev: add support for nbd_export: block-export-add
  blockdev: add blockdev_mirror
  blockdev: change aio on target if io_uring is not default.
  qemu_img convert : add external snapshot support
  blockdev: add backing_chain support
  qcow2: add external snapshot support

 PVE/QemuConfig.pm                             |   4 +-
 PVE/QemuMigrate.pm                            |   2 +-
 PVE/QemuServer.pm                             | 761 +++++++++++-------
 PVE/QemuServer/Blockdev.pm                    | 494 ++++++++++++
 PVE/QemuServer/Drive.pm                       | 201 +++++
 PVE/QemuServer/Makefile                       |   1 +
 test/MigrationTest/QemuMigrateMock.pm         |  10 +-
 test/cfg2cmd/bootorder-blockdev.conf          |  15 +
 test/cfg2cmd/bootorder-blockdev.conf.cmd      |  43 +
 test/cfg2cmd/bootorder-empty-blockdev.conf    |  15 +
 .../cfg2cmd/bootorder-empty-blockdev.conf.cmd |  43 +
 test/cfg2cmd/bootorder-empty.conf             |   1 +
 test/cfg2cmd/bootorder-empty.conf.cmd         |   2 +-
 test/cfg2cmd/bootorder-legacy.conf            |   1 +
 test/cfg2cmd/bootorder-legacy.conf.cmd        |   2 +-
 test/cfg2cmd/bootorder.conf                   |   1 +
 test/cfg2cmd/bootorder.conf.cmd               |   2 +-
 .../cputype-icelake-client-deprecation.conf   |   3 -
 ...putype-icelake-client-deprecation.conf.cmd |   7 +-
 .../custom-cpu-model-defaults.conf.cmd        |   2 +-
 test/cfg2cmd/efi-raw-template.conf.cmd        |   7 +-
 test/cfg2cmd/efi-raw.conf                     |   1 +
 test/cfg2cmd/efi-raw.conf.cmd                 |   2 +-
 test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd |   7 +-
 test/cfg2cmd/efi-secboot-and-tpm.conf.cmd     |   7 +-
 test/cfg2cmd/efidisk-on-rbd.conf              |   1 +
 test/cfg2cmd/efidisk-on-rbd.conf.cmd          |   2 +-
 test/cfg2cmd/i440fx-viommu-virtio.conf.cmd    |   2 +-
 test/cfg2cmd/ide-blockdev.conf                |  14 +
 test/cfg2cmd/ide-blockdev.conf.cmd            |  46 ++
 test/cfg2cmd/ide.conf                         |   1 +
 test/cfg2cmd/ide.conf.cmd                     |   2 +-
 .../cfg2cmd/memory-hotplug-hugepages.conf.cmd |   2 +-
 test/cfg2cmd/memory-hotplug.conf.cmd          |   2 +-
 test/cfg2cmd/memory-hugepages-1g.conf.cmd     |   2 +-
 test/cfg2cmd/memory-hugepages-2m.conf.cmd     |   2 +-
 test/cfg2cmd/minimal-defaults.conf.cmd        |   2 +-
 test/cfg2cmd/netdev-7.1-multiqueues.conf.cmd  |   2 +-
 test/cfg2cmd/netdev-7.1.conf.cmd              |   2 +-
 test/cfg2cmd/q35-ide-blockdev.conf            |  15 +
 test/cfg2cmd/q35-ide-blockdev.conf.cmd        |  45 ++
 test/cfg2cmd/q35-ide.conf                     |   2 +-
 test/cfg2cmd/q35-ide.conf.cmd                 |   3 +-
 .../q35-linux-hostpci-mapping.conf.cmd        |   7 +-
 .../q35-linux-hostpci-multifunction.conf.cmd  |   7 +-
 .../q35-linux-hostpci-template.conf.cmd       |  10 +-
 ...q35-linux-hostpci-x-pci-overrides.conf.cmd |   9 +-
 test/cfg2cmd/q35-linux-hostpci.conf.cmd       |   7 +-
 test/cfg2cmd/q35-simple.conf.cmd              |   7 +-
 test/cfg2cmd/q35-viommu-intel.conf.cmd        |   2 +-
 test/cfg2cmd/q35-viommu-virtio.conf.cmd       |   2 +-
 test/cfg2cmd/seabios_serial.conf              |   4 -
 test/cfg2cmd/seabios_serial.conf.cmd          |   7 +-
 test/cfg2cmd/sev-es.conf.cmd                  |   7 +-
 test/cfg2cmd/sev-snp.conf.cmd                 |   2 +-
 test/cfg2cmd/sev-std.conf.cmd                 |   7 +-
 test/cfg2cmd/simple-backingchain.conf         |  21 +
 test/cfg2cmd/simple-backingchain.conf.cmd     |  30 +
 test/cfg2cmd/simple-btrfs.conf                |   1 +
 test/cfg2cmd/simple-btrfs.conf.cmd            |   2 +-
 .../simple-disk-passthrough-blockdev.conf     |  15 +
 .../simple-disk-passthrough-blockdev.conf.cmd |  40 +
 test/cfg2cmd/simple-disk-passthrough.conf     |   1 +
 test/cfg2cmd/simple-disk-passthrough.conf.cmd |   2 +-
 test/cfg2cmd/simple-rbd-blockdev.conf         |  15 +
 test/cfg2cmd/simple-rbd-blockdev.conf.cmd     |  38 +
 test/cfg2cmd/simple-rbd.conf                  |   1 +
 test/cfg2cmd/simple-rbd.conf.cmd              |   2 +-
 test/cfg2cmd/simple-virtio-blk-blockdev.conf  |  13 +
 .../simple-virtio-blk-blockdev.conf.cmd       |  35 +
 test/cfg2cmd/simple-virtio-blk.conf           |   1 +
 test/cfg2cmd/simple-virtio-blk.conf.cmd       |   2 +-
 .../simple-zfs-over-iscsi-blockdev.conf       |  14 +
 .../simple-zfs-over-iscsi-blockdev.conf.cmd   |  35 +
 test/cfg2cmd/simple-zfs-over-iscsi.conf       |   1 +
 test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd   |   2 +-
 test/cfg2cmd/simple1-blockdev.conf            |  14 +
 test/cfg2cmd/simple1-blockdev.conf.cmd        |  35 +
 test/cfg2cmd/simple1-template.conf.cmd        |  13 +-
 test/cfg2cmd/simple1.conf                     |   1 +
 test/cfg2cmd/simple1.conf.cmd                 |   2 +-
 test/cfg2cmd/vnc-clipboard-spice.conf.cmd     |   2 +-
 test/cfg2cmd/vnc-clipboard-std.conf.cmd       |   2 +-
 test/run_config2command_tests.pl              |  52 ++
 test/run_qemu_img_convert_tests.pl            |  44 +
 test/snapshot-test.pm                         |   4 +-
 86 files changed, 1905 insertions(+), 394 deletions(-)
 create mode 100644 PVE/QemuServer/Blockdev.pm
 create mode 100644 test/cfg2cmd/bootorder-blockdev.conf
 create mode 100644 test/cfg2cmd/bootorder-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/bootorder-empty-blockdev.conf
 create mode 100644 test/cfg2cmd/bootorder-empty-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/ide-blockdev.conf
 create mode 100644 test/cfg2cmd/ide-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/q35-ide-blockdev.conf
 create mode 100644 test/cfg2cmd/q35-ide-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/simple-backingchain.conf
 create mode 100644 test/cfg2cmd/simple-backingchain.conf.cmd
 create mode 100644 test/cfg2cmd/simple-disk-passthrough-blockdev.conf
 create mode 100644 test/cfg2cmd/simple-disk-passthrough-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/simple-rbd-blockdev.conf
 create mode 100644 test/cfg2cmd/simple-rbd-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/simple-virtio-blk-blockdev.conf
 create mode 100644 test/cfg2cmd/simple-virtio-blk-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi-blockdev.conf
 create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi-blockdev.conf.cmd
 create mode 100644 test/cfg2cmd/simple1-blockdev.conf
 create mode 100644 test/cfg2cmd/simple1-blockdev.conf.cmd

pve-qemu:

Alexandre Derumier (1):
  add fake 10.0

 debian/changelog                              |   6 +
 .../extra/0001-bump-version-to-10.0.0.patch   |  19 ++
 ...438a42f7a178504347456b9e50e3d2c50478.patch | 247 ++++++++++++++++++
 ...b0606879ddeddf04bcf8440aabcfe1393b9c.patch | 120 +++++++++
 debian/patches/series                         |   3 +
 5 files changed, 395 insertions(+)
 create mode 100644 debian/patches/extra/0001-bump-version-to-10.0.0.patch
 create mode 100644 debian/patches/extra/0a7c438a42f7a178504347456b9e50e3d2c50478.patch
 create mode 100644 debian/patches/extra/7956b0606879ddeddf04bcf8440aabcfe1393b9c.patch


-- 
2.39.5



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Fiona Ebner]

Am 03.06.25 um 09:55 schrieb Alexandre Derumier via pve-devel:
> This new patch series enable blockdev for qemu machine > 10.0 to avoid breaking efidisk and maybe
> potential migrations bug

Did you see any actual issues with migration or mirroring the EFI disk
now or is this just a gut feeling?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 13370 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "f.ebner@proxmox.com" <f.ebner@proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Thu, 5 Jun 2025 14:08:21 +0000
Message-ID: <0aa862ab988ff9df50f4637f3712f137f23c34c2.camel@groupe-cyllene.com>

Am 03.06.25 um 09:55 schrieb Alexandre Derumier via pve-devel:
> This new patch series enable blockdev for qemu machine > 10.0 to
> avoid breaking efidisk and maybe
> potential migrations bug

>>Did you see any actual issues with migration or mirroring the EFI
>>disk
>>now or is this just a gut feeling?

I don't have tested it to be honest (because don't use efi in
production, and I really don't known how to verify that it's working
fine after a migration+ also the size param) , so it was more by
security

but as it's attached to -machine... directly instead -drive, I really
don't known.  and they was also the version gate for old machine
version without raw support.

We also need to handle migration+storage motion for local dirve between
pve8->pve9, and
I don't have tested if pve8 can still use drive-mirror to pve9 nbd
blockdev.


I only have tested live vm migration with shared storage between pve8
to pve9 without efi,
and the switch from drive to blockdev was working fine.













[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Fiona Ebner]

Am 05.06.25 um 16:08 schrieb DERUMIER, Alexandre:
> Am 03.06.25 um 09:55 schrieb Alexandre Derumier via pve-devel:
>> This new patch series enable blockdev for qemu machine > 10.0 to
>> avoid breaking efidisk and maybe
>> potential migrations bug
> 
>>> Did you see any actual issues with migration or mirroring the EFI
>>> disk
>>> now or is this just a gut feeling?
> 
> I don't have tested it to be honest (because don't use efi in
> production, and I really don't known how to verify that it's working
> fine after a migration+ also the size param) , so it was more by
> security
> 
> but as it's attached to -machine... directly instead -drive, I really
> don't known.  and they was also the version gate for old machine
> version without raw support.
> 
> We also need to handle migration+storage motion for local dirve between
> pve8->pve9, and
> I don't have tested if pve8 can still use drive-mirror to pve9 nbd
> blockdev.
> 
> 
> I only have tested live vm migration with shared storage between pve8
> to pve9 without efi,
> and the switch from drive to blockdev was working fine.

I am inclined to go with version guarding too. I'm doubtful we can cover
all edge cases and all defaults even if we are very careful. There are
just too many possible options and combinations for drives. It's likely
already required for setting the 'write-cache' option on the front-end
drive device, since we didn't do that before. Maybe that won't change
the migration stream either, but it's better to be safe than sorry.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[joao.sousa via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 4691 bytes --]

From: "joao.sousa" <joao.sousa@eurotux.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Fri, 6 Jun 2025 11:17:38 +0100
Message-ID: <do3oxkhuf4myq4dv5reybpj3j5hfz25wyn4i7ahdas7p2tmusl@rrlds5tnevin>

Hi,

I'd like to help out with this feature. I've gone through the developer
documentation and have a basic understanding of the topic, but to be
honest, I'm feeling a bit lost on how to get started.
Could you please point me in the right direction or suggest some initial
steps I can take to begin working on this?

Thanks in advance!

Best regards,
Tiago Sousa


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Fiona Ebner]

Hi Tiago,

Am 06.06.25 um 12:17 schrieb joao.sousa via pve-devel:
> Hi,
> 
> I'd like to help out with this feature. I've gone through the developer
> documentation and have a basic understanding of the topic, but to be
> honest, I'm feeling a bit lost on how to get started.
> Could you please point me in the right direction or suggest some initial
> steps I can take to begin working on this?

What exactly is your goal with the feature? If you want to build
something on top of it, it might be better to wait until it is
finalized/applied. Currently, the most helpful contribution with the
feature itself would be testing and reviewing Alexandre's changes. For
testing, you'll need to apply the changes and build and install the
affected packages. Reviewing requires getting familiar with the code
base first.

Best Regards,
Fiona


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[joao.sousa via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 5964 bytes --]

From: "joao.sousa" <joao.sousa@eurotux.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Wed, 11 Jun 2025 22:28:42 +0100
Message-ID: <5sqpzsnyzmfvvoco3ilo6tg3pcpomhle7h5dsnmbsbdjobpr5a@rqfsbsas36e2>

On 10/06/25 10:54, Fiona Ebner wrote:
> What exactly is your goal with the feature? If you want to build
> something on top of it, it might be better to wait until it is
> finalized/applied. Currently, the most helpful contribution with the
> feature itself would be testing and reviewing Alexandre's changes. For
> testing, you'll need to apply the changes and build and install the
> affected packages. Reviewing requires getting familiar with the code
> base first.

For now, I want to help agile the process of getting the feature to a
stable state. Later on, I'd like to contribute to a version which
enables thin provisioned snapshots.

Regarding the testing, I am having a little bit of trouble applying the
current patches.

This is the output of git am when applying the qemu-server patches:

Applying: blockdev: cmdline: add blockdev syntax support
.git/rebase-apply/patch:972: trailing whitespace.
boot:
error: patch failed: test/cfg2cmd/simple-btrfs.conf.cmd:32
error: test/cfg2cmd/simple-btrfs.conf.cmd: patch does not apply
error: patch failed: test/cfg2cmd/simple-rbd.conf.cmd:34
error: test/cfg2cmd/simple-rbd.conf.cmd: patch does not apply
error: patch failed: test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd:32
error: test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd: patch does not apply
Patch failed at 0001 blockdev: cmdline: add blockdev syntax support
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

It also happens when applying the pve-storage patches. Am I missing something?


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 13114 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "joao.sousa@eurotux.com" <joao.sousa@eurotux.com>, "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Thu, 12 Jun 2025 07:02:24 +0000
Message-ID: <8cbed06bbe3db00e96670dec1e240123b946a917.camel@groupe-cyllene.com>

>> Later on, I'd like to contribute to a version which
>>enables thin provisioned snapshots.

For thin provision, I have done prelimary work in september 2024 (check
the pve-devel mailing),
but I was waiting to have first the snapshot finished.

I can help you if you want about this.




[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Fiona Ebner]

Am 11.06.25 um 23:28 schrieb joao.sousa:
> This is the output of git am when applying the qemu-server patches:
> 
> Applying: blockdev: cmdline: add blockdev syntax support
> .git/rebase-apply/patch:972: trailing whitespace.
> boot:
> error: patch failed: test/cfg2cmd/simple-btrfs.conf.cmd:32
> error: test/cfg2cmd/simple-btrfs.conf.cmd: patch does not apply
> error: patch failed: test/cfg2cmd/simple-rbd.conf.cmd:34
> error: test/cfg2cmd/simple-rbd.conf.cmd: patch does not apply
> error: patch failed: test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd:32
> error: test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd: patch does not apply
> Patch failed at 0001 blockdev: cmdline: add blockdev syntax support
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".
> 
> It also happens when applying the pve-storage patches. Am I missing something?

The tests in qemu-server were updated since then. You can check out an
earlier commit and apply on top of that. For pve-storage, I'd guess it
was the code style changes, so same story:

https://lore.proxmox.com/pve-devel/241b0a74-ed26-4e01-a187-ed7f755e18fa@proxmox.com/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14128 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "joao.sousa@eurotux.com" <joao.sousa@eurotux.com>, "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Thu, 12 Jun 2025 11:33:18 +0000
Message-ID: <a7185aa585b414710be6e09729adda57ed3787b8.camel@groupe-cyllene.com>

> > Later on, I'd like to contribute to a version which
> > enables thin provisioned snapshots.

>>For thin provision, I have done prelimary work in september 2024
>>(check
>>the pve-devel mailing),
>>but I was waiting to have first the snapshot finished.

here the patch series:
https://lists.proxmox.com/pipermail/pve-devel/2024-August/065201.html

The idea is to create a lvm volume smaller than the real qcow2 size,

qemu is sending an event when the used space reach a threshold,

then a daemon should resize the lvm volume by increment.(1GB for
example),
until the next threshold reach....

The only complex thing is to manage some kind of queue in this daemon
and manage
lvm cluster lock, as we can't resize multiple lvm volume at the same
time.
(Not sure if it need a central daemon, or a distributed daemon on each
node with a shared queue in /etc/pve ...)






[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[joao.sousa via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 4407 bytes --]

From: "joao.sousa" <joao.sousa@eurotux.com>
To: Fiona Ebner <f.ebner@proxmox.com>,  Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Fri, 13 Jun 2025 11:25:01 +0100
Message-ID: <zwkygc6va6jmieho2h7gqsx4idrgrxvd7et2kfbi3vyddhjk7c@kfiwufb7auvv>

On 12/06/25 10:12, Fiona Ebner wrote:
> The tests in qemu-server were updated since then. You can check out an
> earlier commit and apply on top of that. For pve-storage, I'd guess it
> was the code style changes, so same story:

I reverted qemu-server to commit 1fd1ca60, however it still fails to
apply patch 12:

[PATCH qemu-server 12/13] blockdev: add backing_chain support

Applying: blockdev: add backing_chain support
error: corrupt patch at line 18
Patch failed at 0012 blockdev: add backing_chain support
hint: Use 'git am --show-current-patch=diff' to see the failed patch


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[joao.sousa--- via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 6013 bytes --]

From: "joao.sousa@eurotux.com" <joao.sousa@eurotux.com>
To: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>,  "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Fri, 13 Jun 2025 17:42:13 +0100
Message-ID: <qss36oltvd4gz5tiss7dqono5h2zmsgwmodj4cqkroboizypwj@kbyz3mymh75a>

On 12/06/25 11:33, DERUMIER, Alexandre wrote:
> here the patch series:
> https://lists.proxmox.com/pipermail/pve-devel/2024-August/065201.html
Thanks I'll definitely be checking it out.

> The idea is to create a lvm volume smaller than the real qcow2 size,
>
> qemu is sending an event when the used space reach a threshold,
>
> then a daemon should resize the lvm volume by increment.(1GB for
> example),
> until the next threshold reach....

I was thinking the same way (probably influenced by oVirt's way of
achieving this)

> The only complex thing is to manage some kind of queue in this daemon
> and manage
> lvm cluster lock, as we can't resize multiple lvm volume at the same
> time.
> (Not sure if it need a central daemon, or a distributed daemon on each
> node with a shared queue in /etc/pve ...)

The solution that makes the most sense to me at the moment would be a
"distributed daemon", as in a daemon running on each node of the
cluster. Each node would maintain its own local queue for disk extends
related to the VMs hosted there. When an extend is needed, the node would
acquire the storage lock, perform the task, and then release the lock
so other nodes can proceed with their operations.

What are your thoughts on this approach?

The help is much appreciated :)


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[DERUMIER, Alexandre via pve-devel]

[-- Attachment #1: Type: message/rfc822, Size: 14801 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "joao.sousa@eurotux.com" <joao.sousa@eurotux.com>, "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Mon, 16 Jun 2025 05:50:52 +0000
Message-ID: <0e7791af247f4183cfd03ed5675bb7688f298d11.camel@groupe-cyllene.com>

>>I was thinking the same way (probably influenced by oVirt's way of
>>achieving this)

Yes, I don't have reinvented the wheel. I have customers with ovirt in
production with this setup, and I known that it's working 

> The only complex thing is to manage some kind of queue in this daemon
> and manage
> lvm cluster lock, as we can't resize multiple lvm volume at the same
> time.
> (Not sure if it need a central daemon, or a distributed daemon on
> each
> node with a shared queue in /etc/pve ...)

>>The solution that makes the most sense to me at the moment would be a
>>"distributed daemon", as in a daemon running on each node of the
>>cluster. Each node would maintain its own local queue for disk
>>extends
>>related to the VMs hosted there. When an extend is needed, the node
>>would
>>acquire the storage lock, perform the task, and then release the lock
>>so other nodes can proceed with their operations.
>>
>>What are your thoughts on this approach?

yes, I was thinking about something like that too.

Worst case could be all vms extendend at the same time, and extend
could not be enough fast, that's why I have also a check in pvestatd
about vm error (if storage is full, the vm will be pause). I think this
could send a notification to the resize daemon too, with high priority
queue.

so maybe something like:

qemu---->qmeventd------> resize queue(/etc/pve/..)<----- resize daemon
         pvestatd------>



>>The help is much appreciated :)

No problem ! (Note that you don't need all the snapshots && blockdev
patches to work  on it, I think that pve-storage patches to add support
on qcow2-lvm should be enough to start)



[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support

[Fiona Ebner]

Am 13.06.25 um 12:25 schrieb joao.sousa:
> On 12/06/25 10:12, Fiona Ebner wrote:
>> The tests in qemu-server were updated since then. You can check out an
>> earlier commit and apply on top of that. For pve-storage, I'd guess it
>> was the code style changes, so same story:
> 
> I reverted qemu-server to commit 1fd1ca60, however it still fails to
> apply patch 12:
> 
> [PATCH qemu-server 12/13] blockdev: add backing_chain support
> 
> Applying: blockdev: add backing_chain support
> error: corrupt patch at line 18
> Patch failed at 0012 blockdev: add backing_chain support
> hint: Use 'git am --show-current-patch=diff' to see the failed patch

Right, that one failed for me as well. Didn't check out why exactly, but
there will be a separate series for the switch to blockdev [0]. After
that, Alexandre can rebase the changes for external snapshot on top and
send another version of the patches.

[0]:
https://lore.proxmox.com/pve-devel/mailman.381.1750055007.395.pve-devel@lists.proxmox.com/T/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel