From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id F1DD81FF15E
for <inbox@lore.proxmox.com>; Tue, 3 Jun 2025 09:57:00 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 0245610F20;
Tue, 3 Jun 2025 09:56:44 +0200 (CEST)
To: pve-devel@lists.proxmox.com
Date: Tue, 3 Jun 2025 09:55:35 +0200
MIME-Version: 1.0
Message-ID: <mailman.197.1748937402.395.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: [pve-devel] [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add
external qcow2 snapshot support
Content-Type: multipart/mixed; boundary="===============1947315644251080764=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============1947315644251080764==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <root@formationkvm1.odiso.net>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 3D74ACB617
for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:42 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 1A137109F7
for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:12 +0200 (CEST)
Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:09 +0200 (CEST)
Received: from formationkvm1.odiso.net (unknown [10.11.201.57])
by bastiontest.odiso.net (Postfix) with ESMTP id 12726860944;
Tue, 3 Jun 2025 09:56:02 +0200 (CEST)
Received: by formationkvm1.odiso.net (Postfix, from userid 0)
id 30F0B110DBD3; Tue, 3 Jun 2025 09:56:01 +0200 (CEST)
From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH-SERIES v6 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Tue, 3 Jun 2025 09:55:35 +0200
Message-Id: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.097 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_NONE 0.1 DMARC none policy
HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different
KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods
RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_NONE 0.001 SPF: sender does not publish an SPF Record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [common.pm,lvmplugin.pm,iscsidirectplugin.pm,esxiplugin.pm,qemuconfig.pm,zfspoolplugin.pm,qemuserver.pm,drive.pm,qemumigratemock.pm,storage.pm,qemumigrate.pm,btrfsplugin.pm,snapshot-test.pm,pbsplugin.pm,plugin.pm,glusterfsplugin.pm,blockdev.pm,lvmthinplugin.pm,dirplugin.pm,rbdplugin.pm]
This patch series implement qcow2 external snapshot support for files && lvm volumes
The current internal qcow2 snapshots have bad write performance because no metadatas can be preallocated.
This is particulary visible on a shared filesystem like ocfs2 or gfs2.
Also other bugs are freeze/lock reported by users since years on snapshots delete on nfs
(The disk access seem to be frozen during all the delete duration)
This also open doors for remote snapshot export-import for storage replication.
The pve-storage patches applied on top of Fiona qemu_blockdev_options patch series
https://lists.proxmox.com/pipermail/pve-devel/2025-May/071204.html
This new patch series enable blockdev for qemu machine > 10.0 to avoid breaking efidisk and maybe
potential migrations bug
Changelog v6:
pve-storage:
- use fiona qemu_blockdev_options patch
- add helpers for qemu-img call
- use qemu-img measure for lvm size (including l2_extended=on)
- add a new helper volume_support_qemu_snapshot
- fix Fabian requested changes
qemu-server:
- enable blockdev for machine version > 10.0
- move blockdev code in QemuServer/Blockdev.pm
- use Fiona qemu_blockdev_options patch for storage plugins
- add raw size for efidisk
- fix drivedel with backing chain
- add tests for qemu-img convert, backing chain,....
- fix Fabian requested changes
pve-qemu:
- add a fake 10.0 qemu bump for testing
MAIN TODO:
- convert/implement all the backup code, dirty bitmap mirro, ...
- add snapshots tests
pve-storage:
Alexandre Derumier (9):
common: add qemu_img_create an preallocation_cmd_option
common: add qemu_img_create_qcow2_backed
common: add qemu_img_info helper
common: add qemu_img_measure
rename_volume: add source && target snap
storage: volume_snapshot: add $running param + api bump
qcow2: add external snapshot support
lvmplugin: add qcow2 snapshot
storage: add volume_support_qemu_snapshot
src/PVE/Storage.pm | 28 +-
src/PVE/Storage/BTRFSPlugin.pm | 2 +-
src/PVE/Storage/Common.pm | 175 ++++++++++
src/PVE/Storage/DirPlugin.pm | 11 +
src/PVE/Storage/ESXiPlugin.pm | 2 +-
src/PVE/Storage/GlusterfsPlugin.pm | 2 +-
src/PVE/Storage/ISCSIDirectPlugin.pm | 2 +-
src/PVE/Storage/LVMPlugin.pm | 467 +++++++++++++++++++++++----
src/PVE/Storage/LvmThinPlugin.pm | 2 +-
src/PVE/Storage/PBSPlugin.pm | 2 +-
src/PVE/Storage/Plugin.pm | 339 ++++++++++++++-----
src/PVE/Storage/RBDPlugin.pm | 8 +-
src/PVE/Storage/ZFSPoolPlugin.pm | 2 +-
13 files changed, 875 insertions(+), 167 deletions(-)
qemu-server:
Alexandre Derumier (13):
blockdev: cmdline: add blockdev syntax support
blockdev: add support for ovmf && efidisk
blockdev: vm_devices_list : fix block-query
blockdev: add support to qemu_driveadd && qemu_drivedel
blockdev: add support to qemu_block_set_io_throttle
blockdev: add support for cdrom media eject/insert
blockdev: add support for block_resize
blockdev: add support for nbd_export: block-export-add
blockdev: add blockdev_mirror
blockdev: change aio on target if io_uring is not default.
qemu_img convert : add external snapshot support
blockdev: add backing_chain support
qcow2: add external snapshot support
PVE/QemuConfig.pm | 4 +-
PVE/QemuMigrate.pm | 2 +-
PVE/QemuServer.pm | 761 +++++++++++-------
PVE/QemuServer/Blockdev.pm | 494 ++++++++++++
PVE/QemuServer/Drive.pm | 201 +++++
PVE/QemuServer/Makefile | 1 +
test/MigrationTest/QemuMigrateMock.pm | 10 +-
test/cfg2cmd/bootorder-blockdev.conf | 15 +
test/cfg2cmd/bootorder-blockdev.conf.cmd | 43 +
test/cfg2cmd/bootorder-empty-blockdev.conf | 15 +
.../cfg2cmd/bootorder-empty-blockdev.conf.cmd | 43 +
test/cfg2cmd/bootorder-empty.conf | 1 +
test/cfg2cmd/bootorder-empty.conf.cmd | 2 +-
test/cfg2cmd/bootorder-legacy.conf | 1 +
test/cfg2cmd/bootorder-legacy.conf.cmd | 2 +-
test/cfg2cmd/bootorder.conf | 1 +
test/cfg2cmd/bootorder.conf.cmd | 2 +-
.../cputype-icelake-client-deprecation.conf | 3 -
...putype-icelake-client-deprecation.conf.cmd | 7 +-
.../custom-cpu-model-defaults.conf.cmd | 2 +-
test/cfg2cmd/efi-raw-template.conf.cmd | 7 +-
test/cfg2cmd/efi-raw.conf | 1 +
test/cfg2cmd/efi-raw.conf.cmd | 2 +-
test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd | 7 +-
test/cfg2cmd/efi-secboot-and-tpm.conf.cmd | 7 +-
test/cfg2cmd/efidisk-on-rbd.conf | 1 +
test/cfg2cmd/efidisk-on-rbd.conf.cmd | 2 +-
test/cfg2cmd/i440fx-viommu-virtio.conf.cmd | 2 +-
test/cfg2cmd/ide-blockdev.conf | 14 +
test/cfg2cmd/ide-blockdev.conf.cmd | 46 ++
test/cfg2cmd/ide.conf | 1 +
test/cfg2cmd/ide.conf.cmd | 2 +-
.../cfg2cmd/memory-hotplug-hugepages.conf.cmd | 2 +-
test/cfg2cmd/memory-hotplug.conf.cmd | 2 +-
test/cfg2cmd/memory-hugepages-1g.conf.cmd | 2 +-
test/cfg2cmd/memory-hugepages-2m.conf.cmd | 2 +-
test/cfg2cmd/minimal-defaults.conf.cmd | 2 +-
test/cfg2cmd/netdev-7.1-multiqueues.conf.cmd | 2 +-
test/cfg2cmd/netdev-7.1.conf.cmd | 2 +-
test/cfg2cmd/q35-ide-blockdev.conf | 15 +
test/cfg2cmd/q35-ide-blockdev.conf.cmd | 45 ++
test/cfg2cmd/q35-ide.conf | 2 +-
test/cfg2cmd/q35-ide.conf.cmd | 3 +-
.../q35-linux-hostpci-mapping.conf.cmd | 7 +-
.../q35-linux-hostpci-multifunction.conf.cmd | 7 +-
.../q35-linux-hostpci-template.conf.cmd | 10 +-
...q35-linux-hostpci-x-pci-overrides.conf.cmd | 9 +-
test/cfg2cmd/q35-linux-hostpci.conf.cmd | 7 +-
test/cfg2cmd/q35-simple.conf.cmd | 7 +-
test/cfg2cmd/q35-viommu-intel.conf.cmd | 2 +-
test/cfg2cmd/q35-viommu-virtio.conf.cmd | 2 +-
test/cfg2cmd/seabios_serial.conf | 4 -
test/cfg2cmd/seabios_serial.conf.cmd | 7 +-
test/cfg2cmd/sev-es.conf.cmd | 7 +-
test/cfg2cmd/sev-snp.conf.cmd | 2 +-
test/cfg2cmd/sev-std.conf.cmd | 7 +-
test/cfg2cmd/simple-backingchain.conf | 21 +
test/cfg2cmd/simple-backingchain.conf.cmd | 30 +
test/cfg2cmd/simple-btrfs.conf | 1 +
test/cfg2cmd/simple-btrfs.conf.cmd | 2 +-
.../simple-disk-passthrough-blockdev.conf | 15 +
.../simple-disk-passthrough-blockdev.conf.cmd | 40 +
test/cfg2cmd/simple-disk-passthrough.conf | 1 +
test/cfg2cmd/simple-disk-passthrough.conf.cmd | 2 +-
test/cfg2cmd/simple-rbd-blockdev.conf | 15 +
test/cfg2cmd/simple-rbd-blockdev.conf.cmd | 38 +
test/cfg2cmd/simple-rbd.conf | 1 +
test/cfg2cmd/simple-rbd.conf.cmd | 2 +-
test/cfg2cmd/simple-virtio-blk-blockdev.conf | 13 +
.../simple-virtio-blk-blockdev.conf.cmd | 35 +
test/cfg2cmd/simple-virtio-blk.conf | 1 +
test/cfg2cmd/simple-virtio-blk.conf.cmd | 2 +-
.../simple-zfs-over-iscsi-blockdev.conf | 14 +
.../simple-zfs-over-iscsi-blockdev.conf.cmd | 35 +
test/cfg2cmd/simple-zfs-over-iscsi.conf | 1 +
test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd | 2 +-
test/cfg2cmd/simple1-blockdev.conf | 14 +
test/cfg2cmd/simple1-blockdev.conf.cmd | 35 +
test/cfg2cmd/simple1-template.conf.cmd | 13 +-
test/cfg2cmd/simple1.conf | 1 +
test/cfg2cmd/simple1.conf.cmd | 2 +-
test/cfg2cmd/vnc-clipboard-spice.conf.cmd | 2 +-
test/cfg2cmd/vnc-clipboard-std.conf.cmd | 2 +-
test/run_config2command_tests.pl | 52 ++
test/run_qemu_img_convert_tests.pl | 44 +
test/snapshot-test.pm | 4 +-
86 files changed, 1905 insertions(+), 394 deletions(-)
create mode 100644 PVE/QemuServer/Blockdev.pm
create mode 100644 test/cfg2cmd/bootorder-blockdev.conf
create mode 100644 test/cfg2cmd/bootorder-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/bootorder-empty-blockdev.conf
create mode 100644 test/cfg2cmd/bootorder-empty-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/ide-blockdev.conf
create mode 100644 test/cfg2cmd/ide-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/q35-ide-blockdev.conf
create mode 100644 test/cfg2cmd/q35-ide-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/simple-backingchain.conf
create mode 100644 test/cfg2cmd/simple-backingchain.conf.cmd
create mode 100644 test/cfg2cmd/simple-disk-passthrough-blockdev.conf
create mode 100644 test/cfg2cmd/simple-disk-passthrough-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/simple-rbd-blockdev.conf
create mode 100644 test/cfg2cmd/simple-rbd-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/simple-virtio-blk-blockdev.conf
create mode 100644 test/cfg2cmd/simple-virtio-blk-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi-blockdev.conf
create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi-blockdev.conf.cmd
create mode 100644 test/cfg2cmd/simple1-blockdev.conf
create mode 100644 test/cfg2cmd/simple1-blockdev.conf.cmd
pve-qemu:
Alexandre Derumier (1):
add fake 10.0
debian/changelog | 6 +
.../extra/0001-bump-version-to-10.0.0.patch | 19 ++
...438a42f7a178504347456b9e50e3d2c50478.patch | 247 ++++++++++++++++++
...b0606879ddeddf04bcf8440aabcfe1393b9c.patch | 120 +++++++++
debian/patches/series | 3 +
5 files changed, 395 insertions(+)
create mode 100644 debian/patches/extra/0001-bump-version-to-10.0.0.patch
create mode 100644 debian/patches/extra/0a7c438a42f7a178504347456b9e50e3d2c50478.patch
create mode 100644 debian/patches/extra/7956b0606879ddeddf04bcf8440aabcfe1393b9c.patch
--
2.39.5
--===============1947315644251080764==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============1947315644251080764==--