From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id 6D72E1FF15E
for <inbox@lore.proxmox.com>; Tue, 22 Apr 2025 13:52:03 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 3FCAF3A72C;
Tue, 22 Apr 2025 13:51:56 +0200 (CEST)
To: pve-devel@lists.proxmox.com
Date: Tue, 22 Apr 2025 13:51:21 +0200
MIME-Version: 1.0
Message-ID: <mailman.16.1745322714.394.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: [pve-devel] [PATCH-SERIES v5 pve-storage/qemu-server/pve-qemu] add
external qcow2 snapshot support
Content-Type: multipart/mixed; boundary="===============3263708610982164954=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============3263708610982164954==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <root@formationkvm1.odiso.net>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id A05E1C8BDD
for <pve-devel@lists.proxmox.com>; Tue, 22 Apr 2025 13:51:53 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 84CE73A5F9
for <pve-devel@lists.proxmox.com>; Tue, 22 Apr 2025 13:51:53 +0200 (CEST)
Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Tue, 22 Apr 2025 13:51:51 +0200 (CEST)
Received: from formationkvm1.odiso.net (unknown [10.11.201.57])
by bastiontest.odiso.net (Postfix) with ESMTP id 59167860941;
Tue, 22 Apr 2025 13:51:44 +0200 (CEST)
Received: by formationkvm1.odiso.net (Postfix, from userid 0)
id 253C71115FBC; Tue, 22 Apr 2025 13:51:43 +0200 (CEST)
From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH-SERIES v5 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support
Date: Tue, 22 Apr 2025 13:51:21 +0200
Message-Id: <20250422115141.808427-1-alexandre.derumier@groupe-cyllene.com>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.107 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_NONE 0.1 DMARC none policy
HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different
KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods
RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_NONE 0.001 SPF: sender does not publish an SPF Record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemumigrate.pm,drive.pm,storage.pm,lvmplugin.pm,qemumigratemock.pm,dirplugin.pm,plugin.pm,rbdplugin.pm,qemuserver.pm,qemuconfig.pm]
This patch series implement qcow2 external snapshot support for files && lvm volumes
The current internal qcow2 snapshots have bad write performance because no metadatas can be preallocated.
This is particulary visible on a shared filesystem like ocfs2 or gfs2.
Also other bugs are freeze/lock reported by users since years on snapshots delete on nfs
(The disk access seem to be frozen during all the delete duration)
This also open doors for remote snapshot export-import for storage replication.
Changelog v5:
pve-storage:
-fix lvm cleanup_worker on removal
- fix Fabian requested changes
qemu-server:
- fix Fabian requested changes
- add suport for iscsi:// blockdev
- convert efidisk && pflash0 to blockdev
- fix cloudinit drive change
- add some units tests for simple rbd,zfs-isci,disk-passthrough
- fix rbd ipv6 monitoring parsing
- fix disk passthrough
- fix qemu-img convert with external snapshot
MAIN TODO:
- convert/implement all the backup code, dirty bitmap mirro, ...
- ceph keyring as blockdev option
- unit tests
Note: Maybe it'll need version guarding finally, as I'm not sure than efidisk && pflash
will be able to live migrate or livemigration+storagemigrate.
(They are already some specific version guard about efidisk size)
I'll look at this for the v6
pve-storage:
Alexandre Derumier (5):
rename_volume: add source && target snap
qcow2: add external snapshot support
lvmplugin: add qcow2 snapshot
storage: vdisk_free: remove external snapshots
volume_has_feature: return storage|qemu_internal|qemu_external
snapshot_type
src/PVE/Storage.pm | 11 +-
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/LVMPlugin.pm | 381 ++++++++++++++++++++++++++++++-----
src/PVE/Storage/Plugin.pm | 320 ++++++++++++++++++++++++-----
src/PVE/Storage/RBDPlugin.pm | 8 +-
5 files changed, 616 insertions(+), 105 deletions(-)
qemu-server
Alexandre Derumier (14):
tests: add cfg2cmd for disk passthrough, rbd,krbd && zfs-over-scsi.
blockdev: cmdline: convert drive to blockdev syntax
blockdev: convert ovmf && efidisk to blockdev
blockdev : convert qemu_driveadd && qemu_drivedel
replace qemu_block_set_io_throttle with qom-set throttlegroup limits
blockdev: vm_devices_list : fix block-query
blockdev: convert cdrom media eject/insert
blockdev: block_resize: convert to blockdev
blockdev: nbd_export: block-export-add : use drive-$id for nodename
blockdev: convert drive_mirror to blockdev_mirror
blockdev: change aio on target if io_uring is not default.
qemu_img convert : add external snapshot support
blockdev: add backing_chain support
qcow2: add external snapshot support
PVE/QemuConfig.pm | 4 +-
PVE/QemuMigrate.pm | 2 +-
PVE/QemuServer.pm | 705 +++++++++---------
PVE/QemuServer/Drive.pm | 461 ++++++++++--
test/MigrationTest/QemuMigrateMock.pm | 10 +-
test/cfg2cmd/bootorder-empty.conf.cmd | 12 +-
test/cfg2cmd/bootorder-legacy.conf.cmd | 12 +-
test/cfg2cmd/bootorder.conf.cmd | 12 +-
...putype-icelake-client-deprecation.conf.cmd | 6 +-
test/cfg2cmd/efi-raw-old.conf.cmd | 7 +-
test/cfg2cmd/efi-raw-template.conf.cmd | 7 +-
test/cfg2cmd/efi-raw.conf.cmd | 7 +-
test/cfg2cmd/efi-secboot-and-tpm-q35.conf.cmd | 7 +-
test/cfg2cmd/efi-secboot-and-tpm.conf.cmd | 7 +-
test/cfg2cmd/efidisk-on-rbd.conf.cmd | 7 +-
test/cfg2cmd/i440fx-win10-hostpci.conf.cmd | 7 +-
test/cfg2cmd/ide.conf.cmd | 23 +-
test/cfg2cmd/pinned-version-pxe-pve.conf.cmd | 6 +-
test/cfg2cmd/pinned-version-pxe.conf.cmd | 6 +-
test/cfg2cmd/pinned-version.conf.cmd | 6 +-
test/cfg2cmd/q35-ide.conf.cmd | 23 +-
.../q35-linux-hostpci-mapping.conf.cmd | 7 +-
.../q35-linux-hostpci-multifunction.conf.cmd | 7 +-
.../q35-linux-hostpci-template.conf.cmd | 10 +-
...q35-linux-hostpci-x-pci-overrides.conf.cmd | 7 +-
test/cfg2cmd/q35-linux-hostpci.conf.cmd | 7 +-
test/cfg2cmd/q35-simple-6.0.conf.cmd | 7 +-
test/cfg2cmd/q35-simple-7.0.conf.cmd | 7 +-
test/cfg2cmd/q35-simple-pinned-6.1.conf.cmd | 7 +-
test/cfg2cmd/q35-simple.conf.cmd | 7 +-
test/cfg2cmd/q35-win10-hostpci.conf.cmd | 7 +-
test/cfg2cmd/seabios_serial.conf.cmd | 6 +-
test/cfg2cmd/sev-es.conf.cmd | 7 +-
test/cfg2cmd/sev-std.conf.cmd | 7 +-
...imple-balloon-free-page-reporting.conf.cmd | 6 +-
test/cfg2cmd/simple-btrfs.conf.cmd | 6 +-
test/cfg2cmd/simple-disk-passthrough.conf | 15 +
test/cfg2cmd/simple-disk-passthrough.conf.cmd | 40 +
test/cfg2cmd/simple-rbd.conf | 15 +
test/cfg2cmd/simple-rbd.conf.cmd | 38 +
test/cfg2cmd/simple-virtio-blk.conf.cmd | 6 +-
test/cfg2cmd/simple-zfs-over-iscsi.conf | 14 +
test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd | 35 +
test/cfg2cmd/simple1-template.conf.cmd | 11 +-
test/cfg2cmd/simple1.conf.cmd | 6 +-
test/run_config2command_tests.pl | 34 +
46 files changed, 1143 insertions(+), 513 deletions(-)
create mode 100644 test/cfg2cmd/simple-disk-passthrough.conf
create mode 100644 test/cfg2cmd/simple-disk-passthrough.conf.cmd
create mode 100644 test/cfg2cmd/simple-rbd.conf
create mode 100644 test/cfg2cmd/simple-rbd.conf.cmd
create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi.conf
create mode 100644 test/cfg2cmd/simple-zfs-over-iscsi.conf.cmd
Alexandre Derumier (1):
add block-commit-replaces option patch
...074-block-commit-add-replaces-option.patch | 137 ++++++++++++++++++
debian/patches/series | 1 +
2 files changed, 138 insertions(+)
create mode 100644 debian/patches/pve/0074-block-commit-add-replaces-option.patch
--
2.39.5
--===============3263708610982164954==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============3263708610982164954==--