From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 4C30E1FF15F for ; Mon, 16 Dec 2024 10:13:37 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9CB5A7398; Mon, 16 Dec 2024 10:13:17 +0100 (CET) To: pve-devel@lists.proxmox.com Date: Mon, 16 Dec 2024 10:12:14 +0100 MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Alexandre Derumier via pve-devel Precedence: list Cc: Alexandre Derumier X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] [PATCH-SERIES v3 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support Content-Type: multipart/mixed; boundary="===============2588629413616728112==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============2588629413616728112== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id CDE55C30EA for ; Mon, 16 Dec 2024 10:13:15 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0987A6CF0 for ; Mon, 16 Dec 2024 10:12:44 +0100 (CET) Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 16 Dec 2024 10:12:39 +0100 (CET) Received: from formationkvm1.odiso.net (unknown [10.11.201.57]) by bastiontest.odiso.net (Postfix) with ESMTP id 1579685A3FF; Mon, 16 Dec 2024 10:12:32 +0100 (CET) Received: by formationkvm1.odiso.net (Postfix, from userid 0) id 480DF1084C68; Mon, 16 Dec 2024 10:12:31 +0100 (CET) From: Alexandre Derumier To: pve-devel@lists.proxmox.com Subject: [PATCH-SERIES v3 pve-storage/qemu-server/pve-qemu] add external qcow2 snapshot support Date: Mon, 16 Dec 2024 10:12:14 +0100 Message-Id: <20241216091229.3142660-1-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.038 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_NONE 0.1 DMARC none policy HEADER_FROM_DIFFERENT_DOMAINS 0.248 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record This patch series implement qcow2 external snapshot support for files && lvm volumes The current internal qcow2 snapshots have bad write performance because no metadatas can be preallocated. This is particulary visible on a shared filesystem like ocfs2 or gfs2. Also other bugs are freeze/lock reported by users since years on snapshots delete on nfs (The disk access seem to be frozen during all the delete duration) This also open doors for remote snapshot export-import for storage replication. Changelog v3: storage: - snapshots files now have the name of the snapshot, and "current" snapshot is the vm volname - allow only qcow2 format for base image for simplication - merge snapshot code in lvmplugin (qcow2 format auto enable snapshot) - the code is a lot more simple now qemu-server: - convertion -drive to modern -blockdev This is needed for blockdev-reopen, where we need to create && switch to snapshot files with same cache,aio,... This is also needed to live rename snapshot files (to keep volname for current snapshot) I have implemented && tested: - disk create,delete,resize,convert,drive_mirror, hotplug,unplug,nbd mirror,cdrom insert/eject - block protocol: file,block_device,rbd,nbd,glusterfs Note that it's currently incomplete: - proxmox backup/restore code need to be converted to blockdev, help is needed - iscsi:// path is not yet implemented (I'll look for the v4) - efi still in drive format (not blocking, but I'll look for v4) The live migration between -drive ---> --blockdev seem to work without breaking, and seem to be transparent for the guest ok, so I think we could try to target pve9 ? storage.cfg example: dir: local2 path /var/liv/vz content snippets,vztmpl,backup,images,iso,rootdir snapext 1 lvm:test vgname test content images pve-storage: Alexandre Derumier (3): qcow2: add external snapshot support lvmplugin: add qcow2 snapshot storage: vdisk_free: remove external snapshots src/PVE/Storage.pm | 18 ++- src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/LVMPlugin.pm | 231 ++++++++++++++++++++++++++--- src/PVE/Storage/Plugin.pm | 207 ++++++++++++++++++++++---- src/test/run_test_zfspoolplugin.pl | 18 +++ 5 files changed, 424 insertions(+), 51 deletions(-) Alexandre Derumier (11): blockdev: cmdline: convert drive to blockdev syntax blockdev: fix cfg2cmd tests blockdev : convert qemu_driveadd && qemu_drivedel blockdev: vm_devices_list : fix block-query blockdev: convert cdrom media eject/insert blockdev: block_resize: convert to blockdev blockdev: nbd_export: block-export-add : use drive-$id for nodename blockdev: convert drive_mirror to blockdev_mirror blockdev: mirror: change aio on target if io_uring is not default. blockdev: add backing_chain support qcow2: add external snapshot support PVE/QemuConfig.pm | 4 +- PVE/QemuMigrate.pm | 2 +- PVE/QemuServer.pm | 946 ++++++++++++++---- test/cfg2cmd/bootorder-empty.conf.cmd | 12 +- test/cfg2cmd/bootorder-legacy.conf.cmd | 12 +- test/cfg2cmd/bootorder.conf.cmd | 12 +- ...putype-icelake-client-deprecation.conf.cmd | 6 +- test/cfg2cmd/ide.conf.cmd | 23 +- test/cfg2cmd/pinned-version-pxe-pve.conf.cmd | 6 +- test/cfg2cmd/pinned-version-pxe.conf.cmd | 6 +- test/cfg2cmd/pinned-version.conf.cmd | 6 +- test/cfg2cmd/q35-ide.conf.cmd | 23 +- .../q35-linux-hostpci-template.conf.cmd | 3 +- test/cfg2cmd/seabios_serial.conf.cmd | 6 +- ...imple-balloon-free-page-reporting.conf.cmd | 6 +- test/cfg2cmd/simple-btrfs.conf.cmd | 6 +- test/cfg2cmd/simple-virtio-blk.conf.cmd | 6 +- test/cfg2cmd/simple1-template.conf.cmd | 11 +- test/cfg2cmd/simple1.conf.cmd | 6 +- 19 files changed, 830 insertions(+), 272 deletions(-) pve-qemu: add block-commit-replaces option patch ...052-block-commit-add-replaces-option.patch | 137 ++++++++++++++++++ debian/patches/series | 1 + 2 files changed, 138 insertions(+) create mode 100644 debian/patches/pve/0052-block-commit-add-replaces-option.patch -- 2.39.5 --===============2588629413616728112== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============2588629413616728112==--