From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 29AFD1FF15E for <inbox@lore.proxmox.com>; Tue, 3 Jun 2025 09:56:05 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 64CB810ABC; Tue, 3 Jun 2025 09:56:15 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Tue, 3 Jun 2025 09:55:40 +0200 In-Reply-To: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com> References: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Message-ID: <mailman.186.1748937373.395.pve-devel@lists.proxmox.com> List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Post: <mailto:pve-devel@lists.proxmox.com> From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com> Precedence: list Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com> X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> Subject: [pve-devel] [PATCH pve-storage 2/9] common: add qemu_img_create_qcow2_backed Content-Type: multipart/mixed; boundary="===============7667739750063995057==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> --===============7667739750063995057== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: <root@formationkvm1.odiso.net> X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 3CD35CB5A4 for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:13 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 14D0310A72 for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:13 +0200 (CEST) Received: from bastiontest.odiso.net (unknown [185.151.190.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-devel@lists.proxmox.com>; Tue, 3 Jun 2025 09:56:11 +0200 (CEST) Received: from formationkvm1.odiso.net (unknown [10.11.201.57]) by bastiontest.odiso.net (Postfix) with ESMTP id 21205860F13; Tue, 3 Jun 2025 09:56:02 +0200 (CEST) Received: by formationkvm1.odiso.net (Postfix, from userid 0) id 4FD1B110DBD3; Tue, 3 Jun 2025 09:56:01 +0200 (CEST) From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com> To: pve-devel@lists.proxmox.com Subject: [PATCH pve-storage 2/9] common: add qemu_img_create_qcow2_backed Date: Tue, 3 Jun 2025 09:55:40 +0200 Message-Id: <20250603075558.627850-6-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com> References: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.051 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_NONE 0.1 DMARC none policy HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record and use it for plugin linked clone This also enable extended_l2=on, as it's mandatory for backing file preallocation. Preallocation was missing previously, so it should increase performance for linked clone now (around x5 in randwrite 4k) cluster_size is set to 128k, as it reduce qcow2 overhead (reduce disk, but also memory needed to cache metadatas) l2_extended is not enabled yet on base image, but it could help too to reduce overhead without impacting performance bench on 100G qcow2 file: fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --name=test fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --name=test base image: randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 20215 randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 22219 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20217 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 21742 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21599 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 22037 clone image with backing file: randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 3912 randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 21476 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20563 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 22265 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 18016 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21611 Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com> --- src/PVE/Storage/Common.pm | 27 +++++++++++++++++++++++++++ src/PVE/Storage/Plugin.pm | 5 +---- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm index 4e4ad30..8910988 100644 --- a/src/PVE/Storage/Common.pm +++ b/src/PVE/Storage/Common.pm @@ -25,6 +25,10 @@ my $RAW_PREALLOCATION = { full => 1, }; +my $QCOW2_CLUSTERS = { + backed => ['extended_l2=on','cluster_size=128k'] +}; + =pod =head1 NAME @@ -179,4 +183,27 @@ sub qemu_img_create { run_command($cmd, errmsg => "unable to create image"); } +=pod + +=head3 qemu_img_create_qcow2_backed + + qemu_img_create_qcow2_backed($scfg, $path, $backing_path, $backing_format) + +Create a new qemu qcow2 image C<$path> using an existing backing image C<$backing_path> with backing_format C<$backing_format>. + +=cut + +sub qemu_img_create_qcow2_backed { + my ($scfg, $path, $backing_path, $backing_format) = @_; + + my $cmd = ['/usr/bin/qemu-img', 'create', '-F', $backing_format, '-b', $backing_path, '-f', 'qcow2', $path]; + + my $options = $QCOW2_CLUSTERS->{backed}; + + push @$options, preallocation_cmd_option($scfg, 'qcow2'); + push @$cmd, '-o', join(',', @$options) if @$options > 0; + + run_command($cmd, errmsg => "unable to create image"); +} + 1; diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index 90627b2..8bf4ed3 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -847,10 +847,7 @@ sub clone_image { eval { local $CWD = $imagedir; - my $cmd = ['/usr/bin/qemu-img', 'create', '-b', "../$basevmid/$basename", - '-F', $format, '-f', 'qcow2', $path]; - - run_command($cmd); + PVE::Storage::Common::qemu_img_create_qcow2_backed($scfg, $path, "../$basevmid/$basename", $format); }; my $err = $@; -- 2.39.5 --===============7667739750063995057== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============7667739750063995057==--