From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	by lore.proxmox.com (Postfix) with ESMTPS id 29AFD1FF15E
	for <inbox@lore.proxmox.com>; Tue,  3 Jun 2025 09:56:05 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id 64CB810ABC;
	Tue,  3 Jun 2025 09:56:15 +0200 (CEST)
To: pve-devel@lists.proxmox.com
Date: Tue,  3 Jun 2025 09:55:40 +0200
In-Reply-To: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>
References: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>
MIME-Version: 1.0
Message-ID: <mailman.186.1748937373.395.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, 
 <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: [pve-devel] [PATCH pve-storage 2/9] common: add
 qemu_img_create_qcow2_backed
Content-Type: multipart/mixed; boundary="===============7667739750063995057=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>

--===============7667739750063995057==
Content-Type: message/rfc822
Content-Disposition: inline

Return-Path: <root@formationkvm1.odiso.net>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
	(No client certificate requested)
	by lists.proxmox.com (Postfix) with ESMTPS id 3CD35CB5A4
	for <pve-devel@lists.proxmox.com>; Tue,  3 Jun 2025 09:56:13 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id 14D0310A72
	for <pve-devel@lists.proxmox.com>; Tue,  3 Jun 2025 09:56:13 +0200 (CEST)
Received: from bastiontest.odiso.net (unknown [185.151.190.228])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits))
	(No client certificate requested)
	by firstgate.proxmox.com (Proxmox) with ESMTPS
	for <pve-devel@lists.proxmox.com>; Tue,  3 Jun 2025 09:56:11 +0200 (CEST)
Received: from formationkvm1.odiso.net (unknown [10.11.201.57])
	by bastiontest.odiso.net (Postfix) with ESMTP id 21205860F13;
	Tue,  3 Jun 2025 09:56:02 +0200 (CEST)
Received: by formationkvm1.odiso.net (Postfix, from userid 0)
	id 4FD1B110DBD3; Tue,  3 Jun 2025 09:56:01 +0200 (CEST)
From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-storage 2/9] common: add qemu_img_create_qcow2_backed
Date: Tue,  3 Jun 2025 09:55:40 +0200
Message-Id: <20250603075558.627850-6-alexandre.derumier@groupe-cyllene.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>
References: <20250603075558.627850-1-alexandre.derumier@groupe-cyllene.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results:  0
	AWL                     0.051 Adjusted score from AWL reputation of From: address
	BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
	DMARC_NONE                0.1 DMARC none policy
	HEADER_FROM_DIFFERENT_DOMAINS  0.001 From and EnvelopeFrom 2nd level mail domains are different
	KAM_DMARC_NONE           0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy
	KAM_DMARC_STATUS         0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
	KAM_LAZY_DOMAIN_SECURITY      1 Sending domain does not have any anti-forgery methods
	RDNS_NONE               0.793 Delivered to internal network by a host with no rDNS
	SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
	SPF_NONE                0.001 SPF: sender does not publish an SPF Record

and use it for plugin linked clone

This also enable extended_l2=on, as it's mandatory for backing file
preallocation.

Preallocation was missing previously, so it should increase performance
for linked clone now (around x5 in randwrite 4k)

cluster_size is set to 128k, as it reduce qcow2 overhead (reduce disk,
but also memory needed to cache metadatas)

l2_extended is not enabled yet on base image, but it could help too
to reduce overhead without impacting performance

bench on 100G qcow2 file:

fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --name=test
fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --name=test

base image:

randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 20215
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 22219
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20217
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 21742
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21599
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 22037

clone image with backing file:

randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 3912
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 21476
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20563
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 22265
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 18016
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21611

Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
---
 src/PVE/Storage/Common.pm | 27 +++++++++++++++++++++++++++
 src/PVE/Storage/Plugin.pm |  5 +----
 2 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 4e4ad30..8910988 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -25,6 +25,10 @@ my $RAW_PREALLOCATION = {
     full => 1,
 };
 
+my $QCOW2_CLUSTERS = {
+   backed => ['extended_l2=on','cluster_size=128k']
+};
+
 =pod
 
 =head1 NAME
@@ -179,4 +183,27 @@ sub qemu_img_create {
     run_command($cmd, errmsg => "unable to create image");
 }
 
+=pod
+
+=head3 qemu_img_create_qcow2_backed
+
+    qemu_img_create_qcow2_backed($scfg, $path, $backing_path, $backing_format)
+
+Create a new qemu qcow2 image C<$path> using an existing backing image C<$backing_path> with backing_format C<$backing_format>.
+
+=cut
+
+sub qemu_img_create_qcow2_backed {
+    my ($scfg, $path, $backing_path, $backing_format) = @_;
+
+    my $cmd = ['/usr/bin/qemu-img', 'create', '-F', $backing_format, '-b', $backing_path, '-f', 'qcow2', $path];
+
+    my $options = $QCOW2_CLUSTERS->{backed};
+
+    push @$options, preallocation_cmd_option($scfg, 'qcow2');
+    push @$cmd, '-o', join(',', @$options) if @$options > 0;
+
+    run_command($cmd, errmsg => "unable to create image");
+}
+
 1;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 90627b2..8bf4ed3 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -847,10 +847,7 @@ sub clone_image {
     eval {
 	local $CWD = $imagedir;
 
-	my $cmd = ['/usr/bin/qemu-img', 'create', '-b', "../$basevmid/$basename",
-		   '-F', $format, '-f', 'qcow2', $path];
-
-	run_command($cmd);
+	PVE::Storage::Common::qemu_img_create_qcow2_backed($scfg, $path, "../$basevmid/$basename", $format);
     };
     my $err = $@;
 
-- 
2.39.5



--===============7667739750063995057==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

--===============7667739750063995057==--