From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 8DBE01FF165
for <inbox@lore.proxmox.com>; Thu, 22 May 2025 15:53:21 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 9DF16368E2;
Thu, 22 May 2025 15:53:19 +0200 (CEST)
To: pve-devel@lists.proxmox.com
Date: Thu, 22 May 2025 15:53:04 +0200
In-Reply-To: <20250522135304.2513284-1-alexandre.derumier@groupe-cyllene.com>
References: <20250522135304.2513284-1-alexandre.derumier@groupe-cyllene.com>
MIME-Version: 1.0
Message-ID: <mailman.560.1747921998.394.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: [pve-devel] [PATCH pve-storage 2/2] common: qemu_img_create: add
backing_file support
Content-Type: multipart/mixed; boundary="===============6078949349019489224=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============6078949349019489224==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <root@formationkvm1.odiso.net>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id B8955D0721
for <pve-devel@lists.proxmox.com>; Thu, 22 May 2025 15:53:17 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id A2DA03677E
for <pve-devel@lists.proxmox.com>; Thu, 22 May 2025 15:53:17 +0200 (CEST)
Received: from bastiontest.odiso.net (unknown [185.151.190.228])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Thu, 22 May 2025 15:53:15 +0200 (CEST)
Received: from formationkvm1.odiso.net (unknown [10.11.201.57])
by bastiontest.odiso.net (Postfix) with ESMTP id 37E86860F0C;
Thu, 22 May 2025 15:53:09 +0200 (CEST)
Received: by formationkvm1.odiso.net (Postfix, from userid 0)
id C24A911F6B05; Thu, 22 May 2025 15:53:05 +0200 (CEST)
From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-storage 2/2] common: qemu_img_create: add backing_file support
Date: Thu, 22 May 2025 15:53:04 +0200
Message-Id: <20250522135304.2513284-3-alexandre.derumier@groupe-cyllene.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250522135304.2513284-1-alexandre.derumier@groupe-cyllene.com>
References: <20250522135304.2513284-1-alexandre.derumier@groupe-cyllene.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.061 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_NONE 0.1 DMARC none policy
HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different
KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information.
RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_NONE 0.001 SPF: sender does not publish an SPF Record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [common.pm,plugin.pm]
and use it for plugin linked clone
This also enable extended_l2=on, as it's mandatory for backing file
preallocation.
Preallocation was missing previously, so it should increase performance
for linked clone now (around x5 in randwrite 4k)
cluster_size is set to 128k, as it reduce qcow2 overhead (reduce disk,
but also memory needed to cache metadatas)
l2_extended is not enabled yet on base image, but it could help too
to reduce overhead without impacting performance
bench on 100G qcow2 file:
fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --name=test
fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --name=test
base image:
randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 20215
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 22219
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20217
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 21742
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21599
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 22037
clone image with backing file:
randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 3912
randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 21476
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20563
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 22265
randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 18016
randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21611
Signed-off-by: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com>
---
src/PVE/Storage/Common.pm | 17 +++++++++++++----
src/PVE/Storage/Plugin.pm | 5 +----
2 files changed, 14 insertions(+), 8 deletions(-)
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 0770d70..01a995a 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -148,14 +148,23 @@ sub preallocation_cmd_option {
}
sub qemu_img_create {
- my ($scfg, $fmt, $size, $path) = @_;
+ my ($scfg, $fmt, $size, $path, $backing_path) = @_;
+
+ die "size can't be specified if backing file is used" if $size && $backing_path;
my $cmd = ['/usr/bin/qemu-img', 'create'];
- my $prealloc_opt = preallocation_cmd_option($scfg, $fmt);
- push @$cmd, '-o', $prealloc_opt if defined($prealloc_opt);
+ my $options = [];
+
+ if($backing_path) {
+ push @$cmd, '-b', $backing_path, '-F', 'qcow2';
+ push @$options, 'extended_l2=on','cluster_size=128k';
+ };
- push @$cmd, '-f', $fmt, $path, "${size}K";
+ push @$options, preallocation_cmd_option($scfg, $fmt);
+ push @$cmd, '-o', join(',', @$options) if @$options > 0;
+ push @$cmd, '-f', $fmt, $path;
+ push @$cmd, "${size}K" if $size;
run_command($cmd, errmsg => "unable to create image");
}
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 7030e4e..a3856ca 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -847,10 +847,7 @@ sub clone_image {
eval {
local $CWD = $imagedir;
- my $cmd = ['/usr/bin/qemu-img', 'create', '-b', "../$basevmid/$basename",
- '-F', $format, '-f', 'qcow2', $path];
-
- run_command($cmd);
+ PVE::Storage::Common::qemu_img_create($scfg, $format, undef, $path, "../$basevmid/$basename");
};
my $err = $@;
--
2.39.5
--===============6078949349019489224==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============6078949349019489224==--