From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pve-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id AED901FF183 for <inbox@lore.proxmox.com>; Mon, 19 May 2025 12:23:54 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3A32A54A7; Mon, 19 May 2025 12:23:55 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Mon, 19 May 2025 12:23:08 +0200 MIME-Version: 1.0 Message-ID: <mailman.456.1747650234.394.pve-devel@lists.proxmox.com> List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Post: <mailto:pve-devel@lists.proxmox.com> From: Alexandre Derumier via pve-devel <pve-devel@lists.proxmox.com> Precedence: list Cc: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com> X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> Subject: [pve-devel] [PATCH pve-storage 0/2] move qemu_img_create to common helpers and enable preallocation on backed images Content-Type: multipart/mixed; boundary="===============0549636375653736657==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com> --===============0549636375653736657== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: <root@formationkvm1.odiso.net> X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F0C6DCF500 for <pve-devel@lists.proxmox.com>; Mon, 19 May 2025 12:23:53 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D2A4D5324 for <pve-devel@lists.proxmox.com>; Mon, 19 May 2025 12:23:23 +0200 (CEST) Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pve-devel@lists.proxmox.com>; Mon, 19 May 2025 12:23:23 +0200 (CEST) Received: from formationkvm1.odiso.net (unknown [10.11.201.57]) by bastiontest.odiso.net (Postfix) with ESMTP id 4DE9D860944; Mon, 19 May 2025 12:23:16 +0200 (CEST) Received: by formationkvm1.odiso.net (Postfix, from userid 0) id 114F11063E48; Mon, 19 May 2025 12:23:13 +0200 (CEST) From: Alexandre Derumier <alexandre.derumier@groupe-cyllene.com> To: pve-devel@lists.proxmox.com Subject: [PATCH pve-storage 0/2] move qemu_img_create to common helpers and enable preallocation on backed images Date: Mon, 19 May 2025 12:23:08 +0200 Message-Id: <20250519102310.911326-1-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.100 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_NONE 0.1 DMARC none policy HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [plugin.pm,glusterfsplugin.pm,common.pm] This is part of my work on qcow2 external snapshot, but could improve current qcow2 linked clone This patch serie move qemu_img_create to common helpers, and enable preallocation on backed_image to increase performance This require l2_extended=on on the backed image I have done some benchmarks on localssd with 100gb qcow2, the performance in randwrite 4k is 5x faster some presentation of l2_extended=on are available here https://www.youtube.com/watch?v=zJetcfDVFNw https://www.youtube.com/watch?v=NfgLCdtkRus I don't have enabled it for base image, as I think that Fabian see performance regression some month ago. but I don't see performance difference in my bench. (can you could test on your side again ?) It could help to reduce qcow2 overhead on disk, and allow to keep more metadatas in memory for bigger image, as qemu default memory l2_cache_size=1MB) https://www.ibm.com/products/tutorials/how-to-tune-qemu-l2-cache-size-and-qcow2-cluster-size Maybe more test with bigger image (>1TB) could be done too to see if it's help bench on 100G qcow2 file: fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --ioengine=libaio --name=test fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --ioengine=libaio --name=test base image: randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 20215 randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 22219 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20217 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 21742 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21599 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 22037 linked clone image with backing file: randwrite 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 3912 randread 4k: prealloc=metadata, l2_extended=off, cluster_size=64k: 21476 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 20563 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=64k: 22265 randwrite 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 18016 randread 4k: prealloc=metadata, l2_extended=on, cluster_size=128k: 21611 Alexandre Derumier (2): common: add qemu_img_create an preallocation_cmd_option common: qemu_img_create: add backing_file support src/PVE/Storage/Common.pm | 62 ++++++++++++++++++++++++++++++ src/PVE/Storage/GlusterfsPlugin.pm | 2 +- src/PVE/Storage/Plugin.pm | 52 +------------------------ 3 files changed, 65 insertions(+), 51 deletions(-) -- 2.39.5 --===============0549636375653736657== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============0549636375653736657==--