From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 51A501FF183 for ; Wed, 16 Jul 2025 08:31:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5179415BD0; Wed, 16 Jul 2025 08:32:10 +0200 (CEST) To: pve-devel@lists.proxmox.com Date: Wed, 16 Jul 2025 08:31:39 +0200 In-Reply-To: <20250716063153.1647681-1-alexandre.derumier@groupe-cyllene.com> References: <20250716063153.1647681-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Alexandre Derumier via pve-devel Precedence: list Cc: Alexandre Derumier X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: [pve-devel] [PATCH FOLLOW-UP storage 02/14] helpers: move qemu_img* to Common module Content-Type: multipart/mixed; boundary="===============1416311603378005569==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============1416311603378005569== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 988A3DACA7 for ; Wed, 16 Jul 2025 08:32:06 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 707FF15887 for ; Wed, 16 Jul 2025 08:32:06 +0200 (CEST) Received: from bastiontest.odiso.net (unknown [IPv6:2a0a:1580:2000:6700::14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 16 Jul 2025 08:32:03 +0200 (CEST) Received: from formationkvm1.odiso.net (unknown [10.11.201.57]) by bastiontest.odiso.net (Postfix) with ESMTP id 24F8F862E45; Wed, 16 Jul 2025 08:31:56 +0200 (CEST) Received: by formationkvm1.odiso.net (Postfix, from userid 0) id 20E4A11F6B1C; Wed, 16 Jul 2025 08:31:56 +0200 (CEST) From: Alexandre Derumier To: pve-devel@lists.proxmox.com Subject: [PATCH FOLLOW-UP storage 02/14] helpers: move qemu_img* to Common module Date: Wed, 16 Jul 2025 08:31:39 +0200 Message-Id: <20250716063153.1647681-5-alexandre.derumier@groupe-cyllene.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250716063153.1647681-1-alexandre.derumier@groupe-cyllene.com> References: <20250716063153.1647681-1-alexandre.derumier@groupe-cyllene.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 1 AWL -1.377 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_NONE 0.1 DMARC none policy HEADER_FROM_DIFFERENT_DOMAINS 0.001 From and EnvelopeFrom 2nd level mail domains are different KAM_DMARC_NONE 0.25 DKIM has Failed or SPF has failed on the message and the domain has no DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record URIBL_BLACK 3 Contains an URL listed in the URIBL blacklist [plugin.pm] From: Fabian Grünbichler Signed-off-by: Fabian Grünbichler Signed-off-by: Alexandre Derumier --- src/PVE/Storage/Common.pm | 147 ++++++++++++++++++++++++++++++--- src/PVE/Storage/LVMPlugin.pm | 10 +-- src/PVE/Storage/Plugin.pm | 153 +++-------------------------------- 3 files changed, 152 insertions(+), 158 deletions(-) diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm index aa89e68..71d123a 100644 --- a/src/PVE/Storage/Common.pm +++ b/src/PVE/Storage/Common.pm @@ -111,18 +111,7 @@ sub deallocate : prototype($$$) { } } -=pod - -=head3 run_qemu_img_json - - run_qemu_img_json($cmd, $timeout) - -Execute qemu_img command C<$cmd> with a timeout C<$timeout>. -Parse the output result and return a json. - -=cut - -sub run_qemu_img_json { +my sub run_qemu_img_json { my ($cmd, $timeout) = @_; my $json = ''; my $err_output = ''; @@ -143,4 +132,138 @@ sub run_qemu_img_json { } return $json; } + +=pod + +=head3 qemu_img_create + + qemu_img_create($fmt, $size, $path, $options) + +Create a new qemu image with a specific format C<$format> and size C<$size> for a target C<$path>. + +C<$options> currently allows setting the C value + +=cut + +sub qemu_img_create { + my ($fmt, $size, $path, $options) = @_; + + my $cmd = ['/usr/bin/qemu-img', 'create']; + + push @$cmd, '-o', "preallocation=$options->{preallocation}" + if defined($options->{preallocation}); + + push @$cmd, '-f', $fmt, $path, "${size}K"; + + run_command($cmd, errmsg => "unable to create image"); +} + +=pod + +=head3 qemu_img_create_qcow2_backed + + qemu_img_create_qcow2_backed($path, $backing_path, $backing_format, $options) + +Create a new qemu qcow2 image C<$path> using an existing backing image C<$backing_path> with backing_format C<$backing_format>. + +C<$options> currently allows setting the C value. + +=cut + +sub qemu_img_create_qcow2_backed { + my ($path, $backing_path, $backing_format, $options) = @_; + + my $cmd = [ + '/usr/bin/qemu-img', + 'create', + '-F', + $backing_format, + '-b', + $backing_path, + '-f', + 'qcow2', + $path, + ]; + + # TODO make this configurable for all volumes/types and pass in via $options + my $opts = ['extended_l2=on', 'cluster_size=128k']; + + push @$opts, "preallocation=$options->{preallocation}" + if defined($options->{preallocation}); + push @$cmd, '-o', join(',', @$opts) if @$opts > 0; + + run_command($cmd, errmsg => "unable to create image"); +} + +=pod + +=head3 qemu_img_info + + qemu_img_info($filename, $file_format, $timeout, $follow_backing_files) + +Returns a json with qemu image C<$filename> informations with format <$file_format>. +If C<$follow_backing_files> option is defined, return a json with the whole chain +of backing files images. + +=cut + +sub qemu_img_info { + my ($filename, $file_format, $timeout, $follow_backing_files) = @_; + + my $cmd = ['/usr/bin/qemu-img', 'info', '--output=json', $filename]; + push $cmd->@*, '-f', $file_format if $file_format; + push $cmd->@*, '--backing-chain' if $follow_backing_files; + + return run_qemu_img_json($cmd, $timeout); +} + +=pod + +=head3 qemu_img_measure + + qemu_img_measure($size, $fmt, $timeout, $options) + +Returns a json with the maximum size including all metadatas overhead for an image with format C<$fmt> and original size C<$size>Kb. + +C<$options> allows specifying qemu-img options that might affect the sizing calculation, such as cluster size. + +=cut + +sub qemu_img_measure { + my ($size, $fmt, $timeout, $options) = @_; + + die "format is missing" if !$fmt; + + my $cmd = ['/usr/bin/qemu-img', 'measure', '--output=json', '--size', "${size}K", '-O', $fmt]; + if ($options) { + push $cmd->@*, '-o', join(',', @$options) if @$options > 0; + } + return run_qemu_img_json($cmd, $timeout); +} + +=pod + +=head3 qemu_img_resize + + qemu_img_resize($scfg, $path, $format, $size, $preallocation, $timeout) + +Resize a qemu image C<$path> with format C<$format> to a target Kb size C<$size>. +Default timeout C<$timeout> is 10s if not specified. +C<$preallocation> allows to specify the preallocation option for the resize operation. + +=cut + +sub qemu_img_resize { + my ($scfg, $path, $format, $size, $preallocation, $timeout) = @_; + + die "format is missing" if !$format; + + my $cmd = ['/usr/bin/qemu-img', 'resize']; + push $cmd->@*, "--preallocation=$preallocation" if $preallocation; + push $cmd->@*, '-f', $format, $path, $size; + + $timeout = 10 if !$timeout; + run_command($cmd, timeout => $timeout); +} + 1; diff --git a/src/PVE/Storage/LVMPlugin.pm b/src/PVE/Storage/LVMPlugin.pm index f90df18..4b60e32 100644 --- a/src/PVE/Storage/LVMPlugin.pm +++ b/src/PVE/Storage/LVMPlugin.pm @@ -584,9 +584,9 @@ my sub lvm_qcow2_format { }; if ($backing_snap) { my $backing_path = $class->path($scfg, $name, $storeid, $backing_snap); - PVE::Storage::Plugin::qemu_img_create_qcow2_backed($path, $backing_path, $fmt, $options); + PVE::Storage::Common::qemu_img_create_qcow2_backed($path, $backing_path, $fmt, $options); } else { - PVE::Storage::Plugin::qemu_img_create($fmt, $size, $path, $options); + PVE::Storage::Common::qemu_img_create($fmt, $size, $path, $options); } } @@ -598,7 +598,7 @@ my sub calculate_lvm_size { my $options = $backing_snap ? ['extended_l2=on', 'cluster_size=128k'] : []; - my $json = PVE::Storage::Plugin::qemu_img_measure($size, $fmt, 5, $options); + my $json = PVE::Storage::Common::qemu_img_measure($size, $fmt, 5, $options); die "failed to query file information with qemu-img measure\n" if !$json; my $info = eval { decode_json($json) }; if ($@) { @@ -871,8 +871,8 @@ sub volume_resize { ); if (!$running && $format eq 'qcow2') { - my $preallocation = PVE::Storage::Plugin::preallocation_cmd_opt($scfg, $fmt); - PVE::Storage::Plugin::qemu_img_resize($path, $format, $size, $preallocation, 10); + my $preallocation = PVE::Storage::Plugin::preallocation_cmd_opt($scfg, $format); + PVE::Storage::Common::qemu_img_resize($path, $format, $size, $preallocation, 10); } return 1; diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm index 6eb91b7..aee145f 100644 --- a/src/PVE/Storage/Plugin.pm +++ b/src/PVE/Storage/Plugin.pm @@ -51,10 +51,6 @@ our $RAW_PREALLOCATION = { full => 1, }; -my $QCOW2_CLUSTERS = { - backed => ['extended_l2=on', 'cluster_size=128k'], -}; - our $MAX_VOLUMES_PER_GUEST = 1024; cfs_register_file( @@ -640,137 +636,6 @@ sub preallocation_cmd_opt { return; } -=pod - -=head3 qemu_img_create - - qemu_img_create($fmt, $size, $path, $options) - -Create a new qemu image with a specific format C<$format> and size C<$size> for a target C<$path>. - -C<$options> currently allows setting the C value - -=cut - -sub qemu_img_create { - my ($fmt, $size, $path, $options) = @_; - - my $cmd = ['/usr/bin/qemu-img', 'create']; - - push @$cmd, '-o', "preallocation=$options->{preallocation}" - if defined($options->{preallocation}); - - push @$cmd, '-f', $fmt, $path, "${size}K"; - - run_command($cmd, errmsg => "unable to create image"); -} - -=pod - -=head3 qemu_img_create_qcow2_backed - - qemu_img_create_qcow2_backed($path, $backing_path, $backing_format, $options) - -Create a new qemu qcow2 image C<$path> using an existing backing image C<$backing_path> with backing_format C<$backing_format>. - -C<$options> currently allows setting the C value. - -=cut - -sub qemu_img_create_qcow2_backed { - my ($path, $backing_path, $backing_format, $options) = @_; - - my $cmd = [ - '/usr/bin/qemu-img', - 'create', - '-F', - $backing_format, - '-b', - $backing_path, - '-f', - 'qcow2', - $path, - ]; - - my $opts = $QCOW2_CLUSTERS->{backed}; - - push @$opts, $options->{preallocation} if defined($options->{preallocation}); - push @$cmd, '-o', join(',', @$opts) if @$opts > 0; - - run_command($cmd, errmsg => "unable to create image"); -} - -=pod - -=head3 qemu_img_info - - qemu_img_info($filename, $file_format, $timeout, $follow_backing_files) - -Returns a json with qemu image C<$filename> informations with format <$file_format>. -If C<$follow_backing_files> option is defined, return a json with the whole chain -of backing files images. - -=cut - -sub qemu_img_info { - my ($filename, $file_format, $timeout, $follow_backing_files) = @_; - - my $cmd = ['/usr/bin/qemu-img', 'info', '--output=json', $filename]; - push $cmd->@*, '-f', $file_format if $file_format; - push $cmd->@*, '--backing-chain' if $follow_backing_files; - - return PVE::Storage::Common::run_qemu_img_json($cmd, $timeout); -} - -=pod - -=head3 qemu_img_measure - - qemu_img_measure($size, $fmt, $timeout, $options) - -Returns a json with the maximum size including all metadatas overhead for an image with format C<$fmt> and original size C<$size>Kb. - -C<$options> allows specifying qemu-img options that might affect the sizing calculation, such as cluster size. - -=cut - -sub qemu_img_measure { - my ($size, $fmt, $timeout, $options) = @_; - - die "format is missing" if !$fmt; - - my $cmd = ['/usr/bin/qemu-img', 'measure', '--output=json', '--size', "${size}K", '-O', $fmt]; - if ($options) { - push $cmd->@*, '-o', join(',', @$options) if @$options > 0; - } - return PVE::Storage::Common::run_qemu_img_json($cmd, $timeout); -} - -=pod - -=head3 qemu_img_resize - - qemu_img_resize($scfg, $path, $format, $size, $preallocation, $timeout) - -Resize a qemu image C<$path> with format C<$format> to a target Kb size C<$size>. -Default timeout C<$timeout> is 10s if not specified. -C<$preallocation> allows to specify the preallocation option for the resize operation. - -=cut - -sub qemu_img_resize { - my ($scfg, $path, $format, $size, $preallocation, $timeout) = @_; - - die "format is missing" if !$format; - - my $cmd = ['/usr/bin/qemu-img', 'resize']; - push $cmd->@*, "--preallocation=$preallocation" if $preallocation; - push $cmd->@*, '-f', $format, $path, $size; - - $timeout = 10 if !$timeout; - run_command($cmd, timeout => $timeout); -} - # Storage implementation # called during addition of storage (before the new storage config got written) @@ -1076,7 +941,9 @@ sub clone_image { my $options = { preallocation => preallocation_cmd_opt($scfg, $format), }; - qemu_img_create_qcow2_backed($path, "../$basevmid/$basename", $format, $options); + PVE::Storage::Common::qemu_img_create_qcow2_backed( + $path, "../$basevmid/$basename", $format, $options, + ); }; my $err = $@; @@ -1117,7 +984,7 @@ sub alloc_image { my $preallocation = preallocation_cmd_opt($scfg, $fmt); my $options = {}; $options->{preallocation} = $preallocation if $preallocation; - eval { qemu_img_create($fmt, $size, $path, $options) }; + eval { PVE::Storage::Common::qemu_img_create($fmt, $size, $path, $options) }; if ($@) { unlink $path; rmdir $imagedir; @@ -1139,7 +1006,11 @@ my sub alloc_backed_image { $options->{preallocation} = $preallocation if $preallocation; my $backing_volname = get_snap_name($class, $volname, $backing_snap); #qemu_img use relative path from base image for the backing_volname by default - eval { qemu_img_create_qcow2_backed($path, $backing_volname, $backing_format, $options) }; + eval { + PVE::Storage::Common::qemu_img_create_qcow2_backed( + $path, $backing_volname, $backing_format, $options, + ); + }; if ($@) { unlink $path; die "$@"; @@ -1271,7 +1142,7 @@ sub file_size_info { "file_size_info: '$filename': falling back to 'raw' from unknown format '$file_format'\n"; $file_format = 'raw'; } - my $json = qemu_img_info($filename, $file_format, $timeout); + my $json = PVE::Storage::Common::qemu_img_info($filename, $file_format, $timeout); if (!$json) { die "failed to query file information with qemu-img\n" if $untrusted; # skip decoding if there was no output, e.g. if there was a timeout. @@ -1387,7 +1258,7 @@ sub volume_resize { my $format = ($class->parse_volname($volname))[6]; my $preallocation = preallocation_cmd_opt($scfg, $format); - qemu_img_resize($path, $format, $size, $preallocation, 10); + PVE::Storage::Common::qemu_img_resize($path, $format, $size, $preallocation, 10); return undef; } @@ -1879,7 +1750,7 @@ sub volume_snapshot_info { my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) = $class->parse_volname($volname); - my $json = qemu_img_info($path, undef, 10, 1); + my $json = PVE::Storage::Common::qemu_img_info($path, undef, 10, 1); die "failed to query file information with qemu-img\n" if !$json; my $json_decode = eval { decode_json($json) }; if ($@) { -- 2.39.5 --===============1416311603378005569== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============1416311603378005569==--