[pve-devel] [PATCH-SERIES v2 qemu-server/storage/guest-common 0/7] slightly improve image format handling

[pve-devel] [PATCH-SERIES v2 qemu-server/storage/guest-common 0/7] slightly improve image format handling

[Fiona Ebner]

v1: https://lore.proxmox.com/pve-devel/20250207125514.42668-1-f.ebner@proxmox.com/

Changes in v2:
* different approach, use existing format
* introduce standard option
* add patches to drop mythological 'cow' format

Allow using 'vmdk' for the 'format' option when allocating an image
with 'pvesm' or storage API.

Drop the ancient 'cow' format from supported qm image formats that was
dropped in QEMU 2.2.

Introduce a standard option for supported image formats on storages
superseding the existing 'pve-storage-format' format.


Dependency bump guest-common -> storage needed for the new standard
option.


qemu-server:

Fiona Ebner (1):
  drive: remove ancient 'cow' from formats

 PVE/QemuServer/Drive.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


storage:

Fiona Ebner (5):
  plugin: file size info: remove ancient 'cow' from formats
  api: code cleanup: remove unused Data::Dumper include
  schema: anchor regex for 'pve-storage-format'
  api: alloc: allow 'vmdk' for the 'format' option
  introduce 'pve-storage-image-format' standard option for image format

 src/PVE/API2/Storage/Content.pm |  8 +++-----
 src/PVE/Storage/Common.pm       | 27 +++++++++++++++++++++++++++
 src/PVE/Storage/Plugin.pm       | 13 ++++++++-----
 3 files changed, 38 insertions(+), 10 deletions(-)


guest-common:

Fiona Ebner (1):
  storage tunnel: use pve-storage-image-format standard option

 src/PVE/StorageTunnel.pm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)


Summary over all repositories:
  5 files changed, 44 insertions(+), 16 deletions(-)

-- 
Generated by git-murpp 0.5.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 qemu-server 1/7] drive: remove ancient 'cow' from formats

[Fiona Ebner]

The format was dropped in QEMU binary version 2.2 with commit
550830f935 ("block: delete cow block driver").

Very old backups might still include this format as a hint (the data
in the backup is present in raw/chunk format in any case), but that is
not an issue. Restore already checks that the target storage supports
a given format and defaults to the default format of the storage if
the hint does not apply.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 PVE/QemuServer/Drive.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/PVE/QemuServer/Drive.pm b/PVE/QemuServer/Drive.pm
index 1041c1dd..81e1aa04 100644
--- a/PVE/QemuServer/Drive.pm
+++ b/PVE/QemuServer/Drive.pm
@@ -26,11 +26,11 @@ parse_drive
 print_drive
 );
 
-our $QEMU_FORMAT_RE = qr/raw|cow|qcow|qcow2|qed|vmdk|cloop/;
+our $QEMU_FORMAT_RE = qr/raw|qcow|qcow2|qed|vmdk|cloop/;
 
 PVE::JSONSchema::register_standard_option('pve-qm-image-format', {
     type => 'string',
-    enum => [qw(raw cow qcow qed qcow2 vmdk cloop)],
+    enum => [qw(raw qcow qed qcow2 vmdk cloop)],
     description => "The drive's backing file's data format.",
     optional => 1,
 });
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 storage 2/7] plugin: file size info: remove ancient 'cow' from formats

[Fiona Ebner]

The format was dropped in QEMU binary version 2.2 with commit
550830f935 ("block: delete cow block driver").

This follows qemu-server commit "drive: remove ancient 'cow' from
formats".

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 65cf43f..79f6f08 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -953,7 +953,7 @@ sub free_image {
 }
 
 # TODO taken from PVE/QemuServer/Drive.pm, avoiding duplication would be nice
-my @checked_qemu_img_formats = qw(raw cow qcow qcow2 qed vmdk cloop);
+my @checked_qemu_img_formats = qw(raw qcow qcow2 qed vmdk cloop);
 
 # set $untrusted if the file in question might be malicious since it isn't
 # created by our stack
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 storage 3/7] api: code cleanup: remove unused Data::Dumper include

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/API2/Storage/Content.pm | 1 -
 1 file changed, 1 deletion(-)

diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index fe0ad4a..3b73e90 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -2,7 +2,6 @@ package PVE::API2::Storage::Content;
 
 use strict;
 use warnings;
-use Data::Dumper;
 
 use PVE::SafeSyslog;
 use PVE::Cluster;
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 storage 4/7] schema: anchor regex for 'pve-storage-format'

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 79f6f08..dfdab16 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -346,7 +346,7 @@ PVE::JSONSchema::register_format('pve-storage-format', \&verify_format);
 sub verify_format {
     my ($fmt, $noerr) = @_;
 
-    if ($fmt !~ m/(raw|qcow2|vmdk|subvol)/) {
+    if ($fmt !~ m/^(raw|qcow2|vmdk|subvol)$/) {
 	return undef if $noerr;
 	die "invalid format '$fmt'\n";
     }
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 storage 5/7] api: alloc: allow 'vmdk' for the 'format' option

[Fiona Ebner]

The API endpoint will automatically detect the format from the
extension for raw, qcow2 and vmdk, but it was not yet possible to
specify the format explicitly via the parameter. This could be
annoying/surprising to users. There also might be third-party plugins
that want to use vmdk, but not require a suffix in the name. Add
'vmdk' as an allowed format to avoid these issues and for consistency
by using the 'pve-storage-format' format.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---

Changes in v2:
* use existing format

 src/PVE/API2/Storage/Content.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 3b73e90..03cc4a1 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -6,6 +6,7 @@ use warnings;
 use PVE::SafeSyslog;
 use PVE::Cluster;
 use PVE::Storage;
+use PVE::Storage::Plugin; # for 'pve-storage-format' format
 use PVE::INotify;
 use PVE::Exception qw(raise_param_exc);
 use PVE::RPCEnvironment;
@@ -179,7 +180,7 @@ __PACKAGE__->register_method ({
 	    },
 	    'format' => {
 		type => 'string',
-		enum => ['raw', 'qcow2', 'subvol'],
+		format => 'pve-storage-format',
 		requires => 'size',
 		optional => 1,
 	    },
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 storage 6/7] introduce 'pve-storage-image-format' standard option for image format

[Fiona Ebner]

The new 'pve-storage-image-format' standard option uses a simple enum
instead of a subroutine verifier. Since the 'pve-storage-format'
format that is replaced by it was used in pve-guest-common's
StorageTunnel, the format cannot be removed without a versioned
breaks.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/API2/Storage/Content.pm |  8 +++-----
 src/PVE/Storage/Common.pm       | 27 +++++++++++++++++++++++++++
 src/PVE/Storage/Plugin.pm       |  9 ++++++---
 3 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 03cc4a1..f84dbb7 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -6,7 +6,7 @@ use warnings;
 use PVE::SafeSyslog;
 use PVE::Cluster;
 use PVE::Storage;
-use PVE::Storage::Plugin; # for 'pve-storage-format' format
+use PVE::Storage::Common; # for 'pve-storage-image-format' standard option
 use PVE::INotify;
 use PVE::Exception qw(raise_param_exc);
 use PVE::RPCEnvironment;
@@ -178,12 +178,10 @@ __PACKAGE__->register_method ({
 		type => 'string',
 		pattern => '\d+[MG]?',
 	    },
-	    'format' => {
-		type => 'string',
-		format => 'pve-storage-format',
+	    format => get_standard_option('pve-storage-image-format', {
 		requires => 'size',
 		optional => 1,
-	    },
+	    }),
 	},
     },
     returns => {
diff --git a/src/PVE/Storage/Common.pm b/src/PVE/Storage/Common.pm
index 3ae20dd..1f81234 100644
--- a/src/PVE/Storage/Common.pm
+++ b/src/PVE/Storage/Common.pm
@@ -3,6 +3,8 @@ package PVE::Storage::Common;
 use strict;
 use warnings;
 
+use PVE::JSONSchema;
+
 =pod
 
 =head1 NAME
@@ -25,6 +27,31 @@ be grouped in a submodule can also be found here.
 
 =back
 
+=head1 STANDARD OPTIONS FOR JSON SCHEMA
+
+=over
+
+=back
+
+=head3 pve-storage-image-format
+
+Possible formats a guest image can have.
+
+=cut
+
+# TODO PVE 9 - Note that currently, qemu-server allows more formats for VM images, so third party
+# storage plugins might potentially allow more too, but none of the plugins we are aware of do that.
+# Those formats should either be allowed here or support for them should be phased out (at least in
+# the storage layer). Can still be added again in the future, should any plugin provider request it.
+
+PVE::JSONSchema::register_standard_option('pve-storage-image-format', {
+    type => 'string',
+    enum => ['raw', 'qcow2', 'subvol', 'vmdk'],
+    description => "Format of the image.",
+});
+
+=pod
+
 =head1 FUNCTIONS
 
 =cut
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index dfdab16..6e23fec 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -184,11 +184,10 @@ my $defaultData = {
 	    type => 'string', format => 'pve-storage-path',
 	    optional => 1,
 	},
-	'format' => {
+	format => get_standard_option('pve-storage-image-format', {
 	    description => "Default image format.",
-	    type => 'string', format => 'pve-storage-format',
 	    optional => 1,
-	},
+	}),
 	preallocation => {
 	    description => "Preallocation mode for raw and qcow2 images. " .
 		"Using 'metadata' on raw images results in preallocation=off.",
@@ -342,6 +341,10 @@ sub verify_content {
     return $ct;
 }
 
+# NOTE the 'pve-storage-format' is deprecated, use the 'pve-storage-image-format' standard option
+# from Storage/Common.pm instead
+# TODO PVE 9 - remove after doing a versioned breaks for pve-guest-common, which was using this
+# format.
 PVE::JSONSchema::register_format('pve-storage-format', \&verify_format);
 sub verify_format {
     my ($fmt, $noerr) = @_;
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v2 guest-common 7/7] storage tunnel: use pve-storage-image-format standard option

[Fiona Ebner]

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
 src/PVE/StorageTunnel.pm | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/PVE/StorageTunnel.pm b/src/PVE/StorageTunnel.pm
index c880889..c0cc0ae 100644
--- a/src/PVE/StorageTunnel.pm
+++ b/src/PVE/StorageTunnel.pm
@@ -7,7 +7,9 @@ use IO::Socket::UNIX;
 use POSIX qw(WNOHANG);
 use Socket qw(SOCK_STREAM);
 
+use PVE::JSONSchema;
 use PVE::Storage;
+use PVE::Storage::Common; # for pve-storage-image-format standard option
 use PVE::Tools;
 use PVE::Tunnel;
 
@@ -153,11 +155,9 @@ our $cmd_schema = {
 	    type => 'string',
 	    description => 'volume name to use as preferred target volume name',
 	},
-	format => {
-	    type => 'string',
-	    format => 'pve-storage-format',
+	format => PVE::JSONSchema::get_standard_option('pve-storage-image-format', {
 	    description => 'preferred format for the target volume',
-	},
+	}),
 	export_formats => {
 	    type => 'string',
 	    description => 'list of supported export formats',
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel