[pve-devel] [PATCH storage/qemu-server/manager v2 0/4] Restore with a specified file format

[pve-devel] [PATCH storage/qemu-server/manager v2 0/4] Restore with a specified file format

[Markus Frank]

This patch series allows to restore all VM disks with a specified format
if supported by the target storage. The existing storage and the new
disk-format options can act as a default/fallback for per disk
storage/format customisation in the future (#4275).

v2:
* renamed diskformat to disk-format
* added storage patch to a new standard option that only allows raw,
 qcow2 and vmdk

storage:

Markus Frank (1):
  add standard option for VM disk formats in file-based storages

 src/PVE/Storage/Plugin.pm | 7 +++++++
 1 file changed, 7 insertions(+)


qemu-server:

Markus Frank (1):
  fix 4888: qmrestore: add disk-format option

 PVE/API2/Qemu.pm     |  6 ++++++
 PVE/CLI/qmrestore.pm |  4 ++++
 PVE/QemuServer.pm    | 10 +++++++---
 3 files changed, 17 insertions(+), 3 deletions(-)


manager:

Markus Frank (2):
  ui: form: add hideFormatWhenStorageEmpty option to DiskStorageSelector
  ui: window: add disk-format option to the restore window

 www/manager6/form/DiskStorageSelector.js |  9 +++++++++
 www/manager6/window/Restore.js           | 16 +++++++++++-----
 2 files changed, 20 insertions(+), 5 deletions(-)

-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH storage v2 1/4] add standard option for VM disk formats in file-based storages

[Markus Frank]

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 src/PVE/Storage/Plugin.pm | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 65cf43f..9f21f0b 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -354,6 +354,13 @@ sub verify_format {
     return $fmt;
 }
 
+PVE::JSONSchema::register_standard_option('pve-storage-qm-image-format', {
+    type => 'string',
+    enum => ['raw', 'qcow2', 'vmdk'],
+    description => "Supported VM disk formats in a file-based storage.",
+    optional => 1,
+});
+
 PVE::JSONSchema::register_format('pve-storage-options', \&verify_options);
 sub verify_options {
     my ($value, $noerr) = @_;
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH qemu-server v2 2/4] fix 4888: qmrestore: add disk-format option

[Markus Frank]

Add an option to choose a file format (qcow2, raw, vmdk) when restoring
a vm backup to file based storage. This options allows all disks to be
recreated with the specified file format if supported by the target
storage.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 PVE/API2/Qemu.pm     |  6 ++++++
 PVE/CLI/qmrestore.pm |  4 ++++
 PVE/QemuServer.pm    | 10 +++++++---
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index 295260e7..617dbe45 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -1013,6 +1013,10 @@ __PACKAGE__->register_method({
 		    default => 0,
 		    description => "Start VM after it was created successfully.",
 		},
+		'disk-format' => get_standard_option('pve-storage-qm-image-format', {
+		    optional => 1,
+		    description => "Image format used for all VM disks when restoring.",
+		}),
 		'import-working-storage' => get_standard_option('pve-storage-id', {
 		    description => "A file-based storage with 'images' content-type enabled, which"
 			." is used as an intermediary extraction storage during import. Defaults to"
@@ -1046,6 +1050,7 @@ __PACKAGE__->register_method({
 	my $storage = extract_param($param, 'storage');
 	my $unique = extract_param($param, 'unique');
 	my $live_restore = extract_param($param, 'live-restore');
+	my $disk_format = extract_param($param, 'disk-format');
 	my $extraction_storage = extract_param($param, 'import-working-storage');
 
 	if (defined(my $ssh_keys = $param->{sshkeys})) {
@@ -1143,6 +1148,7 @@ __PACKAGE__->register_method({
 		    unique => $unique,
 		    bwlimit => $bwlimit,
 		    live => $live_restore,
+		    disk_format => $disk_format,
 		    override_conf => $param,
 		};
 		if (my $volid = $archive->{volid}) {
diff --git a/PVE/CLI/qmrestore.pm b/PVE/CLI/qmrestore.pm
index a47648bd..68c23db5 100755
--- a/PVE/CLI/qmrestore.pm
+++ b/PVE/CLI/qmrestore.pm
@@ -64,6 +64,10 @@ __PACKAGE__->register_method({
 		type => 'boolean',
 		description => "Start the VM immediately from the backup and restore in background. PBS only.",
 	    },
+	    'disk-format' => get_standard_option('pve-storage-qm-image-format', {
+		optional => 1,
+		description => "Restore all VM disks with the specified image format.",
+	    }),
 	},
     },
     returns => {
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 808c0e1c..63836399 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -6660,7 +6660,7 @@ my $parse_backup_hints = sub {
 #
 # Returns: { $virtdev => $volid }
 my $restore_allocate_devices = sub {
-    my ($storecfg, $virtdev_hash, $vmid) = @_;
+    my ($storecfg, $virtdev_hash, $vmid, $disk_format) = @_;
 
     my $map = {};
     foreach my $virtdev (sort keys %$virtdev_hash) {
@@ -6670,6 +6670,10 @@ my $restore_allocate_devices = sub {
 	my $storeid = $d->{storeid};
 	my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
 
+	if ($disk_format) {
+	    $d->{format} = $disk_format;
+	}
+
 	# test if requested format is supported
 	my ($defFormat, $validFormats) = PVE::Storage::storage_default_format($storecfg, $storeid);
 	my $supported = grep { $_ eq $d->{format} } @$validFormats;
@@ -7047,7 +7051,7 @@ sub restore_proxmox_backup_archive {
 	$restore_cleanup_oldconf->($storecfg, $vmid, $oldconf, $virtdev_hash) if $oldconf;
 
 	# allocate volumes
-	my $map = $restore_allocate_devices->($storecfg, $virtdev_hash, $vmid);
+	my $map = $restore_allocate_devices->($storecfg, $virtdev_hash, $vmid, $options->{disk_format});
 
 	foreach my $virtdev (sort keys %$virtdev_hash) {
 	    my $d = $virtdev_hash->{$virtdev};
@@ -7444,7 +7448,7 @@ sub restore_vma_archive {
 	}
 
 	# allocate volumes
-	my $map = $restore_allocate_devices->($cfg, $virtdev_hash, $vmid);
+	my $map = $restore_allocate_devices->($cfg, $virtdev_hash, $vmid, $opts->{disk_format});
 
 	# print restore information to $fifofh
 	foreach my $virtdev (sort keys %$virtdev_hash) {
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager v2 3/4] ui: form: add hideFormatWhenStorageEmpty option to DiskStorageSelector

[Markus Frank]

Prerequisite for "ui: window: add diskformat option to restore window"

The hide condition is copied from the format selector item in the same
file.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 www/manager6/form/DiskStorageSelector.js | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/www/manager6/form/DiskStorageSelector.js b/www/manager6/form/DiskStorageSelector.js
index 0ef48f51..e2064934 100644
--- a/www/manager6/form/DiskStorageSelector.js
+++ b/www/manager6/form/DiskStorageSelector.js
@@ -30,6 +30,7 @@ Ext.define('PVE.form.DiskStorageSelector', {
 
     // hides the format field (e.g. for TPM state)
     hideFormat: false,
+    hideFormatWhenStorageEmpty: false,
 
     // sets the initial size value
     // string because else we get a type confusion
@@ -41,12 +42,20 @@ Ext.define('PVE.form.DiskStorageSelector', {
 	var hdfilesel = me.getComponent('hdimage');
 	var hdsizesel = me.getComponent('disksize');
 
+	// This is needed to make the format selector visible
+	// after it has been hidden because of hideFormatWhenStorageEmpty.
+	let hideFormatCondition = me.hideFormat || me.storageContent === 'rootdir';
+	formatsel.setVisible(!hideFormatCondition);
+
 	// initial store load, and reset/deletion of the storage
 	if (!value) {
 	    hdfilesel.setDisabled(true);
 	    hdfilesel.setVisible(false);
 
 	    formatsel.setDisabled(true);
+	    if (me.hideFormatWhenStorageEmpty) {
+		formatsel.setVisible(false);
+	    }
 	    return;
 	}
 
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager v2 4/4] ui: window: add disk-format option to the restore window

[Markus Frank]

This is done by changing the StorageSelector to a DiskStorageSelector.

Using the hideFormatWhenStorageEmpty option of the DiskStorageSelector
to hide the DiskFormatSelector when no storage is selected, as the
DiskFormatSelector would show the default value qcow2 in a disabled
state, which could confuse users.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 www/manager6/window/Restore.js | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/www/manager6/window/Restore.js b/www/manager6/window/Restore.js
index 6efe1313..9600cdab 100644
--- a/www/manager6/window/Restore.js
+++ b/www/manager6/window/Restore.js
@@ -44,8 +44,11 @@ Ext.define('PVE.window.Restore', {
 	    if (values['live-restore']) {
 		params['live-restore'] = 1;
 	    }
-	    if (values.storage) {
-		params.storage = values.storage;
+	    if (values.hdstorage) {
+		params.storage = values.hdstorage;
+	    }
+	    if (values.diskformat) {
+		params['disk-format'] = values.diskformat;
 	    }
 
 	    ['bwlimit', 'cores', 'name', 'memory', 'sockets'].forEach(opt => {
@@ -141,9 +144,10 @@ Ext.define('PVE.window.Restore', {
 			    view.lookupReference(`${key}Field`).setEmptyText(value);
 			}
 		    });
-
+		    let diskformat = view.down('pveDiskFormatSelector[name=diskformat]');
+		    diskformat.setVisible(false);
 		    if (!allStoragesAvailable) {
-			let storagesel = view.down('pveStorageSelector[name=storage]');
+			let storagesel = view.down('pveStorageSelector[name=hdstorage]');
 			storagesel.allowBlank = false;
 			storagesel.setEmptyText('');
 		    }
@@ -165,12 +169,14 @@ Ext.define('PVE.window.Restore', {
 	    throw "no vmtype specified";
 	}
 
-	let storagesel = Ext.create('PVE.form.StorageSelector', {
+	let storagesel = Ext.create('PVE.form.DiskStorageSelector', {
 	    nodename: me.nodename,
 	    name: 'storage',
 	    value: '',
 	    fieldLabel: gettext('Storage'),
 	    storageContent: me.vmtype === 'lxc' ? 'rootdir' : 'images',
+	    hideSize: true,
+	    hideFormatWhenStorageEmpty: true,
 	    // when restoring a container without specifying a storage, the backend defaults
 	    // to 'local', which is unintuitive and 'rootdir' might not even be allowed on it
 	    allowBlank: me.vmtype !== 'lxc',
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH storage v2 1/4] add standard option for VM disk formats in file-based storages

[Fiona Ebner]

Am 12.02.25 um 14:02 schrieb Markus Frank:
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
>  src/PVE/Storage/Plugin.pm | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
> index 65cf43f..9f21f0b 100644
> --- a/src/PVE/Storage/Plugin.pm
> +++ b/src/PVE/Storage/Plugin.pm
> @@ -354,6 +354,13 @@ sub verify_format {
>      return $fmt;
>  }
>  
> +PVE::JSONSchema::register_standard_option('pve-storage-qm-image-format', {

Nit: There is no real relation to the qm CLI tool. I'd prefer
pve-vm-image-format.

> +    type => 'string',
> +    enum => ['raw', 'qcow2', 'vmdk'],
> +    description => "Supported VM disk formats in a file-based storage.",

Nit: not all file-based storage might (want to) support all. Maybe
simply "VM image format."?

> +    optional => 1,

Not sure if we want to make it optional by default. Of course, can still
opt-out when needed at the use-site, but might be easier to forget this
way around. Having it be required by default and the use-site opt-out
might be more natural in this case (with future use-sites in mind). That
also makes it explicitly visible at the use site if optional. You
already specify optional => 1 for both use sites, so no change would be
needed there :) All that said, fine by me either way.

> +});
> +
>  PVE::JSONSchema::register_format('pve-storage-options', \&verify_options);
>  sub verify_options {
>      my ($value, $noerr) = @_;


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH qemu-server v2 2/4] fix 4888: qmrestore: add disk-format option

[Fiona Ebner]

Am 12.02.25 um 14:02 schrieb Markus Frank:
> Add an option to choose a file format (qcow2, raw, vmdk) when restoring
> a vm backup to file based storage. This options allows all disks to be
> recreated with the specified file format if supported by the target
> storage.
> 
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
>  PVE/API2/Qemu.pm     |  6 ++++++
>  PVE/CLI/qmrestore.pm |  4 ++++
>  PVE/QemuServer.pm    | 10 +++++++---
>  3 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
> index 295260e7..617dbe45 100644
> --- a/PVE/API2/Qemu.pm
> +++ b/PVE/API2/Qemu.pm
> @@ -1013,6 +1013,10 @@ __PACKAGE__->register_method({
>  		    default => 0,
>  		    description => "Start VM after it was created successfully.",
>  		},
> +		'disk-format' => get_standard_option('pve-storage-qm-image-format', {
> +		    optional => 1,

requires => 'archive',

> +		    description => "Image format used for all VM disks when restoring.",

I'd rather say "Preferred image format...", because if the storage
doesn't support it, it won't be used.

> +		}),
>  		'import-working-storage' => get_standard_option('pve-storage-id', {
>  		    description => "A file-based storage with 'images' content-type enabled, which"
>  			." is used as an intermediary extraction storage during import. Defaults to"
> @@ -1046,6 +1050,7 @@ __PACKAGE__->register_method({
>  	my $storage = extract_param($param, 'storage');
>  	my $unique = extract_param($param, 'unique');
>  	my $live_restore = extract_param($param, 'live-restore');
> +	my $disk_format = extract_param($param, 'disk-format');
>  	my $extraction_storage = extract_param($param, 'import-working-storage');
>  
>  	if (defined(my $ssh_keys = $param->{sshkeys})) {
> @@ -1143,6 +1148,7 @@ __PACKAGE__->register_method({
>  		    unique => $unique,
>  		    bwlimit => $bwlimit,
>  		    live => $live_restore,
> +		    disk_format => $disk_format,
>  		    override_conf => $param,
>  		};
>  		if (my $volid = $archive->{volid}) {
> diff --git a/PVE/CLI/qmrestore.pm b/PVE/CLI/qmrestore.pm
> index a47648bd..68c23db5 100755
> --- a/PVE/CLI/qmrestore.pm
> +++ b/PVE/CLI/qmrestore.pm
> @@ -64,6 +64,10 @@ __PACKAGE__->register_method({
>  		type => 'boolean',
>  		description => "Start the VM immediately from the backup and restore in background. PBS only.",
>  	    },
> +	    'disk-format' => get_standard_option('pve-storage-qm-image-format', {

missing
use PVE::Storage; # for 'pve-storage-qm-image-format' standard option
which registers the option, at the beginning of the module

> +		optional => 1,
> +		description => "Restore all VM disks with the specified image format.",

I'd rather say "Preferred image format...", because if the storage
doesn't support it, it won't be used.

> +	    }),
>  	},
>      },
>      returns => {
> diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
> index 808c0e1c..63836399 100644
> --- a/PVE/QemuServer.pm
> +++ b/PVE/QemuServer.pm
> @@ -6660,7 +6660,7 @@ my $parse_backup_hints = sub {
>  #
>  # Returns: { $virtdev => $volid }
>  my $restore_allocate_devices = sub {
> -    my ($storecfg, $virtdev_hash, $vmid) = @_;
> +    my ($storecfg, $virtdev_hash, $vmid, $disk_format) = @_;
>  
>      my $map = {};
>      foreach my $virtdev (sort keys %$virtdev_hash) {
> @@ -6670,6 +6670,10 @@ my $restore_allocate_devices = sub {
>  	my $storeid = $d->{storeid};
>  	my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
>  
> +	if ($disk_format) {
> +	    $d->{format} = $disk_format;

Rather than overriding the format hint, we could consider both:
check if $disk_format is supported, if yes, use that
else, check if $d->{format} is supported, if yes, use that
else, use default format

But arguably quite an edge case, I'm also fine with the behavior in this
patch.

> +	}
> +
>  	# test if requested format is supported
>  	my ($defFormat, $validFormats) = PVE::Storage::storage_default_format($storecfg, $storeid);
>  	my $supported = grep { $_ eq $d->{format} } @$validFormats;


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager v2 3/4] ui: form: add hideFormatWhenStorageEmpty option to DiskStorageSelector

[Fiona Ebner]

"form:" by itself doesn't provide much information as a prefix

Am 12.02.25 um 14:02 schrieb Markus Frank:
> Prerequisite for "ui: window: add diskformat option to restore window"
> 
> The hide condition is copied from the format selector item in the same
> file.
> 
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
>  www/manager6/form/DiskStorageSelector.js | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/www/manager6/form/DiskStorageSelector.js b/www/manager6/form/DiskStorageSelector.js
> index 0ef48f51..e2064934 100644
> --- a/www/manager6/form/DiskStorageSelector.js
> +++ b/www/manager6/form/DiskStorageSelector.js
> @@ -30,6 +30,7 @@ Ext.define('PVE.form.DiskStorageSelector', {
>  
>      // hides the format field (e.g. for TPM state)
>      hideFormat: false,
> +    hideFormatWhenStorageEmpty: false,
>  
>      // sets the initial size value
>      // string because else we get a type confusion
> @@ -41,12 +42,20 @@ Ext.define('PVE.form.DiskStorageSelector', {
>  	var hdfilesel = me.getComponent('hdimage');
>  	var hdsizesel = me.getComponent('disksize');
>  
> +	// This is needed to make the format selector visible
> +	// after it has been hidden because of hideFormatWhenStorageEmpty.
> +	let hideFormatCondition = me.hideFormat || me.storageContent === 'rootdir';
> +	formatsel.setVisible(!hideFormatCondition);

Nit: could be moved further down in the code for better grouping things
to here:

> 	formatsel.setDisabled(me.hideFormat || Ext.Object.getSize(validFormats) <= 1);
> 	formatsel.setValue(selectFormat);

or is there a reason that wouldn't work?

Looking at the next patch, it seems like you manually hide it at first
from the use-site. Maybe we should do it in initComponent() instead?
I.e. also hide if me.hideFormatWhenStorageEmpty && !me.autoSelect.

Going for a view model and bindings might be best to modernize the
component a bit and make such things easier, but not a blocker from my side.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager v2 4/4] ui: window: add disk-format option to the restore window

[Fiona Ebner]

"window:" by itself doesn't provide much information as a prefix. If you
use "restore window:" then you can drop that from the end of the commit
title.

Am 12.02.25 um 14:02 schrieb Markus Frank:
> @@ -141,9 +144,10 @@ Ext.define('PVE.window.Restore', {
>  			    view.lookupReference(`${key}Field`).setEmptyText(value);
>  			}
>  		    });
> -
> +		    let diskformat = view.down('pveDiskFormatSelector[name=diskformat]');
> +		    diskformat.setVisible(false);

I think it's better to do this in the component itself, see my comment
on the previous patch.

>  		    if (!allStoragesAvailable) {
> -			let storagesel = view.down('pveStorageSelector[name=storage]');
> +			let storagesel = view.down('pveStorageSelector[name=hdstorage]');
>  			storagesel.allowBlank = false;
>  			storagesel.setEmptyText('');
>  		    }

Looks good otherwise :)


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel