[pve-devel] [RFC qemu-server] fix #2862: properly backup (all) VM templates

[pve-devel] [RFC qemu-server] fix #2862: properly backup (all) VM templates

[Fabian Grünbichler]

until we maybe have a 'pbs-backup' that links Qemu and PBS like
'pbs-restore', we need to do a regular backup for the template case to
support all storage types and image formats.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---

Notes:
    requires patched pve-qemu to drop read-only check

    alternatively, we could also start writing a pbs-backup (or a
    general pbs-qemu ?) soon, and just disable PBS template backups in
    the meantime.

 PVE/QemuServer.pm        |  6 +++++-
 PVE/VZDump/QemuServer.pm | 22 +++++++---------------
 2 files changed, 12 insertions(+), 16 deletions(-)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 0a09f3a..e789dea 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -3402,6 +3402,8 @@ sub config_to_command {
         }
 
 	my $drive_cmd = print_drive_commandline_full($storecfg, $vmid, $drive);
+	$drive_cmd .= ',readonly' if PVE::QemuConfig->is_template($conf);
+
 	push @$devices, '-drive',$drive_cmd;
 	push @$devices, '-device', print_drivedevice_full($storecfg, $conf, $vmid, $drive, $bridges, $arch, $machine_type);
     });
@@ -4780,7 +4782,8 @@ sub vm_start {
     return PVE::QemuConfig->lock_config($vmid, sub {
 	my $conf = PVE::QemuConfig->load_config($vmid, $migrate_opts->{migratedfrom});
 
-	die "you can't start a vm if it's a template\n" if PVE::QemuConfig->is_template($conf);
+	die "you can't start a vm if it's a template\n"
+	    if !$params->{skiptemplate} && PVE::QemuConfig->is_template($conf);
 
 	my $has_suspended_lock = PVE::QemuConfig->has_lock($conf, 'suspended');
 
@@ -4809,6 +4812,7 @@ sub vm_start {
 # params:
 #   statefile => 'tcp', 'unix' for migration or path/volid for RAM state
 #   skiplock => 0/1, skip checking for config lock
+#   skiptemplate => 0/1, skip checking whether VM is template
 #   forcemachine => to force Qemu machine (rollback/migration)
 #   forcecpu => a QEMU '-cpu' argument string to override get_cpu_options
 #   timeout => in seconds
diff --git a/PVE/VZDump/QemuServer.pm b/PVE/VZDump/QemuServer.pm
index 4614efc..3502111 100644
--- a/PVE/VZDump/QemuServer.pm
+++ b/PVE/VZDump/QemuServer.pm
@@ -395,20 +395,11 @@ sub archive_pbs {
     my $keyfile = PVE::Storage::PBSPlugin::pbs_encryption_key_file_name($scfg, $opts->{storage});
 
     my $diskcount = scalar(@{$task->{disks}});
-    if (PVE::QemuConfig->is_template($self->{vmlist}->{$vmid}) || !$diskcount) {
+    # proxmox-backup-client can only handle raw files and block devs
+    # only use it (directly) for disk-less VMs
+    if (!$diskcount) {
 	my @pathlist;
-	# FIXME: accumulate disk sizes to use for backup job (email) log
-	foreach my $di (@{$task->{disks}}) {
-	    if ($di->{type} eq 'block' || $di->{type} eq 'file') {
-		push @pathlist, "$di->{qmdevice}.img:$di->{path}";
-	    } else {
-		die "implement me (type $di->{type})";
-	    }
-	}
-
-	if (!$diskcount) {
-	    $self->loginfo("backup contains no disks");
-	}
+	$self->loginfo("backup contains no disks");
 
 	local $ENV{PBS_PASSWORD} = $password;
 	local $ENV{PBS_FINGERPRINT} = $fingerprint if defined($fingerprint);
@@ -423,7 +414,6 @@ sub archive_pbs {
 
 	push @$cmd, "qemu-server.conf:$conffile";
 	push @$cmd, "fw.conf:$firewall" if -e $firewall;
-	push @$cmd, @pathlist if scalar(@pathlist);
 
 	$self->loginfo("starting template backup");
 	$self->loginfo(join(' ', @$cmd));
@@ -471,7 +461,8 @@ sub archive_pbs {
 	    $params->{encrypt} = JSON::false;
 	}
 
-	$params->{'use-dirty-bitmap'} = JSON::true if $qemu_support->{'pbs-dirty-bitmap'};
+	$params->{'use-dirty-bitmap'} = JSON::true
+	    if $qemu_support->{'pbs-dirty-bitmap'} && !PVE::QemuConfig->is_template($self->{vmlist}->{$vmid});
 
 	$params->{timeout} = 60; # give some time to connect to the backup server
 
@@ -752,6 +743,7 @@ sub enforce_vm_running_for_backup {
 	# start with skiplock
 	my $params = {
 	    skiplock => 1,
+	    skiptemplate => 1,
 	    paused => 1,
 	};
 	PVE::QemuServer::vm_start($self->{storecfg}, $vmid, $params);
-- 
2.20.1

[pve-devel] [RFC qemu] allow backup of read-only block drives

[Fabian Grünbichler]

this is needed for template backups with PBS until we have the backup
equivalent of 'pbs-restore'.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
did some quick tests and didn't run into any issues - @Dietmar/@Stefan
is that check needed for some non-obvious reason?

 .../0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch   | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/debian/patches/pve/0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch b/debian/patches/pve/0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch
index 56161cc..7b4834a 100644
--- a/debian/patches/pve/0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch
+++ b/debian/patches/pve/0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch
@@ -527,7 +527,7 @@ new file mode 100644
 index 0000000000..9ae89fb679
 --- /dev/null
 +++ b/pve-backup.c
-@@ -0,0 +1,959 @@
+@@ -0,0 +1,955 @@
 +#include "proxmox-backup-client.h"
 +#include "vma.h"
 +
@@ -1050,10 +1050,6 @@ index 0000000000..9ae89fb679
 +            blk = blk_by_name(*d);
 +            if (blk) {
 +                bs = blk_bs(blk);
-+                if (bdrv_is_read_only(bs)) {
-+                    error_setg(task->errp, "Node '%s' is read only", *d);
-+                    goto err;
-+                }
 +                if (!bdrv_is_inserted(bs)) {
 +                    error_setg(task->errp, QERR_DEVICE_HAS_NO_MEDIUM, *d);
 +                    goto err;
-- 
2.20.1

[pve-devel] applied: [RFC qemu] allow backup of read-only block drives

[Thomas Lamprecht]

On 06.08.20 13:13, Fabian Grünbichler wrote:
> this is needed for template backups with PBS until we have the backup
> equivalent of 'pbs-restore'.
> 
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> did some quick tests and didn't run into any issues - @Dietmar/@Stefan
> is that check needed for some non-obvious reason?
> 
>  .../0029-PVE-Backup-proxmox-backup-patches-for-qemu.patch   | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
>

applied, thanks!

[pve-devel] applied: [RFC qemu-server] fix #2862: properly backup (all) VM templates

[Thomas Lamprecht]

On 06.08.20 13:13, Fabian Grünbichler wrote:
> until we maybe have a 'pbs-backup' that links Qemu and PBS like
> 'pbs-restore', we need to do a regular backup for the template case to
> support all storage types and image formats.
> 
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> 
> Notes:
>     requires patched pve-qemu to drop read-only check
> 
>     alternatively, we could also start writing a pbs-backup (or a
>     general pbs-qemu ?) soon, and just disable PBS template backups in
>     the meantime.
> 
>  PVE/QemuServer.pm        |  6 +++++-
>  PVE/VZDump/QemuServer.pm | 22 +++++++---------------
>  2 files changed, 12 insertions(+), 16 deletions(-)
> 
>

While not 100% happy with this, it works OK from what I can see. And we highly
probably won't write a pbs-backup wrapper just for this use-case.

applied, thanks!