From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 011C11FF16E
for <inbox@lore.proxmox.com>; Mon, 31 Mar 2025 15:22:43 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 2B33C53B8;
Mon, 31 Mar 2025 15:20:44 +0200 (CEST)
From: Fiona Ebner <f.ebner@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Mon, 31 Mar 2025 15:20:08 +0200
Message-Id: <20250331132020.105324-26-f.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250331132020.105324-1-f.ebner@proxmox.com>
References: <20250331132020.105324-1-f.ebner@proxmox.com>
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.039 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: [pve-devel] [PATCH qemu-server v6 25/37] backup: implement restore
for external providers
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
First, the provider is asked about what restore mechanism to use.
Currently, only 'qemu-img' is possible. Then the configuration files
are restored, the provider gives information about volumes contained
in the backup and finally the volumes are restored via
'qemu-img convert'.
The code for the restore_external_archive() function was copied and
adapted from the restore_proxmox_backup_archive() function. Together
with restore_vma_archive() it seems sensible to extract the common
parts and use a dedicated module for restore code.
The parse_restore_archive() helper was renamed, because it's not just
parsing.
While currently, the format for the source can only be raw, do an
untrusted check for the source for future-proofing. Still serves as a
basic sanity check currently.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
---
Changes in v6:
* Drop superfluous $storeid argument for $backup_provider->restore_*
method calls.
* Pass raw format to 'qemu-img convert' for source.
* Use zeroinit filter when target volume supports the 'sparseinit'
feature.
* Squash together with patch with untrusted check.
* Adapt to renamed archive_get_*_config() methods.
PVE/API2/Qemu.pm | 30 +++++++++-
PVE/QemuServer.pm | 149 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 176 insertions(+), 3 deletions(-)
diff --git a/PVE/API2/Qemu.pm b/PVE/API2/Qemu.pm
index 156b1c7b..6c7c1d0d 100644
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -927,7 +927,7 @@ __PACKAGE__->register_method({
return $res;
}});
-my $parse_restore_archive = sub {
+my $classify_restore_archive = sub {
my ($storecfg, $archive) = @_;
my ($archive_storeid, $archive_volname) = PVE::Storage::parse_volume_id($archive, 1);
@@ -941,6 +941,22 @@ my $parse_restore_archive = sub {
$res->{type} = 'pbs';
return $res;
}
+ if (PVE::Storage::storage_has_feature($storecfg, $archive_storeid, 'backup-provider')) {
+ my $log_function = sub {
+ my ($log_level, $message) = @_;
+ my $prefix = $log_level eq 'err' ? 'ERROR' : uc($log_level);
+ print "$prefix: $message\n";
+ };
+ my $backup_provider = PVE::Storage::new_backup_provider(
+ $storecfg,
+ $archive_storeid,
+ $log_function,
+ );
+
+ $res->{type} = 'external';
+ $res->{'backup-provider'} = $backup_provider;
+ return $res;
+ }
}
my $path = PVE::Storage::abs_filesystem_path($storecfg, $archive);
$res->{type} = 'file';
@@ -1101,7 +1117,7 @@ __PACKAGE__->register_method({
'backup',
);
- $archive = $parse_restore_archive->($storecfg, $archive);
+ $archive = $classify_restore_archive->($storecfg, $archive);
}
}
@@ -1160,7 +1176,15 @@ __PACKAGE__->register_method({
PVE::QemuServer::check_restore_permissions($rpcenv, $authuser, $merged);
}
}
- if ($archive->{type} eq 'file' || $archive->{type} eq 'pipe') {
+ if (my $backup_provider = $archive->{'backup-provider'}) {
+ PVE::QemuServer::restore_external_archive(
+ $backup_provider,
+ $archive->{volid},
+ $vmid,
+ $authuser,
+ $restore_options,
+ );
+ } elsif ($archive->{type} eq 'file' || $archive->{type} eq 'pipe') {
die "live-restore is only compatible with backup images from a Proxmox Backup Server\n"
if $live_restore;
PVE::QemuServer::restore_file_archive($archive->{path} // '-', $vmid, $authuser, $restore_options);
diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 93f985be..b4796010 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -7175,6 +7175,155 @@ sub restore_proxmox_backup_archive {
}
}
+sub restore_external_archive {
+ my ($backup_provider, $archive, $vmid, $user, $options) = @_;
+
+ die "live restore from backup provider is not implemented\n" if $options->{live};
+
+ my $storecfg = PVE::Storage::config();
+
+ my ($storeid, $volname) = PVE::Storage::parse_volume_id($archive);
+ my $scfg = PVE::Storage::storage_config($storecfg, $storeid);
+
+ my $tmpdir = "/run/qemu-server/vzdumptmp$$";
+ rmtree($tmpdir);
+ mkpath($tmpdir) or die "unable to create $tmpdir\n";
+
+ my $conffile = PVE::QemuConfig->config_file($vmid);
+ # disable interrupts (always do cleanups)
+ local $SIG{INT} =
+ local $SIG{TERM} =
+ local $SIG{QUIT} =
+ local $SIG{HUP} = sub { print STDERR "got interrupt - ignored\n"; };
+
+ # Note: $oldconf is undef if VM does not exists
+ my $cfs_path = PVE::QemuConfig->cfs_config_path($vmid);
+ my $oldconf = PVE::Cluster::cfs_read_file($cfs_path);
+ my $new_conf_raw = '';
+
+ my $rpcenv = PVE::RPCEnvironment::get();
+ my $devinfo = {}; # info about drives included in backup
+ my $virtdev_hash = {}; # info about allocated drives
+
+ eval {
+ # enable interrupts
+ local $SIG{INT} =
+ local $SIG{TERM} =
+ local $SIG{QUIT} =
+ local $SIG{HUP} =
+ local $SIG{PIPE} = sub { die "interrupted by signal\n"; };
+
+ my $cfgfn = "$tmpdir/qemu-server.conf";
+ my $firewall_config_fn = "$tmpdir/fw.conf";
+
+ my $cmd = "restore";
+
+ my ($mechanism, $vmtype) =
+ $backup_provider->restore_get_mechanism($volname);
+ die "mechanism '$mechanism' requested by backup provider is not supported for VMs\n"
+ if $mechanism ne 'qemu-img';
+ die "cannot restore non-VM guest of type '$vmtype'\n" if $vmtype ne 'qemu';
+
+ $devinfo = $backup_provider->restore_vm_init($volname);
+
+ my $data = $backup_provider->archive_get_guest_config($volname)
+ or die "backup provider failed to extract guest configuration\n";
+ PVE::Tools::file_set_contents($cfgfn, $data);
+
+ if ($data = $backup_provider->archive_get_firewall_config($volname)) {
+ PVE::Tools::file_set_contents($firewall_config_fn, $data);
+ my $pve_firewall_dir = '/etc/pve/firewall';
+ mkdir $pve_firewall_dir; # make sure the dir exists
+ PVE::Tools::file_copy($firewall_config_fn, "${pve_firewall_dir}/$vmid.fw");
+ }
+
+ my $fh = IO::File->new($cfgfn, "r") or die "unable to read qemu-server.conf - $!\n";
+
+ $virtdev_hash = $parse_backup_hints->($rpcenv, $user, $storecfg, $fh, $devinfo, $options);
+
+ # create empty/temp config
+ PVE::Tools::file_set_contents($conffile, "memory: 128\nlock: create");
+
+ $restore_cleanup_oldconf->($storecfg, $vmid, $oldconf, $virtdev_hash) if $oldconf;
+
+ # allocate volumes
+ my $map = $restore_allocate_devices->($storecfg, $virtdev_hash, $vmid);
+
+ for my $virtdev (sort keys $virtdev_hash->%*) {
+ my $d = $virtdev_hash->{$virtdev};
+ next if $d->{is_cloudinit}; # no need to restore cloudinit
+
+ my $sparseinit = PVE::Storage::volume_has_feature($storecfg, 'sparseinit', $d->{volid});
+ my $source_format = 'raw';
+
+ my $info = $backup_provider->restore_vm_volume_init($volname, $d->{devname}, {});
+ my $source_path = $info->{'qemu-img-path'}
+ or die "did not get source image path from backup provider\n";
+
+ print "importing drive '$d->{devname}' from '$source_path'\n";
+
+ # safety check for untrusted source image
+ PVE::Storage::file_size_info($source_path, undef, $source_format, 1);
+
+ eval {
+ my $convert_opts = {
+ bwlimit => $options->{bwlimiit},
+ 'is-zero-initialized' => $sparseinit,
+ 'source-path-format' => $source_format,
+ };
+ qemu_img_convert($source_path, $d->{volid}, $d->{size}, $convert_opts);
+ };
+ my $err = $@;
+ eval { $backup_provider->restore_vm_volume_cleanup($volname, $d->{devname}, {}); };
+ if (my $cleanup_err = $@) {
+ die $cleanup_err if !$err;
+ warn $cleanup_err;
+ }
+ die $err if $err
+ }
+
+ $fh->seek(0, 0) || die "seek failed - $!\n";
+
+ my $cookie = { netcount => 0 };
+ while (defined(my $line = <$fh>)) {
+ $new_conf_raw .= restore_update_config_line(
+ $cookie,
+ $map,
+ $line,
+ $options->{unique},
+ );
+ }
+
+ $fh->close();
+ };
+ my $err = $@;
+
+ eval { $backup_provider->restore_vm_cleanup($volname); };
+ warn "backup provider cleanup after restore failed - $@" if $@;
+
+ if ($err) {
+ $restore_deactivate_volumes->($storecfg, $virtdev_hash);
+ }
+
+ rmtree($tmpdir);
+
+ if ($err) {
+ $restore_destroy_volumes->($storecfg, $virtdev_hash);
+ die $err;
+ }
+
+ my $new_conf = restore_merge_config($conffile, $new_conf_raw, $options->{override_conf});
+ check_restore_permissions($rpcenv, $user, $new_conf);
+ PVE::QemuConfig->write_config($vmid, $new_conf);
+
+ eval { rescan($vmid, 1); };
+ warn $@ if $@;
+
+ PVE::AccessControl::add_vm_to_pool($vmid, $options->{pool}) if $options->{pool};
+
+ return;
+}
+
sub pbs_live_restore {
my ($vmid, $conf, $storecfg, $restored_disks, $opts) = @_;
--
2.39.5
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel