From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id F09241FF16F
for <inbox@lore.proxmox.com>; Thu, 13 Feb 2025 18:21:06 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 34430BA89;
Thu, 13 Feb 2025 18:21:01 +0100 (CET)
Message-ID: <a6d503c5-486f-4c2c-813b-39866a054d13@proxmox.com>
Date: Thu, 13 Feb 2025 18:20:56 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
Filip Schauer <f.schauer@proxmox.com>
References: <20250120112842.36450-1-f.schauer@proxmox.com>
<20250120112842.36450-6-f.schauer@proxmox.com>
Content-Language: en-US
From: Fiona Ebner <f.ebner@proxmox.com>
In-Reply-To: <20250120112842.36450-6-f.schauer@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.047 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
Subject: Re: [pve-devel] [PATCH storage v6 5/7] support moving VMA backups
to PBS
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
Am 20.01.25 um 12:28 schrieb Filip Schauer:
> Extend the move API to support moving VMA backups to a Proxmox Backup
> Server.
>
> Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
> ---
> debian/control | 1 +
> src/PVE/API2/Storage/Content.pm | 53 +++++++++++++++------------
> src/PVE/Storage/PBSPlugin.pm | 65 +++++++++++++++++++++++++++++++++
> 3 files changed, 96 insertions(+), 23 deletions(-)
>
> diff --git a/debian/control b/debian/control
> index 4e1a046..0883777 100644
> --- a/debian/control
> +++ b/debian/control
> @@ -46,6 +46,7 @@ Depends: bzip2,
> nfs-common,
> proxmox-backup-client (>= 2.1.10~),
> proxmox-backup-file-restore,
> + proxmox-vma-to-pbs (>= 0.0.2),
> pve-cluster (>= 5.0-32),
> smartmontools,
> smbclient,
> diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
> index 9ee3c51..c0e2a4e 100644
> --- a/src/PVE/API2/Storage/Content.pm
> +++ b/src/PVE/API2/Storage/Content.pm
> @@ -10,6 +10,7 @@ use File::Copy qw(copy move);
> use PVE::SafeSyslog;
> use PVE::Cluster;
> use PVE::Storage;
> +use PVE::Storage::PBSPlugin;
> use PVE::INotify;
> use PVE::Exception qw(raise_param_exc);
> use PVE::RPCEnvironment;
> @@ -557,12 +558,12 @@ __PACKAGE__->register_method ({
> my $user = $rpcenv->get_user();
>
> PVE::Storage::check_volume_access($rpcenv, $user, $cfg, undef, $src_volid);
> + my $src_cfg = PVE::Storage::storage_config($cfg, $src_storeid);
>
> if ($delete) {
> $rpcenv->check($user, "/storage/$src_storeid", ["Datastore.Allocate"]);
>
> if ($vtype eq 'backup') {
> - my $src_cfg = PVE::Storage::storage_config($cfg, $src_storeid);
> my $src_plugin = PVE::Storage::Plugin->lookup($src_cfg->{type});
> my $protected = $src_plugin->get_volume_attribute($src_cfg, $src_storeid, $volname, 'protected');
> die "cannot delete protected backup\n" if $protected;
> @@ -577,30 +578,36 @@ __PACKAGE__->register_method ({
>
> my $worker = sub {
> PVE::Storage::storage_check_enabled($cfg, $dst_storeid, $dst_node);
> - my $sshinfo;
>
> - if ($src_node eq $dst_node) {
> - $sshinfo = {
> - ip => "localhost",
> - name => $dst_node,
> - };
> + my $dst_cfg = PVE::Storage::storage_config($cfg, $dst_storeid);
> + if ($vtype eq 'backup' && $dst_cfg->{type} eq 'pbs') {
Maybe check that the source plugin is path-based here to rule out some
other third-party backup storages early.
> + PVE::Storage::PBSPlugin::vma_to_pbs($src_cfg, $src_volid, $dst_cfg, $dst_storeid);
> } else {
> - $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node);
> - }
> -
> - my $opts = { 'target_volname' => $volname };
> - PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $dst_storeid, $opts);
> -
> - if ($delete) {
> - my $src_path = PVE::Storage::abs_filesystem_path($cfg, $src_volid);
> - PVE::Storage::archive_remove($src_path, 1);
> - }
> -
> - if ($src_node eq $dst_node) {
> - print "Moved volume '$src_volid' to '$dst_storeid'\n";
> - } else {
> - print "Moved volume '$src_volid' on node '$src_node'"
> - ." to '$dst_storeid' on node '$dst_node'\n";
> + my $sshinfo;
> +
> + if ($src_node eq $dst_node) {
> + $sshinfo = {
> + ip => "localhost",
> + name => $dst_node,
> + };
> + } else {
> + $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node);
> + }
> +
> + my $opts = { 'target_volname' => $volname };
> + PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $dst_storeid, $opts);
> +
> + if ($delete) {
> + my $src_path = PVE::Storage::abs_filesystem_path($cfg, $src_volid);
> + PVE::Storage::archive_remove($src_path, 1);
> + }
> +
> + if ($src_node eq $dst_node) {
> + print "Moved volume '$src_volid' to '$dst_storeid'\n";
> + } else {
> + print "Moved volume '$src_volid' on node '$src_node'"
> + ." to '$dst_storeid' on node '$dst_node'\n";
> + }
> }
> };
>
> diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
> index 0808bcc..4f8a05d 100644
> --- a/src/PVE/Storage/PBSPlugin.pm
> +++ b/src/PVE/Storage/PBSPlugin.pm
> @@ -6,6 +6,7 @@ use strict;
> use warnings;
>
> use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
> +use File::Basename;
> use IO::File;
> use JSON;
> use MIME::Base64 qw(decode_base64);
> @@ -971,4 +972,68 @@ sub volume_has_feature {
> return undef;
> }
>
> +sub vma_to_pbs {
> + my ($source_scfg, $source_volid, $target_scfg, $target_storeid) = @_;
> +
> + my $source_plugin = PVE::Storage::Plugin->lookup($source_scfg->{type});
I know I'm being nitpicky, but I'd kinda prefer not having to access the
source plugin here. The required parameters could be passed in.
> + my $target_plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
> + my ($source_storeid, $source_volname) = PVE::Storage::parse_volume_id($source_volid, 0);
> + my $source_path = $source_plugin->path($source_scfg, $source_volname, $source_storeid);
> + my $info = PVE::Storage::archive_info($source_path);
> + die "moving non-VMA backups to a Proxmox Backup Server is not supported\n"
> + if $info->{format} ne 'vma';
> +
> + my $repo = PVE::PBSClient::get_repository($target_scfg);
> + my $vmid = ($source_plugin->parse_volname($source_volname))[2];
> + my $fingerprint = $target_scfg->{fingerprint};
> + my $password = PVE::Storage::PBSPlugin::pbs_password_file_name($target_scfg, $target_storeid);
> + my $namespace = $target_scfg->{namespace};
> + my $keyfile = PVE::Storage::PBSPlugin::pbs_encryption_key_file_name(
> + $target_scfg, $target_storeid);
> + my $master_keyfile = PVE::Storage::PBSPlugin::pbs_master_pubkey_file_name(
> + $target_scfg, $target_storeid);
> +
> + my $comp = $info->{compression};
> + my $backup_time = $info->{ctime};
> + my $source_dirname = dirname($source_path);
> + my $log_file_path = "$source_dirname/$info->{logfilename}";
> + my $notes_file_path = "$source_dirname/$info->{notesfilename}";
> +
> + my $vma_to_pbs_cmd = [
> + "vma-to-pbs",
> + "--repository", $repo,
> + "--vmid", $vmid,
> + "--fingerprint", $fingerprint,
> + "--password-file", $password,
> + "--backup-time", $backup_time,
> + "--compress",
> + ];
> +
> + push @$vma_to_pbs_cmd, "--ns", $namespace if $namespace;
> + push @$vma_to_pbs_cmd, "--log-file", $log_file_path if -e $log_file_path;
> + push @$vma_to_pbs_cmd, "--notes-file", $notes_file_path if -e $notes_file_path;
> + push @$vma_to_pbs_cmd, "--encrypt", "--keyfile", $keyfile if -e $keyfile;
> + push @$vma_to_pbs_cmd, "--master-keyfile", $master_keyfile if -e $master_keyfile;
> +
> + if ($comp) {
> + PVE::Storage::decompress_archive_into_pipe($source_path, $vma_to_pbs_cmd);
> + } else {
> + push @$vma_to_pbs_cmd, $source_path;
> + run_command($vma_to_pbs_cmd);
> + }
> +
> + my $protected = $source_plugin->get_volume_attribute(
> + $source_scfg, $source_storeid, $source_volname, 'protected');
> +
> + if ($protected) {
> + my $target_volid = PVE::Storage::PBSPlugin::print_volid(
> + $target_storeid, 'vm', $vmid, $backup_time);
> + my (undef, $target_volname) = PVE::Storage::parse_volume_id($target_volid, 0);
> + $target_plugin->update_volume_attribute(
> + $target_scfg, $target_storeid, $target_volname, 'protected', 1);
> + }
> +
> + return;
> +}
> +
> 1;
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel