* [pve-devel] [PATCH v4 storage 1/6] plugin: allow volume import of iso, snippets and vztmpl
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 2/6] api: content: implement moving a volume between storages Filip Schauer
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/Storage/Plugin.pm | 67 +++++++++++++++++++++++++--------------
1 file changed, 43 insertions(+), 24 deletions(-)
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 8cc693c..57536c6 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1667,14 +1667,18 @@ sub volume_import {
# XXX: Should we bother with conversion routines at this level? This won't
# happen without manual CLI usage, so for now we just error out...
- die "cannot import format $format into a file of format $file_format\n"
- if $data_format ne $file_format && !($data_format eq 'tar' && $file_format eq 'subvol');
+ if (($vtype eq 'images' || $vtype eq 'rootdir') && $data_format ne $file_format &&
+ !($data_format eq 'tar' && $file_format eq 'subvol')
+ ) {
+ die "cannot import format $format into a file of format $file_format\n";
+ }
# Check for an existing file first since interrupting alloc_image doesn't
# free it.
my $file = $class->path($scfg, $volname, $storeid);
if (-e $file) {
- die "file '$file' already exists\n" if !$allow_rename;
+ die "file '$file' already exists\n"
+ if !$allow_rename || ($vtype ne 'images' && $vtype ne 'rootdir');
warn "file '$file' already exists - importing with a different name\n";
$name = undef;
}
@@ -1682,29 +1686,44 @@ sub volume_import {
my ($size) = read_common_header($fh);
$size = int($size/1024);
- eval {
- my $allocname = $class->alloc_image($storeid, $scfg, $vmid, $file_format, $name, $size);
- my $oldname = $volname;
- $volname = $allocname;
- if (defined($name) && $allocname ne $oldname) {
- die "internal error: unexpected allocated name: '$allocname' != '$oldname'\n";
+ if ($vtype eq 'images' || $vtype eq 'rootdir') {
+ eval {
+ my $allocname = $class->alloc_image($storeid, $scfg, $vmid, $file_format, $name, $size);
+ my $oldname = $volname;
+ $volname = $allocname;
+ if (defined($name) && $allocname ne $oldname) {
+ die "internal error: unexpected allocated name: '$allocname' != '$oldname'\n";
+ }
+ my $file = $class->path($scfg, $volname, $storeid)
+ or die "internal error: failed to get path to newly allocated volume $volname\n";
+ if ($data_format eq 'raw' || $data_format eq 'qcow2' || $data_format eq 'vmdk') {
+ run_command(['dd', "of=$file", 'conv=sparse', 'bs=64k'],
+ input => '<&'.fileno($fh));
+ } elsif ($data_format eq 'tar') {
+ run_command(['tar', @COMMON_TAR_FLAGS, '-C', $file, '-xf', '-'],
+ input => '<&'.fileno($fh));
+ } else {
+ die "volume import format '$format' not available for $class";
+ }
+ };
+ if (my $err = $@) {
+ eval { $class->free_image($storeid, $scfg, $volname, 0, $file_format) };
+ warn $@ if $@;
+ die $err;
}
- my $file = $class->path($scfg, $volname, $storeid)
- or die "internal error: failed to get path to newly allocated volume $volname\n";
- if ($data_format eq 'raw' || $data_format eq 'qcow2' || $data_format eq 'vmdk') {
- run_command(['dd', "of=$file", 'conv=sparse', 'bs=64k'],
- input => '<&'.fileno($fh));
- } elsif ($data_format eq 'tar') {
- run_command(['tar', @COMMON_TAR_FLAGS, '-C', $file, '-xf', '-'],
- input => '<&'.fileno($fh));
- } else {
- die "volume import format '$format' not available for $class";
+ } elsif ($vtype eq 'iso' || $vtype eq 'snippets' || $vtype eq 'vztmpl') {
+ eval {
+ run_command(['dd', "of=$file", 'bs=64k'], input => '<&'.fileno($fh));
+ };
+ if (my $err = $@) {
+ if (-e $file) {
+ eval { unlink($file) };
+ warn $@ if $@;
+ }
+ die $err;
}
- };
- if (my $err = $@) {
- eval { $class->free_image($storeid, $scfg, $volname, 0, $file_format) };
- warn $@ if $@;
- die $err;
+ } else {
+ die "importing volume of type '$vtype' not implemented\n";
}
return "$storeid:$volname";
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 storage 2/6] api: content: implement moving a volume between storages
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 1/6] plugin: allow volume import of iso, snippets and vztmpl Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 3/6] api: content: support moving backups between path based storages Filip Schauer
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
Add the ability to move an iso, snippet or vztmpl between storages and
nodes.
Use curl to call the API method:
```
curl https://$APINODE:8006/api2/json/nodes/$SOURCENODE/storage/$SOURCESTORAGE/content/$SOURCEVOLUME \
--insecure --cookie "$(<cookie)" -H "$(<csrftoken)" -X POST \
--data-raw "target-storage=$TARGETSTORAGE&target-node=$TARGETNODE"
```
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/API2/Storage/Content.pm | 149 ++++++++++++++++++++++++--------
1 file changed, 114 insertions(+), 35 deletions(-)
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index fe0ad4a..6819eca 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -2,7 +2,10 @@ package PVE::API2::Storage::Content;
use strict;
use warnings;
-use Data::Dumper;
+
+use Errno qw(ENOENT);
+use File::Basename;
+use File::Copy qw(copy move);
use PVE::SafeSyslog;
use PVE::Cluster;
@@ -483,30 +486,101 @@ __PACKAGE__->register_method ({
return $upid;
}});
+sub volume_move {
+ my ($cfg, $source_volid, $target_storeid, $delete) = @_;
+
+ my ($source_storeid, $source_volname) = PVE::Storage::parse_volume_id($source_volid, 0);
+
+ die "source and target storage cannot be the same\n" if $source_storeid eq $target_storeid;
+
+ my $source_scfg = PVE::Storage::storage_config($cfg, $source_storeid);
+ my $source_plugin = PVE::Storage::Plugin->lookup($source_scfg->{type});
+ my ($vtype) = $source_plugin->parse_volname($source_volname);
+
+ die "source storage '$source_storeid' does not support volumes of type '$vtype'\n"
+ if !$source_scfg->{content}->{$vtype};
+
+ my $target_scfg = PVE::Storage::storage_config($cfg, $target_storeid);
+ die "target storage '$target_storeid' does not support volumes of type '$vtype'\n"
+ if !$target_scfg->{content}->{$vtype};
+
+ die "use pct move-volume or qm disk move\n" if $vtype eq 'images' || $vtype eq 'rootdir';
+ die "moving volume of type '$vtype' not implemented\n"
+ if ($vtype ne 'iso' && $vtype ne 'vztmpl' && $vtype ne 'snippets');
+
+ PVE::Storage::activate_storage($cfg, $source_storeid);
+
+ die "expected storage '$source_storeid' to be path based\n" if !$source_scfg->{path};
+ my $source_path = $source_plugin->path($source_scfg, $source_volname, $source_storeid);
+ die "$source_path does not exist" if (!-e $source_path);
+ my $source_dirname = dirname($source_path);
+
+ die "expected storage '$target_storeid' to be path based\n" if !$target_scfg->{path};
+ my $target_plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
+ my $target_path = $target_plugin->path($target_scfg, $source_volname, $target_storeid);
+
+ PVE::Storage::activate_storage($cfg, $target_storeid);
+ die "$target_path already exists" if (-e $target_path);
+
+ my @created_files = ();
+
+ eval {
+ if ($delete) {
+ move($source_path, $target_path) or die "failed to move $vtype: $!";
+ } else {
+ copy($source_path, $target_path) or die "failed to copy $vtype: $!";
+ }
+ };
+ if (my $err = $@) {
+ for my $created_file (@created_files) {
+ unlink $created_file or $!{ENOENT} or warn $!;
+ }
+ die $err;
+ }
+
+ PVE::Storage::archive_remove($source_path) if $delete;
+
+ return;
+}
+
__PACKAGE__->register_method ({
- name => 'copy',
+ name => 'move',
path => '{volume}',
method => 'POST',
- description => "Copy a volume. This is experimental code - do not use.",
+ description => "Move a volume.",
+ permissions => {
+ description => "If the --delete option is used, the 'Datastore.Allocate' privilege is " .
+ "required on the source storage. " .
+ "Without --delete, 'Datastore.AllocateSpace' is required on the target storage. " .
+ "When moving a backup, 'VM.Backup' is required on the VM or container.",
+ user => 'all',
+ },
protected => 1,
proxyto => 'node',
parameters => {
- additionalProperties => 0,
+ additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
- storage => get_standard_option('pve-storage-id', { optional => 1}),
+ storage => get_standard_option('pve-storage-id', { optional => 1 }),
volume => {
description => "Source volume identifier",
type => 'string',
},
- target => {
- description => "Target volume identifier",
+ 'target-storage' => {
+ description => "Target storage",
type => 'string',
},
- target_node => get_standard_option('pve-node', {
+ 'target-node' => get_standard_option('pve-node', {
description => "Target node. Default is local node.",
optional => 1,
}),
+ delete => {
+ type => 'boolean',
+ description => "Delete the original volume after a successful copy. " .
+ "By default the original is kept.",
+ optional => 1,
+ default => 0,
+ },
},
},
returns => {
@@ -515,43 +589,48 @@ __PACKAGE__->register_method ({
code => sub {
my ($param) = @_;
- my $rpcenv = PVE::RPCEnvironment::get();
-
- my $user = $rpcenv->get_user();
-
- my $target_node = $param->{target_node} || PVE::INotify::nodename();
- # pvesh examples
- # cd /nodes/localhost/storage/local/content
- # pve:/> create local:103/vm-103-disk-1.raw -target local:103/vm-103-disk-2.raw
- # pve:/> create 103/vm-103-disk-1.raw -target 103/vm-103-disk-3.raw
-
my $src_volid = &$real_volume_id($param->{storage}, $param->{volume});
- my $dst_volid = &$real_volume_id($param->{storage}, $param->{target});
+ my $dst_storeid = $param->{'target-storage'};
+ my ($src_storeid, $volname) = PVE::Storage::parse_volume_id($src_volid);
+ my $src_node = PVE::INotify::nodename();
+ my $dst_node = $param->{'target-node'} || $src_node;
+ my $delete = $param->{delete};
- print "DEBUG: COPY $src_volid TO $dst_volid\n";
+ die "source and target cannot be the same\n"
+ if $src_node eq $dst_node && $src_storeid eq $dst_storeid;
my $cfg = PVE::Storage::config();
- # do all parameter checks first
-
- # then do all short running task (to raise errors before we go to background)
+ my ($vtype) = PVE::Storage::parse_volname($cfg, $src_volid);
+ die "use pct move-volume or qm disk move" if $vtype eq 'images' || $vtype eq 'rootdir';
- # then start the worker task
- my $worker = sub {
- my $upid = shift;
-
- print "DEBUG: starting worker $upid\n";
+ my $rpcenv = PVE::RPCEnvironment::get();
+ my $user = $rpcenv->get_user();
- my ($target_sid, $target_volname) = PVE::Storage::parse_volume_id($dst_volid);
- #my $target_ip = PVE::Cluster::remote_node_ip($target_node);
+ PVE::Storage::check_volume_access($rpcenv, $user, $cfg, undef, $src_volid);
- # you need to get this working (fails currently, because storage_migrate() uses
- # ssh to connect to local host (which is not needed
- my $sshinfo = PVE::SSHInfo::get_ssh_info($target_node);
- PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $target_sid, {'target_volname' => $target_volname});
+ if ($delete) {
+ $rpcenv->check($user, "/storage/$src_storeid", ["Datastore.Allocate"]);
+ } else {
+ $rpcenv->check($user, "/storage/$dst_storeid", ["Datastore.AllocateSpace"]);
+ }
- print "DEBUG: end worker $upid\n";
+ my $worker = sub {
+ if ($src_node eq $dst_node) {
+ volume_move($cfg, $src_volid, $dst_storeid, $delete);
+ } else {
+ PVE::Storage::storage_check_enabled($cfg, $dst_storeid, $dst_node);
+ my $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node);
+ PVE::Storage::storage_migrate(
+ $cfg, $src_volid, $sshinfo, $dst_storeid, {'target_volname' => $volname});
+ if ($delete) {
+ my $src_path = PVE::Storage::abs_filesystem_path($cfg, $src_volid);
+ PVE::Storage::archive_remove($src_path, 1);
+ }
+ }
+ print "Moved volume '$src_volid' on node '$src_node'"
+ ." to '$dst_storeid' on node '$dst_node'\n";
};
return $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 storage 3/6] api: content: support moving backups between path based storages
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 1/6] plugin: allow volume import of iso, snippets and vztmpl Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 2/6] api: content: implement moving a volume between storages Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 4/6] storage: introduce decompress_archive_into_pipe helper Filip Schauer
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/API2/Storage/Content.pm | 41 ++++++++++++++++++++++++++++++--
src/PVE/Storage.pm | 10 +++++++-
src/PVE/Storage/Plugin.pm | 42 ++++++++++++++++++++++++++++++---
3 files changed, 87 insertions(+), 6 deletions(-)
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 6819eca..4a8fe33 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -506,10 +506,17 @@ sub volume_move {
die "use pct move-volume or qm disk move\n" if $vtype eq 'images' || $vtype eq 'rootdir';
die "moving volume of type '$vtype' not implemented\n"
- if ($vtype ne 'iso' && $vtype ne 'vztmpl' && $vtype ne 'snippets');
+ if ($vtype ne 'iso' && $vtype ne 'vztmpl' && $vtype ne 'snippets' && $vtype ne 'backup');
PVE::Storage::activate_storage($cfg, $source_storeid);
+ if ($vtype eq 'backup') {
+ my $protected = $source_plugin->get_volume_attribute(
+ $source_scfg, $source_storeid, $source_volname, 'protected');
+
+ die "cannot delete protected backup\n" if $delete && $protected;
+ }
+
die "expected storage '$source_storeid' to be path based\n" if !$source_scfg->{path};
my $source_path = $source_plugin->path($source_scfg, $source_volname, $source_storeid);
die "$source_path does not exist" if (!-e $source_path);
@@ -525,6 +532,32 @@ sub volume_move {
my @created_files = ();
eval {
+ if ($vtype eq 'backup') {
+ my $target_dirname = dirname($target_path);
+ my $info = PVE::Storage::archive_info($source_path);
+
+ for my $type (qw(log notes)) {
+ my $filename = $info->{"${type}filename"} or next;
+ my $auxiliary_source_path = "$source_dirname/$filename";
+ my $auxiliary_target_path = "$target_dirname/$filename";
+ if (-e $auxiliary_source_path) {
+ copy($auxiliary_source_path, $auxiliary_target_path)
+ or die "copying backup $type file failed: $!";
+ push(@created_files, $auxiliary_target_path);
+ }
+ }
+
+ my $protected = $source_plugin->get_volume_attribute(
+ $source_scfg, $source_storeid, $source_volname, 'protected');
+
+ if ($protected) {
+ my $protection_target_path = PVE::Storage::protection_file_path($target_path);
+ $target_plugin->update_volume_attribute(
+ $target_scfg, $target_storeid, $source_volname, 'protected', 1);
+ push(@created_files, $protection_target_path);
+ }
+ }
+
if ($delete) {
move($source_path, $target_path) or die "failed to move $vtype: $!";
} else {
@@ -601,7 +634,7 @@ __PACKAGE__->register_method ({
my $cfg = PVE::Storage::config();
- my ($vtype) = PVE::Storage::parse_volname($cfg, $src_volid);
+ my ($vtype, undef, $ownervm) = PVE::Storage::parse_volname($cfg, $src_volid);
die "use pct move-volume or qm disk move" if $vtype eq 'images' || $vtype eq 'rootdir';
my $rpcenv = PVE::RPCEnvironment::get();
@@ -615,6 +648,10 @@ __PACKAGE__->register_method ({
$rpcenv->check($user, "/storage/$dst_storeid", ["Datastore.AllocateSpace"]);
}
+ if ($vtype eq 'backup' && $ownervm) {
+ $rpcenv->check($user, "/vms/$ownervm", ['VM.Backup']);
+ }
+
my $worker = sub {
if ($src_node eq $dst_node) {
volume_move($cfg, $src_volid, $dst_storeid, $delete);
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 57b2038..12f7b3f 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -48,7 +48,15 @@ use constant APIVER => 10;
# see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
use constant APIAGE => 1;
-our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];
+our $KNOWN_EXPORT_FORMATS = [
+ 'raw+size',
+ 'tar+size',
+ 'qcow2+size',
+ 'vmdk+size',
+ 'zfs',
+ 'btrfs',
+ 'backup+size'
+];
# load standard plugins
PVE::Storage::DirPlugin->register();
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 57536c6..fc8fe40 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -1579,6 +1579,8 @@ sub prune_backups {
# unprivileged container. In other words, this is from the root user
# namespace's point of view with no uid-mapping in effect.
# As produced via `tar -C vm-100-disk-1.subvol -cpf TheOutputFile.dat .`
+# backup+size: (backups only)
+# A raw binary data stream prefixed with a protection flag and notes.
# Plugins may reuse these helpers. Changes to the header format should be
# reflected by changes to the function prototypes.
@@ -1627,6 +1629,16 @@ sub volume_export {
run_command(['tar', @COMMON_TAR_FLAGS, '-cf', '-', '-C', $file, '.'],
output => '>&'.fileno($fh));
return;
+ } elsif ($format eq 'backup+size') {
+ write_common_header($fh, $size);
+ my $protected = $class->get_volume_attribute(
+ $scfg, $storeid, $volname, 'protected') ? 1 : 0;
+ my $notes = $class->get_volume_attribute($scfg, $storeid, $volname, 'notes') // "";
+ syswrite($fh, pack("C", $protected), 1);
+ syswrite($fh, pack("Q<", length($notes)), 8);
+ syswrite($fh, $notes, length($notes));
+ run_command(['dd', "if=$file", "bs=4k", "status=progress"], output => '>&'.fileno($fh));
+ return;
}
}
die $err_msg;
@@ -1639,6 +1651,10 @@ sub volume_export_formats {
or return;
my ($size, $format) = file_size_info($file);
+ my ($vtype) = $class->parse_volname($volname);
+
+ return ('backup+size') if $vtype eq 'backup';
+
if ($with_snapshots) {
return ($format.'+size') if ($format eq 'qcow2' || $format eq 'vmdk');
return ();
@@ -1654,7 +1670,7 @@ sub volume_import {
my ($class, $scfg, $storeid, $fh, $volname, $format, $snapshot, $base_snapshot, $with_snapshots, $allow_rename) = @_;
die "volume import format '$format' not available for $class\n"
- if $format !~ /^(raw|tar|qcow2|vmdk)\+size$/;
+ if $format !~ /^(raw|tar|qcow2|vmdk|backup)\+size$/;
my $data_format = $1;
die "format $format cannot be imported without snapshots\n"
@@ -1711,7 +1727,18 @@ sub volume_import {
warn $@ if $@;
die $err;
}
- } elsif ($vtype eq 'iso' || $vtype eq 'snippets' || $vtype eq 'vztmpl') {
+ } elsif ($vtype eq 'iso' || $vtype eq 'snippets' || $vtype eq 'vztmpl' || $vtype eq 'backup') {
+ my $protected;
+ my $notes;
+
+ if ($vtype eq 'backup') {
+ sysread($fh, $protected, 1);
+ $protected = unpack('C', $protected);
+ sysread($fh, my $notes_len, 8);
+ $notes_len = unpack('Q<', $notes_len);
+ sysread($fh, $notes, $notes_len);
+ }
+
eval {
run_command(['dd', "of=$file", 'bs=64k'], input => '<&'.fileno($fh));
};
@@ -1722,6 +1749,11 @@ sub volume_import {
}
die $err;
}
+
+ if ($vtype eq 'backup') {
+ $class->update_volume_attribute($scfg, $storeid, $volname, 'protected', $protected);
+ $class->update_volume_attribute($scfg, $storeid, $volname, 'notes', $notes);
+ }
} else {
die "importing volume of type '$vtype' not implemented\n";
}
@@ -1732,7 +1764,11 @@ sub volume_import {
sub volume_import_formats {
my ($class, $scfg, $storeid, $volname, $snapshot, $base_snapshot, $with_snapshots) = @_;
if ($scfg->{path} && !defined($base_snapshot)) {
- my $format = ($class->parse_volname($volname))[6];
+ my ($vtype, $format) = ($class->parse_volname($volname))[0, 6];
+
+ return ('backup+size') if $vtype eq 'backup';
+ return ('raw+size') if !defined($format);
+
if ($with_snapshots) {
return ($format.'+size') if ($format eq 'qcow2' || $format eq 'vmdk');
return ();
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 storage 4/6] storage: introduce decompress_archive_into_pipe helper
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
` (2 preceding siblings ...)
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 3/6] api: content: support moving backups between path based storages Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 5/6] support moving VMA backups to PBS Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 6/6] pvesm: add a move-volume command Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/Storage.pm | 64 +++++++++++++++++++++++++++++-----------------
1 file changed, 40 insertions(+), 24 deletions(-)
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 12f7b3f..e5f5326 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -1682,6 +1682,45 @@ sub extract_vzdump_config_tar {
return wantarray ? ($raw, $file) : $raw;
}
+sub decompress_archive_into_pipe {
+ my ($archive, $cmd, $outfunc) = @_;
+
+ my $info = archive_info($archive);
+ die "archive is not compressed\n" if !$info->{compression};
+ my $decompressor = $info->{decompressor};
+ my $full_cmd = [ [@$decompressor, $archive], $cmd ];
+
+ # lzop/zcat exits with 1 when the pipe is closed early,
+ # detect this and ignore the exit code later
+ my $broken_pipe;
+ my $errstring;
+ my $err = sub {
+ my $output = shift;
+ if (
+ $output =~ m/lzop: Broken pipe: <stdout>/
+ || $output =~ m/gzip: stdout: Broken pipe/
+ || $output =~ m/zstd: error 70 : Write error.*Broken pipe/
+ ) {
+ $broken_pipe = 1;
+ } elsif (!defined ($errstring) && $output !~ m/^\s*$/) {
+ $errstring = "failed to decompress archive: $output\n";
+ }
+ };
+
+ my $rc = eval { run_command($full_cmd, outfunc => $outfunc, errfunc => $err, noerr => 1) };
+ my $rerr = $@;
+
+ $broken_pipe ||= $rc == 141; # broken pipe from cmd POV
+
+ if (!$errstring && !$broken_pipe && $rc != 0) {
+ die "$rerr\n" if $rerr;
+ die "archive decompression failed with exit code $rc\n";
+ }
+ die "$errstring\n" if $errstring;
+
+ return;
+}
+
sub extract_vzdump_config_vma {
my ($archive, $comp) = @_;
@@ -1693,30 +1732,7 @@ sub extract_vzdump_config_vma {
my $decompressor = $info->{decompressor};
if ($comp) {
- my $cmd = [ [@$decompressor, $archive], ["vma", "config", "-"] ];
-
- # lzop/zcat exits with 1 when the pipe is closed early by vma, detect this and ignore the exit code later
- my $broken_pipe;
- my $errstring;
- my $err = sub {
- my $output = shift;
- if ($output =~ m/lzop: Broken pipe: <stdout>/ || $output =~ m/gzip: stdout: Broken pipe/ || $output =~ m/zstd: error 70 : Write error.*Broken pipe/) {
- $broken_pipe = 1;
- } elsif (!defined ($errstring) && $output !~ m/^\s*$/) {
- $errstring = "Failed to extract config from VMA archive: $output\n";
- }
- };
-
- my $rc = eval { run_command($cmd, outfunc => $out, errfunc => $err, noerr => 1) };
- my $rerr = $@;
-
- $broken_pipe ||= $rc == 141; # broken pipe from vma POV
-
- if (!$errstring && !$broken_pipe && $rc != 0) {
- die "$rerr\n" if $rerr;
- die "config extraction failed with exit code $rc\n";
- }
- die "$errstring\n" if $errstring;
+ decompress_archive_into_pipe($archive, ["vma", "config", "-"], $out);
} else {
run_command(["vma", "config", $archive], outfunc => $out);
}
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 storage 5/6] support moving VMA backups to PBS
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
` (3 preceding siblings ...)
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 4/6] storage: introduce decompress_archive_into_pipe helper Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 6/6] pvesm: add a move-volume command Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
Extend the move API to support moving VMA backups to a Proxmox Backup
Server.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
debian/control | 1 +
src/PVE/API2/Storage/Content.pm | 86 ++++++++++++++++++---------------
src/PVE/Storage/PBSPlugin.pm | 65 +++++++++++++++++++++++++
3 files changed, 112 insertions(+), 40 deletions(-)
diff --git a/debian/control b/debian/control
index d7afa98..6b43557 100644
--- a/debian/control
+++ b/debian/control
@@ -42,6 +42,7 @@ Depends: ceph-common (>= 12.2~),
nfs-common,
proxmox-backup-client (>= 2.1.10~),
proxmox-backup-file-restore,
+ proxmox-vma-to-pbs (>= 0.1.0),
pve-cluster (>= 5.0-32),
smartmontools,
smbclient,
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 4a8fe33..6e38a2e 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -10,6 +10,7 @@ use File::Copy qw(copy move);
use PVE::SafeSyslog;
use PVE::Cluster;
use PVE::Storage;
+use PVE::Storage::PBSPlugin;
use PVE::INotify;
use PVE::Exception qw(raise_param_exc);
use PVE::RPCEnvironment;
@@ -495,7 +496,7 @@ sub volume_move {
my $source_scfg = PVE::Storage::storage_config($cfg, $source_storeid);
my $source_plugin = PVE::Storage::Plugin->lookup($source_scfg->{type});
- my ($vtype) = $source_plugin->parse_volname($source_volname);
+ my ($vtype, undef, $vmid) = $source_plugin->parse_volname($source_volname);
die "source storage '$source_storeid' does not support volumes of type '$vtype'\n"
if !$source_scfg->{content}->{$vtype};
@@ -522,53 +523,58 @@ sub volume_move {
die "$source_path does not exist" if (!-e $source_path);
my $source_dirname = dirname($source_path);
- die "expected storage '$target_storeid' to be path based\n" if !$target_scfg->{path};
my $target_plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
- my $target_path = $target_plugin->path($target_scfg, $source_volname, $target_storeid);
-
PVE::Storage::activate_storage($cfg, $target_storeid);
- die "$target_path already exists" if (-e $target_path);
-
- my @created_files = ();
-
- eval {
- if ($vtype eq 'backup') {
- my $target_dirname = dirname($target_path);
- my $info = PVE::Storage::archive_info($source_path);
-
- for my $type (qw(log notes)) {
- my $filename = $info->{"${type}filename"} or next;
- my $auxiliary_source_path = "$source_dirname/$filename";
- my $auxiliary_target_path = "$target_dirname/$filename";
- if (-e $auxiliary_source_path) {
- copy($auxiliary_source_path, $auxiliary_target_path)
- or die "copying backup $type file failed: $!";
- push(@created_files, $auxiliary_target_path);
+
+ if ($vtype eq 'backup' && $target_scfg->{type} eq 'pbs') {
+ PVE::Storage::PBSPlugin::vma_to_pbs(
+ $source_scfg, $source_volid, $target_scfg, $target_storeid);
+ } else {
+ die "expected storage '$target_storeid' to be path based\n" if !$target_scfg->{path};
+ my $target_path = $target_plugin->path($target_scfg, $source_volname, $target_storeid);
+ die "$target_path already exists" if (-e $target_path);
+
+ my @created_files = ();
+
+ eval {
+ if ($vtype eq 'backup') {
+ my $target_dirname = dirname($target_path);
+ my $info = PVE::Storage::archive_info($source_path);
+
+ for my $type (qw(log notes)) {
+ my $filename = $info->{"${type}filename"} or next;
+ my $auxiliary_source_path = "$source_dirname/$filename";
+ my $auxiliary_target_path = "$target_dirname/$filename";
+ if (-e $auxiliary_source_path) {
+ copy($auxiliary_source_path, $auxiliary_target_path)
+ or die "copying backup $type file failed: $!";
+ push(@created_files, $auxiliary_target_path);
+ }
}
- }
- my $protected = $source_plugin->get_volume_attribute(
- $source_scfg, $source_storeid, $source_volname, 'protected');
+ my $protected = $source_plugin->get_volume_attribute(
+ $source_scfg, $source_storeid, $source_volname, 'protected');
- if ($protected) {
- my $protection_target_path = PVE::Storage::protection_file_path($target_path);
- $target_plugin->update_volume_attribute(
- $target_scfg, $target_storeid, $source_volname, 'protected', 1);
- push(@created_files, $protection_target_path);
+ if ($protected) {
+ my $protection_target_path = PVE::Storage::protection_file_path($target_path);
+ $target_plugin->update_volume_attribute(
+ $target_scfg, $target_storeid, $source_volname, 'protected', 1);
+ push(@created_files, $protection_target_path);
+ }
}
- }
- if ($delete) {
- move($source_path, $target_path) or die "failed to move $vtype: $!";
- } else {
- copy($source_path, $target_path) or die "failed to copy $vtype: $!";
- }
- };
- if (my $err = $@) {
- for my $created_file (@created_files) {
- unlink $created_file or $!{ENOENT} or warn $!;
+ if ($delete) {
+ move($source_path, $target_path) or die "failed to move $vtype: $!";
+ } else {
+ copy($source_path, $target_path) or die "failed to copy $vtype: $!";
+ }
+ };
+ if (my $err = $@) {
+ for my $created_file (@created_files) {
+ unlink $created_file or $!{ENOENT} or warn $!;
+ }
+ die $err;
}
- die $err;
}
PVE::Storage::archive_remove($source_path) if $delete;
diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
index 0808bcc..4f8a05d 100644
--- a/src/PVE/Storage/PBSPlugin.pm
+++ b/src/PVE/Storage/PBSPlugin.pm
@@ -6,6 +6,7 @@ use strict;
use warnings;
use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
+use File::Basename;
use IO::File;
use JSON;
use MIME::Base64 qw(decode_base64);
@@ -971,4 +972,68 @@ sub volume_has_feature {
return undef;
}
+sub vma_to_pbs {
+ my ($source_scfg, $source_volid, $target_scfg, $target_storeid) = @_;
+
+ my $source_plugin = PVE::Storage::Plugin->lookup($source_scfg->{type});
+ my $target_plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
+ my ($source_storeid, $source_volname) = PVE::Storage::parse_volume_id($source_volid, 0);
+ my $source_path = $source_plugin->path($source_scfg, $source_volname, $source_storeid);
+ my $info = PVE::Storage::archive_info($source_path);
+ die "moving non-VMA backups to a Proxmox Backup Server is not supported\n"
+ if $info->{format} ne 'vma';
+
+ my $repo = PVE::PBSClient::get_repository($target_scfg);
+ my $vmid = ($source_plugin->parse_volname($source_volname))[2];
+ my $fingerprint = $target_scfg->{fingerprint};
+ my $password = PVE::Storage::PBSPlugin::pbs_password_file_name($target_scfg, $target_storeid);
+ my $namespace = $target_scfg->{namespace};
+ my $keyfile = PVE::Storage::PBSPlugin::pbs_encryption_key_file_name(
+ $target_scfg, $target_storeid);
+ my $master_keyfile = PVE::Storage::PBSPlugin::pbs_master_pubkey_file_name(
+ $target_scfg, $target_storeid);
+
+ my $comp = $info->{compression};
+ my $backup_time = $info->{ctime};
+ my $source_dirname = dirname($source_path);
+ my $log_file_path = "$source_dirname/$info->{logfilename}";
+ my $notes_file_path = "$source_dirname/$info->{notesfilename}";
+
+ my $vma_to_pbs_cmd = [
+ "vma-to-pbs",
+ "--repository", $repo,
+ "--vmid", $vmid,
+ "--fingerprint", $fingerprint,
+ "--password-file", $password,
+ "--backup-time", $backup_time,
+ "--compress",
+ ];
+
+ push @$vma_to_pbs_cmd, "--ns", $namespace if $namespace;
+ push @$vma_to_pbs_cmd, "--log-file", $log_file_path if -e $log_file_path;
+ push @$vma_to_pbs_cmd, "--notes-file", $notes_file_path if -e $notes_file_path;
+ push @$vma_to_pbs_cmd, "--encrypt", "--keyfile", $keyfile if -e $keyfile;
+ push @$vma_to_pbs_cmd, "--master-keyfile", $master_keyfile if -e $master_keyfile;
+
+ if ($comp) {
+ PVE::Storage::decompress_archive_into_pipe($source_path, $vma_to_pbs_cmd);
+ } else {
+ push @$vma_to_pbs_cmd, $source_path;
+ run_command($vma_to_pbs_cmd);
+ }
+
+ my $protected = $source_plugin->get_volume_attribute(
+ $source_scfg, $source_storeid, $source_volname, 'protected');
+
+ if ($protected) {
+ my $target_volid = PVE::Storage::PBSPlugin::print_volid(
+ $target_storeid, 'vm', $vmid, $backup_time);
+ my (undef, $target_volname) = PVE::Storage::parse_volume_id($target_volid, 0);
+ $target_plugin->update_volume_attribute(
+ $target_scfg, $target_storeid, $target_volname, 'protected', 1);
+ }
+
+ return;
+}
+
1;
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH v4 storage 6/6] pvesm: add a move-volume command
2024-09-18 14:49 [pve-devel] [PATCH v4 storage 0/6] support moving volumes between storages Filip Schauer
` (4 preceding siblings ...)
2024-09-18 14:49 ` [pve-devel] [PATCH v4 storage 5/6] support moving VMA backups to PBS Filip Schauer
@ 2024-09-18 14:49 ` Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2024-09-18 14:49 UTC (permalink / raw)
To: pve-devel
The method can be called from the PVE shell with `pvesm move-volume`:
```
pvesm move-volume <source volume> <target storage> [--target-node <node>] [--delete]
```
For example to move a VMA backup to a Proxmox Backup Server:
```
pvesm move-volume \
local:backup/vzdump-qemu-100-2024_06_25-13_08_56.vma.zst pbs
```
Or move a container template to another node and delete the source:
```
pvesm move-volume \
local:vztmpl/devuan-4.0-standard_4.0_amd64.tar.gz local \
--target-node pvenode2 --delete
```
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/CLI/pvesm.pm | 2 ++
1 file changed, 2 insertions(+)
diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm
index 9b9676b..8031fc1 100755
--- a/src/PVE/CLI/pvesm.pm
+++ b/src/PVE/CLI/pvesm.pm
@@ -690,6 +690,8 @@ our $cmddef = {
print "APIVER $res->{apiver}\n";
print "APIAGE $res->{apiage}\n";
}],
+ 'move-volume' => [ "PVE::API2::Storage::Content", 'move', ['volume', 'target-storage'],
+ { node => $nodename } ],
'prune-backups' => [ __PACKAGE__, 'prunebackups', ['storage'], { node => $nodename }, sub {
my $res = shift;
--
2.39.2
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread