* [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages
@ 2025-12-17 13:36 Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 1/6] api: content: implement " Filip Schauer
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
Add the ability to copy a backup, ISO, container template, snippet, or
OVA/OVF between storages and nodes via an API method. Copying a VMA
backup to a Proxmox Backup Server requires the proxmox-vma-to-pbs
package to be installed. Currently only VMA backups can be copied to a
Proxmox Backup Server and copying backups from a Proxmox Backup Server
is currently not supported.
The method can be called from the PVE shell with `pvesm copy-volume`:
```
pvesm copy-volume \
<source volume> <target volume or storage> \
[--target-node <node>] [--delete]
```
For example to copy a VMA backup to a Proxmox Backup Server:
```
pvesm copy-volume \
local:backup/vzdump-qemu-100-2024_06_25-13_08_56.vma.zst pbs
```
Or copy a container template to another node and delete the source:
```
pvesm copy-volume \
local:vztmpl/devuan-4.0-standard_4.0_amd64.tar.gz local \
--target-node pvenode2 --delete
```
Or copy an ISO and rename it:
```
pvesm copy-volume \
local:iso/debian-13.1.0-amd64-netinst.iso cephfs:iso/debian13.1.iso
```
Or use curl to call the API method:
```
curl https://$APINODE:8006/api2/json/nodes/$SOURCENODE/storage/$SOURCESTORAGE/content/$SOURCEVOLUME \
--insecure --cookie "$(<cookie)" -H "$(<csrftoken)" -X POST \
--data-raw "target=$TARGET&target-node=$TARGETNODE"
```
Changes since v8:
* rebase onto newest master (6f49432acc6d)
* fix permission check edge cases
* change "target-storage" parameter back to "target"
and accept either a storage id or a full volume id
Changes since v7:
* rebase onto newest master (9eb914de163d)
Changes since v6:
* introduce $vtype+meta export formats
* remove remnant from rsync-based migration
* avoid ssh when moving a volume locally
* rename 'move' API method to 'copy'
* factor out delete behavior into a helper in storage content API
* fix permission checks in 'copy' API method
* check that the source plugin is path-based before calling vma_to_pbs
* do not pass the source plugin to PVE::Storage::PBSPlugin::vma_to_pbs
Changes since v5:
* Resolve merge conflicts when applying patches 1/7 & 3/7 to current
master (e5f4af47d083).
Changes since v4:
* Remove the volume_move subroutine, instead use storage_migrate for
moving volumes between storages on the same node
* Avoid ssh when moving a volume between storages on the same node
* Add command completion to move-volume parameters
* Make the success message of move-volume less verbose for moves within
the same node
* utf8 encode/decode backup notes during export/import
* Support the new "import" volume type
* Code cleanup
* Add descriptions to single line commit messages
Changes since v3:
* Split changes into multiple commits
* Remove superfluous parentheses from post-ifs
* Move vma_to_pbs branch from move_volume into its own helper inside
PBSPlugin
* Use $! instead of $@ to retrieve unlink error in move_volume
* Also support content type 'rootdir'
* Rework permission checks on the move API method
* Fix permissions description on move API method
* Add error for unimplemented content types
Changes since v2:
* Specify permissions for move method
* Add success message to move method
* Limit the move method to non-vdisk volumes
* Check that source and target are not the same in the move method
* Remove the unnecessary movevolume method from pvesm and make the
move-volume command call the move API method directly
* Fail when trying to move a protected volume with the delete option
enabled, instead of ignoring the protection
* Change "not yet supported" to "not supported" in messages indicating
unimplemented features
* Process auxiliary files first when moving a volume locally on a node
* Move a volume instead of copying it when trying to move a volume
locally on a node with the delete option enabled.
* Use the more general `path` function instead of `filesystem_path` to
get the path of a volume
* Loosen the required privileges to move an ISO or a container template,
or when the delete option is not set.
* Move the volume_move sub from PVE::Storage to
PVE::API2::Storage::Content since it is only used there.
* Explicitly check that storages are path-based in volume_move,
except when moving a vma to a Proxmox Backup Server
Changes since v1:
* Rename pvesm command to move-volume
* Add a delete option to control whether the source volume should be
kept
* Move the API method to the POST endpoint of
/nodes/{node}/storage/{storage}/content/{volume}, replacing the
experimental copy method that has not been used since its introduction
in October 2011 883eeea6.
* Implement migrating volumes between nodes
Filip Schauer (6):
api: content: implement copying volumes between storages
introduce $vtype+meta export formats
api: content: support copying backups between path based storages
storage: introduce decompress_archive_into_pipe helper
support copying VMA backups to PBS
pvesm: add a copy-volume command
debian/control | 1 +
src/PVE/API2/Storage/Content.pm | 149 ++++++++++++++++++++++----------
src/PVE/CLI/pvesm.pm | 6 ++
src/PVE/Storage.pm | 84 +++++++++++-------
src/PVE/Storage/PBSPlugin.pm | 65 ++++++++++++++
src/PVE/Storage/Plugin.pm | 126 ++++++++++++++++++++++++---
6 files changed, 344 insertions(+), 87 deletions(-)
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 1/6] api: content: implement copying volumes between storages
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 2/6] introduce $vtype+meta export formats Filip Schauer
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
Add the ability to copy an `iso`, `snippet`, `vztmpl` or `import` volume
between storages and nodes.
This adapts the experimental storage content copy API method and adds a
few parameters. This also includes a breaking change by renaming
'target_node' to 'target-node'. Since the API method was experimental,
this breaking change is justified by the consistent naming convention of
parameters.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/API2/Storage/Content.pm | 135 +++++++++++++++++++++-----------
1 file changed, 89 insertions(+), 46 deletions(-)
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 1fe7303..201d839 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -435,6 +435,21 @@ __PACKAGE__->register_method({
},
});
+my $volume_remove = sub {
+ my ($cfg, $volid) = @_;
+
+ my ($path, undef, $vtype) = PVE::Storage::path($cfg, $volid);
+
+ PVE::Storage::vdisk_free($cfg, $volid);
+ if (
+ $vtype eq 'backup'
+ && $path =~ /(.*\/vzdump-\w+-\d+-\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2})[^\/]+$/
+ ) {
+ # Remove log file #318 and notes file #3972 if they still exist
+ PVE::Storage::archive_auxiliaries_remove($path);
+ }
+};
+
__PACKAGE__->register_method({
name => 'delete',
path => '{volume}',
@@ -493,15 +508,8 @@ __PACKAGE__->register_method({
}
my $worker = sub {
- PVE::Storage::vdisk_free($cfg, $volid);
+ &$volume_remove($cfg, $volid);
print "Removed volume '$volid'\n";
- if (
- $vtype eq 'backup'
- && $path =~ /(.*\/vzdump-\w+-\d+-\d{4}_\d{2}_\d{2}-\d{2}_\d{2}_\d{2})[^\/]+$/
- ) {
- # Remove log file #318 and notes file #3972 if they still exist
- PVE::Storage::archive_auxiliaries_remove($path);
- }
};
my $id = (defined $ownervm ? "$ownervm@" : '') . $storeid;
@@ -532,29 +540,50 @@ __PACKAGE__->register_method({
name => 'copy',
path => '{volume}',
method => 'POST',
- description => "Copy a volume. This is experimental code - do not use.",
+ description => "Copy a volume.",
+ permissions => {
+ description => "If the --delete option is used, the 'Datastore.Allocate' privilege is"
+ . " required on the source storage. 'Datastore.AllocateSpace' is required on the target"
+ . " storage.",
+ user => 'all',
+ },
protected => 1,
proxyto => 'node',
parameters => {
additionalProperties => 0,
properties => {
node => get_standard_option('pve-node'),
- storage => get_standard_option('pve-storage-id', { optional => 1 }),
+ storage => get_standard_option(
+ 'pve-storage-id',
+ {
+ optional => 1,
+ completion => \&PVE::Storage::complete_storage_enabled,
+ },
+ ),
volume => {
description => "Source volume identifier",
type => 'string',
+ completion => \&PVE::Storage::complete_volume,
},
target => {
- description => "Target volume identifier",
+ description => "Target storage or volume identifier.",
type => 'string',
+ completion => \&PVE::Storage::complete_volume,
},
- target_node => get_standard_option(
+ 'target-node' => get_standard_option(
'pve-node',
{
description => "Target node. Default is local node.",
optional => 1,
},
),
+ delete => {
+ type => 'boolean',
+ description => "Delete the original volume after a successful copy."
+ . " By default the original is kept.",
+ optional => 1,
+ default => 0,
+ },
},
},
returns => {
@@ -563,49 +592,63 @@ __PACKAGE__->register_method({
code => sub {
my ($param) = @_;
- my $rpcenv = PVE::RPCEnvironment::get();
-
- my $user = $rpcenv->get_user();
-
- my $target_node = $param->{target_node} || PVE::INotify::nodename();
- # pvesh examples
- # cd /nodes/localhost/storage/local/content
- # pve:/> create local:103/vm-103-disk-1.raw -target local:103/vm-103-disk-2.raw
- # pve:/> create 103/vm-103-disk-1.raw -target 103/vm-103-disk-3.raw
-
- my $src_volid = &$real_volume_id($param->{storage}, $param->{volume});
- my $dst_volid = &$real_volume_id($param->{storage}, $param->{target});
-
- print "DEBUG: COPY $src_volid TO $dst_volid\n";
-
+ my $src_volid = $real_volume_id->($param->{storage}, $param->{volume});
+ my ($src_storeid, $src_volname) = PVE::Storage::parse_volume_id($src_volid);
my $cfg = PVE::Storage::config();
+ my ($src_path, $ownervm, $vtype) = PVE::Storage::path($cfg, $src_volid);
+
+ die "use pct move-volume or qm disk move\n"
+ if $vtype eq 'images' || $vtype eq 'rootdir';
+ die "moving volume of type '$vtype' not implemented\n"
+ if !grep { $vtype eq $_ } qw(import iso snippets vztmpl);
+
+ my ($dst_storeid, $dst_volname);
+ if ($param->{target} =~ m/:/) {
+ ($dst_storeid, $dst_volname) = PVE::Storage::parse_volume_id($param->{target});
+ my $dst_vtype = (PVE::Storage::parse_volname($cfg, $param->{target}))[0];
+ die "source and target must have the same volume type\n" if $vtype ne $dst_vtype;
+ } else {
+ $dst_storeid = $param->{target};
+ $dst_volname = $src_volname;
+ }
+ my $dst_volid = "$dst_storeid:$dst_volname";
- # do all parameter checks first
+ my $src_node = PVE::INotify::nodename();
+ my $dst_node = $param->{'target-node'} || $src_node;
+ my $delete = $param->{delete};
- # then do all short running task (to raise errors before we go to background)
+ die "source and target cannot be the same\n"
+ if $src_node eq $dst_node && $src_storeid eq $dst_storeid;
- # then start the worker task
- my $worker = sub {
- my $upid = shift;
+ my $rpcenv = PVE::RPCEnvironment::get();
+ my $user = $rpcenv->get_user();
- print "DEBUG: starting worker $upid\n";
+ PVE::Storage::check_volume_access($rpcenv, $user, $cfg, undef, $src_volid);
- my ($target_sid, $target_volname) = PVE::Storage::parse_volume_id($dst_volid);
- #my $target_ip = PVE::Cluster::remote_node_ip($target_node);
+ if ($delete) {
+ $rpcenv->check($user, "/storage/$src_storeid", ["Datastore.Allocate"]);
+ }
- # you need to get this working (fails currently, because storage_migrate() uses
- # ssh to connect to local host (which is not needed
- my $sshinfo = PVE::SSHInfo::get_ssh_info($target_node);
- PVE::Storage::storage_migrate(
- $cfg,
- $src_volid,
- $sshinfo,
- $target_sid,
- { 'target_volname' => $target_volname },
- );
+ $rpcenv->check($user, "/storage/$dst_storeid", ["Datastore.AllocateSpace"]);
- print "DEBUG: end worker $upid\n";
+ my $worker = sub {
+ PVE::Storage::storage_check_enabled($cfg, $src_storeid, $src_node);
+ PVE::Storage::storage_check_enabled($cfg, $dst_storeid, $dst_node);
+ my $sshinfo;
+ $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node) if $src_node ne $dst_node;
+ my $opts = { 'target_volname' => $volname };
+ PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $dst_storeid, $opts);
+
+ if ($delete) {
+ &$volume_remove($cfg, $src_volid);
+ }
+ if ($src_node eq $dst_node) {
+ print "Copied volume '$src_volid' to '$dst_volid'\n";
+ } else {
+ print "Copied volume '$src_volid' on node '$src_node'"
+ . " to '$dst_volid' on node '$dst_node'\n";
+ }
};
return $rpcenv->fork_worker('imgcopy', undef, $user, $worker);
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 2/6] introduce $vtype+meta export formats
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 1/6] api: content: implement " Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 3/6] api: content: support copying backups between path based storages Filip Schauer
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
These new export formats include a JSON metadata header containing the
vtype and the format. This allows for future extensibility without
breaking backward compatibility when adding additional metadata.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Unchanged since v8
src/PVE/Storage.pm | 16 +++++-
src/PVE/Storage/Plugin.pm | 101 +++++++++++++++++++++++++++++++++-----
2 files changed, 103 insertions(+), 14 deletions(-)
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 6e87bac..6163780 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -47,7 +47,21 @@ use constant APIVER => 13;
# see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
use constant APIAGE => 4;
-our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];
+our $KNOWN_EXPORT_FORMATS = [
+ 'raw+size',
+ 'tar+size',
+ 'qcow2+size',
+ 'vmdk+size',
+ 'zfs',
+ 'btrfs',
+ 'images+meta',
+ 'rootdir+meta',
+ 'iso+meta',
+ 'vztmpl+meta',
+ 'backup+meta',
+ 'snippets+meta',
+ 'import+meta',
+];
# load standard plugins
PVE::Storage::DirPlugin->register();
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 6f2e434..3a83459 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -2094,6 +2094,25 @@ sub read_common_header($) {
return $size;
}
+sub write_meta_header($$) {
+ my ($fh, $meta) = @_;
+ my $meta_json = JSON::encode_json($meta);
+ $meta_json = Encode::encode('utf8', $meta_json);
+ syswrite($fh, pack("Q<", length($meta_json)), 8);
+ syswrite($fh, $meta_json, length($meta_json));
+}
+
+sub read_meta_header($) {
+ my ($fh) = @_;
+ sysread($fh, my $meta_size, 8);
+ $meta_size = unpack('Q<', $meta_size);
+ die "import: no meta size found in export header, aborting.\n" if !defined($meta_size);
+ sysread($fh, my $meta_json, $meta_size);
+ $meta_json = Encode::decode('utf8', $meta_json);
+ my $meta = JSON::decode_json($meta_json);
+ return $meta;
+}
+
# Export a volume into a file handle as a stream of desired format.
sub volume_export {
my ($class, $scfg, $storeid, $fh, $volname, $format, $snapshot, $base_snapshot, $with_snapshots)
@@ -2105,7 +2124,7 @@ sub volume_export {
my $err_msg = "volume export format $format not available for $class\n";
if ($scfg->{path} && !defined($snapshot) && !defined($base_snapshot)) {
my ($file) = $class->path($scfg, $volname, $storeid) or die $err_msg;
- my $file_format = ($class->parse_volname($volname))[6];
+ my ($vtype, $file_format) = ($class->parse_volname($volname))[0, 6];
my $size = file_size_info($file, undef, $file_format);
if ($format eq 'raw+size') {
@@ -2149,6 +2168,52 @@ sub volume_export {
output => '>&' . fileno($fh),
);
return;
+ } elsif ($format eq "$vtype+meta") {
+ die "format $file_format cannot be exported without snapshots\n"
+ if !$with_snapshots && $file_format =~ /^qcow2|vmdk$/;
+ die "format $file_format cannot be exported with snapshots\n"
+ if $with_snapshots && $file_format =~ /^raw|subvol$/;
+
+ my $data_format;
+ if ($file_format eq 'subvol') {
+ $data_format = 'tar';
+ } else {
+ $data_format = $file_format;
+ }
+
+ my $meta = {
+ vtype => $vtype,
+ format => $data_format,
+ };
+
+ write_meta_header($fh, $meta);
+ write_common_header($fh, $size);
+ if ($data_format =~ /^(raw|ova|ovf|qcow2|vmdk)$/) {
+ run_command(
+ ['dd', "if=$file", "bs=4k", "status=progress"],
+ output => '>&' . fileno($fh),
+ );
+ } elsif ($data_format eq 'tar') {
+ run_command(
+ ['tar', @COMMON_TAR_FLAGS, '-cf', '-', '-C', $file, '.'],
+ output => '>&' . fileno($fh),
+ );
+ } else {
+ run_command(
+ [
+ 'qemu-img',
+ 'convert',
+ '-f',
+ $file_format,
+ '-O',
+ 'raw',
+ $file,
+ '/dev/stdout',
+ ],
+ output => '>&' . fileno($fh),
+ );
+ }
+ return;
}
}
die $err_msg;
@@ -2160,11 +2225,11 @@ sub volume_export_formats {
my ($vtype, $format) = ($class->parse_volname($volname))[0, 6];
if ($with_snapshots) {
- return ($format . '+size') if ($format eq 'qcow2' || $format eq 'vmdk');
+ return ("$vtype+meta", $format . '+size') if $format =~ /^(qcow2|vmdk)$/;
return ();
}
- return ('tar+size') if $format eq 'subvol';
- return ('raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
+ return ("$vtype+meta", 'tar+size') if $format eq 'subvol';
+ return ("$vtype+meta", 'raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
}
return ();
}
@@ -2184,18 +2249,28 @@ sub volume_import {
$allow_rename,
) = @_;
- die "volume import format '$format' not available for $class\n"
- if $format !~ /^(raw|tar|qcow2|vmdk)\+size$/;
- my $data_format = $1;
+ my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format) =
+ $class->parse_volname($volname);
+
+ my $meta;
+ my $data_format;
+
+ if ($format =~ /^(images|rootdir|iso|vztmpl|backup|snippets|import)\+meta$/) {
+ die "volume type does not match import format\n" if $1 ne $vtype;
+ $meta = read_meta_header($fh);
+ die "volume type does not match metadata header\n" if $meta->{vtype} ne $vtype;
+ $data_format = $meta->{format};
+ } elsif ($format =~ /^(raw|tar|qcow2|vmdk)\+size$/) {
+ $data_format = $1;
+ } else {
+ die "volume import format '$format' not available for $class\n";
+ }
die "format $format cannot be imported without snapshots\n"
if !$with_snapshots && ($data_format eq 'qcow2' || $data_format eq 'vmdk');
die "format $format cannot be imported with snapshots\n"
if $with_snapshots && ($data_format eq 'raw' || $data_format eq 'tar');
- my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $file_format) =
- $class->parse_volname($volname);
-
# XXX: Should we bother with conversion routines at this level? This won't
# happen without manual CLI usage, so for now we just error out...
if (
@@ -2273,11 +2348,11 @@ sub volume_import_formats {
my ($vtype, $format) = ($class->parse_volname($volname))[0, 6];
if ($with_snapshots) {
- return ($format . '+size') if ($format eq 'qcow2' || $format eq 'vmdk');
+ return ("$vtype+meta", $format . '+size') if $format =~ /^(qcow2|vmdk)$/;
return ();
}
- return ('tar+size') if $format eq 'subvol';
- return ('raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
+ return ("$vtype+meta", 'tar+size') if $format eq 'subvol';
+ return ("$vtype+meta", 'raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
}
return ();
}
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 3/6] api: content: support copying backups between path based storages
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 1/6] api: content: implement " Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 2/6] introduce $vtype+meta export formats Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 4/6] storage: introduce decompress_archive_into_pipe helper Filip Schauer
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
This commit adds support for the "backup+meta" export format. When this
format is used, the notes and protection flag of the backup are included
in the metadata header.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
Unchanged since v8
src/PVE/API2/Storage/Content.pm | 10 ++++++++--
src/PVE/Storage/Plugin.pm | 25 ++++++++++++++++++++++++-
2 files changed, 32 insertions(+), 3 deletions(-)
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index 201d839..ea811ec 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -544,7 +544,7 @@ __PACKAGE__->register_method({
permissions => {
description => "If the --delete option is used, the 'Datastore.Allocate' privilege is"
. " required on the source storage. 'Datastore.AllocateSpace' is required on the target"
- . " storage.",
+ . " storage. When moving a backup, 'VM.Backup' is required on the VM or container.",
user => 'all',
},
protected => 1,
@@ -600,7 +600,7 @@ __PACKAGE__->register_method({
die "use pct move-volume or qm disk move\n"
if $vtype eq 'images' || $vtype eq 'rootdir';
die "moving volume of type '$vtype' not implemented\n"
- if !grep { $vtype eq $_ } qw(import iso snippets vztmpl);
+ if !grep { $vtype eq $_ } qw(backup import iso snippets vztmpl);
my ($dst_storeid, $dst_volname);
if ($param->{target} =~ m/:/) {
@@ -627,6 +627,12 @@ __PACKAGE__->register_method({
if ($delete) {
$rpcenv->check($user, "/storage/$src_storeid", ["Datastore.Allocate"]);
+
+ if ($vtype eq 'backup') {
+ my $protected =
+ PVE::Storage::get_volume_attribute($cfg, $src_volid, 'protected');
+ die "cannot delete protected backup\n" if $protected;
+ }
}
$rpcenv->check($user, "/storage/$dst_storeid", ["Datastore.AllocateSpace"]);
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 3a83459..437a677 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -2186,6 +2186,17 @@ sub volume_export {
format => $data_format,
};
+ if ($vtype eq 'backup') {
+ $meta->{protected} =
+ $class->get_volume_attribute($scfg, $storeid, $volname, 'protected')
+ ? JSON::true
+ : JSON::false;
+ $meta->{notes} = Encode::encode(
+ 'utf8',
+ $class->get_volume_attribute($scfg, $storeid, $volname, 'notes'),
+ ) // "";
+ }
+
write_meta_header($fh, $meta);
write_common_header($fh, $size);
if ($data_format =~ /^(raw|ova|ovf|qcow2|vmdk)$/) {
@@ -2229,6 +2240,7 @@ sub volume_export_formats {
return ();
}
return ("$vtype+meta", 'tar+size') if $format eq 'subvol';
+ return ('backup+meta') if $vtype eq 'backup';
return ("$vtype+meta", 'raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
}
return ();
@@ -2324,7 +2336,7 @@ sub volume_import {
warn $@ if $@;
die $err;
}
- } elsif (grep { $vtype eq $_ } qw(import iso snippets vztmpl)) {
+ } elsif (grep { $vtype eq $_ } qw(import iso snippets vztmpl backup)) {
eval {
run_command(['dd', "of=$file", 'conv=excl', 'bs=64k'], input => '<&' . fileno($fh));
};
@@ -2335,6 +2347,16 @@ sub volume_import {
}
die $err;
}
+
+ if ($vtype eq 'backup' && defined($meta)) {
+ if (defined($meta->{notes})) {
+ my $notes = Encode::decode('utf8', $meta->{notes});
+ $class->update_volume_attribute($scfg, $storeid, $volname, 'notes', $notes);
+ }
+
+ $class->update_volume_attribute($scfg, $storeid, $volname, 'protected', 1)
+ if $meta->{protected};
+ }
} else {
die "importing volume of type '$vtype' not implemented\n";
}
@@ -2352,6 +2374,7 @@ sub volume_import_formats {
return ();
}
return ("$vtype+meta", 'tar+size') if $format eq 'subvol';
+ return ('backup+meta') if $vtype eq 'backup';
return ("$vtype+meta", 'raw+size') if $vtype =~ /^(images|iso|snippets|vztmpl|import)$/;
}
return ();
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 4/6] storage: introduce decompress_archive_into_pipe helper
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
` (2 preceding siblings ...)
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 3/6] api: content: support copying backups between path based storages Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 5/6] support copying VMA backups to PBS Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 6/6] pvesm: add a copy-volume command Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
Extract the file decompression code into its own reusable subroutine.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
---
Unchanged since v8
src/PVE/Storage.pm | 68 +++++++++++++++++++++++++++-------------------
1 file changed, 40 insertions(+), 28 deletions(-)
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 6163780..1580929 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -1903,6 +1903,45 @@ sub extract_vzdump_config_tar {
return wantarray ? ($raw, $file) : $raw;
}
+sub decompress_archive_into_pipe {
+ my ($archive, $cmd, $outfunc) = @_;
+
+ my $info = archive_info($archive);
+ die "archive is not compressed\n" if !$info->{compression};
+ my $decompressor = $info->{decompressor};
+ my $full_cmd = [[@$decompressor, $archive], $cmd];
+
+ # lzop/zcat exits with 1 when the pipe is closed early,
+ # detect this and ignore the exit code later
+ my $broken_pipe;
+ my $errstring;
+ my $err = sub {
+ my $output = shift;
+ if (
+ $output =~ m/lzop: Broken pipe: <stdout>/
+ || $output =~ m/gzip: stdout: Broken pipe/
+ || $output =~ m/zstd: error 70 : Write error.*Broken pipe/
+ ) {
+ $broken_pipe = 1;
+ } elsif (!defined($errstring) && $output !~ m/^\s*$/) {
+ $errstring = "failed to decompress archive: $output\n";
+ }
+ };
+
+ my $rc = eval { run_command($full_cmd, outfunc => $outfunc, errfunc => $err, noerr => 1) };
+ my $rerr = $@;
+
+ $broken_pipe ||= $rc == 141; # broken pipe from cmd POV
+
+ if (!$errstring && !$broken_pipe && $rc != 0) {
+ die "$rerr\n" if $rerr;
+ die "archive decompression failed with exit code $rc\n";
+ }
+ die "$errstring\n" if $errstring;
+
+ return;
+}
+
sub extract_vzdump_config_vma {
my ($archive, $comp) = @_;
@@ -1914,34 +1953,7 @@ sub extract_vzdump_config_vma {
my $decompressor = $info->{decompressor};
if ($comp) {
- my $cmd = [[@$decompressor, $archive], ["vma", "config", "-"]];
-
- # lzop/zcat exits with 1 when the pipe is closed early by vma, detect this and ignore the exit code later
- my $broken_pipe;
- my $errstring;
- my $err = sub {
- my $output = shift;
- if (
- $output =~ m/lzop: Broken pipe: <stdout>/
- || $output =~ m/gzip: stdout: Broken pipe/
- || $output =~ m/zstd: error 70 : Write error.*Broken pipe/
- ) {
- $broken_pipe = 1;
- } elsif (!defined($errstring) && $output !~ m/^\s*$/) {
- $errstring = "Failed to extract config from VMA archive: $output\n";
- }
- };
-
- my $rc = eval { run_command($cmd, outfunc => $out, errfunc => $err, noerr => 1) };
- my $rerr = $@;
-
- $broken_pipe ||= $rc == 141; # broken pipe from vma POV
-
- if (!$errstring && !$broken_pipe && $rc != 0) {
- die "$rerr\n" if $rerr;
- die "config extraction failed with exit code $rc\n";
- }
- die "$errstring\n" if $errstring;
+ decompress_archive_into_pipe($archive, ["vma", "config", "-"], $out);
} else {
run_command(["vma", "config", $archive], outfunc => $out);
}
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 5/6] support copying VMA backups to PBS
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
` (3 preceding siblings ...)
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 4/6] storage: introduce decompress_archive_into_pipe helper Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 6/6] pvesm: add a copy-volume command Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
Extend the copy API to support copying VMA backups to a Proxmox Backup
Server.
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
debian/control | 1 +
src/PVE/API2/Storage/Content.pm | 20 ++++++++--
src/PVE/Storage/PBSPlugin.pm | 65 +++++++++++++++++++++++++++++++++
3 files changed, 82 insertions(+), 4 deletions(-)
diff --git a/debian/control b/debian/control
index 6bd55a2..40b0f5a 100644
--- a/debian/control
+++ b/debian/control
@@ -45,6 +45,7 @@ Depends: bzip2,
nfs-common,
proxmox-backup-client (>= 2.1.10~),
proxmox-backup-file-restore,
+ proxmox-vma-to-pbs (>= 0.0.2),
pve-cluster (>= 5.0-32),
smartmontools,
smbclient,
diff --git a/src/PVE/API2/Storage/Content.pm b/src/PVE/API2/Storage/Content.pm
index ea811ec..d6bb4ae 100644
--- a/src/PVE/API2/Storage/Content.pm
+++ b/src/PVE/API2/Storage/Content.pm
@@ -7,6 +7,7 @@ use PVE::SafeSyslog;
use PVE::Cluster;
use PVE::Storage;
use PVE::Storage::Common; # for 'pve-storage-image-format' standard option
+use PVE::Storage::PBSPlugin;
use PVE::INotify;
use PVE::Exception qw(raise_param_exc);
use PVE::RPCEnvironment;
@@ -640,10 +641,21 @@ __PACKAGE__->register_method({
my $worker = sub {
PVE::Storage::storage_check_enabled($cfg, $src_storeid, $src_node);
PVE::Storage::storage_check_enabled($cfg, $dst_storeid, $dst_node);
- my $sshinfo;
- $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node) if $src_node ne $dst_node;
- my $opts = { 'target_volname' => $volname };
- PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $dst_storeid, $opts);
+
+ my $src_cfg = PVE::Storage::storage_config($cfg, $src_storeid);
+ my $dst_cfg = PVE::Storage::storage_config($cfg, $dst_storeid);
+ if ($vtype eq 'backup' && $src_cfg->{path} && $dst_cfg->{type} eq 'pbs') {
+ my $protected =
+ PVE::Storage::get_volume_attribute($cfg, $src_volid, 'protected');
+ PVE::Storage::PBSPlugin::vma_to_pbs(
+ $ownervm, $src_path, $dst_cfg, $dst_storeid, $protected,
+ );
+ } else {
+ my $sshinfo;
+ $sshinfo = PVE::SSHInfo::get_ssh_info($dst_node) if $src_node ne $dst_node;
+ my $opts = { 'target_volname' => $dst_volname };
+ PVE::Storage::storage_migrate($cfg, $src_volid, $sshinfo, $dst_storeid, $opts);
+ }
if ($delete) {
&$volume_remove($cfg, $src_volid);
diff --git a/src/PVE/Storage/PBSPlugin.pm b/src/PVE/Storage/PBSPlugin.pm
index 17e285a..ce78404 100644
--- a/src/PVE/Storage/PBSPlugin.pm
+++ b/src/PVE/Storage/PBSPlugin.pm
@@ -7,6 +7,7 @@ use warnings;
use Encode qw(decode);
use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
+use File::Basename;
use IO::File;
use JSON;
use MIME::Base64 qw(decode_base64);
@@ -1002,4 +1003,68 @@ sub volume_has_feature {
return undef;
}
+sub vma_to_pbs {
+ my ($vmid, $source_path, $target_scfg, $target_storeid, $protected) = @_;
+
+ #my $source_plugin = PVE::Storage::Plugin->lookup($source_scfg->{type});
+ my $target_plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
+ my $info = PVE::Storage::archive_info($source_path);
+ die "moving non-VMA backups to a Proxmox Backup Server is not supported\n"
+ if $info->{format} ne 'vma';
+
+ my $repo = PVE::PBSClient::get_repository($target_scfg);
+ my $fingerprint = $target_scfg->{fingerprint};
+ my $password = PVE::Storage::PBSPlugin::pbs_password_file_name($target_scfg, $target_storeid);
+ my $namespace = $target_scfg->{namespace};
+ my $keyfile =
+ PVE::Storage::PBSPlugin::pbs_encryption_key_file_name($target_scfg, $target_storeid);
+ my $master_keyfile =
+ PVE::Storage::PBSPlugin::pbs_master_pubkey_file_name($target_scfg, $target_storeid);
+
+ my $comp = $info->{compression};
+ my $backup_time = $info->{ctime};
+ my $source_dirname = dirname($source_path);
+ my $log_file_path = "$source_dirname/$info->{logfilename}";
+ my $notes_file_path = "$source_dirname/$info->{notesfilename}";
+
+ my $vma_to_pbs_cmd = [
+ "vma-to-pbs",
+ "--repository",
+ $repo,
+ "--vmid",
+ $vmid,
+ "--fingerprint",
+ $fingerprint,
+ "--password-file",
+ $password,
+ "--backup-time",
+ $backup_time,
+ "--compress",
+ ];
+
+ push @$vma_to_pbs_cmd, "--ns", $namespace if $namespace;
+ push @$vma_to_pbs_cmd, "--log-file", $log_file_path if -e $log_file_path;
+ push @$vma_to_pbs_cmd, "--notes-file", $notes_file_path if -e $notes_file_path;
+ push @$vma_to_pbs_cmd, "--encrypt", "--keyfile", $keyfile if -e $keyfile;
+ push @$vma_to_pbs_cmd, "--master-keyfile", $master_keyfile if -e $master_keyfile;
+
+ if ($comp) {
+ PVE::Storage::decompress_archive_into_pipe($source_path, $vma_to_pbs_cmd);
+ } else {
+ push @$vma_to_pbs_cmd, $source_path;
+ run_command($vma_to_pbs_cmd);
+ }
+
+ if ($protected) {
+ my $target_volid =
+ PVE::Storage::PBSPlugin::print_volid($target_storeid, 'vm', $vmid, $backup_time);
+ my (undef, $target_volname) = PVE::Storage::parse_volume_id($target_volid, 0);
+ $target_plugin->update_volume_attribute(
+ $target_scfg, $target_storeid, $target_volname, 'protected', 1,
+ );
+ }
+
+ return;
+}
+
1;
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [pve-devel] [PATCH storage v9 6/6] pvesm: add a copy-volume command
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
` (4 preceding siblings ...)
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 5/6] support copying VMA backups to PBS Filip Schauer
@ 2025-12-17 13:36 ` Filip Schauer
5 siblings, 0 replies; 7+ messages in thread
From: Filip Schauer @ 2025-12-17 13:36 UTC (permalink / raw)
To: pve-devel
The method can be called from the PVE shell with `pvesm copy-volume`:
```
pvesm copy-volume \
<source volume> <target volume or storage> \
[--target-node <node>] [--delete]
```
For example to copy a VMA backup to a Proxmox Backup Server:
```
pvesm copy-volume \
local:backup/vzdump-qemu-100-2024_06_25-13_08_56.vma.zst pbs
```
Or copy a container template to another node and delete the source:
```
pvesm copy-volume \
local:vztmpl/devuan-4.0-standard_4.0_amd64.tar.gz local \
--target-node pvenode2 --delete
```
Or copy an ISO and rename it:
```
pvesm copy-volume \
local:iso/debian-13.1.0-amd64-netinst.iso cephfs:iso/debian13.1.iso
```
Signed-off-by: Filip Schauer <f.schauer@proxmox.com>
---
src/PVE/CLI/pvesm.pm | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/src/PVE/CLI/pvesm.pm b/src/PVE/CLI/pvesm.pm
index 06bc4c9..4c630c9 100755
--- a/src/PVE/CLI/pvesm.pm
+++ b/src/PVE/CLI/pvesm.pm
@@ -769,6 +769,12 @@ our $cmddef = {
print "APIAGE $res->{apiage}\n";
},
],
+ 'copy-volume' => [
+ "PVE::API2::Storage::Content",
+ 'copy',
+ ['volume', 'target'],
+ { node => $nodename },
+ ],
'prune-backups' => [
__PACKAGE__,
'prunebackups',
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-12-17 13:37 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-12-17 13:36 [pve-devel] [PATCH storage v9 0/6] support copying volumes between storages Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 1/6] api: content: implement " Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 2/6] introduce $vtype+meta export formats Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 3/6] api: content: support copying backups between path based storages Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 4/6] storage: introduce decompress_archive_into_pipe helper Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 5/6] support copying VMA backups to PBS Filip Schauer
2025-12-17 13:36 ` [pve-devel] [PATCH storage v9 6/6] pvesm: add a copy-volume command Filip Schauer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.