From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <w.bumiller@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 388CC70F7D
for <pve-devel@lists.proxmox.com>; Wed, 9 Jun 2021 15:19:37 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id D49F925427
for <pve-devel@lists.proxmox.com>; Wed, 9 Jun 2021 15:19:06 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id 034972506F
for <pve-devel@lists.proxmox.com>; Wed, 9 Jun 2021 15:18:59 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id D0B954674C
for <pve-devel@lists.proxmox.com>; Wed, 9 Jun 2021 15:18:58 +0200 (CEST)
From: Wolfgang Bumiller <w.bumiller@proxmox.com>
To: pve-devel@lists.proxmox.com
Date: Wed, 9 Jun 2021 15:18:48 +0200
Message-Id: <20210609131852.167416-5-w.bumiller@proxmox.com>
X-Mailer: git-send-email 2.30.2
In-Reply-To: <20210609131852.167416-1-w.bumiller@proxmox.com>
References: <20210609131852.167416-1-w.bumiller@proxmox.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.901 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
KAM_SHORT 0.001 Use of a URL Shortener for very short URL
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [gnu.org, pvesm.pm, storage.pm]
Subject: [pve-devel] [PATCH storage 3/4] update import/export storage API
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 09 Jun 2021 13:19:37 -0000
This bumps APIVER, but also APIAGE, see below.
The import methods (volume_import, volume_import_formats):
This additionally gets the '$snapshot' parameter which is
already present on the export side as an informational
piece to know which of the snapshots is the *current* one.
The current "disk" state will be set to this snapshot.
This, too, is required for our btrfs implementation.
`volume_import_formats` can obviously not make much
*use* of this parameter, but it'll still be useful to know
that the information is actually available in the import
call, so its presence will be checked in the btrfs
implementation.
Currently this is intended to be used for btrfs send/recv
support, which in theory could also get additional metadata
similar to how we do the "tar+size" format, however, we
currently only really use this within this repository in
storage_migrate() which has this information readily
available anyway.
On the export side (volume_export, volume_export_formats):
The `$with_snapshots` option can now also be an ordered
array of snapshots to include, as a hint for storages
which need this. (As of the next commit this is only
btrfs, and only when also specifying a base snapshot,
which is a case we can currently not run into except on
the command line interface.)
The current providers of the `with_snapshot` option will
still treat it as a boolean (since eg. for ZFS you cannot
really "skip" snapshots AFAIK).
This is mainly intended for storages which do not have a
strong association between snapshots and the originals, or
an ordering (eg. btrfs and lvm-thin allow creating
arbitrary snapshot trees, and with btrfs you can even
create a "circular" connection between subvolumes, also we
could consider reflink based copies snapshots on xfs in
the future maybe?)
While `import_formats` and `export_formats` now take the same
parameters, I appended the `$snapshot` parameter to the end of
`import_formats` to the end for compatibility.
Therefore we also bump APIAGE, as apart from the btrfs
storage, none of our other storages need the new parameter.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
PVE/CLI/pvesm.pm | 23 +++++++++++++++++++++--
PVE/Storage.pm | 48 ++++++++++++++++++++++++++++++++----------------
2 files changed, 53 insertions(+), 18 deletions(-)
diff --git a/PVE/CLI/pvesm.pm b/PVE/CLI/pvesm.pm
index d28f1ba..b22f759 100755
--- a/PVE/CLI/pvesm.pm
+++ b/PVE/CLI/pvesm.pm
@@ -276,12 +276,23 @@ __PACKAGE__->register_method ({
optional => 1,
default => 0,
},
+ 'snapshot-list' => {
+ description => "Ordered list of snapshots to transfer",
+ type => 'string',
+ format => 'string-list',
+ optional => 1,
+ },
},
},
returns => { type => 'null' },
code => sub {
my ($param) = @_;
+ my $with_snapshots = $param->{'with-snapshots'};
+ if (defined(my $list = $param->{'snapshot-list'})) {
+ $with_snapshots = PVE::Tools::split_list($list);
+ }
+
my $filename = $param->{filename};
my $outfh;
@@ -295,7 +306,7 @@ __PACKAGE__->register_method ({
eval {
my $cfg = PVE::Storage::config();
PVE::Storage::volume_export($cfg, $outfh, $param->{volume}, $param->{format},
- $param->{snapshot}, $param->{base}, $param->{'with-snapshots'});
+ $param->{snapshot}, $param->{base}, $with_snapshots);
};
my $err = $@;
if ($filename ne '-') {
@@ -361,6 +372,13 @@ __PACKAGE__->register_method ({
optional => 1,
default => 0,
},
+ snapshot => {
+ description => "The current-state snapshot if the stream contains snapshots",
+ type => 'string',
+ pattern => qr/[a-z0-9_\-]{1,40}/i,
+ maxLength => 40,
+ optional => 1,
+ },
},
},
returns => { type => 'string' },
@@ -436,7 +454,8 @@ __PACKAGE__->register_method ({
my $volume = $param->{volume};
my $delete = $param->{'delete-snapshot'};
my $imported_volid = PVE::Storage::volume_import($cfg, $infh, $volume, $param->{format},
- $param->{base}, $param->{'with-snapshots'}, $param->{'allow-rename'});
+ $param->{base}, $param->{'with-snapshots'}, $param->{'allow-rename'},
+ $param->{snapshot});
PVE::Storage::volume_snapshot_delete($cfg, $imported_volid, $delete)
if defined($delete);
return $imported_volid;
diff --git a/PVE/Storage.pm b/PVE/Storage.pm
index f9f8d16..15fcedf 100755
--- a/PVE/Storage.pm
+++ b/PVE/Storage.pm
@@ -41,11 +41,11 @@ use PVE::Storage::PBSPlugin;
use PVE::Storage::BTRFSPlugin;
# Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 8;
+use constant APIVER => 9;
# Age is the number of versions we're backward compatible with.
# This is like having 'current=APIVER' and age='APIAGE' in libtool,
# see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 7;
+use constant APIAGE => 8;
# load standard plugins
PVE::Storage::DirPlugin->register();
@@ -716,7 +716,8 @@ sub storage_migrate {
my $send = ['pvesm', 'export', $volid, $format, '-', '-with-snapshots', $with_snapshots];
my $recv = [@$ssh, '--', 'pvesm', 'import', $target_volid, $format, $import_fn, '-with-snapshots', $with_snapshots];
if (defined($snapshot)) {
- push @$send, '-snapshot', $snapshot
+ push @$send, '-snapshot', $snapshot;
+ push @$recv, '-snapshot', $snapshot;
}
if ($migration_snapshot) {
push @$recv, '-delete-snapshot', $snapshot;
@@ -1716,7 +1717,7 @@ sub prune_mark_backup_group {
}
}
-sub volume_export {
+sub volume_export : prototype($$$$$$$) {
my ($cfg, $fh, $volid, $format, $snapshot, $base_snapshot, $with_snapshots) = @_;
my ($storeid, $volname) = parse_volume_id($volid, 1);
@@ -1727,18 +1728,27 @@ sub volume_export {
$snapshot, $base_snapshot, $with_snapshots);
}
-sub volume_import {
- my ($cfg, $fh, $volid, $format, $base_snapshot, $with_snapshots, $allow_rename) = @_;
+sub volume_import : prototype($$$$$$$$) {
+ my ($cfg, $fh, $volid, $format, $base_snapshot, $with_snapshots, $allow_rename, $snapshot) = @_;
my ($storeid, $volname) = parse_volume_id($volid, 1);
die "cannot import into volume '$volid'\n" if !$storeid;
my $scfg = storage_config($cfg, $storeid);
my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
- return $plugin->volume_import($scfg, $storeid, $fh, $volname, $format,
- $base_snapshot, $with_snapshots, $allow_rename) // $volid;
-}
-
-sub volume_export_formats {
+ return $plugin->volume_import(
+ $scfg,
+ $storeid,
+ $fh,
+ $volname,
+ $format,
+ $base_snapshot,
+ $with_snapshots,
+ $allow_rename,
+ $snapshot,
+ ) // $volid;
+}
+
+sub volume_export_formats : prototype($$$$$) {
my ($cfg, $volid, $snapshot, $base_snapshot, $with_snapshots) = @_;
my ($storeid, $volname) = parse_volume_id($volid, 1);
@@ -1750,21 +1760,27 @@ sub volume_export_formats {
$with_snapshots);
}
-sub volume_import_formats {
- my ($cfg, $volid, $base_snapshot, $with_snapshots) = @_;
+sub volume_import_formats : prototype($$$$$) {
+ my ($cfg, $volid, $base_snapshot, $with_snapshots, $snapshot) = @_;
my ($storeid, $volname) = parse_volume_id($volid, 1);
return if !$storeid;
my $scfg = storage_config($cfg, $storeid);
my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
- return $plugin->volume_import_formats($scfg, $storeid, $volname,
- $base_snapshot, $with_snapshots);
+ return $plugin->volume_import_formats(
+ $scfg,
+ $storeid,
+ $volname,
+ $base_snapshot,
+ $with_snapshots,
+ $snapshot,
+ );
}
sub volume_transfer_formats {
my ($cfg, $src_volid, $dst_volid, $snapshot, $base_snapshot, $with_snapshots) = @_;
my @export_formats = volume_export_formats($cfg, $src_volid, $snapshot, $base_snapshot, $with_snapshots);
- my @import_formats = volume_import_formats($cfg, $dst_volid, $base_snapshot, $with_snapshots);
+ my @import_formats = volume_import_formats($cfg, $dst_volid, $base_snapshot, $with_snapshots, $snapshot);
my %import_hash = map { $_ => 1 } @import_formats;
my @common = grep { $import_hash{$_} } @export_formats;
return @common;
--
2.30.2