From: Fabian Ebner <f.ebner@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: Re: [pve-devel] [PATCH v4 storage 3/7] Add API and pvesm call for prune_backups
Date: Mon, 13 Jul 2020 09:04:32 +0200 [thread overview]
Message-ID: <4f9d364c-bc6a-33c4-ff15-0508ed644239@proxmox.com> (raw)
In-Reply-To: <20200709124547.2913-4-f.ebner@proxmox.com>
Am 09.07.20 um 14:45 schrieb Fabian Ebner:
> For the pvesm call use a wrapper and a --dry-run option to redirect
> to the correct API call.
>
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
>
> Changes from v3:
> * allow filtering by type
> * pvesm: redirect to correct API call according to --dry-run
>
> PVE/API2/Storage/Makefile | 2 +-
> PVE/API2/Storage/PruneBackups.pm | 164 +++++++++++++++++++++++++++++++
> PVE/API2/Storage/Status.pm | 7 ++
> PVE/CLI/pvesm.pm | 126 ++++++++++++++++++++++++
> 4 files changed, 298 insertions(+), 1 deletion(-)
> create mode 100644 PVE/API2/Storage/PruneBackups.pm
>
> diff --git a/PVE/API2/Storage/Makefile b/PVE/API2/Storage/Makefile
> index a33525b..3f667e8 100644
> --- a/PVE/API2/Storage/Makefile
> +++ b/PVE/API2/Storage/Makefile
> @@ -1,5 +1,5 @@
>
> -SOURCES= Content.pm Status.pm Config.pm
> +SOURCES= Content.pm Status.pm Config.pm PruneBackups.pm
>
> .PHONY: install
> install:
> diff --git a/PVE/API2/Storage/PruneBackups.pm b/PVE/API2/Storage/PruneBackups.pm
> new file mode 100644
> index 0000000..a84d1c8
> --- /dev/null
> +++ b/PVE/API2/Storage/PruneBackups.pm
> @@ -0,0 +1,164 @@
> +package PVE::API2::Storage::PruneBackups;
> +
> +use strict;
> +use warnings;
> +
> +use PVE::Cluster;
> +use PVE::JSONSchema qw(get_standard_option);
> +use PVE::RESTHandler;
> +use PVE::RPCEnvironment;
> +use PVE::Storage;
> +use PVE::Tools qw(extract_param);
> +
> +use base qw(PVE::RESTHandler);
> +
> +__PACKAGE__->register_method ({
> + name => 'dryrun',
> + path => '',
> + method => 'GET',
> + description => "Get prune information for backups. NOTE: this is only a preview and might not be exactly " .
> + "what a subsequent prune call does, if the hour changes or if backups are removed/added " .
The part "if the hour changes or" should be removed here. It's true that
a new backup can change the pruning selection in different ways
depending on when it is created, but as long as there's the same backups
on the storage, the pruning selection does *not* change.
The same applies to the text further below...
> + "in the meantime.",
> + permissions => {
> + check => ['perm', '/storage/{storage}', ['Datastore.Audit', 'Datastore.AllocateSpace'], any => 1],
> + },
> + protected => 1,
> + proxyto => 'node',
> + parameters => {
> + additionalProperties => 0,
> + properties => {
> + node => get_standard_option('pve-node'),
> + storage => get_standard_option('pve-storage-id', {
> + completion => \&PVE::Storage::complete_storage_enabled,
> + }),
> + 'prune-backups' => get_standard_option('prune-backups', {
> + description => "Use these retention options instead of those from the storage configuration.",
> + optional => 1,
> + }),
> + type => {
> + description => "Either 'qemu' or 'lxc'. Only consider backups for guests of this type.",
> + type => 'string',
> + optional => 1,
> + enum => ['qemu', 'lxc'],
> + },
> + vmid => get_standard_option('pve-vmid', {
> + description => "Only consider backups for this guest.",
> + optional => 1,
> + completion => \&PVE::Cluster::complete_vmid,
> + }),
> + },
> + },
> + returns => {
> + type => 'array',
> + items => {
> + type => 'object',
> + properties => {
> + volid => {
> + description => "Backup volume ID.",
> + type => 'string',
> + },
> + 'ctime' => {
> + description => "Creation time of the backup (seconds since the UNIX epoch).",
> + type => 'integer',
> + },
> + 'mark' => {
> + description => "Whether the backup would be kept or removed. For backups that don't " .
> + "use the standard naming scheme, it's 'protected'.",
> + type => 'string',
> + },
> + type => {
> + description => "One of 'qemu', 'lxc', 'openvz' or 'unknown'.",
> + type => 'string',
> + },
> + 'vmid' => {
> + description => "The VM the backup belongs to.",
> + type => 'integer',
> + optional => 1,
> + },
> + },
> + },
> + },
> + code => sub {
> + my ($param) = @_;
> +
> + my $cfg = PVE::Storage::config();
> +
> + my $vmid = extract_param($param, 'vmid');
> + my $type = extract_param($param, 'type');
> + my $storeid = extract_param($param, 'storage');
> +
> + my $prune_backups = extract_param($param, 'prune-backups');
> + $prune_backups = PVE::JSONSchema::parse_property_string('prune-backups', $prune_backups)
> + if defined($prune_backups);
> +
> + return PVE::Storage::prune_backups($cfg, $storeid, $prune_backups, $vmid, $type, 1);
> + }});
> +
> +__PACKAGE__->register_method ({
> + name => 'delete',
> + path => '',
> + method => 'DELETE',
> + description => "Prune backups. Only those using the standard naming scheme are considered.",
> + permissions => {
> + description => "You need the 'Datastore.Allocate' privilege on the storage " .
> + "(or if a VM ID is specified, 'Datastore.AllocateSpace' and 'VM.Backup' for the VM).",
> + user => 'all',
> + },
> + protected => 1,
> + proxyto => 'node',
> + parameters => {
> + additionalProperties => 0,
> + properties => {
> + node => get_standard_option('pve-node'),
> + storage => get_standard_option('pve-storage-id', {
> + completion => \&PVE::Storage::complete_storage,
> + }),
> + 'prune-backups' => get_standard_option('prune-backups', {
> + description => "Use these retention options instead of those from the storage configuration.",
> + }),
> + type => {
> + description => "Either 'qemu' or 'lxc'. Only consider backups for guests of this type.",
> + type => 'string',
> + optional => 1,
> + enum => ['qemu', 'lxc'],
> + },
> + vmid => get_standard_option('pve-vmid', {
> + description => "Only prune backups for this VM.",
> + completion => \&PVE::Cluster::complete_vmid,
> + optional => 1,
> + }),
> + },
> + },
> + returns => { type => 'string' },
> + code => sub {
> + my ($param) = @_;
> +
> + my $rpcenv = PVE::RPCEnvironment::get();
> + my $authuser = $rpcenv->get_user();
> +
> + my $cfg = PVE::Storage::config();
> +
> + my $vmid = extract_param($param, 'vmid');
> + my $type = extract_param($param, 'type');
> + my $storeid = extract_param($param, 'storage');
> +
> + my $prune_backups = extract_param($param, 'prune-backups');
> + $prune_backups = PVE::JSONSchema::parse_property_string('prune-backups', $prune_backups)
> + if defined($prune_backups);
> +
> + if (defined($vmid)) {
> + $rpcenv->check($authuser, "/storage/$storeid", ['Datastore.AllocateSpace']);
> + $rpcenv->check($authuser, "/vms/$vmid", ['VM.Backup']);
> + } else {
> + $rpcenv->check($authuser, "/storage/$storeid", ['Datastore.Allocate']);
> + }
> +
> + my $id = (defined($vmid) ? "$vmid@" : '') . $storeid;
> + my $worker = sub {
> + PVE::Storage::prune_backups($cfg, $storeid, $prune_backups, $vmid, $type, 0);
> + };
> +
> + return $rpcenv->fork_worker('prunebackups', $id, $authuser, $worker);
> + }});
> +
> +1;
> diff --git a/PVE/API2/Storage/Status.pm b/PVE/API2/Storage/Status.pm
> index d9d9b36..d12643f 100644
> --- a/PVE/API2/Storage/Status.pm
> +++ b/PVE/API2/Storage/Status.pm
> @@ -11,6 +11,7 @@ use PVE::Cluster;
> use PVE::RRD;
> use PVE::Storage;
> use PVE::API2::Storage::Content;
> +use PVE::API2::Storage::PruneBackups;
> use PVE::RESTHandler;
> use PVE::RPCEnvironment;
> use PVE::JSONSchema qw(get_standard_option);
> @@ -18,6 +19,11 @@ use PVE::Exception qw(raise_param_exc);
>
> use base qw(PVE::RESTHandler);
>
> +__PACKAGE__->register_method ({
> + subclass => "PVE::API2::Storage::PruneBackups",
> + path => '{storage}/prunebackups',
> +});
> +
> __PACKAGE__->register_method ({
> subclass => "PVE::API2::Storage::Content",
> # set fragment delimiter (no subdirs) - we need that, because volume
> @@ -214,6 +220,7 @@ __PACKAGE__->register_method ({
> { subdir => 'upload' },
> { subdir => 'rrd' },
> { subdir => 'rrddata' },
> + { subdir => 'prunebackups' },
> ];
>
> return $res;
> diff --git a/PVE/CLI/pvesm.pm b/PVE/CLI/pvesm.pm
> index f223c92..405224c 100755
> --- a/PVE/CLI/pvesm.pm
> +++ b/PVE/CLI/pvesm.pm
> @@ -12,8 +12,10 @@ use PVE::Cluster;
> use PVE::INotify;
> use PVE::RPCEnvironment;
> use PVE::Storage;
> +use PVE::Tools qw(extract_param);
> use PVE::API2::Storage::Config;
> use PVE::API2::Storage::Content;
> +use PVE::API2::Storage::PruneBackups;
> use PVE::API2::Storage::Status;
> use PVE::JSONSchema qw(get_standard_option);
> use PVE::PTY;
> @@ -720,6 +722,99 @@ __PACKAGE__->register_method ({
> return PVE::Storage::scan_zfs();
> }});
>
> +__PACKAGE__->register_method ({
> + name => 'prunebackups',
> + path => 'prunebackups',
> + method => 'GET',
> + description => "Prune backups. This is only a wrapper for the proper API endpoints.",
> + protected => 1,
> + proxyto => 'node',
> + parameters => {
> + additionalProperties => 0,
> + properties => {
> + 'dry-run' => {
> + description => "Only show what would be pruned, don't delete anything.",
> + type => 'boolean',
> + optional => 1,
> + },
> + node => get_standard_option('pve-node'),
> + storage => get_standard_option('pve-storage-id', {
> + completion => \&PVE::Storage::complete_storage_enabled,
> + }),
> + 'prune-backups' => get_standard_option('prune-backups', {
> + description => "Use these retention options instead of those from the storage configuration.",
> + optional => 1,
> + }),
> + type => {
> + description => "Either 'qemu' or 'lxc'. Only consider backups for guests of this type.",
> + type => 'string',
> + optional => 1,
> + enum => ['qemu', 'lxc'],
> + },
> + vmid => get_standard_option('pve-vmid', {
> + description => "Only consider backups for this guest.",
> + optional => 1,
> + completion => \&PVE::Cluster::complete_vmid,
> + }),
> + },
> + },
> + returns => {
> + type => 'object',
> + properties => {
> + dryrun => {
> + description => 'If it was a dry run or not. The list will only be defined in that case.',
> + type => 'boolean',
> + },
> + list => {
> + type => 'array',
> + items => {
> + type => 'object',
> + properties => {
> + volid => {
> + description => "Backup volume ID.",
> + type => 'string',
> + },
> + 'ctime' => {
> + description => "Creation time of the backup (seconds since the UNIX epoch).",
> + type => 'integer',
> + },
> + 'mark' => {
> + description => "Whether the backup would be kept or removed. For backups that don't " .
> + "use the standard naming scheme, it's 'protected'.",
> + type => 'string',
> + },
> + type => {
> + description => "One of 'qemu', 'lxc', 'openvz' or 'unknown'.",
> + type => 'string',
> + },
> + 'vmid' => {
> + description => "The VM the backup belongs to.",
> + type => 'integer',
> + optional => 1,
> + },
> + },
> + },
> + },
> + },
> + },
> + code => sub {
> + my ($param) = @_;
> +
> + my $dryrun = extract_param($param, 'dry-run') ? 1 : 0;
> +
> + my $list = [];
> + if ($dryrun) {
> + $list = PVE::API2::Storage::PruneBackups->dryrun($param);
> + } else {
> + PVE::API2::Storage::PruneBackups->delete($param);
> + }
> +
> + return {
> + dryrun => $dryrun,
> + list => $list,
> + };
> + }});
> +
> our $cmddef = {
> add => [ "PVE::API2::Storage::Config", 'create', ['type', 'storage'] ],
> set => [ "PVE::API2::Storage::Config", 'update', ['storage'] ],
> @@ -819,6 +914,37 @@ our $cmddef = {
> print "APIVER $res->{apiver}\n";
> print "APIAGE $res->{apiage}\n";
> }],
> + 'prune-backups' => [ __PACKAGE__, 'prunebackups', ['storage'], { node => $nodename }, sub {
> + my $res = shift;
> +
> + my ($dryrun, $list) = ($res->{dryrun}, $res->{list});
> +
> + return if !$dryrun;
> +
> + print "NOTE: this is only a preview and might not be exactly what a subsequent prune call does,\n" .
> + "if the hour changes or if backups are removed/added in the meantime.\n\n";
...here
> +
> + my @sorted = sort {
> + my $vmcmp = PVE::Tools::safe_compare($a->{vmid}, $b->{vmid}, sub { $_[0] <=> $_[1] });
> + return $vmcmp if $vmcmp ne 0;
> + return $a->{ctime} <=> $b->{ctime};
> + } @{$list};
> +
> + my $maxlen = 0;
> + foreach my $backup (@sorted) {
> + my $volid = $backup->{volid};
> + $maxlen = length($volid) if length($volid) > $maxlen;
> + }
> + $maxlen+=1;
> +
> + printf("%-${maxlen}s %15s %10s\n", 'Backup', 'Backup-ID', 'Prune-Mark');
> + foreach my $backup (@sorted) {
> + my $type = $backup->{type};
> + my $vmid = $backup->{vmid};
> + my $backup_id = defined($vmid) ? "$type/$vmid" : "$type";
> + printf("%-${maxlen}s %15s %10s\n", $backup->{volid}, $backup_id, $backup->{mark});
> + }
> + }],
> };
>
> 1;
>
next prev parent reply other threads:[~2020-07-13 7:04 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-09 12:45 [pve-devel] [PATCH-SERIES v4] fix #2649: introduce prune-backups property for storages supporting backups Fabian Ebner
2020-07-09 12:45 ` [pve-devel] [PATCH v4 storage 1/7] Introduce prune-backups property for directory-based storages Fabian Ebner
2020-07-24 17:07 ` [pve-devel] applied: " Thomas Lamprecht
2020-07-09 12:45 ` [pve-devel] [PATCH v4 storage 2/7] Add prune_backups to storage API Fabian Ebner
2020-07-24 17:07 ` [pve-devel] applied: " Thomas Lamprecht
2020-07-09 12:45 ` [pve-devel] [PATCH v4 storage 3/7] Add API and pvesm call for prune_backups Fabian Ebner
2020-07-13 7:04 ` Fabian Ebner [this message]
2020-07-24 17:08 ` Thomas Lamprecht
2020-07-09 12:45 ` [pve-devel] [PATCH v4 guest-common 4/7] Add prune-backups option to vzdump parameters Fabian Ebner
2020-08-20 15:39 ` [pve-devel] applied: " Thomas Lamprecht
2020-07-09 12:45 ` [pve-devel] [PATCH v4 manager 5/7] make use of archive_info and archive_remove Fabian Ebner
2020-08-21 11:31 ` [pve-devel] applied: " Thomas Lamprecht
2020-07-09 12:45 ` [pve-devel] [PATCH v4 manager 6/7] Allow prune-backups as an alternative to maxfiles Fabian Ebner
2020-07-09 12:45 ` [pve-devel] [PATCH v4 manager 7/7] Always use prune-backups instead of maxfiles internally Fabian Ebner
2020-08-21 11:33 ` Thomas Lamprecht
2020-08-24 7:25 ` Fabian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4f9d364c-bc6a-33c4-ff15-0508ed644239@proxmox.com \
--to=f.ebner@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox