[pmg-devel] [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration

[pmg-devel] [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration

[Stoiko Ivanov]

changes v2->v3:
* rebased the gui-patches to latest master
* added param_mapping from pve-storage/pvesm (provide an interactive password
  entry)

cover-letter for v2:
changes v1->v2:
* renamed PBSTools to PBSClient, and made it a class (handling the config and
  backup-operations of one PBS instance)
* dropped encryption-key support from the GUI, the API calls and the
  SectionConfig - will be submitted again, when we have a clean solution in PVE
* dropped explicit prune support - each group now gets pruned directly after a
  backup
* one small fixup in debian/control (I mistakenly added two ',' in the
  proxmox-backup-client dependency)
* added /etc/pmg/pbs to the cluster-sync

cover-letter for v1:
changes RFC->v1:
* moved the potentially reusable parts to pve-common (PBSTools.pm, and 2
  functions in Systemd.pm)
* added GUI support (mostly adapted from the LDAPConfig) - huge thanks to
  Dominik for his patience and help!
* added support for encrypted backups
* added support for pruning backups
* added a systemd-timer, which runs a daily prune based on the configured
  settings

differences from the PBS storage-plugin config in PVE:
* the keep-options are kept as separate options instead of using a property
  string for all (like they are in the datastore config in PBS)


cover-letter from the RFC:
This series is a minimal proof-of-concept for integrating PBS into PMG.

The code needs quite a bit of cleanup, and more testing, however I'll send it
as RFC, for an initial sanity-check - to see if I missed some needed
functionality.

It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd
unit manipulation from the PVE::API2::Disks::Directory) and I'd like to
refactor the common parts into pve-common.

The backup consists of the same files, that end up in a locally generated
PMG Backup (instead of creating a tar.gz we send the data to a PBS instance).

What works:
* creating and manipulating 'remotes' (PBS instances where we back up to)
* creating, restoring and forgetting backups to those remotes
* restoring a backup from another PMG installation from a configured remote
* creating schedules for backups (systemd-timer units) for periodic backups
via CLI (pmgbackup, pmgsh)

What's missing:
* quite a bit of refactoring (e.g. move the systemd-timer helpers and most of
  PBSTools to pve-common (and reuse it in the PBS Storage plugin)
  * encryption support
  * pruneing support
  * testing
  * GUI
  * Documentation

  The first patch is a small cosmetic cleanup, and independent of the series.

  A sample session (which I used for testing:
  ```
  pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint  ff:ff:...
  pmgbackup remote list
  pmgbackup pbsjob run pbs1
  pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1
  pmgbackup pbsjob run pbs1
  pmgbackup pbsjob list_backups pbs1
  pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s'
  systemctl list-timers --all
  ```

pve-common:
Stoiko Ivanov (1):
  add PBSClient module

 src/Makefile         |   1 +
 src/PVE/PBSClient.pm | 305 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 306 insertions(+)
 create mode 100644 src/PVE/PBSClient.pm

pmg-api:
Stoiko Ivanov (7):
  debian: drop duplicate ',' in dependencies
  add initial SectionConfig for PBS
  Add API2 module for PBS configuration
  Add API2 module for per-node backups to PBS
  pbs-integration: add CLI calls to pmgbackup
  add scheduled backup to PBS remotes
  add /etc/pmg/pbs to cluster-sync

 debian/control                |   2 +-
 debian/dirs                   |   1 +
 debian/pmg-pbsbackup@.service |   6 +
 debian/rules                  |   1 +
 src/Makefile                  |   6 +-
 src/PMG/API2/Config.pm        |   7 +
 src/PMG/API2/Nodes.pm         |   7 +
 src/PMG/API2/PBS/Job.pm       | 501 ++++++++++++++++++++++++++++++++++
 src/PMG/API2/PBS/Remote.pm    | 231 ++++++++++++++++
 src/PMG/CLI/pmgbackup.pm      |  60 ++++
 src/PMG/Cluster.pm            |   1 +
 src/PMG/PBSConfig.pm          | 195 +++++++++++++
 src/PMG/PBSSchedule.pm        | 104 +++++++
 13 files changed, 1120 insertions(+), 2 deletions(-)
 create mode 100644 debian/pmg-pbsbackup@.service
 create mode 100644 src/PMG/API2/PBS/Job.pm
 create mode 100644 src/PMG/API2/PBS/Remote.pm
 create mode 100644 src/PMG/PBSConfig.pm
 create mode 100644 src/PMG/PBSSchedule.pm

pmg-gui:
Stoiko Ivanov (3):
  Make Backup/Restore panel a menuentry
  refactor RestoreWindow for PBS
  add PBSConfig tab to Backup menu

 js/BackupConfiguration.js |  23 ++
 js/BackupRestore.js       |  67 ++--
 js/Makefile               |   2 +
 js/NavigationTree.js      |   6 +
 js/PBSConfig.js           | 678 ++++++++++++++++++++++++++++++++++++++
 js/SystemConfiguration.js |   4 -
 6 files changed, 748 insertions(+), 32 deletions(-)
 create mode 100644 js/BackupConfiguration.js
 create mode 100644 js/PBSConfig.js

-- 
2.20.1

[pmg-devel] [PATCH pve-common v3 1/1] add PBSClient module

[Stoiko Ivanov]

PBSClient.pm contains methods for:
* handling (sensitive) config-information (passwords, encryption keys)
* creating/restoring/forgetting/listing backups

code is mostly based on the current PBSPlugin in pve-storage

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile         |   1 +
 src/PVE/PBSClient.pm | 305 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 306 insertions(+)
 create mode 100644 src/PVE/PBSClient.pm

diff --git a/src/Makefile b/src/Makefile
index b2a4ac6..098a648 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -21,6 +21,7 @@ LIB_SOURCES = \
 	LDAP.pm \
 	Network.pm \
 	OTP.pm \
+	PBSClient.pm \
 	PTY.pm \
 	ProcFSTools.pm \
 	RESTEnvironment.pm \
diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
new file mode 100644
index 0000000..1d9a9f4
--- /dev/null
+++ b/src/PVE/PBSClient.pm
@@ -0,0 +1,305 @@
+package PVE::PBSClient;
+
+# utility functions for interaction with Proxmox Backup Server
+
+use strict;
+use warnings;
+use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
+use IO::File;
+use JSON;
+use POSIX qw(strftime ENOENT);
+
+use PVE::Tools qw(run_command file_set_contents file_get_contents file_read_firstline);
+use PVE::JSONSchema qw(get_standard_option);
+
+sub new {
+    my ($class, $scfg, $storeid, $sdir) = @_;
+
+    die "no section config provided\n" if ref($scfg) eq '';
+    die "undefined store id\n" if !defined($storeid);
+
+    my $secret_dir = $sdir // '/etc/pve/priv/storage';
+
+    my $self = bless { scfg => $scfg, storeid => $storeid, secret_dir => $secret_dir }, $class;
+}
+
+my sub password_file_name {
+    my ($self) = @_;
+
+    return "$self->{secret_dir}/$self->{storeid}.pw";
+}
+
+sub set_password {
+    my ($self, $password) = @_;
+
+    my $pwfile = password_file_name($self);
+    mkdir $self->{secret_dir};
+
+    PVE::Tools::file_set_contents($pwfile, "$password\n", 0600);
+};
+
+sub delete_password {
+    my ($self) = @_;
+
+    my $pwfile = password_file_name($self);
+
+    unlink $pwfile;
+};
+
+sub get_password {
+    my ($self) = @_;
+
+    my $pwfile = password_file_name($self);
+
+    return PVE::Tools::file_read_firstline($pwfile);
+}
+
+sub encryption_key_file_name {
+    my ($self) = @_;
+
+    return "$self->{secret_dir}/$self->{storeid}.enc";
+};
+
+sub set_encryption_key {
+    my ($self, $key) = @_;
+
+    my $encfile = encryption_key_file_name($self);
+    mkdir $self->{secret_dir};
+
+    PVE::Tools::file_set_contents($encfile, "$key\n", 0600);
+};
+
+sub delete_encryption_key {
+    my ($self) = @_;
+
+    my $encfile = encryption_key_file_name($self);
+
+    if (!unlink $encfile) {
+	return if $! == ENOENT;
+	die "failed to delete encryption key! $!\n";
+    }
+};
+
+# Returns a file handle if there is an encryption key, or `undef` if there is not. Dies on error.
+my sub open_encryption_key {
+    my ($self) = @_;
+
+    my $encryption_key_file = encryption_key_file_name($self);
+
+    my $keyfd;
+    if (!open($keyfd, '<', $encryption_key_file)) {
+	return undef if $! == ENOENT;
+	die "failed to open encryption key: $encryption_key_file: $!\n";
+    }
+
+    return $keyfd;
+}
+
+my $USE_CRYPT_PARAMS = {
+    backup => 1,
+    restore => 1,
+    'upload-log' => 1,
+};
+
+my sub do_raw_client_cmd {
+    my ($self, $client_cmd, $param, %opts) = @_;
+
+    my $use_crypto = $USE_CRYPT_PARAMS->{$client_cmd};
+
+    my $client_exe = '/usr/bin/proxmox-backup-client';
+    die "executable not found '$client_exe'! Proxmox backup client not installed?\n"
+	if ! -x $client_exe;
+
+    my $scfg = $self->{scfg};
+    my $server = $scfg->{server};
+    my $datastore = $scfg->{datastore};
+    my $username = $scfg->{username} // 'root@pam';
+
+    my $userns_cmd = delete $opts{userns_cmd};
+
+    my $cmd = [];
+
+    push @$cmd, @$userns_cmd if defined($userns_cmd);
+
+    push @$cmd, $client_exe, $client_cmd;
+
+    # This must live in the top scope to not get closed before the `run_command`
+    my $keyfd;
+    if ($use_crypto) {
+	if (defined($keyfd = open_encryption_key($self))) {
+	    my $flags = fcntl($keyfd, F_GETFD, 0)
+		// die "failed to get file descriptor flags: $!\n";
+	    fcntl($keyfd, F_SETFD, $flags & ~FD_CLOEXEC)
+		or die "failed to remove FD_CLOEXEC from encryption key file descriptor\n";
+	    push @$cmd, '--crypt-mode=encrypt', '--keyfd='.fileno($keyfd);
+	} else {
+	    push @$cmd, '--crypt-mode=none';
+	}
+    }
+
+    push @$cmd, @$param if defined($param);
+
+    push @$cmd, "--repository", "$username\@$server:$datastore";
+
+    local $ENV{PBS_PASSWORD} = get_password($self);
+
+    local $ENV{PBS_FINGERPRINT} = $scfg->{fingerprint};
+
+    # no ascii-art on task logs
+    local $ENV{PROXMOX_OUTPUT_NO_BORDER} = 1;
+    local $ENV{PROXMOX_OUTPUT_NO_HEADER} = 1;
+
+    if (my $logfunc = $opts{logfunc}) {
+	$logfunc->("run: " . join(' ', @$cmd));
+    }
+
+    run_command($cmd, %opts);
+}
+
+my sub run_raw_client_cmd {
+    my ($self, $client_cmd, $param, %opts) = @_;
+    return do_raw_client_cmd($self, $client_cmd, $param, %opts);
+}
+
+my sub run_client_cmd {
+    my ($self, $client_cmd, $param, $no_output) = @_;
+
+    my $json_str = '';
+    my $outfunc = sub { $json_str .= "$_[0]\n" };
+
+    $param = [] if !defined($param);
+    $param = [ $param ] if !ref($param);
+
+    $param = [@$param, '--output-format=json'] if !$no_output;
+
+    do_raw_client_cmd($self, $client_cmd, $param,
+		      outfunc => $outfunc, errmsg => 'proxmox-backup-client failed');
+
+    return undef if $no_output;
+
+    my $res = decode_json($json_str);
+
+    return $res;
+}
+
+sub autogen_encryption_key {
+    my ($self) = @_;
+    my $encfile = encryption_key_file_name($self);
+    run_command(['proxmox-backup-client', 'key', 'create', '--kdf', 'none', $encfile]);
+};
+
+sub get_snapshots {
+    my ($self, $opts) = @_;
+
+    my $param = [];
+    if (defined($opts->{group})) {
+	push @$param, $opts->{group};
+    }
+
+    return run_client_cmd($self, "snapshots", $param);
+};
+
+sub backup_tree {
+    my ($self, $opts) = @_;
+
+    my $type = delete $opts->{type};
+    die "backup-type not provided\n" if !defined($type);
+    my $id = delete $opts->{id};
+    die "backup-id not provided\n" if !defined($id);
+    my $root = delete $opts->{root};
+    die "root dir not provided\n" if !defined($root);
+    my $pxarname = delete $opts->{pxarname};
+    die "archive name not provided\n" if !defined($pxarname);
+    my $time = delete $opts->{time};
+
+    my $param = [];
+
+    push @$param, "$pxarname.pxar:$root";
+    push @$param, '--backup-type', $type;
+    push @$param, '--backup-id', $id;
+    push @$param, '--backup-time', $time if defined($time);
+
+    return run_raw_client_cmd($self, 'backup', $param, %$opts);
+};
+
+sub restore_pxar {
+    my ($self, $opts) = @_;
+
+    my $snapshot = delete $opts->{snapshot};
+    die "snapshot not provided\n" if !defined($snapshot);
+    my $pxarname = delete $opts->{pxarname};
+    die "archive name not provided\n" if !defined($pxarname);
+    my $target = delete $opts->{target};
+    die "restore-target not provided\n" if !defined($target);
+    #my $time = delete $opts->{time};
+
+    my $param = [];
+
+    push @$param, "$snapshot";
+    push @$param, "$pxarname.pxar";
+    push @$param, "$target";
+    push @$param, "--allow-existing-dirs", 0;
+
+    return run_raw_client_cmd($self, 'restore', $param, %$opts);
+};
+
+sub forget_snapshot {
+    my ($self, $snapshot) = @_;
+
+    die "snapshot not provided\n" if !defined($snapshot);
+
+    my $param = [];
+
+    push @$param, "$snapshot";
+
+    return run_raw_client_cmd($self, 'forget', $param);
+};
+
+sub prune_group {
+    my ($self, $opts, $prune_opts, $group) = @_;
+
+    die "group not provided\n" if !defined($group);
+
+    # do nothing if no keep options specified for remote
+    return [] if scalar(keys %$prune_opts) == 0;
+
+    my $param = [];
+
+    push @$param, "--quiet";
+
+    if (defined($opts->{'dry-run'}) && $opts->{'dry-run'}) {
+	push @$param, "--dry-run", $opts->{'dry-run'};
+    }
+
+    foreach my $keep_opt (keys %$prune_opts) {
+	push @$param, "--$keep_opt", $prune_opts->{$keep_opt};
+    }
+    push @$param, "$group";
+
+    return run_client_cmd($self, 'prune', $param);
+};
+
+sub status {
+    my ($self) = @_;
+
+    my $total = 0;
+    my $free = 0;
+    my $used = 0;
+    my $active = 0;
+
+    eval {
+	my $res = run_client_cmd($self, "status");
+
+	$active = 1;
+	$total = $res->{total};
+	$used = $res->{used};
+	$free = $res->{avail};
+    };
+    if (my $err = $@) {
+	warn $err;
+    }
+
+    return ($total, $free, $used, $active);
+};
+
+1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 1/7] debian: drop duplicate ', ' in dependencies

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/control | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/debian/control b/debian/control
index 2460411..1afbbb5 100644
--- a/debian/control
+++ b/debian/control
@@ -76,7 +76,7 @@ Depends: apt,
          pmg-log-tracker,
          postfix (>= 2.5.5),
          postgresql-11,
-         proxmox-backup-client,,
+         proxmox-backup-client,
          proxmox-mini-journalreader,
          proxmox-spamassassin,
          pve-xtermjs (>= 1.0-1),
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 2/7] add initial SectionConfig for PBS

[Stoiko Ivanov]

add a SectionConfig definition to hold information about PBS-remotes used
for backing up PMG.

Mostly adapted from the PBSPlugin.pm in pve-storage.

This commit needs a versioned dependency on pve-common

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/dirs          |   1 +
 src/Makefile         |   1 +
 src/PMG/PBSConfig.pm | 195 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 197 insertions(+)
 create mode 100644 src/PMG/PBSConfig.pm

diff --git a/debian/dirs b/debian/dirs
index f7ac2e7..f138bb4 100644
--- a/debian/dirs
+++ b/debian/dirs
@@ -1,4 +1,5 @@
 /etc/pmg
 /etc/pmg/dkim
+/etc/pmg/pbs
 /var/lib/pmg
 /var/lib/pmg/backup
diff --git a/src/Makefile b/src/Makefile
index 05d9598..daa9d46 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -66,6 +66,7 @@ LIBSOURCES =				\
 	PMG/SMTP.pm			\
 	PMG/Unpack.pm			\
 	PMG/Backup.pm			\
+	PMG/PBSConfig.pm		\
 	PMG/RuleCache.pm		\
 	PMG/Statistic.pm		\
 	PMG/UserConfig.pm		\
diff --git a/src/PMG/PBSConfig.pm b/src/PMG/PBSConfig.pm
new file mode 100644
index 0000000..36479ce
--- /dev/null
+++ b/src/PMG/PBSConfig.pm
@@ -0,0 +1,195 @@
+package PMG::PBSConfig;
+
+# section config implementation for PBS integration in PMG
+
+use strict;
+use warnings;
+
+use PVE::Tools qw(extract_param);
+use PVE::SectionConfig;
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::PBSClient;
+
+use base qw(PVE::SectionConfig);
+
+my $inotify_file_id = 'pmg-pbs.conf';
+my $secret_dir = '/etc/pmg/pbs';
+my $config_filename = "${secret_dir}/pbs.conf";
+
+
+my %prune_option = (
+    optional => 1,
+    type => 'integer', minimum => '0',
+    format_description => 'N',
+);
+
+my %prune_properties = (
+    'keep-last' => {
+	%prune_option,
+	description => 'Keep the last <N> backups.',
+    },
+    'keep-hourly' => {
+	%prune_option,
+	description => 'Keep backups for the last <N> different hours. If there is more' .
+		       'than one backup for a single hour, only the latest one is kept.'
+    },
+    'keep-daily' => {
+	%prune_option,
+	description => 'Keep backups for the last <N> different days. If there is more' .
+		       'than one backup for a single day, only the latest one is kept.'
+    },
+    'keep-weekly' => {
+	%prune_option,
+	description => 'Keep backups for the last <N> different weeks. If there is more' .
+		       'than one backup for a single week, only the latest one is kept.'
+    },
+    'keep-monthly' => {
+	%prune_option,
+	description => 'Keep backups for the last <N> different months. If there is more' .
+		       'than one backup for a single month, only the latest one is kept.'
+    },
+    'keep-yearly' => {
+	%prune_option,
+	description => 'Keep backups for the last <N> different years. If there is more' .
+		       'than one backup for a single year, only the latest one is kept.'
+    },
+);
+
+my $defaultData = {
+    propertyList => {
+	type => { description => "Section type." },
+	remote => {
+	    description => "Proxmox Backup Server ID.",
+	    type => 'string', format => 'pve-configid',
+	},
+    },
+};
+
+sub properties {
+    return {
+	datastore => {
+	    description => "Proxmox backup server datastore name.",
+	    type => 'string',
+	},
+	server => {
+	    description => "Proxmox backup server address.",
+	    type => 'string', format => 'address',
+	    maxLength => 256,
+	},
+	disable => {
+	    description => "Flag to disable/deactivate the entry.",
+	    type => 'boolean',
+	    optional => 1,
+	},
+	password => {
+	    description => "Password for the user on the Proxmox backup server.",
+	    type => 'string',
+	    optional => 1,
+	},
+	username => get_standard_option('pmg-email-address', {
+	    description => "Username on the Proxmox backup server"
+	}),
+	fingerprint => get_standard_option('fingerprint-sha256'),
+	%prune_properties,
+    };
+}
+
+sub options {
+    return {
+	server => {},
+	datastore => {},
+	disable => { optional => 1 },
+	username => { optional => 1 },
+	password => { optional => 1 },
+	fingerprint => { optional => 1 },
+	'keep-last' => { optional => 1 },
+	'keep-hourly' =>  { optional => 1 },
+	'keep-daily' => { optional => 1 },
+	'keep-weekly' => { optional => 1 },
+	'keep-monthly' => { optional => 1 },
+	'keep-yearly' => { optional => 1 },
+    };
+}
+
+sub type {
+    return 'pbs';
+}
+
+sub private {
+    return $defaultData;
+}
+
+sub prune_options {
+    my ($self, $remote) = @_;
+
+    my $remote_cfg = $self->{ids}->{$remote};
+
+    my $res = {};
+
+    foreach my $keep_opt (keys %prune_properties) {
+
+	if (defined($remote_cfg->{$keep_opt})) {
+	    $res->{$keep_opt} = $remote_cfg->{$keep_opt};
+	}
+    }
+    return $res;
+}
+
+sub new {
+    my ($type) = @_;
+
+    my $class = ref($type) || $type;
+
+    my $cfg = PVE::INotify::read_file($inotify_file_id);
+
+    $cfg->{secret_dir} = $secret_dir;
+
+    return bless $cfg, $class;
+}
+
+sub write {
+    my ($self) = @_;
+
+    PVE::INotify::write_file($inotify_file_id, $self);
+}
+
+my $lockfile = "/var/lock/pmgpbsconfig.lck";
+
+sub lock_config {
+    my ($code, $errmsg) = @_;
+
+    my $p = PVE::Tools::lock_file($lockfile, undef, $code);
+    if (my $err = $@) {
+	$errmsg ? die "$errmsg: $err" : die $err;
+    }
+}
+
+
+__PACKAGE__->register();
+__PACKAGE__->init();
+
+sub read_pmg_pbs_conf {
+    my ($filename, $fh) = @_;
+
+    local $/ = undef; # slurp mode
+
+    my $raw = defined($fh) ? <$fh> : '';
+
+    return __PACKAGE__->parse_config($filename, $raw);
+}
+
+sub write_pmg_pbs_conf {
+    my ($filename, $fh, $cfg) = @_;
+
+    my $raw = __PACKAGE__->write_config($filename, $cfg);
+
+    PVE::Tools::safe_print($filename, $fh, $raw);
+}
+
+PVE::INotify::register_file($inotify_file_id, $config_filename,
+			    \&read_pmg_pbs_conf,
+			    \&write_pmg_pbs_conf,
+			    undef,
+			    always_call_parser => 1);
+
+1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 3/7] Add API2 module for PBS configuration

[Stoiko Ivanov]

The module provides the API methods for creating/updating/listing/deleting
PBS remotes

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile               |   1 +
 src/PMG/API2/Config.pm     |   7 ++
 src/PMG/API2/PBS/Remote.pm | 231 +++++++++++++++++++++++++++++++++++++
 3 files changed, 239 insertions(+)
 create mode 100644 src/PMG/API2/PBS/Remote.pm

diff --git a/src/Makefile b/src/Makefile
index daa9d46..5add6af 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -137,6 +137,7 @@ LIBSOURCES =				\
 	PMG/API2/Statistics.pm		\
 	PMG/API2/MailTracker.pm		\
 	PMG/API2/Backup.pm		\
+	PMG/API2/PBS/Remote.pm		\
 	PMG/API2/Nodes.pm		\
 	PMG/API2/Postfix.pm		\
 	PMG/API2/Quarantine.pm		\
diff --git a/src/PMG/API2/Config.pm b/src/PMG/API2/Config.pm
index d4a9679..e11eb3f 100644
--- a/src/PMG/API2/Config.pm
+++ b/src/PMG/API2/Config.pm
@@ -25,6 +25,7 @@ use PMG::API2::Fetchmail;
 use PMG::API2::DestinationTLSPolicy;
 use PMG::API2::DKIMSign;
 use PMG::API2::SACustom;
+use PMG::API2::PBS::Remote;
 
 use base qw(PVE::RESTHandler);
 
@@ -93,6 +94,11 @@ __PACKAGE__->register_method({
     path => 'customscores',
 });
 
+__PACKAGE__->register_method ({
+    subclass => "PMG::API2::PBS::Remote",
+    path => 'pbs',
+});
+
 __PACKAGE__->register_method ({
     name => 'index', 
     path => '',
@@ -131,6 +137,7 @@ __PACKAGE__->register_method ({
 	push @$res, { section => 'regextest' };
 	push @$res, { section => 'tlspolicy' };
 	push @$res, { section => 'dkim' };
+	push @$res, { section => 'pbs' };
 
 	return $res;
     }});
diff --git a/src/PMG/API2/PBS/Remote.pm b/src/PMG/API2/PBS/Remote.pm
new file mode 100644
index 0000000..3af90c3
--- /dev/null
+++ b/src/PMG/API2/PBS/Remote.pm
@@ -0,0 +1,231 @@
+package PMG::API2::PBS::Remote;
+
+use strict;
+use warnings;
+
+use PVE::SafeSyslog;
+use PVE::Tools qw(extract_param);
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::PBSClient;
+
+use PMG::PBSConfig;
+
+use base qw(PVE::RESTHandler);
+
+__PACKAGE__->register_method ({
+    name => 'list',
+    path => '',
+    method => 'GET',
+    description => "List all configured Proxmox Backup Server instances.",
+    permissions => { check => [ 'admin', 'audit' ] },
+    proxyto => 'master',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {}
+    },
+    returns => {
+        type => "array",
+        items => PMG::PBSConfig->createSchema(1),
+        links => [ { rel => 'child', href => "{remote}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $res = [];
+
+	my $conf = PMG::PBSConfig->new();
+
+	if (defined($conf)) {
+	    foreach my $remote (keys %{$conf->{ids}}) {
+		my $d = $conf->{ids}->{$remote};
+		my $entry = {
+		    remote => $remote,
+		    server => $d->{server},
+		    datastore => $d->{datastore},
+		    username => $d->{username},
+		    disable => $d->{disable},
+		};
+		push @$res, $entry;
+	    }
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'create',
+    path => '',
+    method => 'POST',
+    description => "Add Proxmox Backup Server instance.",
+    permissions => { check => [ 'admin' ] },
+    proxyto => 'master',
+    protected => 1,
+    parameters => PMG::PBSConfig->createSchema(1),
+    returns => { type => 'null' } ,
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    $conf->{ids} //= {};
+	    my $ids = $conf->{ids};
+
+	    my $remote = extract_param($param, 'remote');
+	    die "PBS remote '$remote' already exists\n"
+		if $ids->{$remote};
+
+	    my $remotecfg = PMG::PBSConfig->check_config($remote, $param, 1);
+
+	    my $password = extract_param($remotecfg, 'password');
+
+	    my $pbs = PVE::PBSClient->new($remotecfg, $remote, $conf->{secret_dir});
+	    $pbs->set_password($password) if defined($password);
+
+	    $ids->{$remote} = $remotecfg;
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "add PBS remote failed");
+
+	return undef;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'read_config',
+    path => '{remote}',
+    method => 'GET',
+    description => "Get PBS remote configuration.",
+    proxyto => 'master',
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 1,
+	properties => {
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {},
+    code => sub {
+	my ($param) = @_;
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote = $param->{remote};
+
+	my $data = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$data;
+
+	delete $data->{type};
+
+	$data->{digest} = $conf->{digest};
+	$data->{remote} = $remote;
+
+	return $data;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'update_config',
+    path => '{remote}',
+    method => 'PUT',
+    description => "Update PBS remote settings.",
+    permissions => { check => [ 'admin' ] },
+    protected => 1,
+    proxyto => 'master',
+    parameters => PMG::PBSConfig->updateSchema(),
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    my $ids = $conf->{ids};
+
+	    my $digest = extract_param($param, 'digest');
+	    PVE::SectionConfig::assert_if_modified($conf, $digest);
+
+	    my $remote = extract_param($param, 'remote');
+
+	    die "PBS remote '$remote' does not exist\n"
+		if !$ids->{$remote};
+
+	    my $delete_str = extract_param($param, 'delete');
+	    die "no options specified\n"
+		if !$delete_str && !scalar(keys %$param);
+
+	    my $pbs = PVE::PBSClient->new($ids->{$remote}, $remote, $conf->{secret_dir});
+	    foreach my $opt (PVE::Tools::split_list($delete_str)) {
+		if ($opt eq 'password') {
+		    $pbs->delete_password();
+		}
+
+		delete $ids->{$remote}->{$opt};
+	    }
+
+	    if (defined(my $password = extract_param($param, 'password'))) {
+		$pbs->set_password($password);
+	    }
+
+	    my $remoteconfig = PMG::PBSConfig->check_config($remote, $param, 0, 1);
+
+	    foreach my $p (keys %$remoteconfig) {
+		$ids->{$remote}->{$p} = $remoteconfig->{$p};
+	    }
+
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "update PBS remote failed");
+
+	return undef;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'delete',
+    path => '{remote}',
+    method => 'DELETE',
+    description => "Delete an PBS remote",
+    permissions => { check => [ 'admin' ] },
+    protected => 1,
+    proxyto => 'master',
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    remote => {
+		description => "Profile ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	}
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    my $ids = $conf->{ids};
+
+	    my $remote = $param->{remote};
+
+	    die "PBS remote '$remote' does not exist\n"
+		if !$ids->{$remote};
+
+	    my $pbs = PVE::PBSClient->new($ids->{$remote}, $remote, $conf->{secret_dir});
+	    $pbs->delete_password($remote);
+	    delete $ids->{$remote};
+
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "delete PBS remote failed");
+
+	return undef;
+    }});
+
+1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 4/7] Add API2 module for per-node backups to PBS

[Stoiko Ivanov]

The module adds API2 methods for:

* creating/restoring/listing/forgetting backups on a configured PBS remote

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile            |   1 +
 src/PMG/API2/Nodes.pm   |   7 +
 src/PMG/API2/PBS/Job.pm | 371 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 379 insertions(+)
 create mode 100644 src/PMG/API2/PBS/Job.pm

diff --git a/src/Makefile b/src/Makefile
index 5add6af..fb42f21 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -137,6 +137,7 @@ LIBSOURCES =				\
 	PMG/API2/Statistics.pm		\
 	PMG/API2/MailTracker.pm		\
 	PMG/API2/Backup.pm		\
+	PMG/API2/PBS/Job.pm		\
 	PMG/API2/PBS/Remote.pm		\
 	PMG/API2/Nodes.pm		\
 	PMG/API2/Postfix.pm		\
diff --git a/src/PMG/API2/Nodes.pm b/src/PMG/API2/Nodes.pm
index 96aa146..259f8f3 100644
--- a/src/PMG/API2/Nodes.pm
+++ b/src/PMG/API2/Nodes.pm
@@ -26,6 +26,7 @@ use PMG::API2::SpamAssassin;
 use PMG::API2::Postfix;
 use PMG::API2::MailTracker;
 use PMG::API2::Backup;
+use PMG::API2::PBS::Job;
 
 use base qw(PVE::RESTHandler);
 
@@ -79,6 +80,11 @@ __PACKAGE__->register_method ({
     path => 'backup',
 });
 
+__PACKAGE__->register_method ({
+    subclass => "PMG::API2::PBS::Job",
+    path => 'pbs',
+});
+
 __PACKAGE__->register_method ({
     name => 'index',
     path => '',
@@ -105,6 +111,7 @@ __PACKAGE__->register_method ({
 	my $result = [
 	    { name => 'apt' },
 	    { name => 'backup' },
+	    { name => 'pbs' },
 	    { name => 'clamav' },
 	    { name => 'spamassassin' },
 	    { name => 'postfix' },
diff --git a/src/PMG/API2/PBS/Job.pm b/src/PMG/API2/PBS/Job.pm
new file mode 100644
index 0000000..dee1754
--- /dev/null
+++ b/src/PMG/API2/PBS/Job.pm
@@ -0,0 +1,371 @@
+package PMG::API2::PBS::Job;
+
+use strict;
+use warnings;
+
+use POSIX qw(strftime);
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::SafeSyslog;
+use PVE::Tools qw(extract_param);
+use PVE::PBSClient;
+
+use PMG::RESTEnvironment;
+use PMG::Backup;
+use PMG::PBSConfig;
+
+use base qw(PVE::RESTHandler);
+
+__PACKAGE__->register_method ({
+    name => 'list',
+    path => '',
+    method => 'GET',
+    description => "List all configured Proxmox Backup Server jobs.",
+    permissions => { check => [ 'admin', 'audit' ] },
+    proxyto => 'node',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	},
+    },
+    returns => {
+        type => "array",
+        items => PMG::PBSConfig->createSchema(1),
+        links => [ { rel => 'child', href => "{remote}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $res = [];
+
+	my $conf = PMG::PBSConfig->new();
+	if (defined($conf)) {
+	    foreach my $remote (keys %{$conf->{ids}}) {
+		my $d = $conf->{ids}->{$remote};
+		my $entry = {
+		    remote => $remote,
+		    server => $d->{server},
+		    datastore => $d->{datastore},
+		};
+		push @$res, $entry;
+	    }
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'remote_index',
+    path => '{remote}',
+    method => 'GET',
+    description => "Backup Job index.",
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {
+	type => 'array',
+	items => {
+	    type => "object",
+	    properties => { section => { type => 'string'} },
+	},
+	links => [ { rel => 'child', href => "{section}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $result = [
+	    { section => 'snapshots' },
+	    { section => 'backup' },
+	    { section => 'restore' },
+	    { section => 'timer' },
+	];
+	return $result;
+}});
+
+__PACKAGE__->register_method ({
+    name => 'get_snapshots',
+    path => '{remote}/snapshots',
+    method => 'GET',
+    description => "Get snapshots stored on remote.",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {
+	type => 'array',
+	items => {
+	    type => "object",
+	    properties => {
+		time => { type => 'string'},
+		ctime => { type => 'string'},
+		size => { type => 'integer'},
+	    },
+	},
+	links => [ { rel => 'child', href => "{time}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	return [] if $remote_config->{disable};
+
+	my $snap_param = {
+	    group => "host/$node",
+	};
+
+	my $pbs = PVE::PBSClient->new($remote_config, $remote, $conf->{secret_dir});
+	my $snapshots = $pbs->get_snapshots($snap_param);
+	my $res = [];
+	foreach my $item (@$snapshots) {
+	    my $btype = $item->{"backup-type"};
+	    my $bid = $item->{"backup-id"};
+	    my $epoch = $item->{"backup-time"};
+	    my $size = $item->{size} // 1;
+
+	    my @pxar = grep { $_->{filename} eq 'pmgbackup.pxar.didx' } @{$item->{files}};
+	    die "unexpected number of pmgbackup archives in snapshot\n" if (scalar(@pxar) != 1);
+
+
+	    next if !($btype eq 'host');
+	    next if !($bid eq $node);
+
+	    my $time = strftime("%FT%TZ", gmtime($epoch));
+
+	    my $info = {
+		time => $time,
+		ctime => $epoch,
+		size => $size,
+	    };
+
+	    push @$res, $info;
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'forget_snapshot',
+    path => '{remote}/snapshots/{time}',
+    method => 'DELETE',
+    description => "Forget a snapshot",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    time => {
+		description => "Backup time in RFC 3399 format",
+		type => 'string',
+	    },
+	},
+    },
+    returns => {type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+	my $time = $param->{time};
+
+	my $snapshot = "host/$node/$time";
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+	die "PBS remote '$remote' is disabled\n" if $remote_config->{disable};
+
+	my $pbs = PVE::PBSClient->new($remote_config, $remote, $conf->{secret_dir});
+
+	eval {
+	    $pbs->forget_snapshot($snapshot);
+	};
+	die "Forgetting backup failed: $@" if $@;
+
+	return;
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'run_backup',
+    path => '{remote}/backup',
+    method => 'POST',
+    description => "run backup and prune the backupgroup afterwards.",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => "string" },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PMG::RESTEnvironment->get();
+	my $authuser = $rpcenv->get_user();
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+	die "PBS remote '$remote' is disabled\n" if $remote_config->{disable};
+
+	my $pbs = PVE::PBSClient->new($remote_config, $remote, $conf->{secret_dir});
+	my $backup_dir = "/var/lib/pmg/backup/current";
+
+	my $worker = sub {
+	    my $upid = shift;
+
+	    print "starting update of current backup state\n";
+
+	    -d $backup_dir || mkdir $backup_dir;
+	    PMG::Backup::pmg_backup($backup_dir, $param->{statistic});
+	    my $pbs_opts = {
+		type => 'host',
+		id => $node,
+		pxarname => 'pmgbackup',
+		root => $backup_dir,
+	    };
+
+	    $pbs->backup_tree($pbs_opts);
+
+	    print "backup finished\n";
+
+	    my $group = "host/$node";
+	    print "starting prune of $group\n";
+	    my $prune_opts = $conf->prune_options($remote);
+	    my $res = $pbs->prune_group(undef, $prune_opts, $group);
+
+	    foreach my $pruned (@$res){
+		my $time = strftime("%FT%TZ", gmtime($pruned->{'backup-time'}));
+		my $snap = $pruned->{'backup-type'} . '/' . $pruned->{'backup-id'} . '/' .  $time;
+		print "pruned snapshot: $snap\n";
+	    }
+
+	    print "prune finished\n";
+
+	    return;
+	};
+
+	return $rpcenv->fork_worker('pbs_backup', undef, $authuser, $worker);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'restore',
+    path => '{remote}/restore',
+    method => 'POST',
+    description => "Restore the system configuration.",
+    permissions => { check => [ 'admin' ] },
+    proxyto => 'node',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    PMG::Backup::get_restore_options(),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    'backup-time' => {description=> "backup-time to restore",
+		optional => 1, type => 'string'
+	    },
+	    'backup-id' => {description => "backup-id (hostname) of backup snapshot",
+		optional => 1, type => 'string'
+	    },
+	},
+    },
+    returns => { type => "string" },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PMG::RESTEnvironment->get();
+	my $authuser = $rpcenv->get_user();
+
+	my $remote = $param->{remote};
+	my $backup_id = $param->{'backup-id'} // $param->{node};
+	my $snapshot = "host/$backup_id";
+	$snapshot .= "/$param->{'backup-time'}" if defined($param->{'backup-time'});
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+	die "PBS remote '$remote' is disabled\n" if $remote_config->{disable};
+
+	my $pbs = PVE::PBSClient->new($remote_config, $remote, $conf->{secret_dir});
+
+	my $time = time;
+	my $dirname = "/tmp/proxrestore_$$.$time";
+
+	$param->{database} //= 1;
+
+	die "nothing selected - please select what you want to restore (config or database?)\n"
+	    if !($param->{database} || $param->{config});
+
+	my $pbs_opts = {
+	    pxarname => 'pmgbackup',
+	    target => $dirname,
+	    snapshot => $snapshot,
+	};
+
+	my $worker = sub {
+	    my $upid = shift;
+
+	    print "starting restore of $snapshot from $remote\n";
+
+	    $pbs->restore_pxar($pbs_opts);
+	    print "starting restore of PMG config\n";
+	    PMG::Backup::pmg_restore($dirname, $param->{database},
+		 $param->{config}, $param->{statistic});
+	    print "restore finished\n";
+
+	    return;
+	};
+
+	return $rpcenv->fork_worker('pbs_restore', undef, $authuser, $worker);
+    }});
+
+1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 5/7] pbs-integration: add CLI calls to pmgbackup

[Stoiko Ivanov]

This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores, pruning

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/CLI/pmgbackup.pm | 54 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/src/PMG/CLI/pmgbackup.pm b/src/PMG/CLI/pmgbackup.pm
index 69224e5..228f5ab 100644
--- a/src/PMG/CLI/pmgbackup.pm
+++ b/src/PMG/CLI/pmgbackup.pm
@@ -3,19 +3,55 @@ package PMG::CLI::pmgbackup;
 use strict;
 use warnings;
 use Data::Dumper;
+use POSIX qw(strftime);
 
 use PVE::Tools;
 use PVE::SafeSyslog;
 use PVE::INotify;
 use PVE::CLIHandler;
+use PVE::CLIFormatter;
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::PTY;
 
 use PMG::RESTEnvironment;
 use PMG::API2::Backup;
+use PMG::API2::PBS::Remote;
+use PMG::API2::PBS::Job;
 
 use base qw(PVE::CLIHandler);
 
 my $nodename = PVE::INotify::nodename();
 
+sub param_mapping {
+    my ($name) = @_;
+
+    my $password_map = PVE::CLIHandler::get_standard_mapping('pve-password', {
+	func => sub {
+	    my ($value) = @_;
+	    return $value if $value;
+	    return PVE::PTY::get_confirmed_password();
+	},
+    });
+
+    my $enc_key_map = {
+	name => 'encryption-key',
+	desc => 'a file containing an encryption key, or the special value "autogen"',
+	func => sub {
+	    my ($value) = @_;
+	    return $value if $value eq 'autogen';
+	    return PVE::Tools::file_get_contents($value);
+	}
+    };
+
+
+    my $mapping = {
+	'create' => [ $password_map, $enc_key_map ],
+	'update_config' => [ $password_map, $enc_key_map ],
+    };
+    return $mapping->{$name};
+}
+
+
 sub setup_environment {
     PMG::RESTEnvironment->setup_default_cli_env();
 }
@@ -32,6 +68,24 @@ our $cmddef = {
     backup => [ 'PMG::API2::Backup', 'backup', undef, { node => $nodename } ],
     restore => [ 'PMG::API2::Backup', 'restore', undef, { node => $nodename } ],
     list => [ 'PMG::API2::Backup', 'list', undef, { node => $nodename }, $format_backup_list ],
+    remote => {
+	list => ['PMG::API2::PBS::Remote', 'list', undef, undef,  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['remote', 'server', 'datastore', 'username' ], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
+	add => ['PMG::API2::PBS::Remote', 'create', ['remote'] ],
+	remove => ['PMG::API2::PBS::Remote', 'delete', ['remote'] ],
+	set => ['PMG::API2::PBS::Remote', 'update_config', ['remote'] ],
+    },
+    pbsjob => {
+	list_backups => ['PMG::API2::PBS::Job', 'get_snapshots', ['remote'] , { node => $nodename },  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['time', 'size'], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
+	forget => ['PMG::API2::PBS::Job', 'forget_snapshot', ['remote', 'time'], { node => $nodename} ],
+	run => ['PMG::API2::PBS::Job', 'run_backup', ['remote'], { node => $nodename} ],
+	restore => ['PMG::API2::PBS::Job', 'restore', ['remote'], { node => $nodename} ],
+    },
 };
 
 1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 6/7] add scheduled backup to PBS remotes

[Stoiko Ivanov]

PMG::PBSSchedule contains methods for creating/deleting systemd-timer units,
which will run a backup to a configured PBS remote.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/pmg-pbsbackup@.service |   6 ++
 debian/rules                  |   1 +
 src/Makefile                  |   3 +-
 src/PMG/API2/PBS/Job.pm       | 130 ++++++++++++++++++++++++++++++++++
 src/PMG/CLI/pmgbackup.pm      |   6 ++
 src/PMG/PBSSchedule.pm        | 104 +++++++++++++++++++++++++++
 6 files changed, 249 insertions(+), 1 deletion(-)
 create mode 100644 debian/pmg-pbsbackup@.service
 create mode 100644 src/PMG/PBSSchedule.pm

diff --git a/debian/pmg-pbsbackup@.service b/debian/pmg-pbsbackup@.service
new file mode 100644
index 0000000..37aa23b
--- /dev/null
+++ b/debian/pmg-pbsbackup@.service
@@ -0,0 +1,6 @@
+[Unit]
+Description=Backup to PBS remote %I
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/pmgbackup pbsjob run %I
diff --git a/debian/rules b/debian/rules
index bab4d98..5a2cf7a 100755
--- a/debian/rules
+++ b/debian/rules
@@ -20,6 +20,7 @@ override_dh_installinit:
 	dh_systemd_enable --name=pmgspamreport pmgspamreport.service
 	dh_systemd_enable --name=pmgreport pmgreport.service
 	dh_systemd_enable --name=pmgsync pmgsync.service
+	dh_systemd_enable --no-enable --name=pmg-pbsbackup@ pmg-pbsbackup@.service
 
 override_dh_systemd_start:
 	dh_systemd_start pmg-hourly.timer pmg-daily.timer pmgspamreport.timer pmgreport.timer
diff --git a/src/Makefile b/src/Makefile
index fb42f21..9d5c335 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -15,7 +15,7 @@ CRONSCRIPTS = pmg-hourly pmg-daily
 
 CLI_CLASSES = $(addprefix PMG/CLI/, $(addsuffix .pm, ${CLITOOLS}))
 SERVICE_CLASSES = $(addprefix PMG/Service/, $(addsuffix .pm, ${SERVICES}))
-SERVICE_UNITS = $(addprefix debian/, $(addsuffix .service, ${SERVICES}))
+SERVICE_UNITS = $(addprefix debian/, $(addsuffix .service, ${SERVICES} pmg-pbsbackup@))
 TIMER_UNITS = $(addprefix debian/, $(addsuffix .timer, ${CRONSCRIPTS} pmgspamreport pmgreport))
 
 CLI_BINARIES = $(addprefix bin/, ${CLITOOLS} ${CLISCRIPTS} ${CRONSCRIPTS})
@@ -67,6 +67,7 @@ LIBSOURCES =				\
 	PMG/Unpack.pm			\
 	PMG/Backup.pm			\
 	PMG/PBSConfig.pm		\
+	PMG/PBSSchedule.pm		\
 	PMG/RuleCache.pm		\
 	PMG/Statistic.pm		\
 	PMG/UserConfig.pm		\
diff --git a/src/PMG/API2/PBS/Job.pm b/src/PMG/API2/PBS/Job.pm
index dee1754..4b686ec 100644
--- a/src/PMG/API2/PBS/Job.pm
+++ b/src/PMG/API2/PBS/Job.pm
@@ -14,6 +14,7 @@ use PVE::PBSClient;
 use PMG::RESTEnvironment;
 use PMG::Backup;
 use PMG::PBSConfig;
+use PMG::PBSSchedule;
 
 use base qw(PVE::RESTHandler);
 
@@ -368,4 +369,133 @@ __PACKAGE__->register_method ({
 	return $rpcenv->fork_worker('pbs_restore', undef, $authuser, $worker);
     }});
 
+__PACKAGE__->register_method ({
+    name => 'create_timer',
+    path => '{remote}/timer',
+    method => 'POST',
+    description => "Create backup schedule",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    schedule => {
+		description => "Schedule for the backup (OnCalendar setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z*.:,\-/ ]+',
+		default => 'daily', optional => 1,
+	    },
+	    delay => {
+		description => "Randomized delay to add to the starttime (RandomizedDelaySec setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z. ]+',
+		default => 'daily', optional => 1,
+	    },
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $schedule = $param->{schedule} // 'daily';
+	my $delay = $param->{delay} // '5min';
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+	die "PBS remote '$remote' is disabled\n" if $remote_config->{disable};
+
+	PMG::PBSSchedule::create_schedule($remote, $schedule, $delay);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'delete_timer',
+    path => '{remote}/timer',
+    method => 'DELETE',
+    description => "Delete backup schedule",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+
+	PMG::PBSSchedule::delete_schedule($remote);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'list_timer',
+    path => '{remote}/timer',
+    method => 'GET',
+    description => "Get timer specification",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => 'object', properties => {
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+		optional => 1,
+	    },
+	    schedule => {
+		description => "Schedule for the backup (OnCalendar setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z*.:,\-/ ]+',
+		default => 'daily', optional => 1,
+	    },
+	    delay => {
+		description => "Randomized delay to add to the starttime (RandomizedDelaySec setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z. ]+',
+		default => 'daily', optional => 1,
+	    },
+	    unitfile => {
+		description => "unit file for the systemd.timer unit",
+		type => 'string', optional => 1,
+	    },
+	}},
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+
+	my $schedules = PMG::PBSSchedule::get_schedules();
+	my @data = grep {$_->{remote} eq $remote} @$schedules;
+
+	my $res = {};
+	if (scalar(@data) == 1) {
+	    $res = $data[0];
+	}
+
+	return $res
+    }});
+
 1;
diff --git a/src/PMG/CLI/pmgbackup.pm b/src/PMG/CLI/pmgbackup.pm
index 228f5ab..e40da25 100644
--- a/src/PMG/CLI/pmgbackup.pm
+++ b/src/PMG/CLI/pmgbackup.pm
@@ -85,6 +85,12 @@ our $cmddef = {
 	forget => ['PMG::API2::PBS::Job', 'forget_snapshot', ['remote', 'time'], { node => $nodename} ],
 	run => ['PMG::API2::PBS::Job', 'run_backup', ['remote'], { node => $nodename} ],
 	restore => ['PMG::API2::PBS::Job', 'restore', ['remote'], { node => $nodename} ],
+	create => ['PMG::API2::PBS::Job', 'create_timer', ['remote'], { node => $nodename }],
+	delete => ['PMG::API2::PBS::Job', 'delete_timer', ['remote'], { node => $nodename }],
+	schedule => ['PMG::API2::PBS::Job', 'list_timer', ['remote'], { node => $nodename },  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['remote', 'schedule', 'delay'], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
     },
 };
 
diff --git a/src/PMG/PBSSchedule.pm b/src/PMG/PBSSchedule.pm
new file mode 100644
index 0000000..6663f55
--- /dev/null
+++ b/src/PMG/PBSSchedule.pm
@@ -0,0 +1,104 @@
+package PMG::PBSSchedule;
+
+use strict;
+use warnings;
+
+use PVE::Tools qw(run_command file_set_contents file_get_contents trim dir_glob_foreach);
+use PVE::Systemd;
+
+# systemd timer
+sub get_schedules {
+    my ($param) = @_;
+
+    my $result = [];
+
+    my $systemd_dir = '/etc/systemd/system';
+
+    dir_glob_foreach($systemd_dir, '^pmg-pbsbackup@.+\.timer$', sub {
+	my ($filename) = @_;
+	my $remote;
+	if ($filename =~ /^pmg-pbsbackup\@(.+)\.timer$/) {
+	    $remote = PVE::Systemd::unescape_unit($1);
+	} else {
+	    die 'Unrecognized timer name!\n';
+	}
+
+	my $unitfile = "$systemd_dir/$filename";
+	my $unit = PVE::Systemd::read_ini($unitfile);
+
+	push @$result, {
+	    unitfile => $unitfile,
+	    remote => $remote,
+	    schedule => $unit->{'Timer'}->{'OnCalendar'},
+	    delay => $unit->{'Timer'}->{'RandomizedDelaySec'},
+	};
+    });
+
+    return $result;
+
+}
+
+sub create_schedule {
+    my ($remote, $schedule, $delay) = @_;
+
+    my $unit_name = 'pmg-pbsbackup@' . PVE::Systemd::escape_unit($remote);
+    #my $service_unit = $unit_name . '.service';
+    my $timer_unit = $unit_name . '.timer';
+    my $timer_unit_path = "/etc/systemd/system/$timer_unit";
+
+    # create systemd timer
+    run_command(['systemd-analyze', 'calendar', $schedule], errmsg => "Invalid schedule specification", outfunc => sub {});
+    run_command(['systemd-analyze', 'timespan', $delay], errmsg => "Invalid delay specification", outfunc => sub {});
+    my $timer = {
+	'Unit' => {
+	    'Description' => "Timer for PBS Backup to remote $remote",
+	},
+	'Timer' => {
+	    'OnCalendar' => $schedule,
+	    'RandomizedDelaySec' => $delay,
+	},
+	'Install' => {
+	    'WantedBy' => 'timers.target',
+	},
+    };
+
+    eval {
+	PVE::Systemd::write_ini($timer, $timer_unit_path);
+	run_command(['systemctl', 'daemon-reload']);
+	run_command(['systemctl', 'enable', $timer_unit]);
+	run_command(['systemctl', 'start', $timer_unit]);
+
+    };
+    if (my $err = $@) {
+	die "Creating backup schedule for $remote failed: $err\n";
+    }
+
+    return;
+}
+
+sub delete_schedule {
+    my ($remote) = @_;
+
+    my $schedules = get_schedules();
+
+    die "Schedule for $remote not found!\n" if !grep {$_->{remote} eq $remote} @$schedules;
+
+    my $unit_name = 'pmg-pbsbackup@' . PVE::Systemd::escape_unit($remote);
+    my $service_unit = $unit_name . '.service';
+    my $timer_unit = $unit_name . '.timer';
+    my $timer_unit_path = "/etc/systemd/system/$timer_unit";
+
+    eval {
+	run_command(['systemctl', 'disable', $timer_unit]);
+	unlink($timer_unit_path) || die "delete '$timer_unit_path' failed - $!\n";
+	run_command(['systemctl', 'daemon-reload']);
+
+    };
+    if (my $err = $@) {
+	die "Removing backup schedule for $remote failed: $err\n";
+    }
+
+    return;
+}
+
+1;
-- 
2.20.1

[pmg-devel] [PATCH pmg-api v3 7/7] add /etc/pmg/pbs to cluster-sync

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/Cluster.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/PMG/Cluster.pm b/src/PMG/Cluster.pm
index f99232d..ce4f257 100644
--- a/src/PMG/Cluster.pm
+++ b/src/PMG/Cluster.pm
@@ -409,6 +409,7 @@ sub sync_config_from_master {
     my $dirs = [
 	'templates',
 	'dkim',
+	'pbs',
     ];
 
     foreach my $dir (@$dirs) {
-- 
2.20.1

[pmg-devel] [PATCH pmg-gui v3 1/3] Make Backup/Restore panel a menuentry

[Stoiko Ivanov]

Move it away from the tab list in the Configuration entry to a submenu in
preparation for adding PBS integration

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 js/BackupConfiguration.js | 18 ++++++++++++++++++
 js/Makefile               |  1 +
 js/NavigationTree.js      |  6 ++++++
 js/SystemConfiguration.js |  4 ----
 4 files changed, 25 insertions(+), 4 deletions(-)
 create mode 100644 js/BackupConfiguration.js

diff --git a/js/BackupConfiguration.js b/js/BackupConfiguration.js
new file mode 100644
index 0000000..35b50a4
--- /dev/null
+++ b/js/BackupConfiguration.js
@@ -0,0 +1,18 @@
+Ext.define('PMG.BackupConfiguration', {
+    extend: 'Ext.tab.Panel',
+    alias: 'widget.pmgBackupConfiguration',
+
+    title: gettext('Backup'),
+
+    border: false,
+    defaults: { border: false },
+
+    items: [
+	{
+	    itemId: 'local',
+	    title: gettext('Local Backup/Restore'),
+	    xtype: 'pmgBackupRestore',
+	},
+   ],
+});
+
diff --git a/js/Makefile b/js/Makefile
index e60d638..47eabb7 100644
--- a/js/Makefile
+++ b/js/Makefile
@@ -35,6 +35,7 @@ JSSRC=							\
 	RuleConfiguration.js				\
 	SystemOptions.js				\
 	Subscription.js					\
+	BackupConfiguration.js				\
 	BackupRestore.js				\
 	SystemConfiguration.js				\
 	MailProxyRelaying.js				\
diff --git a/js/NavigationTree.js b/js/NavigationTree.js
index 0ea0d2f..ac01fd6 100644
--- a/js/NavigationTree.js
+++ b/js/NavigationTree.js
@@ -86,6 +86,12 @@ Ext.define('PMG.store.NavigationStore', {
 			path: 'pmgSubscription',
 			leaf: true,
 		    },
+		    {
+			text: gettext('Backup/Restore'),
+			iconCls: 'fa fa-floppy-o',
+			path: 'pmgBackupConfiguration',
+			leaf: true,
+		    },
 		],
 	    },
 	    {
diff --git a/js/SystemConfiguration.js b/js/SystemConfiguration.js
index 37cb3e4..51b558a 100644
--- a/js/SystemConfiguration.js
+++ b/js/SystemConfiguration.js
@@ -49,10 +49,6 @@ Ext.define('PMG.SystemConfiguration', {
             title: gettext('Options'),
 	    xtype: 'pmgSystemOptions',
 	},
-	{
-	    itemId: 'backup',
-	    xtype: 'pmgBackupRestore',
-	},
     ],
 
     initComponent: function() {
-- 
2.20.1

[pmg-devel] [PATCH pmg-gui v3 2/3] refactor RestoreWindow for PBS

[Stoiko Ivanov]

by moving the item definition to initComponent, and changing the check
for a provided filename, we can reuse the window for restores from PBS

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 js/BackupRestore.js | 58 +++++++++++++++++++++++----------------------
 1 file changed, 30 insertions(+), 28 deletions(-)

diff --git a/js/BackupRestore.js b/js/BackupRestore.js
index b66a80e..2c90f2e 100644
--- a/js/BackupRestore.js
+++ b/js/BackupRestore.js
@@ -26,39 +26,41 @@ Ext.define('PMG.RestoreWindow', {
     fieldDefaults: {
 	labelWidth: 150,
     },
-    items: [
-	{
-	    xtype: 'proxmoxcheckbox',
-	    name: 'config',
-	    fieldLabel: gettext('System Configuration'),
-	},
-	{
-	    xtype: 'proxmoxcheckbox',
-	    name: 'database',
-	    value: 1,
-	    uncheckedValue: 0,
-	    fieldLabel: gettext('Rule Database'),
-	    listeners: {
-		change: function(field, value) {
-		    field.nextSibling('field[name=statistic]').setDisabled(!value);
-		},
-	    },
-	},
-	{
-	    xtype: 'proxmoxcheckbox',
-	    name: 'statistic',
-	    fieldLabel: gettext('Statistic'),
-	},
-    ],
 
     initComponent: function() {
 	let me = this;
 
-	if (!me.filename) {
-	    throw "no filename given";
-	}
+	me.items = [
+	    {
+		xtype: 'proxmoxcheckbox',
+		name: 'config',
+		fieldLabel: gettext('System Configuration'),
+	    },
+	    {
+		xtype: 'proxmoxcheckbox',
+		name: 'database',
+		value: 1,
+		uncheckedValue: 0,
+		fieldLabel: gettext('Rule Database'),
+		listeners: {
+		    change: function(field, value) {
+			field.nextSibling('field[name=statistic]').setDisabled(!value);
+		    },
+		},
+	    },
+	    {
+		xtype: 'proxmoxcheckbox',
+		name: 'statistic',
+		fieldLabel: gettext('Statistic'),
+	    },
+	];
 
-	me.url = `/nodes/${Proxmox.NodeName}/backup/${encodeURIComponent(me.filename)}`;
+	let initurl = "/nodes/" + Proxmox.NodeName;
+	if (me.filename) {
+	    me.url = initurl + "/backup/" + encodeURIComponent(me.filename);
+	} else {
+	    throw "neither filename nor snapshot given";
+	}
 
 	me.callParent();
     },
-- 
2.20.1

[pmg-devel] [PATCH pmg-gui v3 3/3] add PBSConfig tab to Backup menu

[Stoiko Ivanov]

The PBSConfig panel enables creation/editing/deletion of PBS instances.
Each instance can lists its snapshots and each snapshot can be restored

Inspired by the LDAPConfig panel and PBSEdit from pve-manager.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 js/BackupConfiguration.js |   5 +
 js/BackupRestore.js       |   9 +
 js/Makefile               |   1 +
 js/PBSConfig.js           | 678 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 693 insertions(+)
 create mode 100644 js/PBSConfig.js

diff --git a/js/BackupConfiguration.js b/js/BackupConfiguration.js
index 35b50a4..e21771f 100644
--- a/js/BackupConfiguration.js
+++ b/js/BackupConfiguration.js
@@ -13,6 +13,11 @@ Ext.define('PMG.BackupConfiguration', {
 	    title: gettext('Local Backup/Restore'),
 	    xtype: 'pmgBackupRestore',
 	},
+	{
+	    itemId: 'proxmoxbackupserver',
+	    title: 'Proxmox Backup Server',
+	    xtype: 'pmgPBSConfig',
+	},
    ],
 });
 
diff --git a/js/BackupRestore.js b/js/BackupRestore.js
index 2c90f2e..2b9ce53 100644
--- a/js/BackupRestore.js
+++ b/js/BackupRestore.js
@@ -58,6 +58,15 @@ Ext.define('PMG.RestoreWindow', {
 	let initurl = "/nodes/" + Proxmox.NodeName;
 	if (me.filename) {
 	    me.url = initurl + "/backup/" + encodeURIComponent(me.filename);
+	} else if (me.backup_time) {
+	    me.items.push(
+		{
+		    xtype: 'hiddenfield',
+		    name: 'backup-time',
+		    value: me.backup_time,
+		},
+	    );
+	    me.url = initurl + "/pbs/" + me.name + '/restore';
 	} else {
 	    throw "neither filename nor snapshot given";
 	}
diff --git a/js/Makefile b/js/Makefile
index 47eabb7..42eaeb0 100644
--- a/js/Makefile
+++ b/js/Makefile
@@ -37,6 +37,7 @@ JSSRC=							\
 	Subscription.js					\
 	BackupConfiguration.js				\
 	BackupRestore.js				\
+	PBSConfig.js					\
 	SystemConfiguration.js				\
 	MailProxyRelaying.js				\
 	MailProxyPorts.js				\
diff --git a/js/PBSConfig.js b/js/PBSConfig.js
new file mode 100644
index 0000000..9a14d6d
--- /dev/null
+++ b/js/PBSConfig.js
@@ -0,0 +1,678 @@
+Ext.define('Proxmox.form.PBSEncryptionCheckbox', {
+    extend: 'Ext.form.field.Checkbox',
+    xtype: 'pbsEncryptionCheckbox',
+
+    inputValue: true,
+
+    viewModel: {
+	data: {
+	    value: null,
+	    originalValue: null,
+	},
+	formulas: {
+	    blabel: (get) => {
+		let v = get('value');
+		let original = get('originalValue');
+		if (!get('isCreate') && original) {
+		    if (!v) {
+			return gettext('Warning: Existing encryption key will be deleted!');
+		    }
+		    return gettext('Active');
+		} else {
+		    return gettext('Auto-generate a client encryption key, saved privately in /etc/pmg');
+		}
+	    },
+	},
+    },
+
+    bind: {
+	value: '{value}',
+	boxLabel: '{blabel}',
+    },
+    resetOriginalValue: function() {
+	let me = this;
+	let vm = me.getViewModel();
+	vm.set('originalValue', me.value);
+
+	me.callParent(arguments);
+    },
+
+    getSubmitData: function() {
+	let me = this;
+	let val = me.getSubmitValue();
+	if (!me.isCreate) {
+	    if (val === null) {
+	       return { 'delete': 'encryption-key' };
+	    } else if (val && !!val !== !!me.originalValue) {
+	       return { 'encryption-key': 'autogen' };
+	    }
+	} else if (val) {
+	   return { 'encryption-key': 'autogen' };
+	}
+	return null;
+    },
+
+    initComponent: function() {
+	let me = this;
+	me.callParent();
+
+	let vm = me.getViewModel();
+	vm.set('isCreate', me.isCreate);
+    },
+});
+
+Ext.define('PMG.PBSInputPanel', {
+    extend: 'Ext.tab.Panel',
+    xtype: 'pmgPBSInputPanel',
+
+    bodyPadding: 10,
+    remoteId: undefined,
+
+    initComponent: function() {
+	let me = this;
+
+	me.items = [
+	    {
+		title: gettext('Backup Server'),
+		xtype: 'inputpanel',
+		reference: 'remoteeditpanel',
+		onGetValues: function(values) {
+		    values.disable = values.enable ? 0 : 1;
+		    delete values.enable;
+
+		    return values;
+		},
+
+		column1: [
+		    {
+			xtype: me.isCreate ? 'textfield' : 'displayfield',
+			name: 'remote',
+			value: me.isCreate ? null : undefined,
+			fieldLabel: gettext('ID'),
+			allowBlank: false,
+		    },
+		    {
+			xtype: 'proxmoxtextfield',
+			name: 'server',
+			value: me.isCreate ? null : undefined,
+			vtype: 'DnsOrIp',
+			fieldLabel: gettext('Server'),
+			allowBlank: false,
+		    },
+		    {
+			xtype: 'proxmoxtextfield',
+			name: 'datastore',
+			value: me.isCreate ? null : undefined,
+			fieldLabel: 'Datastore',
+			allowBlank: false,
+		    },
+		],
+		column2: [
+		    {
+			xtype: 'proxmoxtextfield',
+			name: 'username',
+			value: me.isCreate ? null : undefined,
+			emptyText: gettext('Example') + ': admin@pbs',
+			fieldLabel: gettext('Username'),
+			regex: /\S+@\w+/,
+			regexText: gettext('Example') + ': admin@pbs',
+			allowBlank: false,
+		    },
+		    {
+			xtype: 'proxmoxtextfield',
+			inputType: 'password',
+			name: 'password',
+			value: me.isCreate ? null : undefined,
+			emptyText: me.isCreate ? gettext('None') : '********',
+			fieldLabel: gettext('Password'),
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'proxmoxcheckbox',
+			name: 'enable',
+			checked: true,
+			uncheckedValue: 0,
+			fieldLabel: gettext('Enable'),
+		    },
+		],
+		columnB: [
+		    {
+			xtype: 'proxmoxtextfield',
+			name: 'fingerprint',
+			value: me.isCreate ? null : undefined,
+			fieldLabel: gettext('Fingerprint'),
+			emptyText: gettext('Server certificate SHA-256 fingerprint, required for self-signed certificates'),
+			regex: /[A-Fa-f0-9]{2}(:[A-Fa-f0-9]{2}){31}/,
+			regexText: gettext('Example') + ': AB:CD:EF:...',
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'pbsEncryptionCheckbox',
+			name: 'encryption-key',
+			isCreate: me.isCreate,
+			fieldLabel: gettext('Encryption Key'),
+		    },
+		    {
+			xtype: 'displayfield',
+			userCls: 'pmx-hint',
+			value: `Proxmox Backup Server is currently in beta.`,
+		    },
+		],
+	    },
+	    {
+		title: gettext('Prune Options'),
+		xtype: 'inputpanel',
+		reference: 'prunepanel',
+		column1: [
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Last'),
+			name: 'keep-last',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Daily'),
+			name: 'keep-daily',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Monthly'),
+			name: 'keep-monthly',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		],
+		column2: [
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Hourly'),
+			name: 'keep-hourly',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Weekly'),
+			name: 'keep-weekly',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		    {
+			xtype: 'proxmoxintegerfield',
+			fieldLabel: gettext('Keep Yearly'),
+			name: 'keep-yearly',
+			cbind: {
+			    deleteEmpty: '{!isCreate}',
+			},
+			minValue: 1,
+			allowBlank: true,
+		    },
+		],
+	    },
+	];
+
+	me.callParent();
+    },
+
+});
+
+Ext.define('PMG.PBSEdit', {
+    extend: 'Proxmox.window.Edit',
+    xtype: 'pmgPBSEdit',
+
+    subject: 'Proxmox Backup Server',
+    isAdd: true,
+
+    bodyPadding: 0,
+
+    initComponent: function() {
+	let me = this;
+
+	me.isCreate = !me.remoteId;
+
+	if (me.isCreate) {
+            me.url = '/api2/extjs/config/pbs';
+            me.method = 'POST';
+	} else {
+            me.url = '/api2/extjs/config/pbs/' + me.remoteId;
+            me.method = 'PUT';
+	}
+
+	let ipanel = Ext.create('PMG.PBSInputPanel', {
+	    isCreate: me.isCreate,
+	    remoteId: me.remoteId,
+	});
+
+	me.items = [ipanel];
+
+	me.fieldDefaults = {
+	    labelWidth: 150,
+	};
+
+	me.callParent();
+
+	if (!me.isCreate) {
+	    me.load({
+		success: function(response, options) {
+		    let values = response.result.data;
+
+		    values.enable = values.disable ? 0 : 1;
+		    me.down('inputpanel[reference=remoteeditpanel]').setValues(values);
+		    me.down('inputpanel[reference=prunepanel]').setValues(values);
+		},
+	    });
+	}
+    },
+});
+
+Ext.define('PMG.PBSScheduleEdit', {
+    extend: 'Proxmox.window.Edit',
+    xtype: 'pmgPBSScheduleEdit',
+
+    isAdd: true,
+    method: 'POST',
+    subject: gettext('Scheduled Backup'),
+    autoLoad: true,
+    items: [
+	{
+	    xtype: 'proxmoxKVComboBox',
+	    name: 'schedule',
+	    fieldLabel: gettext('Schedule'),
+	    comboItems: [
+		['daily', 'daily'],
+		['hourly', 'hourly'],
+		['weekly', 'weekly'],
+		['monthly', 'monthly'],
+	    ],
+	    editable: true,
+	    emptyText: 'Systemd Calender Event',
+	},
+	{
+	    xtype: 'proxmoxKVComboBox',
+	    name: 'delay',
+	    fieldLabel: gettext('Random Delay'),
+	    comboItems: [
+		['0s', 'no delay'],
+		['15 minutes', '15 Minutes'],
+		['6 hours', '6 hours'],
+	    ],
+	    editable: true,
+	    emptyText: 'Systemd TimeSpan',
+	},
+    ],
+    initComponent: function() {
+	let me = this;
+
+	me.url = '/nodes/' + Proxmox.NodeName + '/pbs/' + me.remote + '/timer';
+	me.callParent();
+    },
+});
+
+Ext.define('PMG.PBSConfig', {
+    extend: 'Ext.panel.Panel',
+    xtype: 'pmgPBSConfig',
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	callRestore: function(grid, record) {
+	    let name = this.getViewModel().get('name');
+	    Ext.create('PMG.RestoreWindow', {
+		name: name,
+		backup_time: record.data.time,
+	    }).show();
+	},
+
+	restoreSnapshot: function(button) {
+	    let me = this;
+	    let view = me.lookup('pbsremotegrid');
+	    let record = view.getSelection()[0];
+	    me.callRestore(view, record);
+	},
+
+	runBackup: function(button) {
+	    let me = this;
+	    let view = me.lookup('pbsremotegrid');
+	    let name = me.getViewModel().get('name');
+	    Proxmox.Utils.API2Request({
+		url: "/nodes/" + Proxmox.NodeName + "/pbs/" + name + "/backup",
+		method: 'POST',
+		waitMsgTarget: view,
+		failure: function(response, opts) {
+		    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+		},
+		success: function(response, opts) {
+		    let upid = response.result.data;
+
+		    let win = Ext.create('Proxmox.window.TaskViewer', {
+			upid: upid,
+		    });
+		    win.show();
+		    me.mon(win, 'close', function() { view.getStore().load(); });
+		},
+	    });
+	},
+
+	reload: function(grid) {
+	    let me = this;
+	    let selection = grid.getSelection();
+	    me.showInfo(grid, selection);
+	},
+
+	showInfo: function(grid, selected) {
+	    let me = this;
+	    let viewModel = me.getViewModel();
+	    if (selected[0]) {
+		let name = selected[0].data.remote;
+		viewModel.set('selected', true);
+		viewModel.set('name', name);
+
+		// set grid stores and load them
+		let remstore = me.lookup('pbsremotegrid').getStore();
+		remstore.getProxy().setUrl('/api2/json/nodes/' + Proxmox.NodeName + '/pbs/' + name + '/snapshots');
+		remstore.load();
+	    } else {
+		viewModel.set('selected', false);
+	    }
+	},
+	reloadSnapshots: function() {
+	    let me = this;
+	    let grid = me.lookup('grid');
+	    let selection = grid.getSelection();
+	    me.showInfo(grid, selection);
+	},
+	init: function(view) {
+	    let me = this;
+	    me.lookup('grid').relayEvents(view, ['activate']);
+	    let pbsremotegrid = me.lookup('pbsremotegrid');
+
+	    Proxmox.Utils.monStoreErrors(pbsremotegrid, pbsremotegrid.getStore(), true);
+	},
+
+	control: {
+	    'grid[reference=grid]': {
+		selectionchange: 'showInfo',
+		load: 'reload',
+	    },
+	    'grid[reference=pbsremotegrid]': {
+		itemdblclick: 'restoreSnapshot',
+	    },
+	},
+    },
+
+    viewModel: {
+	data: {
+	    name: '',
+	    selected: false,
+	},
+    },
+
+    layout: 'border',
+
+    items: [
+	{
+	    region: 'center',
+	    reference: 'grid',
+	    xtype: 'pmgPBSConfigGrid',
+	    border: false,
+	},
+	{
+	    xtype: 'grid',
+	    region: 'south',
+	    reference: 'pbsremotegrid',
+	    hidden: true,
+	    height: '70%',
+	    border: false,
+	    split: true,
+	    emptyText: gettext('No backups on remote'),
+	    tbar: [
+		{
+		    xtype: 'proxmoxButton',
+		    text: gettext('Backup'),
+		    handler: 'runBackup',
+		    selModel: false,
+		},
+		{
+		    xtype: 'proxmoxButton',
+		    text: gettext('Restore'),
+		    handler: 'restoreSnapshot',
+		    disabled: true,
+		},
+		{
+		    xtype: 'proxmoxStdRemoveButton',
+		    text: gettext('Forget Snapshot'),
+		    disabled: true,
+		    getUrl: function(rec) {
+			let me = this;
+			let remote = me.lookupViewModel().get('name');
+			return '/nodes/' + Proxmox.NodeName + '/pbs/' + remote +'/snapshots/'+ rec.data.time;
+		    },
+		    confirmMsg: function(rec) {
+			let me = this;
+			let time = rec.data.time;
+			return Ext.String.format(gettext('Are you sure you want to forget snapshot {0}'), `'${time}'`);
+		    },
+		    callback: 'reloadSnapshots',
+		},
+	    ],
+	    store: {
+		fields: ['time', 'size', 'ctime', 'encrypted'],
+		proxy: { type: 'proxmox' },
+		sorters: [
+		    {
+			property: 'time',
+			direction: 'DESC',
+		    },
+		],
+	    },
+	    bind: {
+		title: Ext.String.format(gettext("Backup snapshots on '{0}'"), '{name}'),
+		hidden: '{!selected}',
+	    },
+	    columns: [
+		{
+		    text: 'Time',
+		    dataIndex: 'time',
+		    flex: 1,
+		},
+		{
+		    text: 'Size',
+		    dataIndex: 'size',
+		    flex: 1,
+		},
+		{
+		    text: 'Encrypted',
+		    dataIndex: 'encrypted',
+		    renderer: Proxmox.Utils.format_boolean,
+		    flex: 1,
+		},
+	    ],
+	},
+    ],
+
+});
+
+Ext.define('pmg-pbs-config', {
+    extend: 'Ext.data.Model',
+    fields: ['remote', 'server', 'datastore', 'username', 'disabled'],
+    proxy: {
+	type: 'proxmox',
+	url: '/api2/json/config/pbs',
+    },
+    idProperty: 'remote',
+});
+
+Ext.define('PMG.PBSConfigGrid', {
+    extend: 'Ext.grid.GridPanel',
+    xtype: 'pmgPBSConfigGrid',
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	run_editor: function() {
+	    let me = this;
+	    let view = me.getView();
+	    let rec = view.getSelection()[0];
+	    if (!rec) {
+		return;
+	    }
+
+	    let win = Ext.createWidget('pmgPBSEdit', {
+		remoteId: rec.data.remote,
+	    });
+	    win.on('destroy', me.reload, me);
+	    win.load();
+	    win.show();
+	},
+
+	newRemote: function() {
+	    let me = this;
+	    let win = Ext.createWidget('pmgPBSEdit', {});
+	    win.on('destroy', me.reload, me);
+	    win.show();
+	},
+
+
+	reload: function() {
+	    let me = this;
+	    let view = me.getView();
+	    view.getStore().load();
+	    view.fireEvent('load', view);
+	},
+
+	createSchedule: function() {
+	    let me = this;
+	    let view = me.getView();
+	    let rec = view.getSelection()[0];
+	    let remotename = rec.data.remote;
+	    let win = Ext.createWidget('pmgPBSScheduleEdit', {
+		remote: remotename,
+	    });
+	    win.on('destroy', me.reload, me);
+	    win.show();
+	},
+
+	init: function(view) {
+	    let me = this;
+	    Proxmox.Utils.monStoreErrors(view, view.getStore(), true);
+	},
+
+    },
+
+    store: {
+	model: 'pmg-pbs-config',
+	sorters: [{
+	    property: 'remote',
+	    order: 'DESC',
+	}],
+    },
+
+    tbar: [
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Edit'),
+	    disabled: true,
+	    handler: 'run_editor',
+	},
+	{
+	    text: gettext('Create'),
+	    handler: 'newRemote',
+	},
+	{
+	    xtype: 'proxmoxStdRemoveButton',
+	    baseurl: '/config/pbs',
+	    callback: 'reload',
+	},
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Schedule'),
+	    enableFn: function(rec) {
+		return !rec.data.disable;
+	    },
+	    disabled: true,
+	    handler: 'createSchedule',
+	},
+	{
+	    xtype: 'proxmoxStdRemoveButton',
+	    baseurl: '/nodes/' + Proxmox.NodeName + '/pbs/',
+	    callback: 'reload',
+	    text: gettext('Remove Schedule'),
+	    confirmMsg: function(rec) {
+		let me = this;
+		let name = rec.getId();
+		return Ext.String.format(gettext('Are you sure you want to remove the schedule for {0}'), `'${name}'`);
+	    },
+	    getUrl: function(rec) {
+		let me = this;
+		return me.baseurl + '/' + rec.getId() + '/timer';
+	    },
+	},
+    ],
+
+    listeners: {
+	itemdblclick: 'run_editor',
+	activate: 'reload',
+    },
+
+    columns: [
+	{
+	    header: gettext('Backup Server name'),
+	    sortable: true,
+	    dataIndex: 'remote',
+	    flex: 2,
+	},
+	{
+	    header: gettext('Server'),
+	    sortable: true,
+	    dataIndex: 'server',
+	    flex: 2,
+	},
+	{
+	    header: gettext('Datastore'),
+	    sortable: true,
+	    dataIndex: 'datastore',
+	    flex: 1,
+	},
+	{
+	    header: gettext('User ID'),
+	    sortable: true,
+	    dataIndex: 'username',
+	    flex: 1,
+	},
+	{
+	    header: gettext('Encryption'),
+	    width: 80,
+	    sortable: true,
+	    dataIndex: 'encryption-key',
+	    renderer: Proxmox.Utils.format_boolean,
+	},
+	{
+	    header: gettext('Enabled'),
+	    width: 80,
+	    sortable: true,
+	    dataIndex: 'disable',
+	    renderer: Proxmox.Utils.format_neg_boolean,
+	},
+    ],
+
+});
+
-- 
2.20.1

[pmg-devel] applied: [PATCH pve-common v3 1/1] add PBSClient module

[Thomas Lamprecht]

On 16.11.20 12:01, Stoiko Ivanov wrote:
> PBSClient.pm contains methods for:
> * handling (sensitive) config-information (passwords, encryption keys)
> * creating/restoring/forgetting/listing backups
> 
> code is mostly based on the current PBSPlugin in pve-storage
> 
> Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
> ---
>  src/Makefile         |   1 +
>  src/PVE/PBSClient.pm | 305 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 306 insertions(+)
>  create mode 100644 src/PVE/PBSClient.pm
> 
>

applied, thanks!

Added some followups, mostly smaller cleanups though:

pbs: restore pxar: add required parameters explicitly in method signature
pbs: code cleanup param array assembly
pbs: autogen key: adapt recent changes in storage module
PBS client: use our normal blessed method call style

[pmg-devel] applied-series: [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration

[Thomas Lamprecht]

On 16.11.20 12:01, Stoiko Ivanov wrote:
> changes v2->v3:
> * rebased the gui-patches to latest master
> * added param_mapping from pve-storage/pvesm (provide an interactive password
>   entry)
> 

applied series, with a few followups as discussed off-list, thanks!