[pmg-devel] [RFC pmg-api 00/12] POC PBS integration

[pmg-devel] [RFC pmg-api 00/12] POC PBS integration

[Stoiko Ivanov]

This series is a minimal proof-of-concept for integrating PBS into PMG.

The code needs quite a bit of cleanup, and more testing, however I'll send it
as RFC, for an initial sanity-check - to see if I missed some needed
functionality.

It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd
unit manipulation from the PVE::API2::Disks::Directory) and I'd like to
refactor the common parts into pve-common.

The backup consists of the same files, that end up in a locally generated
PMG Backup (instead of creating a tar.gz we send the data to a PBS instance).

What works:
* creating and manipulating 'remotes' (PBS instances where we back up to)
* creating, restoring and forgetting backups to those remotes
* restoring a backup from another PMG installation from a configured remote
* creating schedules for backups (systemd-timer units) for periodic backups
via CLI (pmgbackup, pmgsh)

What's missing:
* quite a bit of refactoring (e.g. move the systemd-timer helpers and most of
  PBSTools to pve-common (and reuse it in the PBS Storage plugin)
* encryption support
* pruneing support
* testing
* GUI
* Documentation

The first patch is a small cosmetic cleanup, and independent of the series.

A sample session (which I used for testing:
```
pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint  ff:ff:...
pmgbackup remote list
pmgbackup pbsjob run pbs1
pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1
pmgbackup pbsjob run pbs1
pmgbackup pbsjob list_backups pbs1
pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s'
systemctl list-timers --all
```

Stoiko Ivanov (12):
  drop left-over commented out code
  Backup: split backup creation and creating tar
  Restore: optionally restore from directory
  Backup: push restore options to PMG::Backup
  debian: add dependency on proxmox-backup-client
  add helper module for handling PBS Integration
  PBSTools: add methods for managing backups
  PBSTools: add systemd-timer helpers
  add initial SectionConfig for pbs
  Add API2 module for PBS configuration
  Add API2 module for per-node backups to PBS
  pbs-integration: add CLI calls to pmgbackup

 debian/control                |   1 +
 debian/dirs                   |   1 +
 debian/pmg-pbsbackup@.service |   6 +
 debian/rules                  |   1 +
 src/Makefile                  |   6 +-
 src/PMG/API2/Backup.pm        |  21 +-
 src/PMG/API2/Config.pm        |   7 +
 src/PMG/API2/Nodes.pm         |   7 +
 src/PMG/API2/PBS/Job.pm       | 467 ++++++++++++++++++++++++++++++++++
 src/PMG/API2/PBS/Remote.pm    | 218 ++++++++++++++++
 src/PMG/Backup.pm             |  86 +++++--
 src/PMG/CLI/pmgbackup.pm      |  27 ++
 src/PMG/PBSConfig.pm          | 168 ++++++++++++
 src/PMG/PBSTools.pm           | 466 +++++++++++++++++++++++++++++++++
 14 files changed, 1447 insertions(+), 35 deletions(-)
 create mode 100644 debian/pmg-pbsbackup@.service
 create mode 100644 src/PMG/API2/PBS/Job.pm
 create mode 100644 src/PMG/API2/PBS/Remote.pm
 create mode 100644 src/PMG/PBSConfig.pm
 create mode 100644 src/PMG/PBSTools.pm

-- 
2.20.1

[pmg-devel] [RFC pmg-api 01/12] drop left-over commented out code

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/Backup.pm | 2 --
 1 file changed, 2 deletions(-)

diff --git a/src/PMG/Backup.pm b/src/PMG/Backup.pm
index 11eff5f..025bac2 100644
--- a/src/PMG/Backup.pm
+++ b/src/PMG/Backup.pm
@@ -186,8 +186,6 @@ sub pmg_backup {
 
 	push @$extra_cfgs, $sa_custom_config_fn;
 
-	#push @$extra_cfgs, '/etc/postfix/tls_policy';
-
 	my $extradb = $include_statistics ? $statfn : '';
 
 	my $extra = join(' ', @$extra_cfgs);
-- 
2.20.1

[pmg-devel] [RFC pmg-api 02/12] Backup: split backup creation and creating tar

[Stoiko Ivanov]

In preparation for integrating PMG with PBS split the current creation of
a PMG backup into 2 methods:
* create all files in a backup in a target directory
* create a tarball from a backup in a temporary directory

use the changed method in the backup API call.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/API2/Backup.pm |  2 +-
 src/PMG/Backup.pm      | 41 ++++++++++++++++++++++++++++++++---------
 2 files changed, 33 insertions(+), 10 deletions(-)

diff --git a/src/PMG/API2/Backup.pm b/src/PMG/API2/Backup.pm
index 0bfcfc9..08c06b5 100644
--- a/src/PMG/API2/Backup.pm
+++ b/src/PMG/API2/Backup.pm
@@ -131,7 +131,7 @@ __PACKAGE__->register_method ({
 
 	    print "starting backup to: $filename\n";
 
-	    PMG::Backup::pmg_backup($filename, $param->{statistic});
+	    PMG::Backup::pmg_backup_pack($filename, $param->{statistic});
 
 	    print "backup finished\n";
 
diff --git a/src/PMG/Backup.pm b/src/PMG/Backup.pm
index 025bac2..3854a5d 100644
--- a/src/PMG/Backup.pm
+++ b/src/PMG/Backup.pm
@@ -133,10 +133,11 @@ sub dumpstatdb {
 }
 
 sub pmg_backup {
-    my ($filename, $include_statistics) = @_;
+    my ($backupdir, $include_statistics) = @_;
+
 
     my $time = time;
-    my $dirname = "/tmp/proxbackup_$$.$time";
+    my $dirname = $backupdir // '/var/lib/pmg/backup/current';
     my $dbfn = "Proxmox_ruledb.sql";
     my $statfn = "Proxmox_statdb.sql";
     my $tarfn = "config_backup.tar";
@@ -145,12 +146,7 @@ sub pmg_backup {
 
     eval {
 
-	my $targetdir = dirname($filename);
-	mkdir $targetdir; # try to create target dir
-	-d $targetdir ||
-	    die "unable to access target directory '$targetdir'\n";
-
-	# create a temporary directory
+	# create backup directory
 	mkdir $dirname;
 
 	# dump the database first
@@ -197,7 +193,34 @@ sub pmg_backup {
 	system("cd $dirname; md5sum $tarfn $dbfn $extradb $verfn> $sigfn") == 0 ||
 	    die "unable to create backup signature: ERROR";
 
-	system("rm -f $filename; tar czf $filename -C $dirname $verfn $sigfn $dbfn $extradb $tarfn") == 0 ||
+    };
+    my $err = $@;
+
+    if ($err) {
+	die $err;
+    }
+}
+
+sub pmg_backup_pack {
+    my ($filename, $include_statistics) = @_;
+
+    my $time = time;
+    my $dirname = "/tmp/proxbackup_$$.$time";
+
+    eval {
+
+	my $targetdir = dirname($filename);
+	mkdir $targetdir; # try to create target dir
+	-d $targetdir ||
+	    die "unable to access target directory '$targetdir'\n";
+
+	rmtree $dirname;
+	# create backup directory
+	mkdir $dirname;
+
+	pmg_backup($dirname, $include_statistics);
+
+	system("rm -f $filename; tar czf $filename --strip-components=1 -C $dirname .") == 0 ||
 	    die "unable to create backup archive: ERROR";
     };
     my $err = $@;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 03/12] Restore: optionally restore from directory

[Stoiko Ivanov]

In preparation for integrating PMG with PBS decided based on the type of the
provided filename, whether or not to untar:
* if it's a directory skip untarring (PBS)
* if it's a filename untar (local backup)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/Backup.pm | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/src/PMG/Backup.pm b/src/PMG/Backup.pm
index 3854a5d..fd7274c 100644
--- a/src/PMG/Backup.pm
+++ b/src/PMG/Backup.pm
@@ -245,12 +245,23 @@ sub pmg_restore {
     my $tarfn = "config_backup.tar";
     my $sigfn = "proxmox_backup_v1.md5";
 
+    my $untar = 1;
+
+    # directory indicates that the files were restored from a PBS remote
+    if ( -d $filename ) {
+	$dirname = $filename;
+	$untar = 0;
+    }
+
     eval {
-	# create a temporary directory
-	mkdir $dirname;
 
-	system("cd $dirname; tar xzf $filename >/dev/null 2>&1") == 0 ||
-	    die "unable to extract backup archive: ERROR";
+	if ($untar) {
+	    # create a temporary directory
+	    mkdir $dirname;
+
+	    system("cd $dirname; tar xzf $filename >/dev/null 2>&1") == 0 ||
+		die "unable to extract backup archive: ERROR";
+	}
 
 	system("cd $dirname; md5sum -c $sigfn") == 0 ||
 	    die "proxmox backup signature check failed: ERROR";
-- 
2.20.1

[pmg-devel] [RFC pmg-api 04/12] Backup: push restore options to PMG::Backup

[Stoiko Ivanov]

that way they can be reused for the PBS restore API call

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/API2/Backup.pm | 19 +------------------
 src/PMG/Backup.pm      | 24 ++++++++++++++++++++++++
 2 files changed, 25 insertions(+), 18 deletions(-)

diff --git a/src/PMG/API2/Backup.pm b/src/PMG/API2/Backup.pm
index 08c06b5..4ea28d1 100644
--- a/src/PMG/API2/Backup.pm
+++ b/src/PMG/API2/Backup.pm
@@ -202,26 +202,9 @@ __PACKAGE__->register_method ({
     parameters => {
 	additionalProperties => 0,
 	properties => {
+	    PMG::Backup::get_restore_options(),
 	    node => get_standard_option('pve-node'),
 	    filename => $backup_filename_property,
-	    config => {
-		description => "Restore system configuration.",
-		type => 'boolean',
-		optional => 1,
-		default => 0,
-	    },
-	    database => {
-		description => "Restore the rule database. This is the default.",
-		type => 'boolean',
-		optional => 1,
-		default => 1,
-	    },
-	    statistic => {
-		description => "Restore statistic databases. Only considered when you restore the 'database'.",
-		type => 'boolean',
-		optional => 1,
-		default => 0,
-	    },
 	},
     },
     returns => { type => "string" },
diff --git a/src/PMG/Backup.pm b/src/PMG/Backup.pm
index fd7274c..aba17ad 100644
--- a/src/PMG/Backup.pm
+++ b/src/PMG/Backup.pm
@@ -6,6 +6,7 @@ use Data::Dumper;
 use File::Basename;
 use File::Path;
 
+use PVE::JSONSchema qw(get_standard_option);
 use PVE::Tools;
 
 use PMG::pmgcfg;
@@ -14,6 +15,29 @@ use PMG::Utils qw(postgres_admin_cmd);
 
 my $sa_custom_config_fn = "/etc/mail/spamassassin/custom.cf";
 
+sub get_restore_options {
+    return (
+	node => get_standard_option('pve-node'),
+	config => {
+	    description => "Restore system configuration.",
+	    type => 'boolean',
+	    optional => 1,
+	    default => 0,
+	},
+	database => {
+	    description => "Restore the rule database. This is the default.",
+	    type => 'boolean',
+	    optional => 1,
+	    default => 1,
+	},
+	statistic => {
+	    description => "Restore statistic databases. Only considered when you restore the 'database'.",
+	    type => 'boolean',
+	    optional => 1,
+	    default => 0,
+	});
+}
+
 sub dump_table {
     my ($dbh, $table, $ofh, $seq, $seqcol) = @_;
 
-- 
2.20.1

[pmg-devel] [RFC pmg-api 05/12] debian: add dependency on proxmox-backup-client

[Stoiko Ivanov]

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/control | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/control b/debian/control
index af10bc0..2460411 100644
--- a/debian/control
+++ b/debian/control
@@ -76,6 +76,7 @@ Depends: apt,
          pmg-log-tracker,
          postfix (>= 2.5.5),
          postgresql-11,
+         proxmox-backup-client,,
          proxmox-mini-journalreader,
          proxmox-spamassassin,
          pve-xtermjs (>= 1.0-1),
-- 
2.20.1

[pmg-devel] [RFC pmg-api 06/12] add helper module for handling PBS Integration

[Stoiko Ivanov]

PBSTools.pm contains methods which eventually should be shared between
PVE and PMG, for:
* handling (sensitive) config-information (passwords, encryption keys)
* running PBS operations

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile        |   1 +
 src/PMG/PBSTools.pm | 239 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 240 insertions(+)
 create mode 100644 src/PMG/PBSTools.pm

diff --git a/src/Makefile b/src/Makefile
index 05d9598..7f9726b 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -66,6 +66,7 @@ LIBSOURCES =				\
 	PMG/SMTP.pm			\
 	PMG/Unpack.pm			\
 	PMG/Backup.pm			\
+	PMG/PBSTools.pm			\
 	PMG/RuleCache.pm		\
 	PMG/Statistic.pm		\
 	PMG/UserConfig.pm		\
diff --git a/src/PMG/PBSTools.pm b/src/PMG/PBSTools.pm
new file mode 100644
index 0000000..e2a2b60
--- /dev/null
+++ b/src/PMG/PBSTools.pm
@@ -0,0 +1,239 @@
+package PMG::PBSTools;
+
+# utility functions to talk with Proxmox Backup Server
+
+use strict;
+use warnings;
+# FIXME: cleanup to only needed modules:
+use Fcntl qw(F_GETFD F_SETFD FD_CLOEXEC);
+use HTTP::Request;
+use IO::File;
+use JSON;
+use LWP::UserAgent;
+use POSIX qw(strftime ENOENT);
+
+use PVE::Tools qw(run_command file_set_contents file_get_contents file_read_firstline trim dir_glob_foreach);
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::Systemd;
+
+# Helpers
+my $secret_dir;
+
+sub set_secret_dir {
+    my ($dir) = @_;
+    $secret_dir = $dir;
+}
+
+sub pbs_password_file_name {
+    my ($scfg, $storeid) = @_;
+
+    return "${secret_dir}/${storeid}.pw";
+}
+
+sub pbs_set_password {
+    my ($scfg, $storeid, $password) = @_;
+
+    my $pwfile = pbs_password_file_name($scfg, $storeid);
+    mkdir $secret_dir;
+
+    PVE::Tools::file_set_contents($pwfile, "$password\n", 0600);
+}
+
+sub pbs_delete_password {
+    my ($scfg, $storeid) = @_;
+
+    my $pwfile = pbs_password_file_name($scfg, $storeid);
+
+    unlink $pwfile;
+}
+
+sub pbs_get_password {
+    my ($scfg, $storeid) = @_;
+
+    my $pwfile = pbs_password_file_name($scfg, $storeid);
+
+    return PVE::Tools::file_read_firstline($pwfile);
+}
+
+sub pbs_encryption_key_file_name {
+    my ($scfg, $storeid) = @_;
+
+    return "${secret_dir}/${storeid}.enc";
+}
+
+sub pbs_set_encryption_key {
+    my ($scfg, $storeid, $key) = @_;
+
+    my $encfile = pbs_encryption_key_file_name($scfg, $storeid);
+    mkdir $secret_dir;
+
+    PVE::Tools::file_set_contents($encfile, "$key\n", 0600);
+}
+
+sub pbs_delete_encryption_key {
+    my ($scfg, $storeid) = @_;
+
+    my $encfile = pbs_encryption_key_file_name($scfg, $storeid);
+
+    if (!unlink $encfile) {
+	return if $! == ENOENT;
+	die "failed to delete encryption key! $!\n";
+    }
+}
+
+sub pbs_get_encryption_key {
+    my ($scfg, $storeid) = @_;
+
+    my $encfile = pbs_encryption_key_file_name($scfg, $storeid);
+
+    return PVE::Tools::file_get_contents($encfile);
+}
+
+# Returns a file handle if there is an encryption key, or `undef` if there is not. Dies on error.
+sub pbs_open_encryption_key {
+    my ($scfg, $storeid) = @_;
+
+    my $encryption_key_file = pbs_encryption_key_file_name($scfg, $storeid);
+
+    my $keyfd;
+    if (!open($keyfd, '<', $encryption_key_file)) {
+	return undef if $! == ENOENT;
+	die "failed to open encryption key: $encryption_key_file: $!\n";
+    }
+
+    return $keyfd;
+}
+
+my $USE_CRYPT_PARAMS = {
+    backup => 1,
+    restore => 1,
+    'upload-log' => 1,
+};
+
+my sub do_raw_client_cmd {
+    my ($scfg, $storeid, $client_cmd, $param, %opts) = @_;
+
+    #FIXME: my $use_crypto = $USE_CRYPT_PARAMS->{$client_cmd};
+    my $use_crypto = 0;
+
+    my $client_exe = '/usr/bin/proxmox-backup-client';
+    die "executable not found '$client_exe'! Proxmox backup client not installed?\n"
+	if ! -x $client_exe;
+
+    my $server = $scfg->{server};
+    my $datastore = $scfg->{datastore};
+    my $username = $scfg->{username} // 'root@pam';
+
+    my $userns_cmd = delete $opts{userns_cmd};
+
+    my $cmd = [];
+
+    push @$cmd, @$userns_cmd if defined($userns_cmd);
+
+    push @$cmd, $client_exe, $client_cmd;
+
+    # This must live in the top scope to not get closed before the `run_command`
+    my $keyfd;
+    if ($use_crypto) {
+    if (defined($keyfd = pbs_open_encryption_key($scfg, $storeid))) {
+	    my $flags = fcntl($keyfd, F_GETFD, 0)
+		// die "failed to get file descriptor flags: $!\n";
+	    fcntl($keyfd, F_SETFD, $flags & ~FD_CLOEXEC)
+		or die "failed to remove FD_CLOEXEC from encryption key file descriptor\n";
+	    push @$cmd, '--crypt-mode=encrypt', '--keyfd='.fileno($keyfd);
+	} else {
+	    push @$cmd, '--crypt-mode=none';
+	}
+    }
+
+    push @$cmd, @$param if defined($param);
+
+    push @$cmd, "--repository", "$username\@$server:$datastore";
+
+    local $ENV{PBS_PASSWORD} = pbs_get_password($scfg, $storeid);
+
+    local $ENV{PBS_FINGERPRINT} = $scfg->{fingerprint};
+
+    # no ascii-art on task logs
+    local $ENV{PROXMOX_OUTPUT_NO_BORDER} = 1;
+    local $ENV{PROXMOX_OUTPUT_NO_HEADER} = 1;
+
+    if (my $logfunc = $opts{logfunc}) {
+	$logfunc->("run: " . join(' ', @$cmd));
+    }
+
+    run_command($cmd, %opts);
+}
+
+# FIXME: External perl code should NOT have access to this.
+#
+# There should be separate functions to
+# - make backups
+# - restore backups
+# - restore files
+# with a sane API
+sub run_raw_client_cmd {
+    my ($scfg, $storeid, $client_cmd, $param, %opts) = @_;
+    return do_raw_client_cmd($scfg, $storeid, $client_cmd, $param, %opts);
+}
+
+sub run_client_cmd {
+    my ($scfg, $storeid, $client_cmd, $param, $no_output) = @_;
+
+    my $json_str = '';
+    my $outfunc = sub { $json_str .= "$_[0]\n" };
+
+    $param = [] if !defined($param);
+    $param = [ $param ] if !ref($param);
+
+    $param = [@$param, '--output-format=json'] if !$no_output;
+
+    do_raw_client_cmd($scfg, $storeid, $client_cmd, $param,
+		      outfunc => $outfunc, errmsg => 'proxmox-backup-client failed');
+
+    return undef if $no_output;
+
+    my $res = decode_json($json_str);
+
+    return $res;
+}
+
+my $autogen_encryption_key = sub {
+    my ($scfg, $storeid) = @_;
+    my $encfile = pbs_encryption_key_file_name($storeid);
+    run_command(['proxmox-backup-client', 'key', 'create', '--kdf', 'none', $encfile]);
+};
+
+sub print_snapshot {
+    my ($storeid, $btype, $bid, $btime) = @_;
+
+    my $time_str = strftime("%FT%TZ", gmtime($btime));
+    my $volname = "backup/${btype}/${bid}/${time_str}";
+
+    return "${storeid}:${volname}";
+}
+
+sub status {
+    my ($class, $storeid, $scfg) = @_;
+
+    my $total = 0;
+    my $free = 0;
+    my $used = 0;
+    my $active = 0;
+
+    eval {
+	my $res = run_client_cmd($scfg, $storeid, "status");
+
+	$active = 1;
+	$total = $res->{total};
+	$used = $res->{used};
+	$free = $res->{avail};
+    };
+    if (my $err = $@) {
+	warn $err;
+    }
+
+    return ($total, $free, $used, $active);
+}
+
+1;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 07/12] PBSTools: add methods for managing backups

[Stoiko Ivanov]

* creating/restoring/forgetting/listing backups

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/PBSTools.pm | 62 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/src/PMG/PBSTools.pm b/src/PMG/PBSTools.pm
index e2a2b60..9efb2ce 100644
--- a/src/PMG/PBSTools.pm
+++ b/src/PMG/PBSTools.pm
@@ -213,6 +213,68 @@ sub print_snapshot {
     return "${storeid}:${volname}";
 }
 
+sub get_snapshots {
+    my ($scfg, $remote) = @_;
+
+    return run_client_cmd($scfg, $remote, "snapshots");
+}
+
+sub backup_tree {
+    my ($scfg, $remote, $opts) = @_;
+
+    my $type = delete $opts->{type};
+    die "backup-type not provided\n" if !defined($type);
+    my $id = delete $opts->{id};
+    die "backup-id not provided\n" if !defined($id);
+    my $root = delete $opts->{root};
+    die "root dir not provided\n" if !defined($root);
+    my $pxarname = delete $opts->{pxarname};
+    die "archive name not provided\n" if !defined($pxarname);
+    my $time = delete $opts->{time};
+
+    my $param = [];
+
+    push @$param, "$pxarname.pxar:$root";
+    push @$param, '--backup-type', $type;
+    push @$param, '--backup-id', $id;
+    push @$param, '--backup-time', $time if defined($time);
+
+    return run_raw_client_cmd($scfg, $remote, 'backup', $param, %$opts);
+}
+
+sub restore_pxar {
+    my ($scfg, $remote, $opts) = @_;
+
+    my $snapshot = delete $opts->{snapshot};
+    die "snapshot not provided\n" if !defined($snapshot);
+    my $pxarname = delete $opts->{pxarname};
+    die "archive name not provided\n" if !defined($pxarname);
+    my $target = delete $opts->{target};
+    die "restore-target not provided\n" if !defined($target);
+    #my $time = delete $opts->{time};
+
+    my $param = [];
+
+    push @$param, "$snapshot";
+    push @$param, "$pxarname.pxar";
+    push @$param, "$target";
+    push @$param, "--allow-existing-dirs", 0;
+
+    return run_raw_client_cmd($scfg, $remote, 'restore', $param, %$opts);
+}
+
+sub forget_snapshot {
+    my ($scfg, $remote, $snapshot) = @_;
+
+    die "snapshot not provided\n" if !defined($snapshot);
+
+    my $param = [];
+
+    push @$param, "$snapshot";
+
+    return run_raw_client_cmd($scfg, $remote, 'forget', $param);
+}
+
 sub status {
     my ($class, $storeid, $scfg) = @_;
 
-- 
2.20.1

[pmg-devel] [RFC pmg-api 08/12] PBSTools: add systemd-timer helpers

[Stoiko Ivanov]

add helper methods to create systemd-timer units, which run backups to a
PBS-Remote regularly.

Unit-file handling taken from pve-storage/PVE/API2/Disks/Directory.pm

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/pmg-pbsbackup@.service |   6 ++
 debian/rules                  |   1 +
 src/Makefile                  |   2 +-
 src/PMG/PBSTools.pm           | 165 ++++++++++++++++++++++++++++++++++
 4 files changed, 173 insertions(+), 1 deletion(-)
 create mode 100644 debian/pmg-pbsbackup@.service

diff --git a/debian/pmg-pbsbackup@.service b/debian/pmg-pbsbackup@.service
new file mode 100644
index 0000000..37aa23b
--- /dev/null
+++ b/debian/pmg-pbsbackup@.service
@@ -0,0 +1,6 @@
+[Unit]
+Description=Backup to PBS remote %I
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/pmgbackup pbsjob run %I
diff --git a/debian/rules b/debian/rules
index bab4d98..5a2cf7a 100755
--- a/debian/rules
+++ b/debian/rules
@@ -20,6 +20,7 @@ override_dh_installinit:
 	dh_systemd_enable --name=pmgspamreport pmgspamreport.service
 	dh_systemd_enable --name=pmgreport pmgreport.service
 	dh_systemd_enable --name=pmgsync pmgsync.service
+	dh_systemd_enable --no-enable --name=pmg-pbsbackup@ pmg-pbsbackup@.service
 
 override_dh_systemd_start:
 	dh_systemd_start pmg-hourly.timer pmg-daily.timer pmgspamreport.timer pmgreport.timer
diff --git a/src/Makefile b/src/Makefile
index 7f9726b..a460048 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -15,7 +15,7 @@ CRONSCRIPTS = pmg-hourly pmg-daily
 
 CLI_CLASSES = $(addprefix PMG/CLI/, $(addsuffix .pm, ${CLITOOLS}))
 SERVICE_CLASSES = $(addprefix PMG/Service/, $(addsuffix .pm, ${SERVICES}))
-SERVICE_UNITS = $(addprefix debian/, $(addsuffix .service, ${SERVICES}))
+SERVICE_UNITS = $(addprefix debian/, $(addsuffix .service, ${SERVICES})) debian/pmg-pbsbackup@.service
 TIMER_UNITS = $(addprefix debian/, $(addsuffix .timer, ${CRONSCRIPTS} pmgspamreport pmgreport))
 
 CLI_BINARIES = $(addprefix bin/, ${CLITOOLS} ${CLISCRIPTS} ${CRONSCRIPTS})
diff --git a/src/PMG/PBSTools.pm b/src/PMG/PBSTools.pm
index 9efb2ce..fcef74a 100644
--- a/src/PMG/PBSTools.pm
+++ b/src/PMG/PBSTools.pm
@@ -298,4 +298,169 @@ sub status {
     return ($total, $free, $used, $active);
 }
 
+# systemd timer
+my $read_ini = sub {
+    my ($filename) = @_;
+
+    my $content = file_get_contents($filename);
+    my @lines = split /\n/, $content;
+
+    my $result = {};
+    my $section;
+
+    foreach my $line (@lines) {
+	$line = trim($line);
+	if ($line =~ m/^\[([^\]]+)\]/) {
+	    $section = $1;
+	    if (!defined($result->{$section})) {
+		$result->{$section} = {};
+	    }
+	} elsif ($line =~ m/^(.*?)=(.*)$/) {
+	    my ($key, $val) = ($1, $2);
+	    if (!$section) {
+		warn "key value pair found without section, skipping\n";
+		next;
+	    }
+
+	    if ($result->{$section}->{$key}) {
+		# make duplicate properties to arrays to keep the order
+		my $prop = $result->{$section}->{$key};
+		if (ref($prop) eq 'ARRAY') {
+		    push @$prop, $val;
+		} else {
+		    $result->{$section}->{$key} = [$prop, $val];
+		}
+	    } else {
+		$result->{$section}->{$key} = $val;
+	    }
+	}
+	# ignore everything else
+    }
+
+    return $result;
+};
+
+my $write_ini = sub {
+    my ($ini, $filename) = @_;
+
+    my $content = "";
+
+    foreach my $sname (sort keys %$ini) {
+	my $section = $ini->{$sname};
+
+	$content .= "[$sname]\n";
+
+	foreach my $pname (sort keys %$section) {
+	    my $prop = $section->{$pname};
+
+	    if (!ref($prop)) {
+		$content .= "$pname=$prop\n";
+	    } elsif (ref($prop) eq 'ARRAY') {
+		foreach my $val (@$prop) {
+		    $content .= "$pname=$val\n";
+		}
+	    } else {
+		die "invalid property '$pname'\n";
+	    }
+	}
+	$content .= "\n";
+    }
+
+    file_set_contents($filename, $content);
+};
+
+sub get_schedules {
+    my ($param) = @_;
+
+    my $result = [];
+
+    my $systemd_dir = '/etc/systemd/system';
+
+    dir_glob_foreach($systemd_dir, '^pmg-pbsbackup@.+\.timer$', sub {
+	my ($filename) = @_;
+	my $remote;
+	if ($filename =~ /^pmg-pbsbackup\@(.+)\.timer$/) {
+	    $remote = PVE::Systemd::unescape_unit($1);
+	} else {
+	    die 'Unrecognized timer name!\n';
+	}
+
+	my $unitfile = "$systemd_dir/$filename";
+	my $unit = $read_ini->($unitfile);
+
+	push @$result, {
+	    unitfile => $unitfile,
+	    remote => $remote,
+	    schedule => $unit->{'Timer'}->{'OnCalendar'},
+	    delay => $unit->{'Timer'}->{'RandomizedDelaySec'},
+	};
+    });
+
+    return $result;
+
+}
+
+sub create_schedule {
+    my ($remote, $schedule, $delay) = @_;
+
+    my $unit_name = 'pmg-pbsbackup@' . PVE::Systemd::escape_unit($remote);
+    #my $service_unit = $unit_name . '.service';
+    my $timer_unit = $unit_name . '.timer';
+    my $timer_unit_path = "/etc/systemd/system/$timer_unit";
+
+    # create systemd timer
+    run_command(['systemd-analyze', 'calendar', $schedule], errmsg => "Invalid schedule specification", outfunc => sub {});
+    run_command(['systemd-analyze', 'timespan', $delay], errmsg => "Invalid delay specification", outfunc => sub {});
+    my $timer = {
+	'Unit' => {
+	    'Description' => "Timer for PBS Backup to remote $remote",
+	},
+	'Timer' => {
+	    'OnCalendar' => $schedule,
+	    'RandomizedDelaySec' => $delay,
+	},
+	'Install' => {
+	    'WantedBy' => 'timers.target',
+	},
+    };
+
+    eval {
+	$write_ini->($timer, $timer_unit_path);
+	run_command(['systemctl', 'daemon-reload']);
+	run_command(['systemctl', 'enable', $timer_unit]);
+	run_command(['systemctl', 'start', $timer_unit]);
+
+    };
+    if (my $err = $@) {
+	die "Creating backup schedule for $remote failed: $err\n";
+    }
+
+    return;
+}
+
+sub delete_schedule {
+    my ($remote) = @_;
+
+    my $schedules = get_schedules();
+
+    die "Schedule for $remote not found!\n" if !grep {$_->{remote} eq $remote} @$schedules;
+
+    my $unit_name = 'pmg-pbsbackup@' . PVE::Systemd::escape_unit($remote);
+    my $service_unit = $unit_name . '.service';
+    my $timer_unit = $unit_name . '.timer';
+    my $timer_unit_path = "/etc/systemd/system/$timer_unit";
+
+    eval {
+	run_command(['systemctl', 'disable', $timer_unit]);
+	unlink($timer_unit_path) || die "delete '$timer_unit_path' failed - $!\n";
+	run_command(['systemctl', 'daemon-reload']);
+
+    };
+    if (my $err = $@) {
+	die "Removing backup schedule for $remote failed: $err\n";
+    }
+
+    return;
+}
+
 1;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 09/12] add initial SectionConfig for pbs

[Stoiko Ivanov]

add a SectionConfig definition to hold information about PBS-remotes used
for backing up PMG.

Mostly adapted from the PBSPlugin.pm in pve-storage.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 debian/dirs          |   1 +
 src/Makefile         |   1 +
 src/PMG/PBSConfig.pm | 168 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 170 insertions(+)
 create mode 100644 src/PMG/PBSConfig.pm

diff --git a/debian/dirs b/debian/dirs
index f7ac2e7..f138bb4 100644
--- a/debian/dirs
+++ b/debian/dirs
@@ -1,4 +1,5 @@
 /etc/pmg
 /etc/pmg/dkim
+/etc/pmg/pbs
 /var/lib/pmg
 /var/lib/pmg/backup
diff --git a/src/Makefile b/src/Makefile
index a460048..001cb57 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -67,6 +67,7 @@ LIBSOURCES =				\
 	PMG/Unpack.pm			\
 	PMG/Backup.pm			\
 	PMG/PBSTools.pm			\
+	PMG/PBSConfig.pm		\
 	PMG/RuleCache.pm		\
 	PMG/Statistic.pm		\
 	PMG/UserConfig.pm		\
diff --git a/src/PMG/PBSConfig.pm b/src/PMG/PBSConfig.pm
new file mode 100644
index 0000000..d290161
--- /dev/null
+++ b/src/PMG/PBSConfig.pm
@@ -0,0 +1,168 @@
+package PMG::PBSConfig;
+
+# section config implementation for PBS integration in PMG
+
+use strict;
+use warnings;
+
+use PVE::Tools qw(extract_param);
+use PVE::SectionConfig;
+use PVE::JSONSchema qw(get_standard_option);
+use PMG::PBSTools;
+
+use base qw(PVE::SectionConfig);
+
+my $inotify_file_id = 'pmg-pbs.conf';
+my $secret_dir = '/etc/pmg/pbs';
+my $config_filename = "${secret_dir}/pbs.conf";
+
+my $get_secret_dir = sub {
+    return $secret_dir;
+};
+
+my $defaultData = {
+    propertyList => {
+	type => { description => "Section type." },
+	remote => {
+	    description => "Proxmox Backup Server ID.",
+	    type => 'string', format => 'pve-configid',
+	},
+    },
+};
+
+sub properties {
+    return {
+	datastore => {
+	    description => "Proxmox backup server datastore name.",
+	    type => 'string',
+	},
+	server => {
+	    description => "Proxmox backup server address.",
+	    type => 'string', format => 'address',
+	    maxLength => 256,
+	},
+	disable => {
+	    description => "Flag to disable/deactivate the entry.",
+	    type => 'boolean',
+	    optional => 1,
+	},
+	password => {
+	    description => "Password for the user on the Proxmox backup server.",
+	    type => 'string',
+	    optional => 1,
+	},
+	username => get_standard_option('pmg-email-address', {
+	    description => "Username on the Proxmox backup server"
+	}),
+	# openssl s_client -connect <host>:8007 2>&1 |openssl x509 -fingerprint -sha256
+	fingerprint => get_standard_option('fingerprint-sha256'),
+	'encryption-key' => {
+	    description => "Encryption key. Use 'autogen' to generate one automatically without passphrase.",
+	    type => 'string',
+	    optional => 1,
+	},
+    };
+}
+
+sub options {
+    return {
+	server => { fixed => 1 },
+	datastore => { fixed => 1 },
+	disable => { optional => 1},
+	username => { optional => 1 },
+	password => { optional => 1 },
+	'encryption-key' => { optional => 1 },
+	fingerprint => { optional => 1 },
+    };
+}
+
+sub type {
+    return 'pbs';
+}
+
+sub private {
+    return $defaultData;
+}
+
+sub parse_config {
+    my ($class, $filename, $raw) = @_;
+
+    my $cfg = $class->SUPER::parse_config($filename, $raw);
+
+    PMG::PBSTools::set_secret_dir($secret_dir);
+
+    return $cfg;
+}
+
+sub write_config {
+    my ($class, $filename, $cfg) = @_;
+
+    foreach my $pbs (keys %{$cfg->{ids}}) {
+	my $data = $cfg->{ids}->{$pbs};
+
+	my $password = extract_param($data, 'password');
+	PMG::PBSTools::pbs_set_password($data, $pbs, $password) if defined($password);
+
+	my $encryption_key = extract_param($data, 'encryption-key');
+	PMG::PBSTools::pbs_set_encryption_key($data, $pbs, $encryption_key) if defined($encryption_key);
+    }
+
+    $class->SUPER::write_config($filename, $cfg);
+}
+
+sub new {
+    my ($type) = @_;
+
+    my $class = ref($type) || $type;
+
+    my $cfg = PVE::INotify::read_file($inotify_file_id);
+
+    return bless $cfg, $class;
+}
+
+sub write {
+    my ($self) = @_;
+
+    PVE::INotify::write_file($inotify_file_id, $self);
+}
+
+my $lockfile = "/var/lock/pmgpbsconfig.lck";
+
+sub lock_config {
+    my ($code, $errmsg) = @_;
+
+    my $p = PVE::Tools::lock_file($lockfile, undef, $code);
+    if (my $err = $@) {
+	$errmsg ? die "$errmsg: $err" : die $err;
+    }
+}
+
+
+__PACKAGE__->register();
+__PACKAGE__->init();
+
+sub read_pmg_pbs_conf {
+    my ($filename, $fh) = @_;
+
+    local $/ = undef; # slurp mode
+
+    my $raw = defined($fh) ? <$fh> : '';
+
+    return __PACKAGE__->parse_config($filename, $raw);
+}
+
+sub write_pmg_pbs_conf {
+    my ($filename, $fh, $cfg) = @_;
+
+    my $raw = __PACKAGE__->write_config($filename, $cfg);
+
+    PVE::Tools::safe_print($filename, $fh, $raw);
+}
+
+PVE::INotify::register_file($inotify_file_id, $config_filename,
+			    \&read_pmg_pbs_conf,
+			    \&write_pmg_pbs_conf,
+			    undef,
+			    always_call_parser => 1);
+
+1;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 10/12] Add API2 module for PBS configuration

[Stoiko Ivanov]

The module provides the API methods for creating/updating/listing/deleting
PBS remotes

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile               |   1 +
 src/PMG/API2/Config.pm     |   7 ++
 src/PMG/API2/PBS/Remote.pm | 218 +++++++++++++++++++++++++++++++++++++
 3 files changed, 226 insertions(+)
 create mode 100644 src/PMG/API2/PBS/Remote.pm

diff --git a/src/Makefile b/src/Makefile
index 001cb57..e1546a8 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -138,6 +138,7 @@ LIBSOURCES =				\
 	PMG/API2/Statistics.pm		\
 	PMG/API2/MailTracker.pm		\
 	PMG/API2/Backup.pm		\
+	PMG/API2/PBS/Remote.pm		\
 	PMG/API2/Nodes.pm		\
 	PMG/API2/Postfix.pm		\
 	PMG/API2/Quarantine.pm		\
diff --git a/src/PMG/API2/Config.pm b/src/PMG/API2/Config.pm
index d4a9679..e11eb3f 100644
--- a/src/PMG/API2/Config.pm
+++ b/src/PMG/API2/Config.pm
@@ -25,6 +25,7 @@ use PMG::API2::Fetchmail;
 use PMG::API2::DestinationTLSPolicy;
 use PMG::API2::DKIMSign;
 use PMG::API2::SACustom;
+use PMG::API2::PBS::Remote;
 
 use base qw(PVE::RESTHandler);
 
@@ -93,6 +94,11 @@ __PACKAGE__->register_method({
     path => 'customscores',
 });
 
+__PACKAGE__->register_method ({
+    subclass => "PMG::API2::PBS::Remote",
+    path => 'pbs',
+});
+
 __PACKAGE__->register_method ({
     name => 'index', 
     path => '',
@@ -131,6 +137,7 @@ __PACKAGE__->register_method ({
 	push @$res, { section => 'regextest' };
 	push @$res, { section => 'tlspolicy' };
 	push @$res, { section => 'dkim' };
+	push @$res, { section => 'pbs' };
 
 	return $res;
     }});
diff --git a/src/PMG/API2/PBS/Remote.pm b/src/PMG/API2/PBS/Remote.pm
new file mode 100644
index 0000000..fcc7362
--- /dev/null
+++ b/src/PMG/API2/PBS/Remote.pm
@@ -0,0 +1,218 @@
+package PMG::API2::PBS::Remote;
+
+use strict;
+use warnings;
+
+use PVE::SafeSyslog;
+use PVE::Tools qw(extract_param);
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+
+use PMG::PBSTools;
+use PMG::PBSConfig;
+
+use base qw(PVE::RESTHandler);
+
+__PACKAGE__->register_method ({
+    name => 'list',
+    path => '',
+    method => 'GET',
+    description => "List all configured Proxmox Backup Server instances.",
+    permissions => { check => [ 'admin', 'audit' ] },
+    proxyto => 'master',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {}
+    },
+    returns => {
+        type => "array",
+        items => PMG::PBSConfig->createSchema(1),
+        links => [ { rel => 'child', href => "{remote}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $res = [];
+
+	my $conf = PMG::PBSConfig->new();
+
+	if (defined($conf)) {
+	    foreach my $remote (keys %{$conf->{ids}}) {
+		my $d = $conf->{ids}->{$remote};
+		my $entry = {
+		    remote => $remote,
+		    server => $d->{server},
+		    datastore => $d->{datastore},
+		    username => $d->{username},
+		};
+		push @$res, $entry;
+	    }
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'create',
+    path => '',
+    method => 'POST',
+    description => "Add Proxmox Backup Server instance.",
+    permissions => { check => [ 'admin' ] },
+    proxyto => 'master',
+    protected => 1,
+    parameters => PMG::PBSConfig->createSchema(1),
+    returns => { type => 'null' } ,
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    $conf->{ids} //= {};
+	    my $ids = $conf->{ids};
+
+	    my $remote = extract_param($param, 'remote');
+	    die "PBS remote '$remote' already exists\n"
+		if $ids->{$remote};
+
+	    my $remotecfg = PMG::PBSConfig->check_config($remote, $param, 1);
+
+	    $ids->{$remote} = $remotecfg;
+
+
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "add PBS remote failed");
+
+	return undef;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'read_config',
+    path => '{remote}',
+    method => 'GET',
+    description => "Get PBS remote configuration.",
+    proxyto => 'master',
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 1,
+	properties => {
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {},
+    code => sub {
+	my ($param) = @_;
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote = $param->{remote};
+
+	my $data = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$data;
+
+	delete $data->{password};
+	delete $data->{type};
+
+	$data->{digest} = $conf->{digest};
+
+	return $data;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'update_config',
+    path => '{remote}',
+    method => 'PUT',
+    description => "Update PBS remote settings.",
+    permissions => { check => [ 'admin' ] },
+    protected => 1,
+    proxyto => 'master',
+    parameters => PMG::PBSConfig->updateSchema(),
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    my $ids = $conf->{ids};
+
+	    my $digest = extract_param($param, 'digest');
+	    PVE::SectionConfig::assert_if_modified($conf, $digest);
+
+	    my $remote = extract_param($param, 'remote');
+
+	    die "PBS remote '$remote' does not exist\n"
+		if !$ids->{$remote};
+
+	    my $delete_str = extract_param($param, 'delete');
+	    die "no options specified\n"
+		if !$delete_str && !scalar(keys %$param);
+
+	    foreach my $opt (PVE::Tools::split_list($delete_str)) {
+		delete $ids->{$remote}->{$opt};
+	    }
+
+	    my $remoteconfig = PMG::PBSConfig->check_config($remote, $param, 0, 1);
+
+	    foreach my $p (keys %$remoteconfig) {
+		$ids->{$remote}->{$p} = $remoteconfig->{$p};
+	    }
+
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "update PBS remote failed");
+
+	return undef;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'delete',
+    path => '{remote}',
+    method => 'DELETE',
+    description => "Delete an PBS remote",
+    permissions => { check => [ 'admin' ] },
+    protected => 1,
+    proxyto => 'master',
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    remote => {
+		description => "Profile ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	}
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $code = sub {
+
+	    my $conf = PMG::PBSConfig->new();
+	    my $ids = $conf->{ids};
+
+	    my $remote = $param->{remote};
+
+	    die "PBS remote '$remote' does not exist\n"
+		if !$ids->{$remote};
+
+	    PMG::PBSTools::pbs_delete_password($ids->{$remote}, $remote);
+	    PMG::PBSTools::pbs_delete_encryption_key($ids->{$remote}, $remote);
+	    delete $ids->{$remote};
+
+	    $conf->write();
+	};
+
+	PMG::PBSConfig::lock_config($code, "delete PBS remote failed");
+
+	return undef;
+    }});
+
+1;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 11/12] Add API2 module for per-node backups to PBS

[Stoiko Ivanov]

The module adds API2 methods for:

* creating/restoring/listing/forgetting backups to a configured PBS remote
* creating backup schedules (using systemd-timers)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/Makefile            |   1 +
 src/PMG/API2/Nodes.pm   |   7 +
 src/PMG/API2/PBS/Job.pm | 467 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 475 insertions(+)
 create mode 100644 src/PMG/API2/PBS/Job.pm

diff --git a/src/Makefile b/src/Makefile
index e1546a8..b978cfa 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -138,6 +138,7 @@ LIBSOURCES =				\
 	PMG/API2/Statistics.pm		\
 	PMG/API2/MailTracker.pm		\
 	PMG/API2/Backup.pm		\
+	PMG/API2/PBS/Job.pm		\
 	PMG/API2/PBS/Remote.pm		\
 	PMG/API2/Nodes.pm		\
 	PMG/API2/Postfix.pm		\
diff --git a/src/PMG/API2/Nodes.pm b/src/PMG/API2/Nodes.pm
index 96aa146..259f8f3 100644
--- a/src/PMG/API2/Nodes.pm
+++ b/src/PMG/API2/Nodes.pm
@@ -26,6 +26,7 @@ use PMG::API2::SpamAssassin;
 use PMG::API2::Postfix;
 use PMG::API2::MailTracker;
 use PMG::API2::Backup;
+use PMG::API2::PBS::Job;
 
 use base qw(PVE::RESTHandler);
 
@@ -79,6 +80,11 @@ __PACKAGE__->register_method ({
     path => 'backup',
 });
 
+__PACKAGE__->register_method ({
+    subclass => "PMG::API2::PBS::Job",
+    path => 'pbs',
+});
+
 __PACKAGE__->register_method ({
     name => 'index',
     path => '',
@@ -105,6 +111,7 @@ __PACKAGE__->register_method ({
 	my $result = [
 	    { name => 'apt' },
 	    { name => 'backup' },
+	    { name => 'pbs' },
 	    { name => 'clamav' },
 	    { name => 'spamassassin' },
 	    { name => 'postfix' },
diff --git a/src/PMG/API2/PBS/Job.pm b/src/PMG/API2/PBS/Job.pm
new file mode 100644
index 0000000..8fa3a19
--- /dev/null
+++ b/src/PMG/API2/PBS/Job.pm
@@ -0,0 +1,467 @@
+package PMG::API2::PBS::Job;
+
+use strict;
+use warnings;
+
+use POSIX qw(strftime);
+
+use PVE::JSONSchema qw(get_standard_option);
+use PVE::RESTHandler;
+use PVE::SafeSyslog;
+use PVE::Tools qw(extract_param);
+
+use PMG::RESTEnvironment;
+use PMG::Backup;
+use PMG::PBSTools;
+use PMG::PBSConfig;
+
+use base qw(PVE::RESTHandler);
+
+__PACKAGE__->register_method ({
+    name => 'list',
+    path => '',
+    method => 'GET',
+    description => "List all configured Proxmox Backup Server jobs.",
+    permissions => { check => [ 'admin', 'audit' ] },
+    proxyto => 'node',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	},
+    },
+    returns => {
+        type => "array",
+        items => PMG::PBSConfig->createSchema(1),
+        links => [ { rel => 'child', href => "{remote}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $res = [];
+
+	my $conf = PMG::PBSConfig->new();
+	if (defined($conf)) {
+	    foreach my $remote (keys %{$conf->{ids}}) {
+		my $d = $conf->{ids}->{$remote};
+		my $entry = {
+		    remote => $remote,
+		    server => $d->{server},
+		    datastore => $d->{datastore},
+		};
+		push @$res, $entry;
+	    }
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'remote_index',
+    path => '{remote}',
+    method => 'GET',
+    description => "Backup Job index.",
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {
+	type => 'array',
+	items => {
+	    type => "object",
+	    properties => { section => { type => 'string'} },
+	},
+	links => [ { rel => 'child', href => "{section}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $result = [
+	    { section => 'snapshots' },
+	    { section => 'backup' },
+	    { section => 'restore' },
+	    { section => 'timer' },
+	];
+	return $result;
+}});
+
+__PACKAGE__->register_method ({
+    name => 'get_snapshots',
+    path => '{remote}/snapshots',
+    method => 'GET',
+    description => "Get snapshots stored on remote.",
+    proxyto => 'node',
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => {
+	type => 'array',
+	items => {
+	    type => "object",
+	    properties => {
+		time => { type => 'string'},
+		ctime => { type => 'string'},
+		size => { type => 'integer'},
+	    },
+	},
+	links => [ { rel => 'child', href => "{time}" } ],
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	my $snapshots = PMG::PBSTools::get_snapshots($remote_config, $remote);
+	my $res = [];
+	foreach my $item (@$snapshots) {
+	    my $btype = $item->{"backup-type"};
+	    my $bid = $item->{"backup-id"};
+	    my $epoch = $item->{"backup-time"};
+	    my $size = $item->{size} // 1;
+
+	    next if !($btype eq 'host');
+	    next if !($bid eq $node);
+
+	    my $time = strftime("%FT%TZ", gmtime($epoch));
+
+	    my $info = {
+		time => $time,
+		ctime => $epoch,
+		size => $size,
+	    };
+
+	    push @$res, $info;
+	}
+
+	return $res;
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'forget_snapshot',
+    path => '{remote}/snapshots/{time}',
+    method => 'DELETE',
+    description => "Forget a snapshot",
+    proxyto => 'node',
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    time => {
+		description => "Backup time in RFC 3399 format",
+		type => 'string',
+	    },
+	},
+    },
+    returns => {type => 'string' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+	my $time = $param->{time};
+
+	my $snapshot = "host/$node/$time";
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $rpcenv = PMG::RESTEnvironment->get();
+	my $authuser = $rpcenv->get_user();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	my $worker = sub {
+	    eval {
+		PMG::PBSTools::forget_snapshot($remote_config, $remote, $snapshot);
+	    };
+	    die "Forgetting backup failed: $@" if $@;
+
+	    return;
+	};
+
+	return $rpcenv->fork_worker('pbs_forget', undef, $authuser, $worker);
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'run_backup',
+    path => '{remote}/backup',
+    method => 'POST',
+    description => "run backup",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => "string" },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PMG::RESTEnvironment->get();
+	my $authuser = $rpcenv->get_user();
+
+	my $remote = $param->{remote};
+	my $node = $param->{node};
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	my $backup_dir = "/var/lib/pmg/backup";
+	my $currentdir = "${backup_dir}/current";
+
+	my $worker = sub {
+	    my $upid = shift;
+
+	    print "starting update of current backup state\n";
+
+	    PMG::Backup::pmg_backup($currentdir, $param->{statistic});
+	    my $pbs_opts = {
+		type => 'host',
+		id => $node,
+		pxarname => 'pmgbackup',
+		root => $currentdir,
+	    };
+
+	    PMG::PBSTools::backup_tree($remote_config, $remote, $pbs_opts);
+
+	    print "backup finished\n";
+
+	    return;
+	};
+
+	return $rpcenv->fork_worker('pbs_backup', undef, $authuser, $worker);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'restore',
+    path => '{remote}/restore',
+    method => 'POST',
+    description => "Restore the system configuration.",
+    permissions => { check => [ 'admin' ] },
+    proxyto => 'node',
+    protected => 1,
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    PMG::Backup::get_restore_options(),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    snapshot => {description=> "Backup-time to restore",
+		optional => 1, type => 'string'
+	    },
+	    althost => {description => "hostname of backup snapshot",
+		optional => 1, type => 'string'
+	    },
+	},
+    },
+    returns => { type => "string" },
+    code => sub {
+	my ($param) = @_;
+
+	my $rpcenv = PMG::RESTEnvironment->get();
+	my $authuser = $rpcenv->get_user();
+
+	my $remote = $param->{remote};
+	my $node = $param->{althost} // $param->{node};
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	my $time = time;
+	my $dirname = "/tmp/proxrestore_$$.$time";
+
+	$param->{database} //= 1;
+
+	die "nothing selected - please select what you want to restore (config or database?)\n"
+	    if !($param->{database} || $param->{config});
+
+	my $pbs_opts = {
+	    pxarname => 'pmgbackup',
+	    target => $dirname,
+	};
+
+	$pbs_opts->{snapshot} = $param->{snapshot} // "host/$node";
+
+	my $worker = sub {
+	    my $upid = shift;
+
+	    print "starting restore of $pbs_opts->{snapshot} from $remote\n";
+
+	    PMG::PBSTools::restore_pxar($remote_config, $remote, $pbs_opts);
+	    print "starting restore of PMG config\n";
+	    PMG::Backup::pmg_restore($dirname, $param->{database},
+		 $param->{config}, $param->{statistic});
+	    print "restore finished\n";
+
+	    return;
+	};
+
+	return $rpcenv->fork_worker('pbs_restore', undef, $authuser, $worker);
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'create_timer',
+    path => '{remote}/timer',
+    method => 'POST',
+    description => "Create backup schedule",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    schedule => {
+		description => "Schedule for the backup (OnCalendar setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z*.:,\-/ ]+',
+		default => 'daily', optional => 1,
+	    },
+	    delay => {
+		description => "Randomized delay to add to the starttime (RandomizedDelaySec setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z. ]+',
+		default => 'daily', optional => 1,
+	    },
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+	my $schedule = $param->{schedule} // 'daily';
+	my $delay = $param->{delay} // '5min';
+
+	my $conf = PMG::PBSConfig->new();
+
+	my $remote_config = $conf->{ids}->{$remote};
+	die "PBS remote '$remote' does not exist\n" if !$remote_config;
+
+	PMG::PBSTools::create_schedule($remote, $schedule, $delay);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'delete_timer',
+    path => '{remote}/timer',
+    method => 'DELETE',
+    description => "Delete backup schedule",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => 'null' },
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+
+	PMG::PBSTools::delete_schedule($remote);
+
+    }});
+
+__PACKAGE__->register_method ({
+    name => 'list_timer',
+    path => '{remote}/timer',
+    method => 'GET',
+    description => "Get timer specification",
+    proxyto => 'node',
+    protected => 1,
+    permissions => { check => [ 'admin', 'audit' ] },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	},
+    },
+    returns => { type => 'object', properties => {
+	    remote => {
+		description => "Proxmox Backup Server ID.",
+		type => 'string', format => 'pve-configid',
+	    },
+	    schedule => {
+		description => "Schedule for the backup (OnCalendar setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z*.:,\-/ ]+',
+		default => 'daily', optional => 1,
+	    },
+	    delay => {
+		description => "Randomized delay to add to the starttime (RandomizedDelaySec setting of the systemd.timer)",
+		type => 'string', pattern => '[0-9a-zA-Z. ]+',
+		default => 'daily', optional => 1,
+	    },
+	    unitfile => {
+		description => "unit file for the systemd.timer unit",
+		type => 'string',
+	    },
+	}},
+    code => sub {
+	my ($param) = @_;
+
+	my $remote = $param->{remote};
+
+	my $schedules = PMG::PBSTools::get_schedules();
+
+	my @data = grep {$_->{remote} eq $remote} @$schedules;
+
+	die "Schedule for $remote not found!\n" if (scalar(@data != 1));
+	my $res = $data[0];
+
+	return $res
+    }});
+1;
-- 
2.20.1

[pmg-devel] [RFC pmg-api 12/12] pbs-integration: add CLI calls to pmgbackup

[Stoiko Ivanov]

This patch adds to new categories for commands to pmgbackup:
* pmgbackup remote - for managing PBS instances' configuration, cluster-wide
* pmgbackup pbsjob - for managing backups, restores and schedules to those
  remotes, per node

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
---
 src/PMG/CLI/pmgbackup.pm | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/src/PMG/CLI/pmgbackup.pm b/src/PMG/CLI/pmgbackup.pm
index 69224e5..06bf997 100644
--- a/src/PMG/CLI/pmgbackup.pm
+++ b/src/PMG/CLI/pmgbackup.pm
@@ -8,9 +8,12 @@ use PVE::Tools;
 use PVE::SafeSyslog;
 use PVE::INotify;
 use PVE::CLIHandler;
+use PVE::CLIFormatter;
 
 use PMG::RESTEnvironment;
 use PMG::API2::Backup;
+use PMG::API2::PBS::Remote;
+use PMG::API2::PBS::Job;
 
 use base qw(PVE::CLIHandler);
 
@@ -32,6 +35,30 @@ our $cmddef = {
     backup => [ 'PMG::API2::Backup', 'backup', undef, { node => $nodename } ],
     restore => [ 'PMG::API2::Backup', 'restore', undef, { node => $nodename } ],
     list => [ 'PMG::API2::Backup', 'list', undef, { node => $nodename }, $format_backup_list ],
+    remote => {
+	list => ['PMG::API2::PBS::Remote', 'list', undef, undef,  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['remote', 'server', 'datastore', 'username' ], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
+	add => ['PMG::API2::PBS::Remote', 'create', ['remote'] ],
+	remove => ['PMG::API2::PBS::Remote', 'delete', ['remote'] ],
+	set => ['PMG::API2::PBS::Remote', 'update_config', ['remote'] ],
+    },
+    pbsjob => {
+	list_backups => ['PMG::API2::PBS::Job', 'get_snapshots', ['remote'] , { node => $nodename },  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['time', 'size'], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
+	forget => ['PMG::API2::PBS::Job', 'forget_snapshot', ['remote', 'time'], { node => $nodename} ],
+	run => ['PMG::API2::PBS::Job', 'run_backup', ['remote'], { node => $nodename} ],
+	restore => ['PMG::API2::PBS::Job', 'restore', ['remote'], { node => $nodename} ],
+	create => ['PMG::API2::PBS::Job', 'create_timer', ['remote'], { node => $nodename }],
+	delete => ['PMG::API2::PBS::Job', 'delete_timer', ['remote'], { node => $nodename }],
+	schedule => ['PMG::API2::PBS::Job', 'list_timer', ['remote'], { node => $nodename },  sub {
+	    my ($data, $schema, $options) = @_;
+	    PVE::CLIFormatter::print_api_result($data, $schema, ['remote', 'schedule', 'delay'], $options);
+	}, $PVE::RESTHandler::standard_output_options ],
+    },
 };
 
 1;
-- 
2.20.1

Re: [pmg-devel] [RFC pmg-api 02/12] Backup: split backup creation and creating tar

[Dietmar Maurer]

>  
>  sub pmg_backup {
> -    my ($filename, $include_statistics) = @_;
> +    my ($backupdir, $include_statistics) = @_;
> +
>  
>      my $time = time;
> -    my $dirname = "/tmp/proxbackup_$$.$time";
> +    my $dirname = $backupdir // '/var/lib/pmg/backup/current';

Why is $backupdir optional? And why do you use different paths (/tmp vs. /var/lib)?