public inbox for pmg-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Stoiko Ivanov <s.ivanov@proxmox.com>
To: pmg-devel@lists.proxmox.com
Subject: [pmg-devel] [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration
Date: Mon, 16 Nov 2020 12:01:07 +0100	[thread overview]
Message-ID: <20201116110118.7483-1-s.ivanov@proxmox.com> (raw)

changes v2->v3:
* rebased the gui-patches to latest master
* added param_mapping from pve-storage/pvesm (provide an interactive password
  entry)

cover-letter for v2:
changes v1->v2:
* renamed PBSTools to PBSClient, and made it a class (handling the config and
  backup-operations of one PBS instance)
* dropped encryption-key support from the GUI, the API calls and the
  SectionConfig - will be submitted again, when we have a clean solution in PVE
* dropped explicit prune support - each group now gets pruned directly after a
  backup
* one small fixup in debian/control (I mistakenly added two ',' in the
  proxmox-backup-client dependency)
* added /etc/pmg/pbs to the cluster-sync

cover-letter for v1:
changes RFC->v1:
* moved the potentially reusable parts to pve-common (PBSTools.pm, and 2
  functions in Systemd.pm)
* added GUI support (mostly adapted from the LDAPConfig) - huge thanks to
  Dominik for his patience and help!
* added support for encrypted backups
* added support for pruning backups
* added a systemd-timer, which runs a daily prune based on the configured
  settings

differences from the PBS storage-plugin config in PVE:
* the keep-options are kept as separate options instead of using a property
  string for all (like they are in the datastore config in PBS)


cover-letter from the RFC:
This series is a minimal proof-of-concept for integrating PBS into PMG.

The code needs quite a bit of cleanup, and more testing, however I'll send it
as RFC, for an initial sanity-check - to see if I missed some needed
functionality.

It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd
unit manipulation from the PVE::API2::Disks::Directory) and I'd like to
refactor the common parts into pve-common.

The backup consists of the same files, that end up in a locally generated
PMG Backup (instead of creating a tar.gz we send the data to a PBS instance).

What works:
* creating and manipulating 'remotes' (PBS instances where we back up to)
* creating, restoring and forgetting backups to those remotes
* restoring a backup from another PMG installation from a configured remote
* creating schedules for backups (systemd-timer units) for periodic backups
via CLI (pmgbackup, pmgsh)

What's missing:
* quite a bit of refactoring (e.g. move the systemd-timer helpers and most of
  PBSTools to pve-common (and reuse it in the PBS Storage plugin)
  * encryption support
  * pruneing support
  * testing
  * GUI
  * Documentation

  The first patch is a small cosmetic cleanup, and independent of the series.

  A sample session (which I used for testing:
  ```
  pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint  ff:ff:...
  pmgbackup remote list
  pmgbackup pbsjob run pbs1
  pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1
  pmgbackup pbsjob run pbs1
  pmgbackup pbsjob list_backups pbs1
  pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s'
  systemctl list-timers --all
  ```

pve-common:
Stoiko Ivanov (1):
  add PBSClient module

 src/Makefile         |   1 +
 src/PVE/PBSClient.pm | 305 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 306 insertions(+)
 create mode 100644 src/PVE/PBSClient.pm

pmg-api:
Stoiko Ivanov (7):
  debian: drop duplicate ',' in dependencies
  add initial SectionConfig for PBS
  Add API2 module for PBS configuration
  Add API2 module for per-node backups to PBS
  pbs-integration: add CLI calls to pmgbackup
  add scheduled backup to PBS remotes
  add /etc/pmg/pbs to cluster-sync

 debian/control                |   2 +-
 debian/dirs                   |   1 +
 debian/pmg-pbsbackup@.service |   6 +
 debian/rules                  |   1 +
 src/Makefile                  |   6 +-
 src/PMG/API2/Config.pm        |   7 +
 src/PMG/API2/Nodes.pm         |   7 +
 src/PMG/API2/PBS/Job.pm       | 501 ++++++++++++++++++++++++++++++++++
 src/PMG/API2/PBS/Remote.pm    | 231 ++++++++++++++++
 src/PMG/CLI/pmgbackup.pm      |  60 ++++
 src/PMG/Cluster.pm            |   1 +
 src/PMG/PBSConfig.pm          | 195 +++++++++++++
 src/PMG/PBSSchedule.pm        | 104 +++++++
 13 files changed, 1120 insertions(+), 2 deletions(-)
 create mode 100644 debian/pmg-pbsbackup@.service
 create mode 100644 src/PMG/API2/PBS/Job.pm
 create mode 100644 src/PMG/API2/PBS/Remote.pm
 create mode 100644 src/PMG/PBSConfig.pm
 create mode 100644 src/PMG/PBSSchedule.pm

pmg-gui:
Stoiko Ivanov (3):
  Make Backup/Restore panel a menuentry
  refactor RestoreWindow for PBS
  add PBSConfig tab to Backup menu

 js/BackupConfiguration.js |  23 ++
 js/BackupRestore.js       |  67 ++--
 js/Makefile               |   2 +
 js/NavigationTree.js      |   6 +
 js/PBSConfig.js           | 678 ++++++++++++++++++++++++++++++++++++++
 js/SystemConfiguration.js |   4 -
 6 files changed, 748 insertions(+), 32 deletions(-)
 create mode 100644 js/BackupConfiguration.js
 create mode 100644 js/PBSConfig.js

-- 
2.20.1





             reply	other threads:[~2020-11-16 11:01 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-16 11:01 Stoiko Ivanov [this message]
2020-11-16 11:01 ` [pmg-devel] [PATCH pve-common v3 1/1] add PBSClient module Stoiko Ivanov
2020-11-17  8:49   ` [pmg-devel] applied: " Thomas Lamprecht
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 1/7] debian: drop duplicate ', ' in dependencies Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 2/7] add initial SectionConfig for PBS Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 3/7] Add API2 module for PBS configuration Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 4/7] Add API2 module for per-node backups to PBS Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 5/7] pbs-integration: add CLI calls to pmgbackup Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 6/7] add scheduled backup to PBS remotes Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-api v3 7/7] add /etc/pmg/pbs to cluster-sync Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-gui v3 1/3] Make Backup/Restore panel a menuentry Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-gui v3 2/3] refactor RestoreWindow for PBS Stoiko Ivanov
2020-11-16 11:01 ` [pmg-devel] [PATCH pmg-gui v3 3/3] add PBSConfig tab to Backup menu Stoiko Ivanov
2020-11-17 17:22 ` [pmg-devel] applied-series: [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201116110118.7483-1-s.ivanov@proxmox.com \
    --to=s.ivanov@proxmox.com \
    --cc=pmg-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal