From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <s.ivanov@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4584E63CAF for <pmg-devel@pve.proxmox.com>; Wed, 28 Oct 2020 19:54:50 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0625A22A24 for <pmg-devel@pve.proxmox.com>; Wed, 28 Oct 2020 19:54:50 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 9822B229CA for <pmg-devel@pve.proxmox.com>; Wed, 28 Oct 2020 19:54:47 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 4027345F7E for <pmg-devel@pve.proxmox.com>; Wed, 28 Oct 2020 19:54:47 +0100 (CET) From: Stoiko Ivanov <s.ivanov@proxmox.com> To: pmg-devel@pve.proxmox.com Date: Wed, 28 Oct 2020 19:54:16 +0100 Message-Id: <20201028185432.23067-1-s.ivanov@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.289 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [pbstools.pm, pmgbackup.pm, remote.pm, pbsschedule.pm, job.pm, config.pm, pbsconfig.pm, systemd.pm, nodes.pm, backup.pm] URIBL_SBL 0.644 Contains an URL's NS IP listed in the Spamhaus SBL blocklist [backup.pm] URIBL_SBL_A 0.1 Contains URL's A record listed in the Spamhaus SBL blocklist [backup.pm] Subject: [pmg-devel] [PATCH pve-common/api/gui] add initial PBS integration X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion <pmg-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pmg-devel/> List-Post: <mailto:pmg-devel@lists.proxmox.com> List-Help: <mailto:pmg-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Wed, 28 Oct 2020 18:54:50 -0000 changes RFC->v1: * moved the potentially reusable parts to pve-common (PBSTools.pm, and 2 functions in Systemd.pm) * added GUI support (mostly adapted from the LDAPConfig) - huge thanks to Dominik for his patience and help! * added support for encrypted backups * added support for pruning backups * added a systemd-timer, which runs a daily prune based on the configured settings differences from the PBS storage-plugin config in PVE: * the keep-options are kept as separate options instead of using a property string for all (like they are in the datastore config in PBS) cover-letter from the RFC: This series is a minimal proof-of-concept for integrating PBS into PMG. The code needs quite a bit of cleanup, and more testing, however I'll send it as RFC, for an initial sanity-check - to see if I missed some needed functionality. It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd unit manipulation from the PVE::API2::Disks::Directory) and I'd like to refactor the common parts into pve-common. The backup consists of the same files, that end up in a locally generated PMG Backup (instead of creating a tar.gz we send the data to a PBS instance). What works: * creating and manipulating 'remotes' (PBS instances where we back up to) * creating, restoring and forgetting backups to those remotes * restoring a backup from another PMG installation from a configured remote * creating schedules for backups (systemd-timer units) for periodic backups via CLI (pmgbackup, pmgsh) What's missing: * quite a bit of refactoring (e.g. move the systemd-timer helpers and most of PBSTools to pve-common (and reuse it in the PBS Storage plugin) * encryption support * pruneing support * testing * GUI * Documentation The first patch is a small cosmetic cleanup, and independent of the series. A sample session (which I used for testing: ``` pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint ff:ff:... pmgbackup remote list pmgbackup pbsjob run pbs1 pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1 pmgbackup pbsjob run pbs1 pmgbackup pbsjob list_backups pbs1 pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s' systemctl list-timers --all ``` pve-common: Stoiko Ivanov (2): Systemd: add helpers for parsing unit files add helper module for handling PBS Integration src/Makefile | 1 + src/PVE/PBSTools.pm | 309 ++++++++++++++++++++++++++++++++++++++++++++ src/PVE/Systemd.pm | 72 +++++++++++ 3 files changed, 382 insertions(+) create mode 100644 src/PVE/PBSTools.pm pmg-api: Stoiko Ivanov (11): drop left-over commented out code Backup: split backup creation and creating tar Restore: optionally restore from directory Backup: push restore options to PMG::Backup debian: add dependency on proxmox-backup-client add initial SectionConfig for pbs Add API2 module for PBS configuration Add API2 module for per-node backups to PBS pbs-integration: add CLI calls to pmgbackup add scheduled backup to PBS remotes add daily timer for pruning configured remotes debian/control | 1 + debian/dirs | 1 + debian/pmg-pbsbackup@.service | 6 + debian/pmg-pbsprune.service | 6 + debian/pmg-pbsprune.timer | 10 + debian/rules | 4 +- src/Makefile | 8 +- src/PMG/API2/Backup.pm | 21 +- src/PMG/API2/Config.pm | 7 + src/PMG/API2/Nodes.pm | 7 + src/PMG/API2/PBS/Job.pm | 544 ++++++++++++++++++++++++++++++++++ src/PMG/API2/PBS/Remote.pm | 248 ++++++++++++++++ src/PMG/Backup.pm | 86 +++++- src/PMG/CLI/pmgbackup.pm | 102 +++++++ src/PMG/PBSConfig.pm | 209 +++++++++++++ src/PMG/PBSSchedule.pm | 104 +++++++ 16 files changed, 1327 insertions(+), 37 deletions(-) create mode 100644 debian/pmg-pbsbackup@.service create mode 100644 debian/pmg-pbsprune.service create mode 100644 debian/pmg-pbsprune.timer create mode 100644 src/PMG/API2/PBS/Job.pm create mode 100644 src/PMG/API2/PBS/Remote.pm create mode 100644 src/PMG/PBSConfig.pm create mode 100644 src/PMG/PBSSchedule.pm pmg-gui: Stoiko Ivanov (3): Make Backup/Restore panel a menuentry refactor RestoreWindow for PBS add PBSConfig tab to Backup menu js/BackupConfiguration.js | 23 ++ js/BackupRestore.js | 68 ++-- js/Makefile | 2 + js/NavigationTree.js | 6 + js/PBSConfig.js | 680 ++++++++++++++++++++++++++++++++++++++ js/SystemConfiguration.js | 4 - 6 files changed, 750 insertions(+), 33 deletions(-) create mode 100644 js/BackupConfiguration.js create mode 100644 js/PBSConfig.js -- 2.20.1