From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <s.ivanov@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E8FE769DB7 for <pmg-devel@lists.proxmox.com>; Mon, 16 Nov 2020 12:01:34 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DAFFD2DB0E for <pmg-devel@lists.proxmox.com>; Mon, 16 Nov 2020 12:01:34 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 35CEF2DAC7 for <pmg-devel@lists.proxmox.com>; Mon, 16 Nov 2020 12:01:32 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E325343662 for <pmg-devel@lists.proxmox.com>; Mon, 16 Nov 2020 12:01:31 +0100 (CET) From: Stoiko Ivanov <s.ivanov@proxmox.com> To: pmg-devel@lists.proxmox.com Date: Mon, 16 Nov 2020 12:01:07 +0100 Message-Id: <20201116110118.7483-1-s.ivanov@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.085 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [pbsschedule.pm, pbsclient.pm, config.pm, pmgbackup.pm, cluster.pm, remote.pm, pbsconfig.pm, job.pm, nodes.pm] Subject: [pmg-devel] [PATCH pve-common/pmg-api/pmg-gui v3] add initial PBS integration X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion <pmg-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pmg-devel/> List-Post: <mailto:pmg-devel@lists.proxmox.com> List-Help: <mailto:pmg-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Mon, 16 Nov 2020 11:01:34 -0000 changes v2->v3: * rebased the gui-patches to latest master * added param_mapping from pve-storage/pvesm (provide an interactive password entry) cover-letter for v2: changes v1->v2: * renamed PBSTools to PBSClient, and made it a class (handling the config and backup-operations of one PBS instance) * dropped encryption-key support from the GUI, the API calls and the SectionConfig - will be submitted again, when we have a clean solution in PVE * dropped explicit prune support - each group now gets pruned directly after a backup * one small fixup in debian/control (I mistakenly added two ',' in the proxmox-backup-client dependency) * added /etc/pmg/pbs to the cluster-sync cover-letter for v1: changes RFC->v1: * moved the potentially reusable parts to pve-common (PBSTools.pm, and 2 functions in Systemd.pm) * added GUI support (mostly adapted from the LDAPConfig) - huge thanks to Dominik for his patience and help! * added support for encrypted backups * added support for pruning backups * added a systemd-timer, which runs a daily prune based on the configured settings differences from the PBS storage-plugin config in PVE: * the keep-options are kept as separate options instead of using a property string for all (like they are in the datastore config in PBS) cover-letter from the RFC: This series is a minimal proof-of-concept for integrating PBS into PMG. The code needs quite a bit of cleanup, and more testing, however I'll send it as RFC, for an initial sanity-check - to see if I missed some needed functionality. It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd unit manipulation from the PVE::API2::Disks::Directory) and I'd like to refactor the common parts into pve-common. The backup consists of the same files, that end up in a locally generated PMG Backup (instead of creating a tar.gz we send the data to a PBS instance). What works: * creating and manipulating 'remotes' (PBS instances where we back up to) * creating, restoring and forgetting backups to those remotes * restoring a backup from another PMG installation from a configured remote * creating schedules for backups (systemd-timer units) for periodic backups via CLI (pmgbackup, pmgsh) What's missing: * quite a bit of refactoring (e.g. move the systemd-timer helpers and most of PBSTools to pve-common (and reuse it in the PBS Storage plugin) * encryption support * pruneing support * testing * GUI * Documentation The first patch is a small cosmetic cleanup, and independent of the series. A sample session (which I used for testing: ``` pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint ff:ff:... pmgbackup remote list pmgbackup pbsjob run pbs1 pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1 pmgbackup pbsjob run pbs1 pmgbackup pbsjob list_backups pbs1 pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s' systemctl list-timers --all ``` pve-common: Stoiko Ivanov (1): add PBSClient module src/Makefile | 1 + src/PVE/PBSClient.pm | 305 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 306 insertions(+) create mode 100644 src/PVE/PBSClient.pm pmg-api: Stoiko Ivanov (7): debian: drop duplicate ',' in dependencies add initial SectionConfig for PBS Add API2 module for PBS configuration Add API2 module for per-node backups to PBS pbs-integration: add CLI calls to pmgbackup add scheduled backup to PBS remotes add /etc/pmg/pbs to cluster-sync debian/control | 2 +- debian/dirs | 1 + debian/pmg-pbsbackup@.service | 6 + debian/rules | 1 + src/Makefile | 6 +- src/PMG/API2/Config.pm | 7 + src/PMG/API2/Nodes.pm | 7 + src/PMG/API2/PBS/Job.pm | 501 ++++++++++++++++++++++++++++++++++ src/PMG/API2/PBS/Remote.pm | 231 ++++++++++++++++ src/PMG/CLI/pmgbackup.pm | 60 ++++ src/PMG/Cluster.pm | 1 + src/PMG/PBSConfig.pm | 195 +++++++++++++ src/PMG/PBSSchedule.pm | 104 +++++++ 13 files changed, 1120 insertions(+), 2 deletions(-) create mode 100644 debian/pmg-pbsbackup@.service create mode 100644 src/PMG/API2/PBS/Job.pm create mode 100644 src/PMG/API2/PBS/Remote.pm create mode 100644 src/PMG/PBSConfig.pm create mode 100644 src/PMG/PBSSchedule.pm pmg-gui: Stoiko Ivanov (3): Make Backup/Restore panel a menuentry refactor RestoreWindow for PBS add PBSConfig tab to Backup menu js/BackupConfiguration.js | 23 ++ js/BackupRestore.js | 67 ++-- js/Makefile | 2 + js/NavigationTree.js | 6 + js/PBSConfig.js | 678 ++++++++++++++++++++++++++++++++++++++ js/SystemConfiguration.js | 4 - 6 files changed, 748 insertions(+), 32 deletions(-) create mode 100644 js/BackupConfiguration.js create mode 100644 js/PBSConfig.js -- 2.20.1