From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D754060DBC for ; Mon, 19 Oct 2020 21:02:27 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D57452F1BC for ; Mon, 19 Oct 2020 21:02:27 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 8115E2F133 for ; Mon, 19 Oct 2020 21:02:23 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 525C145E33 for ; Mon, 19 Oct 2020 21:02:23 +0200 (CEST) From: Stoiko Ivanov To: pmg-devel@lists.proxmox.com Date: Mon, 19 Oct 2020 21:01:57 +0200 Message-Id: <20201019190209.11495-1-s.ivanov@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.329 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_SBL 0.644 Contains an URL's NS IP listed in the Spamhaus SBL blocklist [backup.pm] URIBL_SBL_A 0.1 Contains URL's A record listed in the Spamhaus SBL blocklist [backup.pm] Subject: [pmg-devel] [RFC pmg-api 00/12] POC PBS integration X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2020 19:02:27 -0000 This series is a minimal proof-of-concept for integrating PBS into PMG. The code needs quite a bit of cleanup, and more testing, however I'll send it as RFC, for an initial sanity-check - to see if I missed some needed functionality. It reuses quite a bit of code from pve-storage (PBSPlugin, and the systemd unit manipulation from the PVE::API2::Disks::Directory) and I'd like to refactor the common parts into pve-common. The backup consists of the same files, that end up in a locally generated PMG Backup (instead of creating a tar.gz we send the data to a PBS instance). What works: * creating and manipulating 'remotes' (PBS instances where we back up to) * creating, restoring and forgetting backups to those remotes * restoring a backup from another PMG installation from a configured remote * creating schedules for backups (systemd-timer units) for periodic backups via CLI (pmgbackup, pmgsh) What's missing: * quite a bit of refactoring (e.g. move the systemd-timer helpers and most of PBSTools to pve-common (and reuse it in the PBS Storage plugin) * encryption support * pruneing support * testing * GUI * Documentation The first patch is a small cosmetic cleanup, and independent of the series. A sample session (which I used for testing: ``` pmgbackup remote add pbs1 --datastore local --server 192.0.2.11 --user root@pam --password xxx --fingerprint ff:ff:... pmgbackup remote list pmgbackup pbsjob run pbs1 pmgbackup pbsjob restore pbs1 -althost pmg-live -config 1 -database 1 -statistic 1 pmgbackup pbsjob run pbs1 pmgbackup pbsjob list_backups pbs1 pmgbackup pbsjob create pbs1 --schedule 'minutely' --delay '0s' systemctl list-timers --all ``` Stoiko Ivanov (12): drop left-over commented out code Backup: split backup creation and creating tar Restore: optionally restore from directory Backup: push restore options to PMG::Backup debian: add dependency on proxmox-backup-client add helper module for handling PBS Integration PBSTools: add methods for managing backups PBSTools: add systemd-timer helpers add initial SectionConfig for pbs Add API2 module for PBS configuration Add API2 module for per-node backups to PBS pbs-integration: add CLI calls to pmgbackup debian/control | 1 + debian/dirs | 1 + debian/pmg-pbsbackup@.service | 6 + debian/rules | 1 + src/Makefile | 6 +- src/PMG/API2/Backup.pm | 21 +- src/PMG/API2/Config.pm | 7 + src/PMG/API2/Nodes.pm | 7 + src/PMG/API2/PBS/Job.pm | 467 ++++++++++++++++++++++++++++++++++ src/PMG/API2/PBS/Remote.pm | 218 ++++++++++++++++ src/PMG/Backup.pm | 86 +++++-- src/PMG/CLI/pmgbackup.pm | 27 ++ src/PMG/PBSConfig.pm | 168 ++++++++++++ src/PMG/PBSTools.pm | 466 +++++++++++++++++++++++++++++++++ 14 files changed, 1447 insertions(+), 35 deletions(-) create mode 100644 debian/pmg-pbsbackup@.service create mode 100644 src/PMG/API2/PBS/Job.pm create mode 100644 src/PMG/API2/PBS/Remote.pm create mode 100644 src/PMG/PBSConfig.pm create mode 100644 src/PMG/PBSTools.pm -- 2.20.1