From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 39A321FF2A0 for ; Mon, 15 Jul 2024 12:15:58 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3E82B37B70; Mon, 15 Jul 2024 12:16:20 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Mon, 15 Jul 2024 12:15:38 +0200 Message-Id: <20240715101602.274244-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.021 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [RFC proxmox-backup 00/24] fix #3044: push datastore to remote X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" While being mostly implemented, there are still some implementation details to be clarified, therefore requesting comments on the current state of the patch series. This patch series implements the functionality to extend the current sync jobs in pull direction by an additional push direction, allowing to push contents of a local source datastore to a remote target. The series implements this by using the REST API of the remote target for fetching, creating and/or deleting namespaces, groups and backups, and reuses the clients backup writer functionality to create snapshots by writing a manifeset on the remote target and sync the fixed index, dynamic index or blobs contained in the source manifest to the remote, preserving also encryption information. The patch series is structured as follows: - patches 1 to 5 are cleanup patches - patches 6 to 11 are patches restructuring the current code so that functionality of the current pull implementation can be reused for the push implementation as well - patches 12 and 13 extend the backup writers functionality to be able to push snapshots to the target - patches 14 to 16 are once again preparatory patches for shared implementation of sync jobs in pull and push direction - patch 17 defines the required permission acls and roles - patch 18 implements almost all of the logic required for the push, including pushing of the datastore, namespace, groups and snapshots, taking into account also filters and additional sync flags - patch 19 extends the current sync job configuration by a flag allowing to set the direction the sync job should operate, defaulting to pull. - patches 20 to 24 finally expose the new sync job direction via the API, CLI and WebUI. While most of the functionality is already in place, some open questions remain: - Remove vanished stats would require to expose additional information to the REST API endpoints for deletion of namespaces and groups - Performance for push: The current implementation only allows to re-upload known chunks which have been read from the local datastore, there is no download of previous snapshots to optimize this. - Permissions and roles: Currently, a dedicated role is implemented for a push operator. It remains to be clarified if the given permissions are to open, or if additional subsets of permissions might be warranted, e.g. to allow for removed vanished. Christian Ebner (24): datastore: data blob: fix typos in comments server: pull: be more specific in module comment server: pull: silence clippy to many arguments warning www: sync edit: indetation style fix server: pull: fix sync info message for root namespace server: sync: move sync related stats to common module server: sync: move reader trait to common sync module server: sync: move source to common sync module client: backup writer: bundle upload stats counters client: backup writer: factor out merged chunk stream upload client: backup writer: add chunk count and duration stats client: backup writer: allow push uploading index and chunks api: backup: add ignore-previous flag to backup endpoint server: sync: move skip info/reason to common sync module server: sync: make skip reason message more genenric server: sync: factor out namespace depth check into sync module api types: define remote permissions and roles for push sync fix #3044: server: implement push support for sync operations api: config: extend sync job config by sync direction api: push: implement endpoint for sync in push direction api: sync: move sync job invocation to common module bin: manager: add datastore push cli command form: group filter: allow to set namespace for local datastore www: sync edit: allow to set sync direction for sync jobs examples/upload-speed.rs | 1 + pbs-api-types/src/acl.rs | 17 + pbs-api-types/src/jobs.rs | 49 ++ pbs-client/src/backup_writer.rs | 232 +++++-- pbs-datastore/src/data_blob.rs | 6 +- proxmox-backup-client/src/benchmark.rs | 1 + proxmox-backup-client/src/main.rs | 1 + src/api2/admin/sync.rs | 2 +- src/api2/backup/mod.rs | 6 +- src/api2/config/sync.rs | 9 + src/api2/mod.rs | 3 + src/api2/pull.rs | 108 ---- src/api2/push.rs | 207 ++++++ src/api2/sync.rs | 142 +++++ src/bin/proxmox-backup-manager.rs | 216 +++++-- src/bin/proxmox-backup-proxy.rs | 2 +- src/server/mod.rs | 2 + src/server/pull.rs | 656 ++----------------- src/server/push.rs | 835 +++++++++++++++++++++++++ src/server/sync.rs | 570 +++++++++++++++++ www/form/GroupFilter.js | 18 +- www/window/SyncJobEdit.js | 64 +- 22 files changed, 2343 insertions(+), 804 deletions(-) create mode 100644 src/api2/push.rs create mode 100644 src/api2/sync.rs create mode 100644 src/server/push.rs create mode 100644 src/server/sync.rs -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel