From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 60B201FF179 for ; Thu, 1 Aug 2024 09:44:50 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 944A7306E9; Thu, 1 Aug 2024 09:44:53 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Thu, 1 Aug 2024 09:43:32 +0200 Message-Id: <20240801074403.36229-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.019 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH v2 proxmox-backup 00/31] fix #3044: push datastore to remote target X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" This patch series implements the functionality to extend the current sync jobs in pull direction by an additional push direction, allowing to push contents of a local source datastore to a remote target. The series implements this by using the REST API of the remote target for fetching, creating and/or deleting namespaces, groups and backups, and reuses the clients backup writer functionality to create snapshots by writing a manifeset on the remote target and sync the fixed index, dynamic index or blobs contained in the source manifest to the remote, preserving also encryption information. Thanks a lot to Thomas and Gabriel for initial feedback on the previous version, their suggestions have been addressed in this new version of the patch series. Most notable changes since version 1 of the patch series include: - Do not share known chunk state over all backup writer instances. This fixes an issue present in the previous series, where a known chunk upload might have failed, because seemingly unknown to the server. This stems from the fact that the reupload as known chunks is only possible for chunks stored in the the sever side's `BackupEnvironment`s state, which only lives as long as each individual backup writer instance, and only covers chunks from the previous snapshot of the backup group (an the chunks seen by the current writer instance). - Instead, download the index file as listed in the previous snapshot of the group already present on the server and deduplicate based on that. Although this is an orthogonal optimization, it should be the better approach as it mimics the current incremental backup. - Clearly separate sync job configs in pull and push direction, by introducing a dedicated config type `sync-push` for sync jobs in push direction. This avoids potential misconfiguration. - Clearly separate sync jobs in pull and push direction in the WebUI, by using a separate grid to list the jobs. - Rename the `ignore-previous` flag to `no-timestamp-check`, which is more fitting and now actually only does skip the check, as the previous manifest is require to fetch the known chunk list now. The patch series is structured as follows in this version: - patch 1 is a cleanup patch fixing typos in api documentation. - patches 2 to 7 are patches restructuring the current code so that functionality of the current pull implementation can be reused for the push implementation as well. - patches 8 and 9 extend the backup writers functionality to be able to push snapshots to the target. - patches 10 to 12 are once again preparatory patches for shared implementation of sync jobs in pull and push direction. - patch 13 defines the required permission acls and roles. - patch 14 implements almost all of the logic required for the push, including pushing of the datastore, namespace, groups and snapshots, taking into account also filters and additional sync flags. - patch 15 extends the current sync job configuration by a new config type `sync-push` allowing to configure sync jobs in push direction while limiting possible misconfiguration errors. - patches 16 to 26 expose the new sync job direction via the API, CLI and WebUI. - patches 27 to 31 finally are followup patches, changing the return type for the backup group and namespace delete REST API endpoints to return statistics on the deleted snapshots, groups and namespaces, which are then used to include this information in the task log. As this is an API breaking change, the patches are kept independent from the other patches. Link to issue on bugtracker: https://bugzilla.proxmox.com/show_bug.cgi?id=3044 Christian Ebner (31): api: datastore: add missing whitespace in description server: sync: move sync related stats to common module server: sync: move reader trait to common sync module server: sync: move source to common sync module client: backup writer: bundle upload stats counters client: backup writer: factor out merged chunk stream upload client: backup writer: add chunk count and duration stats client: backup writer: allow push uploading index and chunks api: backup: add no-timestamp-check flag to backup endpoint server: sync: move skip info/reason to common sync module server: sync: make skip reason message more genenric server: sync: factor out namespace depth check into sync module api types: define remote permissions and roles for push sync fix #3044: server: implement push support for sync operations config: jobs: add `sync-push` config type for push sync jobs api: push: implement endpoint for sync in push direction api: sync: move sync job invocation to common module api: sync jobs: expose optional `sync-direction` parameter bin: manager: add datastore push cli command ui: group filter: allow to set namespace for local datastore ui: sync edit: source group filters based on sync direction ui: add view with separate grids for pull and push sync jobs ui: sync job: adapt edit window to be used for pull and push ui: sync: pass sync-direction to allow removing push jobs ui: sync view: do not use data model proxy for store ui: sync view: set sync direction when invoking run task via api datastore: move `BackupGroupDeleteStats` to api types api types: implement api type for `BackupGroupDeleteStats` datastore: increment deleted group counter when removing group api: datastore/namespace: return backup groups delete stats on remove server: sync job: use delete stats provided by the api examples/upload-speed.rs | 1 + pbs-api-types/src/acl.rs | 17 + pbs-api-types/src/datastore.rs | 64 ++ pbs-api-types/src/jobs.rs | 52 ++ pbs-client/src/backup_writer.rs | 232 +++++-- pbs-config/src/sync.rs | 11 +- pbs-datastore/src/backup_info.rs | 34 +- pbs-datastore/src/datastore.rs | 27 +- proxmox-backup-client/src/benchmark.rs | 1 + proxmox-backup-client/src/main.rs | 1 + src/api2/admin/datastore.rs | 24 +- src/api2/admin/namespace.rs | 20 +- src/api2/admin/sync.rs | 27 +- src/api2/backup/mod.rs | 4 +- src/api2/config/datastore.rs | 22 +- src/api2/config/notifications/mod.rs | 15 +- src/api2/config/sync.rs | 53 +- src/api2/mod.rs | 3 + src/api2/pull.rs | 108 ---- src/api2/push.rs | 207 ++++++ src/api2/sync.rs | 131 ++++ src/bin/proxmox-backup-manager.rs | 216 +++++-- src/bin/proxmox-backup-proxy.rs | 25 +- src/server/mod.rs | 2 + src/server/pull.rs | 651 ++----------------- src/server/push.rs | 833 +++++++++++++++++++++++++ src/server/sync.rs | 570 +++++++++++++++++ www/Makefile | 1 + www/config/SyncPullPushView.js | 60 ++ www/config/SyncView.js | 47 +- www/datastore/DataStoreList.js | 2 +- www/datastore/Panel.js | 2 +- www/form/GroupFilter.js | 18 +- www/window/SyncJobEdit.js | 41 +- 34 files changed, 2623 insertions(+), 899 deletions(-) create mode 100644 src/api2/push.rs create mode 100644 src/api2/sync.rs create mode 100644 src/server/push.rs create mode 100644 src/server/sync.rs create mode 100644 www/config/SyncPullPushView.js -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel