From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 1DA101FF16B for ; Thu, 31 Oct 2024 13:16:27 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3261F85A6; Thu, 31 Oct 2024 13:16:25 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Thu, 31 Oct 2024 13:14:50 +0100 Message-Id: <20241031121519.434337-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.029 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox-backup-proxy.rs, lib.rs, jobs.rs, pull.rs, push.rs, mod.rs, namespace.rs, acl.rs, datastore.rs, proxmox.com, proxmox-backup-manager.rs, version.rs, sync.rs] Subject: [pbs-devel] [PATCH v6 proxmox-backup 00/29] fix #3044: push datastore to remote target X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" This patch series implements the functionality to extend the current sync jobs in pull direction by an additional push direction, allowing to push contents of a local source datastore to a remote target. The series implements this by using the REST API of the remote target for fetching, creating and/or deleting namespaces, groups and backups, and reuses the clients backup writer functionality to create snapshots by writing a manifeset on the remote target and sync the fixed index, dynamic index or blobs contained in the source manifest to the remote, preserving also encryption information. Thanks to Fabian and Dominik for further feedback to the previous patch series version. Changes since version 5 of the patch series - Split roles and permissions for separate remote datastore prune and remote datastoe modify roles. - Fetch target groups filtered by ownership, so to not try to push or remove unowned groups. - Add documentation, highlight the caveats of conflicting push jobs when using shared remotes. - Check also for optional `PRIV_DATASTORE_BACKUP` as opposed to only `PRIV_DATASTORE_READ` on source datastore namespace, that user can read the contents from there as well. - Drop `sync-direction` parameter from API endpoints where not needed, determine it from the corresponding jobs configuration instead. - Adapt layout of split job view in WebUI to use more general, less component specific values - Introduce `remote_acl_path` helpers for `BackupNamespace` and `SyncJobConfig`. - Refactor upload counters to bundle and update counters by chunk variant. - Rework `version` endpoint and supported api feature check to be based on `supported_features` rather than a hardcoded version, allowing for more flexibility. - `PushParameters` now always have the remote version for check stored unconditionally. - Renamed `igonre-protected` to a less misinterpretable `error-on-protected` and inverted boolean logic. - Squashed and reorderd patches, the delete stats are not followup patches as they are now fully backwards compatible. Changes since version 4 of the patch series: - Rebased onto current master Most notable changes since version 3 of the patch series include: - Rework access control permission checks to resemble the pull based logic more closely. In order to perform a full sync in push direction, including permissions for pruning contents with remove vansished, a acl.cfg looks like below: ``` acl:1:/datastore/source-store:syncoperator@pbs:DatastoreReader acl:1:/remote:syncoperator@pbs:RemoteAudit acl:1:/remote/remote-target/target-store:syncoperator@pbs:RemoteDatastorePrune,RemoteSyncPushOperator ``` - Modify access to sync jobs now requires `DatastoreAudit` for both, pull and push sync jobs - Fix previously incorrect privs required for removing target namespaces - Fix performance bottleneck by not reading known chunks from source, by sending `MergedChunkInfo` instead of `ChunkInfo` over to the upload stream - Factor upload statistic counters and structs out into their own module and provide methods for easy conversion - Implement `map_to_target` helper for easier/more readable source to target mapping for namespaces - Optimize namespace creation on target, only try creating non pre-existing namespace components. - Avoid temp file for manifest and upload source manifest directly - Not failing on deletion for protected snapshots is now opt-in - Refactor api endpoint `version` in order to be able to fetch api version for target - Reworked `SyncDirection` api type, use `api` macro to reduce code Most notable changes since version 2 of the patch series include: - Add checks and extend roles and privs to allow for restricting a local users access to remote datastore operations. In order to perform a full sync in push direction, including permissions for namespace creation and deleting contents with remove vansished, a acl.cfg looks like below: ``` acl:1:/datastore/datastore:syncoperator@pbs:DatastoreAudit acl:1:/remote:syncoperator@pbs:RemoteSyncOperator acl:1:/remote/local/pushme:syncoperator@pbs:RemoteDatastoreModify,RemoteDatastorePrune,RemoteSyncPushOperator ``` Based on further feedback, privs might get further grouped or an additional role containing most of these can be created. - Drop patch introducing `no-timestamp-check` flag for backup client, as pointed out by Fabian this is not needed, as only backups newer than the currently last available will be pushed. - Fix read snapshots from source by using the correct namespace. - Rename PullParameters `owner` to more fitting `local_user`. - Fix typos in remote sync push operator comment. - Fix comments not matching the functionality for the cli implementations. Link to issue on bugtracker: https://bugzilla.proxmox.com/show_bug.cgi?id=3044 Christian Ebner (29): client: backup writer: refactor backup and upload stats counters client: backup writer: factor out merged chunk stream upload client: backup writer: allow push uploading index and chunks config: acl: refactor acl path component check for datastore config: acl: allow namespace components for remote datastores api types: add remote acl path method for `BackupNamespace` api types: implement remote acl path method for sync job api types: define remote permissions and roles for push sync datastore: move `BackupGroupDeleteStats` to api types api types: implement api type for `BackupGroupDeleteStats` datastore: increment deleted group counter when removing group api/api-types: refactor api endpoint version, add api types fix #3044: server: implement push support for sync operations api types/config: add `sync-push` config type for push sync jobs api: push: implement endpoint for sync in push direction api: sync: move sync job invocation to server sync module api: config: Require PRIV_DATASTORE_AUDIT to modify sync job api: config: factor out sync job owner check api: sync jobs: expose optional `sync-direction` parameter api: admin: avoid duplicate name for list sync jobs api method bin: manager: add datastore push cli command ui: group filter: allow to set namespace for local datastore ui: sync edit: source group filters based on sync direction ui: add view with separate grids for pull and push sync jobs ui: sync job: adapt edit window to be used for pull and push ui: sync view: set proxy on view instead of model api: datastore/namespace: return backup groups delete stats on remove api: version: add 'prune-delete-stats' as supported feature docs: add section for sync jobs in push direction docs/managing-remotes.rst | 36 + pbs-api-types/src/acl.rs | 31 + pbs-api-types/src/datastore.rs | 76 +- pbs-api-types/src/jobs.rs | 46 ++ pbs-api-types/src/lib.rs | 3 + pbs-api-types/src/version.rs | 109 +++ pbs-client/src/backup_stats.rs | 119 +++ pbs-client/src/backup_writer.rs | 318 +++++--- pbs-client/src/inject_reused_chunks.rs | 14 +- pbs-client/src/lib.rs | 4 + pbs-config/src/acl.rs | 11 +- pbs-config/src/sync.rs | 16 +- pbs-datastore/src/backup_info.rs | 34 +- pbs-datastore/src/datastore.rs | 27 +- src/api2/admin/datastore.rs | 29 +- src/api2/admin/namespace.rs | 31 +- src/api2/admin/sync.rs | 43 +- src/api2/config/datastore.rs | 15 +- src/api2/config/notifications/mod.rs | 21 +- src/api2/config/sync.rs | 298 ++++++-- src/api2/mod.rs | 2 + src/api2/pull.rs | 108 --- src/api2/push.rs | 183 +++++ src/api2/version.rs | 42 +- src/bin/proxmox-backup-manager.rs | 216 ++++-- src/bin/proxmox-backup-proxy.rs | 24 +- src/server/mod.rs | 2 + src/server/pull.rs | 6 +- src/server/push.rs | 980 +++++++++++++++++++++++++ src/server/sync.rs | 150 +++- www/Makefile | 1 + www/config/SyncPullPushView.js | 61 ++ www/config/SyncView.js | 18 +- www/datastore/DataStoreList.js | 2 +- www/datastore/Panel.js | 2 +- www/form/GroupFilter.js | 21 +- www/window/SyncJobEdit.js | 45 +- 37 files changed, 2644 insertions(+), 500 deletions(-) create mode 100644 pbs-api-types/src/version.rs create mode 100644 pbs-client/src/backup_stats.rs create mode 100644 src/api2/push.rs create mode 100644 src/server/push.rs create mode 100644 www/config/SyncPullPushView.js -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel