From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 7F8A61FF168 for ; Mon, 14 Oct 2024 13:04:35 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 12FDD32C80; Mon, 14 Oct 2024 13:05:07 +0200 (CEST) Date: Mon, 14 Oct 2024 13:04:58 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox Backup Server development discussion References: <20240912143322.548839-1-c.ebner@proxmox.com> In-Reply-To: <20240912143322.548839-1-c.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1728903796.p0pah9uq7h.astroid@yuna.none> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.047 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] partially-applied: [PATCH v3 proxmox-backup 00/33] fix #3044: push datastore to remote target X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" applied the initial seven refactoring patches 1-4 & 9-11, leaving out the backup writer ones for now since those might still see some changes. On September 12, 2024 4:32 pm, Christian Ebner wrote: > This patch series implements the functionality to extend the current > sync jobs in pull direction by an additional push direction, allowing > to push contents of a local source datastore to a remote target. > > The series implements this by using the REST API of the remote target > for fetching, creating and/or deleting namespaces, groups and backups, > and reuses the clients backup writer functionality to create snapshots > by writing a manifeset on the remote target and sync the fixed index, > dynamic index or blobs contained in the source manifest to the remote, > preserving also encryption information. > > Thanks to Fabian for further feedback to the previous version of the > patches, especially regarding users and ACLs. > > Most notable changes since version 2 of the patch series include: > - Add checks and extend roles and privs to allow for restricting a local > users access to remote datastore operations. In order to perform a > full sync in push direction, including permissions for namespace > creation and deleting contents with remove vansished, a acl.cfg looks > like below: > ``` > acl:1:/datastore/datastore:syncoperator@pbs:DatastoreAudit > acl:1:/remote:syncoperator@pbs:RemoteSyncOperator > acl:1:/remote/local/pushme:syncoperator@pbs:RemoteDatastoreModify,RemoteDatastorePrune,RemoteSyncPushOperator > ``` > Based on further feedback, privs might get further grouped or an > additional role containing most of these can be created. > - Drop patch introducing `no-timestamp-check` flag for backup client, as pointed > out by Fabian this is not needed, as only backups newer than the currently > last available will be pushed. > - Fix read snapshots from source by using the correct namespace. > - Rename PullParameters `owner` to more fitting `local_user`. > - Fix typos in remote sync push operator comment. > - Fix comments not matching the functionality for the cli implementations. > > The patch series is structured as follows in this version: > - patch 1 is a cleanup patch fixing typos in api documentation. > - patches 2 to 7 are patches restructuring the current code so that > functionality of the current pull implementation can be reused for > the push implementation as well. > - patch 8 extens the backup writers functionality to be able to push > snapshots to the target. > - patches 9 to 11 are once again preparatory patches for shared > implementation of sync jobs in pull and push direction. > - patches 12 to 14 define the required permission acls and roles. > - patch 15 implements almost all of the logic required for the push, > including pushing of the datastore, namespace, groups and snapshots, > taking into account also filters and additional sync flags. > - patch 16 extends the current sync job configuration by a new config > type `sync-push` allowing to configure sync jobs in push direction > while limiting possible misconfiguration errors. > - patches 17 to 28 expose the new sync job direction via the API, CLI > and WebUI. > - patches 29 to 33 finally are followup patches, changing the return > type for the backup group and namespace delete REST API endpoints > to return statistics on the deleted snapshots, groups and namespaces, > which are then used to include this information in the task log. > As this is an API breaking change, the patches are kept independent > from the other patches. > > Link to issue on bugtracker: > https://bugzilla.proxmox.com/show_bug.cgi?id=3044 > > Christian Ebner (33): > api: datastore: add missing whitespace in description > server: sync: move sync related stats to common module > server: sync: move reader trait to common sync module > server: sync: move source to common sync module > client: backup writer: bundle upload stats counters > client: backup writer: factor out merged chunk stream upload > client: backup writer: add chunk count and duration stats > client: backup writer: allow push uploading index and chunks > server: sync: move skip info/reason to common sync module > server: sync: make skip reason message more genenric > server: sync: factor out namespace depth check into sync module > config: acl: mention optional namespace acl path component > config: acl: allow namespace components for remote datastores > api types: define remote permissions and roles for push sync > fix #3044: server: implement push support for sync operations > config: jobs: add `sync-push` config type for push sync jobs > api: push: implement endpoint for sync in push direction > api: sync: move sync job invocation to server sync module > api: sync jobs: expose optional `sync-direction` parameter > api: sync: add permission checks for push sync jobs > bin: manager: add datastore push cli command > ui: group filter: allow to set namespace for local datastore > ui: sync edit: source group filters based on sync direction > ui: add view with separate grids for pull and push sync jobs > ui: sync job: adapt edit window to be used for pull and push > ui: sync: pass sync-direction to allow removing push jobs > ui: sync view: do not use data model proxy for store > ui: sync view: set sync direction when invoking run task via api > datastore: move `BackupGroupDeleteStats` to api types > api types: implement api type for `BackupGroupDeleteStats` > datastore: increment deleted group counter when removing group > api: datastore/namespace: return backup groups delete stats on remove > server: sync job: use delete stats provided by the api > > pbs-api-types/src/acl.rs | 32 + > pbs-api-types/src/datastore.rs | 64 ++ > pbs-api-types/src/jobs.rs | 52 ++ > pbs-client/src/backup_writer.rs | 228 +++++-- > pbs-config/src/acl.rs | 7 +- > pbs-config/src/sync.rs | 11 +- > pbs-datastore/src/backup_info.rs | 34 +- > pbs-datastore/src/datastore.rs | 27 +- > src/api2/admin/datastore.rs | 24 +- > src/api2/admin/namespace.rs | 20 +- > src/api2/admin/sync.rs | 45 +- > src/api2/config/datastore.rs | 22 +- > src/api2/config/notifications/mod.rs | 15 +- > src/api2/config/sync.rs | 84 ++- > src/api2/mod.rs | 2 + > src/api2/pull.rs | 108 ---- > src/api2/push.rs | 182 ++++++ > src/bin/proxmox-backup-manager.rs | 216 +++++-- > src/bin/proxmox-backup-proxy.rs | 25 +- > src/server/mod.rs | 3 + > src/server/pull.rs | 658 ++------------------ > src/server/push.rs | 883 +++++++++++++++++++++++++++ > src/server/sync.rs | 700 +++++++++++++++++++++ > www/Makefile | 1 + > www/config/SyncPullPushView.js | 60 ++ > www/config/SyncView.js | 47 +- > www/datastore/DataStoreList.js | 2 +- > www/datastore/Panel.js | 2 +- > www/form/GroupFilter.js | 18 +- > www/window/SyncJobEdit.js | 45 +- > 30 files changed, 2706 insertions(+), 911 deletions(-) > create mode 100644 src/api2/push.rs > create mode 100644 src/server/push.rs > create mode 100644 src/server/sync.rs > create mode 100644 www/config/SyncPullPushView.js > > -- > 2.39.2 > > > > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel > > > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel