[pbs-devel] [PATCH v7 proxmox-backup 00/31] fix #3044: push datastore to remote target

[Christian Ebner <c.ebner@proxmox.com>]

This patch series implements the functionality to extend the current
sync jobs in pull direction by an additional push direction, allowing
to push contents of a local source datastore to a remote target.

The series implements this by using the REST API of the remote target
for fetching, creating and/or deleting namespaces, groups and backups,
and reuses the clients backup writer functionality to create snapshots
by writing a manifeset on the remote target and sync the fixed index,
dynamic index or blobs contained in the source manifest to the remote,
preserving also encryption information.

Thanks to Fabian for further feedback to the previous patch series
version.

Changes since version 6 of the patch series:
- Fix permission check for sync job modify access, correctly check local
  datastore access if job not owned by sync user.
- Pre-filter source namespaces, so namespaces which the sync user has no
  access to cannot be leaked.
- Avoid possibly removing unrelated target namespaces during remove
  vanished by only removing sub-namespaces of the remote target namespace.
- Fix issues with local/target namespace mapping, make clear which are
  which by adapting variable names accordingly.
- Adapt roles related to remote datastore access to mimic roles for
  local datastore access.
- Uncoditionally pass namespace parameter and early check and fail if
  remote does not support namespaces.
- Fetch previous snapshots index to initialize known chunks correctly.
- Adapt snapshot filter for excluding snapshots older than current last
  snapshot already present on target.
- Fix incorrect owner header label in sync job grid for push direction.
- Use `BackupGroup`s `cmp::Ord` for sorting, for pull and push
- Update some comments and docs.

Changes since version 5 of the patch series:
- Split roles and permissions for separate remote datastore prune and remote
  datastoe modify roles.
- Fetch target groups filtered by ownership, so to not try to push or remove
  unowned groups.
- Add documentation, highlight the caveats of conflicting push jobs when using
  shared remotes.
- Check also for optional `PRIV_DATASTORE_BACKUP` as opposed to only
  `PRIV_DATASTORE_READ` on source datastore namespace, that user can read the
  contents from there as well.
- Drop `sync-direction` parameter from API endpoints where not needed, determine
  it from the corresponding jobs configuration instead.
- Adapt layout of split job view in WebUI to use more general, less component
  specific values
- Introduce `remote_acl_path` helpers for `BackupNamespace` and `SyncJobConfig`.
- Refactor upload counters to bundle and update counters by chunk variant.
- Rework `version` endpoint and supported api feature check to be based on
  `supported_features` rather than a hardcoded version, allowing for more
  flexibility.
- `PushParameters` now always have the remote version for check stored
  unconditionally.
- Renamed `igonre-protected` to a less misinterpretable `error-on-protected` and
  inverted boolean logic.
- Squashed and reorderd patches, the delete stats are not followup patches as
  they are now fully backwards compatible.

Changes since version 4 of the patch series:
- Rebased onto current master

Most notable changes since version 3 of the patch series include:
- Rework access control permission checks to resemble the pull based
  logic more closely.
  In order to perform a full sync in push direction, including
  permissions for pruning contents with remove vansished, a acl.cfg
  looks like below:
  ```
  acl:1:/datastore/source-store:syncoperator@pbs:DatastoreReader
  acl:1:/remote:syncoperator@pbs:RemoteAudit
  acl:1:/remote/remote-target/target-store:syncoperator@pbs:RemoteDatastorePrune,RemoteSyncPushOperator
  ```
- Modify access to sync jobs now requires `DatastoreAudit` for both,
  pull and push sync jobs
- Fix previously incorrect privs required for removing target
  namespaces
- Fix performance bottleneck by not reading known chunks from source,
  by sending `MergedChunkInfo` instead of `ChunkInfo` over to the
  upload stream
- Factor upload statistic counters and structs out into their own
  module and provide methods for easy conversion
- Implement `map_to_target` helper for easier/more readable source to
  target mapping for namespaces
- Optimize namespace creation on target, only try creating non
  pre-existing namespace components.
- Avoid temp file for manifest and upload source manifest directly
- Not failing on deletion for protected snapshots is now opt-in
- Refactor api endpoint `version` in order to be able to fetch api
  version for target
- Reworked `SyncDirection` api type, use `api` macro to reduce code

Most notable changes since version 2 of the patch series include:
- Add checks and extend roles and privs to allow for restricting a local
  users access to remote datastore operations. In order to perform a
  full sync in push direction, including permissions for namespace
  creation and deleting contents with remove vansished, a acl.cfg looks
  like below:
  ```
  acl:1:/datastore/datastore:syncoperator@pbs:DatastoreAudit
  acl:1:/remote:syncoperator@pbs:RemoteSyncOperator
  acl:1:/remote/local/pushme:syncoperator@pbs:RemoteDatastoreModify,RemoteDatastorePrune,RemoteSyncPushOperator
  ```
  Based on further feedback, privs might get further grouped or an
  additional role containing most of these can be created.
- Drop patch introducing `no-timestamp-check` flag for backup client, as pointed
  out by Fabian this is not needed, as only backups newer than the currently
  last available will be pushed.
- Fix read snapshots from source by using the correct namespace.
- Rename PullParameters `owner` to more fitting `local_user`.
- Fix typos in remote sync push operator comment.
- Fix comments not matching the functionality for the cli implementations.

Link to issue on bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=3044

Christian Ebner (31):
  sync: pull: optimize backup group sorting
  sync: extend sync source's list namespaces method by filter callback
  client: backup writer: refactor backup and upload stats counters
  client: backup writer: factor out merged chunk stream upload
  client: backup writer: allow push uploading index and chunks
  config: acl: refactor acl path component check for datastore
  config: acl: allow namespace components for remote datastores
  api types: add remote acl path method for `BackupNamespace`
  api types: implement remote acl path method for sync job
  api types: define remote permissions and roles for push sync
  datastore: move `BackupGroupDeleteStats` to api types
  api types: implement api type for `BackupGroupDeleteStats`
  datastore: increment deleted group counter when removing group
  api/api-types: refactor api endpoint version, add api types
  fix #3044: server: implement push support for sync operations
  api types/config: add `sync-push` config type for push sync jobs
  api: push: implement endpoint for sync in push direction
  api: sync: move sync job invocation to server sync module
  api: config: Require PRIV_DATASTORE_AUDIT to modify sync job
  api: config: factor out sync job owner check
  api: sync jobs: expose optional `sync-direction` parameter
  api: admin: avoid duplicate name for list sync jobs api method
  bin: manager: add datastore push cli command
  ui: group filter: allow to set namespace for local datastore
  ui: sync edit: source group filters based on sync direction
  ui: add view with separate grids for pull and push sync jobs
  ui: sync job: adapt edit window to be used for pull and push
  ui: sync view: set proxy on view instead of model
  api: datastore/namespace: return backup groups delete stats on remove
  api: version: add 'prune-delete-stats' as supported feature
  docs: add section for sync jobs in push direction

 docs/managing-remotes.rst              |  40 +
 pbs-api-types/src/acl.rs               |  38 +
 pbs-api-types/src/datastore.rs         |  76 +-
 pbs-api-types/src/jobs.rs              |  46 ++
 pbs-api-types/src/lib.rs               |   3 +
 pbs-api-types/src/version.rs           |  88 +++
 pbs-client/src/backup_stats.rs         | 119 +++
 pbs-client/src/backup_writer.rs        | 318 +++++---
 pbs-client/src/inject_reused_chunks.rs |  14 +-
 pbs-client/src/lib.rs                  |   4 +
 pbs-config/src/acl.rs                  |  11 +-
 pbs-config/src/sync.rs                 |  16 +-
 pbs-datastore/src/backup_info.rs       |  34 +-
 pbs-datastore/src/datastore.rs         |  27 +-
 src/api2/admin/datastore.rs            |  29 +-
 src/api2/admin/namespace.rs            |  31 +-
 src/api2/admin/sync.rs                 |  43 +-
 src/api2/config/datastore.rs           |  15 +-
 src/api2/config/notifications/mod.rs   |  21 +-
 src/api2/config/sync.rs                | 296 ++++++--
 src/api2/mod.rs                        |   2 +
 src/api2/pull.rs                       | 108 ---
 src/api2/push.rs                       | 175 +++++
 src/api2/version.rs                    |  42 +-
 src/bin/proxmox-backup-manager.rs      | 216 ++++--
 src/bin/proxmox-backup-proxy.rs        |  24 +-
 src/server/mod.rs                      |   2 +
 src/server/pull.rs                     |  33 +-
 src/server/push.rs                     | 994 +++++++++++++++++++++++++
 src/server/sync.rs                     | 179 ++++-
 www/Makefile                           |   1 +
 www/config/SyncPullPushView.js         |  61 ++
 www/config/SyncView.js                 |  29 +-
 www/datastore/DataStoreList.js         |   2 +-
 www/datastore/Panel.js                 |   2 +-
 www/form/GroupFilter.js                |  21 +-
 www/window/SyncJobEdit.js              |  49 +-
 37 files changed, 2694 insertions(+), 515 deletions(-)
 create mode 100644 pbs-api-types/src/version.rs
 create mode 100644 pbs-client/src/backup_stats.rs
 create mode 100644 src/api2/push.rs
 create mode 100644 src/server/push.rs
 create mode 100644 www/config/SyncPullPushView.js

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel