[PATCH proxmox{-widget-toolkit,,-backup} v2 00/43] append-only sync jobs and snapshot retention timespan

[Christian Ebner <c.ebner@proxmox.com>]

Currently sync jobs cannot be configured to be fully append-only
since namespace creation requires datastore modify privileges to do
so. Further, permissions would also allow to restore or modify owned
content.

This patch series therefore extends the current permissions
and roles to allow for append only sync jobs, by only allowing
the minimally required permissions and roles.

In particular, for push the sync jobs local user on the source
requires RemoteSyncAppendOperator as well as DatastoreReader on the
source datastore, with DatastoreAppend and DatastoreAudit (latter for
listing privs of pre-existing contents without restore) permissions
for the user on the remote instance used for connection.

For pull, the user on the target must be able to append to the
datastore via DatastoreAppend and able to read from the remote source
by the respective RemoteSyncOperator permissions on the remote and
by either DatastoreBackup or DatastoreReader permissions to access the
contents.

Further, sync jobs are extended to allow setting a retention timespan
for which synced snapshots cannot be pruned, neither by the sync job,
nor by prune jobs. Only root@pam is allowed to change the retention
period. After the retention period, snapshots behave like regular
snapshots again and can be pruned.

To protect from sync jobs or clients setting unintended retention
timespans, a maximum retention timespan must be configured on the
datastore before any retention can be set. Optionally, a default
retention timespan can be configured as fallback for new snapshots.

S3 object locking is still work in progress, so not included as part
of this patch series. This will be either a followup or included in
future versions of this patch series, if needed.

Changes since RFC version 1 (thanks to Robert and Thomas for feedback):
- New patches for proxmox-widget-toolkit to fix proxmoxDateTimeField
  behaviour in order to use it for retention time picking.
- Retention can only be set or extended, never shortened or cleared.
  The endpoint to modify it is now limited to the superuser.
- Setting a retention requires a configured max-retention-timespan on
  the datastore, enforced for backups, sync jobs, the API endpoint and
  tape restores.
- Add an optional default-retention-timespan datastore option, used
  for new snapshots if neither client nor sync job request a
  retention. It must not exceed the max. Both values are exposed in
  the datastore status, so clients can check the limits upfront.
- Add client options to set the retention on backup and a CLI command
  to extend it for existing snapshots.
- Snapshots with active retention are now treated like protected
  ones: prune keeps them without them taking up keep slots, while
  group removal and remove-vanished skip them instead of failing.
- Pull no longer replaces the whole manifest when setting the
  retention. Only the unprotected part is changed, keeping the file
  list and signature intact, and updates of the unprotected part on
  the source are still synced. Repeated pulls no longer abort the
  group for retained snapshots.
- Resync of corrupt snapshots honors the retention, retained ones are
  skipped with a warning. With append-only privileges, corrupt
  snapshots are skipped with a warning instead of the whole namespace.
- Tape restores no longer re-lock snapshots with already expired
  retention.
- Retention now also covers the second of the retention timestamp
  itself, consistently for all checks.
- Drop the bulk S3 object removal on recursive namespace deletion, as
  it did not honor group locks and bypassed per-snapshot checks.
- Do not leak the group owner in backup owner check errors.
- Do not fail the client status output for users lacking privileges
  to see the storage usage.

Design decisions:
- The retention is only set on snapshot creation by backup or sync,
  it is not renewed on resync. Retention timestamps of the sync source
  are never propagated, the target sets its own based on the sync job
  or datastore default.
- Tape restores shorten retention timestamps exceeding the datastore
  limit, or strip them if no limit is configured. s3-refresh only
  warns about retention exceeding the limit, as the bucket is intended
  to become the dominant retention once retention is enforced via S3
  object lock.
- Snapshots with a manifest that cannot be parsed can no longer be
  removed without manual intervention, as their retention cannot be
  determined.
- Retention support must be explicitley enabled on the datastore by
  setting the max-retention-timespan. This is enforced also for the
  default-retention-timespan and helps to protect against misuse.


proxmox-widget-toolkit:

Christian Ebner (5):
  form: date time field: init with now instead of epoch on empty fields
  form: date time field: set min/max value limits passed via config
  form: date time field: update to new limits when switching day/time
  form: date time field: submit combined datetime instead of item fields
  form: date time field: track value like other form fields

 src/form/DateTimeField.js | 88 ++++++++++++++++++++++++++++++++-------
 1 file changed, 72 insertions(+), 16 deletions(-)


proxmox:

Christian Ebner (6):
  pbs-api-types: add append only privilege and role
  pbs-api-types: add remote datastore append privilege and role
  pbs-api-types: extend snapshot list items by retention timestamp
  pbs-api-types: extend sync job config by retention-timespan parameter
  pbs-api-types: add max/default retention timespan property to
    datastore
  pbs-api-types: store: expose max/default retention timespans in status

 pbs-api-types/src/acl.rs       | 24 +++++++++++++
 pbs-api-types/src/datastore.rs | 31 ++++++++++++++--
 pbs-api-types/src/jobs.rs      | 64 +++++++++++++++++++++++++++++++++-
 3 files changed, 116 insertions(+), 3 deletions(-)


proxmox-backup:

Christian Ebner (32):
  config/tools: add new fields for sync and snapshot/datastore list
    items
  api: admin: improve code style for status endpoint
  client: avoid error in status if user lacks permissions
  server: allow iterating contents for Datastore.Append permissions
  api: backup: fix possible information leak in multi-tenant datastores
  datastore: s3: fix namespace level bulk remove not honoring group
    locks
  api: backup: allow backup for user/token with append permission
  api: allow namespace creation on append permissions
  api: sync: allow pull to target for user/token with append permission
  sync: pull: allow pulling for user/token with append permission
  sync: push: allow push and ns creation on Remote.DatastoreAppend
  datastore: conditionally treat missing manifest as error or benign
  api: backup: provide retain-until timestamp for extended prune
    protection
  tools: include retain-until timestamp in snapshot list items
  client: backup writer: allow to send retain-until timestamp on backup
  sync: push: allow to set retention timestamp for synced snapshots
  sync: pull: set retention timestamp from sync job for new snapshots
  datastore/config: parse and enforce maximum retention timespan
  api: config: allow to set or delete retention timespan for sync jobs
  api: admin: allow to modify snapshot retention for root user
  tape/s3-refresh: check for retentions outside datastore limits
  ui: add retention timespan form and use it for sync job edit window
  ui: allow datastore wide max retention timespan configuration
  ui: show snapshot retention in datastore contents and allow extension
  group: destroy: s3: skip snapshots with active retentions
  prune: skip snapshots with active retentions
  api/ui: expose default retention setting as datastore config option
  api: datastore: expose max/default retention timespans in status
    listing
  backup/sync: fallback to datastore default-retention-timespan
  client: add flags to set snapshot retention via timestamp/timespan
  docs: describe the privileges and roles for append only backup/sync
  docs: document snapshot retention for new backup snapshots

 docs/backup-client.rst                     |  28 ++++
 docs/maintenance.rst                       |  22 +++
 docs/managing-remotes.rst                  |  26 +++-
 docs/storage.rst                           |   7 +
 docs/user-management.rst                   |  16 +++
 examples/upload-speed.rs                   |   1 +
 pbs-client/src/backup_writer.rs            |   5 +
 pbs-datastore/src/backup_info.rs           | 110 +++++++++++++--
 pbs-datastore/src/datastore.rs             | 113 +++++++++++----
 pbs-datastore/src/manifest.rs              |  50 +++++++
 pbs-datastore/src/prune.rs                 |  54 +++++++-
 proxmox-backup-client/src/benchmark.rs     |   1 +
 proxmox-backup-client/src/main.rs          |  94 +++++++++++--
 proxmox-backup-client/src/snapshot.rs      |  98 ++++++++++++-
 src/api2/admin/datastore.rs                |  93 ++++++++++---
 src/api2/admin/namespace.rs                |   8 +-
 src/api2/backup/environment.rs             |  14 ++
 src/api2/backup/mod.rs                     |  37 ++++-
 src/api2/config/datastore.rs               |  44 ++++++
 src/api2/config/sync.rs                    |  51 ++++++-
 src/api2/pull.rs                           |  19 ++-
 src/api2/push.rs                           |  17 ++-
 src/api2/tape/restore.rs                   |  22 ++-
 src/backup/hierarchy.rs                    |  23 +++-
 src/backup/verify.rs                       |   1 +
 src/server/prune_job.rs                    |   5 +-
 src/server/pull.rs                         | 152 +++++++++++++++++++--
 src/server/push.rs                         |  46 ++++++-
 src/server/sync.rs                         |   1 +
 src/tools/mod.rs                           |  23 ++--
 tests/prune.rs                             |  49 ++++++-
 www/Makefile                               |   4 +
 www/datastore/Content.js                   | 103 +++++++++++++-
 www/datastore/OptionView.js                |  16 +++
 www/datastore/Prune.js                     |   4 +
 www/form/RetentionTimespan.js              |  22 +++
 www/window/DefaultRetentionTimespanEdit.js |  25 ++++
 www/window/MaxRetentionTimespanEdit.js     |  25 ++++
 www/window/RetentionEdit.js                |  56 ++++++++
 www/window/SyncJobEdit.js                  |   9 ++
 40 files changed, 1344 insertions(+), 150 deletions(-)
 create mode 100644 www/form/RetentionTimespan.js
 create mode 100644 www/window/DefaultRetentionTimespanEdit.js
 create mode 100644 www/window/MaxRetentionTimespanEdit.js
 create mode 100644 www/window/RetentionEdit.js


Summary over all repositories:
  44 files changed, 1532 insertions(+), 169 deletions(-)

-- 
Generated by murpp 0.11.0