From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2AB8B1FF0AF for ; Fri, 06 Nov 2026 18:23:50 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2A9882120A; Fri, 06 Nov 2026 18:23:18 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox{-widget-toolkit,,-backup} v2 00/43] append-only sync jobs and snapshot retention timespan Date: Fri, 6 Nov 2026 18:21:52 +0100 Message-ID: <20261106172235.890542-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793985772019 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.464 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6CG3VIXGPPULT4M5IWX5YRKW5JNF4BOM X-Message-ID-Hash: 6CG3VIXGPPULT4M5IWX5YRKW5JNF4BOM X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Currently sync jobs cannot be configured to be fully append-only since namespace creation requires datastore modify privileges to do so. Further, permissions would also allow to restore or modify owned content. This patch series therefore extends the current permissions and roles to allow for append only sync jobs, by only allowing the minimally required permissions and roles. In particular, for push the sync jobs local user on the source requires RemoteSyncAppendOperator as well as DatastoreReader on the source datastore, with DatastoreAppend and DatastoreAudit (latter for listing privs of pre-existing contents without restore) permissions for the user on the remote instance used for connection. For pull, the user on the target must be able to append to the datastore via DatastoreAppend and able to read from the remote source by the respective RemoteSyncOperator permissions on the remote and by either DatastoreBackup or DatastoreReader permissions to access the contents. Further, sync jobs are extended to allow setting a retention timespan for which synced snapshots cannot be pruned, neither by the sync job, nor by prune jobs. Only root@pam is allowed to change the retention period. After the retention period, snapshots behave like regular snapshots again and can be pruned. To protect from sync jobs or clients setting unintended retention timespans, a maximum retention timespan must be configured on the datastore before any retention can be set. Optionally, a default retention timespan can be configured as fallback for new snapshots. S3 object locking is still work in progress, so not included as part of this patch series. This will be either a followup or included in future versions of this patch series, if needed. Changes since RFC version 1 (thanks to Robert and Thomas for feedback): - New patches for proxmox-widget-toolkit to fix proxmoxDateTimeField behaviour in order to use it for retention time picking. - Retention can only be set or extended, never shortened or cleared. The endpoint to modify it is now limited to the superuser. - Setting a retention requires a configured max-retention-timespan on the datastore, enforced for backups, sync jobs, the API endpoint and tape restores. - Add an optional default-retention-timespan datastore option, used for new snapshots if neither client nor sync job request a retention. It must not exceed the max. Both values are exposed in the datastore status, so clients can check the limits upfront. - Add client options to set the retention on backup and a CLI command to extend it for existing snapshots. - Snapshots with active retention are now treated like protected ones: prune keeps them without them taking up keep slots, while group removal and remove-vanished skip them instead of failing. - Pull no longer replaces the whole manifest when setting the retention. Only the unprotected part is changed, keeping the file list and signature intact, and updates of the unprotected part on the source are still synced. Repeated pulls no longer abort the group for retained snapshots. - Resync of corrupt snapshots honors the retention, retained ones are skipped with a warning. With append-only privileges, corrupt snapshots are skipped with a warning instead of the whole namespace. - Tape restores no longer re-lock snapshots with already expired retention. - Retention now also covers the second of the retention timestamp itself, consistently for all checks. - Drop the bulk S3 object removal on recursive namespace deletion, as it did not honor group locks and bypassed per-snapshot checks. - Do not leak the group owner in backup owner check errors. - Do not fail the client status output for users lacking privileges to see the storage usage. Design decisions: - The retention is only set on snapshot creation by backup or sync, it is not renewed on resync. Retention timestamps of the sync source are never propagated, the target sets its own based on the sync job or datastore default. - Tape restores shorten retention timestamps exceeding the datastore limit, or strip them if no limit is configured. s3-refresh only warns about retention exceeding the limit, as the bucket is intended to become the dominant retention once retention is enforced via S3 object lock. - Snapshots with a manifest that cannot be parsed can no longer be removed without manual intervention, as their retention cannot be determined. - Retention support must be explicitley enabled on the datastore by setting the max-retention-timespan. This is enforced also for the default-retention-timespan and helps to protect against misuse. proxmox-widget-toolkit: Christian Ebner (5): form: date time field: init with now instead of epoch on empty fields form: date time field: set min/max value limits passed via config form: date time field: update to new limits when switching day/time form: date time field: submit combined datetime instead of item fields form: date time field: track value like other form fields src/form/DateTimeField.js | 88 ++++++++++++++++++++++++++++++++------- 1 file changed, 72 insertions(+), 16 deletions(-) proxmox: Christian Ebner (6): pbs-api-types: add append only privilege and role pbs-api-types: add remote datastore append privilege and role pbs-api-types: extend snapshot list items by retention timestamp pbs-api-types: extend sync job config by retention-timespan parameter pbs-api-types: add max/default retention timespan property to datastore pbs-api-types: store: expose max/default retention timespans in status pbs-api-types/src/acl.rs | 24 +++++++++++++ pbs-api-types/src/datastore.rs | 31 ++++++++++++++-- pbs-api-types/src/jobs.rs | 64 +++++++++++++++++++++++++++++++++- 3 files changed, 116 insertions(+), 3 deletions(-) proxmox-backup: Christian Ebner (32): config/tools: add new fields for sync and snapshot/datastore list items api: admin: improve code style for status endpoint client: avoid error in status if user lacks permissions server: allow iterating contents for Datastore.Append permissions api: backup: fix possible information leak in multi-tenant datastores datastore: s3: fix namespace level bulk remove not honoring group locks api: backup: allow backup for user/token with append permission api: allow namespace creation on append permissions api: sync: allow pull to target for user/token with append permission sync: pull: allow pulling for user/token with append permission sync: push: allow push and ns creation on Remote.DatastoreAppend datastore: conditionally treat missing manifest as error or benign api: backup: provide retain-until timestamp for extended prune protection tools: include retain-until timestamp in snapshot list items client: backup writer: allow to send retain-until timestamp on backup sync: push: allow to set retention timestamp for synced snapshots sync: pull: set retention timestamp from sync job for new snapshots datastore/config: parse and enforce maximum retention timespan api: config: allow to set or delete retention timespan for sync jobs api: admin: allow to modify snapshot retention for root user tape/s3-refresh: check for retentions outside datastore limits ui: add retention timespan form and use it for sync job edit window ui: allow datastore wide max retention timespan configuration ui: show snapshot retention in datastore contents and allow extension group: destroy: s3: skip snapshots with active retentions prune: skip snapshots with active retentions api/ui: expose default retention setting as datastore config option api: datastore: expose max/default retention timespans in status listing backup/sync: fallback to datastore default-retention-timespan client: add flags to set snapshot retention via timestamp/timespan docs: describe the privileges and roles for append only backup/sync docs: document snapshot retention for new backup snapshots docs/backup-client.rst | 28 ++++ docs/maintenance.rst | 22 +++ docs/managing-remotes.rst | 26 +++- docs/storage.rst | 7 + docs/user-management.rst | 16 +++ examples/upload-speed.rs | 1 + pbs-client/src/backup_writer.rs | 5 + pbs-datastore/src/backup_info.rs | 110 +++++++++++++-- pbs-datastore/src/datastore.rs | 113 +++++++++++---- pbs-datastore/src/manifest.rs | 50 +++++++ pbs-datastore/src/prune.rs | 54 +++++++- proxmox-backup-client/src/benchmark.rs | 1 + proxmox-backup-client/src/main.rs | 94 +++++++++++-- proxmox-backup-client/src/snapshot.rs | 98 ++++++++++++- src/api2/admin/datastore.rs | 93 ++++++++++--- src/api2/admin/namespace.rs | 8 +- src/api2/backup/environment.rs | 14 ++ src/api2/backup/mod.rs | 37 ++++- src/api2/config/datastore.rs | 44 ++++++ src/api2/config/sync.rs | 51 ++++++- src/api2/pull.rs | 19 ++- src/api2/push.rs | 17 ++- src/api2/tape/restore.rs | 22 ++- src/backup/hierarchy.rs | 23 +++- src/backup/verify.rs | 1 + src/server/prune_job.rs | 5 +- src/server/pull.rs | 152 +++++++++++++++++++-- src/server/push.rs | 46 ++++++- src/server/sync.rs | 1 + src/tools/mod.rs | 23 ++-- tests/prune.rs | 49 ++++++- www/Makefile | 4 + www/datastore/Content.js | 103 +++++++++++++- www/datastore/OptionView.js | 16 +++ www/datastore/Prune.js | 4 + www/form/RetentionTimespan.js | 22 +++ www/window/DefaultRetentionTimespanEdit.js | 25 ++++ www/window/MaxRetentionTimespanEdit.js | 25 ++++ www/window/RetentionEdit.js | 56 ++++++++ www/window/SyncJobEdit.js | 9 ++ 40 files changed, 1344 insertions(+), 150 deletions(-) create mode 100644 www/form/RetentionTimespan.js create mode 100644 www/window/DefaultRetentionTimespanEdit.js create mode 100644 www/window/MaxRetentionTimespanEdit.js create mode 100644 www/window/RetentionEdit.js Summary over all repositories: 44 files changed, 1532 insertions(+), 169 deletions(-) -- Generated by murpp 0.11.0