From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id CCB0C1FF13C for ; Thu, 19 Mar 2026 17:13:18 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A5C6D4B33; Thu, 19 Mar 2026 17:13:33 +0100 (CET) From: Hannes Laimer To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox-backup v5 0/9] fixes #6195: add support for moving groups and namespaces Date: Thu, 19 Mar 2026 17:13:16 +0100 Message-ID: <20260319161325.206846-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1773936763742 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.984 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.408 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.819 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.903 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KUF2ZXE3OWDTJ7UOEYZZ7GFA36JWPMAO X-Message-ID-Hash: KUF2ZXE3OWDTJ7UOEYZZ7GFA36JWPMAO X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Implements moving both backup groups and namespaces within a datastore. This also adds namespace locking, this allows moves to happen within a datastore that is otherwise in-use. # Namespace locking To make move_group and move_namespace safe against concurrent operations, a namespace-level locking scheme is introduced in the first patch. Locks are acquired on the directory itself at: /run/proxmox-backup/locks// Two lock modes are used: - Shared: held by operations that read from or write into a namespace (backup, pull/push sync, verify per snapshot, prune per group, move_group on both source and target ns). Acquiring a shared lock on ns also acquires shared locks on all non-root ancestors, so an exclusive lock on any ancestor blocks all active operations below it. Uses a 2 second timeout so that worker tasks briefly wait for a concurrent move to finish rather than immediately skipping. - Exclusive: held by operations that structurally modify a namespace (move_namespace, delete_namespace). Non-blocking (timeout=0). Acquiring an exclusive lock on ns also acquires shared locks on all non-root ancestors, mirroring the shared variant so that two concurrent structural operations on related namespaces contend correctly. Locking up the ancestor chain rather than down the subtree keeps the cost O(depth), bounded by MAX_NAMESPACE_DEPTH (8). Verify and prune skip gracefully when a namespace lock cannot be acquired, since a concurrent move is a transient condition. create_locked_backup_group now returns (owner, ns_guard, group_guard), all callers updated. # Moving ## Groups 1. lock source ns (shared), lock target ns (shared), lock group (exclusive) 2. create target type directory 3. FS: rename group directory S3: copy objects, rename cache, delete source objects ## Namespace 1. lock source ns (exclusive), lock target ns (exclusive) 2. FS: rename namespace directory S3: - create target ns dirs and markers, both on S3 and local cache - move groups one by one, if copying fails the group stays at the source and can be moved afterwards manually - clean up empty source ns dirs v5, thanks @Chris!: - lock dir instead of `.ns-lock` file - explicitly drop ns lock guards in specific order - improve cleanup of partially failed s3 moves, we now create the local empty dir+owner file before we start copying s3 objects, if any of the s3 ops fail, the dir stays behind and can be deleted through the UI(which also triggers a prefix cleanup on the s3 storage) - update parameters for `DataStore::lookup_datastore()` - ui: re-ordered actions, `move` now next to `verify` - ui: add move to right-click context menu - ui: show empty groups in the UI - add cli commands for both ns and group moves - add 2s ns lock timeout for worker tasks *note*: given the UI change to show empty groups it could make sense to not auto-delete a group if the last snapshot is deleted. For this series though that is not relevant since we just need empty groups to be deletable through the UI for partially failed s3 moves Hannes Laimer (9): ui: show empty groups datastore: add namespace-level locking datastore: add move_group datastore: add move_namespace api: add PUT endpoint for move_group api: add PUT endpoint for move_namespace ui: add move group action ui: add move namespace action cli: add move-namespace and move-group commands pbs-datastore/src/backup_info.rs | 261 ++++++++++++++- pbs-datastore/src/datastore.rs | 345 +++++++++++++++++++- src/api2/admin/datastore.rs | 78 ++++- src/api2/admin/namespace.rs | 86 ++++- src/api2/backup/environment.rs | 4 + src/api2/backup/mod.rs | 14 +- src/api2/tape/restore.rs | 9 +- src/backup/verify.rs | 19 +- src/bin/proxmox_backup_manager/datastore.rs | 85 ++++- src/server/prune_job.rs | 11 + src/server/pull.rs | 8 +- src/server/push.rs | 6 + www/Makefile | 2 + www/datastore/Content.js | 134 +++++++- www/form/NamespaceSelector.js | 11 + www/window/GroupMove.js | 56 ++++ www/window/NamespaceMove.js | 79 +++++ 17 files changed, 1166 insertions(+), 42 deletions(-) create mode 100644 www/window/GroupMove.js create mode 100644 www/window/NamespaceMove.js -- 2.47.3