From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 416DD1FF137 for ; Tue, 31 Mar 2026 14:34:33 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B9C3F1C3BE; Tue, 31 Mar 2026 14:35:00 +0200 (CEST) Message-ID: <4a15ffe7-76cd-4a5b-aab0-db08b408ff69@proxmox.com> Date: Tue, 31 Mar 2026 14:34:56 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: superseded: [PATCH proxmox-backup v5 0/9] fixes #6195: add support for moving groups and namespaces To: pbs-devel@lists.proxmox.com References: <20260319161325.206846-1-h.laimer@proxmox.com> Content-Language: en-US From: Hannes Laimer In-Reply-To: <20260319161325.206846-1-h.laimer@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1774960440997 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.084 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 7HLMIRHTWTLQJR4QZH4CQFAXDJUORR3J X-Message-ID-Hash: 7HLMIRHTWTLQJR4QZH4CQFAXDJUORR3J X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: superseded-by: https://lore.proxmox.com/pbs-devel/20260331123409.198353-1-h.laimer@proxmox.com/T/#t On 2026-03-19 17:12, Hannes Laimer wrote: > Implements moving both backup groups and namespaces within a datastore. > This also adds namespace locking, this allows moves to happen within a > datastore that is otherwise in-use. > > > # Namespace locking > > To make move_group and move_namespace safe against concurrent > operations, a namespace-level locking scheme is introduced in the first > patch. > > Locks are acquired on the directory itself at: > /run/proxmox-backup/locks// > > Two lock modes are used: > > - Shared: held by operations that read from or write into a namespace > (backup, pull/push sync, verify per snapshot, prune per group, > move_group on both source and target ns). > Acquiring a shared lock on ns also acquires shared locks on all > non-root ancestors, so an exclusive lock on any ancestor blocks > all active operations below it. > Uses a 2 second timeout so that worker tasks briefly wait for a > concurrent move to finish rather than immediately skipping. > > - Exclusive: held by operations that structurally modify a namespace > (move_namespace, delete_namespace). Non-blocking (timeout=0). > Acquiring an exclusive lock on ns also acquires shared locks on all > non-root ancestors, mirroring the shared variant so that two > concurrent structural operations on related namespaces contend > correctly. > > Locking up the ancestor chain rather than down the subtree keeps the > cost O(depth), bounded by MAX_NAMESPACE_DEPTH (8). > > Verify and prune skip gracefully when a namespace lock cannot be > acquired, since a concurrent move is a transient condition. > create_locked_backup_group now returns (owner, ns_guard, group_guard), > all callers updated. > > # Moving > > ## Groups > 1. lock source ns (shared), lock target ns (shared), lock group (exclusive) > 2. create target type directory > 3. FS: rename group directory > S3: copy objects, rename cache, delete source objects > > ## Namespace > 1. lock source ns (exclusive), lock target ns (exclusive) > 2. FS: rename namespace directory > S3: - create target ns dirs and markers, both on S3 and local cache > - move groups one by one, if copying fails the group stays at > the source and can be moved afterwards manually > - clean up empty source ns dirs > > > v5, thanks @Chris!: > - lock dir instead of `.ns-lock` file > - explicitly drop ns lock guards in specific order > - improve cleanup of partially failed s3 moves, we now create the local > empty dir+owner file before we start copying s3 objects, if any of > the s3 ops fail, the dir stays behind and can be deleted through the > UI(which also triggers a prefix cleanup on the s3 storage) > - update parameters for `DataStore::lookup_datastore()` > - ui: re-ordered actions, `move` now next to `verify` > - ui: add move to right-click context menu > - ui: show empty groups in the UI > - add cli commands for both ns and group moves > - add 2s ns lock timeout for worker tasks > > *note*: given the UI change to show empty groups it could make sense to > not auto-delete a group if the last snapshot is deleted. For this series > though that is not relevant since we just need empty groups to be > deletable through the UI for partially failed s3 moves > > > > Hannes Laimer (9): > ui: show empty groups > datastore: add namespace-level locking > datastore: add move_group > datastore: add move_namespace > api: add PUT endpoint for move_group > api: add PUT endpoint for move_namespace > ui: add move group action > ui: add move namespace action > cli: add move-namespace and move-group commands > > pbs-datastore/src/backup_info.rs | 261 ++++++++++++++- > pbs-datastore/src/datastore.rs | 345 +++++++++++++++++++- > src/api2/admin/datastore.rs | 78 ++++- > src/api2/admin/namespace.rs | 86 ++++- > src/api2/backup/environment.rs | 4 + > src/api2/backup/mod.rs | 14 +- > src/api2/tape/restore.rs | 9 +- > src/backup/verify.rs | 19 +- > src/bin/proxmox_backup_manager/datastore.rs | 85 ++++- > src/server/prune_job.rs | 11 + > src/server/pull.rs | 8 +- > src/server/push.rs | 6 + > www/Makefile | 2 + > www/datastore/Content.js | 134 +++++++- > www/form/NamespaceSelector.js | 11 + > www/window/GroupMove.js | 56 ++++ > www/window/NamespaceMove.js | 79 +++++ > 17 files changed, 1166 insertions(+), 42 deletions(-) > create mode 100644 www/window/GroupMove.js > create mode 100644 www/window/NamespaceMove.js >