From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 608111FF13B for ; Wed, 25 Mar 2026 09:55:41 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D737FC439; Wed, 25 Mar 2026 09:56:01 +0100 (CET) Date: Wed, 25 Mar 2026 09:55:53 +0100 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [PATCH proxmox-backup v5 4/9] datastore: add move_namespace To: Hannes Laimer , pbs-devel@lists.proxmox.com References: <20260319161325.206846-1-h.laimer@proxmox.com> <20260319161325.206846-5-h.laimer@proxmox.com> <1774356676.j651zha210.astroid@yuna.none> In-Reply-To: MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1774427682.fqtrys91x3.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1774428908754 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.053 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: H54KU2CVTTARMT53GFUAVBFZ6WJCQ72U X-Message-ID-Hash: H54KU2CVTTARMT53GFUAVBFZ6WJCQ72U X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On March 25, 2026 7:32 am, Hannes Laimer wrote: > On 2026-03-24 13:59, Fabian Gr=C3=BCnbichler wrote: >> On March 19, 2026 5:13 pm, Hannes Laimer wrote: >>> move_namespace relocates an entire namespace subtree (the given >>> namespace, all child namespaces, and their groups) to a new location >>> within the same datastore. >>> >>> For the filesystem backend the entire subtree is relocated with a singl= e >>> atomic rename. For the S3 backend groups are moved one at a time via >>> BackupGroup::move_to(). Groups that fail are left at the source and >>> listed as an error in the task log so they can be retried with >>> move_group individually. Source namespaces where all groups succeeded >>> have their S3 markers and local cache directories removed, >>> deepest-first. >>> >>> Signed-off-by: Hannes Laimer >>> --- >>> pbs-datastore/src/datastore.rs | 249 ++++++++++++++++++++++++++++++++- >>> 1 file changed, 248 insertions(+), 1 deletion(-) >>> >>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datasto= re.rs >>> index d4c88452..3187cbc3 100644 >>> --- a/pbs-datastore/src/datastore.rs >>> +++ b/pbs-datastore/src/datastore.rs >>> @@ -31,7 +31,7 @@ use pbs_api_types::{ >>> ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, Back= upType, ChunkOrder, >>> DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType, Dat= astoreFSyncLevel, >>> DatastoreTuning, GarbageCollectionCacheStats, GarbageCollectionSta= tus, MaintenanceMode, >>> - MaintenanceType, Operation, UPID, >>> + MaintenanceType, Operation, MAX_NAMESPACE_DEPTH, UPID, >>> }; >>> use pbs_config::s3::S3_CFG_TYPE_ID; >>> use pbs_config::{BackupLockGuard, ConfigVersionCache}; >>> @@ -1003,6 +1003,253 @@ impl DataStore { >>> Ok((removed_all_requested, stats)) >>> } >>> =20 >>> + /// Move a backup namespace (including all child namespaces and gr= oups) to a new location. >>> + /// >>> + /// The entire subtree rooted at `source_ns` is relocated to `targ= et_ns`. Exclusive namespace >>> + /// locks are held on both source and target namespaces for the du= ration to block concurrent >>> + /// readers and writers. >>> + /// >>> + /// For the filesystem backend the rename is atomic. For the S3 ba= ckend groups are moved >>> + /// one at a time. A group that fails to copy is left at the sourc= e and can be moved >>> + /// individually with `move_group`. The operation returns an error= listing any such groups. >>=20 >> I think we want these semantics in both cases, since a move is only >> atomic if you ignore a lot of the invariants you need to violate to make >> it so, even in the filesystem case. and if we have such semantics, and >> handle the invariants correctly, we only need the namespace locks if we >> basically want to make the namespace lock a "per-NS-maintenance-mode" in >> disguise.. which I don't think is what users actually want - they want >> an option to bulk move groups from A to B without interrupting the >> world.. >>=20 >=20 > yes, i see that use-case. but isn't a group lock also just a > per-group-maintenance-mode :P in a way. my point was the scope. locking a single group (and its snapshot) at a time is much less invasive than locking a whole namespace. the current approach pretty much requires(!) stopping all readers and writers (which includes things like verification, prune job, GC, ..) before I can move a namespace, and the move can then take arbitrarily long with S3. this is rather cumbersome in practice, because the person doing the moving and the person(s) controlling the rest of the tasks are often not the same. that's why I noted the parallelism to maintenance mode - taking an exclusive lock on a namespace forces a lot of unrelated actions to be blocked, just like maintance mode does. taking an exclusive lock on a group just prevents deleting it or adding a new snapshot, it does not prevent verifying it, for example. the other approach (recursively locking groups and moving them one-by-one) has downsides: - if a reader/writer conflicts with the move, that group either needs to be copied or skipped, and will remain in the source namespace - it's more work in the file system case - if aborted half-way through for whatever reason, you have a partial move all of those points are already somewhat true with S3 anyway. the big upside is that by reducing the granularity of the "unit of work", the amount of conflicts goes down as well. so it meshes better with how we handle other operations. this is particularly important if you think of bulk moving of *contents*, instead of moving of namespaces themselves. and I think that is an important use case, in particular considering future extensions: - move groups into an "archive" namespace based on certain criteria (no backups in the past N weeks?) - move snapshots older than N into an archive namespace (creating groups if needed) the current move namespace feature here is more akin to a rename namespace feature, which is something fundamentally different than moving groups. like I said, we can still have it, but it might be better coupled with a special maintenance mode (which is an even bigger hammer, but one we already have that doesn't otherwise affect the regular hot path). > you're point makes sense, i think the "extra" locking it introduces > everywhere is the best reason against ns locks. as for scoping, i think > locking a whole ns before moving that ns does make sense though. >=20 > there may be leftovers regardless of locks in case of s3 backends. but > these would then have to be moved manually, and if that is more than a > few it gets cumbersome rather fast. what we could is is just lock > everything, then move.. the same is true when deleting a namespace though - it is best effort, and if something remains at the end you need to retry or live with it ;) I don't think locking all groups and their snapshots at the same time is an option, but I also don't think partially moving is that bad. we could even collect groups which have conflicting tasks (failure to obtain exclusive lock), and then retry them at the end? one question is what to do with groups where a snapshot in the middle is not lockable - do we want to move everything up to that point, and leave the rest? or do we want to lock all snapshots up-front, and only move the group wholesale if possible, and not at all if not? when destroying a group, we remove snapshots until the first one fails, so there is some precedent for doing it one-by-one at that level as well.. > either way thanks for the feedback! > not opposed to dropping ns locks, you do have very good points :) >=20 >> I hope my comments on the other patches make it clear what I mean - I am >> open for arguments for the current approach with namespace locks, but at >> the moment I am not yet convinced this is the better approach.. >>=20 >> skipping the rest of the implementation below (for now) >>=20 >>> + /// >>> + /// Fails if: >>> + /// - `source_ns` is the root namespace >>> + /// - `source_ns` =3D=3D `target_ns` >>> + /// - `source_ns` does not exist >>> + /// - `target_ns` already exists (to prevent silent merging) >>> + /// - `target_ns`'s parent does not exist >>> + /// - `source_ns` is an ancestor of `target_ns` >>> + /// - the move would exceed the maximum namespace depth >>> + pub fn move_namespace( >>=20 >> other than noting that this particular fn is *very* long ;) >>=20 >>> [..] >=20 >=20