From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 7BFEB1FF13F for ; Thu, 23 Apr 2026 18:30:09 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 56E591F7BE; Thu, 23 Apr 2026 18:30:09 +0200 (CEST) Content-Type: text/plain; charset=UTF-8 Date: Thu, 23 Apr 2026 18:29:34 +0200 Message-Id: Subject: Re: [PATCH proxmox-backup v8 00/13] fixes #6195: add support for moving groups and namespaces From: =?utf-8?q?Michael_K=C3=B6ppl?= To: "Hannes Laimer" , Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 X-Mailer: aerc 0.21.0 References: <20260422133951.192862-1-h.laimer@proxmox.com> In-Reply-To: <20260422133951.192862-1-h.laimer@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776961685075 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.097 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [lib.rs,pull.rs,namespace.rs,sync.rs,datastore.rs,push.rs] Message-ID-Hash: KS42BE6UVZSIKXM2EGQJOVGMDLASC5KS X-Message-ID-Hash: KS42BE6UVZSIKXM2EGQJOVGMDLASC5KS X-MailFrom: m.koeppl@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Had a closer look at this series. Tested the following: - generally moving group as well as namespaces in various ways - checked that the merge invariants hold by explicitly creating groups where snapshot times overlap and that moving is not possible if ownership does not match - checked with an s3-backed datastore that moving notes can be repeated in case any of the steps fail (made the rename of the notes file fail by making the file immutable) and also that the source stays intact - ran moves concurrently to see if I'd run into problems w.r.t. where the group is moved or if any data is lost Noticed the following: - The values of unchecked checkboxes aren't sent to the backend, resulting in them being interpreted as set to true (the default). Left comments on the affected patches. Using proxmox-backup-manager like this: `proxmox-backup-manager datastore move-namespace --ns ns2 --target-ns ns1` generally to cause problems. There's nothing really telling me that anything didn't work in its output, but doing this resulted in the following problems for me: - When moving a namespace with groups in it as described above, I'm afterwards unable to delete the moved group (in its new namespace) or the new namespace containing it because the removal of the snapshot fails because the snapshot lock seemingly cannot be acquired. - Given namespaces ns1, ns2, ns2/a with a group in ns2/a, when using `proxmox-backup-manager datastore move-namespace --ns ns2 --target-ns ns1` the resulting ns1/a namespace did not show the group in ns1/a for me until many seconds (~15-20s) later even though the log suggests everything is done: 2026-04-23T17:52:50+02:00: moving namespace 'ns2' -> 'ns1': 2 namespace= s 2026-04-23T17:52:50+02:00: moving group 'ct/104' from 'ns2/a' to 'ns1/a= ' 2026-04-23T17:52:50+02:00: removing backup group "/datastore/test/ns/ns= 2/ns/a/ct/104" 2026-04-23T17:52:50+02:00: moved group 1 ('ct/104' in 'ns2/a') 2026-04-23T17:52:50+02:00: TASK OK Also tested this with `--ns ns2 --target-ns ns1/ns2` and it happens there as well. - It's possible to use namespaces that don't exist yet for --target-ns. I suppose it's on purpose, but I could imagine that it's frustrating if users want to move a namespace, have a typo in their --target-ns and then the content of the namespace is in different namespace than they intended. Wanted to note that since the UI also doesn't allow using namespaces that don't exist. On Wed Apr 22, 2026 at 3:39 PM CEST, Hannes Laimer wrote: > Add support for moving backup groups and entire namespace subtrees to > a different location within the same datastore. > > Groups are moved with exclusive per-group and per-snapshot locking. > For S3, objects are copied to the target prefix before deleting the > source. Namespace moves process groups individually, deferring and > retrying lock conflicts once, so partially failed moves can be > completed with move-group. > > > v8, thanks @Chris and @Fabian: > - shared check_move_group/check_move_namespace > - have BackupGroup::destroy consume the group lock so move can hand over > cleanup instead of duplicating it > - split remove_namespace into flat and recursive variants, and lift the > empty-type-dir cleanup so the flat path prunes after move > - lift check_namespace_depth_limit into pbs-datastore > - move snapshot-move to BackupDir::move_to with typed target and > same-store assertion > - check_merge_invariants: take the target group directly, stream > min/first-overlap instead of collecting, include both namespaces in > the error > - reuse create_locked_backup_group (new non-pub variant) for the target > group, differentiating soft/hard errors > - rename MoveGroupError -> BackupGroupOpError > - move_notes_to: S3 upload first, local rename second, with upload > failures propagated as fatal instead of warn+continue > - source-ns cleanup: distinguish depth-limited (silent) from concurrent > (warn, named) children > - iter_backup_groups errors recorded per-ns instead of aborting > - running moved-group counter in the task log, final summary splits > fatal from lock(rerun may succeed) > - docs: fix merge-groups default, document permission asymmetry, note > stacked destructive defaults > - cli: --ns, --target-ns completion > - new patch 5/13, thanks @Chris!: fix data-loss race between > move-group/move-namespace and a concurrent GC phase 1 with a > per-datastore write-ahead journal at > /run/proxmox-backup/locks//move-journal > > v7, thanks @Fabian and @Arthur!: > - allow moving into existing target namespaces > - merge groups with the same name when owner matches and source > snapshots are strictly newer than target snapshots > - lock and move snapshots in batches(512) to avoid FD exhaustion > - assert new-style locking upfront > - pre-create target group dir with source owner > - use remove_dir for source cleanup, log errors instead of swallowing > - rename 'new-ns' to 'target-ns', use dedicated POST endpoints > /move-group and /move-namespace > - relax move-group permissions: DATASTORE_PRUNE + ownership on > source, DATASTORE_BACKUP + ownership on target > - add max-depth, delete-source, and merge-group/merge-groups flags > - add docs section and online help link for move dialogs > - make ns and target-ns required in CLI, no need to keep the > empty -> root thing since this is new > > v6, thanks @Fabian and @Dominik!: > - drop ns locks, lock everything directly, like we do for delete > - ui: disable prune for empty groups > - ui: dont render verification status for empty groups > > v5, thanks @Chris!: > - lock dir instead of .ns-lock file > - improve cleanup of partially failed s3 moves > - ui: show empty groups, re-order actions, add context menu > - add cli commands for both ns and group moves > > > > Hannes Laimer (13): > ui: show empty groups > datastore: lift check_namespace_depth_limit to pbs-datastore > datastore: have BackupGroup::destroy consume the group lock > datastore: split remove_namespace into flat and recursive variants > datastore: add move journal for coordinating with gc phase 1 > datastore: add move-group > datastore: add move-namespace > docs: add section on moving namespaces and groups > api: add POST endpoint for move-group > api: add POST endpoint for move-namespace > ui: add move group action > ui: add move namespace action > cli: add move-namespace and move-group commands > > docs/storage.rst | 60 +- > pbs-datastore/src/backup_info.rs | 269 ++++++- > pbs-datastore/src/datastore.rs | 732 ++++++++++++++++++-- > pbs-datastore/src/lib.rs | 7 +- > pbs-datastore/src/move_journal.rs | 149 ++++ > src/api2/admin/datastore.rs | 94 +++ > src/api2/admin/namespace.rs | 85 ++- > src/bin/proxmox_backup_manager/datastore.rs | 113 ++- > src/server/pull.rs | 16 +- > src/server/push.rs | 7 +- > src/server/sync.rs | 21 - > www/Makefile | 2 + > www/OnlineHelpInfo.js | 4 + > www/datastore/Content.js | 167 ++++- > www/form/NamespaceSelector.js | 11 + > www/window/GroupMove.js | 71 ++ > www/window/NamespaceMove.js | 136 ++++ > 17 files changed, 1792 insertions(+), 152 deletions(-) > create mode 100644 pbs-datastore/src/move_journal.rs > create mode 100644 www/window/GroupMove.js > create mode 100644 www/window/NamespaceMove.js