Re: [PATCH proxmox-backup v8 00/13] fixes #6195: add support for moving groups and namespaces

["Michael Köppl" <m.koeppl@proxmox.com>]

Had a closer look at this series. Tested the following:

- generally moving group as well as namespaces in various ways
- checked that the merge invariants hold by explicitly creating groups
  where snapshot times overlap and that moving is not possible if
  ownership does not match
- checked with an s3-backed datastore that moving notes can be repeated
  in case any of the steps fail (made the rename of the notes file fail
  by making the file immutable) and also that the source stays intact
- ran moves concurrently to see if I'd run into problems w.r.t. where
  the group is moved or if any data is lost

Noticed the following:
- The values of unchecked checkboxes aren't sent to the backend,
  resulting in them being interpreted as set to true (the default).
  Left comments on the affected patches.

Using proxmox-backup-manager like this:
  `proxmox-backup-manager datastore move-namespace
    <datastore> --ns ns2 --target-ns ns1`
generally to cause problems. There's nothing really telling me that
anything didn't work in its output, but doing this resulted in the
following problems for me:
- When moving a namespace with groups in it as described above, I'm
  afterwards unable to delete the moved group (in its new namespace) or
  the new namespace containing it because the removal of the snapshot
  fails because the snapshot lock seemingly cannot be acquired.
- Given namespaces ns1, ns2, ns2/a with a group in ns2/a, when using
  `proxmox-backup-manager datastore move-namespace
    <datastore> --ns ns2 --target-ns ns1`
  the resulting ns1/a namespace did not show the group in ns1/a for me
  until many seconds (~15-20s) later even though the log suggests
  everything is done:
    2026-04-23T17:52:50+02:00: moving namespace 'ns2' -> 'ns1': 2 namespaces
    2026-04-23T17:52:50+02:00: moving group 'ct/104' from 'ns2/a' to 'ns1/a'
    2026-04-23T17:52:50+02:00: removing backup group "/datastore/test/ns/ns2/ns/a/ct/104"
    2026-04-23T17:52:50+02:00: moved group 1 ('ct/104' in 'ns2/a')
    2026-04-23T17:52:50+02:00: TASK OK
  Also tested this with `--ns ns2 --target-ns ns1/ns2` and it happens
  there as well.
- It's possible to use namespaces that don't exist yet for --target-ns.
  I suppose it's on purpose, but I could imagine that it's frustrating
  if users want to move a namespace, have a typo in their --target-ns
  and then the content of the namespace is in different namespace than
  they intended. Wanted to note that since the UI also doesn't allow
  using namespaces that don't exist.

On Wed Apr 22, 2026 at 3:39 PM CEST, Hannes Laimer wrote:
> Add support for moving backup groups and entire namespace subtrees to
> a different location within the same datastore.
>
> Groups are moved with exclusive per-group and per-snapshot locking.
> For S3, objects are copied to the target prefix before deleting the
> source. Namespace moves process groups individually, deferring and
> retrying lock conflicts once, so partially failed moves can be
> completed with move-group.
>
>
> v8, thanks @Chris and @Fabian:
> - shared check_move_group/check_move_namespace
> - have BackupGroup::destroy consume the group lock so move can hand over
>   cleanup instead of duplicating it
> - split remove_namespace into flat and recursive variants, and lift the
>   empty-type-dir cleanup so the flat path prunes after move
> - lift check_namespace_depth_limit into pbs-datastore
> - move snapshot-move to BackupDir::move_to with typed target and
>   same-store assertion
> - check_merge_invariants: take the target group directly, stream
>   min/first-overlap instead of collecting, include both namespaces in
>   the error
> - reuse create_locked_backup_group (new non-pub variant) for the target
>   group, differentiating soft/hard errors
> - rename MoveGroupError -> BackupGroupOpError
> - move_notes_to: S3 upload first, local rename second, with upload
>   failures propagated as fatal instead of warn+continue
> - source-ns cleanup: distinguish depth-limited (silent) from concurrent
>   (warn, named) children
> - iter_backup_groups errors recorded per-ns instead of aborting
> - running moved-group counter in the task log, final summary splits
>   fatal from lock(rerun may succeed)
> - docs: fix merge-groups default, document permission asymmetry, note
>   stacked destructive defaults
> - cli: --ns, --target-ns completion
> - new patch 5/13, thanks @Chris!: fix data-loss race between
>   move-group/move-namespace and a concurrent GC phase 1 with a
>   per-datastore write-ahead journal at
>   /run/proxmox-backup/locks/<datastore>/move-journal
>
> v7, thanks @Fabian and @Arthur!:
>  - allow moving into existing target namespaces
>  - merge groups with the same name when owner matches and source
>    snapshots are strictly newer than target snapshots
>  - lock and move snapshots in batches(512) to avoid FD exhaustion
>  - assert new-style locking upfront
>  - pre-create target group dir with source owner
>  - use remove_dir for source cleanup, log errors instead of swallowing
>  - rename 'new-ns' to 'target-ns', use dedicated POST endpoints
>    /move-group and /move-namespace
>  - relax move-group permissions: DATASTORE_PRUNE + ownership on
>    source, DATASTORE_BACKUP + ownership on target
>  - add max-depth, delete-source, and merge-group/merge-groups flags
>  - add docs section and online help link for move dialogs
>  - make ns and target-ns required in CLI, no need to keep the
>    empty -> root thing since this is new
>
> v6, thanks @Fabian and @Dominik!:
>  - drop ns locks, lock everything directly, like we do for delete
>  - ui: disable prune for empty groups
>  - ui: dont render verification status for empty groups
>
> v5, thanks @Chris!:
>  - lock dir instead of .ns-lock file
>  - improve cleanup of partially failed s3 moves
>  - ui: show empty groups, re-order actions, add context menu
>  - add cli commands for both ns and group moves
>
>
>
> Hannes Laimer (13):
>   ui: show empty groups
>   datastore: lift check_namespace_depth_limit to pbs-datastore
>   datastore: have BackupGroup::destroy consume the group lock
>   datastore: split remove_namespace into flat and recursive variants
>   datastore: add move journal for coordinating with gc phase 1
>   datastore: add move-group
>   datastore: add move-namespace
>   docs: add section on moving namespaces and groups
>   api: add POST endpoint for move-group
>   api: add POST endpoint for move-namespace
>   ui: add move group action
>   ui: add move namespace action
>   cli: add move-namespace and move-group commands
>
>  docs/storage.rst                            |  60 +-
>  pbs-datastore/src/backup_info.rs            | 269 ++++++-
>  pbs-datastore/src/datastore.rs              | 732 ++++++++++++++++++--
>  pbs-datastore/src/lib.rs                    |   7 +-
>  pbs-datastore/src/move_journal.rs           | 149 ++++
>  src/api2/admin/datastore.rs                 |  94 +++
>  src/api2/admin/namespace.rs                 |  85 ++-
>  src/bin/proxmox_backup_manager/datastore.rs | 113 ++-
>  src/server/pull.rs                          |  16 +-
>  src/server/push.rs                          |   7 +-
>  src/server/sync.rs                          |  21 -
>  www/Makefile                                |   2 +
>  www/OnlineHelpInfo.js                       |   4 +
>  www/datastore/Content.js                    | 167 ++++-
>  www/form/NamespaceSelector.js               |  11 +
>  www/window/GroupMove.js                     |  71 ++
>  www/window/NamespaceMove.js                 | 136 ++++
>  17 files changed, 1792 insertions(+), 152 deletions(-)
>  create mode 100644 pbs-datastore/src/move_journal.rs
>  create mode 100644 www/window/GroupMove.js
>  create mode 100644 www/window/NamespaceMove.js