[PATCH proxmox-backup v8 00/13] fixes #6195: add support for moving groups and namespaces

[Hannes Laimer <h.laimer@proxmox.com>]

Add support for moving backup groups and entire namespace subtrees to
a different location within the same datastore.

Groups are moved with exclusive per-group and per-snapshot locking.
For S3, objects are copied to the target prefix before deleting the
source. Namespace moves process groups individually, deferring and
retrying lock conflicts once, so partially failed moves can be
completed with move-group.


v8, thanks @Chris and @Fabian:
- shared check_move_group/check_move_namespace
- have BackupGroup::destroy consume the group lock so move can hand over
  cleanup instead of duplicating it
- split remove_namespace into flat and recursive variants, and lift the
  empty-type-dir cleanup so the flat path prunes after move
- lift check_namespace_depth_limit into pbs-datastore
- move snapshot-move to BackupDir::move_to with typed target and
  same-store assertion
- check_merge_invariants: take the target group directly, stream
  min/first-overlap instead of collecting, include both namespaces in
  the error
- reuse create_locked_backup_group (new non-pub variant) for the target
  group, differentiating soft/hard errors
- rename MoveGroupError -> BackupGroupOpError
- move_notes_to: S3 upload first, local rename second, with upload
  failures propagated as fatal instead of warn+continue
- source-ns cleanup: distinguish depth-limited (silent) from concurrent
  (warn, named) children
- iter_backup_groups errors recorded per-ns instead of aborting
- running moved-group counter in the task log, final summary splits
  fatal from lock(rerun may succeed)
- docs: fix merge-groups default, document permission asymmetry, note
  stacked destructive defaults
- cli: --ns, --target-ns completion
- new patch 5/13, thanks @Chris!: fix data-loss race between
  move-group/move-namespace and a concurrent GC phase 1 with a
  per-datastore write-ahead journal at
  /run/proxmox-backup/locks/<datastore>/move-journal

v7, thanks @Fabian and @Arthur!:
 - allow moving into existing target namespaces
 - merge groups with the same name when owner matches and source
   snapshots are strictly newer than target snapshots
 - lock and move snapshots in batches(512) to avoid FD exhaustion
 - assert new-style locking upfront
 - pre-create target group dir with source owner
 - use remove_dir for source cleanup, log errors instead of swallowing
 - rename 'new-ns' to 'target-ns', use dedicated POST endpoints
   /move-group and /move-namespace
 - relax move-group permissions: DATASTORE_PRUNE + ownership on
   source, DATASTORE_BACKUP + ownership on target
 - add max-depth, delete-source, and merge-group/merge-groups flags
 - add docs section and online help link for move dialogs
 - make ns and target-ns required in CLI, no need to keep the
   empty -> root thing since this is new

v6, thanks @Fabian and @Dominik!:
 - drop ns locks, lock everything directly, like we do for delete
 - ui: disable prune for empty groups
 - ui: dont render verification status for empty groups

v5, thanks @Chris!:
 - lock dir instead of .ns-lock file
 - improve cleanup of partially failed s3 moves
 - ui: show empty groups, re-order actions, add context menu
 - add cli commands for both ns and group moves



Hannes Laimer (13):
  ui: show empty groups
  datastore: lift check_namespace_depth_limit to pbs-datastore
  datastore: have BackupGroup::destroy consume the group lock
  datastore: split remove_namespace into flat and recursive variants
  datastore: add move journal for coordinating with gc phase 1
  datastore: add move-group
  datastore: add move-namespace
  docs: add section on moving namespaces and groups
  api: add POST endpoint for move-group
  api: add POST endpoint for move-namespace
  ui: add move group action
  ui: add move namespace action
  cli: add move-namespace and move-group commands

 docs/storage.rst                            |  60 +-
 pbs-datastore/src/backup_info.rs            | 269 ++++++-
 pbs-datastore/src/datastore.rs              | 732 ++++++++++++++++++--
 pbs-datastore/src/lib.rs                    |   7 +-
 pbs-datastore/src/move_journal.rs           | 149 ++++
 src/api2/admin/datastore.rs                 |  94 +++
 src/api2/admin/namespace.rs                 |  85 ++-
 src/bin/proxmox_backup_manager/datastore.rs | 113 ++-
 src/server/pull.rs                          |  16 +-
 src/server/push.rs                          |   7 +-
 src/server/sync.rs                          |  21 -
 www/Makefile                                |   2 +
 www/OnlineHelpInfo.js                       |   4 +
 www/datastore/Content.js                    | 167 ++++-
 www/form/NamespaceSelector.js               |  11 +
 www/window/GroupMove.js                     |  71 ++
 www/window/NamespaceMove.js                 | 136 ++++
 17 files changed, 1792 insertions(+), 152 deletions(-)
 create mode 100644 pbs-datastore/src/move_journal.rs
 create mode 100644 www/window/GroupMove.js
 create mode 100644 www/window/NamespaceMove.js

-- 
2.47.3