[pbs-devel] [PATCH proxmox-backup v14 00/26] add removable datastores

[Hannes Laimer <h.laimer@proxmox.com>]

These patches add support for removable datastores. All removable
datastores have a backing-device(a UUID) associated with them. Removable
datastores work like normal ones, just that they can be unplugged. It is
possible to create a removable datastore, sync backups onto it, unplug
it and use it on a different PBS.

The datastore path specified is relative to the root of the used device.
Removable datastores are bind mounted to /mnt/datastore/<NAME>.
Multiple datastores can be created on a single device, but only device with 
a single datastore on them will be auto-mounted.

When a removable datastore is deleted and 'destroy-data' is set, the
device has to be mounted. If 'destroy-data' is not set the datastore
can be deleted even if the device is not present. Removable datastores
are automatically mounted when plugged in. 

 v14: thanks @Fabian and @Maximiliano
 * add two functions to get mount status, or ensure mounted for rm.ds.,
    avoiding repeating things
 * use enum for `mount_status` instead of Option<bool>
 * fix problme with unmounting, now check for unmounting maintenance mode before actually unmounting,
    manually chaning the maintenance mode during unmounting will not prevent unmounting
 * improve logging for mounting: add context and adjust logging levels
 * improve uuid_mount function: load config file directly and call do_mount function directly without 
    going through the API
 * add logging for cleanup on ds deletion
 * move check for nesting into do_create_datastore, and check fir all datastore(not just removable)
 * remove redundant check when deleting through directory
 * use single worker when creating removable datastore through dir endpoint
 * drop get_mount_point function
 * ui: stop loading status after first failed attempt, prevents logs spamming

v13: thanks @Fabian
 * allow multiple datastore on devices
 * replace `is_datastore_available` by a more specific function, it is now
    removable datastore specific and won't be called for normal ones
 * replace removable/is_available in status structs with mount_state,
    which is `None` for normal datastore as it makes it
    less ambiguous what is meant  
 * remove notion of 'available' from normal datastores and replace it with
    mounted/mount_status for removable ones, as it never really made sense 
    in the first place
 * abort of an unmount task will now reset the maintanance mode
 * add check for race when setting maintenance at end of unmounting task
 * improve documentation and commit messages
 * remove not needed tokio::spawn
 * only auto mount devices with single datastore on them
 * drop ptach that added flag for excluding used partitions
 * make auto mount service not dynamic
 * add debug command to scan devices for datastores they may contain
 * rebase onto master

v12: thanks @Wolfgang
 * use bind mounts, so now
    <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
    this is a bit cleaner and allows for multiple datastores
    on a single device to be mounted individually, if we
    want to allow that in the future
 * small code improvements


v11:
 * rebase onto master

v10: thanks @Gabriel and @Wolfgang
 * make is_datastore_available more robust
 * fix a lot of wording
 * drop format on uuid_mount command for UUID
 * only gather_disk_stats if datastore is available
 * overall code improvements
 * ui: include model in partition selector
 * rebased onto master

v9:
 * change mount point to `/mnt/datastore/<name>`
 * update "Directory" list UI
 * add `absolute_path()` from Dietmar's RFC
 * update docs

v8:
 * still depends on [1]
 * paths for removable datastores are now relative to 
    `/mnt/removable_datastore/<UUID>`
 * add support for creation of removable datastore through the 
    "create directory" endpoint (last 3 patches)
 * update datastore creation UI
 * update docs

v7:
 * depends on [1]
 * improve logging when waiting for tasks
 * drop `update-datatore-cache` refactoring
 * fix some commit messages

[1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html

v6:
 * remove 'drop' flag in datastore cache
 * use maintenance-mode 'unmount' for unmounting process, only for the
    unmounting not for being unmounted
 * rename/simplify update-datastore-cache command
 * ui: integrate new unmounting maintenance mode
 * basically a mix of v3 and v4

v5: thanks @Dietmar and @Christian
 * drop --force for unmount since it'll always fail if tasks are still running, and if
    there are not normal unount will work
 * improve several commit messages
 * improve error message wording
 * add removable datastore section to docs
 * add documentation for is_datastore_available

v4: thanks a lot @Dietmar and @Christian
 * make check if mounted wayyy faster
 * don't keep track of mounting state
 * drop Unplugged maintenance mode
 * use UUID_FORMAT for uuid field
 * a lot of small things, like use of bail!, inline format!, ...
 * include improvement to cache handling

v3:
 * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
 * fix CLI (un)mount command, thanks @Gabriel
 * add removable datastore CLI autocomplete helper
 * rebase onto master
 * move ui patches to the end

thanks @Lukas and @Thomas for the feedback
v2:
 * fix datastore 'add' button in the UI
 * some format!("{}", a) -> format!("{a}")
 * replace `const` with `let` in js code
 * change icon `fa-usb` -> `fa-plug`
 * add some docs
 * add JDoc for parseMaintenanceMode
 * proxmox-schema dep bump

Dietmar Maurer (1):
  maintenance: add 'Unmount' maintenance type

Hannes Laimer (25):
  pbs-api-types: add backing-device to DataStoreConfig
  maintenance: make is_offline more generic
  datastore: add helper for checking if a datastore is mounted
  api: admin: add (un)mount endpoint for removable datastores
  api: removable datastore creation
  api: add check for nested datastores on creation
  pbs-api-types: add mount_status field to DataStoreListItem
  bin: manager: add (un)mount command
  add auto-mounting for removable datastores
  datastore: handle deletion of removable datastore properly
  docs: add removable datastores section
  ui: add partition selector form
  ui: add removable datastore creation support
  ui: add (un)mount button to summary
  ui: tree: render unmounted datastores correctly
  ui: utils: make parseMaintenanceMode more robust
  ui: add datastore status mask for unmounted removable datastores
  ui: maintenance: fix disable msg field if no type is selected
  ui: render 'unmount' maintenance mode correctly
  api: node: allow creation of removable datastore through directory
    endpoint
  api: node: include removable datastores in directory list
  node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
  ui: support create removable datastore through directory creation
  bin: debug: add inspect device command
  api: disks: only return UUID of partitions if it actually is one

 debian/proxmox-backup-server.install        |   1 +
 debian/proxmox-backup-server.udev           |   3 +
 docs/storage.rst                            |  38 +++
 etc/Makefile                                |   1 +
 etc/removable-device-attach@.service        |   8 +
 pbs-api-types/src/datastore.rs              |  47 ++-
 pbs-api-types/src/maintenance.rs            |  12 +-
 pbs-config/src/datastore.rs                 |  14 +
 pbs-datastore/src/datastore.rs              |  83 +++++-
 pbs-datastore/src/lib.rs                    |   4 +-
 src/api2/admin/datastore.rs                 | 310 +++++++++++++++++++-
 src/api2/config/datastore.rs                | 107 ++++++-
 src/api2/node/disks/directory.rs            |  74 +++--
 src/api2/status/mod.rs                      |  30 +-
 src/bin/proxmox_backup_debug/inspect.rs     | 149 ++++++++++
 src/bin/proxmox_backup_manager/datastore.rs | 127 +++++++-
 src/server/metric_collection/mod.rs         |   4 +
 src/tools/disks/mod.rs                      |   5 +-
 www/DirectoryList.js                        |  13 +
 www/Makefile                                |   1 +
 www/NavigationTree.js                       |  17 +-
 www/Utils.js                                |  33 ++-
 www/css/ext6-pbs.css                        |  20 ++
 www/datastore/DataStoreListSummary.js       |   1 +
 www/datastore/Summary.js                    | 115 +++++++-
 www/form/PartitionSelector.js               |  81 +++++
 www/window/CreateDirectory.js               |  14 +
 www/window/DataStoreEdit.js                 |  37 +++
 www/window/MaintenanceOptions.js            |  17 +-
 29 files changed, 1274 insertions(+), 92 deletions(-)
 create mode 100644 etc/removable-device-attach@.service
 create mode 100644 www/form/PartitionSelector.js

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel