Re: [pbs-devel] [PATCH proxmox-backup v14 00/26] add removable datastores

[Hannes Laimer <h.laimer@proxmox.com>]

On 11/22/24 15:46, Hannes Laimer wrote:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
> 
> The datastore path specified is relative to the root of the used device.
> Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> Multiple datastores can be created on a single device, but only device with
> a single datastore on them will be auto-mounted.
> 
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in.
> 
>   v14: thanks @Fabian and @Maximiliano
>   * add two functions to get mount status, or ensure mounted for rm.ds.,
>      avoiding repeating things
>   * use enum for `mount_status` instead of Option<bool>
>   * fix problme with unmounting, now check for unmounting maintenance mode before actually unmounting,
>      manually chaning the maintenance mode during unmounting will not prevent unmounting
                                                                      ^ *now
>   * improve logging for mounting: add context and adjust logging levels
>   * improve uuid_mount function: load config file directly and call do_mount function directly without
>      going through the API
>   * add logging for cleanup on ds deletion
>   * move check for nesting into do_create_datastore, and check fir all datastore(not just removable)
>   * remove redundant check when deleting through directory
>   * use single worker when creating removable datastore through dir endpoint
>   * drop get_mount_point function
>   * ui: stop loading status after first failed attempt, prevents logs spamming
> 
> v13: thanks @Fabian
>   * allow multiple datastore on devices
>   * replace `is_datastore_available` by a more specific function, it is now
>      removable datastore specific and won't be called for normal ones
>   * replace removable/is_available in status structs with mount_state,
>      which is `None` for normal datastore as it makes it
>      less ambiguous what is meant
>   * remove notion of 'available' from normal datastores and replace it with
>      mounted/mount_status for removable ones, as it never really made sense
>      in the first place
>   * abort of an unmount task will now reset the maintanance mode
>   * add check for race when setting maintenance at end of unmounting task
>   * improve documentation and commit messages
>   * remove not needed tokio::spawn
>   * only auto mount devices with single datastore on them
>   * drop ptach that added flag for excluding used partitions
>   * make auto mount service not dynamic
>   * add debug command to scan devices for datastores they may contain
>   * rebase onto master
> 
> v12: thanks @Wolfgang
>   * use bind mounts, so now
>      <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
>      this is a bit cleaner and allows for multiple datastores
>      on a single device to be mounted individually, if we
>      want to allow that in the future
>   * small code improvements
> 
> 
> v11:
>   * rebase onto master
> 
> v10: thanks @Gabriel and @Wolfgang
>   * make is_datastore_available more robust
>   * fix a lot of wording
>   * drop format on uuid_mount command for UUID
>   * only gather_disk_stats if datastore is available
>   * overall code improvements
>   * ui: include model in partition selector
>   * rebased onto master
> 
> v9:
>   * change mount point to `/mnt/datastore/<name>`
>   * update "Directory" list UI
>   * add `absolute_path()` from Dietmar's RFC
>   * update docs
> 
> v8:
>   * still depends on [1]
>   * paths for removable datastores are now relative to
>      `/mnt/removable_datastore/<UUID>`
>   * add support for creation of removable datastore through the
>      "create directory" endpoint (last 3 patches)
>   * update datastore creation UI
>   * update docs
> 
> v7:
>   * depends on [1]
>   * improve logging when waiting for tasks
>   * drop `update-datatore-cache` refactoring
>   * fix some commit messages
> 
> [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
> 
> v6:
>   * remove 'drop' flag in datastore cache
>   * use maintenance-mode 'unmount' for unmounting process, only for the
>      unmounting not for being unmounted
>   * rename/simplify update-datastore-cache command
>   * ui: integrate new unmounting maintenance mode
>   * basically a mix of v3 and v4
> 
> v5: thanks @Dietmar and @Christian
>   * drop --force for unmount since it'll always fail if tasks are still running, and if
>      there are not normal unount will work
>   * improve several commit messages
>   * improve error message wording
>   * add removable datastore section to docs
>   * add documentation for is_datastore_available
> 
> v4: thanks a lot @Dietmar and @Christian
>   * make check if mounted wayyy faster
>   * don't keep track of mounting state
>   * drop Unplugged maintenance mode
>   * use UUID_FORMAT for uuid field
>   * a lot of small things, like use of bail!, inline format!, ...
>   * include improvement to cache handling
> 
> v3:
>   * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
>   * fix CLI (un)mount command, thanks @Gabriel
>   * add removable datastore CLI autocomplete helper
>   * rebase onto master
>   * move ui patches to the end
> 
> thanks @Lukas and @Thomas for the feedback
> v2:
>   * fix datastore 'add' button in the UI
>   * some format!("{}", a) -> format!("{a}")
>   * replace `const` with `let` in js code
>   * change icon `fa-usb` -> `fa-plug`
>   * add some docs
>   * add JDoc for parseMaintenanceMode
>   * proxmox-schema dep bump
> 
> Dietmar Maurer (1):
>    maintenance: add 'Unmount' maintenance type
> 
> Hannes Laimer (25):
>    pbs-api-types: add backing-device to DataStoreConfig
>    maintenance: make is_offline more generic
>    datastore: add helper for checking if a datastore is mounted
>    api: admin: add (un)mount endpoint for removable datastores
>    api: removable datastore creation
>    api: add check for nested datastores on creation
>    pbs-api-types: add mount_status field to DataStoreListItem
>    bin: manager: add (un)mount command
>    add auto-mounting for removable datastores
>    datastore: handle deletion of removable datastore properly
>    docs: add removable datastores section
>    ui: add partition selector form
>    ui: add removable datastore creation support
>    ui: add (un)mount button to summary
>    ui: tree: render unmounted datastores correctly
>    ui: utils: make parseMaintenanceMode more robust
>    ui: add datastore status mask for unmounted removable datastores
>    ui: maintenance: fix disable msg field if no type is selected
>    ui: render 'unmount' maintenance mode correctly
>    api: node: allow creation of removable datastore through directory
>      endpoint
>    api: node: include removable datastores in directory list
>    node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
>    ui: support create removable datastore through directory creation
>    bin: debug: add inspect device command
>    api: disks: only return UUID of partitions if it actually is one
> 
>   debian/proxmox-backup-server.install        |   1 +
>   debian/proxmox-backup-server.udev           |   3 +
>   docs/storage.rst                            |  38 +++
>   etc/Makefile                                |   1 +
>   etc/removable-device-attach@.service        |   8 +
>   pbs-api-types/src/datastore.rs              |  47 ++-
>   pbs-api-types/src/maintenance.rs            |  12 +-
>   pbs-config/src/datastore.rs                 |  14 +
>   pbs-datastore/src/datastore.rs              |  83 +++++-
>   pbs-datastore/src/lib.rs                    |   4 +-
>   src/api2/admin/datastore.rs                 | 310 +++++++++++++++++++-
>   src/api2/config/datastore.rs                | 107 ++++++-
>   src/api2/node/disks/directory.rs            |  74 +++--
>   src/api2/status/mod.rs                      |  30 +-
>   src/bin/proxmox_backup_debug/inspect.rs     | 149 ++++++++++
>   src/bin/proxmox_backup_manager/datastore.rs | 127 +++++++-
>   src/server/metric_collection/mod.rs         |   4 +
>   src/tools/disks/mod.rs                      |   5 +-
>   www/DirectoryList.js                        |  13 +
>   www/Makefile                                |   1 +
>   www/NavigationTree.js                       |  17 +-
>   www/Utils.js                                |  33 ++-
>   www/css/ext6-pbs.css                        |  20 ++
>   www/datastore/DataStoreListSummary.js       |   1 +
>   www/datastore/Summary.js                    | 115 +++++++-
>   www/form/PartitionSelector.js               |  81 +++++
>   www/window/CreateDirectory.js               |  14 +
>   www/window/DataStoreEdit.js                 |  37 +++
>   www/window/MaintenanceOptions.js            |  17 +-
>   29 files changed, 1274 insertions(+), 92 deletions(-)
>   create mode 100644 etc/removable-device-attach@.service
>   create mode 100644 www/form/PartitionSelector.js
> 



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel