From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 467001FF186 for ; Wed, 13 Nov 2024 16:01:44 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5F09618A83; Wed, 13 Nov 2024 16:01:41 +0100 (CET) From: Hannes Laimer To: pbs-devel@lists.proxmox.com Date: Wed, 13 Nov 2024 16:00:36 +0100 Message-Id: <20241113150102.164820-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.021 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox-backup v13 00/26] add removable datastores X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" These patches add support for removable datastores. All removable datastores have a backing-device(a UUID) associated with them. Removable datastores work like normal ones, just that they can be unplugged. It is possible to create a removable datastore, sync backups onto it, unplug it and use it on a different PBS. The datastore path specified is relative to the root of the used device. Removable datastores are bind mounted to /mnt/datastore/. Multiple datastores can be created on a single device, but only device with a single datastore on them will be auto-mounted. When a removable datastore is deleted and 'destroy-data' is set, the device has to be mounted. If 'destroy-data' is not set the datastore can be deleted even if the device is not present. Removable datastores are automatically mounted when plugged in. v13: thanks @Fabian * allow multiple datastore on devices * replace `is_datastore_available` by a more specific function, it is now removable datastore specific and won't be called for normal ones * replace removable/is_available in status structs with mount_state, which is `None` for normal datastore as it makes it less ambiguous what is meant * remove notion of 'available' from normal datastores and replace it with mounted/mount_status for removable ones, as it never really made sense in the first place * abort of an unmount task will now reset the maintanance mode * add check for race when setting maintenance at end of unmounting task * improve documentation and commit messages * remove not needed tokio::spawn * only auto mount devices with single datastore on them * drop ptach that added flag for excluding used partitions * make auto mount service not dynamic * add debug command to scan devices for datastores they may contain * rebase onto master v12: thanks @Wolfgang * use bind mounts, so now /path/to/ds is mounted to /mnt/datastore/ this is a bit cleaner and allows for multiple datastores on a single device to be mounted individually, if we want to allow that in the future * small code improvements v11: * rebase onto master v10: thanks @Gabriel and @Wolfgang * make is_datastore_available more robust * fix a lot of wording * drop format on uuid_mount command for UUID * only gather_disk_stats if datastore is available * overall code improvements * ui: include model in partition selector * rebased onto master v9: * change mount point to `/mnt/datastore/` * update "Directory" list UI * add `absolute_path()` from Dietmar's RFC * update docs v8: * still depends on [1] * paths for removable datastores are now relative to `/mnt/removable_datastore/` * add support for creation of removable datastore through the "create directory" endpoint (last 3 patches) * update datastore creation UI * update docs v7: * depends on [1] * improve logging when waiting for tasks * drop `update-datatore-cache` refactoring * fix some commit messages [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html v6: * remove 'drop' flag in datastore cache * use maintenance-mode 'unmount' for unmounting process, only for the unmounting not for being unmounted * rename/simplify update-datastore-cache command * ui: integrate new unmounting maintenance mode * basically a mix of v3 and v4 v5: thanks @Dietmar and @Christian * drop --force for unmount since it'll always fail if tasks are still running, and if there are not normal unount will work * improve several commit messages * improve error message wording * add removable datastore section to docs * add documentation for is_datastore_available v4: thanks a lot @Dietmar and @Christian * make check if mounted wayyy faster * don't keep track of mounting state * drop Unplugged maintenance mode * use UUID_FORMAT for uuid field * a lot of small things, like use of bail!, inline format!, ... * include improvement to cache handling v3: * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied) * fix CLI (un)mount command, thanks @Gabriel * add removable datastore CLI autocomplete helper * rebase onto master * move ui patches to the end thanks @Lukas and @Thomas for the feedback v2: * fix datastore 'add' button in the UI * some format!("{}", a) -> format!("{a}") * replace `const` with `let` in js code * change icon `fa-usb` -> `fa-plug` * add some docs * add JDoc for parseMaintenanceMode * proxmox-schema dep bump Dietmar Maurer (2): config: factor out method to get the absolute datastore path maintenance: add 'Unmount' maintenance type Hannes Laimer (24): tools: add disks utility functions pbs-api-types: add backing-device to DataStoreConfig disks: add UUID to partition info datastore: add helper for checking if a datastore is mounted api: admin: add (un)mount endpoint for removable datastores api: removable datastore creation pbs-api-types: add mount_status field to DataStoreListItem bin: manager: add (un)mount command add auto-mounting for removable datastores datastore: handle deletion of removable datastore properly docs: add removable datastores section ui: add partition selector form ui: add removable datastore creation support ui: add (un)mount button to summary ui: tree: render unmounted datastores correctly ui: utils: make parseMaintenanceMode more robust ui: add datastore status mask for unmounted removable datastores ui: maintenance: fix disable msg field if no type is selected ui: render 'unmount' maintenance mode correctly api: node: allow creation of removable datastore through directory endpoint api: node: include removable datastores in directory list node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR ui: support create removable datastore through directory creation bin: debug: add inspect device command debian/proxmox-backup-server.install | 1 + debian/proxmox-backup-server.udev | 3 + docs/storage.rst | 38 +++ etc/Makefile | 3 +- etc/removable-device-attach@.service | 8 + pbs-api-types/src/datastore.rs | 46 +++- pbs-api-types/src/maintenance.rs | 7 +- pbs-config/src/datastore.rs | 14 + pbs-datastore/src/datastore.rs | 88 +++++- pbs-datastore/src/lib.rs | 2 +- src/api2/admin/datastore.rs | 289 ++++++++++++++++++-- src/api2/config/datastore.rs | 87 +++++- src/api2/node/disks/directory.rs | 104 ++++++- src/api2/status/mod.rs | 29 +- src/bin/proxmox_backup_debug/inspect.rs | 149 ++++++++++ src/bin/proxmox_backup_manager/datastore.rs | 136 ++++++++- src/server/metric_collection/mod.rs | 18 +- src/tools/disks/mod.rs | 39 ++- www/DirectoryList.js | 13 + www/Makefile | 1 + www/NavigationTree.js | 17 +- www/Utils.js | 33 ++- www/css/ext6-pbs.css | 20 ++ www/datastore/DataStoreListSummary.js | 1 + www/datastore/Summary.js | 113 +++++++- www/form/PartitionSelector.js | 81 ++++++ www/window/CreateDirectory.js | 14 + www/window/DataStoreEdit.js | 37 +++ www/window/MaintenanceOptions.js | 17 +- 29 files changed, 1328 insertions(+), 80 deletions(-) create mode 100644 etc/removable-device-attach@.service create mode 100644 www/form/PartitionSelector.js -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel