From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 432051FF173 for ; Mon, 25 Nov 2024 21:48:41 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DFE651B650; Mon, 25 Nov 2024 21:48:39 +0100 (CET) Message-ID: Date: Mon, 25 Nov 2024 21:48:05 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Proxmox Backup Server development discussion , Hannes Laimer References: <20241125162213.157011-1-h.laimer@proxmox.com> Content-Language: en-GB, de-AT From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <20241125162213.157011-1-h.laimer@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL -0.047 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] applied-series: [PATCH proxmox-backup v15 00/26] add removable datastores X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Am 25.11.24 um 17:21 schrieb Hannes Laimer: > These patches add support for removable datastores. All removable > datastores have a backing-device(a UUID) associated with them. Removable > datastores work like normal ones, just that they can be unplugged. It is > possible to create a removable datastore, sync backups onto it, unplug > it and use it on a different PBS. > > The datastore path specified is relative to the root of the used device. > Removable datastores are bind mounted to /mnt/datastore/. > Multiple datastores can be created on a single device, but only device with > a single datastore on them will be auto-mounted. > > When a removable datastore is deleted and 'destroy-data' is set, the > device has to be mounted. If 'destroy-data' is not set the datastore > can be deleted even if the device is not present. Removable datastores > are automatically mounted when plugged in. > Dietmar Maurer (1): > maintenance: add 'Unmount' maintenance type > > Hannes Laimer (25): > pbs-api-types: add backing-device to DataStoreConfig > maintenance: make is_offline more generic > datastore: add helper for checking if a datastore is mounted > api: admin: add (un)mount endpoint for removable datastores > api: removable datastore creation > api: add check for nested datastores on creation > pbs-api-types: add mount_status field to DataStoreListItem > bin: manager: add (un)mount command > add auto-mounting for removable datastores > datastore: handle deletion of removable datastore properly > docs: add removable datastores section > ui: add partition selector form > ui: add removable datastore creation support > ui: add (un)mount button to summary > ui: tree: render unmounted datastores correctly > ui: utils: make parseMaintenanceMode more robust > ui: add datastore status mask for unmounted removable datastores > ui: maintenance: fix disable msg field if no type is selected > ui: render 'unmount' maintenance mode correctly > api: node: allow creation of removable datastore through directory > endpoint > api: node: include removable datastores in directory list > node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR > ui: support create removable datastore through directory creation > bin: debug: add inspect device command > api: disks: only return UUID of partitions if it actually is one > > debian/proxmox-backup-server.install | 1 + > debian/proxmox-backup-server.udev | 3 + > docs/storage.rst | 38 +++ > etc/Makefile | 1 + > etc/removable-device-attach@.service | 8 + > pbs-api-types/src/datastore.rs | 47 ++- > pbs-api-types/src/maintenance.rs | 12 +- > pbs-config/src/datastore.rs | 14 + > pbs-datastore/src/datastore.rs | 83 ++++- > pbs-datastore/src/lib.rs | 4 +- > src/api2/admin/datastore.rs | 316 +++++++++++++++++++- > src/api2/config/datastore.rs | 122 +++++++- > src/api2/node/disks/directory.rs | 74 +++-- > src/api2/status/mod.rs | 30 +- > src/bin/proxmox_backup_debug/inspect.rs | 149 +++++++++ > src/bin/proxmox_backup_manager/datastore.rs | 126 +++++++- > src/server/metric_collection/mod.rs | 4 + > src/tools/disks/mod.rs | 5 +- > www/DirectoryList.js | 13 + > www/Makefile | 1 + > www/NavigationTree.js | 18 +- > www/Utils.js | 33 +- > www/css/ext6-pbs.css | 20 ++ > www/datastore/DataStoreListSummary.js | 1 + > www/datastore/Summary.js | 122 +++++++- > www/form/PartitionSelector.js | 81 +++++ > www/window/CreateDirectory.js | 14 + > www/window/DataStoreEdit.js | 32 ++ > www/window/MaintenanceOptions.js | 17 +- > 29 files changed, 1298 insertions(+), 91 deletions(-) > create mode 100644 etc/removable-device-attach@.service > create mode 100644 www/form/PartitionSelector.js > applied series, thanks! I did a handful of follow-up commits for the UI behavior, especially w.r.t. how mount and unmount are processed. Just check the commits themselves, in general I tried to be descriptive. Please take a look at them to cross-check for any unforeseen regressions. What would be still good is adding to the docs how jobs like sync, GC, ... are handled when the datastore is not plugged and if they are caught up on or not when it's plugged again. That should have a bit higher priority. Additionally looking into a "remove" event udev hook to clean up a mount point, e.g. if a user just pulls out the USB pen drive; while that naturally can never be fully hedged against, especially in terms of data safety, it would be still nice if re-plugging it works again. For the use case where one just uses an external drive as cheap physical offsite copy, i.e. as cheap tape replacement, nothing should be permanently broken, as one can just reinsert the drive and trigger a new sync and then do a clean unmount. In my tests with some XFS formatted USB pen drive the mount entry is kept when I suddenly remove the drive (while the datastore is inactive though) and thus I get an error about the filesystem UUID already existing when the udev hook tries to mount it again when I re-plug the USB pen drive. We do not have to bend backwards for this, but atm. I basically only can resolve this by unmounting the old mountpoint manually, e.g. through a root shell (or reboot I guess), both not _that_ nice UX wise. In any way: thanks for going the distance here, this is definitively one of those features were initial envisioned amount of work was rather way off due to quite some bike shedding potential but also all the ugliness of dealing with transient mount points and resources that are not persistent. Lets fix rough edges and potential issues as follow-ups. _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel