From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
	by lore.proxmox.com (Postfix) with ESMTPS id 432051FF173
	for <inbox@lore.proxmox.com>; Mon, 25 Nov 2024 21:48:41 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id DFE651B650;
	Mon, 25 Nov 2024 21:48:39 +0100 (CET)
Message-ID: <b87d0ed9-4358-44bb-a0cd-b4fa2e5e8a04@proxmox.com>
Date: Mon, 25 Nov 2024 21:48:05 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird Beta
To: Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>, Hannes Laimer <h.laimer@proxmox.com>
References: <20241125162213.157011-1-h.laimer@proxmox.com>
Content-Language: en-GB, de-AT
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
Autocrypt: addr=t.lamprecht@proxmox.com; keydata=
 xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J
 uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w
 OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD
 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7
 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+
 wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6
 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY
 virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt
 dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ
 jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt
 cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO
 R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK
 CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/
 bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R
 G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF
 sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB
 FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB
 UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk
 tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1
 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI
 tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu
 cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR
 D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95
 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy
 HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD
 txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6
 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY
 ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM
 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X
 s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l
 gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9
 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk
 IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j
 /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk
 IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0
 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS
 RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj
 C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA
 kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF
 BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg
 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja
 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx
In-Reply-To: <20241125162213.157011-1-h.laimer@proxmox.com>
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.047 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DMARC_MISSING             0.1 Missing DMARC policy
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
 Validity was blocked. See
 https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
 information.
 RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
 Validity was blocked. See
 https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
 information.
 RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
 Validity was blocked. See
 https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
 information.
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] applied-series: [PATCH proxmox-backup v15 00/26] add
 removable datastores
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>

Am 25.11.24 um 17:21 schrieb Hannes Laimer:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
> 
> The datastore path specified is relative to the root of the used device.
> Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> Multiple datastores can be created on a single device, but only device with 
> a single datastore on them will be auto-mounted.
> 
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in. 

> Dietmar Maurer (1):
>   maintenance: add 'Unmount' maintenance type
> 
> Hannes Laimer (25):
>   pbs-api-types: add backing-device to DataStoreConfig
>   maintenance: make is_offline more generic
>   datastore: add helper for checking if a datastore is mounted
>   api: admin: add (un)mount endpoint for removable datastores
>   api: removable datastore creation
>   api: add check for nested datastores on creation
>   pbs-api-types: add mount_status field to DataStoreListItem
>   bin: manager: add (un)mount command
>   add auto-mounting for removable datastores
>   datastore: handle deletion of removable datastore properly
>   docs: add removable datastores section
>   ui: add partition selector form
>   ui: add removable datastore creation support
>   ui: add (un)mount button to summary
>   ui: tree: render unmounted datastores correctly
>   ui: utils: make parseMaintenanceMode more robust
>   ui: add datastore status mask for unmounted removable datastores
>   ui: maintenance: fix disable msg field if no type is selected
>   ui: render 'unmount' maintenance mode correctly
>   api: node: allow creation of removable datastore through directory
>     endpoint
>   api: node: include removable datastores in directory list
>   node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
>   ui: support create removable datastore through directory creation
>   bin: debug: add inspect device command
>   api: disks: only return UUID of partitions if it actually is one
> 
>  debian/proxmox-backup-server.install        |   1 +
>  debian/proxmox-backup-server.udev           |   3 +
>  docs/storage.rst                            |  38 +++
>  etc/Makefile                                |   1 +
>  etc/removable-device-attach@.service        |   8 +
>  pbs-api-types/src/datastore.rs              |  47 ++-
>  pbs-api-types/src/maintenance.rs            |  12 +-
>  pbs-config/src/datastore.rs                 |  14 +
>  pbs-datastore/src/datastore.rs              |  83 ++++-
>  pbs-datastore/src/lib.rs                    |   4 +-
>  src/api2/admin/datastore.rs                 | 316 +++++++++++++++++++-
>  src/api2/config/datastore.rs                | 122 +++++++-
>  src/api2/node/disks/directory.rs            |  74 +++--
>  src/api2/status/mod.rs                      |  30 +-
>  src/bin/proxmox_backup_debug/inspect.rs     | 149 +++++++++
>  src/bin/proxmox_backup_manager/datastore.rs | 126 +++++++-
>  src/server/metric_collection/mod.rs         |   4 +
>  src/tools/disks/mod.rs                      |   5 +-
>  www/DirectoryList.js                        |  13 +
>  www/Makefile                                |   1 +
>  www/NavigationTree.js                       |  18 +-
>  www/Utils.js                                |  33 +-
>  www/css/ext6-pbs.css                        |  20 ++
>  www/datastore/DataStoreListSummary.js       |   1 +
>  www/datastore/Summary.js                    | 122 +++++++-
>  www/form/PartitionSelector.js               |  81 +++++
>  www/window/CreateDirectory.js               |  14 +
>  www/window/DataStoreEdit.js                 |  32 ++
>  www/window/MaintenanceOptions.js            |  17 +-
>  29 files changed, 1298 insertions(+), 91 deletions(-)
>  create mode 100644 etc/removable-device-attach@.service
>  create mode 100644 www/form/PartitionSelector.js
> 


applied series, thanks!

I did a handful of follow-up commits for the UI behavior, especially w.r.t. how mount
and unmount are processed. Just check the commits themselves, in general I tried to be
descriptive. Please take a look at them to cross-check for any unforeseen regressions.

What would be still good is adding to the docs how jobs like sync, GC, ... are handled
when the datastore is not plugged and if they are caught up on or not when it's plugged
again. That should have a bit higher priority.

Additionally looking into a "remove" event udev hook to clean up a mount point, e.g. if
a user just pulls out the USB pen drive; while that naturally can never be fully hedged
against, especially in terms of data safety, it would be still nice if re-plugging it
works again. For the use case where one just uses an external drive as cheap physical
offsite copy, i.e. as cheap tape replacement, nothing should be permanently broken, as
one can just reinsert the drive and trigger a new sync and then do a clean unmount.
In my tests with some XFS formatted USB pen drive the mount entry is kept when I suddenly
remove the drive (while the datastore is inactive though) and thus I get an error about
the filesystem UUID already existing when the udev hook tries to mount it again when I
re-plug the USB pen drive.
We do not have to bend backwards for this, but atm. I basically only can resolve this
by unmounting the old mountpoint manually, e.g. through a root shell (or reboot I guess),
both not _that_ nice UX wise.

In any way: thanks for going the distance here, this is definitively one of those
features were initial envisioned amount of work was rather way off due to quite some bike
shedding potential but also all the ugliness of dealing with transient mount points and
resources that are not persistent. Lets fix rough edges and potential issues as follow-ups.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel