[pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores

[pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores

[Hannes Laimer]

These patches add support for removable datastores. All removable
datastores have a backing-device(a UUID) associated with them. Removable
datastores work like normal ones, just that they can be unplugged. It is
possible to create a removable datastore, sync backups onto it, unplug
it and use it on a different PBS.

The datastore path specified is relative to the root of the used device.
Removable datastores are bind mounted to /mnt/datastore/<NAME>.
If we decide to allow multiple datastores on the same device in the future
they can be mounted/unmounted individually.

When a removable datastore is deleted and 'destroy-data' is set, the
device has to be mounted. If 'destroy-data' is not set the datastore
can be deleted even if the device is not present. Removable datastores
are automatically mounted when plugged in. 

v12: thanks @Wolfgang
 * use bind mounts, so now
    <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
    this is a bit cleaner and allows for multiple datastores
    on a single device to be mounted individually, if we
    want to allow that in the future
 * small code improvements


v11:
 * rebase onto master

v10: thanks @Gabriel and @Wolfgang
 * make is_datastore_available more robust
 * fix a lot of wording
 * drop format on uuid_mount command for UUID
 * only gather_disk_stats if datastore is available
 * overall code improvements
 * ui: include model in partition selector
 * rebased onto master

v9:
 * change mount point to `/mnt/datastore/<name>`
 * update "Directory" list UI
 * add `absolute_path()` from Dietmar's RFC
 * update docs

v8:
 * still depends on [1]
 * paths for removable datastores are now relative to 
    `/mnt/removable_datastore/<UUID>`
 * add support for creation of removable datastore through the 
    "create directory" endpoint (last 3 patches)
 * update datastore creation UI
 * update docs

v7:
 * depends on [1]
 * improve logging when waiting for tasks
 * drop `update-datatore-cache` refactoring
 * fix some commit messages

[1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html

v6:
 * remove 'drop' flag in datastore cache
 * use maintenance-mode 'unmount' for unmounting process, only for the
    unmounting not for being unmounted
 * rename/simplify update-datastore-cache command
 * ui: integrate new unmounting maintenance mode
 * basically a mix of v3 and v4

v5: thanks @Dietmar and @Christian
 * drop --force for unmount since it'll always fail if tasks are still running, and if
    there are not normal unount will work
 * improve several commit messages
 * improve error message wording
 * add removable datastore section to docs
 * add documentation for is_datastore_available

v4: thanks a lot @Dietmar and @Christian
 * make check if mounted wayyy faster
 * don't keep track of mounting state
 * drop Unplugged maintenance mode
 * use UUID_FORMAT for uuid field
 * a lot of small things, like use of bail!, inline format!, ...
 * include improvement to cache handling

v3:
 * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
 * fix CLI (un)mount command, thanks @Gabriel
 * add removable datastore CLI autocomplete helper
 * rebase onto master
 * move ui patches to the end

thanks @Lukas and @Thomas for the feedback
v2:
 * fix datastore 'add' button in the UI
 * some format!("{}", a) -> format!("{a}")
 * replace `const` with `let` in js code
 * change icon `fa-usb` -> `fa-plug`
 * add some docs
 * add JDoc for parseMaintenanceMode
 * proxmox-schema dep bump

Dietmar Maurer (2):
  config: factor out method to get the absolute datastore path
  maintenance: add 'Unmount' maintenance type

Hannes Laimer (24):
  tools: add disks utility functions
  pbs-api-types: add backing-device to DataStoreConfig
  disks: add UUID to partition info
  datastore: add helper for checking if a removable datastore is
    available
  api: admin: add (un)mount endpoint for removable datastores
  api: removable datastore creation
  api: disks list: add exclude-used flag
  pbs-api-types: add removable/is-available flag to DataStoreListItem
  bin: manager: add (un)mount command
  add auto-mounting for removable datastores
  datastore: handle deletion of removable datastore properly
  docs: add removable datastores section
  ui: add partition selector form
  ui: add removable datastore creation support
  ui: add (un)mount button to summary
  ui: tree: render unmounted datastores correctly
  ui: utils: make parseMaintenanceMode more robust
  ui: add datastore status mask for unmounted removable datastores
  ui: maintenance: fix disable msg field if no type is selected
  ui: render 'unmount' maintenance mode correctly
  api: node: allow creation of removable datastore through directory
    endpoint
  api: node: include removable datastores in directory list
  node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
  ui: support create removable datastore through directory creation

 debian/proxmox-backup-server.install        |   1 +
 debian/proxmox-backup-server.udev           |   3 +
 docs/storage.rst                            |  29 +++
 etc/Makefile                                |   3 +-
 etc/removable-device-attach@.service.in     |   8 +
 pbs-api-types/src/datastore.rs              |  42 +++-
 pbs-api-types/src/maintenance.rs            |  11 +-
 pbs-config/src/datastore.rs                 |  14 ++
 pbs-datastore/src/datastore.rs              |  73 +++++-
 pbs-datastore/src/lib.rs                    |   2 +-
 src/api2/admin/datastore.rs                 | 260 ++++++++++++++++++--
 src/api2/config/datastore.rs                |  69 +++++-
 src/api2/node/disks/directory.rs            | 102 ++++++--
 src/api2/node/disks/mod.rs                  |   8 +
 src/api2/status.rs                          |  18 +-
 src/bin/proxmox-backup-proxy.rs             |   9 +-
 src/bin/proxmox_backup_manager/datastore.rs | 129 +++++++++-
 src/tools/disks/mod.rs                      |  99 +++++++-
 www/DirectoryList.js                        |  13 +
 www/Makefile                                |   1 +
 www/NavigationTree.js                       |  15 +-
 www/Utils.js                                |  33 ++-
 www/css/ext6-pbs.css                        |  20 ++
 www/datastore/DataStoreListSummary.js       |   1 +
 www/datastore/Summary.js                    | 113 ++++++++-
 www/form/PartitionSelector.js               |  81 ++++++
 www/window/CreateDirectory.js               |  14 ++
 www/window/DataStoreEdit.js                 |  37 +++
 www/window/MaintenanceOptions.js            |  17 +-
 29 files changed, 1141 insertions(+), 84 deletions(-)
 create mode 100644 etc/removable-device-attach@.service.in
 create mode 100644 www/form/PartitionSelector.js

-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions

[Hannes Laimer]

... for mounting and unmounting

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/tools/disks/mod.rs | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index c729c26a..9d8ff05c 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -1315,3 +1315,33 @@ pub fn get_fs_uuid(disk: &Disk) -> Result<String, Error> {
 
     bail!("get_fs_uuid failed - missing UUID");
 }
+
+/// Mount a disk by its UUID and the mount point.
+pub fn mount_by_uuid(uuid: &str, mount_point: &Path) -> Result<(), Error> {
+    let mut command = std::process::Command::new("mount");
+    command.arg(&format!("UUID={uuid}"));
+    command.arg(mount_point);
+
+    proxmox_sys::command::run_command(command, None)?;
+    Ok(())
+}
+
+/// Create bind mount.
+pub fn bind_mount(path: &Path, target: &Path) -> Result<(), Error> {
+    let mut command = std::process::Command::new("mount");
+    command.arg("--bind");
+    command.arg(path);
+    command.arg(target);
+
+    proxmox_sys::command::run_command(command, None)?;
+    Ok(())
+}
+
+/// Unmount a disk by its mount point.
+pub fn unmount_by_mountpoint(path: &str) -> Result<(), Error> {
+    let mut command = std::process::Command::new("umount");
+    command.arg(path);
+
+    proxmox_sys::command::run_command(command, None)?;
+    Ok(())
+}
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path

[Hannes Laimer]

From: Dietmar Maurer <dietmar@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-api-types/src/datastore.rs   |  5 +++++
 pbs-datastore/src/datastore.rs   | 11 +++++++----
 src/api2/node/disks/directory.rs |  4 ++--
 src/bin/proxmox-backup-proxy.rs  |  4 ++--
 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 31767417..a5704c93 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -357,6 +357,11 @@ impl DataStoreConfig {
         }
     }
 
+    /// Returns the absolute path to the datastore content.
+    pub fn absolute_path(&self) -> String {
+        self.path.clone()
+    }
+
     pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
         self.maintenance_mode.as_ref().and_then(|str| {
             MaintenanceMode::deserialize(proxmox_schema::de::SchemaDeserializer::new(
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index d0f3c53a..fb37bd5a 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -178,7 +178,7 @@ impl DataStore {
             )?;
             Arc::new(ChunkStore::open(
                 name,
-                &config.path,
+                config.absolute_path(),
                 tuning.sync_level.unwrap_or_default(),
             )?)
         };
@@ -262,8 +262,11 @@ impl DataStore {
             DatastoreTuning::API_SCHEMA
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
         )?;
-        let chunk_store =
-            ChunkStore::open(&name, &config.path, tuning.sync_level.unwrap_or_default())?;
+        let chunk_store = ChunkStore::open(
+            &name,
+            config.absolute_path(),
+            tuning.sync_level.unwrap_or_default(),
+        )?;
         let inner = Arc::new(Self::with_store_and_config(
             Arc::new(chunk_store),
             config,
@@ -1387,7 +1390,7 @@ impl DataStore {
             bail!("datastore is currently in use");
         }
 
-        let base = PathBuf::from(&datastore_config.path);
+        let base = PathBuf::from(datastore_config.absolute_path());
 
         let mut ok = true;
         if destroy_data {
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index e971f40d..c0c53d84 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -247,12 +247,12 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
     let (config, _) = pbs_config::datastore::config()?;
     let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
     let conflicting_datastore: Option<DataStoreConfig> =
-        datastores.into_iter().find(|ds| ds.path == path);
+        datastores.into_iter().find(|ds| ds.absolute_path() == path);
 
     if let Some(conflicting_datastore) = conflicting_datastore {
         bail!(
             "Can't remove '{}' since it's required by datastore '{}'",
-            conflicting_datastore.path,
+            conflicting_datastore.absolute_path(),
             conflicting_datastore.name
         );
     }
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 041f3aff..6ccf2f96 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -1132,8 +1132,8 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
                 {
                     continue;
                 }
-                let path = std::path::Path::new(&config.path);
-                datastores.push(gather_disk_stats(disk_manager.clone(), path, &config.name));
+                let path = std::path::PathBuf::from(config.absolute_path());
+                datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
             }
         }
         Err(err) => {
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-api-types/src/datastore.rs | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index a5704c93..863ed317 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -160,6 +160,9 @@ pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema =
         .minimum(1)
         .schema();
 
+/// Base directory where datastores are mounted
+pub const DATASTORE_MOUNT_DIR: &str = "/mnt/datastore";
+
 #[api]
 #[derive(Debug, Default, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
 #[serde(rename_all = "lowercase")]
@@ -273,6 +276,12 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
             format: &ApiStringFormat::PropertyString(&MaintenanceMode::API_SCHEMA),
             type: String,
         },
+        "backing-device": {
+            description: "The UUID of the filesystem partition for removable datastores.",
+            optional: true,
+            format: &proxmox_schema::api_types::UUID_FORMAT,
+            type: String,
+        }
     }
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -320,6 +329,11 @@ pub struct DataStoreConfig {
     /// Maintenance mode, type is either 'offline' or 'read-only', message should be enclosed in "
     #[serde(skip_serializing_if = "Option::is_none")]
     pub maintenance_mode: Option<String>,
+
+    /// The UUID of the device(for removable datastores)
+    #[updater(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub backing_device: Option<String>,
 }
 
 #[api]
@@ -354,12 +368,23 @@ impl DataStoreConfig {
             notification_mode: None,
             tuning: None,
             maintenance_mode: None,
+            backing_device: None,
         }
     }
 
     /// Returns the absolute path to the datastore content.
     pub fn absolute_path(&self) -> String {
-        self.path.clone()
+        if self.backing_device.is_some() {
+            format!("{DATASTORE_MOUNT_DIR}/{}", self.name)
+        } else {
+            self.path.clone()
+        }
+    }
+
+    pub fn get_mount_point(&self) -> Option<String> {
+        self.backing_device
+            .is_some()
+            .then(|| format!("{DATASTORE_MOUNT_DIR}/{}", self.name))
     }
 
     pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type

[Hannes Laimer]

From: Dietmar Maurer <dietmar@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
---
 pbs-api-types/src/datastore.rs   | 3 +++
 pbs-api-types/src/maintenance.rs | 5 ++++-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 863ed317..d6e2f97e 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -404,6 +404,9 @@ impl DataStoreConfig {
         match current_type {
             Some(MaintenanceType::ReadOnly) => { /* always OK  */ }
             Some(MaintenanceType::Offline) => { /* always OK  */ }
+            Some(MaintenanceType::Unmount) => {
+                bail!("datastore is being unmounted");
+            }
             Some(MaintenanceType::Delete) => {
                 match new_type {
                     Some(MaintenanceType::Delete) => { /* allow to delete a deleted storage */ }
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 1e3413dc..fd4d3416 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -38,7 +38,6 @@ pub enum Operation {
 /// Maintenance type.
 pub enum MaintenanceType {
     // TODO:
-    //  - Add "unmounting" once we got pluggable datastores
     //  - Add "GarbageCollection" or "DeleteOnly" as type and track GC (or all deletes) as separate
     //    operation, so that one can enable a mode where nothing new can be added but stuff can be
     //    cleaned
@@ -48,6 +47,8 @@ pub enum MaintenanceType {
     Offline,
     /// The datastore is being deleted.
     Delete,
+    /// The (removable) datastore is being unmounted.
+    Unmount,
 }
 serde_plain::derive_display_from_serialize!(MaintenanceType);
 serde_plain::derive_fromstr_from_deserialize!(MaintenanceType);
@@ -94,6 +95,8 @@ impl MaintenanceMode {
 
         if let Some(Operation::Lookup) = operation {
             return Ok(());
+        } else if self.ty == MaintenanceType::Unmount {
+            bail!("datastore is being unmounted");
         } else if self.ty == MaintenanceType::Offline {
             bail!("offline maintenance mode: {}", message);
         } else if self.ty == MaintenanceType::ReadOnly {
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 05/26] disks: add UUID to partition info

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/tools/disks/mod.rs | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 9d8ff05c..93c1a882 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -55,6 +55,8 @@ pub struct LsblkInfo {
     /// File system label.
     #[serde(rename = "fstype")]
     file_system_type: Option<String>,
+    /// File system UUID.
+    uuid: Option<String>,
 }
 
 impl DiskManage {
@@ -613,7 +615,7 @@ pub struct BlockDevStat {
 /// Use lsblk to read partition type uuids and file system types.
 pub fn get_lsblk_info() -> Result<Vec<LsblkInfo>, Error> {
     let mut command = std::process::Command::new("lsblk");
-    command.args(["--json", "-o", "path,parttype,fstype"]);
+    command.args(["--json", "-o", "path,parttype,fstype,uuid"]);
 
     let output = proxmox_sys::command::run_command(command, None)?;
 
@@ -697,6 +699,8 @@ pub struct PartitionInfo {
     pub size: Option<u64>,
     /// GPT partition
     pub gpt: bool,
+    /// UUID
+    pub uuid: Option<String>,
 }
 
 #[api(
@@ -887,8 +891,10 @@ fn get_partitions_info(
 
             let mounted = disk.is_mounted().unwrap_or(false);
             let mut filesystem = None;
+            let mut uuid = None;
             if let (Some(devpath), Some(infos)) = (devpath.as_ref(), lsblk_infos.as_ref()) {
                 for info in infos.iter().filter(|i| i.path.eq(devpath)) {
+                    uuid = info.uuid.clone();
                     used = match info.partition_type.as_deref() {
                         Some("21686148-6449-6e6f-744e-656564454649") => PartitionUsageType::BIOS,
                         Some("c12a7328-f81f-11d2-ba4b-00a0c93ec93b") => PartitionUsageType::EFI,
@@ -911,6 +917,7 @@ fn get_partitions_info(
                 filesystem,
                 size: disk.size().ok(),
                 gpt: disk.has_gpt(),
+                uuid,
             }
         })
         .collect()
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available

[Hannes Laimer]

Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-api-types/src/maintenance.rs |  2 ++
 pbs-datastore/src/datastore.rs   | 58 ++++++++++++++++++++++++++++++++
 pbs-datastore/src/lib.rs         |  2 +-
 src/bin/proxmox-backup-proxy.rs  |  5 ++-
 4 files changed, 65 insertions(+), 2 deletions(-)

diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index fd4d3416..9f51292e 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -82,6 +82,8 @@ impl MaintenanceMode {
     /// task finishes, so all open files are closed.
     pub fn is_offline(&self) -> bool {
         self.ty == MaintenanceType::Offline
+            || self.ty == MaintenanceType::Unmount
+            || self.ty == MaintenanceType::Delete
     }
 
     pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index fb37bd5a..29f98b37 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1,5 +1,6 @@
 use std::collections::{HashMap, HashSet};
 use std::io::{self, Write};
+use std::os::unix::ffi::OsStrExt;
 use std::os::unix::io::AsRawFd;
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, LazyLock, Mutex};
@@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
 use proxmox_sys::error::SysError;
 use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
 use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
+use proxmox_sys::linux::procfs::MountInfo;
 use proxmox_sys::process_locker::ProcessLockSharedGuard;
 use proxmox_worker_task::WorkerTaskContext;
 
@@ -46,6 +48,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
     Ok(())
 }
 
+/// check if a removable datastore is currently available/mounted by
+/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
+/// /proc/self/mountinfo
+pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
+    use nix::sys::stat::SFlag;
+
+    let uuid = match config.backing_device.as_deref() {
+        Some(dev) => dev,
+        None => return true,
+    };
+
+    let Some(store_mount_point) = config.get_mount_point() else {
+        return true;
+    };
+    let store_mount_point = Path::new(&store_mount_point);
+
+    let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
+        Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
+        _ => return false,
+    };
+
+    let Ok(mount_info) = MountInfo::read() else {
+        return false;
+    };
+
+    for (_, entry) in mount_info {
+        let Some(source) = entry.mount_source else {
+            continue;
+        };
+
+        if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
+            continue;
+        }
+
+        if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
+            let sflag = SFlag::from_bits_truncate(stat.st_mode);
+
+            if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
+                return true;
+            }
+        }
+    }
+
+    false
+}
+
 /// Datastore Management
 ///
 /// A Datastore can store severals backups, and provides the
@@ -155,6 +203,12 @@ impl DataStore {
             }
         }
 
+        if config.backing_device.is_some() && !is_datastore_available(&config) {
+            let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+            datastore_cache.remove(&config.name);
+            bail!("Removable Datastore is not mounted");
+        }
+
         let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
         let entry = datastore_cache.get(name);
 
@@ -258,6 +312,10 @@ impl DataStore {
     ) -> Result<Arc<Self>, Error> {
         let name = config.name.clone();
 
+        if !is_datastore_available(&config) {
+            bail!("Datastore is not available")
+        }
+
         let tuning: DatastoreTuning = serde_json::from_value(
             DatastoreTuning::API_SCHEMA
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 202b0955..6d58f327 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -204,7 +204,7 @@ pub use manifest::BackupManifest;
 pub use store_progress::StoreProgress;
 
 mod datastore;
-pub use datastore::{check_backup_owner, DataStore};
+pub use datastore::{check_backup_owner, is_datastore_available, DataStore};
 
 mod hierarchy;
 pub use hierarchy::{
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 6ccf2f96..58fb8b84 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -23,7 +23,7 @@ use proxmox_sys::fs::{CreateOptions, FileSystemInformation};
 use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsStat};
 use proxmox_sys::logrotate::LogRotate;
 
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
 
 use proxmox_rest_server::{
     cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
@@ -1132,6 +1132,9 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
                 {
                     continue;
                 }
+                if !is_datastore_available(&config) {
+                    continue;
+                }
                 let path = std::path::PathBuf::from(config.absolute_path());
                 datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
             }
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-api-types/src/maintenance.rs |   4 +
 src/api2/admin/datastore.rs      | 243 +++++++++++++++++++++++++++++--
 2 files changed, 237 insertions(+), 10 deletions(-)

diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 9f51292e..60181258 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -78,6 +78,10 @@ pub struct MaintenanceMode {
 }
 
 impl MaintenanceMode {
+    pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
+        Self { ty, message }
+    }
+
     /// Used for deciding whether the datastore is cleared from the internal cache after the last
     /// task finishes, so all open files are closed.
     pub fn is_offline(&self) -> bool {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 976617d9..3c95888d 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -3,7 +3,7 @@
 use std::collections::HashSet;
 use std::ffi::OsStr;
 use std::os::unix::ffi::OsStrExt;
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 use anyhow::{bail, format_err, Error};
@@ -13,7 +13,7 @@ use hyper::{header, Body, Response, StatusCode};
 use serde::Deserialize;
 use serde_json::{json, Value};
 use tokio_stream::wrappers::ReceiverStream;
-use tracing::{info, warn};
+use tracing::{debug, info, warn};
 
 use proxmox_async::blocking::WrappedReaderStream;
 use proxmox_async::{io::AsyncChannelWriter, stream::AsyncReaderStream};
@@ -29,6 +29,7 @@ use proxmox_sys::fs::{
     file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
 };
 use proxmox_time::CalendarEvent;
+use proxmox_worker_task::WorkerTaskContext;
 
 use pxar::accessor::aio::Accessor;
 use pxar::EntryKind;
@@ -36,12 +37,12 @@ use pxar::EntryKind;
 use pbs_api_types::{
     print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
     Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
-    GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
-    PruneJobOptions, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
-    BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
-    DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
-    PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
+    GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
+    MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
+    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
+    BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
+    NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
+    PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
     VERIFICATION_OUTDATED_AFTER_SCHEMA,
 };
 use pbs_client::pxar::{create_tar, create_zip};
@@ -57,8 +58,8 @@ use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
 use pbs_datastore::prune::compute_prune_info;
 use pbs_datastore::{
-    check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
-    StoreProgress, CATALOG_NAME,
+    check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
+    LocalChunkReader, StoreProgress, CATALOG_NAME,
 };
 use pbs_tools::json::required_string_param;
 use proxmox_rest_server::{formatter, WorkerTask};
@@ -2384,6 +2385,226 @@ pub async fn set_backup_owner(
     .await?
 }
 
+/// Here we
+///
+/// 1. mount the removable device to `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
+/// 2. bind mount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>/<datastore.path>` to `/mnt/datastore/<datastore.name>`
+/// 3. unmount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
+///
+/// leaving us with the datastore being mounted directly with its name under /mnt/datastore/...
+///
+/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
+/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
+/// technically possible.
+pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
+    if let (Some(uuid), Some(mount_point)) = (
+        datastore.backing_device.as_ref(),
+        datastore.get_mount_point(),
+    ) {
+        if pbs_datastore::is_datastore_available(&datastore) {
+            bail!("datastore '{}' is already mounted", datastore.name);
+        }
+        let tmp_mount_path = format!(
+            "{}/{:x}",
+            pbs_buildcfg::rundir!("/mount"),
+            proxmox_uuid::Uuid::generate()
+        );
+
+        let default_options = proxmox_sys::fs::CreateOptions::new();
+        proxmox_sys::fs::create_path(
+            &tmp_mount_path,
+            Some(default_options.clone()),
+            Some(default_options.clone()),
+        )?;
+
+        debug!("mounting '{uuid}' to '{}'", tmp_mount_path);
+        crate::tools::disks::mount_by_uuid(uuid, Path::new(&tmp_mount_path))?;
+
+        let full_store_path = format!(
+            "{tmp_mount_path}/{}",
+            datastore.path.trim_start_matches('/')
+        );
+
+        proxmox_sys::fs::create_path(
+            &mount_point,
+            Some(default_options.clone()),
+            Some(default_options.clone()),
+        )?;
+
+        // can't be created before it is mounted, so we have to do it here
+        proxmox_sys::fs::create_path(
+            &full_store_path,
+            Some(default_options.clone()),
+            Some(default_options.clone()),
+        )?;
+
+        info!(
+            "mounting '{}'({}) to '{}'",
+            datastore.name, datastore.path, mount_point
+        );
+        if let Err(err) =
+            crate::tools::disks::bind_mount(Path::new(&full_store_path), Path::new(&mount_point))
+        {
+            debug!("unmounting '{}'", tmp_mount_path);
+            let _ = crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path);
+            let _ = std::fs::remove_dir(std::path::Path::new(&tmp_mount_path));
+            return Err(format_err!(
+                "Datastore '{}' cound not be mounted: {}.",
+                datastore.name,
+                err
+            ));
+        }
+
+        debug!("unmounting '{}'", tmp_mount_path);
+        crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path)?;
+        std::fs::remove_dir(std::path::Path::new(&tmp_mount_path))?;
+
+        Ok(())
+    } else {
+        Err(format_err!(
+            "Datastore '{}' cannot be mounted because it is not removable.",
+            datastore.name
+        ))
+    }
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+        }
+    },
+    returns: {
+        schema: UPID_SCHEMA,
+    },
+    access: {
+        permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
+    },
+)]
+/// Mount removable datastore.
+pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let (section_config, _digest) = pbs_config::datastore::config()?;
+    let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+    if datastore.backing_device.is_none() {
+        bail!("datastore '{store}' is not removable");
+    }
+
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+    let upid = WorkerTask::new_thread(
+        "mount-device",
+        Some(store),
+        auth_id.to_string(),
+        to_stdout,
+        move |_worker| do_mount_device(datastore),
+    )?;
+
+    Ok(json!(upid))
+}
+
+fn do_unmount_device(
+    datastore: DataStoreConfig,
+    worker: Option<&dyn WorkerTaskContext>,
+) -> Result<(), Error> {
+    let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
+    let mut old_status = String::new();
+    while active_operations.read + active_operations.write > 0 {
+        if let Some(worker) = worker {
+            if worker.abort_requested() {
+                bail!("aborted, due to user request");
+            }
+            let status = format!(
+                "cannot unmount yet, still {} read and {} write operations active",
+                active_operations.read, active_operations.write
+            );
+            if status != old_status {
+                info!("{status}");
+                old_status = status;
+            }
+        }
+        std::thread::sleep(std::time::Duration::from_millis(250));
+        active_operations = task_tracking::get_active_operations(&datastore.name)?;
+    }
+    if let Some(mount_point) = datastore.get_mount_point() {
+        crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
+
+        let _lock = pbs_config::datastore::lock_config()?;
+        let (mut section_config, _digest) = pbs_config::datastore::config()?;
+        let mut store_config: DataStoreConfig =
+            section_config.lookup("datastore", &datastore.name)?;
+        store_config.maintenance_mode = None;
+        section_config.set_data(&datastore.name, "datastore", &store_config)?;
+        pbs_config::datastore::save_config(&section_config)?;
+    }
+    Ok(())
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+        },
+    },
+    returns: {
+        schema: UPID_SCHEMA,
+    },
+    access: {
+        permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, true),
+    }
+)]
+/// Unmount a removable device that is associated with the datastore
+pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let _lock = pbs_config::datastore::lock_config()?;
+    let (mut section_config, _digest) = pbs_config::datastore::config()?;
+    let mut datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+    if datastore.backing_device.is_none() {
+        bail!("datastore '{store}' is not removable");
+    }
+
+    if !is_datastore_available(&datastore) {
+        bail!("datastore '{store}' is not mounted");
+    }
+
+    datastore.set_maintenance_mode(Some(MaintenanceMode::new(MaintenanceType::Unmount, None)))?;
+    section_config.set_data(&store, "datastore", &datastore)?;
+    pbs_config::datastore::save_config(&section_config)?;
+
+    drop(_lock);
+
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+    if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+    {
+        let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
+        let _ = proxmox_daemon::command_socket::send_raw(
+            sock,
+            &format!(
+                "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
+                &store
+            ),
+        )
+        .await;
+    }
+
+    let upid = WorkerTask::new_thread(
+        "unmount-device",
+        Some(store),
+        auth_id.to_string(),
+        to_stdout,
+        move |worker| do_unmount_device(datastore, Some(&worker)),
+    )?;
+
+    Ok(json!(upid))
+}
+
 #[sortable]
 const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
     (
@@ -2422,6 +2643,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
             .get(&API_METHOD_LIST_GROUPS)
             .delete(&API_METHOD_DELETE_GROUP),
     ),
+    ("mount", &Router::new().post(&API_METHOD_MOUNT)),
     (
         "namespace",
         // FIXME: move into datastore:: sub-module?!
@@ -2456,6 +2678,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
             .delete(&API_METHOD_DELETE_SNAPSHOT),
     ),
     ("status", &Router::new().get(&API_METHOD_STATUS)),
+    ("unmount", &Router::new().post(&API_METHOD_UNMOUNT)),
     (
         "upload-backup-log",
         &Router::new().upload(&API_METHOD_UPLOAD_BACKUP_LOG),
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/config/datastore.rs | 37 ++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index ca6edf05..c24b6e9d 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -4,7 +4,7 @@ use ::serde::{Deserialize, Serialize};
 use anyhow::Error;
 use hex::FromHex;
 use serde_json::Value;
-use tracing::warn;
+use tracing::{info, warn};
 
 use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
 use proxmox_schema::{api, param_bail, ApiType};
@@ -20,7 +20,8 @@ use pbs_config::BackupLockGuard;
 use pbs_datastore::chunk_store::ChunkStore;
 
 use crate::api2::admin::{
-    prune::list_prune_jobs, sync::list_sync_jobs, verify::list_verification_jobs,
+    datastore::do_mount_device, prune::list_prune_jobs, sync::list_sync_jobs,
+    verify::list_verification_jobs,
 };
 use crate::api2::config::prune::{delete_prune_job, do_create_prune_job};
 use crate::api2::config::sync::delete_sync_job;
@@ -71,13 +72,30 @@ pub(crate) fn do_create_datastore(
     mut config: SectionConfigData,
     datastore: DataStoreConfig,
 ) -> Result<(), Error> {
-    let path: PathBuf = datastore.path.clone().into();
+    let path: PathBuf = datastore.absolute_path().into();
+    let backup_user = pbs_config::backup_user()?;
+    if let Some(store_mount_point) = datastore.get_mount_point() {
+        let default_options = proxmox_sys::fs::CreateOptions::new();
+        proxmox_sys::fs::create_path(
+            store_mount_point,
+            Some(default_options.clone()),
+            Some(default_options.clone()),
+        )?;
+        do_mount_device(datastore.clone())?;
+
+        if path.join(".chunks").is_dir() {
+            config.set_data(&datastore.name, "datastore", &datastore)?;
+            pbs_config::datastore::save_config(&config)?;
+            jobstate::create_state_file("garbage_collection", &datastore.name)?;
+            info!("created removable datastore, chunkstore already exists");
+            return Ok(());
+        }
+    }
 
     let tuning: DatastoreTuning = serde_json::from_value(
         DatastoreTuning::API_SCHEMA
             .parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
     )?;
-    let backup_user = pbs_config::backup_user()?;
     let _store = ChunkStore::create(
         &datastore.name,
         path,
@@ -120,6 +138,17 @@ pub fn create_datastore(
         param_bail!("name", "datastore '{}' already exists.", config.name);
     }
 
+    if let Some(uuid) = &config.backing_device {
+        for (datastore_name, (_, config)) in &section_config.sections {
+            if config["backing-device"].as_str() == Some(uuid) {
+                param_bail!(
+                    "backing-device",
+                    "device already in use by datastore '{datastore_name}'",
+                );
+            }
+        }
+    }
+
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
 
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag

[Hannes Laimer]

... used by the partition selector for removable datastore creation.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/node/disks/mod.rs |  8 +++++
 src/tools/disks/mod.rs     | 60 ++++++++++++++++++++++++++++++++------
 2 files changed, 59 insertions(+), 9 deletions(-)

diff --git a/src/api2/node/disks/mod.rs b/src/api2/node/disks/mod.rs
index 4ef4ee2b..5bb7eb94 100644
--- a/src/api2/node/disks/mod.rs
+++ b/src/api2/node/disks/mod.rs
@@ -41,6 +41,12 @@ pub mod zfs;
                 optional: true,
                 default: false,
             },
+            "exclude-used": {
+                description: "Exclude partitions already used for removable datastores or mounted directories.",
+                type: bool,
+                optional: true,
+                default: false,
+            },
             "usage-type": {
                 type: DiskUsageType,
                 optional: true,
@@ -62,6 +68,7 @@ pub mod zfs;
 pub fn list_disks(
     skipsmart: bool,
     include_partitions: bool,
+    exclude_used: bool,
     usage_type: Option<DiskUsageType>,
 ) -> Result<Vec<DiskUsageInfo>, Error> {
     let mut list = Vec::new();
@@ -69,6 +76,7 @@ pub fn list_disks(
     for (_, info) in DiskUsageQuery::new()
         .smart(!skipsmart)
         .partitions(include_partitions)
+        .exclude_used(exclude_used)
         .query()?
     {
         if let Some(ref usage_type) = usage_type {
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 93c1a882..8d84e312 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -30,6 +30,7 @@ pub use zpool_list::*;
 mod lvm;
 pub use lvm::*;
 mod smart;
+use crate::api2::node::disks::directory::list_datastore_mounts;
 pub use smart::*;
 
 static ISCSI_PATH_REGEX: LazyLock<regex::Regex> =
@@ -824,6 +825,7 @@ fn scan_partitions(
 pub struct DiskUsageQuery {
     smart: bool,
     partitions: bool,
+    exclude_used: bool,
 }
 
 impl DiskUsageQuery {
@@ -831,6 +833,7 @@ impl DiskUsageQuery {
         Self {
             smart: true,
             partitions: false,
+            exclude_used: false,
         }
     }
 
@@ -844,12 +847,22 @@ impl DiskUsageQuery {
         self
     }
 
+    pub fn exclude_used(&mut self, exclude_used: bool) -> &mut Self {
+        self.exclude_used = exclude_used;
+        self
+    }
+
     pub fn query(&self) -> Result<HashMap<String, DiskUsageInfo>, Error> {
-        get_disks(None, !self.smart, self.partitions)
+        get_disks(None, !self.smart, self.partitions, self.exclude_used)
     }
 
     pub fn find(&self, disk: &str) -> Result<DiskUsageInfo, Error> {
-        let mut map = get_disks(Some(vec![disk.to_string()]), !self.smart, self.partitions)?;
+        let mut map = get_disks(
+            Some(vec![disk.to_string()]),
+            !self.smart,
+            self.partitions,
+            self.exclude_used,
+        )?;
         if let Some(info) = map.remove(disk) {
             Ok(info)
         } else {
@@ -858,7 +871,7 @@ impl DiskUsageQuery {
     }
 
     pub fn find_all(&self, disks: Vec<String>) -> Result<HashMap<String, DiskUsageInfo>, Error> {
-        get_disks(Some(disks), !self.smart, self.partitions)
+        get_disks(Some(disks), !self.smart, self.partitions, self.exclude_used)
     }
 }
 
@@ -931,6 +944,8 @@ fn get_disks(
     no_smart: bool,
     // include partitions
     include_partitions: bool,
+    // skip partitions which are in use
+    exclude_used: bool,
 ) -> Result<HashMap<String, DiskUsageInfo>, Error> {
     let disk_manager = DiskManage::new();
 
@@ -948,6 +963,31 @@ fn get_disks(
 
     // fixme: ceph journals/volumes
 
+    let uuids_in_use = if exclude_used && include_partitions {
+        let (config, _digest) = pbs_config::datastore::config()?;
+
+        let uuids_from_datastores: Vec<String> = config
+            .sections
+            .iter()
+            .filter_map(|(_, (_, data))| {
+                data.as_object()
+                    .and_then(|cfg| cfg.get("backing-device"))
+                    .and_then(serde_json::Value::as_str)
+                    .map(String::from)
+            })
+            .collect();
+
+        let uuids_from_mounts: Vec<String> = list_datastore_mounts()?
+            .into_iter()
+            // FIXME: include UUID in DatastoreMountInfo
+            .filter_map(|mount| mount.device.split('/').last().map(String::from))
+            .collect();
+
+        [&uuids_from_datastores[..], &uuids_from_mounts[..]].concat()
+    } else {
+        Vec::new()
+    };
+
     let mut result = HashMap::new();
 
     for item in proxmox_sys::fs::scan_subdir(libc::AT_FDCWD, "/sys/block", &BLOCKDEVICE_NAME_REGEX)?
@@ -1020,12 +1060,14 @@ fn get_disks(
 
         let partitions: Option<Vec<PartitionInfo>> = if include_partitions {
             disk.partitions().map_or(None, |parts| {
-                Some(get_partitions_info(
-                    parts,
-                    &lvm_devices,
-                    &zfs_devices,
-                    &file_system_devices,
-                ))
+                Some(
+                    get_partitions_info(parts, &lvm_devices, &zfs_devices, &file_system_devices)
+                        .into_iter()
+                        .filter(|part| {
+                            !part.uuid.as_ref().is_some_and(|u| uuids_in_use.contains(u))
+                        })
+                        .collect(),
+                )
             })
         } else {
             None
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-api-types/src/datastore.rs |  9 ++++++++-
 src/api2/admin/datastore.rs    | 17 +++++++++--------
 src/api2/status.rs             | 18 +++++++++++++++---
 3 files changed, 32 insertions(+), 12 deletions(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index d6e2f97e..6a999372 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -454,6 +454,10 @@ impl DataStoreConfig {
 pub struct DataStoreListItem {
     pub store: String,
     pub comment: Option<String>,
+    /// Datastore is removable
+    pub removable: bool,
+    /// Datastore is available
+    pub available: bool,
     /// If the datastore is in maintenance mode, information about it
     #[serde(skip_serializing_if = "Option::is_none")]
     pub maintenance: Option<String>,
@@ -1453,6 +1457,8 @@ pub struct DataStoreStatusListItem {
     /// The available bytes of the underlying storage. (-1 on error)
     #[serde(skip_serializing_if = "Option::is_none")]
     pub avail: Option<u64>,
+    /// The datastore is available, relevant if removable
+    pub is_available: bool,
     /// A list of usages of the past (last Month).
     #[serde(skip_serializing_if = "Option::is_none")]
     pub history: Option<Vec<Option<f64>>>,
@@ -1477,12 +1483,13 @@ pub struct DataStoreStatusListItem {
 }
 
 impl DataStoreStatusListItem {
-    pub fn empty(store: &str, err: Option<String>) -> Self {
+    pub fn empty(store: &str, err: Option<String>, is_available: bool) -> Self {
         DataStoreStatusListItem {
             store: store.to_owned(),
             total: None,
             used: None,
             avail: None,
+            is_available,
             history: None,
             history_start: None,
             history_delta: None,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 3c95888d..0b5dede7 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -1310,8 +1310,8 @@ pub fn get_datastore_list(
 
     let mut list = Vec::new();
 
-    for (store, (_, data)) in &config.sections {
-        let acl_path = &["datastore", store];
+    for (store, (_, data)) in config.sections {
+        let acl_path = &["datastore", &store];
         let user_privs = user_info.lookup_privs(&auth_id, acl_path);
         let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
 
@@ -1322,15 +1322,16 @@ pub fn get_datastore_list(
             }
         }
 
+        let store_config: DataStoreConfig = serde_json::from_value(data)?;
+        let is_available = pbs_datastore::is_datastore_available(&store_config);
+
         if allowed || allow_id {
             list.push(DataStoreListItem {
                 store: store.clone(),
-                comment: if !allowed {
-                    None
-                } else {
-                    data["comment"].as_str().map(String::from)
-                },
-                maintenance: data["maintenance-mode"].as_str().map(String::from),
+                comment: store_config.comment.filter(|_| allowed),
+                removable: store_config.backing_device.is_some(),
+                available: is_available,
+                maintenance: store_config.maintenance_mode,
             });
         }
     }
diff --git a/src/api2/status.rs b/src/api2/status.rs
index f1ae0ef5..a0dd46b6 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -13,7 +13,7 @@ use pbs_api_types::{
 };
 
 use pbs_config::CachedUserInfo;
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
 
 use crate::rrd_cache::extract_rrd_data;
 use crate::tools::statistics::linear_regression;
@@ -48,10 +48,17 @@ pub async fn datastore_status(
     for (store, (_, _)) in &config.sections {
         let user_privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
         let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
+
+        let store_config = config.lookup("datastore", store)?;
+        if !is_datastore_available(&store_config) {
+            list.push(DataStoreStatusListItem::empty(store, None, false));
+            continue;
+        }
+
         if !allowed {
             if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
                 if can_access_any_namespace(datastore, &auth_id, &user_info) {
-                    list.push(DataStoreStatusListItem::empty(store, None));
+                    list.push(DataStoreStatusListItem::empty(store, None, true));
                 }
             }
             continue;
@@ -60,7 +67,11 @@ pub async fn datastore_status(
         let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
             Ok(datastore) => datastore,
             Err(err) => {
-                list.push(DataStoreStatusListItem::empty(store, Some(err.to_string())));
+                list.push(DataStoreStatusListItem::empty(
+                    store,
+                    Some(err.to_string()),
+                    true,
+                ));
                 continue;
             }
         };
@@ -71,6 +82,7 @@ pub async fn datastore_status(
             total: Some(status.total),
             used: Some(status.used),
             avail: Some(status.available),
+            is_available: true,
             history: None,
             history_start: None,
             history_delta: None,
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-config/src/datastore.rs                 | 14 ++++
 src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
 2 files changed, 89 insertions(+), 1 deletion(-)

diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
index dc5bb3da..396dcb37 100644
--- a/pbs-config/src/datastore.rs
+++ b/pbs-config/src/datastore.rs
@@ -62,6 +62,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
     }
 }
 
+pub fn complete_removable_datastore_name(
+    _arg: &str,
+    _param: &HashMap<String, String>,
+) -> Vec<String> {
+    match config() {
+        Ok((data, _digest)) => data
+            .sections
+            .into_iter()
+            .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
+            .collect(),
+        Err(_) => Vec::new(),
+    }
+}
+
 pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
     let mut list = vec![
         String::from("/"),
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 383bcd24..f2795b39 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::Error;
+use anyhow::{format_err, Error};
 use serde_json::Value;
 
 use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
     Ok(Value::Null)
 }
 
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+)]
+/// Mount a removable datastore.
+async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    param["node"] = "localhost".into();
+
+    let info = &api2::admin::datastore::API_METHOD_MOUNT;
+    let result = match info.handler {
+        ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
+        _ => unreachable!(),
+    };
+
+    crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+    Ok(())
+}
+
 #[api(
     input: {
         properties: {
@@ -99,6 +127,34 @@ async fn create_datastore(mut param: Value) -> Result<Value, Error> {
     Ok(Value::Null)
 }
 
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+)]
+/// Unmount a removable datastore.
+async fn unmount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+    param["node"] = "localhost".into();
+
+    let info = &api2::admin::datastore::API_METHOD_UNMOUNT;
+    let result = match info.handler {
+        ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
+        _ => unreachable!(),
+    };
+
+    crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+    Ok(())
+}
+
 #[api(
     protected: true,
     input: {
@@ -142,6 +198,15 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
 pub fn datastore_commands() -> CommandLineInterface {
     let cmd_def = CliCommandMap::new()
         .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
+        .insert(
+            "mount",
+            CliCommand::new(&API_METHOD_MOUNT_DATASTORE)
+                .arg_param(&["store"])
+                .completion_cb(
+                    "store",
+                    pbs_config::datastore::complete_removable_datastore_name,
+                ),
+        )
         .insert(
             "show",
             CliCommand::new(&API_METHOD_SHOW_DATASTORE)
@@ -152,6 +217,15 @@ pub fn datastore_commands() -> CommandLineInterface {
             "create",
             CliCommand::new(&API_METHOD_CREATE_DATASTORE).arg_param(&["name", "path"]),
         )
+        .insert(
+            "unmount",
+            CliCommand::new(&API_METHOD_UNMOUNT_DATASTORE)
+                .arg_param(&["store"])
+                .completion_cb(
+                    "store",
+                    pbs_config::datastore::complete_removable_datastore_name,
+                ),
+        )
         .insert(
             "update",
             CliCommand::new(&api2::config::datastore::API_METHOD_UPDATE_DATASTORE)
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 debian/proxmox-backup-server.install        |  1 +
 debian/proxmox-backup-server.udev           |  3 ++
 etc/Makefile                                |  3 +-
 etc/removable-device-attach@.service.in     |  8 +++
 src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
 5 files changed, 68 insertions(+), 2 deletions(-)
 create mode 100644 etc/removable-device-attach@.service.in

diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
index ef1e9ba1..34bfc454 100644
--- a/debian/proxmox-backup-server.install
+++ b/debian/proxmox-backup-server.install
@@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
 etc/proxmox-backup-daily-update.timer /lib/systemd/system/
 etc/proxmox-backup-proxy.service /lib/systemd/system/
 etc/proxmox-backup.service /lib/systemd/system/
+etc/removable-device-attach@.service /lib/systemd/system/
 usr/bin/pmt
 usr/bin/pmtx
 usr/bin/proxmox-tape
diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
index afdfb2bc..e21b8bc7 100644
--- a/debian/proxmox-backup-server.udev
+++ b/debian/proxmox-backup-server.udev
@@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
   SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
 
 LABEL="persistent_storage_tape_end"
+
+# triggers the mounting of a removable device
+ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
\ No newline at end of file
diff --git a/etc/Makefile b/etc/Makefile
index 42f639f6..730de4f8 100644
--- a/etc/Makefile
+++ b/etc/Makefile
@@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
 	proxmox-backup-banner.service \
 	proxmox-backup-daily-update.service \
 	proxmox-backup.service \
-	proxmox-backup-proxy.service
+	proxmox-backup-proxy.service \
+	removable-device-attach@.service
 
 all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
 
diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
new file mode 100644
index 00000000..e10d1ea3
--- /dev/null
+++ b/etc/removable-device-attach@.service.in
@@ -0,0 +1,8 @@
+[Unit]
+Description=Try to mount the removable device of a datastore with uuid '%i'.
+After=proxmox-backup-proxy.service
+Requires=proxmox-backup-proxy.service
+
+[Service]
+Type=simple
+ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index f2795b39..e74c8537 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::{format_err, Error};
+use anyhow::{bail, format_err, Error};
 use serde_json::Value;
 
 use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
     Ok(())
 }
 
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            uuid: {
+                type: String,
+                description: "The UUID of the device that should be mounted",
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        },
+    },
+)]
+/// Try mounting a removable datastore given the UUID.
+async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let uuid = param["uuid"]
+        .as_str()
+        .ok_or_else(|| format_err!("uuid has to be specified"))?;
+
+    let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
+    let data: Value = match info.handler {
+        ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
+        _ => unreachable!(),
+    };
+
+    let store_name = data.as_array().and_then(|list| {
+        list.iter()
+            .filter_map(Value::as_object)
+            .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
+            .and_then(|s| s.get("name").and_then(Value::as_str))
+    });
+
+    if let Some(store_name) = store_name {
+        let info = &api2::admin::datastore::API_METHOD_MOUNT;
+        let mount_param = serde_json::json!({
+            "store": store_name,
+        });
+        let result = match info.handler {
+            ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
+            _ => unreachable!(),
+        };
+        crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+        return Ok(Value::Null);
+    }
+    bail!("'{uuid}' is not associated with any datastore")
+}
+
 pub fn datastore_commands() -> CommandLineInterface {
     let cmd_def = CliCommandMap::new()
         .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
@@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
                     pbs_config::datastore::complete_calendar_event,
                 ),
         )
+        .insert(
+            "uuid-mount",
+            CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
+        )
         .insert(
             "remove",
             CliCommand::new(&API_METHOD_DELETE_DATASTORE)
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pbs-datastore/src/datastore.rs |  4 +++-
 src/api2/config/datastore.rs   | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 29f98b37..df038d62 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1510,7 +1510,9 @@ impl DataStore {
                         // weird, but ok
                     }
                     Err(err) if err.is_errno(nix::errno::Errno::EBUSY) => {
-                        warn!("Cannot delete datastore directory (is it a mount point?).")
+                        if datastore_config.backing_device.is_none() {
+                            warn!("Cannot delete datastore directory (is it a mount point?).")
+                        }
                     }
                     Err(err) if err.is_errno(nix::errno::Errno::ENOTEMPTY) => {
                         warn!("Datastore directory not empty, not deleting.")
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index c24b6e9d..c7d59c44 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -29,9 +29,11 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
 use crate::api2::config::verify::delete_verification_job;
 use pbs_config::CachedUserInfo;
 
+use pbs_datastore::is_datastore_available;
 use proxmox_rest_server::WorkerTask;
 
 use crate::server::jobstate;
+use crate::tools::disks::unmount_by_mountpoint;
 
 #[api(
     input: {
@@ -523,6 +525,14 @@ pub async fn delete_datastore(
         http_bail!(NOT_FOUND, "datastore '{}' does not exist.", name);
     }
 
+    let store_config: DataStoreConfig = config.lookup("datastore", &name)?;
+    if destroy_data && !is_datastore_available(&store_config) {
+        http_bail!(
+            BAD_REQUEST,
+            "cannot destroy data on '{name}' unless the datastore is mounted"
+        );
+    }
+
     if !keep_job_configs {
         for job in list_verification_jobs(Some(name.clone()), Value::Null, rpcenv)? {
             delete_verification_job(job.config.id, None, rpcenv)?
@@ -549,6 +559,22 @@ pub async fn delete_datastore(
 
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+    let name_copy = name.clone();
+    tokio::spawn(async move {
+        if let Ok(proxy_pid) =
+            proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+        {
+            let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
+            let _ = proxmox_daemon::command_socket::send_raw(
+                sock,
+                &format!(
+                    "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
+                    name_copy
+                ),
+            )
+            .await;
+        }
+    });
 
     let upid = WorkerTask::new_thread(
         "delete-datastore",
@@ -561,6 +587,12 @@ pub async fn delete_datastore(
             // ignore errors
             let _ = jobstate::remove_state_file("prune", &name);
             let _ = jobstate::remove_state_file("garbage_collection", &name);
+            if destroy_data {
+                if let Some(mount_point) = store_config.get_mount_point() {
+                    let _ = unmount_by_mountpoint(&mount_point);
+                    let _ = std::fs::remove_dir(&mount_point);
+                }
+            }
 
             if let Err(err) =
                 proxmox_async::runtime::block_on(crate::server::notify_datastore_removed())
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 docs/storage.rst | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/docs/storage.rst b/docs/storage.rst
index f1e15d52..8713352a 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -165,6 +165,35 @@ following command creates a new datastore called ``store1`` on
   # proxmox-backup-manager datastore create store1 /backup/disk1/store1
 
 
+Removable Datastores
+^^^^^^^^^^^^^^^^^^^^
+Removable datastores have a ``backing-device`` associated with them, they can be
+mounted and unmounted. Other than that they behave the same way a normal datastore
+would.
+
+They can be created on already correctly formatted partitions, which, as with normal
+datastores, should be either ``ext4`` or ``xfs``.  It is also possible to create them
+on completely unused disks through "Admnistartion" > "Disks / Storage" > "Directory",
+using this method the disk will be partitioned and formatted automatically for the datastore.
+
+They are mounted automatically when plugged in. It is possible to create a
+removable datastore on one PBS and use it on multiple, the device just has to be added
+on all as a removable datastore. This can be done using the normal `Create Datastore`
+form, by checking **Removable datastore**, selecting the device and setting the correct path.
+If the device already contains a datastore at the specified path it'll just be added as
+a new datastore to the PBS instance and will be mounted whenever plugged in. Unmounting has
+to be done through the UI by clicking "Unmount" on the summary page or using the CLI
+
+.. code-block:: console
+
+  # proxmox-backup-manager datastore unmount store1
+
+both will wait for any running tasks to finish and unmount the device.
+
+All removable datastores are mounted under /mnt/datastore/<name>, and the specified path
+refers to the path on the device.
+
+
 Managing Datastores
 ^^^^^^^^^^^^^^^^^^^
 
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 15/26] ui: add partition selector form

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/Makefile                  |  1 +
 www/form/PartitionSelector.js | 81 +++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+)
 create mode 100644 www/form/PartitionSelector.js

diff --git a/www/Makefile b/www/Makefile
index 609a0ba6..45adfc54 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -49,6 +49,7 @@ JSSRC=							\
 	form/NamespaceMaxDepth.js			\
 	form/CalendarEvent.js				\
 	form/PermissionPathSelector.js			\
+	form/PartitionSelector.js			\
 	form/GroupSelector.js				\
 	form/GroupFilter.js				\
 	form/VerifyOutdatedAfter.js			\
diff --git a/www/form/PartitionSelector.js b/www/form/PartitionSelector.js
new file mode 100644
index 00000000..3d6eadd5
--- /dev/null
+++ b/www/form/PartitionSelector.js
@@ -0,0 +1,81 @@
+Ext.define('pbs-partition-list', {
+    extend: 'Ext.data.Model',
+    fields: ['name', 'uuid', 'filesystem', 'devpath', 'size', 'model'],
+    proxy: {
+	type: 'proxmox',
+	url: "/api2/json/nodes/localhost/disks/list?skipsmart=1&include-partitions=1&exclude-used=1",
+	reader: {
+	    transform: (rawData) => rawData.data
+		.flatMap(disk => (disk.partitions
+			.map(part => ({ ...part, model: disk.model })) ?? [])
+			.filter(partition => partition.used === 'filesystem')),
+	},
+    },
+    idProperty: 'devpath',
+
+});
+
+Ext.define('PBS.form.PartitionSelector', {
+    extend: 'Proxmox.form.ComboGrid',
+    alias: 'widget.pbsPartitionSelector',
+
+    allowBlank: false,
+    autoSelect: false,
+    submitEmpty: false,
+    valueField: 'uuid',
+    displayField: 'devpath',
+
+    store: {
+	model: 'pbs-partition-list',
+	autoLoad: true,
+	sorters: 'devpath',
+    },
+    getSubmitData: function() {
+	let me = this;
+	let data = null;
+	if (!me.disabled && me.submitValue && !me.isFileUpload()) {
+	    let val = me.getSubmitValue();
+	    if (val !== undefined && val !== null && val !== '') {
+		data = {};
+		data[me.getName()] = val;
+	    } else if (me.getDeleteEmpty()) {
+		data = {};
+		data.delete = me.getName();
+	    }
+	}
+	return data;
+    },
+    listConfig: {
+	columns: [
+	    {
+		header: gettext('Path'),
+		sortable: true,
+		dataIndex: 'devpath',
+		renderer: (v, metaData, rec) => Ext.String.htmlEncode(v),
+		flex: 1,
+	    },
+	    {
+		header: gettext('Filesystem'),
+		sortable: true,
+		dataIndex: 'filesystem',
+		flex: 1,
+	    },
+	    {
+		header: gettext('Size'),
+		sortable: true,
+		dataIndex: 'size',
+		renderer: Proxmox.Utils.format_size,
+		flex: 1,
+	    },
+	    {
+		header: gettext('Model'),
+		sortable: true,
+		dataIndex: 'model',
+		flex: 1,
+	    },
+	],
+	viewConfig: {
+	    emptyText: 'No usable partitions present',
+	},
+    },
+});
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 16/26] ui: add removable datastore creation support

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/window/DataStoreEdit.js | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index b6115460..54929382 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -62,6 +62,20 @@ Ext.define('PBS.DataStoreEdit', {
 			fieldLabel: gettext('Backing Path'),
 			emptyText: gettext('An absolute path'),
 		    },
+		    {
+			xtype: 'pmxDisplayEditField',
+			fieldLabel: gettext('Device'),
+			name: 'backing-device',
+			disabled: true,
+			cbind: {
+			    editable: '{isCreate}',
+			},
+			editConfig: {
+			    xtype: 'pbsPartitionSelector',
+			    allowBlank: true,
+			},
+			emptyText: gettext('Device path'),
+		    },
 		],
 		column2: [
 		    {
@@ -87,6 +101,29 @@ Ext.define('PBS.DataStoreEdit', {
 		    },
 		],
 		columnB: [
+		    {
+			xtype: 'checkbox',
+			boxLabel: gettext('Removable datastore'),
+			submitValue: false,
+			listeners: {
+			    change: function(checkbox, isRemovable) {
+				let inputPanel = checkbox.up('inputpanel');
+				let pathField = inputPanel.down('[name=path]');
+				let uuidField = inputPanel.down('pbsPartitionSelector[name=backing-device]');
+				let uuidEditField = inputPanel.down('[name=backing-device]');
+
+				uuidField.allowBlank = !isRemovable;
+				uuidEditField.setDisabled(!isRemovable);
+				uuidField.setDisabled(!isRemovable);
+				uuidField.setValue('');
+				if (isRemovable) {
+				    pathField.setFieldLabel(gettext('On device path'));
+				} else {
+				    pathField.setFieldLabel(gettext('Backing Path'));
+				}
+			    },
+			},
+		    },
 		    {
 			xtype: 'textfield',
 			name: 'comment',
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 17/26] ui: add (un)mount button to summary

[Hannes Laimer]

And only try to load datastore information if the datastore is
available.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/datastore/Summary.js | 92 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 90 insertions(+), 2 deletions(-)

diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index a932b4e0..49aa3b3c 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -309,7 +309,82 @@ Ext.define('PBS.DataStoreSummary', {
 	    model: 'pve-rrd-datastore',
 	});
 
-	me.callParent();
+	me.statusStore = Ext.create('Proxmox.data.ObjectStore', {
+	    url: `/api2/json/admin/datastore/${me.datastore}/status`,
+	    interval: 1000,
+	});
+
+	let unmountBtn = Ext.create('Ext.Button', {
+	    text: gettext('Unmount'),
+	    hidden: true,
+	    handler: () => {
+		Proxmox.Utils.API2Request({
+		    url: `/admin/datastore/${me.datastore}/unmount`,
+		    method: 'POST',
+		    failure: function(response) {
+			Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+		    },
+		    success: function(response, options) {
+			Ext.create('Proxmox.window.TaskViewer', {
+			    upid: response.result.data,
+			}).show();
+		    },
+		});
+	    },
+	});
+
+	let mountBtn = Ext.create('Ext.Button', {
+	    text: gettext('Mount'),
+	    hidden: true,
+	    handler: () => {
+		Proxmox.Utils.API2Request({
+		    url: `/admin/datastore/${me.datastore}/mount`,
+		    method: 'POST',
+		    failure: function(response) {
+			Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+		    },
+		    success: function(response, options) {
+			Ext.create('Proxmox.window.TaskViewer', {
+			    upid: response.result.data,
+			}).show();
+		    },
+		});
+	    },
+	});
+
+	Ext.apply(me, {
+	    tbar: [unmountBtn, mountBtn, '->', { xtype: 'proxmoxRRDTypeSelector' }],
+	});
+
+	me.mon(me.statusStore, 'load', (s, records, success) => {
+	    if (!success) {
+		me.down('pbsDataStoreInfo').fireEvent('deactivate');
+		Proxmox.Utils.API2Request({
+		    url: `/config/datastore/${me.datastore}`,
+		    success: response => {
+			let mode = response.result.data['maintenance-mode'];
+			let [type, _message] = PBS.Utils.parseMaintenanceMode(mode);
+			if (!response.result.data['backing-device']) {
+			    return;
+			}
+			if (!type || type === 'read-only') {
+			    unmountBtn.setDisabled(true);
+			    mountBtn.setDisabled(false);
+			} else if (type === 'unmount') {
+			    unmountBtn.setDisabled(true);
+			    mountBtn.setDisabled(true);
+			} else {
+			    unmountBtn.setDisabled(false);
+			    mountBtn.setDisabled(false);
+			}
+		    },
+		});
+	    } else {
+		me.down('pbsDataStoreInfo').fireEvent('activate');
+		unmountBtn.setDisabled(false);
+		mountBtn.setDisabled(true);
+	    }
+	});
 
 	let sp = Ext.state.Manager.getProvider();
 	me.mon(sp, 'statechange', function(provider, key, value) {
@@ -322,11 +397,17 @@ Ext.define('PBS.DataStoreSummary', {
 	    Proxmox.Utils.updateColumns(me);
 	});
 
+	me.callParent();
+
 	Proxmox.Utils.API2Request({
 	    url: `/config/datastore/${me.datastore}`,
 	    waitMsgTarget: me.down('pbsDataStoreInfo'),
 	    success: function(response) {
-		let path = Ext.htmlEncode(response.result.data.path);
+		let data = response.result.data;
+		let path = Ext.htmlEncode(data.path);
+		const removable = Object.prototype.hasOwnProperty.call(data, "backing-device");
+		unmountBtn.setHidden(!removable);
+		mountBtn.setHidden(!removable);
 		me.down('pbsDataStoreInfo').setTitle(`${me.datastore} (${path})`);
 		me.down('pbsDataStoreNotes').setNotes(response.result.data.comment);
 	    },
@@ -344,6 +425,13 @@ Ext.define('PBS.DataStoreSummary', {
 	    let hasIoTicks = records?.some((rec) => rec?.data?.io_ticks !== undefined);
 	    me.down('#ioDelayChart').setVisible(!success || hasIoTicks);
 	}, undefined, { single: true });
+	me.on('afterrender', () => {
+	    me.statusStore.startUpdate();
+	});
+
+	me.on('destroy', () => {
+	    me.statusStore.stopUpdate();
+	});
 
 	me.query('proxmoxRRDChart').forEach((chart) => {
 	    chart.setStore(me.rrdstore);
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 18/26] ui: tree: render unmounted datastores correctly

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/NavigationTree.js                 | 15 ++++++++++++---
 www/css/ext6-pbs.css                  |  8 ++++++++
 www/datastore/DataStoreListSummary.js |  1 +
 3 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index 53c8daff..4f1a8fd5 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -267,13 +267,22 @@ Ext.define('PBS.view.main.NavigationTree', {
 		    j++;
 		}
 
-		let [qtip, iconCls] = ['', 'fa fa-database'];
+		let mainIcon = `fa fa-${records[i].data.removable ? 'plug' : 'database'}`;
+		let [qtip, iconCls] = ['', mainIcon];
 		const maintenance = records[i].data.maintenance;
+		const removable_not_available = records[i].data.removable && !records[i].data.available;
+		if (removable_not_available) {
+		    iconCls = `${mainIcon} pmx-tree-icon-custom unplugged`;
+		    qtip = gettext('Removable datastore not mounted');
+		}
 		if (maintenance) {
 		    const [type, message] = PBS.Utils.parseMaintenanceMode(maintenance);
 		    qtip = `${type}${message ? ': ' + message : ''}`;
-		    let maintenanceTypeCls = type === 'delete' ? 'destroying' : 'maintenance';
-		    iconCls = `fa fa-database pmx-tree-icon-custom ${maintenanceTypeCls}`;
+		    let maintenanceTypeCls = 'maintenance';
+		    if (type === 'delete') {
+			maintenanceTypeCls = 'destroying';
+		    }
+		    iconCls = `${mainIcon} pmx-tree-icon-custom ${maintenanceTypeCls}`;
 		}
 
 		if (getChildTextAt(j).localeCompare(name) !== 0) {
diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index c33ce684..706e681e 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -271,6 +271,10 @@ span.snapshot-comment-column {
     content: "\ ";
 }
 
+.x-treelist-item-icon.fa-plug, .pmx-tree-icon-custom.fa-plug {
+    font-size: 12px;
+}
+
 /* datastore maintenance */
 .pmx-tree-icon-custom.maintenance:after {
     content: "\f0ad";
@@ -290,6 +294,10 @@ span.snapshot-comment-column {
     color: #888;
 }
 
+.pmx-tree-icon-custom.unplugged:before {
+    color: #888;
+}
+
 /*' PBS specific icons */
 
 .pbs-icon-tape {
diff --git a/www/datastore/DataStoreListSummary.js b/www/datastore/DataStoreListSummary.js
index b908034d..f7ea83e7 100644
--- a/www/datastore/DataStoreListSummary.js
+++ b/www/datastore/DataStoreListSummary.js
@@ -22,6 +22,7 @@ Ext.define('PBS.datastore.DataStoreListSummary', {
 	    stillbad: 0,
 	    deduplication: 1.0,
 	    error: "",
+	    removable: false,
 	    maintenance: '',
 	},
     },
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 19/26] ui: utils: make parseMaintenanceMode more robust

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/Utils.js | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/www/Utils.js b/www/Utils.js
index 4853be36..7756e9b5 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -740,14 +740,29 @@ Ext.define('PBS.Utils', {
 	return `${icon} ${value}`;
     },
 
-    // FIXME: this "parser" is brittle and relies on the order the arguments will appear in
+    /**
+     * Parses maintenance mode property string.
+     * Examples:
+     *   "offline,message=foo" -> ["offline", "foo"]
+     *   "offline" -> ["offline", null]
+     *   "message=foo,offline" -> ["offline", "foo"]
+     *   null/undefined -> [null, null]
+     *
+     * @param {string|null} mode - Maintenance mode string to parse.
+     * @return {Array<string|null>} - Parsed maintenance mode values.
+     */
     parseMaintenanceMode: function(mode) {
-	let [type, message] = mode.split(/,(.+)/);
-	type = type.split("=").pop();
-	message = message ? message.split("=")[1]
-	    .replace(/^"(.*)"$/, '$1')
-	    .replaceAll('\\"', '"') : null;
-	return [type, message];
+	if (!mode) {
+	    return [null, null];
+	}
+	return mode.split(',').reduce(([m, msg], pair) => {
+	    const [key, value] = pair.split('=');
+	    if (key === 'message') {
+		return [m, value.replace(/^"(.*)"$/, '$1').replace(/\\"/g, '"')];
+	    } else {
+		return [value ?? key, msg];
+	    }
+	}, [null, null]);
     },
 
     renderMaintenance: function(mode, activeTasks) {
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 20/26] ui: add datastore status mask for unmounted removable datastores

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/css/ext6-pbs.css     | 12 ++++++++++++
 www/datastore/Summary.js | 21 +++++++++++++--------
 2 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index 706e681e..891189ae 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -261,6 +261,18 @@ span.snapshot-comment-column {
     content: "\f0ad";
 }
 
+.pbs-unplugged-mask div.x-mask-msg-text {
+    background: None;
+    padding: 12px 0 0;
+}
+
+.pbs-unplugged-mask:before {
+    font-size: 3em;
+    display: flex;
+    justify-content: center;
+    content: "\f1e6";
+}
+
 /* the small icons TODO move to proxmox-widget-toolkit */
 .pmx-tree-icon-custom:after {
     position: relative;
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index 49aa3b3c..33b4e18a 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -61,16 +61,21 @@ Ext.define('PBS.DataStoreInfo', {
 		Proxmox.Utils.API2Request({
 		    url: `/config/datastore/${me.view.datastore}`,
 		    success: function(response) {
-			const config = response.result.data;
-			if (config['maintenance-mode']) {
-			    const [_type, msg] = PBS.Utils.parseMaintenanceMode(config['maintenance-mode']);
-			    me.view.el.mask(
-				`${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`,
-				'fa pbs-maintenance-mask',
-			    );
-			} else {
+			let maintenanceString = response.result.data['maintenance-mode'];
+			let removable = !!response.result.data['backing-device'];
+			if (!maintenanceString && !removable) {
 			    me.view.el.mask(gettext('Datastore is not available'));
+			    return;
 			}
+
+			let [_type, msg] = PBS.Utils.parseMaintenanceMode(maintenanceString);
+			let isUnplugged = !maintenanceString && removable;
+			let maskMessage = isUnplugged
+			    ? gettext('Datastore is not mounted')
+			    : `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`;
+
+			let maskIcon = isUnplugged ? 'fa pbs-unplugged-mask' : 'fa pbs-maintenance-mask';
+			me.view.el.mask(maskMessage, maskIcon);
 		    },
 		});
 		return;
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 21/26] ui: maintenance: fix disable msg field if no type is selected

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/window/MaintenanceOptions.js | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 1ee92542..527c3698 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -56,12 +56,17 @@ Ext.define('PBS.window.MaintenanceOptions', {
 		fieldLabel: gettext('Maintenance Type'),
 		value: '__default__',
 		deleteEmpty: true,
+		listeners: {
+		    change: (field, newValue) => {
+			Ext.getCmp('message-field').setDisabled(newValue === '__default__');
+		    },
+		},
 	    },
 	    {
 		xtype: 'proxmoxtextfield',
+		id: 'message-field',
 		name: 'maintenance-msg',
 		fieldLabel: gettext('Description'),
-		// FIXME: disable if maintenance type is none
 	    },
 	],
     },
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 22/26] ui: render 'unmount' maintenance mode correctly

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 www/Utils.js                     |  4 +++-
 www/window/MaintenanceOptions.js | 10 ++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/www/Utils.js b/www/Utils.js
index 7756e9b5..6bae9b70 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -775,7 +775,7 @@ Ext.define('PBS.Utils', {
 	let extra = '';
 
 	if (activeTasks !== undefined) {
-	    const conflictingTasks = activeTasks.write + (type === 'offline' ? activeTasks.read : 0);
+	    const conflictingTasks = activeTasks.write + (type === 'offline' || type === 'unmount' ? activeTasks.read : 0);
 
 	    if (conflictingTasks > 0) {
 		extra += '| <i class="fa fa-spinner fa-pulse fa-fw"></i> ';
@@ -795,6 +795,8 @@ Ext.define('PBS.Utils', {
 		break;
 	    case 'offline': modeText = gettext("Offline");
 		break;
+	    case 'unmount': modeText = gettext("Unmounting");
+		break;
 	}
 	return `${modeText} ${extra}`;
     },
diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 527c3698..d7348cb4 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -52,6 +52,7 @@ Ext.define('PBS.window.MaintenanceOptions', {
 	items: [
 	    {
 		xtype: 'pbsMaintenanceType',
+		id: 'type-field',
 		name: 'maintenance-type',
 		fieldLabel: gettext('Maintenance Type'),
 		value: '__default__',
@@ -85,6 +86,15 @@ Ext.define('PBS.window.MaintenanceOptions', {
 	    };
 	}
 
+        let unmounting = options['maintenance-type'] === 'unmount';
+        let defaultType = options['maintenance-type'] === '__default__';
+        if (unmounting) {
+            options['maintenance-type'] = '';
+        }
+
 	me.callParent([options]);
+
+        Ext.ComponentManager.get('type-field').setDisabled(unmounting);
+        Ext.ComponentManager.get('message-field').setDisabled(unmounting || defaultType);
     },
 });
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 23/26] api: node: allow creation of removable datastore through directory endpoint

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/node/disks/directory.rs | 57 +++++++++++++++++++++++++++++---
 1 file changed, 52 insertions(+), 5 deletions(-)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index c0c53d84..cd816414 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -123,6 +123,11 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
                 description: "Configure a datastore using the directory.",
                 type: bool,
                 optional: true,
+                default: false,
+            },
+            "removable-datastore": {
+                description: "The added datastore is removable.",
+                type: bool,
             },
             filesystem: {
                 type: FileSystemType,
@@ -141,7 +146,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
 pub fn create_datastore_disk(
     name: String,
     disk: String,
-    add_datastore: Option<bool>,
+    add_datastore: bool,
+    removable_datastore: bool,
     filesystem: Option<FileSystemType>,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<String, Error> {
@@ -155,8 +161,49 @@ pub fn create_datastore_disk(
         bail!("disk '{}' is already in use.", disk);
     }
 
-    let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+    if add_datastore && removable_datastore {
+        let upid_str = WorkerTask::new_thread(
+            "dircreate",
+            Some(name.clone()),
+            auth_id,
+            to_stdout,
+            move |_worker| {
+                info!("create removable datastore '{name}' on disk {disk}");
+
+                let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
+
+                let manager = DiskManage::new();
+
+                let disk = manager.disk_by_name(&disk)?;
+
+                let partition = create_single_linux_partition(&disk)?;
+                create_file_system(&partition, filesystem)?;
+
+                let uuid = get_fs_uuid(&partition)?;
+
+                let lock = pbs_config::datastore::lock_config()?;
+                let datastore: DataStoreConfig = serde_json::from_value(
+                    json!({ "name": name, "path": name, "backing-device": uuid }),
+                )?;
+
+                let (config, _digest) = pbs_config::datastore::config()?;
+
+                if config.sections.get(&datastore.name).is_some() {
+                    bail!("datastore '{}' already exists.", datastore.name);
+                }
+
+                // we don't have to check if the UUID is already in use since we just created the
+                // fs ourself
 
+                crate::api2::config::datastore::do_create_datastore(lock, config, datastore)?;
+
+                Ok(())
+            },
+        )?;
+        return Ok(upid_str);
+    };
+
+    let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
     // check if the default path exists already.
     // bail if it is not empty or another filesystem mounted on top
     let default_path = std::path::PathBuf::from(&mount_point);
@@ -183,7 +230,6 @@ pub fn create_datastore_disk(
         move |_worker| {
             info!("create datastore '{name}' on disk {disk}");
 
-            let add_datastore = add_datastore.unwrap_or(false);
             let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
 
             let manager = DiskManage::new();
@@ -246,8 +292,9 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
     // path of datastore cannot be changed
     let (config, _) = pbs_config::datastore::config()?;
     let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
-    let conflicting_datastore: Option<DataStoreConfig> =
-        datastores.into_iter().find(|ds| ds.absolute_path() == path);
+    let conflicting_datastore: Option<DataStoreConfig> = datastores.into_iter().find(|ds| {
+        ds.absolute_path() == path || ds.get_mount_point().map_or(false, |mp| mp == path)
+    });
 
     if let Some(conflicting_datastore) = conflicting_datastore {
         bail!(
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 24/26] api: node: include removable datastores in directory list

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/node/disks/directory.rs | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index cd816414..ed0bd03d 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -45,6 +45,8 @@ pub struct DatastoreMountInfo {
     pub path: String,
     /// The mounted device.
     pub device: String,
+    /// This is removable
+    pub removable: bool,
     /// File system type
     pub filesystem: Option<String>,
     /// Mount options
@@ -61,7 +63,7 @@ pub struct DatastoreMountInfo {
         }
     },
     returns: {
-        description: "List of systemd datastore mount units.",
+        description: "List of removable-datastore devices and systemd datastore mount units.",
         type: Array,
         items: {
             type: DatastoreMountInfo,
@@ -100,6 +102,31 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
             path: data.Where,
             filesystem: data.Type,
             options: data.Options,
+            removable: false,
+        });
+    }
+
+    let (config, _digest) = pbs_config::datastore::config()?;
+    let store_list: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
+
+    for item in store_list
+        .into_iter()
+        .filter(|store| store.backing_device.is_some())
+    {
+        let Some(backing_device) = item.backing_device.as_deref() else {
+            continue;
+        };
+        let Some(mount_point) = item.get_mount_point() else {
+            continue;
+        };
+        list.push(DatastoreMountInfo {
+            unitfile: "datastore config".to_string(),
+            name: item.name.clone(),
+            device: format!("/dev/disk/by-uuid/{backing_device}"),
+            path: mount_point,
+            filesystem: None,
+            options: None,
+            removable: true,
         });
     }
 
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/node/disks/directory.rs | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index ed0bd03d..2525642d 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -11,8 +11,8 @@ use proxmox_schema::api;
 use proxmox_section_config::SectionConfigData;
 
 use pbs_api_types::{
-    DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_SCHEMA, NODE_SCHEMA, PRIV_SYS_AUDIT,
-    PRIV_SYS_MODIFY, UPID_SCHEMA,
+    DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_MOUNT_DIR, DATASTORE_SCHEMA, NODE_SCHEMA,
+    PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, UPID_SCHEMA,
 };
 
 use crate::tools::disks::{
@@ -23,8 +23,6 @@ use crate::tools::systemd::{self, types::*};
 
 use proxmox_rest_server::WorkerTask;
 
-const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
-
 #[api(
     properties: {
         "filesystem": {
@@ -91,7 +89,7 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
 
         let name = data
             .Where
-            .strip_prefix(BASE_MOUNT_DIR)
+            .strip_prefix(DATASTORE_MOUNT_DIR)
             .unwrap_or(&data.Where)
             .to_string();
 
@@ -230,7 +228,7 @@ pub fn create_datastore_disk(
         return Ok(upid_str);
     };
 
-    let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+    let mount_point = format!("{}/{}", DATASTORE_MOUNT_DIR, &name);
     // check if the default path exists already.
     // bail if it is not empty or another filesystem mounted on top
     let default_path = std::path::PathBuf::from(&mount_point);
@@ -238,7 +236,7 @@ pub fn create_datastore_disk(
     match std::fs::metadata(&default_path) {
         Err(_) => {} // path does not exist
         Ok(stat) => {
-            let basedir_dev = std::fs::metadata(BASE_MOUNT_DIR)?.st_dev();
+            let basedir_dev = std::fs::metadata(DATASTORE_MOUNT_DIR)?.st_dev();
             if stat.st_dev() != basedir_dev {
                 bail!("path {default_path:?} already exists and is mountpoint");
             }
@@ -315,7 +313,7 @@ pub fn create_datastore_disk(
 )]
 /// Remove a Filesystem mounted under `/mnt/datastore/<name>`.
 pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
-    let path = format!("{}{}", BASE_MOUNT_DIR, name);
+    let path = format!("{}/{}", DATASTORE_MOUNT_DIR, name);
     // path of datastore cannot be changed
     let (config, _) = pbs_config::datastore::config()?;
     let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v12 26/26] ui: support create removable datastore through directory creation

[Hannes Laimer]

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/api2/node/disks/directory.rs |  2 ++
 www/DirectoryList.js             | 13 +++++++++++++
 www/window/CreateDirectory.js    | 14 ++++++++++++++
 3 files changed, 29 insertions(+)

diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 2525642d..356e102f 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -153,6 +153,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
             "removable-datastore": {
                 description: "The added datastore is removable.",
                 type: bool,
+                optional: true,
+                default: false,
             },
             filesystem: {
                 type: FileSystemType,
diff --git a/www/DirectoryList.js b/www/DirectoryList.js
index adefa9ab..25921a62 100644
--- a/www/DirectoryList.js
+++ b/www/DirectoryList.js
@@ -121,6 +121,19 @@ Ext.define('PBS.admin.Directorylist', {
     ],
 
     columns: [
+	{
+	    text: '<span class="fa fa-plug"/>',
+	    flex: 0,
+	    width: 35,
+	    dataIndex: 'removable',
+	    renderer: function(_text, _, row) {
+		if (row.data.removable) {
+		    return `<i class="fa fa-check"/>`;
+		} else {
+		    return '';
+		}
+	    },
+	},
 	{
 	    text: gettext('Path'),
 	    dataIndex: 'path',
diff --git a/www/window/CreateDirectory.js b/www/window/CreateDirectory.js
index 6aabe21a..38d6979d 100644
--- a/www/window/CreateDirectory.js
+++ b/www/window/CreateDirectory.js
@@ -43,6 +43,20 @@ Ext.define('PBS.window.CreateDirectory', {
 	    name: 'add-datastore',
 	    fieldLabel: gettext('Add as Datastore'),
 	    value: '1',
+	    listeners: {
+		change(field, newValue, _oldValue) {
+		    let form = field.up('form');
+		    let rmBox = form.down('[name=removable-datastore]');
+
+		    rmBox.setDisabled(!newValue);
+		    rmBox.setValue(false);
+		},
+	    },
+	},
+	{
+	    xtype: 'proxmoxcheckbox',
+	    name: 'removable-datastore',
+	    fieldLabel: gettext('is removable'),
 	},
     ],
 });
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel