* [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores
@ 2024-04-25 6:52 Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions Hannes Laimer
` (27 more replies)
0 siblings, 28 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
These patches add support for removable datastores. All removable
datastores have a backing-device(a UUID) associated with them. Removable
datastores work like normal ones, just that they can be unplugged. It is
possible to create a removable datastore, sync backups onto it, unplug
it and use it on a different PBS.
The datastore path specified is relative to `/mnt/removable_datastore/<name>`.
When a removable datastore is deleted and 'destroy-data' is set, the
device has to be mounted in. If 'destroy-data' is not set the datastore
can be deleted even if the device is not present. Removable datastores
are automatically mounted when plugged in.
v10: thanks @Gabriel and @Wolfgang
* make is_datastore_available more robust
* fix a lot of wording
* drop format on uuid_mount command for UUID
* only gather_disk_stats if datastore is available
* overall code improvements
* ui: include model in partition selector
* rebased onto master
v9:
* change mount point to `/mnt/datastore/<name>`
* update "Directory" list UI
* add `absolute_path()` from Dietmar's RFC
* update docs
v8:
* still depends on [1]
* paths for removable datastores are now relative to
`/mnt/removable_datastore/<UUID>`
* add support for creation of removable datastore through the
"create directory" endpoint (last 3 patches)
* update datastore creation UI
* update docs
v7:
* depends on [1]
* improve logging when waiting for tasks
* drop `update-datatore-cache` refactoring
* fix some commit messages
[1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
v6:
* remove 'drop' flag in datastore cache
* use maintenance-mode 'unmount' for unmounting process, only for the
unmounting not for being unmounted
* rename/simplify update-datastore-cache command
* ui: integrate new unmounting maintenance mode
* basically a mix of v3 and v4
v5: thanks @Dietmar and @Christian
* drop --force for unmount since it'll always fail if tasks are still running, and if
there are not normal unount will work
* improve several commit messages
* improve error message wording
* add removable datastore section to docs
* add documentation for is_datastore_available
v4: thanks a lot @Dietmar and @Christian
* make check if mounted wayyy faster
* don't keep track of mounting state
* drop Unplugged maintenance mode
* use UUID_FORMAT for uuid field
* a lot of small things, like use of bail!, inline format!, ...
* include improvement to cache handling
v3:
* remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
* fix CLI (un)mount command, thanks @Gabriel
* add removable datastore CLI autocomplete helper
* rebase onto master
* move ui patches to the end
thanks @Lukas and @Thomas for the feedback
v2:
* fix datastore 'add' button in the UI
* some format!("{}", a) -> format!("{a}")
* replace `const` with `let` in js code
* change icon `fa-usb` -> `fa-plug`
* add some docs
* add JDoc for parseMaintenanceMode
* proxmox-schema dep bump
Dietmar Maurer (2):
config: factor out method to get the absolute datastore path
maintenance: add 'Unmount' maintenance type
Hannes Laimer (24):
tools: add disks utility functions
pbs-api-types: add backing-device to DataStoreConfig
disks: add UUID to partition info
datastore: add helper for checking if a removable datastore is
available
api: admin: add (un)mount endpoint for removable datastores
api: removable datastore creation
api: disks list: add exclude-used flag
pbs-api-types: add removable/is-available flag to DataStoreListItem
bin: manager: add (un)mount command
add auto-mounting for removable datastores
datastore: handle deletion of removable datastore properly
docs: add removable datastores section
ui: add partition selector form
ui: add removable datastore creation support
ui: add (un)mount button to summary
ui: tree: render unmounted datastores correctly
ui: utils: make parseMaintenanceMode more robust
ui: add datastore status mask for unmounted removable datastores
ui: maintenance: fix disable msg field if no type is selected
ui: render 'unmount' maintenance mode correctly
api: node: allow creation of removable datastore through directory
endpoint
api: node: include removable datastores in directory list
node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
ui: support create removable datastore through directory creation
debian/proxmox-backup-server.install | 1 +
debian/proxmox-backup-server.udev | 3 +
docs/storage.rst | 29 +++
etc/Makefile | 3 +-
etc/removable-device-attach@.service.in | 8 +
pbs-api-types/src/datastore.rs | 42 +++-
pbs-api-types/src/maintenance.rs | 11 +-
pbs-config/src/datastore.rs | 14 ++
pbs-datastore/src/datastore.rs | 79 +++++++-
pbs-datastore/src/lib.rs | 2 +-
src/api2/admin/datastore.rs | 203 ++++++++++++++++++--
src/api2/config/datastore.rs | 69 ++++++-
src/api2/node/disks/directory.rs | 112 +++++++++--
src/api2/node/disks/mod.rs | 8 +
src/api2/status.rs | 18 +-
src/bin/proxmox-backup-proxy.rs | 9 +-
src/bin/proxmox_backup_manager/datastore.rs | 129 ++++++++++++-
src/tools/disks/mod.rs | 89 ++++++++-
www/DirectoryList.js | 13 ++
www/Makefile | 1 +
www/NavigationTree.js | 15 +-
www/Utils.js | 33 +++-
www/css/ext6-pbs.css | 20 ++
www/datastore/DataStoreListSummary.js | 1 +
www/datastore/Summary.js | 113 ++++++++++-
www/form/PartitionSelector.js | 81 ++++++++
www/window/CreateDirectory.js | 14 ++
www/window/DataStoreEdit.js | 37 ++++
www/window/MaintenanceOptions.js | 17 +-
29 files changed, 1086 insertions(+), 88 deletions(-)
create mode 100644 etc/removable-device-attach@.service.in
create mode 100644 www/form/PartitionSelector.js
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-05-03 11:33 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
` (26 subsequent siblings)
27 siblings, 1 reply; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
... for mounting and unmounting
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/tools/disks/mod.rs | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 94f89e0a..303fe4f1 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -1323,3 +1323,22 @@ pub fn get_fs_uuid(disk: &Disk) -> Result<String, Error> {
bail!("get_fs_uuid failed - missing UUID");
}
+
+/// Mount a disk by its UUID and the mount point.
+pub fn mount_by_uuid(uuid: &str, mount_point: &Path) -> Result<(), Error> {
+ let mut command = std::process::Command::new("mount");
+ command.arg(&format!("UUID={uuid}"));
+ command.arg(mount_point);
+
+ proxmox_sys::command::run_command(command, None)?;
+ Ok(())
+}
+
+/// Unmount a disk by its mount point.
+pub fn unmount_by_mountpoint(path: &str) -> Result<(), Error> {
+ let mut command = std::process::Command::new("umount");
+ command.arg(path);
+
+ proxmox_sys::command::run_command(command, None)?;
+ Ok(())
+}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 02/26] config: factor out method to get the absolute datastore path
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
` (25 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
From: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 5 +++++
pbs-datastore/src/datastore.rs | 11 +++++++----
src/api2/node/disks/directory.rs | 4 ++--
src/bin/proxmox-backup-proxy.rs | 4 ++--
4 files changed, 16 insertions(+), 8 deletions(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 31767417..a5704c93 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -357,6 +357,11 @@ impl DataStoreConfig {
}
}
+ /// Returns the absolute path to the datastore content.
+ pub fn absolute_path(&self) -> String {
+ self.path.clone()
+ }
+
pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
self.maintenance_mode.as_ref().and_then(|str| {
MaintenanceMode::deserialize(proxmox_schema::de::SchemaDeserializer::new(
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index f95da761..f9b986c1 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -185,7 +185,7 @@ impl DataStore {
)?;
Arc::new(ChunkStore::open(
name,
- &config.path,
+ config.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
@@ -265,8 +265,11 @@ impl DataStore {
DatastoreTuning::API_SCHEMA
.parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
)?;
- let chunk_store =
- ChunkStore::open(&name, &config.path, tuning.sync_level.unwrap_or_default())?;
+ let chunk_store = ChunkStore::open(
+ &name,
+ config.absolute_path(),
+ tuning.sync_level.unwrap_or_default(),
+ )?;
let inner = Arc::new(Self::with_store_and_config(
Arc::new(chunk_store),
config,
@@ -1405,7 +1408,7 @@ impl DataStore {
bail!("datastore is currently in use");
}
- let base = PathBuf::from(&datastore_config.path);
+ let base = PathBuf::from(datastore_config.absolute_path());
let mut ok = true;
if destroy_data {
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 9f1112a9..73af92cc 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -251,12 +251,12 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
let conflicting_datastore: Option<DataStoreConfig> =
- datastores.into_iter().find(|ds| ds.path == path);
+ datastores.into_iter().find(|ds| ds.absolute_path() == path);
if let Some(conflicting_datastore) = conflicting_datastore {
bail!(
"Can't remove '{}' since it's required by datastore '{}'",
- conflicting_datastore.path,
+ conflicting_datastore.absolute_path(),
conflicting_datastore.name
);
}
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 15444685..3a05b1f3 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -1143,8 +1143,8 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
{
continue;
}
- let path = std::path::Path::new(&config.path);
- datastores.push(gather_disk_stats(disk_manager.clone(), path, &config.name));
+ let path = std::path::PathBuf::from(config.absolute_path());
+ datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
}
}
Err(err) => {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
` (24 subsequent siblings)
27 siblings, 1 reply; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index a5704c93..e5c5cfcf 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -160,6 +160,9 @@ pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema =
.minimum(1)
.schema();
+/// Base directory where datastores are mounted
+pub const DATASTORE_MOUNT_DIR: &str = "/mnt/datastore";
+
#[api]
#[derive(Debug, Default, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
@@ -273,6 +276,12 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
format: &ApiStringFormat::PropertyString(&MaintenanceMode::API_SCHEMA),
type: String,
},
+ "backing-device": {
+ description: "The UUID of the filesystem partition for removable datastores.",
+ optional: true,
+ format: &proxmox_schema::api_types::UUID_FORMAT,
+ type: String,
+ }
}
)]
#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -320,6 +329,11 @@ pub struct DataStoreConfig {
/// Maintenance mode, type is either 'offline' or 'read-only', message should be enclosed in "
#[serde(skip_serializing_if = "Option::is_none")]
pub maintenance_mode: Option<String>,
+
+ /// The UUID of the device(for removable datastores)
+ #[updater(skip)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub backing_device: Option<String>,
}
#[api]
@@ -354,12 +368,23 @@ impl DataStoreConfig {
notification_mode: None,
tuning: None,
maintenance_mode: None,
+ backing_device: None,
}
}
/// Returns the absolute path to the datastore content.
pub fn absolute_path(&self) -> String {
- self.path.clone()
+ if let Some(mount_point) = self.get_mount_point() {
+ format!("{mount_point}/{}", self.path.trim_matches('/'))
+ } else {
+ self.path.clone()
+ }
+ }
+
+ pub fn get_mount_point(&self) -> Option<String> {
+ self.backing_device
+ .is_some()
+ .then(|| format!("{DATASTORE_MOUNT_DIR}/{}", self.name))
}
pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 04/26] maintenance: add 'Unmount' maintenance type
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (2 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info Hannes Laimer
` (23 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
From: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
---
pbs-api-types/src/datastore.rs | 3 +++
pbs-api-types/src/maintenance.rs | 5 ++++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index e5c5cfcf..af914fa4 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -404,6 +404,9 @@ impl DataStoreConfig {
match current_type {
Some(MaintenanceType::ReadOnly) => { /* always OK */ }
Some(MaintenanceType::Offline) => { /* always OK */ }
+ Some(MaintenanceType::Unmount) => {
+ bail!("datastore is being unmounted");
+ }
Some(MaintenanceType::Delete) => {
match new_type {
Some(MaintenanceType::Delete) => { /* allow to delete a deleted storage */ }
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 1e3413dc..fd4d3416 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -38,7 +38,6 @@ pub enum Operation {
/// Maintenance type.
pub enum MaintenanceType {
// TODO:
- // - Add "unmounting" once we got pluggable datastores
// - Add "GarbageCollection" or "DeleteOnly" as type and track GC (or all deletes) as separate
// operation, so that one can enable a mode where nothing new can be added but stuff can be
// cleaned
@@ -48,6 +47,8 @@ pub enum MaintenanceType {
Offline,
/// The datastore is being deleted.
Delete,
+ /// The (removable) datastore is being unmounted.
+ Unmount,
}
serde_plain::derive_display_from_serialize!(MaintenanceType);
serde_plain::derive_fromstr_from_deserialize!(MaintenanceType);
@@ -94,6 +95,8 @@ impl MaintenanceMode {
if let Some(Operation::Lookup) = operation {
return Ok(());
+ } else if self.ty == MaintenanceType::Unmount {
+ bail!("datastore is being unmounted");
} else if self.ty == MaintenanceType::Offline {
bail!("offline maintenance mode: {}", message);
} else if self.ty == MaintenanceType::ReadOnly {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (3 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
` (22 subsequent siblings)
27 siblings, 1 reply; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/tools/disks/mod.rs | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 303fe4f1..6227948d 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -59,6 +59,8 @@ pub struct LsblkInfo {
/// File system label.
#[serde(rename = "fstype")]
file_system_type: Option<String>,
+ /// File system UUID.
+ uuid: Option<String>,
}
impl DiskManage {
@@ -617,7 +619,7 @@ pub struct BlockDevStat {
/// Use lsblk to read partition type uuids and file system types.
pub fn get_lsblk_info() -> Result<Vec<LsblkInfo>, Error> {
let mut command = std::process::Command::new("lsblk");
- command.args(["--json", "-o", "path,parttype,fstype"]);
+ command.args(["--json", "-o", "path,parttype,fstype,uuid"]);
let output = proxmox_sys::command::run_command(command, None)?;
@@ -701,6 +703,8 @@ pub struct PartitionInfo {
pub size: Option<u64>,
/// GPT partition
pub gpt: bool,
+ /// UUID
+ pub uuid: Option<String>,
}
#[api(
@@ -891,8 +895,10 @@ fn get_partitions_info(
let mounted = disk.is_mounted().unwrap_or(false);
let mut filesystem = None;
+ let mut uuid = None;
if let (Some(devpath), Some(infos)) = (devpath.as_ref(), lsblk_infos.as_ref()) {
for info in infos.iter().filter(|i| i.path.eq(devpath)) {
+ uuid = info.uuid.clone();
used = match info.partition_type.as_deref() {
Some("21686148-6449-6e6f-744e-656564454649") => PartitionUsageType::BIOS,
Some("c12a7328-f81f-11d2-ba4b-00a0c93ec93b") => PartitionUsageType::EFI,
@@ -915,6 +921,7 @@ fn get_partitions_info(
filesystem,
size: disk.size().ok(),
gpt: disk.has_gpt(),
+ uuid,
}
})
.collect()
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 06/26] datastore: add helper for checking if a removable datastore is available
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (4 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
` (21 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/maintenance.rs | 2 +-
pbs-datastore/src/datastore.rs | 58 ++++++++++++++++++++++++++++++++
pbs-datastore/src/lib.rs | 2 +-
src/bin/proxmox-backup-proxy.rs | 5 ++-
4 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index fd4d3416..4f653ec7 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -81,7 +81,7 @@ impl MaintenanceMode {
/// Used for deciding whether the datastore is cleared from the internal cache after the last
/// task finishes, so all open files are closed.
pub fn is_offline(&self) -> bool {
- self.ty == MaintenanceType::Offline
+ self.ty == MaintenanceType::Offline || self.ty == MaintenanceType::Unmount
}
pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index f9b986c1..4f7865fb 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1,5 +1,6 @@
use std::collections::{HashMap, HashSet};
use std::io::{self, Write};
+use std::os::unix::ffi::OsStrExt;
use std::os::unix::io::AsRawFd;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
@@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
+use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::ProcessLockSharedGuard;
use proxmox_sys::WorkerTaskContext;
use proxmox_sys::{task_log, task_warn};
@@ -49,6 +51,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
Ok(())
}
+/// check if a removable datastore is currently available/mounted by
+/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
+/// /proc/self/mountinfo
+pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
+ use nix::sys::stat::SFlag;
+
+ let uuid = match config.backing_device.as_deref() {
+ Some(dev) => dev,
+ None => return true,
+ };
+
+ let Some(store_mount_point) = config.get_mount_point() else {
+ return false;
+ };
+ let store_mount_point = Path::new(&store_mount_point);
+
+ let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
+ Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
+ _ => return false,
+ };
+
+ let Ok(mount_info) = MountInfo::read() else {
+ return false;
+ };
+
+ for (_, entry) in mount_info {
+ let Some(source) = entry.mount_source else {
+ continue;
+ };
+
+ if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
+ continue;
+ }
+
+ if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
+ let sflag = SFlag::from_bits_truncate(stat.st_mode);
+
+ if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
+ return true;
+ }
+ }
+ }
+
+ false
+}
+
/// Datastore Management
///
/// A Datastore can store severals backups, and provides the
@@ -158,6 +206,12 @@ impl DataStore {
}
}
+ if config.backing_device.is_some() && !is_datastore_available(&config) {
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+ datastore_cache.remove(&config.name);
+ bail!("Removable Datastore is not mounted");
+ }
+
if let Some(operation) = operation {
update_active_operations(name, operation, 1)?;
}
@@ -261,6 +315,10 @@ impl DataStore {
) -> Result<Arc<Self>, Error> {
let name = config.name.clone();
+ if !is_datastore_available(&config) {
+ bail!("Datastore is not available")
+ }
+
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
.parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 43050162..458f93d9 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -206,7 +206,7 @@ pub use manifest::BackupManifest;
pub use store_progress::StoreProgress;
mod datastore;
-pub use datastore::{check_backup_owner, DataStore};
+pub use datastore::{check_backup_owner, is_datastore_available, DataStore};
mod hierarchy;
pub use hierarchy::{
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 3a05b1f3..9c7a472d 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -20,7 +20,7 @@ use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsSta
use proxmox_sys::logrotate::LogRotate;
use proxmox_sys::{task_log, task_warn};
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
use proxmox_rest_server::{
cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
@@ -1143,6 +1143,9 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
{
continue;
}
+ if !is_datastore_available(&config) {
+ continue;
+ }
let path = std::path::PathBuf::from(config.absolute_path());
datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 07/26] api: admin: add (un)mount endpoint for removable datastores
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (5 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 08/26] api: removable datastore creation Hannes Laimer
` (20 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/maintenance.rs | 4 +
src/api2/admin/datastore.rs | 186 +++++++++++++++++++++++++++++--
2 files changed, 180 insertions(+), 10 deletions(-)
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 4f653ec7..469dc25d 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -78,6 +78,10 @@ pub struct MaintenanceMode {
}
impl MaintenanceMode {
+ pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
+ Self { ty, message }
+ }
+
/// Used for deciding whether the datastore is cleared from the internal cache after the last
/// task finishes, so all open files are closed.
pub fn is_offline(&self) -> bool {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 3ea17499..327ffffc 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -26,7 +26,7 @@ use proxmox_sortable_macro::sortable;
use proxmox_sys::fs::{
file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
};
-use proxmox_sys::{task_log, task_warn};
+use proxmox_sys::{task_log, task_warn, WorkerTaskContext};
use proxmox_time::CalendarEvent;
use pxar::accessor::aio::Accessor;
@@ -35,13 +35,13 @@ use pxar::EntryKind;
use pbs_api_types::{
print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
- GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
- PruneJobOptions, RRDMode, RRDTimeFrame, SnapshotListItem, SnapshotVerifyState,
- BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
- BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
- NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
- PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
- VERIFICATION_OUTDATED_AFTER_SCHEMA,
+ GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
+ MaintenanceType, Operation, PruneJobOptions, RRDMode, RRDTimeFrame, SnapshotListItem,
+ SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+ BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA,
+ MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
+ PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID,
+ UPID_SCHEMA, VERIFICATION_OUTDATED_AFTER_SCHEMA,
};
use pbs_client::pxar::{create_tar, create_zip};
use pbs_config::CachedUserInfo;
@@ -56,8 +56,8 @@ use pbs_datastore::index::IndexFile;
use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
use pbs_datastore::prune::compute_prune_info;
use pbs_datastore::{
- check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
- StoreProgress, CATALOG_NAME,
+ check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
+ LocalChunkReader, StoreProgress, CATALOG_NAME,
};
use pbs_tools::json::required_string_param;
use proxmox_rest_server::{formatter, WorkerTask};
@@ -2330,6 +2330,170 @@ pub async fn set_backup_owner(
.await?
}
+pub fn do_mount_device(
+ datastore: DataStoreConfig,
+ worker: Option<&dyn WorkerTaskContext>,
+) -> Result<(), Error> {
+ if let (Some(uuid), Some(mount_point)) = (
+ datastore.backing_device.as_ref(),
+ datastore.get_mount_point(),
+ ) {
+ if pbs_datastore::is_datastore_available(&datastore) {
+ bail!("device '{uuid}' is already mounted");
+ }
+ let mount_point_path = std::path::Path::new(&mount_point);
+ if let Some(worker) = worker {
+ task_log!(
+ worker,
+ "mounting '{uuid}' for store {} to '{mount_point}'",
+ datastore.name,
+ );
+ }
+ crate::tools::disks::mount_by_uuid(uuid, mount_point_path)?;
+
+ Ok(())
+ } else {
+ Err(format_err!(
+ "Datastore '{}' cannot be mounted because it is not removable.",
+ datastore.name
+ ))
+ }
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ }
+ },
+ returns: {
+ schema: UPID_SCHEMA,
+ },
+ access: {
+ permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
+ },
+)]
+/// Mount removable datastore.
+pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let (section_config, _digest) = pbs_config::datastore::config()?;
+ let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+ if datastore.backing_device.is_none() {
+ bail!("datastore '{store}' is not removable");
+ }
+
+ let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+ let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+ let upid = WorkerTask::new_thread(
+ "mount-device",
+ Some(store),
+ auth_id.to_string(),
+ to_stdout,
+ move |worker| do_mount_device(datastore, Some(&worker)),
+ )?;
+
+ Ok(json!(upid))
+}
+
+fn do_unmount_device(
+ datastore: DataStoreConfig,
+ worker: Option<&dyn WorkerTaskContext>,
+) -> Result<(), Error> {
+ let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
+ let mut old_status = String::new();
+ while active_operations.read + active_operations.write > 0 {
+ if let Some(worker) = worker {
+ if worker.abort_requested() {
+ bail!("aborted, due to user request");
+ }
+ let status = format!(
+ "cannot unmount yet, still {} read and {} write operations active",
+ active_operations.read, active_operations.write
+ );
+ if status != old_status {
+ task_log!(worker, "{status}");
+ old_status = status;
+ }
+ }
+ std::thread::sleep(std::time::Duration::from_millis(250));
+ active_operations = task_tracking::get_active_operations(&datastore.name)?;
+ }
+ if let Some(mount_point) = datastore.get_mount_point() {
+ crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
+
+ let _lock = pbs_config::datastore::lock_config()?;
+ let (mut section_config, _digest) = pbs_config::datastore::config()?;
+ let mut store_config: DataStoreConfig =
+ section_config.lookup("datastore", &datastore.name)?;
+ store_config.maintenance_mode = None;
+ section_config.set_data(&datastore.name, "datastore", &store_config)?;
+ pbs_config::datastore::save_config(§ion_config)?;
+ }
+ Ok(())
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: { schema: DATASTORE_SCHEMA },
+ },
+ },
+ returns: {
+ schema: UPID_SCHEMA,
+ },
+ access: {
+ permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, true),
+ }
+)]
+/// Unmount a removable device that is associated with the datastore
+pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let _lock = pbs_config::datastore::lock_config()?;
+ let (mut section_config, _digest) = pbs_config::datastore::config()?;
+ let mut datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+ if datastore.backing_device.is_none() {
+ bail!("datastore '{store}' is not removable");
+ }
+
+ if !is_datastore_available(&datastore) {
+ bail!("datastore '{store}' is not mounted");
+ }
+
+ datastore.set_maintenance_mode(Some(MaintenanceMode::new(MaintenanceType::Unmount, None)))?;
+ section_config.set_data(&store, "datastore", &datastore)?;
+ pbs_config::datastore::save_config(§ion_config)?;
+
+ drop(_lock);
+
+ let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+ let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+ if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+ {
+ let sock = proxmox_rest_server::ctrl_sock_from_pid(proxy_pid);
+ let _ = proxmox_rest_server::send_raw_command(
+ sock,
+ &format!("{{\"command\":\"update-datastore-cache\",\"args\":\"{store}\"}}\n"),
+ )
+ .await;
+ }
+
+ let upid = WorkerTask::new_thread(
+ "unmount-device",
+ Some(store),
+ auth_id.to_string(),
+ to_stdout,
+ move |worker| do_unmount_device(datastore, Some(&worker)),
+ )?;
+
+ Ok(json!(upid))
+}
+
#[sortable]
const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
(
@@ -2368,6 +2532,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
.get(&API_METHOD_LIST_GROUPS)
.delete(&API_METHOD_DELETE_GROUP),
),
+ ("mount", &Router::new().post(&API_METHOD_MOUNT)),
(
"namespace",
// FIXME: move into datastore:: sub-module?!
@@ -2402,6 +2567,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
.delete(&API_METHOD_DELETE_SNAPSHOT),
),
("status", &Router::new().get(&API_METHOD_STATUS)),
+ ("unmount", &Router::new().post(&API_METHOD_UNMOUNT)),
(
"upload-backup-log",
&Router::new().upload(&API_METHOD_UPLOAD_BACKUP_LOG),
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 08/26] api: removable datastore creation
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (6 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 09/26] api: disks list: add exclude-used flag Hannes Laimer
` (19 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/config/datastore.rs | 53 +++++++++++++++++++++++++++++++++---
1 file changed, 49 insertions(+), 4 deletions(-)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 6b742acb..401fbccf 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -8,7 +8,7 @@ use serde_json::Value;
use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
use proxmox_schema::{api, param_bail, ApiType};
use proxmox_section_config::SectionConfigData;
-use proxmox_sys::{task_warn, WorkerTaskContext};
+use proxmox_sys::{task_log, task_warn, WorkerTaskContext};
use proxmox_uuid::Uuid;
use pbs_api_types::{
@@ -20,7 +20,8 @@ use pbs_config::BackupLockGuard;
use pbs_datastore::chunk_store::ChunkStore;
use crate::api2::admin::{
- prune::list_prune_jobs, sync::list_sync_jobs, verify::list_verification_jobs,
+ datastore::do_mount_device, prune::list_prune_jobs, sync::list_sync_jobs,
+ verify::list_verification_jobs,
};
use crate::api2::config::prune::{delete_prune_job, do_create_prune_job};
use crate::api2::config::sync::delete_sync_job;
@@ -72,13 +73,35 @@ pub(crate) fn do_create_datastore(
datastore: DataStoreConfig,
worker: Option<&dyn WorkerTaskContext>,
) -> Result<(), Error> {
- let path: PathBuf = datastore.path.clone().into();
+ let path: PathBuf = datastore.absolute_path().into();
+ let backup_user = pbs_config::backup_user()?;
+ if let Some(store_mount_point) = datastore.get_mount_point() {
+ let default_options = proxmox_sys::fs::CreateOptions::new();
+ proxmox_sys::fs::create_path(
+ store_mount_point,
+ Some(default_options.clone()),
+ Some(default_options.clone()),
+ )?;
+ do_mount_device(datastore.clone(), worker)?;
+
+ if path.join(".chunks").is_dir() {
+ config.set_data(&datastore.name, "datastore", &datastore)?;
+ pbs_config::datastore::save_config(&config)?;
+ jobstate::create_state_file("garbage_collection", &datastore.name)?;
+ if let Some(worker) = worker {
+ task_log!(
+ worker,
+ "created removable datastore, chunkstore already exists"
+ );
+ }
+ return Ok(());
+ }
+ }
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
.parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
)?;
- let backup_user = pbs_config::backup_user()?;
let _store = ChunkStore::create(
&datastore.name,
path,
@@ -122,6 +145,28 @@ pub fn create_datastore(
param_bail!("name", "datastore '{}' already exists.", config.name);
}
+ if let Some(uuid) = &config.backing_device {
+ let already_used_by = section_config
+ .sections
+ .iter()
+ .flat_map(|(datastore_name, (_, config))| {
+ config
+ .as_object()
+ .and_then(|cfg| cfg.get("backing-device"))
+ .and_then(|backing_device| backing_device.as_str())
+ .filter(|&device_uuid| device_uuid == uuid)
+ .map(|_| datastore_name)
+ })
+ .next();
+
+ if let Some(datastore_name) = already_used_by {
+ param_bail!(
+ "backing-device",
+ "device already in use by datastore '{datastore_name}'",
+ );
+ }
+ }
+
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 09/26] api: disks list: add exclude-used flag
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (7 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 08/26] api: removable datastore creation Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
` (18 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
... used by the partition selector for removable datastore creation.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/mod.rs | 8 +++++
src/tools/disks/mod.rs | 61 ++++++++++++++++++++++++++++++++------
2 files changed, 60 insertions(+), 9 deletions(-)
diff --git a/src/api2/node/disks/mod.rs b/src/api2/node/disks/mod.rs
index 711dae7b..f63bbd9b 100644
--- a/src/api2/node/disks/mod.rs
+++ b/src/api2/node/disks/mod.rs
@@ -41,6 +41,12 @@ pub mod zfs;
optional: true,
default: false,
},
+ "exclude-used": {
+ description: "Exclude partitions already used for removable datastores or mounted directories.",
+ type: bool,
+ optional: true,
+ default: false,
+ },
"usage-type": {
type: DiskUsageType,
optional: true,
@@ -62,6 +68,7 @@ pub mod zfs;
pub fn list_disks(
skipsmart: bool,
include_partitions: bool,
+ exclude_used: bool,
usage_type: Option<DiskUsageType>,
) -> Result<Vec<DiskUsageInfo>, Error> {
let mut list = Vec::new();
@@ -69,6 +76,7 @@ pub fn list_disks(
for (_, info) in DiskUsageQuery::new()
.smart(!skipsmart)
.partitions(include_partitions)
+ .exclude_used(exclude_used)
.query()?
{
if let Some(ref usage_type) = usage_type {
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 6227948d..a8018834 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -20,6 +20,7 @@ use proxmox_rest_server::WorkerTask;
use proxmox_schema::api;
use proxmox_sys::linux::procfs::{mountinfo::Device, MountInfo};
use proxmox_sys::task_log;
+use serde_json::Value;
use pbs_api_types::{BLOCKDEVICE_DISK_AND_PARTITION_NAME_REGEX, BLOCKDEVICE_NAME_REGEX};
@@ -32,6 +33,7 @@ pub use zpool_list::*;
mod lvm;
pub use lvm::*;
mod smart;
+use crate::api2::node::disks::directory::list_datastore_mounts;
pub use smart::*;
lazy_static::lazy_static! {
@@ -828,6 +830,7 @@ fn scan_partitions(
pub struct DiskUsageQuery {
smart: bool,
partitions: bool,
+ exclude_used: bool,
}
impl DiskUsageQuery {
@@ -835,6 +838,7 @@ impl DiskUsageQuery {
Self {
smart: true,
partitions: false,
+ exclude_used: false,
}
}
@@ -848,12 +852,22 @@ impl DiskUsageQuery {
self
}
+ pub fn exclude_used(&mut self, exclude_used: bool) -> &mut Self {
+ self.exclude_used = exclude_used;
+ self
+ }
+
pub fn query(&self) -> Result<HashMap<String, DiskUsageInfo>, Error> {
- get_disks(None, !self.smart, self.partitions)
+ get_disks(None, !self.smart, self.partitions, self.exclude_used)
}
pub fn find(&self, disk: &str) -> Result<DiskUsageInfo, Error> {
- let mut map = get_disks(Some(vec![disk.to_string()]), !self.smart, self.partitions)?;
+ let mut map = get_disks(
+ Some(vec![disk.to_string()]),
+ !self.smart,
+ self.partitions,
+ self.exclude_used,
+ )?;
if let Some(info) = map.remove(disk) {
Ok(info)
} else {
@@ -862,7 +876,7 @@ impl DiskUsageQuery {
}
pub fn find_all(&self, disks: Vec<String>) -> Result<HashMap<String, DiskUsageInfo>, Error> {
- get_disks(Some(disks), !self.smart, self.partitions)
+ get_disks(Some(disks), !self.smart, self.partitions, self.exclude_used)
}
}
@@ -935,6 +949,8 @@ fn get_disks(
no_smart: bool,
// include partitions
include_partitions: bool,
+ // skip partitions which are in use
+ exclude_used: bool,
) -> Result<HashMap<String, DiskUsageInfo>, Error> {
let disk_manager = DiskManage::new();
@@ -952,6 +968,31 @@ fn get_disks(
// fixme: ceph journals/volumes
+ let uuids_in_use = if exclude_used && include_partitions {
+ let (config, _digest) = pbs_config::datastore::config()?;
+
+ let uuids_from_datastores: Vec<String> = config
+ .sections
+ .iter()
+ .filter_map(|(_, (_, data))| {
+ data.as_object()
+ .and_then(|cfg| cfg.get("backing-device"))
+ .and_then(Value::as_str)
+ .map(String::from)
+ })
+ .collect();
+
+ let uuids_from_mounts: Vec<String> = list_datastore_mounts()?
+ .into_iter()
+ // FIXME: include UUID in DatastoreMountInfo
+ .filter_map(|mount| mount.device.split('/').last().map(String::from))
+ .collect();
+
+ [&uuids_from_datastores[..], &uuids_from_mounts[..]].concat()
+ } else {
+ Vec::new()
+ };
+
let mut result = HashMap::new();
for item in proxmox_sys::fs::scan_subdir(libc::AT_FDCWD, "/sys/block", &BLOCKDEVICE_NAME_REGEX)?
@@ -1024,12 +1065,14 @@ fn get_disks(
let partitions: Option<Vec<PartitionInfo>> = if include_partitions {
disk.partitions().map_or(None, |parts| {
- Some(get_partitions_info(
- parts,
- &lvm_devices,
- &zfs_devices,
- &file_system_devices,
- ))
+ Some(
+ get_partitions_info(parts, &lvm_devices, &zfs_devices, &file_system_devices)
+ .into_iter()
+ .filter(|part| {
+ !part.uuid.as_ref().is_some_and(|u| uuids_in_use.contains(u))
+ })
+ .collect(),
+ )
})
} else {
None
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (8 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 09/26] api: disks list: add exclude-used flag Hannes Laimer
@ 2024-04-25 6:52 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 11/26] bin: manager: add (un)mount command Hannes Laimer
` (17 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 9 ++++++++-
src/api2/admin/datastore.rs | 17 +++++++++--------
src/api2/status.rs | 18 +++++++++++++++---
3 files changed, 32 insertions(+), 12 deletions(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index af914fa4..0ad59b15 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -454,6 +454,10 @@ impl DataStoreConfig {
pub struct DataStoreListItem {
pub store: String,
pub comment: Option<String>,
+ /// Datastore is removable
+ pub removable: bool,
+ /// Datastore is available
+ pub available: bool,
/// If the datastore is in maintenance mode, information about it
#[serde(skip_serializing_if = "Option::is_none")]
pub maintenance: Option<String>,
@@ -1453,6 +1457,8 @@ pub struct DataStoreStatusListItem {
/// The available bytes of the underlying storage. (-1 on error)
#[serde(skip_serializing_if = "Option::is_none")]
pub avail: Option<u64>,
+ /// The datastore is available, relevant if removable
+ pub is_available: bool,
/// A list of usages of the past (last Month).
#[serde(skip_serializing_if = "Option::is_none")]
pub history: Option<Vec<Option<f64>>>,
@@ -1477,12 +1483,13 @@ pub struct DataStoreStatusListItem {
}
impl DataStoreStatusListItem {
- pub fn empty(store: &str, err: Option<String>) -> Self {
+ pub fn empty(store: &str, err: Option<String>, is_available: bool) -> Self {
DataStoreStatusListItem {
store: store.to_owned(),
total: None,
used: None,
avail: None,
+ is_available,
history: None,
history_start: None,
history_delta: None,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 327ffffc..df181ead 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -1313,8 +1313,8 @@ pub fn get_datastore_list(
let mut list = Vec::new();
- for (store, (_, data)) in &config.sections {
- let acl_path = &["datastore", store];
+ for (store, (_, data)) in config.sections {
+ let acl_path = &["datastore", &store];
let user_privs = user_info.lookup_privs(&auth_id, acl_path);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
@@ -1325,15 +1325,16 @@ pub fn get_datastore_list(
}
}
+ let store_config: DataStoreConfig = serde_json::from_value(data)?;
+ let is_available = pbs_datastore::is_datastore_available(&store_config);
+
if allowed || allow_id {
list.push(DataStoreListItem {
store: store.clone(),
- comment: if !allowed {
- None
- } else {
- data["comment"].as_str().map(String::from)
- },
- maintenance: data["maintenance-mode"].as_str().map(String::from),
+ comment: store_config.comment.filter(|_| allowed),
+ removable: store_config.backing_device.is_some(),
+ available: is_available,
+ maintenance: store_config.maintenance_mode,
});
}
}
diff --git a/src/api2/status.rs b/src/api2/status.rs
index 78bc06b5..e1d33ccf 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -13,7 +13,7 @@ use pbs_api_types::{
};
use pbs_config::CachedUserInfo;
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
use crate::rrd_cache::extract_rrd_data;
use crate::tools::statistics::linear_regression;
@@ -48,10 +48,17 @@ pub async fn datastore_status(
for (store, (_, _)) in &config.sections {
let user_privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
+
+ let store_config = config.lookup("datastore", store)?;
+ if !is_datastore_available(&store_config) {
+ list.push(DataStoreStatusListItem::empty(store, None, false));
+ continue;
+ }
+
if !allowed {
if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
if can_access_any_namespace(datastore, &auth_id, &user_info) {
- list.push(DataStoreStatusListItem::empty(store, None));
+ list.push(DataStoreStatusListItem::empty(store, None, true));
}
}
continue;
@@ -60,7 +67,11 @@ pub async fn datastore_status(
let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
Ok(datastore) => datastore,
Err(err) => {
- list.push(DataStoreStatusListItem::empty(store, Some(err.to_string())));
+ list.push(DataStoreStatusListItem::empty(
+ store,
+ Some(err.to_string()),
+ true,
+ ));
continue;
}
};
@@ -71,6 +82,7 @@ pub async fn datastore_status(
total: Some(status.total),
used: Some(status.used),
avail: Some(status.available),
+ is_available: true,
history: None,
history_start: None,
history_delta: None,
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 11/26] bin: manager: add (un)mount command
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (9 preceding siblings ...)
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores Hannes Laimer
` (16 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-config/src/datastore.rs | 14 ++++
src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
2 files changed, 89 insertions(+), 1 deletion(-)
diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
index 5844a174..a540788f 100644
--- a/pbs-config/src/datastore.rs
+++ b/pbs-config/src/datastore.rs
@@ -63,6 +63,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
}
}
+pub fn complete_removable_datastore_name(
+ _arg: &str,
+ _param: &HashMap<String, String>,
+) -> Vec<String> {
+ match config() {
+ Ok((data, _digest)) => data
+ .sections
+ .into_iter()
+ .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
+ .collect(),
+ Err(_) => Vec::new(),
+ }
+}
+
pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
let mut list = vec![
String::from("/"),
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 383bcd24..f2795b39 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::Error;
+use anyhow::{format_err, Error};
use serde_json::Value;
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
Ok(Value::Null)
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Mount a removable datastore.
+async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+ param["node"] = "localhost".into();
+
+ let info = &api2::admin::datastore::API_METHOD_MOUNT;
+ let result = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
+ _ => unreachable!(),
+ };
+
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ Ok(())
+}
+
#[api(
input: {
properties: {
@@ -99,6 +127,34 @@ async fn create_datastore(mut param: Value) -> Result<Value, Error> {
Ok(Value::Null)
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Unmount a removable datastore.
+async fn unmount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+ param["node"] = "localhost".into();
+
+ let info = &api2::admin::datastore::API_METHOD_UNMOUNT;
+ let result = match info.handler {
+ ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
+ _ => unreachable!(),
+ };
+
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ Ok(())
+}
+
#[api(
protected: true,
input: {
@@ -142,6 +198,15 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
pub fn datastore_commands() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
.insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
+ .insert(
+ "mount",
+ CliCommand::new(&API_METHOD_MOUNT_DATASTORE)
+ .arg_param(&["store"])
+ .completion_cb(
+ "store",
+ pbs_config::datastore::complete_removable_datastore_name,
+ ),
+ )
.insert(
"show",
CliCommand::new(&API_METHOD_SHOW_DATASTORE)
@@ -152,6 +217,15 @@ pub fn datastore_commands() -> CommandLineInterface {
"create",
CliCommand::new(&API_METHOD_CREATE_DATASTORE).arg_param(&["name", "path"]),
)
+ .insert(
+ "unmount",
+ CliCommand::new(&API_METHOD_UNMOUNT_DATASTORE)
+ .arg_param(&["store"])
+ .completion_cb(
+ "store",
+ pbs_config::datastore::complete_removable_datastore_name,
+ ),
+ )
.insert(
"update",
CliCommand::new(&api2::config::datastore::API_METHOD_UPDATE_DATASTORE)
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (10 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 11/26] bin: manager: add (un)mount command Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
` (15 subsequent siblings)
27 siblings, 1 reply; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
debian/proxmox-backup-server.install | 1 +
debian/proxmox-backup-server.udev | 3 ++
etc/Makefile | 3 +-
etc/removable-device-attach@.service.in | 8 +++
src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
5 files changed, 68 insertions(+), 2 deletions(-)
create mode 100644 etc/removable-device-attach@.service.in
diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
index ef1e9ba1..34bfc454 100644
--- a/debian/proxmox-backup-server.install
+++ b/debian/proxmox-backup-server.install
@@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
etc/proxmox-backup-daily-update.timer /lib/systemd/system/
etc/proxmox-backup-proxy.service /lib/systemd/system/
etc/proxmox-backup.service /lib/systemd/system/
+etc/removable-device-attach@.service /lib/systemd/system/
usr/bin/pmt
usr/bin/pmtx
usr/bin/proxmox-tape
diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
index afdfb2bc..e21b8bc7 100644
--- a/debian/proxmox-backup-server.udev
+++ b/debian/proxmox-backup-server.udev
@@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
LABEL="persistent_storage_tape_end"
+
+# triggers the mounting of a removable device
+ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
\ No newline at end of file
diff --git a/etc/Makefile b/etc/Makefile
index 42f639f6..730de4f8 100644
--- a/etc/Makefile
+++ b/etc/Makefile
@@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
proxmox-backup-banner.service \
proxmox-backup-daily-update.service \
proxmox-backup.service \
- proxmox-backup-proxy.service
+ proxmox-backup-proxy.service \
+ removable-device-attach@.service
all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
new file mode 100644
index 00000000..e10d1ea3
--- /dev/null
+++ b/etc/removable-device-attach@.service.in
@@ -0,0 +1,8 @@
+[Unit]
+Description=Try to mount the removable device of a datastore with uuid '%i'.
+After=proxmox-backup-proxy.service
+Requires=proxmox-backup-proxy.service
+
+[Service]
+Type=simple
+ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index f2795b39..e74c8537 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::{format_err, Error};
+use anyhow::{bail, format_err, Error};
use serde_json::Value;
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
Ok(())
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ uuid: {
+ type: String,
+ description: "The UUID of the device that should be mounted",
+ },
+ "output-format": {
+ schema: OUTPUT_FORMAT,
+ optional: true,
+ },
+ },
+ },
+)]
+/// Try mounting a removable datastore given the UUID.
+async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let uuid = param["uuid"]
+ .as_str()
+ .ok_or_else(|| format_err!("uuid has to be specified"))?;
+
+ let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
+ let data: Value = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
+ _ => unreachable!(),
+ };
+
+ let store_name = data.as_array().and_then(|list| {
+ list.iter()
+ .filter_map(Value::as_object)
+ .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
+ .and_then(|s| s.get("name").and_then(Value::as_str))
+ });
+
+ if let Some(store_name) = store_name {
+ let info = &api2::admin::datastore::API_METHOD_MOUNT;
+ let mount_param = serde_json::json!({
+ "store": store_name,
+ });
+ let result = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
+ _ => unreachable!(),
+ };
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ return Ok(Value::Null);
+ }
+ bail!("'{uuid}' is not associated with any datastore")
+}
+
pub fn datastore_commands() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
.insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
@@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
pbs_config::datastore::complete_calendar_event,
),
)
+ .insert(
+ "uuid-mount",
+ CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
+ )
.insert(
"remove",
CliCommand::new(&API_METHOD_DELETE_DATASTORE)
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 13/26] datastore: handle deletion of removable datastore properly
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (11 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section Hannes Laimer
` (14 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-datastore/src/datastore.rs | 10 ++++++----
src/api2/config/datastore.rs | 16 ++++++++++++++++
2 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 4f7865fb..3b8af833 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1528,10 +1528,12 @@ impl DataStore {
// weird, but ok
}
Err(err) if err.is_errno(nix::errno::Errno::EBUSY) => {
- task_warn!(
- worker,
- "Cannot delete datastore directory (is it a mount point?)."
- )
+ if datastore_config.backing_device.is_none() {
+ task_warn!(
+ worker,
+ "Cannot delete datastore directory (is it a mount point?)."
+ )
+ }
}
Err(err) if err.is_errno(nix::errno::Errno::ENOTEMPTY) => {
task_warn!(worker, "Datastore directory not empty, not deleting.")
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 401fbccf..fae44a75 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -29,9 +29,11 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
use crate::api2::config::verify::delete_verification_job;
use pbs_config::CachedUserInfo;
+use pbs_datastore::is_datastore_available;
use proxmox_rest_server::WorkerTask;
use crate::server::jobstate;
+use crate::tools::disks::unmount_by_mountpoint;
#[api(
input: {
@@ -541,6 +543,14 @@ pub async fn delete_datastore(
http_bail!(NOT_FOUND, "datastore '{}' does not exist.", name);
}
+ let store_config: DataStoreConfig = config.lookup("datastore", &name)?;
+ if destroy_data && is_datastore_available(&store_config) {
+ http_bail!(
+ BAD_REQUEST,
+ "cannot destroy data on '{name}' unless the datastore is mounted"
+ );
+ }
+
if !keep_job_configs {
for job in list_verification_jobs(Some(name.clone()), Value::Null, rpcenv)? {
delete_verification_job(job.config.id, None, rpcenv)?
@@ -579,6 +589,12 @@ pub async fn delete_datastore(
// ignore errors
let _ = jobstate::remove_state_file("prune", &name);
let _ = jobstate::remove_state_file("garbage_collection", &name);
+ if destroy_data {
+ if let Some(mount_point) = store_config.get_mount_point() {
+ let _ = unmount_by_mountpoint(&mount_point);
+ let _ = std::fs::remove_dir(&mount_point);
+ }
+ }
if let Err(err) =
proxmox_async::runtime::block_on(crate::server::notify_datastore_removed())
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (12 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 15/26] ui: add partition selector form Hannes Laimer
` (13 subsequent siblings)
27 siblings, 1 reply; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
docs/storage.rst | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/docs/storage.rst b/docs/storage.rst
index 4444c423..0a2f8f8a 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -165,6 +165,35 @@ following command creates a new datastore called ``store1`` on
# proxmox-backup-manager datastore create store1 /backup/disk1/store1
+Removable Datastores
+^^^^^^^^^^^^^^^^^^^^
+Removable datastores have a ``backing-device`` associated with them, they can be
+mounted and unmounted. Other than that they behave the same way a normal datastore
+would.
+
+They can be created on already correctly formatted partitions, which, as with normal
+datastores, should be either ``ext4`` or ``xfs``. It is also possible to create them
+on completely unused disks through "Admnistartion" > "Disks / Storage" > "Directory",
+using this method the disk will be partitioned and formatted automatically for the datastore.
+
+They are mounted automatically when plugged in. It is possible to create a
+removable datastore on one PBS and use it on multiple, the device just has to be added
+on all as a removable datastore. This can be done using the normal `Create Datastore`
+form, by checking **Removable datastore**, selecting the device and setting the correct path.
+If the device already contains a datastore at the specified path it'll just be added as
+a new datastore to the PBS instance and will be mounted whenever plugged in. Unmounting has
+to be done through the UI by clicking "Unmount" on the summary page or using the CLI
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore unmount store1
+
+both will wait for any running tasks to finish and unmount the device.
+
+All removable datastores are mounted under /mnt/datastore/<name>, and the specified path
+refers to the path on the device.
+
+
Managing Datastores
^^^^^^^^^^^^^^^^^^^
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 15/26] ui: add partition selector form
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (13 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 16/26] ui: add removable datastore creation support Hannes Laimer
` (12 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Makefile | 1 +
www/form/PartitionSelector.js | 81 +++++++++++++++++++++++++++++++++++
2 files changed, 82 insertions(+)
create mode 100644 www/form/PartitionSelector.js
diff --git a/www/Makefile b/www/Makefile
index 609a0ba6..45adfc54 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -49,6 +49,7 @@ JSSRC= \
form/NamespaceMaxDepth.js \
form/CalendarEvent.js \
form/PermissionPathSelector.js \
+ form/PartitionSelector.js \
form/GroupSelector.js \
form/GroupFilter.js \
form/VerifyOutdatedAfter.js \
diff --git a/www/form/PartitionSelector.js b/www/form/PartitionSelector.js
new file mode 100644
index 00000000..3d6eadd5
--- /dev/null
+++ b/www/form/PartitionSelector.js
@@ -0,0 +1,81 @@
+Ext.define('pbs-partition-list', {
+ extend: 'Ext.data.Model',
+ fields: ['name', 'uuid', 'filesystem', 'devpath', 'size', 'model'],
+ proxy: {
+ type: 'proxmox',
+ url: "/api2/json/nodes/localhost/disks/list?skipsmart=1&include-partitions=1&exclude-used=1",
+ reader: {
+ transform: (rawData) => rawData.data
+ .flatMap(disk => (disk.partitions
+ .map(part => ({ ...part, model: disk.model })) ?? [])
+ .filter(partition => partition.used === 'filesystem')),
+ },
+ },
+ idProperty: 'devpath',
+
+});
+
+Ext.define('PBS.form.PartitionSelector', {
+ extend: 'Proxmox.form.ComboGrid',
+ alias: 'widget.pbsPartitionSelector',
+
+ allowBlank: false,
+ autoSelect: false,
+ submitEmpty: false,
+ valueField: 'uuid',
+ displayField: 'devpath',
+
+ store: {
+ model: 'pbs-partition-list',
+ autoLoad: true,
+ sorters: 'devpath',
+ },
+ getSubmitData: function() {
+ let me = this;
+ let data = null;
+ if (!me.disabled && me.submitValue && !me.isFileUpload()) {
+ let val = me.getSubmitValue();
+ if (val !== undefined && val !== null && val !== '') {
+ data = {};
+ data[me.getName()] = val;
+ } else if (me.getDeleteEmpty()) {
+ data = {};
+ data.delete = me.getName();
+ }
+ }
+ return data;
+ },
+ listConfig: {
+ columns: [
+ {
+ header: gettext('Path'),
+ sortable: true,
+ dataIndex: 'devpath',
+ renderer: (v, metaData, rec) => Ext.String.htmlEncode(v),
+ flex: 1,
+ },
+ {
+ header: gettext('Filesystem'),
+ sortable: true,
+ dataIndex: 'filesystem',
+ flex: 1,
+ },
+ {
+ header: gettext('Size'),
+ sortable: true,
+ dataIndex: 'size',
+ renderer: Proxmox.Utils.format_size,
+ flex: 1,
+ },
+ {
+ header: gettext('Model'),
+ sortable: true,
+ dataIndex: 'model',
+ flex: 1,
+ },
+ ],
+ viewConfig: {
+ emptyText: 'No usable partitions present',
+ },
+ },
+});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 16/26] ui: add removable datastore creation support
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (14 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 15/26] ui: add partition selector form Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 17/26] ui: add (un)mount button to summary Hannes Laimer
` (11 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/window/DataStoreEdit.js | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index b6115460..54929382 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -62,6 +62,20 @@ Ext.define('PBS.DataStoreEdit', {
fieldLabel: gettext('Backing Path'),
emptyText: gettext('An absolute path'),
},
+ {
+ xtype: 'pmxDisplayEditField',
+ fieldLabel: gettext('Device'),
+ name: 'backing-device',
+ disabled: true,
+ cbind: {
+ editable: '{isCreate}',
+ },
+ editConfig: {
+ xtype: 'pbsPartitionSelector',
+ allowBlank: true,
+ },
+ emptyText: gettext('Device path'),
+ },
],
column2: [
{
@@ -87,6 +101,29 @@ Ext.define('PBS.DataStoreEdit', {
},
],
columnB: [
+ {
+ xtype: 'checkbox',
+ boxLabel: gettext('Removable datastore'),
+ submitValue: false,
+ listeners: {
+ change: function(checkbox, isRemovable) {
+ let inputPanel = checkbox.up('inputpanel');
+ let pathField = inputPanel.down('[name=path]');
+ let uuidField = inputPanel.down('pbsPartitionSelector[name=backing-device]');
+ let uuidEditField = inputPanel.down('[name=backing-device]');
+
+ uuidField.allowBlank = !isRemovable;
+ uuidEditField.setDisabled(!isRemovable);
+ uuidField.setDisabled(!isRemovable);
+ uuidField.setValue('');
+ if (isRemovable) {
+ pathField.setFieldLabel(gettext('On device path'));
+ } else {
+ pathField.setFieldLabel(gettext('Backing Path'));
+ }
+ },
+ },
+ },
{
xtype: 'textfield',
name: 'comment',
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 17/26] ui: add (un)mount button to summary
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (15 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 16/26] ui: add removable datastore creation support Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 18/26] ui: tree: render unmounted datastores correctly Hannes Laimer
` (10 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
And only try to load datastore information if the datastore is
available.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/datastore/Summary.js | 92 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 90 insertions(+), 2 deletions(-)
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index a932b4e0..49aa3b3c 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -309,7 +309,82 @@ Ext.define('PBS.DataStoreSummary', {
model: 'pve-rrd-datastore',
});
- me.callParent();
+ me.statusStore = Ext.create('Proxmox.data.ObjectStore', {
+ url: `/api2/json/admin/datastore/${me.datastore}/status`,
+ interval: 1000,
+ });
+
+ let unmountBtn = Ext.create('Ext.Button', {
+ text: gettext('Unmount'),
+ hidden: true,
+ handler: () => {
+ Proxmox.Utils.API2Request({
+ url: `/admin/datastore/${me.datastore}/unmount`,
+ method: 'POST',
+ failure: function(response) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, options) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ }).show();
+ },
+ });
+ },
+ });
+
+ let mountBtn = Ext.create('Ext.Button', {
+ text: gettext('Mount'),
+ hidden: true,
+ handler: () => {
+ Proxmox.Utils.API2Request({
+ url: `/admin/datastore/${me.datastore}/mount`,
+ method: 'POST',
+ failure: function(response) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, options) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ }).show();
+ },
+ });
+ },
+ });
+
+ Ext.apply(me, {
+ tbar: [unmountBtn, mountBtn, '->', { xtype: 'proxmoxRRDTypeSelector' }],
+ });
+
+ me.mon(me.statusStore, 'load', (s, records, success) => {
+ if (!success) {
+ me.down('pbsDataStoreInfo').fireEvent('deactivate');
+ Proxmox.Utils.API2Request({
+ url: `/config/datastore/${me.datastore}`,
+ success: response => {
+ let mode = response.result.data['maintenance-mode'];
+ let [type, _message] = PBS.Utils.parseMaintenanceMode(mode);
+ if (!response.result.data['backing-device']) {
+ return;
+ }
+ if (!type || type === 'read-only') {
+ unmountBtn.setDisabled(true);
+ mountBtn.setDisabled(false);
+ } else if (type === 'unmount') {
+ unmountBtn.setDisabled(true);
+ mountBtn.setDisabled(true);
+ } else {
+ unmountBtn.setDisabled(false);
+ mountBtn.setDisabled(false);
+ }
+ },
+ });
+ } else {
+ me.down('pbsDataStoreInfo').fireEvent('activate');
+ unmountBtn.setDisabled(false);
+ mountBtn.setDisabled(true);
+ }
+ });
let sp = Ext.state.Manager.getProvider();
me.mon(sp, 'statechange', function(provider, key, value) {
@@ -322,11 +397,17 @@ Ext.define('PBS.DataStoreSummary', {
Proxmox.Utils.updateColumns(me);
});
+ me.callParent();
+
Proxmox.Utils.API2Request({
url: `/config/datastore/${me.datastore}`,
waitMsgTarget: me.down('pbsDataStoreInfo'),
success: function(response) {
- let path = Ext.htmlEncode(response.result.data.path);
+ let data = response.result.data;
+ let path = Ext.htmlEncode(data.path);
+ const removable = Object.prototype.hasOwnProperty.call(data, "backing-device");
+ unmountBtn.setHidden(!removable);
+ mountBtn.setHidden(!removable);
me.down('pbsDataStoreInfo').setTitle(`${me.datastore} (${path})`);
me.down('pbsDataStoreNotes').setNotes(response.result.data.comment);
},
@@ -344,6 +425,13 @@ Ext.define('PBS.DataStoreSummary', {
let hasIoTicks = records?.some((rec) => rec?.data?.io_ticks !== undefined);
me.down('#ioDelayChart').setVisible(!success || hasIoTicks);
}, undefined, { single: true });
+ me.on('afterrender', () => {
+ me.statusStore.startUpdate();
+ });
+
+ me.on('destroy', () => {
+ me.statusStore.stopUpdate();
+ });
me.query('proxmoxRRDChart').forEach((chart) => {
chart.setStore(me.rrdstore);
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 18/26] ui: tree: render unmounted datastores correctly
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (16 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 17/26] ui: add (un)mount button to summary Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 19/26] ui: utils: make parseMaintenanceMode more robust Hannes Laimer
` (9 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/NavigationTree.js | 15 ++++++++++++---
www/css/ext6-pbs.css | 8 ++++++++
www/datastore/DataStoreListSummary.js | 1 +
3 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index a5ea390f..4f1a8fd5 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -267,13 +267,22 @@ Ext.define('PBS.view.main.NavigationTree', {
j++;
}
- let [qtip, iconCls] = ['', 'fa fa-database'];
+ let mainIcon = `fa fa-${records[i].data.removable ? 'plug' : 'database'}`;
+ let [qtip, iconCls] = ['', mainIcon];
const maintenance = records[i].data.maintenance;
+ const removable_not_available = records[i].data.removable && !records[i].data.available;
+ if (removable_not_available) {
+ iconCls = `${mainIcon} pmx-tree-icon-custom unplugged`;
+ qtip = gettext('Removable datastore not mounted');
+ }
if (maintenance) {
const [type, message] = PBS.Utils.parseMaintenanceMode(maintenance);
qtip = `${type}${message ? ': ' + message : ''}`;
- let mainenanceTypeCls = type === 'delete' ? 'destroying' : 'maintenance';
- iconCls = `fa fa-database pmx-tree-icon-custom ${mainenanceTypeCls}`;
+ let maintenanceTypeCls = 'maintenance';
+ if (type === 'delete') {
+ maintenanceTypeCls = 'destroying';
+ }
+ iconCls = `${mainIcon} pmx-tree-icon-custom ${maintenanceTypeCls}`;
}
if (getChildTextAt(j).localeCompare(name) !== 0) {
diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index c33ce684..706e681e 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -271,6 +271,10 @@ span.snapshot-comment-column {
content: "\ ";
}
+.x-treelist-item-icon.fa-plug, .pmx-tree-icon-custom.fa-plug {
+ font-size: 12px;
+}
+
/* datastore maintenance */
.pmx-tree-icon-custom.maintenance:after {
content: "\f0ad";
@@ -290,6 +294,10 @@ span.snapshot-comment-column {
color: #888;
}
+.pmx-tree-icon-custom.unplugged:before {
+ color: #888;
+}
+
/*' PBS specific icons */
.pbs-icon-tape {
diff --git a/www/datastore/DataStoreListSummary.js b/www/datastore/DataStoreListSummary.js
index b908034d..f7ea83e7 100644
--- a/www/datastore/DataStoreListSummary.js
+++ b/www/datastore/DataStoreListSummary.js
@@ -22,6 +22,7 @@ Ext.define('PBS.datastore.DataStoreListSummary', {
stillbad: 0,
deduplication: 1.0,
error: "",
+ removable: false,
maintenance: '',
},
},
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 19/26] ui: utils: make parseMaintenanceMode more robust
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (17 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 18/26] ui: tree: render unmounted datastores correctly Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 20/26] ui: add datastore status mask for unmounted removable datastores Hannes Laimer
` (8 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Utils.js | 29 ++++++++++++++++++++++-------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/www/Utils.js b/www/Utils.js
index 40e7cbff..5b27b480 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -708,14 +708,29 @@ Ext.define('PBS.Utils', {
return `${icon} ${value}`;
},
- // FIXME: this "parser" is brittle and relies on the order the arguments will appear in
+ /**
+ * Parses maintenance mode property string.
+ * Examples:
+ * "offline,message=foo" -> ["offline", "foo"]
+ * "offline" -> ["offline", null]
+ * "message=foo,offline" -> ["offline", "foo"]
+ * null/undefined -> [null, null]
+ *
+ * @param {string|null} mode - Maintenance mode string to parse.
+ * @return {Array<string|null>} - Parsed maintenance mode values.
+ */
parseMaintenanceMode: function(mode) {
- let [type, message] = mode.split(/,(.+)/);
- type = type.split("=").pop();
- message = message ? message.split("=")[1]
- .replace(/^"(.*)"$/, '$1')
- .replaceAll('\\"', '"') : null;
- return [type, message];
+ if (!mode) {
+ return [null, null];
+ }
+ return mode.split(',').reduce(([m, msg], pair) => {
+ const [key, value] = pair.split('=');
+ if (key === 'message') {
+ return [m, value.replace(/^"(.*)"$/, '$1').replace(/\\"/g, '"')];
+ } else {
+ return [value ?? key, msg];
+ }
+ }, [null, null]);
},
renderMaintenance: function(mode, activeTasks) {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 20/26] ui: add datastore status mask for unmounted removable datastores
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (18 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 19/26] ui: utils: make parseMaintenanceMode more robust Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 21/26] ui: maintenance: fix disable msg field if no type is selected Hannes Laimer
` (7 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/css/ext6-pbs.css | 12 ++++++++++++
www/datastore/Summary.js | 21 +++++++++++++--------
2 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index 706e681e..891189ae 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -261,6 +261,18 @@ span.snapshot-comment-column {
content: "\f0ad";
}
+.pbs-unplugged-mask div.x-mask-msg-text {
+ background: None;
+ padding: 12px 0 0;
+}
+
+.pbs-unplugged-mask:before {
+ font-size: 3em;
+ display: flex;
+ justify-content: center;
+ content: "\f1e6";
+}
+
/* the small icons TODO move to proxmox-widget-toolkit */
.pmx-tree-icon-custom:after {
position: relative;
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index 49aa3b3c..33b4e18a 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -61,16 +61,21 @@ Ext.define('PBS.DataStoreInfo', {
Proxmox.Utils.API2Request({
url: `/config/datastore/${me.view.datastore}`,
success: function(response) {
- const config = response.result.data;
- if (config['maintenance-mode']) {
- const [_type, msg] = PBS.Utils.parseMaintenanceMode(config['maintenance-mode']);
- me.view.el.mask(
- `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`,
- 'fa pbs-maintenance-mask',
- );
- } else {
+ let maintenanceString = response.result.data['maintenance-mode'];
+ let removable = !!response.result.data['backing-device'];
+ if (!maintenanceString && !removable) {
me.view.el.mask(gettext('Datastore is not available'));
+ return;
}
+
+ let [_type, msg] = PBS.Utils.parseMaintenanceMode(maintenanceString);
+ let isUnplugged = !maintenanceString && removable;
+ let maskMessage = isUnplugged
+ ? gettext('Datastore is not mounted')
+ : `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`;
+
+ let maskIcon = isUnplugged ? 'fa pbs-unplugged-mask' : 'fa pbs-maintenance-mask';
+ me.view.el.mask(maskMessage, maskIcon);
},
});
return;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 21/26] ui: maintenance: fix disable msg field if no type is selected
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (19 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 20/26] ui: add datastore status mask for unmounted removable datastores Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 22/26] ui: render 'unmount' maintenance mode correctly Hannes Laimer
` (6 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/window/MaintenanceOptions.js | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 1ee92542..527c3698 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -56,12 +56,17 @@ Ext.define('PBS.window.MaintenanceOptions', {
fieldLabel: gettext('Maintenance Type'),
value: '__default__',
deleteEmpty: true,
+ listeners: {
+ change: (field, newValue) => {
+ Ext.getCmp('message-field').setDisabled(newValue === '__default__');
+ },
+ },
},
{
xtype: 'proxmoxtextfield',
+ id: 'message-field',
name: 'maintenance-msg',
fieldLabel: gettext('Description'),
- // FIXME: disable if maintenance type is none
},
],
},
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 22/26] ui: render 'unmount' maintenance mode correctly
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (20 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 21/26] ui: maintenance: fix disable msg field if no type is selected Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 23/26] api: node: allow creation of removable datastore through directory endpoint Hannes Laimer
` (5 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Utils.js | 4 +++-
www/window/MaintenanceOptions.js | 10 ++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/www/Utils.js b/www/Utils.js
index 5b27b480..029f690d 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -743,7 +743,7 @@ Ext.define('PBS.Utils', {
let extra = '';
if (activeTasks !== undefined) {
- const conflictingTasks = activeTasks.write + (type === 'offline' ? activeTasks.read : 0);
+ const conflictingTasks = activeTasks.write + (type === 'offline' || type === 'unmount' ? activeTasks.read : 0);
if (conflictingTasks > 0) {
extra += '| <i class="fa fa-spinner fa-pulse fa-fw"></i> ';
@@ -763,6 +763,8 @@ Ext.define('PBS.Utils', {
break;
case 'offline': modeText = gettext("Offline");
break;
+ case 'unmount': modeText = gettext("Unmounting");
+ break;
}
return `${modeText} ${extra}`;
},
diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 527c3698..d7348cb4 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -52,6 +52,7 @@ Ext.define('PBS.window.MaintenanceOptions', {
items: [
{
xtype: 'pbsMaintenanceType',
+ id: 'type-field',
name: 'maintenance-type',
fieldLabel: gettext('Maintenance Type'),
value: '__default__',
@@ -85,6 +86,15 @@ Ext.define('PBS.window.MaintenanceOptions', {
};
}
+ let unmounting = options['maintenance-type'] === 'unmount';
+ let defaultType = options['maintenance-type'] === '__default__';
+ if (unmounting) {
+ options['maintenance-type'] = '';
+ }
+
me.callParent([options]);
+
+ Ext.ComponentManager.get('type-field').setDisabled(unmounting);
+ Ext.ComponentManager.get('message-field').setDisabled(unmounting || defaultType);
},
});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 23/26] api: node: allow creation of removable datastore through directory endpoint
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (21 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 22/26] ui: render 'unmount' maintenance mode correctly Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 24/26] api: node: include removable datastores in directory list Hannes Laimer
` (4 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 67 +++++++++++++++++++++++++++++---
1 file changed, 62 insertions(+), 5 deletions(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 73af92cc..25d43c79 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -122,6 +122,11 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
description: "Configure a datastore using the directory.",
type: bool,
optional: true,
+ default: false,
+ },
+ "removable-datastore": {
+ description: "The added datastore is removable.",
+ type: bool,
},
filesystem: {
type: FileSystemType,
@@ -140,7 +145,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
pub fn create_datastore_disk(
name: String,
disk: String,
- add_datastore: Option<bool>,
+ add_datastore: bool,
+ removable_datastore: bool,
filesystem: Option<FileSystemType>,
rpcenv: &mut dyn RpcEnvironment,
) -> Result<String, Error> {
@@ -154,8 +160,59 @@ pub fn create_datastore_disk(
bail!("disk '{}' is already in use.", disk);
}
- let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+ if add_datastore && removable_datastore {
+ let upid_str = WorkerTask::new_thread(
+ "dircreate",
+ Some(name.clone()),
+ auth_id,
+ to_stdout,
+ move |worker| {
+ task_log!(
+ worker,
+ "create removable datastore '{}' on disk {}",
+ name,
+ disk
+ );
+
+ let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
+
+ let manager = DiskManage::new();
+
+ let disk = manager.disk_by_name(&disk)?;
+
+ let partition = create_single_linux_partition(&disk)?;
+ create_file_system(&partition, filesystem)?;
+
+ let uuid = get_fs_uuid(&partition)?;
+ let lock = pbs_config::datastore::lock_config()?;
+ let datastore: DataStoreConfig = serde_json::from_value(
+ json!({ "name": name, "path": name, "backing-device": uuid }),
+ )?;
+
+ let (config, _digest) = pbs_config::datastore::config()?;
+
+ if config.sections.get(&datastore.name).is_some() {
+ bail!("datastore '{}' already exists.", datastore.name);
+ }
+
+ // we don't have to check if the UUID is already in use since we just created the
+ // fs ourself
+
+ crate::api2::config::datastore::do_create_datastore(
+ lock,
+ config,
+ datastore,
+ Some(&worker),
+ )?;
+
+ Ok(())
+ },
+ )?;
+ return Ok(upid_str);
+ };
+
+ let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
// check if the default path exists already.
// bail if it is not empty or another filesystem mounted on top
let default_path = std::path::PathBuf::from(&mount_point);
@@ -182,7 +239,6 @@ pub fn create_datastore_disk(
move |worker| {
task_log!(worker, "create datastore '{}' on disk {}", name, disk);
- let add_datastore = add_datastore.unwrap_or(false);
let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
let manager = DiskManage::new();
@@ -250,8 +306,9 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
// path of datastore cannot be changed
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
- let conflicting_datastore: Option<DataStoreConfig> =
- datastores.into_iter().find(|ds| ds.absolute_path() == path);
+ let conflicting_datastore: Option<DataStoreConfig> = datastores.into_iter().find(|ds| {
+ ds.absolute_path() == path || ds.get_mount_point().map_or(false, |mp| mp == path)
+ });
if let Some(conflicting_datastore) = conflicting_datastore {
bail!(
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 24/26] api: node: include removable datastores in directory list
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (22 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 23/26] api: node: allow creation of removable datastore through directory endpoint Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
` (3 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 25d43c79..55bf6be0 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -43,6 +43,8 @@ pub struct DatastoreMountInfo {
pub path: String,
/// The mounted device.
pub device: String,
+ /// This is removable
+ pub removable: bool,
/// File system type
pub filesystem: Option<String>,
/// Mount options
@@ -59,7 +61,7 @@ pub struct DatastoreMountInfo {
}
},
returns: {
- description: "List of systemd datastore mount units.",
+ description: "List of removable-datastore devices and systemd datastore mount units.",
type: Array,
items: {
type: DatastoreMountInfo,
@@ -99,6 +101,31 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
path: data.Where,
filesystem: data.Type,
options: data.Options,
+ removable: false,
+ });
+ }
+
+ let (config, _digest) = pbs_config::datastore::config()?;
+ let store_list: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
+
+ for item in store_list
+ .into_iter()
+ .filter(|store| store.backing_device.is_some())
+ {
+ let Some(backing_device) = item.backing_device.as_deref() else {
+ continue;
+ };
+ let Some(mount_point) = item.get_mount_point() else {
+ continue;
+ };
+ list.push(DatastoreMountInfo {
+ unitfile: "datastore config".to_string(),
+ name: item.name.clone(),
+ device: format!("/dev/disk/by-uuid/{backing_device}"),
+ path: mount_point,
+ filesystem: None,
+ options: None,
+ removable: true,
});
}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (23 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 24/26] api: node: include removable datastores in directory list Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 26/26] ui: support create removable datastore through directory creation Hannes Laimer
` (2 subsequent siblings)
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 55bf6be0..94d271ff 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -9,8 +9,8 @@ use proxmox_section_config::SectionConfigData;
use proxmox_sys::task_log;
use pbs_api_types::{
- DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_SCHEMA, NODE_SCHEMA, PRIV_SYS_AUDIT,
- PRIV_SYS_MODIFY, UPID_SCHEMA,
+ DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_MOUNT_DIR, DATASTORE_SCHEMA, NODE_SCHEMA,
+ PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, UPID_SCHEMA,
};
use crate::tools::disks::{
@@ -21,8 +21,6 @@ use crate::tools::systemd::{self, types::*};
use proxmox_rest_server::WorkerTask;
-const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
-
#[api(
properties: {
"filesystem": {
@@ -90,7 +88,7 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
let name = data
.Where
- .strip_prefix(BASE_MOUNT_DIR)
+ .strip_prefix(DATASTORE_MOUNT_DIR)
.unwrap_or(&data.Where)
.to_string();
@@ -239,7 +237,7 @@ pub fn create_datastore_disk(
return Ok(upid_str);
};
- let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+ let mount_point = format!("{}/{}", DATASTORE_MOUNT_DIR, &name);
// check if the default path exists already.
// bail if it is not empty or another filesystem mounted on top
let default_path = std::path::PathBuf::from(&mount_point);
@@ -247,7 +245,7 @@ pub fn create_datastore_disk(
match std::fs::metadata(&default_path) {
Err(_) => {} // path does not exist
Ok(stat) => {
- let basedir_dev = std::fs::metadata(BASE_MOUNT_DIR)?.st_dev();
+ let basedir_dev = std::fs::metadata(DATASTORE_MOUNT_DIR)?.st_dev();
if stat.st_dev() != basedir_dev {
bail!("path {default_path:?} already exists and is mountpoint");
}
@@ -329,7 +327,7 @@ pub fn create_datastore_disk(
)]
/// Remove a Filesystem mounted under `/mnt/datastore/<name>`.
pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
- let path = format!("{}{}", BASE_MOUNT_DIR, name);
+ let path = format!("{}/{}", DATASTORE_MOUNT_DIR, name);
// path of datastore cannot be changed
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v10 26/26] ui: support create removable datastore through directory creation
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (24 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
@ 2024-04-25 6:53 ` Hannes Laimer
2024-05-03 11:34 ` [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Max Carrara
2024-08-20 15:47 ` Hannes Laimer
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-04-25 6:53 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 2 ++
www/DirectoryList.js | 13 +++++++++++++
www/window/CreateDirectory.js | 14 ++++++++++++++
3 files changed, 29 insertions(+)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 94d271ff..3a6cf8c3 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -152,6 +152,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
"removable-datastore": {
description: "The added datastore is removable.",
type: bool,
+ optional: true,
+ default: false,
},
filesystem: {
type: FileSystemType,
diff --git a/www/DirectoryList.js b/www/DirectoryList.js
index adefa9ab..25921a62 100644
--- a/www/DirectoryList.js
+++ b/www/DirectoryList.js
@@ -121,6 +121,19 @@ Ext.define('PBS.admin.Directorylist', {
],
columns: [
+ {
+ text: '<span class="fa fa-plug"/>',
+ flex: 0,
+ width: 35,
+ dataIndex: 'removable',
+ renderer: function(_text, _, row) {
+ if (row.data.removable) {
+ return `<i class="fa fa-check"/>`;
+ } else {
+ return '';
+ }
+ },
+ },
{
text: gettext('Path'),
dataIndex: 'path',
diff --git a/www/window/CreateDirectory.js b/www/window/CreateDirectory.js
index 6aabe21a..38d6979d 100644
--- a/www/window/CreateDirectory.js
+++ b/www/window/CreateDirectory.js
@@ -43,6 +43,20 @@ Ext.define('PBS.window.CreateDirectory', {
name: 'add-datastore',
fieldLabel: gettext('Add as Datastore'),
value: '1',
+ listeners: {
+ change(field, newValue, _oldValue) {
+ let form = field.up('form');
+ let rmBox = form.down('[name=removable-datastore]');
+
+ rmBox.setDisabled(!newValue);
+ rmBox.setValue(false);
+ },
+ },
+ },
+ {
+ xtype: 'proxmoxcheckbox',
+ name: 'removable-datastore',
+ fieldLabel: gettext('is removable'),
},
],
});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions Hannes Laimer
@ 2024-05-03 11:33 ` Max Carrara
0 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:33 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:52 AM CEST, Hannes Laimer wrote:
> ... for mounting and unmounting
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/tools/disks/mod.rs | 19 +++++++++++++++++++
> 1 file changed, 19 insertions(+)
>
> diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
> index 94f89e0a..303fe4f1 100644
> --- a/src/tools/disks/mod.rs
> +++ b/src/tools/disks/mod.rs
> @@ -1323,3 +1323,22 @@ pub fn get_fs_uuid(disk: &Disk) -> Result<String, Error> {
>
> bail!("get_fs_uuid failed - missing UUID");
> }
> +
> +/// Mount a disk by its UUID and the mount point.
> +pub fn mount_by_uuid(uuid: &str, mount_point: &Path) -> Result<(), Error> {
Wouldn't it make sense to use our own UUID type from `proxmox_uuid`
here?
> + let mut command = std::process::Command::new("mount");
> + command.arg(&format!("UUID={uuid}"));
> + command.arg(mount_point);
> +
> + proxmox_sys::command::run_command(command, None)?;
> + Ok(())
> +}
> +
> +/// Unmount a disk by its mount point.
> +pub fn unmount_by_mountpoint(path: &str) -> Result<(), Error> {
> + let mut command = std::process::Command::new("umount");
> + command.arg(path);
> +
> + proxmox_sys::command::run_command(command, None)?;
> + Ok(())
> +}
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
@ 2024-05-03 11:34 ` Max Carrara
0 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:34 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:52 AM CEST, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/datastore.rs | 27 ++++++++++++++++++++++++++-
> 1 file changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> index a5704c93..e5c5cfcf 100644
> --- a/pbs-api-types/src/datastore.rs
> +++ b/pbs-api-types/src/datastore.rs
> @@ -160,6 +160,9 @@ pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema =
> .minimum(1)
> .schema();
>
> +/// Base directory where datastores are mounted
> +pub const DATASTORE_MOUNT_DIR: &str = "/mnt/datastore";
> +
> #[api]
> #[derive(Debug, Default, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
> #[serde(rename_all = "lowercase")]
> @@ -273,6 +276,12 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
> format: &ApiStringFormat::PropertyString(&MaintenanceMode::API_SCHEMA),
> type: String,
> },
> + "backing-device": {
> + description: "The UUID of the filesystem partition for removable datastores.",
> + optional: true,
> + format: &proxmox_schema::api_types::UUID_FORMAT,
> + type: String,
> + }
> }
> )]
> #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
> @@ -320,6 +329,11 @@ pub struct DataStoreConfig {
> /// Maintenance mode, type is either 'offline' or 'read-only', message should be enclosed in "
> #[serde(skip_serializing_if = "Option::is_none")]
> pub maintenance_mode: Option<String>,
> +
> + /// The UUID of the device(for removable datastores)
> + #[updater(skip)]
> + #[serde(skip_serializing_if = "Option::is_none")]
> + pub backing_device: Option<String>,
Here as well, maybe better to use our own UUID type. It even has serde
support :)
> }
>
> #[api]
> @@ -354,12 +368,23 @@ impl DataStoreConfig {
> notification_mode: None,
> tuning: None,
> maintenance_mode: None,
> + backing_device: None,
> }
> }
>
> /// Returns the absolute path to the datastore content.
> pub fn absolute_path(&self) -> String {
> - self.path.clone()
> + if let Some(mount_point) = self.get_mount_point() {
> + format!("{mount_point}/{}", self.path.trim_matches('/'))
> + } else {
> + self.path.clone()
> + }
> + }
> +
> + pub fn get_mount_point(&self) -> Option<String> {
> + self.backing_device
> + .is_some()
> + .then(|| format!("{DATASTORE_MOUNT_DIR}/{}", self.name))
> }
>
> pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info Hannes Laimer
@ 2024-05-03 11:34 ` Max Carrara
0 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:34 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:52 AM CEST, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/tools/disks/mod.rs | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
> index 303fe4f1..6227948d 100644
> --- a/src/tools/disks/mod.rs
> +++ b/src/tools/disks/mod.rs
> @@ -59,6 +59,8 @@ pub struct LsblkInfo {
> /// File system label.
> #[serde(rename = "fstype")]
> file_system_type: Option<String>,
> + /// File system UUID.
> + uuid: Option<String>,
Maybe use `proxmox_uuid` here as well.
> }
>
> impl DiskManage {
> @@ -617,7 +619,7 @@ pub struct BlockDevStat {
> /// Use lsblk to read partition type uuids and file system types.
> pub fn get_lsblk_info() -> Result<Vec<LsblkInfo>, Error> {
> let mut command = std::process::Command::new("lsblk");
> - command.args(["--json", "-o", "path,parttype,fstype"]);
> + command.args(["--json", "-o", "path,parttype,fstype,uuid"]);
>
> let output = proxmox_sys::command::run_command(command, None)?;
>
> @@ -701,6 +703,8 @@ pub struct PartitionInfo {
> pub size: Option<u64>,
> /// GPT partition
> pub gpt: bool,
> + /// UUID
> + pub uuid: Option<String>,
.. and here too.
> }
>
> #[api(
> @@ -891,8 +895,10 @@ fn get_partitions_info(
>
> let mounted = disk.is_mounted().unwrap_or(false);
> let mut filesystem = None;
> + let mut uuid = None;
> if let (Some(devpath), Some(infos)) = (devpath.as_ref(), lsblk_infos.as_ref()) {
> for info in infos.iter().filter(|i| i.path.eq(devpath)) {
> + uuid = info.uuid.clone();
> used = match info.partition_type.as_deref() {
> Some("21686148-6449-6e6f-744e-656564454649") => PartitionUsageType::BIOS,
> Some("c12a7328-f81f-11d2-ba4b-00a0c93ec93b") => PartitionUsageType::EFI,
> @@ -915,6 +921,7 @@ fn get_partitions_info(
> filesystem,
> size: disk.size().ok(),
> gpt: disk.has_gpt(),
> + uuid,
> }
> })
> .collect()
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores Hannes Laimer
@ 2024-05-03 11:34 ` Max Carrara
0 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:34 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:53 AM CEST, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> debian/proxmox-backup-server.install | 1 +
> debian/proxmox-backup-server.udev | 3 ++
> etc/Makefile | 3 +-
> etc/removable-device-attach@.service.in | 8 +++
> src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
> 5 files changed, 68 insertions(+), 2 deletions(-)
> create mode 100644 etc/removable-device-attach@.service.in
>
> diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
> index ef1e9ba1..34bfc454 100644
> --- a/debian/proxmox-backup-server.install
> +++ b/debian/proxmox-backup-server.install
> @@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
> etc/proxmox-backup-daily-update.timer /lib/systemd/system/
> etc/proxmox-backup-proxy.service /lib/systemd/system/
> etc/proxmox-backup.service /lib/systemd/system/
> +etc/removable-device-attach@.service /lib/systemd/system/
> usr/bin/pmt
> usr/bin/pmtx
> usr/bin/proxmox-tape
> diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
> index afdfb2bc..e21b8bc7 100644
> --- a/debian/proxmox-backup-server.udev
> +++ b/debian/proxmox-backup-server.udev
> @@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
> SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
>
> LABEL="persistent_storage_tape_end"
> +
> +# triggers the mounting of a removable device
> +ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
> \ No newline at end of file
> diff --git a/etc/Makefile b/etc/Makefile
> index 42f639f6..730de4f8 100644
> --- a/etc/Makefile
> +++ b/etc/Makefile
> @@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
> proxmox-backup-banner.service \
> proxmox-backup-daily-update.service \
> proxmox-backup.service \
> - proxmox-backup-proxy.service
> + proxmox-backup-proxy.service \
> + removable-device-attach@.service
>
> all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
>
> diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
> new file mode 100644
> index 00000000..e10d1ea3
> --- /dev/null
> +++ b/etc/removable-device-attach@.service.in
> @@ -0,0 +1,8 @@
> +[Unit]
> +Description=Try to mount the removable device of a datastore with uuid '%i'.
> +After=proxmox-backup-proxy.service
> +Requires=proxmox-backup-proxy.service
> +
> +[Service]
> +Type=simple
> +ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
> diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> index f2795b39..e74c8537 100644
> --- a/src/bin/proxmox_backup_manager/datastore.rs
> +++ b/src/bin/proxmox_backup_manager/datastore.rs
> @@ -1,4 +1,4 @@
> -use anyhow::{format_err, Error};
> +use anyhow::{bail, format_err, Error};
> use serde_json::Value;
>
> use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> @@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> Ok(())
> }
>
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + uuid: {
> + type: String,
> + description: "The UUID of the device that should be mounted",
Not sure if we can parse the UUID with our own type from `proxmox_uuid`
here, but would be cool if that worked.
> + },
> + "output-format": {
> + schema: OUTPUT_FORMAT,
> + optional: true,
> + },
> + },
> + },
> +)]
> +/// Try mounting a removable datastore given the UUID.
> +async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> + let uuid = param["uuid"]
> + .as_str()
> + .ok_or_else(|| format_err!("uuid has to be specified"))?;
Otherwise, I'd use our own UUID type here too.
> +
> + let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
> + let data: Value = match info.handler {
> + ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
> + _ => unreachable!(),
> + };
> +
> + let store_name = data.as_array().and_then(|list| {
> + list.iter()
> + .filter_map(Value::as_object)
> + .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
> + .and_then(|s| s.get("name").and_then(Value::as_str))
> + });
> +
> + if let Some(store_name) = store_name {
> + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> + let mount_param = serde_json::json!({
> + "store": store_name,
> + });
> + let result = match info.handler {
> + ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
> + _ => unreachable!(),
> + };
> + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> + return Ok(Value::Null);
> + }
> + bail!("'{uuid}' is not associated with any datastore")
> +}
> +
> pub fn datastore_commands() -> CommandLineInterface {
> let cmd_def = CliCommandMap::new()
> .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> @@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
> pbs_config::datastore::complete_calendar_event,
> ),
> )
> + .insert(
> + "uuid-mount",
> + CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
> + )
> .insert(
> "remove",
> CliCommand::new(&API_METHOD_DELETE_DATASTORE)
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section Hannes Laimer
@ 2024-05-03 11:34 ` Max Carrara
0 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:34 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:53 AM CEST, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> docs/storage.rst | 29 +++++++++++++++++++++++++++++
> 1 file changed, 29 insertions(+)
>
> diff --git a/docs/storage.rst b/docs/storage.rst
> index 4444c423..0a2f8f8a 100644
> --- a/docs/storage.rst
> +++ b/docs/storage.rst
> @@ -165,6 +165,35 @@ following command creates a new datastore called ``store1`` on
> # proxmox-backup-manager datastore create store1 /backup/disk1/store1
>
>
> +Removable Datastores
> +^^^^^^^^^^^^^^^^^^^^
> +Removable datastores have a ``backing-device`` associated with them, they can be
> +mounted and unmounted. Other than that they behave the same way a normal datastore
> +would.
> +
> +They can be created on already correctly formatted partitions, which, as with normal
> +datastores, should be either ``ext4`` or ``xfs``. It is also possible to create them
> +on completely unused disks through "Admnistartion" > "Disks / Storage" > "Directory",
Typo here and wrong name here - should be:
"Administration" > "Storage / Disks"
> +using this method the disk will be partitioned and formatted automatically for the datastore.
> +
> +They are mounted automatically when plugged in. It is possible to create a
> +removable datastore on one PBS and use it on multiple, the device just has to be added
> +on all as a removable datastore. This can be done using the normal `Create Datastore`
> +form, by checking **Removable datastore**, selecting the device and setting the correct path.
> +If the device already contains a datastore at the specified path it'll just be added as
> +a new datastore to the PBS instance and will be mounted whenever plugged in. Unmounting has
> +to be done through the UI by clicking "Unmount" on the summary page or using the CLI
> +
> +.. code-block:: console
> +
> + # proxmox-backup-manager datastore unmount store1
> +
> +both will wait for any running tasks to finish and unmount the device.
> +
> +All removable datastores are mounted under /mnt/datastore/<name>, and the specified path
> +refers to the path on the device.
> +
> +
> Managing Datastores
> ^^^^^^^^^^^^^^^^^^^
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (25 preceding siblings ...)
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 26/26] ui: support create removable datastore through directory creation Hannes Laimer
@ 2024-05-03 11:34 ` Max Carrara
2024-08-20 15:47 ` Hannes Laimer
27 siblings, 0 replies; 34+ messages in thread
From: Max Carrara @ 2024-05-03 11:34 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Thu Apr 25, 2024 at 8:52 AM CEST, Hannes Laimer wrote:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
Looks pretty solid! Applied the patches and gave it a spin on my test
instance. There are some more comments inline, mostly suggesting to use
our own UUID type from `proxmox_uuid` instead of plain strings.
Otherwise, see the sections below.
Testing
=======
* Attached a new disk to my VM
* Created a new removable datastore via the Storage / Disks menu
* Configured the new datastore in my host's storage
* Ran a couple backups
* Set up Prune, GC and Verify jobs
* Set up Sync job to pull from other local datastore
* Manually unmounted and mounted the datastore, UI reacted smoothly
(very dank actually)
Now to the more fun part: I decided to ignore the docs and detach the
(SCSI) disk I set the datastore up on in the GUI of my host. Lo and
behold, the datastore was still unmounted correctly; at least the UI
didn't complain. As soon as I attached the disk again, the datastore was
almost immediately found. Very robust, I like it!
Code Review
===========
I've got nothing to complain - apart from the comments inline regarding
UUIDs. The vast majority of patches are very easy to follow, each change
is kept comprehensible. The context of the commits is very clear - no
commit tries to do too many things at once. Very good work!
Regarding Failed Jobs
=====================
One thing I noticed is that jobs will still run and consequently fail if
the datastore is unmounted. I guess that's behaving as expected, but if
I put myself in the user's shoes, I'd prefer those jobs to not run at
all instead of failing on every run.
If I (for example) have a job that runs every hour, that would amount to
24 errors a day, which might be a lot of noise - if I have 168 failed
jobs in a week and some other e.g. *very important* job happens to fail
some time inbetween, that very important job now gets lost between those
(IMO) unnecessary fails.
That's IMHO the only thing that requires some more attention
(disclaimer: I'm not sure if there's been some discussion about this
already); I've got no complaints otherwise. Pretty solid work I gotta
say!
Bikeshedding
============
The lesser important thing I wanted to bring up is: Is "Removable
Datastore" really the right name? I would personally prefer something
like "Pluggable Datastore" or something similar, because all datastores
can be removed - it's just that the features added in this series allows
*some* of them to be re-attached again.
Specifically, there's a "Remove Datastore" button already in the options
section of datastores; I think the name would conflict with the meaning
of this button. (Sure, you could argue that it's context-dependent, but
I still don't think that those should clash.)
>
> The datastore path specified is relative to `/mnt/removable_datastore/<name>`.
>
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted in. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in.
>
>
> v10: thanks @Gabriel and @Wolfgang
> * make is_datastore_available more robust
> * fix a lot of wording
> * drop format on uuid_mount command for UUID
> * only gather_disk_stats if datastore is available
> * overall code improvements
> * ui: include model in partition selector
> * rebased onto master
>
> v9:
> * change mount point to `/mnt/datastore/<name>`
> * update "Directory" list UI
> * add `absolute_path()` from Dietmar's RFC
> * update docs
>
> v8:
> * still depends on [1]
> * paths for removable datastores are now relative to
> `/mnt/removable_datastore/<UUID>`
> * add support for creation of removable datastore through the
> "create directory" endpoint (last 3 patches)
> * update datastore creation UI
> * update docs
>
> v7:
> * depends on [1]
> * improve logging when waiting for tasks
> * drop `update-datatore-cache` refactoring
> * fix some commit messages
>
> [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
>
> v6:
> * remove 'drop' flag in datastore cache
> * use maintenance-mode 'unmount' for unmounting process, only for the
> unmounting not for being unmounted
> * rename/simplify update-datastore-cache command
> * ui: integrate new unmounting maintenance mode
> * basically a mix of v3 and v4
>
> v5: thanks @Dietmar and @Christian
> * drop --force for unmount since it'll always fail if tasks are still running, and if
> there are not normal unount will work
> * improve several commit messages
> * improve error message wording
> * add removable datastore section to docs
> * add documentation for is_datastore_available
>
> v4: thanks a lot @Dietmar and @Christian
> * make check if mounted wayyy faster
> * don't keep track of mounting state
> * drop Unplugged maintenance mode
> * use UUID_FORMAT for uuid field
> * a lot of small things, like use of bail!, inline format!, ...
> * include improvement to cache handling
>
> v3:
> * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
> * fix CLI (un)mount command, thanks @Gabriel
> * add removable datastore CLI autocomplete helper
> * rebase onto master
> * move ui patches to the end
>
> thanks @Lukas and @Thomas for the feedback
> v2:
> * fix datastore 'add' button in the UI
> * some format!("{}", a) -> format!("{a}")
> * replace `const` with `let` in js code
> * change icon `fa-usb` -> `fa-plug`
> * add some docs
> * add JDoc for parseMaintenanceMode
> * proxmox-schema dep bump
>
> Dietmar Maurer (2):
> config: factor out method to get the absolute datastore path
> maintenance: add 'Unmount' maintenance type
>
> Hannes Laimer (24):
> tools: add disks utility functions
> pbs-api-types: add backing-device to DataStoreConfig
> disks: add UUID to partition info
> datastore: add helper for checking if a removable datastore is
> available
> api: admin: add (un)mount endpoint for removable datastores
> api: removable datastore creation
> api: disks list: add exclude-used flag
> pbs-api-types: add removable/is-available flag to DataStoreListItem
> bin: manager: add (un)mount command
> add auto-mounting for removable datastores
> datastore: handle deletion of removable datastore properly
> docs: add removable datastores section
> ui: add partition selector form
> ui: add removable datastore creation support
> ui: add (un)mount button to summary
> ui: tree: render unmounted datastores correctly
> ui: utils: make parseMaintenanceMode more robust
> ui: add datastore status mask for unmounted removable datastores
> ui: maintenance: fix disable msg field if no type is selected
> ui: render 'unmount' maintenance mode correctly
> api: node: allow creation of removable datastore through directory
> endpoint
> api: node: include removable datastores in directory list
> node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
> ui: support create removable datastore through directory creation
>
> debian/proxmox-backup-server.install | 1 +
> debian/proxmox-backup-server.udev | 3 +
> docs/storage.rst | 29 +++
> etc/Makefile | 3 +-
> etc/removable-device-attach@.service.in | 8 +
> pbs-api-types/src/datastore.rs | 42 +++-
> pbs-api-types/src/maintenance.rs | 11 +-
> pbs-config/src/datastore.rs | 14 ++
> pbs-datastore/src/datastore.rs | 79 +++++++-
> pbs-datastore/src/lib.rs | 2 +-
> src/api2/admin/datastore.rs | 203 ++++++++++++++++++--
> src/api2/config/datastore.rs | 69 ++++++-
> src/api2/node/disks/directory.rs | 112 +++++++++--
> src/api2/node/disks/mod.rs | 8 +
> src/api2/status.rs | 18 +-
> src/bin/proxmox-backup-proxy.rs | 9 +-
> src/bin/proxmox_backup_manager/datastore.rs | 129 ++++++++++++-
> src/tools/disks/mod.rs | 89 ++++++++-
> www/DirectoryList.js | 13 ++
> www/Makefile | 1 +
> www/NavigationTree.js | 15 +-
> www/Utils.js | 33 +++-
> www/css/ext6-pbs.css | 20 ++
> www/datastore/DataStoreListSummary.js | 1 +
> www/datastore/Summary.js | 113 ++++++++++-
> www/form/PartitionSelector.js | 81 ++++++++
> www/window/CreateDirectory.js | 14 ++
> www/window/DataStoreEdit.js | 37 ++++
> www/window/MaintenanceOptions.js | 17 +-
> 29 files changed, 1086 insertions(+), 88 deletions(-)
> create mode 100644 etc/removable-device-attach@.service.in
> create mode 100644 www/form/PartitionSelector.js
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
` (26 preceding siblings ...)
2024-05-03 11:34 ` [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Max Carrara
@ 2024-08-20 15:47 ` Hannes Laimer
27 siblings, 0 replies; 34+ messages in thread
From: Hannes Laimer @ 2024-08-20 15:47 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
outdated, v11 -> [1]
[1] https://lists.proxmox.com/pipermail/pbs-devel/2024-August/010620.html
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 34+ messages in thread
end of thread, other threads:[~2024-08-20 15:47 UTC | newest]
Thread overview: 34+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-25 6:52 [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 01/26] tools: add disks utility functions Hannes Laimer
2024-05-03 11:33 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 05/26] disks: add UUID to partition info Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 08/26] api: removable datastore creation Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 09/26] api: disks list: add exclude-used flag Hannes Laimer
2024-04-25 6:52 ` [pbs-devel] [PATCH proxmox-backup v10 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 11/26] bin: manager: add (un)mount command Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 12/26] add auto-mounting for removable datastores Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 14/26] docs: add removable datastores section Hannes Laimer
2024-05-03 11:34 ` Max Carrara
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 15/26] ui: add partition selector form Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 16/26] ui: add removable datastore creation support Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 17/26] ui: add (un)mount button to summary Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 18/26] ui: tree: render unmounted datastores correctly Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 19/26] ui: utils: make parseMaintenanceMode more robust Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 20/26] ui: add datastore status mask for unmounted removable datastores Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 21/26] ui: maintenance: fix disable msg field if no type is selected Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 22/26] ui: render 'unmount' maintenance mode correctly Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 23/26] api: node: allow creation of removable datastore through directory endpoint Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 24/26] api: node: include removable datastores in directory list Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
2024-04-25 6:53 ` [pbs-devel] [PATCH proxmox-backup v10 26/26] ui: support create removable datastore through directory creation Hannes Laimer
2024-05-03 11:34 ` [pbs-devel] [PATCH proxmox-backup v10 00/26] add removable datastores Max Carrara
2024-08-20 15:47 ` Hannes Laimer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox