* [pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
` (25 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
... for mounting and unmounting
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/tools/disks/mod.rs | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index c729c26a..9d8ff05c 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -1315,3 +1315,33 @@ pub fn get_fs_uuid(disk: &Disk) -> Result<String, Error> {
bail!("get_fs_uuid failed - missing UUID");
}
+
+/// Mount a disk by its UUID and the mount point.
+pub fn mount_by_uuid(uuid: &str, mount_point: &Path) -> Result<(), Error> {
+ let mut command = std::process::Command::new("mount");
+ command.arg(&format!("UUID={uuid}"));
+ command.arg(mount_point);
+
+ proxmox_sys::command::run_command(command, None)?;
+ Ok(())
+}
+
+/// Create bind mount.
+pub fn bind_mount(path: &Path, target: &Path) -> Result<(), Error> {
+ let mut command = std::process::Command::new("mount");
+ command.arg("--bind");
+ command.arg(path);
+ command.arg(target);
+
+ proxmox_sys::command::run_command(command, None)?;
+ Ok(())
+}
+
+/// Unmount a disk by its mount point.
+pub fn unmount_by_mountpoint(path: &str) -> Result<(), Error> {
+ let mut command = std::process::Command::new("umount");
+ command.arg(path);
+
+ proxmox_sys::command::run_command(command, None)?;
+ Ok(())
+}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> ... for mounting and unmounting
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/tools/disks/mod.rs | 30 ++++++++++++++++++++++++++++++
> 1 file changed, 30 insertions(+)
>
> diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
> index c729c26a..9d8ff05c 100644
> --- a/src/tools/disks/mod.rs
> +++ b/src/tools/disks/mod.rs
> @@ -1315,3 +1315,33 @@ pub fn get_fs_uuid(disk: &Disk) -> Result<String, Error> {
>
> bail!("get_fs_uuid failed - missing UUID");
> }
> +
> +/// Mount a disk by its UUID and the mount point.
> +pub fn mount_by_uuid(uuid: &str, mount_point: &Path) -> Result<(), Error> {
path
> + let mut command = std::process::Command::new("mount");
> + command.arg(&format!("UUID={uuid}"));
> + command.arg(mount_point);
> +
> + proxmox_sys::command::run_command(command, None)?;
> + Ok(())
> +}
> +
> +/// Create bind mount.
> +pub fn bind_mount(path: &Path, target: &Path) -> Result<(), Error> {
path(s)
> + let mut command = std::process::Command::new("mount");
> + command.arg("--bind");
> + command.arg(path);
> + command.arg(target);
> +
> + proxmox_sys::command::run_command(command, None)?;
> + Ok(())
> +}
> +
> +/// Unmount a disk by its mount point.
> +pub fn unmount_by_mountpoint(path: &str) -> Result<(), Error> {
str? seems inconsistent, or is there a reason for this?
> + let mut command = std::process::Command::new("umount");
> + command.arg(path);
> +
> + proxmox_sys::command::run_command(command, None)?;
> + Ok(())
> +}
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
` (24 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
From: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 5 +++++
pbs-datastore/src/datastore.rs | 11 +++++++----
src/api2/node/disks/directory.rs | 4 ++--
src/bin/proxmox-backup-proxy.rs | 4 ++--
4 files changed, 16 insertions(+), 8 deletions(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 31767417..a5704c93 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -357,6 +357,11 @@ impl DataStoreConfig {
}
}
+ /// Returns the absolute path to the datastore content.
+ pub fn absolute_path(&self) -> String {
+ self.path.clone()
+ }
+
pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
self.maintenance_mode.as_ref().and_then(|str| {
MaintenanceMode::deserialize(proxmox_schema::de::SchemaDeserializer::new(
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index d0f3c53a..fb37bd5a 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -178,7 +178,7 @@ impl DataStore {
)?;
Arc::new(ChunkStore::open(
name,
- &config.path,
+ config.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
@@ -262,8 +262,11 @@ impl DataStore {
DatastoreTuning::API_SCHEMA
.parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
)?;
- let chunk_store =
- ChunkStore::open(&name, &config.path, tuning.sync_level.unwrap_or_default())?;
+ let chunk_store = ChunkStore::open(
+ &name,
+ config.absolute_path(),
+ tuning.sync_level.unwrap_or_default(),
+ )?;
let inner = Arc::new(Self::with_store_and_config(
Arc::new(chunk_store),
config,
@@ -1387,7 +1390,7 @@ impl DataStore {
bail!("datastore is currently in use");
}
- let base = PathBuf::from(&datastore_config.path);
+ let base = PathBuf::from(datastore_config.absolute_path());
let mut ok = true;
if destroy_data {
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index e971f40d..c0c53d84 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -247,12 +247,12 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
let conflicting_datastore: Option<DataStoreConfig> =
- datastores.into_iter().find(|ds| ds.path == path);
+ datastores.into_iter().find(|ds| ds.absolute_path() == path);
if let Some(conflicting_datastore) = conflicting_datastore {
bail!(
"Can't remove '{}' since it's required by datastore '{}'",
- conflicting_datastore.path,
+ conflicting_datastore.absolute_path(),
conflicting_datastore.name
);
}
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 041f3aff..6ccf2f96 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -1132,8 +1132,8 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
{
continue;
}
- let path = std::path::Path::new(&config.path);
- datastores.push(gather_disk_stats(disk_manager.clone(), path, &config.name));
+ let path = std::path::PathBuf::from(config.absolute_path());
+ datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
}
}
Err(err) => {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> From: Dietmar Maurer <dietmar@proxmox.com>
a reason for such a change is always nice to have, e.g.:
removable datastores will have a PBS-managed mountpoint as path, direct
access to the field needs to be replaced with a helper that can account
for this.
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/datastore.rs | 5 +++++
> pbs-datastore/src/datastore.rs | 11 +++++++----
> src/api2/node/disks/directory.rs | 4 ++--
> src/bin/proxmox-backup-proxy.rs | 4 ++--
> 4 files changed, 16 insertions(+), 8 deletions(-)
>
> diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> index 31767417..a5704c93 100644
> --- a/pbs-api-types/src/datastore.rs
> +++ b/pbs-api-types/src/datastore.rs
> @@ -357,6 +357,11 @@ impl DataStoreConfig {
> }
> }
>
> + /// Returns the absolute path to the datastore content.
> + pub fn absolute_path(&self) -> String {
> + self.path.clone()
> + }
> +
> pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
> self.maintenance_mode.as_ref().and_then(|str| {
> MaintenanceMode::deserialize(proxmox_schema::de::SchemaDeserializer::new(
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index d0f3c53a..fb37bd5a 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -178,7 +178,7 @@ impl DataStore {
> )?;
> Arc::new(ChunkStore::open(
> name,
> - &config.path,
> + config.absolute_path(),
> tuning.sync_level.unwrap_or_default(),
> )?)
> };
> @@ -262,8 +262,11 @@ impl DataStore {
> DatastoreTuning::API_SCHEMA
> .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
> )?;
> - let chunk_store =
> - ChunkStore::open(&name, &config.path, tuning.sync_level.unwrap_or_default())?;
> + let chunk_store = ChunkStore::open(
> + &name,
> + config.absolute_path(),
> + tuning.sync_level.unwrap_or_default(),
> + )?;
> let inner = Arc::new(Self::with_store_and_config(
> Arc::new(chunk_store),
> config,
> @@ -1387,7 +1390,7 @@ impl DataStore {
> bail!("datastore is currently in use");
> }
>
> - let base = PathBuf::from(&datastore_config.path);
> + let base = PathBuf::from(datastore_config.absolute_path());
>
> let mut ok = true;
> if destroy_data {
> diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
> index e971f40d..c0c53d84 100644
> --- a/src/api2/node/disks/directory.rs
> +++ b/src/api2/node/disks/directory.rs
> @@ -247,12 +247,12 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
> let (config, _) = pbs_config::datastore::config()?;
> let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
> let conflicting_datastore: Option<DataStoreConfig> =
> - datastores.into_iter().find(|ds| ds.path == path);
> + datastores.into_iter().find(|ds| ds.absolute_path() == path);
>
> if let Some(conflicting_datastore) = conflicting_datastore {
> bail!(
> "Can't remove '{}' since it's required by datastore '{}'",
> - conflicting_datastore.path,
> + conflicting_datastore.absolute_path(),
> conflicting_datastore.name
> );
> }
> diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
> index 041f3aff..6ccf2f96 100644
> --- a/src/bin/proxmox-backup-proxy.rs
> +++ b/src/bin/proxmox-backup-proxy.rs
> @@ -1132,8 +1132,8 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
> {
> continue;
> }
> - let path = std::path::Path::new(&config.path);
> - datastores.push(gather_disk_stats(disk_manager.clone(), path, &config.name));
> + let path = std::path::PathBuf::from(config.absolute_path());
> + datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
> }
> }
> Err(err) => {
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 01/26] tools: add disks utility functions Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 02/26] config: factor out method to get the absolute datastore path Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
` (23 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index a5704c93..863ed317 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -160,6 +160,9 @@ pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema =
.minimum(1)
.schema();
+/// Base directory where datastores are mounted
+pub const DATASTORE_MOUNT_DIR: &str = "/mnt/datastore";
+
#[api]
#[derive(Debug, Default, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
@@ -273,6 +276,12 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
format: &ApiStringFormat::PropertyString(&MaintenanceMode::API_SCHEMA),
type: String,
},
+ "backing-device": {
+ description: "The UUID of the filesystem partition for removable datastores.",
+ optional: true,
+ format: &proxmox_schema::api_types::UUID_FORMAT,
+ type: String,
+ }
}
)]
#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -320,6 +329,11 @@ pub struct DataStoreConfig {
/// Maintenance mode, type is either 'offline' or 'read-only', message should be enclosed in "
#[serde(skip_serializing_if = "Option::is_none")]
pub maintenance_mode: Option<String>,
+
+ /// The UUID of the device(for removable datastores)
+ #[updater(skip)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub backing_device: Option<String>,
}
#[api]
@@ -354,12 +368,23 @@ impl DataStoreConfig {
notification_mode: None,
tuning: None,
maintenance_mode: None,
+ backing_device: None,
}
}
/// Returns the absolute path to the datastore content.
pub fn absolute_path(&self) -> String {
- self.path.clone()
+ if self.backing_device.is_some() {
+ format!("{DATASTORE_MOUNT_DIR}/{}", self.name)
+ } else {
+ self.path.clone()
+ }
+ }
+
+ pub fn get_mount_point(&self) -> Option<String> {
+ self.backing_device
+ .is_some()
+ .then(|| format!("{DATASTORE_MOUNT_DIR}/{}", self.name))
}
pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
following the rest of the series.. `path` is now either
- the absolute path to the datastore directory (regular datastores)
- the relative path to the datastore directory on a removable device (removable datastores)
the schema only tells us it's a "Directory name". I see room for improvement there ;)
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/datastore.rs | 27 ++++++++++++++++++++++++++-
> 1 file changed, 26 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> index a5704c93..863ed317 100644
> --- a/pbs-api-types/src/datastore.rs
> +++ b/pbs-api-types/src/datastore.rs
> @@ -160,6 +160,9 @@ pub const PRUNE_SCHEMA_KEEP_YEARLY: Schema =
> .minimum(1)
> .schema();
>
> +/// Base directory where datastores are mounted
> +pub const DATASTORE_MOUNT_DIR: &str = "/mnt/datastore";
> +
> #[api]
> #[derive(Debug, Default, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)]
> #[serde(rename_all = "lowercase")]
> @@ -273,6 +276,12 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
> format: &ApiStringFormat::PropertyString(&MaintenanceMode::API_SCHEMA),
> type: String,
> },
> + "backing-device": {
> + description: "The UUID of the filesystem partition for removable datastores.",
> + optional: true,
> + format: &proxmox_schema::api_types::UUID_FORMAT,
> + type: String,
> + }
> }
> )]
> #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
> @@ -320,6 +329,11 @@ pub struct DataStoreConfig {
> /// Maintenance mode, type is either 'offline' or 'read-only', message should be enclosed in "
> #[serde(skip_serializing_if = "Option::is_none")]
> pub maintenance_mode: Option<String>,
> +
> + /// The UUID of the device(for removable datastores)
> + #[updater(skip)]
> + #[serde(skip_serializing_if = "Option::is_none")]
> + pub backing_device: Option<String>,
> }
>
> #[api]
> @@ -354,12 +368,23 @@ impl DataStoreConfig {
> notification_mode: None,
> tuning: None,
> maintenance_mode: None,
> + backing_device: None,
> }
> }
>
> /// Returns the absolute path to the datastore content.
> pub fn absolute_path(&self) -> String {
> - self.path.clone()
> + if self.backing_device.is_some() {
> + format!("{DATASTORE_MOUNT_DIR}/{}", self.name)
see comment on patch #7
> + } else {
> + self.path.clone()
> + }
> + }
> +
> + pub fn get_mount_point(&self) -> Option<String> {
> + self.backing_device
> + .is_some()
> + .then(|| format!("{DATASTORE_MOUNT_DIR}/{}", self.name))
> }
>
> pub fn get_maintenance_mode(&self) -> Option<MaintenanceMode> {
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (2 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 03/26] pbs-api-types: add backing-device to DataStoreConfig Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 05/26] disks: add UUID to partition info Hannes Laimer
` (22 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
From: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
---
pbs-api-types/src/datastore.rs | 3 +++
pbs-api-types/src/maintenance.rs | 5 ++++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 863ed317..d6e2f97e 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -404,6 +404,9 @@ impl DataStoreConfig {
match current_type {
Some(MaintenanceType::ReadOnly) => { /* always OK */ }
Some(MaintenanceType::Offline) => { /* always OK */ }
+ Some(MaintenanceType::Unmount) => {
+ bail!("datastore is being unmounted");
+ }
Some(MaintenanceType::Delete) => {
match new_type {
Some(MaintenanceType::Delete) => { /* allow to delete a deleted storage */ }
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 1e3413dc..fd4d3416 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -38,7 +38,6 @@ pub enum Operation {
/// Maintenance type.
pub enum MaintenanceType {
// TODO:
- // - Add "unmounting" once we got pluggable datastores
// - Add "GarbageCollection" or "DeleteOnly" as type and track GC (or all deletes) as separate
// operation, so that one can enable a mode where nothing new can be added but stuff can be
// cleaned
@@ -48,6 +47,8 @@ pub enum MaintenanceType {
Offline,
/// The datastore is being deleted.
Delete,
+ /// The (removable) datastore is being unmounted.
+ Unmount,
}
serde_plain::derive_display_from_serialize!(MaintenanceType);
serde_plain::derive_fromstr_from_deserialize!(MaintenanceType);
@@ -94,6 +95,8 @@ impl MaintenanceMode {
if let Some(Operation::Lookup) = operation {
return Ok(());
+ } else if self.ty == MaintenanceType::Unmount {
+ bail!("datastore is being unmounted");
} else if self.ty == MaintenanceType::Offline {
bail!("offline maintenance mode: {}", message);
} else if self.ty == MaintenanceType::ReadOnly {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> From: Dietmar Maurer <dietmar@proxmox.com>
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
so is this a patch from Dietmar or (partially) from you?
> ---
> pbs-api-types/src/datastore.rs | 3 +++
> pbs-api-types/src/maintenance.rs | 5 ++++-
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> index 863ed317..d6e2f97e 100644
> --- a/pbs-api-types/src/datastore.rs
> +++ b/pbs-api-types/src/datastore.rs
> @@ -404,6 +404,9 @@ impl DataStoreConfig {
> match current_type {
> Some(MaintenanceType::ReadOnly) => { /* always OK */ }
> Some(MaintenanceType::Offline) => { /* always OK */ }
> + Some(MaintenanceType::Unmount) => {
> + bail!("datastore is being unmounted");
> + }
> Some(MaintenanceType::Delete) => {
> match new_type {
> Some(MaintenanceType::Delete) => { /* allow to delete a deleted storage */ }
> diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> index 1e3413dc..fd4d3416 100644
> --- a/pbs-api-types/src/maintenance.rs
> +++ b/pbs-api-types/src/maintenance.rs
> @@ -38,7 +38,6 @@ pub enum Operation {
> /// Maintenance type.
> pub enum MaintenanceType {
> // TODO:
> - // - Add "unmounting" once we got pluggable datastores
> // - Add "GarbageCollection" or "DeleteOnly" as type and track GC (or all deletes) as separate
> // operation, so that one can enable a mode where nothing new can be added but stuff can be
> // cleaned
> @@ -48,6 +47,8 @@ pub enum MaintenanceType {
> Offline,
> /// The datastore is being deleted.
> Delete,
> + /// The (removable) datastore is being unmounted.
> + Unmount,
> }
> serde_plain::derive_display_from_serialize!(MaintenanceType);
> serde_plain::derive_fromstr_from_deserialize!(MaintenanceType);
> @@ -94,6 +95,8 @@ impl MaintenanceMode {
>
> if let Some(Operation::Lookup) = operation {
> return Ok(());
> + } else if self.ty == MaintenanceType::Unmount {
> + bail!("datastore is being unmounted");
> } else if self.ty == MaintenanceType::Offline {
> bail!("offline maintenance mode: {}", message);
> } else if self.ty == MaintenanceType::ReadOnly {
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 05/26] disks: add UUID to partition info
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (3 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 04/26] maintenance: add 'Unmount' maintenance type Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
` (21 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/tools/disks/mod.rs | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 9d8ff05c..93c1a882 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -55,6 +55,8 @@ pub struct LsblkInfo {
/// File system label.
#[serde(rename = "fstype")]
file_system_type: Option<String>,
+ /// File system UUID.
+ uuid: Option<String>,
}
impl DiskManage {
@@ -613,7 +615,7 @@ pub struct BlockDevStat {
/// Use lsblk to read partition type uuids and file system types.
pub fn get_lsblk_info() -> Result<Vec<LsblkInfo>, Error> {
let mut command = std::process::Command::new("lsblk");
- command.args(["--json", "-o", "path,parttype,fstype"]);
+ command.args(["--json", "-o", "path,parttype,fstype,uuid"]);
let output = proxmox_sys::command::run_command(command, None)?;
@@ -697,6 +699,8 @@ pub struct PartitionInfo {
pub size: Option<u64>,
/// GPT partition
pub gpt: bool,
+ /// UUID
+ pub uuid: Option<String>,
}
#[api(
@@ -887,8 +891,10 @@ fn get_partitions_info(
let mounted = disk.is_mounted().unwrap_or(false);
let mut filesystem = None;
+ let mut uuid = None;
if let (Some(devpath), Some(infos)) = (devpath.as_ref(), lsblk_infos.as_ref()) {
for info in infos.iter().filter(|i| i.path.eq(devpath)) {
+ uuid = info.uuid.clone();
used = match info.partition_type.as_deref() {
Some("21686148-6449-6e6f-744e-656564454649") => PartitionUsageType::BIOS,
Some("c12a7328-f81f-11d2-ba4b-00a0c93ec93b") => PartitionUsageType::EFI,
@@ -911,6 +917,7 @@ fn get_partitions_info(
filesystem,
size: disk.size().ok(),
gpt: disk.has_gpt(),
+ uuid,
}
})
.collect()
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (4 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 05/26] disks: add UUID to partition info Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
` (20 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/maintenance.rs | 2 ++
pbs-datastore/src/datastore.rs | 58 ++++++++++++++++++++++++++++++++
pbs-datastore/src/lib.rs | 2 +-
src/bin/proxmox-backup-proxy.rs | 5 ++-
4 files changed, 65 insertions(+), 2 deletions(-)
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index fd4d3416..9f51292e 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -82,6 +82,8 @@ impl MaintenanceMode {
/// task finishes, so all open files are closed.
pub fn is_offline(&self) -> bool {
self.ty == MaintenanceType::Offline
+ || self.ty == MaintenanceType::Unmount
+ || self.ty == MaintenanceType::Delete
}
pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index fb37bd5a..29f98b37 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1,5 +1,6 @@
use std::collections::{HashMap, HashSet};
use std::io::{self, Write};
+use std::os::unix::ffi::OsStrExt;
use std::os::unix::io::AsRawFd;
use std::path::{Path, PathBuf};
use std::sync::{Arc, LazyLock, Mutex};
@@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
+use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::ProcessLockSharedGuard;
use proxmox_worker_task::WorkerTaskContext;
@@ -46,6 +48,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
Ok(())
}
+/// check if a removable datastore is currently available/mounted by
+/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
+/// /proc/self/mountinfo
+pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
+ use nix::sys::stat::SFlag;
+
+ let uuid = match config.backing_device.as_deref() {
+ Some(dev) => dev,
+ None => return true,
+ };
+
+ let Some(store_mount_point) = config.get_mount_point() else {
+ return true;
+ };
+ let store_mount_point = Path::new(&store_mount_point);
+
+ let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
+ Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
+ _ => return false,
+ };
+
+ let Ok(mount_info) = MountInfo::read() else {
+ return false;
+ };
+
+ for (_, entry) in mount_info {
+ let Some(source) = entry.mount_source else {
+ continue;
+ };
+
+ if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
+ continue;
+ }
+
+ if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
+ let sflag = SFlag::from_bits_truncate(stat.st_mode);
+
+ if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
+ return true;
+ }
+ }
+ }
+
+ false
+}
+
/// Datastore Management
///
/// A Datastore can store severals backups, and provides the
@@ -155,6 +203,12 @@ impl DataStore {
}
}
+ if config.backing_device.is_some() && !is_datastore_available(&config) {
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+ datastore_cache.remove(&config.name);
+ bail!("Removable Datastore is not mounted");
+ }
+
let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
let entry = datastore_cache.get(name);
@@ -258,6 +312,10 @@ impl DataStore {
) -> Result<Arc<Self>, Error> {
let name = config.name.clone();
+ if !is_datastore_available(&config) {
+ bail!("Datastore is not available")
+ }
+
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
.parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 202b0955..6d58f327 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -204,7 +204,7 @@ pub use manifest::BackupManifest;
pub use store_progress::StoreProgress;
mod datastore;
-pub use datastore::{check_backup_owner, DataStore};
+pub use datastore::{check_backup_owner, is_datastore_available, DataStore};
mod hierarchy;
pub use hierarchy::{
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 6ccf2f96..58fb8b84 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -23,7 +23,7 @@ use proxmox_sys::fs::{CreateOptions, FileSystemInformation};
use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsStat};
use proxmox_sys::logrotate::LogRotate;
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
use proxmox_rest_server::{
cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
@@ -1132,6 +1132,9 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
{
continue;
}
+ if !is_datastore_available(&config) {
+ continue;
+ }
let path = std::path::PathBuf::from(config.absolute_path());
datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 14:04 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/maintenance.rs | 2 ++
> pbs-datastore/src/datastore.rs | 58 ++++++++++++++++++++++++++++++++
> pbs-datastore/src/lib.rs | 2 +-
> src/bin/proxmox-backup-proxy.rs | 5 ++-
> 4 files changed, 65 insertions(+), 2 deletions(-)
>
> diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> index fd4d3416..9f51292e 100644
> --- a/pbs-api-types/src/maintenance.rs
> +++ b/pbs-api-types/src/maintenance.rs
> @@ -82,6 +82,8 @@ impl MaintenanceMode {
> /// task finishes, so all open files are closed.
> pub fn is_offline(&self) -> bool {
> self.ty == MaintenanceType::Offline
> + || self.ty == MaintenanceType::Unmount
> + || self.ty == MaintenanceType::Delete
> }
>
> pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index fb37bd5a..29f98b37 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -1,5 +1,6 @@
> use std::collections::{HashMap, HashSet};
> use std::io::{self, Write};
> +use std::os::unix::ffi::OsStrExt;
> use std::os::unix::io::AsRawFd;
> use std::path::{Path, PathBuf};
> use std::sync::{Arc, LazyLock, Mutex};
> @@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
> use proxmox_sys::error::SysError;
> use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
> use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
> +use proxmox_sys::linux::procfs::MountInfo;
> use proxmox_sys::process_locker::ProcessLockSharedGuard;
> use proxmox_worker_task::WorkerTaskContext;
>
> @@ -46,6 +48,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
> Ok(())
> }
>
> +/// check if a removable datastore is currently available/mounted by
> +/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
> +/// /proc/self/mountinfo
check if a *removable* datastore is ..
> +pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
> + use nix::sys::stat::SFlag;
> +
> + let uuid = match config.backing_device.as_deref() {
> + Some(dev) => dev,
> + None => return true,
returns true if not a removable datastore?
> + };
> +
> + let Some(store_mount_point) = config.get_mount_point() else {
> + return true;
same here. also see further below - this should either be removable
datastore specific, in which case it could take uuid and mountpoint as
parameters, or it could take any DataStoreConfig, then the doc comment
should reflect that and clearly describe the semantics..
> + };
> + let store_mount_point = Path::new(&store_mount_point);
> +
> + let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
> + Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
> + _ => return false,
shouldn't this differentiate between:
- stat failed with ENOENT
- stat failed for other reasons
- stat worked but result is not as expected
?
also, that code (continued below at (A))
> + };
> +
> + let Ok(mount_info) = MountInfo::read() else {
> + return false;
shouldn't this be an error?
> + };
> +
> + for (_, entry) in mount_info {
> + let Some(source) = entry.mount_source else {
> + continue;
> + };
> +
> + if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
> + continue;
> + }
> +
> + if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
> + let sflag = SFlag::from_bits_truncate(stat.st_mode);
> +
> + if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
> + return true;
> + }
(A) and this code could go into a helper..
> + }
> + }
> +
> + false
> +}
> +
> /// Datastore Management
> ///
> /// A Datastore can store severals backups, and provides the
> @@ -155,6 +203,12 @@ impl DataStore {
> }
> }
>
> + if config.backing_device.is_some() && !is_datastore_available(&config) {
> + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> + datastore_cache.remove(&config.name);
> + bail!("Removable Datastore is not mounted");
> + }
so here the helper is only called for removable datastores..
> +
> let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> let entry = datastore_cache.get(name);
>
> @@ -258,6 +312,10 @@ impl DataStore {
> ) -> Result<Arc<Self>, Error> {
> let name = config.name.clone();
>
> + if !is_datastore_available(&config) {
> + bail!("Datastore is not available")
> + }
but here it is called for all datastores
> +
> let tuning: DatastoreTuning = serde_json::from_value(
> DatastoreTuning::API_SCHEMA
> .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
> diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
> index 202b0955..6d58f327 100644
> --- a/pbs-datastore/src/lib.rs
> +++ b/pbs-datastore/src/lib.rs
> @@ -204,7 +204,7 @@ pub use manifest::BackupManifest;
> pub use store_progress::StoreProgress;
>
> mod datastore;
> -pub use datastore::{check_backup_owner, DataStore};
> +pub use datastore::{check_backup_owner, is_datastore_available, DataStore};
>
> mod hierarchy;
> pub use hierarchy::{
> diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
> index 6ccf2f96..58fb8b84 100644
> --- a/src/bin/proxmox-backup-proxy.rs
> +++ b/src/bin/proxmox-backup-proxy.rs
> @@ -23,7 +23,7 @@ use proxmox_sys::fs::{CreateOptions, FileSystemInformation};
> use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsStat};
> use proxmox_sys::logrotate::LogRotate;
>
> -use pbs_datastore::DataStore;
> +use pbs_datastore::{is_datastore_available, DataStore};
>
> use proxmox_rest_server::{
> cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
> @@ -1132,6 +1132,9 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
> {
> continue;
> }
> + if !is_datastore_available(&config) {
> + continue;
> + }
and here as well..
> let path = std::path::PathBuf::from(config.absolute_path());
> datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
> }
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 14:04 ` Hannes Laimer
2024-10-30 9:45 ` Fabian Grünbichler
0 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 14:04 UTC (permalink / raw)
To: Proxmox Backup Server development discussion; +Cc: pbs-devel
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > pbs-api-types/src/maintenance.rs | 2 ++
> > pbs-datastore/src/datastore.rs | 58 ++++++++++++++++++++++++++++++++
> > pbs-datastore/src/lib.rs | 2 +-
> > src/bin/proxmox-backup-proxy.rs | 5 ++-
> > 4 files changed, 65 insertions(+), 2 deletions(-)
> >
> > diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> > index fd4d3416..9f51292e 100644
> > --- a/pbs-api-types/src/maintenance.rs
> > +++ b/pbs-api-types/src/maintenance.rs
> > @@ -82,6 +82,8 @@ impl MaintenanceMode {
> > /// task finishes, so all open files are closed.
> > pub fn is_offline(&self) -> bool {
> > self.ty == MaintenanceType::Offline
> > + || self.ty == MaintenanceType::Unmount
> > + || self.ty == MaintenanceType::Delete
> > }
> >
> > pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
> > diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> > index fb37bd5a..29f98b37 100644
> > --- a/pbs-datastore/src/datastore.rs
> > +++ b/pbs-datastore/src/datastore.rs
> > @@ -1,5 +1,6 @@
> > use std::collections::{HashMap, HashSet};
> > use std::io::{self, Write};
> > +use std::os::unix::ffi::OsStrExt;
> > use std::os::unix::io::AsRawFd;
> > use std::path::{Path, PathBuf};
> > use std::sync::{Arc, LazyLock, Mutex};
> > @@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
> > use proxmox_sys::error::SysError;
> > use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
> > use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
> > +use proxmox_sys::linux::procfs::MountInfo;
> > use proxmox_sys::process_locker::ProcessLockSharedGuard;
> > use proxmox_worker_task::WorkerTaskContext;
> >
> > @@ -46,6 +48,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
> > Ok(())
> > }
> >
> > +/// check if a removable datastore is currently available/mounted by
> > +/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
> > +/// /proc/self/mountinfo
>
> check if a *removable* datastore is ..
>
The idea was to be able to work with normal and removable datastores the
same way throughout the codebase, not having this would mean having to
disginguish between them when using datastores.
It might make sense to add a check for the existance on `.chunks/`, so
this wouldn't do nothing for normal datastores.
> > +pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
> > + use nix::sys::stat::SFlag;
> > +
> > + let uuid = match config.backing_device.as_deref() {
> > + Some(dev) => dev,
> > + None => return true,
>
> returns true if not a removable datastore?
>
> > + };
> > +
> > + let Some(store_mount_point) = config.get_mount_point() else {
> > + return true;
>
> same here. also see further below - this should either be removable
> datastore specific, in which case it could take uuid and mountpoint as
> parameters, or it could take any DataStoreConfig, then the doc comment
> should reflect that and clearly describe the semantics..
>
yes, I'll update the docs
> > + };
> > + let store_mount_point = Path::new(&store_mount_point);
> > +
> > + let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
> > + Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
> > + _ => return false,
>
> shouldn't this differentiate between:
> - stat failed with ENOENT
> - stat failed for other reasons
> - stat worked but result is not as expected
>
> ?
>
I guess it comes down to wether we want to differantiate between, "it is
not there" or "we can't check if it is there". Since this used mostly
done in the background and quite frequently, I opted for treating the
two cases the same.
> also, that code (continued below at (A))
>
> > + };
> > +
> > + let Ok(mount_info) = MountInfo::read() else {
> > + return false;
>
> shouldn't this be an error?
>
... same here
> > + };
> > +
> > + for (_, entry) in mount_info {
> > + let Some(source) = entry.mount_source else {
> > + continue;
> > + };
> > +
> > + if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
> > + continue;
> > + }
> > +
> > + if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
> > + let sflag = SFlag::from_bits_truncate(stat.st_mode);
> > +
> > + if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
> > + return true;
> > + }
>
> (A) and this code could go into a helper..
>
yup, I'll do that
> > + }
> > + }
> > +
> > + false
> > +}
> > +
> > /// Datastore Management
> > ///
> > /// A Datastore can store severals backups, and provides the
> > @@ -155,6 +203,12 @@ impl DataStore {
> > }
> > }
> >
> > + if config.backing_device.is_some() && !is_datastore_available(&config) {
> > + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> > + datastore_cache.remove(&config.name);
> > + bail!("Removable Datastore is not mounted");
> > + }
>
> so here the helper is only called for removable datastores..
>
the thing we do here is only relevant for removable datastores, so if
we would also need to drop cache entries for non-removable datastores,
we could drop the is-removable check, yes.
> > +
> > let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> > let entry = datastore_cache.get(name);
> >
> > @@ -258,6 +312,10 @@ impl DataStore {
> > ) -> Result<Arc<Self>, Error> {
> > let name = config.name.clone();
> >
> > + if !is_datastore_available(&config) {
> > + bail!("Datastore is not available")
> > + }
>
> but here it is called for all datastores
>
... here what this check guards is relevant for both normal and
removable. In my head this made sense, I hope it also does outside :)
> > +
> > let tuning: DatastoreTuning = serde_json::from_value(
> > DatastoreTuning::API_SCHEMA
> > .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
> > diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
> > index 202b0955..6d58f327 100644
> > --- a/pbs-datastore/src/lib.rs
> > +++ b/pbs-datastore/src/lib.rs
> > @@ -204,7 +204,7 @@ pub use manifest::BackupManifest;
> > pub use store_progress::StoreProgress;
> >
> > mod datastore;
> > -pub use datastore::{check_backup_owner, DataStore};
> > +pub use datastore::{check_backup_owner, is_datastore_available, DataStore};
> >
> > mod hierarchy;
> > pub use hierarchy::{
> > diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
> > index 6ccf2f96..58fb8b84 100644
> > --- a/src/bin/proxmox-backup-proxy.rs
> > +++ b/src/bin/proxmox-backup-proxy.rs
> > @@ -23,7 +23,7 @@ use proxmox_sys::fs::{CreateOptions, FileSystemInformation};
> > use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsStat};
> > use proxmox_sys::logrotate::LogRotate;
> >
> > -use pbs_datastore::DataStore;
> > +use pbs_datastore::{is_datastore_available, DataStore};
> >
> > use proxmox_rest_server::{
> > cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
> > @@ -1132,6 +1132,9 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
> > {
> > continue;
> > }
> > + if !is_datastore_available(&config) {
> > + continue;
> > + }
>
> and here as well..
>
> > let path = std::path::PathBuf::from(config.absolute_path());
> > datastores.push(gather_disk_stats(disk_manager.clone(), &path, &config.name));
> > }
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available
2024-10-29 14:04 ` Hannes Laimer
@ 2024-10-30 9:45 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-30 9:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
Quoting Hannes Laimer (2024-10-29 15:04:25)
> On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> > On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > > Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> > > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > > ---
> > > pbs-api-types/src/maintenance.rs | 2 ++
> > > pbs-datastore/src/datastore.rs | 58 ++++++++++++++++++++++++++++++++
> > > pbs-datastore/src/lib.rs | 2 +-
> > > src/bin/proxmox-backup-proxy.rs | 5 ++-
> > > 4 files changed, 65 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> > > index fd4d3416..9f51292e 100644
> > > --- a/pbs-api-types/src/maintenance.rs
> > > +++ b/pbs-api-types/src/maintenance.rs
> > > @@ -82,6 +82,8 @@ impl MaintenanceMode {
> > > /// task finishes, so all open files are closed.
> > > pub fn is_offline(&self) -> bool {
> > > self.ty == MaintenanceType::Offline
> > > + || self.ty == MaintenanceType::Unmount
> > > + || self.ty == MaintenanceType::Delete
> > > }
> > >
> > > pub fn check(&self, operation: Option<Operation>) -> Result<(), Error> {
> > > diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> > > index fb37bd5a..29f98b37 100644
> > > --- a/pbs-datastore/src/datastore.rs
> > > +++ b/pbs-datastore/src/datastore.rs
> > > @@ -1,5 +1,6 @@
> > > use std::collections::{HashMap, HashSet};
> > > use std::io::{self, Write};
> > > +use std::os::unix::ffi::OsStrExt;
> > > use std::os::unix::io::AsRawFd;
> > > use std::path::{Path, PathBuf};
> > > use std::sync::{Arc, LazyLock, Mutex};
> > > @@ -14,6 +15,7 @@ use proxmox_schema::ApiType;
> > > use proxmox_sys::error::SysError;
> > > use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
> > > use proxmox_sys::fs::{lock_dir_noblock, DirLockGuard};
> > > +use proxmox_sys::linux::procfs::MountInfo;
> > > use proxmox_sys::process_locker::ProcessLockSharedGuard;
> > > use proxmox_worker_task::WorkerTaskContext;
> > >
> > > @@ -46,6 +48,52 @@ pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error>
> > > Ok(())
> > > }
> > >
> > > +/// check if a removable datastore is currently available/mounted by
> > > +/// comparing the `st_rdev` values of `/dev/disk/by-uuid/<uuid>` and the source device in
> > > +/// /proc/self/mountinfo
> >
> > check if a *removable* datastore is ..
> >
>
> The idea was to be able to work with normal and removable datastores the
> same way throughout the codebase, not having this would mean having to
> disginguish between them when using datastores.
which you already do in parts, but
> It might make sense to add a check for the existance on `.chunks/`, so
> this wouldn't do nothing for normal datastores.
I don't mind this helper working for both, I just care about its intended use
case and semantics being clear :)
> > > +pub fn is_datastore_available(config: &DataStoreConfig) -> bool {
> > > + use nix::sys::stat::SFlag;
> > > +
> > > + let uuid = match config.backing_device.as_deref() {
> > > + Some(dev) => dev,
> > > + None => return true,
> >
> > returns true if not a removable datastore?
> >
> > > + };
> > > +
> > > + let Some(store_mount_point) = config.get_mount_point() else {
> > > + return true;
> >
> > same here. also see further below - this should either be removable
> > datastore specific, in which case it could take uuid and mountpoint as
> > parameters, or it could take any DataStoreConfig, then the doc comment
> > should reflect that and clearly describe the semantics..
> >
>
> yes, I'll update the docs
>
> > > + };
> > > + let store_mount_point = Path::new(&store_mount_point);
> > > +
> > > + let dev_node = match nix::sys::stat::stat(format!("/dev/disk/by-uuid/{uuid}").as_str()) {
> > > + Ok(stat) if SFlag::from_bits_truncate(stat.st_mode) == SFlag::S_IFBLK => stat.st_rdev,
> > > + _ => return false,
> >
> > shouldn't this differentiate between:
> > - stat failed with ENOENT
> > - stat failed for other reasons
> > - stat worked but result is not as expected
> >
> > ?
> >
>
> I guess it comes down to wether we want to differantiate between, "it is
> not there" or "we can't check if it is there". Since this used mostly
> done in the background and quite frequently, I opted for treating the
> two cases the same.
I don't really see how the frequency of calls makes a difference for whether
this should differentiate between an error and an expected state.. it can be
okay to map errors to "not available" if we don't ever want to differentiate,
but then this should be clearly communicated ;)
>
> > also, that code (continued below at (A))
> >
> > > + };
> > > +
> > > + let Ok(mount_info) = MountInfo::read() else {
> > > + return false;
> >
> > shouldn't this be an error?
> >
>
> ... same here
>
> > > + };
> > > +
> > > + for (_, entry) in mount_info {
> > > + let Some(source) = entry.mount_source else {
> > > + continue;
> > > + };
> > > +
> > > + if entry.mount_point != store_mount_point || !source.as_bytes().starts_with(b"/") {
> > > + continue;
> > > + }
> > > +
> > > + if let Ok(stat) = nix::sys::stat::stat(source.as_os_str()) {
> > > + let sflag = SFlag::from_bits_truncate(stat.st_mode);
> > > +
> > > + if sflag == SFlag::S_IFBLK && stat.st_rdev == dev_node {
> > > + return true;
> > > + }
> >
> > (A) and this code could go into a helper..
> >
>
> yup, I'll do that
>
> > > + }
> > > + }
> > > +
> > > + false
> > > +}
> > > +
> > > /// Datastore Management
> > > ///
> > > /// A Datastore can store severals backups, and provides the
> > > @@ -155,6 +203,12 @@ impl DataStore {
> > > }
> > > }
> > >
> > > + if config.backing_device.is_some() && !is_datastore_available(&config) {
> > > + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> > > + datastore_cache.remove(&config.name);
> > > + bail!("Removable Datastore is not mounted");
> > > + }
> >
> > so here the helper is only called for removable datastores..
> >
>
> the thing we do here is only relevant for removable datastores, so if
> we would also need to drop cache entries for non-removable datastores,
> we could drop the is-removable check, yes.
the first thing is_datastore_available does is check whether a backing device
is set, and return true otherwise.. so the extra condition here doesn't really
make sense, it's just checked twice in case of a removable datastore..
>
> > > +
> > > let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> > > let entry = datastore_cache.get(name);
> > >
> > > @@ -258,6 +312,10 @@ impl DataStore {
> > > ) -> Result<Arc<Self>, Error> {
> > > let name = config.name.clone();
> > >
> > > + if !is_datastore_available(&config) {
> > > + bail!("Datastore is not available")
> > > + }
> >
> > but here it is called for all datastores
> >
>
> ... here what this check guards is relevant for both normal and
> removable. In my head this made sense, I hope it also does outside :)
>
yes, I just wanted to point out that it doesn't really make sense. a call for a
regular datastore will always return true, so guarding it by whether a backing
device is set doesn't make sense with the current implementation. either all
calls should be made unconditional (since it's a nop for regular datastores
anyway), or all calls should be made conditional and it should only be called
for removable datastores. then all the redundant checks within can be dropped.
a mix of both is just confusing.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (5 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 06/26] datastore: add helper for checking if a removable datastore is available Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation Hannes Laimer
` (19 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/maintenance.rs | 4 +
src/api2/admin/datastore.rs | 243 +++++++++++++++++++++++++++++--
2 files changed, 237 insertions(+), 10 deletions(-)
diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
index 9f51292e..60181258 100644
--- a/pbs-api-types/src/maintenance.rs
+++ b/pbs-api-types/src/maintenance.rs
@@ -78,6 +78,10 @@ pub struct MaintenanceMode {
}
impl MaintenanceMode {
+ pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
+ Self { ty, message }
+ }
+
/// Used for deciding whether the datastore is cleared from the internal cache after the last
/// task finishes, so all open files are closed.
pub fn is_offline(&self) -> bool {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 976617d9..3c95888d 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -3,7 +3,7 @@
use std::collections::HashSet;
use std::ffi::OsStr;
use std::os::unix::ffi::OsStrExt;
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
use std::sync::Arc;
use anyhow::{bail, format_err, Error};
@@ -13,7 +13,7 @@ use hyper::{header, Body, Response, StatusCode};
use serde::Deserialize;
use serde_json::{json, Value};
use tokio_stream::wrappers::ReceiverStream;
-use tracing::{info, warn};
+use tracing::{debug, info, warn};
use proxmox_async::blocking::WrappedReaderStream;
use proxmox_async::{io::AsyncChannelWriter, stream::AsyncReaderStream};
@@ -29,6 +29,7 @@ use proxmox_sys::fs::{
file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
};
use proxmox_time::CalendarEvent;
+use proxmox_worker_task::WorkerTaskContext;
use pxar::accessor::aio::Accessor;
use pxar::EntryKind;
@@ -36,12 +37,12 @@ use pxar::EntryKind;
use pbs_api_types::{
print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
- GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
- PruneJobOptions, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
- BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
- DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
- PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
- PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
+ GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
+ MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
+ BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
+ BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
+ NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
+ PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
VERIFICATION_OUTDATED_AFTER_SCHEMA,
};
use pbs_client::pxar::{create_tar, create_zip};
@@ -57,8 +58,8 @@ use pbs_datastore::index::IndexFile;
use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
use pbs_datastore::prune::compute_prune_info;
use pbs_datastore::{
- check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
- StoreProgress, CATALOG_NAME,
+ check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
+ LocalChunkReader, StoreProgress, CATALOG_NAME,
};
use pbs_tools::json::required_string_param;
use proxmox_rest_server::{formatter, WorkerTask};
@@ -2384,6 +2385,226 @@ pub async fn set_backup_owner(
.await?
}
+/// Here we
+///
+/// 1. mount the removable device to `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
+/// 2. bind mount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>/<datastore.path>` to `/mnt/datastore/<datastore.name>`
+/// 3. unmount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
+///
+/// leaving us with the datastore being mounted directly with its name under /mnt/datastore/...
+///
+/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
+/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
+/// technically possible.
+pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
+ if let (Some(uuid), Some(mount_point)) = (
+ datastore.backing_device.as_ref(),
+ datastore.get_mount_point(),
+ ) {
+ if pbs_datastore::is_datastore_available(&datastore) {
+ bail!("datastore '{}' is already mounted", datastore.name);
+ }
+ let tmp_mount_path = format!(
+ "{}/{:x}",
+ pbs_buildcfg::rundir!("/mount"),
+ proxmox_uuid::Uuid::generate()
+ );
+
+ let default_options = proxmox_sys::fs::CreateOptions::new();
+ proxmox_sys::fs::create_path(
+ &tmp_mount_path,
+ Some(default_options.clone()),
+ Some(default_options.clone()),
+ )?;
+
+ debug!("mounting '{uuid}' to '{}'", tmp_mount_path);
+ crate::tools::disks::mount_by_uuid(uuid, Path::new(&tmp_mount_path))?;
+
+ let full_store_path = format!(
+ "{tmp_mount_path}/{}",
+ datastore.path.trim_start_matches('/')
+ );
+
+ proxmox_sys::fs::create_path(
+ &mount_point,
+ Some(default_options.clone()),
+ Some(default_options.clone()),
+ )?;
+
+ // can't be created before it is mounted, so we have to do it here
+ proxmox_sys::fs::create_path(
+ &full_store_path,
+ Some(default_options.clone()),
+ Some(default_options.clone()),
+ )?;
+
+ info!(
+ "mounting '{}'({}) to '{}'",
+ datastore.name, datastore.path, mount_point
+ );
+ if let Err(err) =
+ crate::tools::disks::bind_mount(Path::new(&full_store_path), Path::new(&mount_point))
+ {
+ debug!("unmounting '{}'", tmp_mount_path);
+ let _ = crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path);
+ let _ = std::fs::remove_dir(std::path::Path::new(&tmp_mount_path));
+ return Err(format_err!(
+ "Datastore '{}' cound not be mounted: {}.",
+ datastore.name,
+ err
+ ));
+ }
+
+ debug!("unmounting '{}'", tmp_mount_path);
+ crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path)?;
+ std::fs::remove_dir(std::path::Path::new(&tmp_mount_path))?;
+
+ Ok(())
+ } else {
+ Err(format_err!(
+ "Datastore '{}' cannot be mounted because it is not removable.",
+ datastore.name
+ ))
+ }
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ }
+ },
+ returns: {
+ schema: UPID_SCHEMA,
+ },
+ access: {
+ permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
+ },
+)]
+/// Mount removable datastore.
+pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let (section_config, _digest) = pbs_config::datastore::config()?;
+ let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+ if datastore.backing_device.is_none() {
+ bail!("datastore '{store}' is not removable");
+ }
+
+ let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+ let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+ let upid = WorkerTask::new_thread(
+ "mount-device",
+ Some(store),
+ auth_id.to_string(),
+ to_stdout,
+ move |_worker| do_mount_device(datastore),
+ )?;
+
+ Ok(json!(upid))
+}
+
+fn do_unmount_device(
+ datastore: DataStoreConfig,
+ worker: Option<&dyn WorkerTaskContext>,
+) -> Result<(), Error> {
+ let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
+ let mut old_status = String::new();
+ while active_operations.read + active_operations.write > 0 {
+ if let Some(worker) = worker {
+ if worker.abort_requested() {
+ bail!("aborted, due to user request");
+ }
+ let status = format!(
+ "cannot unmount yet, still {} read and {} write operations active",
+ active_operations.read, active_operations.write
+ );
+ if status != old_status {
+ info!("{status}");
+ old_status = status;
+ }
+ }
+ std::thread::sleep(std::time::Duration::from_millis(250));
+ active_operations = task_tracking::get_active_operations(&datastore.name)?;
+ }
+ if let Some(mount_point) = datastore.get_mount_point() {
+ crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
+
+ let _lock = pbs_config::datastore::lock_config()?;
+ let (mut section_config, _digest) = pbs_config::datastore::config()?;
+ let mut store_config: DataStoreConfig =
+ section_config.lookup("datastore", &datastore.name)?;
+ store_config.maintenance_mode = None;
+ section_config.set_data(&datastore.name, "datastore", &store_config)?;
+ pbs_config::datastore::save_config(§ion_config)?;
+ }
+ Ok(())
+}
+
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: { schema: DATASTORE_SCHEMA },
+ },
+ },
+ returns: {
+ schema: UPID_SCHEMA,
+ },
+ access: {
+ permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, true),
+ }
+)]
+/// Unmount a removable device that is associated with the datastore
+pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let _lock = pbs_config::datastore::lock_config()?;
+ let (mut section_config, _digest) = pbs_config::datastore::config()?;
+ let mut datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+
+ if datastore.backing_device.is_none() {
+ bail!("datastore '{store}' is not removable");
+ }
+
+ if !is_datastore_available(&datastore) {
+ bail!("datastore '{store}' is not mounted");
+ }
+
+ datastore.set_maintenance_mode(Some(MaintenanceMode::new(MaintenanceType::Unmount, None)))?;
+ section_config.set_data(&store, "datastore", &datastore)?;
+ pbs_config::datastore::save_config(§ion_config)?;
+
+ drop(_lock);
+
+ let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+ let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+
+ if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+ {
+ let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
+ let _ = proxmox_daemon::command_socket::send_raw(
+ sock,
+ &format!(
+ "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
+ &store
+ ),
+ )
+ .await;
+ }
+
+ let upid = WorkerTask::new_thread(
+ "unmount-device",
+ Some(store),
+ auth_id.to_string(),
+ to_stdout,
+ move |worker| do_unmount_device(datastore, Some(&worker)),
+ )?;
+
+ Ok(json!(upid))
+}
+
#[sortable]
const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
(
@@ -2422,6 +2643,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
.get(&API_METHOD_LIST_GROUPS)
.delete(&API_METHOD_DELETE_GROUP),
),
+ ("mount", &Router::new().post(&API_METHOD_MOUNT)),
(
"namespace",
// FIXME: move into datastore:: sub-module?!
@@ -2456,6 +2678,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
.delete(&API_METHOD_DELETE_SNAPSHOT),
),
("status", &Router::new().get(&API_METHOD_STATUS)),
+ ("unmount", &Router::new().post(&API_METHOD_UNMOUNT)),
(
"upload-backup-log",
&Router::new().upload(&API_METHOD_UPLOAD_BACKUP_LOG),
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 14:58 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/maintenance.rs | 4 +
> src/api2/admin/datastore.rs | 243 +++++++++++++++++++++++++++++--
> 2 files changed, 237 insertions(+), 10 deletions(-)
a pretty substantial patch - no commit message at all??
>
> diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> index 9f51292e..60181258 100644
> --- a/pbs-api-types/src/maintenance.rs
> +++ b/pbs-api-types/src/maintenance.rs
> @@ -78,6 +78,10 @@ pub struct MaintenanceMode {
> }
>
> impl MaintenanceMode {
> + pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
> + Self { ty, message }
> + }
> +
nit: if we want this, it should be in its own patch and convert existing
constructions as well.. or this could be dropped and the call to `new`
below could just init the struct..
> /// Used for deciding whether the datastore is cleared from the internal cache after the last
> /// task finishes, so all open files are closed.
> pub fn is_offline(&self) -> bool {
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 976617d9..3c95888d 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -3,7 +3,7 @@
> use std::collections::HashSet;
> use std::ffi::OsStr;
> use std::os::unix::ffi::OsStrExt;
> -use std::path::PathBuf;
> +use std::path::{Path, PathBuf};
> use std::sync::Arc;
>
> use anyhow::{bail, format_err, Error};
> @@ -13,7 +13,7 @@ use hyper::{header, Body, Response, StatusCode};
> use serde::Deserialize;
> use serde_json::{json, Value};
> use tokio_stream::wrappers::ReceiverStream;
> -use tracing::{info, warn};
> +use tracing::{debug, info, warn};
>
> use proxmox_async::blocking::WrappedReaderStream;
> use proxmox_async::{io::AsyncChannelWriter, stream::AsyncReaderStream};
> @@ -29,6 +29,7 @@ use proxmox_sys::fs::{
> file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
> };
> use proxmox_time::CalendarEvent;
> +use proxmox_worker_task::WorkerTaskContext;
>
> use pxar::accessor::aio::Accessor;
> use pxar::EntryKind;
> @@ -36,12 +37,12 @@ use pxar::EntryKind;
> use pbs_api_types::{
> print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
> Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
> - GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
> - PruneJobOptions, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
> - BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
> - DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
> - PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
> - PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> + GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
> + MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
> + BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
> + BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
> + NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
> + PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> VERIFICATION_OUTDATED_AFTER_SCHEMA,
> };
> use pbs_client::pxar::{create_tar, create_zip};
> @@ -57,8 +58,8 @@ use pbs_datastore::index::IndexFile;
> use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
> use pbs_datastore::prune::compute_prune_info;
> use pbs_datastore::{
> - check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
> - StoreProgress, CATALOG_NAME,
> + check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
> + LocalChunkReader, StoreProgress, CATALOG_NAME,
> };
> use pbs_tools::json::required_string_param;
> use proxmox_rest_server::{formatter, WorkerTask};
> @@ -2384,6 +2385,226 @@ pub async fn set_backup_owner(
> .await?
> }
>
> +/// Here we
> +///
> +/// 1. mount the removable device to `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
> +/// 2. bind mount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>/<datastore.path>` to `/mnt/datastore/<datastore.name>`
> +/// 3. unmount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
couldn't we instead mount directly, and adapt get_absolute_path to
return /mnt/datastore/{datastore.name}/{datastore.path} ?
the "concurrent mounts" can be solved by locking (using the UUID or
datastore name as lock scope) or by going via a Mount maintenance mode..
seems to me like there's a lot less that could go wrong/fail in that
case? but maybe I am missing some background that should have been in
the commit message ;)
if I think of the semantics of "removable datastores", then if we want
to support multiple such datastores on a single device (at least as an
option in the future), then unmounting only makes sense when thinking of
the backing device (after all, I want to unmount to unplug it, and I
don't want to do that 10 times if the backing devices contains 10
datastores, and keep track of them all, and ensure nothing mounts them
again in the meantime, ..).. so only mounting it once in the first place
(and pointing the datastore at the relevant subdir of the mountpoint)
seems like the better option?
> +///
> +/// leaving us with the datastore being mounted directly with its name under /mnt/datastore/...
> +///
> +/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
> +/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
> +/// technically possible.
> +pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
> + if let (Some(uuid), Some(mount_point)) = (
> + datastore.backing_device.as_ref(),
> + datastore.get_mount_point(),
> + ) {
> + if pbs_datastore::is_datastore_available(&datastore) {
> + bail!("datastore '{}' is already mounted", datastore.name);
> + }
> + let tmp_mount_path = format!(
> + "{}/{:x}",
> + pbs_buildcfg::rundir!("/mount"),
> + proxmox_uuid::Uuid::generate()
> + );
> +
> + let default_options = proxmox_sys::fs::CreateOptions::new();
> + proxmox_sys::fs::create_path(
> + &tmp_mount_path,
> + Some(default_options.clone()),
> + Some(default_options.clone()),
> + )?;
> +
> + debug!("mounting '{uuid}' to '{}'", tmp_mount_path);
> + crate::tools::disks::mount_by_uuid(uuid, Path::new(&tmp_mount_path))?;
> +
> + let full_store_path = format!(
> + "{tmp_mount_path}/{}",
> + datastore.path.trim_start_matches('/')
> + );
> +
> + proxmox_sys::fs::create_path(
> + &mount_point,
> + Some(default_options.clone()),
> + Some(default_options.clone()),
> + )?;
> +
> + // can't be created before it is mounted, so we have to do it here
> + proxmox_sys::fs::create_path(
> + &full_store_path,
> + Some(default_options.clone()),
> + Some(default_options.clone()),
> + )?;
shouldn't this create_path call be limited to the initial creation of
the datastore? in all other cases it should already be there..
> +
> + info!(
> + "mounting '{}'({}) to '{}'",
> + datastore.name, datastore.path, mount_point
> + );
> + if let Err(err) =
> + crate::tools::disks::bind_mount(Path::new(&full_store_path), Path::new(&mount_point))
> + {
> + debug!("unmounting '{}'", tmp_mount_path);
> + let _ = crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path);
> + let _ = std::fs::remove_dir(std::path::Path::new(&tmp_mount_path));
> + return Err(format_err!(
> + "Datastore '{}' cound not be mounted: {}.",
> + datastore.name,
> + err
> + ));
> + }
> +
> + debug!("unmounting '{}'", tmp_mount_path);
> + crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path)?;
> + std::fs::remove_dir(std::path::Path::new(&tmp_mount_path))?;
> +
> + Ok(())
> + } else {
> + Err(format_err!(
> + "Datastore '{}' cannot be mounted because it is not removable.",
> + datastore.name
> + ))
> + }
> +}
> +
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + }
> + },
> + returns: {
> + schema: UPID_SCHEMA,
> + },
> + access: {
> + permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
> + },
> +)]
> +/// Mount removable datastore.
> +pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> + let (section_config, _digest) = pbs_config::datastore::config()?;
> + let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
> +
> + if datastore.backing_device.is_none() {
> + bail!("datastore '{store}' is not removable");
> + }
> +
> + let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> + let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> +
> + let upid = WorkerTask::new_thread(
> + "mount-device",
> + Some(store),
> + auth_id.to_string(),
> + to_stdout,
> + move |_worker| do_mount_device(datastore),
> + )?;
> +
> + Ok(json!(upid))
> +}
> +
> +fn do_unmount_device(
> + datastore: DataStoreConfig,
> + worker: Option<&dyn WorkerTaskContext>,
> +) -> Result<(), Error> {
> + let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
> + let mut old_status = String::new();
> + while active_operations.read + active_operations.write > 0 {
> + if let Some(worker) = worker {
> + if worker.abort_requested() {
> + bail!("aborted, due to user request");
> + }
> + let status = format!(
> + "cannot unmount yet, still {} read and {} write operations active",
> + active_operations.read, active_operations.write
> + );
> + if status != old_status {
> + info!("{status}");
> + old_status = status;
> + }
> + }
> + std::thread::sleep(std::time::Duration::from_millis(250));
I think once per second is probably enough? on a busy/big server there
might be a lot of operations that we are waiting for (and thus a lot of
decrements of the counters)..
> + active_operations = task_tracking::get_active_operations(&datastore.name)?;
> + }
> + if let Some(mount_point) = datastore.get_mount_point() {
shouldn't this if be right at the start?
> + crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
> +
> + let _lock = pbs_config::datastore::lock_config()?;
> + let (mut section_config, _digest) = pbs_config::datastore::config()?;
> + let mut store_config: DataStoreConfig =
> + section_config.lookup("datastore", &datastore.name)?;
should we re-check the current maintenance_mode here to avoid races?
> + store_config.maintenance_mode = None;
should use set_maintenance_mode
> + section_config.set_data(&datastore.name, "datastore", &store_config)?;
> + pbs_config::datastore::save_config(§ion_config)?;
> + }
> + Ok(())
> +}
> +
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + store: { schema: DATASTORE_SCHEMA },
> + },
> + },
> + returns: {
> + schema: UPID_SCHEMA,
> + },
> + access: {
> + permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, true),
> + }
> +)]
> +/// Unmount a removable device that is associated with the datastore
> +pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> + let _lock = pbs_config::datastore::lock_config()?;
> + let (mut section_config, _digest) = pbs_config::datastore::config()?;
> + let mut datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
> +
> + if datastore.backing_device.is_none() {
> + bail!("datastore '{store}' is not removable");
> + }
> +
> + if !is_datastore_available(&datastore) {
> + bail!("datastore '{store}' is not mounted");
> + }
> +
> + datastore.set_maintenance_mode(Some(MaintenanceMode::new(MaintenanceType::Unmount, None)))?;
> + section_config.set_data(&store, "datastore", &datastore)?;
> + pbs_config::datastore::save_config(§ion_config)?;
> +
> + drop(_lock);
> +
> + let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> + let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> +
> + if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
> + {
> + let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
> + let _ = proxmox_daemon::command_socket::send_raw(
> + sock,
> + &format!(
> + "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
> + &store
> + ),
> + )
> + .await;
> + }
> +
> + let upid = WorkerTask::new_thread(
> + "unmount-device",
> + Some(store),
> + auth_id.to_string(),
> + to_stdout,
> + move |worker| do_unmount_device(datastore, Some(&worker)),
> + )?;
> +
> + Ok(json!(upid))
> +}
> +
> #[sortable]
> const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> (
> @@ -2422,6 +2643,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> .get(&API_METHOD_LIST_GROUPS)
> .delete(&API_METHOD_DELETE_GROUP),
> ),
> + ("mount", &Router::new().post(&API_METHOD_MOUNT)),
> (
> "namespace",
> // FIXME: move into datastore:: sub-module?!
> @@ -2456,6 +2678,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> .delete(&API_METHOD_DELETE_SNAPSHOT),
> ),
> ("status", &Router::new().get(&API_METHOD_STATUS)),
> + ("unmount", &Router::new().post(&API_METHOD_UNMOUNT)),
> (
> "upload-backup-log",
> &Router::new().upload(&API_METHOD_UPLOAD_BACKUP_LOG),
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 14:58 ` Hannes Laimer
2024-10-30 10:07 ` Fabian Grünbichler
0 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 14:58 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > pbs-api-types/src/maintenance.rs | 4 +
> > src/api2/admin/datastore.rs | 243 +++++++++++++++++++++++++++++--
> > 2 files changed, 237 insertions(+), 10 deletions(-)
>
> a pretty substantial patch - no commit message at all??
>
> >
> > diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> > index 9f51292e..60181258 100644
> > --- a/pbs-api-types/src/maintenance.rs
> > +++ b/pbs-api-types/src/maintenance.rs
> > @@ -78,6 +78,10 @@ pub struct MaintenanceMode {
> > }
> >
> > impl MaintenanceMode {
> > + pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
> > + Self { ty, message }
> > + }
> > +
>
> nit: if we want this, it should be in its own patch and convert existing
> constructions as well.. or this could be dropped and the call to `new`
> below could just init the struct..
>
> > /// Used for deciding whether the datastore is cleared from the internal cache after the last
> > /// task finishes, so all open files are closed.
> > pub fn is_offline(&self) -> bool {
> > diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> > index 976617d9..3c95888d 100644
> > --- a/src/api2/admin/datastore.rs
> > +++ b/src/api2/admin/datastore.rs
> > @@ -3,7 +3,7 @@
> > use std::collections::HashSet;
> > use std::ffi::OsStr;
> > use std::os::unix::ffi::OsStrExt;
> > -use std::path::PathBuf;
> > +use std::path::{Path, PathBuf};
> > use std::sync::Arc;
> >
> > use anyhow::{bail, format_err, Error};
> > @@ -13,7 +13,7 @@ use hyper::{header, Body, Response, StatusCode};
> > use serde::Deserialize;
> > use serde_json::{json, Value};
> > use tokio_stream::wrappers::ReceiverStream;
> > -use tracing::{info, warn};
> > +use tracing::{debug, info, warn};
> >
> > use proxmox_async::blocking::WrappedReaderStream;
> > use proxmox_async::{io::AsyncChannelWriter, stream::AsyncReaderStream};
> > @@ -29,6 +29,7 @@ use proxmox_sys::fs::{
> > file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
> > };
> > use proxmox_time::CalendarEvent;
> > +use proxmox_worker_task::WorkerTaskContext;
> >
> > use pxar::accessor::aio::Accessor;
> > use pxar::EntryKind;
> > @@ -36,12 +37,12 @@ use pxar::EntryKind;
> > use pbs_api_types::{
> > print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
> > Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
> > - GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
> > - PruneJobOptions, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
> > - BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
> > - DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
> > - PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
> > - PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> > + GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
> > + MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
> > + BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
> > + BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
> > + NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
> > + PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> > VERIFICATION_OUTDATED_AFTER_SCHEMA,
> > };
> > use pbs_client::pxar::{create_tar, create_zip};
> > @@ -57,8 +58,8 @@ use pbs_datastore::index::IndexFile;
> > use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
> > use pbs_datastore::prune::compute_prune_info;
> > use pbs_datastore::{
> > - check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
> > - StoreProgress, CATALOG_NAME,
> > + check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
> > + LocalChunkReader, StoreProgress, CATALOG_NAME,
> > };
> > use pbs_tools::json::required_string_param;
> > use proxmox_rest_server::{formatter, WorkerTask};
> > @@ -2384,6 +2385,226 @@ pub async fn set_backup_owner(
> > .await?
> > }
> >
> > +/// Here we
> > +///
> > +/// 1. mount the removable device to `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
> > +/// 2. bind mount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>/<datastore.path>` to `/mnt/datastore/<datastore.name>`
> > +/// 3. unmount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
>
> couldn't we instead mount directly, and adapt get_absolute_path to
> return /mnt/datastore/{datastore.name}/{datastore.path} ?
>
but then we would either mount all, or none, of the datastores on the
device.
> the "concurrent mounts" can be solved by locking (using the UUID or
> datastore name as lock scope) or by going via a Mount maintenance mode..
>
yes, I'm just not sure that locking really adds something. We
can mount in paralell, we just have to make sure we don't use the same tmp
dir name for it.
> seems to me like there's a lot less that could go wrong/fail in that
> case? but maybe I am missing some background that should have been in
> the commit message ;)
>
most of what can go wrong comes from needing a bunch of tmp things and
the (bind) mounting that happens, that would still happen, even
non-concurrently.
> if I think of the semantics of "removable datastores", then if we want
> to support multiple such datastores on a single device (at least as an
> option in the future), then unmounting only makes sense when thinking of
> the backing device (after all, I want to unmount to unplug it, and I
> don't want to do that 10 times if the backing devices contains 10
> datastores, and keep track of them all, and ensure nothing mounts them
> again in the meantime, ..).. so only mounting it once in the first place
> (and pointing the datastore at the relevant subdir of the mountpoint)
> seems like the better option?
>
We could not auto mount if a device that contains multiple datastores, so
what someone mounted they also have to unmount. I feel like clicking
"unmount" on one datastore and 4 other ones also end up un-mounted is
not really something I'd expect, also we'd have to track active
operations on all of them when unounting, which we can do, but idk...
This could make sense if we had the notion of a device, and unmounting
the deivce unmounts all this datastores, but I don't think that is what we want.
> > +///
> > +/// leaving us with the datastore being mounted directly with its name under /mnt/datastore/...
> > +///
> > +/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
> > +/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
> > +/// technically possible.
> > +pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
> > + if let (Some(uuid), Some(mount_point)) = (
> > + datastore.backing_device.as_ref(),
> > + datastore.get_mount_point(),
> > + ) {
> > + if pbs_datastore::is_datastore_available(&datastore) {
> > + bail!("datastore '{}' is already mounted", datastore.name);
> > + }
> > + let tmp_mount_path = format!(
> > + "{}/{:x}",
> > + pbs_buildcfg::rundir!("/mount"),
> > + proxmox_uuid::Uuid::generate()
> > + );
> > +
> > + let default_options = proxmox_sys::fs::CreateOptions::new();
> > + proxmox_sys::fs::create_path(
> > + &tmp_mount_path,
> > + Some(default_options.clone()),
> > + Some(default_options.clone()),
> > + )?;
> > +
> > + debug!("mounting '{uuid}' to '{}'", tmp_mount_path);
> > + crate::tools::disks::mount_by_uuid(uuid, Path::new(&tmp_mount_path))?;
> > +
> > + let full_store_path = format!(
> > + "{tmp_mount_path}/{}",
> > + datastore.path.trim_start_matches('/')
> > + );
> > +
> > + proxmox_sys::fs::create_path(
> > + &mount_point,
> > + Some(default_options.clone()),
> > + Some(default_options.clone()),
> > + )?;
> > +
> > + // can't be created before it is mounted, so we have to do it here
> > + proxmox_sys::fs::create_path(
> > + &full_store_path,
> > + Some(default_options.clone()),
> > + Some(default_options.clone()),
> > + )?;
>
> shouldn't this create_path call be limited to the initial creation of
> the datastore? in all other cases it should already be there..
>
we'd have to check if `.chunks/` exists for that, and
create_path already kind of does that. But manually checking
is probably a little faster(?)
> > +
> > + info!(
> > + "mounting '{}'({}) to '{}'",
> > + datastore.name, datastore.path, mount_point
> > + );
> > + if let Err(err) =
> > + crate::tools::disks::bind_mount(Path::new(&full_store_path), Path::new(&mount_point))
> > + {
> > + debug!("unmounting '{}'", tmp_mount_path);
> > + let _ = crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path);
> > + let _ = std::fs::remove_dir(std::path::Path::new(&tmp_mount_path));
> > + return Err(format_err!(
> > + "Datastore '{}' cound not be mounted: {}.",
> > + datastore.name,
> > + err
> > + ));
> > + }
> > +
> > + debug!("unmounting '{}'", tmp_mount_path);
> > + crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path)?;
> > + std::fs::remove_dir(std::path::Path::new(&tmp_mount_path))?;
> > +
> > + Ok(())
> > + } else {
> > + Err(format_err!(
> > + "Datastore '{}' cannot be mounted because it is not removable.",
> > + datastore.name
> > + ))
> > + }
> > +}
> > +
> > +#[api(
> > + protected: true,
> > + input: {
> > + properties: {
> > + store: {
> > + schema: DATASTORE_SCHEMA,
> > + },
> > + }
> > + },
> > + returns: {
> > + schema: UPID_SCHEMA,
> > + },
> > + access: {
> > + permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
> > + },
> > +)]
> > +/// Mount removable datastore.
> > +pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> > + let (section_config, _digest) = pbs_config::datastore::config()?;
> > + let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
> > +
> > + if datastore.backing_device.is_none() {
> > + bail!("datastore '{store}' is not removable");
> > + }
> > +
> > + let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> > + let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> > +
> > + let upid = WorkerTask::new_thread(
> > + "mount-device",
> > + Some(store),
> > + auth_id.to_string(),
> > + to_stdout,
> > + move |_worker| do_mount_device(datastore),
> > + )?;
> > +
> > + Ok(json!(upid))
> > +}
> > +
> > +fn do_unmount_device(
> > + datastore: DataStoreConfig,
> > + worker: Option<&dyn WorkerTaskContext>,
> > +) -> Result<(), Error> {
> > + let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
> > + let mut old_status = String::new();
> > + while active_operations.read + active_operations.write > 0 {
> > + if let Some(worker) = worker {
> > + if worker.abort_requested() {
> > + bail!("aborted, due to user request");
> > + }
> > + let status = format!(
> > + "cannot unmount yet, still {} read and {} write operations active",
> > + active_operations.read, active_operations.write
> > + );
> > + if status != old_status {
> > + info!("{status}");
> > + old_status = status;
> > + }
> > + }
> > + std::thread::sleep(std::time::Duration::from_millis(250));
>
> I think once per second is probably enough? on a busy/big server there
> might be a lot of operations that we are waiting for (and thus a lot of
> decrements of the counters)..
>
makes sense
> > + active_operations = task_tracking::get_active_operations(&datastore.name)?;
> > + }
> > + if let Some(mount_point) = datastore.get_mount_point() {
>
> shouldn't this if be right at the start?
>
this fn is only called in the unmount endpoint, which does this check.
But checking again wouldn't hurt, I guess
> > + crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
> > +
> > + let _lock = pbs_config::datastore::lock_config()?;
> > + let (mut section_config, _digest) = pbs_config::datastore::config()?;
> > + let mut store_config: DataStoreConfig =
> > + section_config.lookup("datastore", &datastore.name)?;
>
> should we re-check the current maintenance_mode here to avoid races?
>
I think the only thing we could race against are other unmount tasks,
and both would set it to `None`
> > + store_config.maintenance_mode = None;
>
> should use set_maintenance_mode
>
we set the maintenance to `Unmount` before starting the unmounting, we
can't leave that except by setting it like this. If something should go
wrong we want this locked in, so to say. So manually examinig the
situation, and editing the config file is needed.
> > + section_config.set_data(&datastore.name, "datastore", &store_config)?;
> > + pbs_config::datastore::save_config(§ion_config)?;
> > + }
> > + Ok(())
> > +}
> > +
> > +#[api(
> > + protected: true,
> > + input: {
> > + properties: {
> > + store: { schema: DATASTORE_SCHEMA },
> > + },
> > + },
> > + returns: {
> > + schema: UPID_SCHEMA,
> > + },
> > + access: {
> > + permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, true),
> > + }
> > +)]
> > +/// Unmount a removable device that is associated with the datastore
> > +pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> > + let _lock = pbs_config::datastore::lock_config()?;
> > + let (mut section_config, _digest) = pbs_config::datastore::config()?;
> > + let mut datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
> > +
> > + if datastore.backing_device.is_none() {
> > + bail!("datastore '{store}' is not removable");
> > + }
> > +
> > + if !is_datastore_available(&datastore) {
> > + bail!("datastore '{store}' is not mounted");
> > + }
> > +
> > + datastore.set_maintenance_mode(Some(MaintenanceMode::new(MaintenanceType::Unmount, None)))?;
> > + section_config.set_data(&store, "datastore", &datastore)?;
> > + pbs_config::datastore::save_config(§ion_config)?;
> > +
> > + drop(_lock);
> > +
> > + let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> > + let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> > +
> > + if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
> > + {
> > + let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
> > + let _ = proxmox_daemon::command_socket::send_raw(
> > + sock,
> > + &format!(
> > + "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
> > + &store
> > + ),
> > + )
> > + .await;
> > + }
> > +
> > + let upid = WorkerTask::new_thread(
> > + "unmount-device",
> > + Some(store),
> > + auth_id.to_string(),
> > + to_stdout,
> > + move |worker| do_unmount_device(datastore, Some(&worker)),
> > + )?;
> > +
> > + Ok(json!(upid))
> > +}
> > +
> > #[sortable]
> > const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> > (
> > @@ -2422,6 +2643,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> > .get(&API_METHOD_LIST_GROUPS)
> > .delete(&API_METHOD_DELETE_GROUP),
> > ),
> > + ("mount", &Router::new().post(&API_METHOD_MOUNT)),
> > (
> > "namespace",
> > // FIXME: move into datastore:: sub-module?!
> > @@ -2456,6 +2678,7 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
> > .delete(&API_METHOD_DELETE_SNAPSHOT),
> > ),
> > ("status", &Router::new().get(&API_METHOD_STATUS)),
> > + ("unmount", &Router::new().post(&API_METHOD_UNMOUNT)),
> > (
> > "upload-backup-log",
> > &Router::new().upload(&API_METHOD_UPLOAD_BACKUP_LOG),
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores
2024-10-29 14:58 ` Hannes Laimer
@ 2024-10-30 10:07 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-30 10:07 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
Quoting Hannes Laimer (2024-10-29 15:58:25)
> On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> > On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > > ---
> > > pbs-api-types/src/maintenance.rs | 4 +
> > > src/api2/admin/datastore.rs | 243 +++++++++++++++++++++++++++++--
> > > 2 files changed, 237 insertions(+), 10 deletions(-)
> >
> > a pretty substantial patch - no commit message at all??
> >
> > >
> > > diff --git a/pbs-api-types/src/maintenance.rs b/pbs-api-types/src/maintenance.rs
> > > index 9f51292e..60181258 100644
> > > --- a/pbs-api-types/src/maintenance.rs
> > > +++ b/pbs-api-types/src/maintenance.rs
> > > @@ -78,6 +78,10 @@ pub struct MaintenanceMode {
> > > }
> > >
> > > impl MaintenanceMode {
> > > + pub fn new(ty: MaintenanceType, message: Option<String>) -> Self {
> > > + Self { ty, message }
> > > + }
> > > +
> >
> > nit: if we want this, it should be in its own patch and convert existing
> > constructions as well.. or this could be dropped and the call to `new`
> > below could just init the struct..
> >
> > > /// Used for deciding whether the datastore is cleared from the internal cache after the last
> > > /// task finishes, so all open files are closed.
> > > pub fn is_offline(&self) -> bool {
> > > diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> > > index 976617d9..3c95888d 100644
> > > --- a/src/api2/admin/datastore.rs
> > > +++ b/src/api2/admin/datastore.rs
> > > @@ -3,7 +3,7 @@
> > > use std::collections::HashSet;
> > > use std::ffi::OsStr;
> > > use std::os::unix::ffi::OsStrExt;
> > > -use std::path::PathBuf;
> > > +use std::path::{Path, PathBuf};
> > > use std::sync::Arc;
> > >
> > > use anyhow::{bail, format_err, Error};
> > > @@ -13,7 +13,7 @@ use hyper::{header, Body, Response, StatusCode};
> > > use serde::Deserialize;
> > > use serde_json::{json, Value};
> > > use tokio_stream::wrappers::ReceiverStream;
> > > -use tracing::{info, warn};
> > > +use tracing::{debug, info, warn};
> > >
> > > use proxmox_async::blocking::WrappedReaderStream;
> > > use proxmox_async::{io::AsyncChannelWriter, stream::AsyncReaderStream};
> > > @@ -29,6 +29,7 @@ use proxmox_sys::fs::{
> > > file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
> > > };
> > > use proxmox_time::CalendarEvent;
> > > +use proxmox_worker_task::WorkerTaskContext;
> > >
> > > use pxar::accessor::aio::Accessor;
> > > use pxar::EntryKind;
> > > @@ -36,12 +37,12 @@ use pxar::EntryKind;
> > > use pbs_api_types::{
> > > print_ns_and_snapshot, print_store_and_ns, Authid, BackupContent, BackupNamespace, BackupType,
> > > Counts, CryptMode, DataStoreConfig, DataStoreListItem, DataStoreStatus,
> > > - GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, Operation,
> > > - PruneJobOptions, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
> > > - BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
> > > - DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
> > > - PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
> > > - PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> > > + GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
> > > + MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
> > > + BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
> > > + BACKUP_TYPE_SCHEMA, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH,
> > > + NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY,
> > > + PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, UPID, UPID_SCHEMA,
> > > VERIFICATION_OUTDATED_AFTER_SCHEMA,
> > > };
> > > use pbs_client::pxar::{create_tar, create_zip};
> > > @@ -57,8 +58,8 @@ use pbs_datastore::index::IndexFile;
> > > use pbs_datastore::manifest::{BackupManifest, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME};
> > > use pbs_datastore::prune::compute_prune_info;
> > > use pbs_datastore::{
> > > - check_backup_owner, task_tracking, BackupDir, BackupGroup, DataStore, LocalChunkReader,
> > > - StoreProgress, CATALOG_NAME,
> > > + check_backup_owner, is_datastore_available, task_tracking, BackupDir, BackupGroup, DataStore,
> > > + LocalChunkReader, StoreProgress, CATALOG_NAME,
> > > };
> > > use pbs_tools::json::required_string_param;
> > > use proxmox_rest_server::{formatter, WorkerTask};
> > > @@ -2384,6 +2385,226 @@ pub async fn set_backup_owner(
> > > .await?
> > > }
> > >
> > > +/// Here we
> > > +///
> > > +/// 1. mount the removable device to `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
> > > +/// 2. bind mount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>/<datastore.path>` to `/mnt/datastore/<datastore.name>`
> > > +/// 3. unmount `<PBS_RUN_DIR>/mount/<RANDOM_UUID>`
> >
> > couldn't we instead mount directly, and adapt get_absolute_path to
> > return /mnt/datastore/{datastore.name}/{datastore.path} ?
> >
>
> but then we would either mount all, or none, of the datastores on the
> device.
which I think is probably the expected behaviour, but I can see the argument
against it as well..
>
> > the "concurrent mounts" can be solved by locking (using the UUID or
> > datastore name as lock scope) or by going via a Mount maintenance mode..
> >
>
> yes, I'm just not sure that locking really adds something. We
> can mount in paralell, we just have to make sure we don't use the same tmp
> dir name for it.
>
> > seems to me like there's a lot less that could go wrong/fail in that
> > case? but maybe I am missing some background that should have been in
> > the commit message ;)
> >
>
> most of what can go wrong comes from needing a bunch of tmp things and
> the (bind) mounting that happens, that would still happen, even
> non-concurrently.
well, with one mount per device there would be no need for tmp things and bind mounts at all?
>
> > if I think of the semantics of "removable datastores", then if we want
> > to support multiple such datastores on a single device (at least as an
> > option in the future), then unmounting only makes sense when thinking of
> > the backing device (after all, I want to unmount to unplug it, and I
> > don't want to do that 10 times if the backing devices contains 10
> > datastores, and keep track of them all, and ensure nothing mounts them
> > again in the meantime, ..).. so only mounting it once in the first place
> > (and pointing the datastore at the relevant subdir of the mountpoint)
> > seems like the better option?
> >
>
> We could not auto mount if a device that contains multiple datastores, so
> what someone mounted they also have to unmount. I feel like clicking
> "unmount" on one datastore and 4 other ones also end up un-mounted is
> not really something I'd expect, also we'd have to track active
> operations on all of them when unounting, which we can do, but idk...
> This could make sense if we had the notion of a device, and unmounting
> the deivce unmounts all this datastores, but I don't think that is what we want.
well the whole purpose of removable datastores is to allow rotating those
devices or putting them in cold storage.. so if we want to support multiple
datastores on the same device, then a single unmount for all of them would make
a lot of sense IMHO.. just unmounting one of N datastores on the same device
has no purpose?
>
> > > +///
> > > +/// leaving us with the datastore being mounted directly with its name under /mnt/datastore/...
> > > +///
> > > +/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
> > > +/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
> > > +/// technically possible.
> > > +pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
> > > + if let (Some(uuid), Some(mount_point)) = (
> > > + datastore.backing_device.as_ref(),
> > > + datastore.get_mount_point(),
> > > + ) {
> > > + if pbs_datastore::is_datastore_available(&datastore) {
> > > + bail!("datastore '{}' is already mounted", datastore.name);
> > > + }
> > > + let tmp_mount_path = format!(
> > > + "{}/{:x}",
> > > + pbs_buildcfg::rundir!("/mount"),
> > > + proxmox_uuid::Uuid::generate()
> > > + );
> > > +
> > > + let default_options = proxmox_sys::fs::CreateOptions::new();
> > > + proxmox_sys::fs::create_path(
> > > + &tmp_mount_path,
> > > + Some(default_options.clone()),
> > > + Some(default_options.clone()),
> > > + )?;
> > > +
> > > + debug!("mounting '{uuid}' to '{}'", tmp_mount_path);
> > > + crate::tools::disks::mount_by_uuid(uuid, Path::new(&tmp_mount_path))?;
> > > +
> > > + let full_store_path = format!(
> > > + "{tmp_mount_path}/{}",
> > > + datastore.path.trim_start_matches('/')
> > > + );
> > > +
> > > + proxmox_sys::fs::create_path(
> > > + &mount_point,
> > > + Some(default_options.clone()),
> > > + Some(default_options.clone()),
> > > + )?;
> > > +
> > > + // can't be created before it is mounted, so we have to do it here
> > > + proxmox_sys::fs::create_path(
> > > + &full_store_path,
> > > + Some(default_options.clone()),
> > > + Some(default_options.clone()),
> > > + )?;
> >
> > shouldn't this create_path call be limited to the initial creation of
> > the datastore? in all other cases it should already be there..
> >
>
> we'd have to check if `.chunks/` exists for that, and
> create_path already kind of does that. But manually checking
> is probably a little faster(?)
>
> > > +
> > > + info!(
> > > + "mounting '{}'({}) to '{}'",
> > > + datastore.name, datastore.path, mount_point
> > > + );
> > > + if let Err(err) =
> > > + crate::tools::disks::bind_mount(Path::new(&full_store_path), Path::new(&mount_point))
> > > + {
> > > + debug!("unmounting '{}'", tmp_mount_path);
> > > + let _ = crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path);
> > > + let _ = std::fs::remove_dir(std::path::Path::new(&tmp_mount_path));
> > > + return Err(format_err!(
> > > + "Datastore '{}' cound not be mounted: {}.",
> > > + datastore.name,
> > > + err
> > > + ));
> > > + }
> > > +
> > > + debug!("unmounting '{}'", tmp_mount_path);
> > > + crate::tools::disks::unmount_by_mountpoint(&tmp_mount_path)?;
> > > + std::fs::remove_dir(std::path::Path::new(&tmp_mount_path))?;
> > > +
> > > + Ok(())
> > > + } else {
> > > + Err(format_err!(
> > > + "Datastore '{}' cannot be mounted because it is not removable.",
> > > + datastore.name
> > > + ))
> > > + }
> > > +}
> > > +
> > > +#[api(
> > > + protected: true,
> > > + input: {
> > > + properties: {
> > > + store: {
> > > + schema: DATASTORE_SCHEMA,
> > > + },
> > > + }
> > > + },
> > > + returns: {
> > > + schema: UPID_SCHEMA,
> > > + },
> > > + access: {
> > > + permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_AUDIT, false),
> > > + },
> > > +)]
> > > +/// Mount removable datastore.
> > > +pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> > > + let (section_config, _digest) = pbs_config::datastore::config()?;
> > > + let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
> > > +
> > > + if datastore.backing_device.is_none() {
> > > + bail!("datastore '{store}' is not removable");
> > > + }
> > > +
> > > + let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> > > + let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> > > +
> > > + let upid = WorkerTask::new_thread(
> > > + "mount-device",
> > > + Some(store),
> > > + auth_id.to_string(),
> > > + to_stdout,
> > > + move |_worker| do_mount_device(datastore),
> > > + )?;
> > > +
> > > + Ok(json!(upid))
> > > +}
> > > +
> > > +fn do_unmount_device(
> > > + datastore: DataStoreConfig,
> > > + worker: Option<&dyn WorkerTaskContext>,
> > > +) -> Result<(), Error> {
> > > + let mut active_operations = task_tracking::get_active_operations(&datastore.name)?;
> > > + let mut old_status = String::new();
> > > + while active_operations.read + active_operations.write > 0 {
> > > + if let Some(worker) = worker {
> > > + if worker.abort_requested() {
> > > + bail!("aborted, due to user request");
> > > + }
> > > + let status = format!(
> > > + "cannot unmount yet, still {} read and {} write operations active",
> > > + active_operations.read, active_operations.write
> > > + );
> > > + if status != old_status {
> > > + info!("{status}");
> > > + old_status = status;
> > > + }
> > > + }
> > > + std::thread::sleep(std::time::Duration::from_millis(250));
> >
> > I think once per second is probably enough? on a busy/big server there
> > might be a lot of operations that we are waiting for (and thus a lot of
> > decrements of the counters)..
> >
>
> makes sense
>
> > > + active_operations = task_tracking::get_active_operations(&datastore.name)?;
> > > + }
> > > + if let Some(mount_point) = datastore.get_mount_point() {
> >
> > shouldn't this if be right at the start?
> >
>
> this fn is only called in the unmount endpoint, which does this check.
> But checking again wouldn't hurt, I guess
>
> > > + crate::tools::disks::unmount_by_mountpoint(&mount_point)?;
> > > +
> > > + let _lock = pbs_config::datastore::lock_config()?;
> > > + let (mut section_config, _digest) = pbs_config::datastore::config()?;
> > > + let mut store_config: DataStoreConfig =
> > > + section_config.lookup("datastore", &datastore.name)?;
> >
> > should we re-check the current maintenance_mode here to avoid races?
> >
>
> I think the only thing we could race against are other unmount tasks,
> and both would set it to `None`
or an admin setting it to something else manually because they gave up on
unmounting and we give them no way to cancel that ;)
>
> > > + store_config.maintenance_mode = None;
> >
> > should use set_maintenance_mode
> >
>
> we set the maintenance to `Unmount` before starting the unmounting, we
> can't leave that except by setting it like this. If something should go
> wrong we want this locked in, so to say. So manually examinig the
> situation, and editing the config file is needed.
wouldn't it make sense to allow cancelling an unmount by setting the
maintenance mode to something else again?
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (6 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 07/26] api: admin: add (un)mount endpoint for removable datastores Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag Hannes Laimer
` (18 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/config/datastore.rs | 37 ++++++++++++++++++++++++++++++++----
1 file changed, 33 insertions(+), 4 deletions(-)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index ca6edf05..c24b6e9d 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -4,7 +4,7 @@ use ::serde::{Deserialize, Serialize};
use anyhow::Error;
use hex::FromHex;
use serde_json::Value;
-use tracing::warn;
+use tracing::{info, warn};
use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
use proxmox_schema::{api, param_bail, ApiType};
@@ -20,7 +20,8 @@ use pbs_config::BackupLockGuard;
use pbs_datastore::chunk_store::ChunkStore;
use crate::api2::admin::{
- prune::list_prune_jobs, sync::list_sync_jobs, verify::list_verification_jobs,
+ datastore::do_mount_device, prune::list_prune_jobs, sync::list_sync_jobs,
+ verify::list_verification_jobs,
};
use crate::api2::config::prune::{delete_prune_job, do_create_prune_job};
use crate::api2::config::sync::delete_sync_job;
@@ -71,13 +72,30 @@ pub(crate) fn do_create_datastore(
mut config: SectionConfigData,
datastore: DataStoreConfig,
) -> Result<(), Error> {
- let path: PathBuf = datastore.path.clone().into();
+ let path: PathBuf = datastore.absolute_path().into();
+ let backup_user = pbs_config::backup_user()?;
+ if let Some(store_mount_point) = datastore.get_mount_point() {
+ let default_options = proxmox_sys::fs::CreateOptions::new();
+ proxmox_sys::fs::create_path(
+ store_mount_point,
+ Some(default_options.clone()),
+ Some(default_options.clone()),
+ )?;
+ do_mount_device(datastore.clone())?;
+
+ if path.join(".chunks").is_dir() {
+ config.set_data(&datastore.name, "datastore", &datastore)?;
+ pbs_config::datastore::save_config(&config)?;
+ jobstate::create_state_file("garbage_collection", &datastore.name)?;
+ info!("created removable datastore, chunkstore already exists");
+ return Ok(());
+ }
+ }
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
.parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
)?;
- let backup_user = pbs_config::backup_user()?;
let _store = ChunkStore::create(
&datastore.name,
path,
@@ -120,6 +138,17 @@ pub fn create_datastore(
param_bail!("name", "datastore '{}' already exists.", config.name);
}
+ if let Some(uuid) = &config.backing_device {
+ for (datastore_name, (_, config)) in §ion_config.sections {
+ if config["backing-device"].as_str() == Some(uuid) {
+ param_bail!(
+ "backing-device",
+ "device already in use by datastore '{datastore_name}'",
+ );
+ }
+ }
+ }
+
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 15:15 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/api2/config/datastore.rs | 37 ++++++++++++++++++++++++++++++++----
> 1 file changed, 33 insertions(+), 4 deletions(-)
>
> diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
> index ca6edf05..c24b6e9d 100644
> --- a/src/api2/config/datastore.rs
> +++ b/src/api2/config/datastore.rs
> @@ -4,7 +4,7 @@ use ::serde::{Deserialize, Serialize};
> use anyhow::Error;
> use hex::FromHex;
> use serde_json::Value;
> -use tracing::warn;
> +use tracing::{info, warn};
>
> use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
> use proxmox_schema::{api, param_bail, ApiType};
> @@ -20,7 +20,8 @@ use pbs_config::BackupLockGuard;
> use pbs_datastore::chunk_store::ChunkStore;
>
> use crate::api2::admin::{
> - prune::list_prune_jobs, sync::list_sync_jobs, verify::list_verification_jobs,
> + datastore::do_mount_device, prune::list_prune_jobs, sync::list_sync_jobs,
> + verify::list_verification_jobs,
> };
> use crate::api2::config::prune::{delete_prune_job, do_create_prune_job};
> use crate::api2::config::sync::delete_sync_job;
> @@ -71,13 +72,30 @@ pub(crate) fn do_create_datastore(
> mut config: SectionConfigData,
> datastore: DataStoreConfig,
> ) -> Result<(), Error> {
> - let path: PathBuf = datastore.path.clone().into();
> + let path: PathBuf = datastore.absolute_path().into();
> + let backup_user = pbs_config::backup_user()?;
> + if let Some(store_mount_point) = datastore.get_mount_point() {
> + let default_options = proxmox_sys::fs::CreateOptions::new();
> + proxmox_sys::fs::create_path(
> + store_mount_point,
> + Some(default_options.clone()),
> + Some(default_options.clone()),
> + )?;
> + do_mount_device(datastore.clone())?;
> +
> + if path.join(".chunks").is_dir() {
> + config.set_data(&datastore.name, "datastore", &datastore)?;
> + pbs_config::datastore::save_config(&config)?;
> + jobstate::create_state_file("garbage_collection", &datastore.name)?;
> + info!("created removable datastore, chunkstore already exists");
> + return Ok(());
> + }
> + }
>
> let tuning: DatastoreTuning = serde_json::from_value(
> DatastoreTuning::API_SCHEMA
> .parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
> )?;
> - let backup_user = pbs_config::backup_user()?;
why is this line moved?
> let _store = ChunkStore::create(
> &datastore.name,
> path,
> @@ -120,6 +138,17 @@ pub fn create_datastore(
> param_bail!("name", "datastore '{}' already exists.", config.name);
> }
>
> + if let Some(uuid) = &config.backing_device {
> + for (datastore_name, (_, config)) in §ion_config.sections {
> + if config["backing-device"].as_str() == Some(uuid) {
> + param_bail!(
> + "backing-device",
> + "device already in use by datastore '{datastore_name}'",
> + );
that's a pretty harsh limitation.. that could be easily lifted? since
datastores are the trust boundary, having more than one on a single
removable device can make a lot of sense (e.g., for cold storage of
multiple non-removable datastores).
we actually only require path+UUID to be unique (well, that and path not
being nested for each unique UUID, but that is also true for
non-removable datastores and IIRC we don't check it there anywhere
either ;)).
> + }
> + }
> + }
> +
> let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 15:15 ` Hannes Laimer
0 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 15:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > src/api2/config/datastore.rs | 37 ++++++++++++++++++++++++++++++++----
> > 1 file changed, 33 insertions(+), 4 deletions(-)
> >
> > diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
> > index ca6edf05..c24b6e9d 100644
> > --- a/src/api2/config/datastore.rs
> > +++ b/src/api2/config/datastore.rs
> > @@ -4,7 +4,7 @@ use ::serde::{Deserialize, Serialize};
> > use anyhow::Error;
> > use hex::FromHex;
> > use serde_json::Value;
> > -use tracing::warn;
> > +use tracing::{info, warn};
> >
> > use proxmox_router::{http_bail, Permission, Router, RpcEnvironment, RpcEnvironmentType};
> > use proxmox_schema::{api, param_bail, ApiType};
> > @@ -20,7 +20,8 @@ use pbs_config::BackupLockGuard;
> > use pbs_datastore::chunk_store::ChunkStore;
> >
> > use crate::api2::admin::{
> > - prune::list_prune_jobs, sync::list_sync_jobs, verify::list_verification_jobs,
> > + datastore::do_mount_device, prune::list_prune_jobs, sync::list_sync_jobs,
> > + verify::list_verification_jobs,
> > };
> > use crate::api2::config::prune::{delete_prune_job, do_create_prune_job};
> > use crate::api2::config::sync::delete_sync_job;
> > @@ -71,13 +72,30 @@ pub(crate) fn do_create_datastore(
> > mut config: SectionConfigData,
> > datastore: DataStoreConfig,
> > ) -> Result<(), Error> {
> > - let path: PathBuf = datastore.path.clone().into();
> > + let path: PathBuf = datastore.absolute_path().into();
> > + let backup_user = pbs_config::backup_user()?;
> > + if let Some(store_mount_point) = datastore.get_mount_point() {
> > + let default_options = proxmox_sys::fs::CreateOptions::new();
> > + proxmox_sys::fs::create_path(
> > + store_mount_point,
> > + Some(default_options.clone()),
> > + Some(default_options.clone()),
> > + )?;
> > + do_mount_device(datastore.clone())?;
> > +
> > + if path.join(".chunks").is_dir() {
> > + config.set_data(&datastore.name, "datastore", &datastore)?;
> > + pbs_config::datastore::save_config(&config)?;
> > + jobstate::create_state_file("garbage_collection", &datastore.name)?;
> > + info!("created removable datastore, chunkstore already exists");
> > + return Ok(());
> > + }
> > + }
> >
> > let tuning: DatastoreTuning = serde_json::from_value(
> > DatastoreTuning::API_SCHEMA
> > .parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
> > )?;
> > - let backup_user = pbs_config::backup_user()?;
>
> why is this line moved?
>
good question, must have missied it from an earlier version
> > let _store = ChunkStore::create(
> > &datastore.name,
> > path,
> > @@ -120,6 +138,17 @@ pub fn create_datastore(
> > param_bail!("name", "datastore '{}' already exists.", config.name);
> > }
> >
> > + if let Some(uuid) = &config.backing_device {
> > + for (datastore_name, (_, config)) in §ion_config.sections {
> > + if config["backing-device"].as_str() == Some(uuid) {
> > + param_bail!(
> > + "backing-device",
> > + "device already in use by datastore '{datastore_name}'",
> > + );
>
> that's a pretty harsh limitation.. that could be easily lifted? since
> datastores are the trust boundary, having more than one on a single
> removable device can make a lot of sense (e.g., for cold storage of
> multiple non-removable datastores).
>
> we actually only require path+UUID to be unique (well, that and path not
> being nested for each unique UUID, but that is also true for
> non-removable datastores and IIRC we don't check it there anywhere
> either ;)).
>
yes, that would make sense, but it wouldn't really have worked with the path
layout from before intrucing bind mounts, this is more of a leftover.
I'll replace this check
> > + }
> > + }
> > + }
> > +
> > let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> > let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> >
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (7 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 08/26] api: removable datastore creation Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
` (17 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
... used by the partition selector for removable datastore creation.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/mod.rs | 8 +++++
src/tools/disks/mod.rs | 60 ++++++++++++++++++++++++++++++++------
2 files changed, 59 insertions(+), 9 deletions(-)
diff --git a/src/api2/node/disks/mod.rs b/src/api2/node/disks/mod.rs
index 4ef4ee2b..5bb7eb94 100644
--- a/src/api2/node/disks/mod.rs
+++ b/src/api2/node/disks/mod.rs
@@ -41,6 +41,12 @@ pub mod zfs;
optional: true,
default: false,
},
+ "exclude-used": {
+ description: "Exclude partitions already used for removable datastores or mounted directories.",
+ type: bool,
+ optional: true,
+ default: false,
+ },
"usage-type": {
type: DiskUsageType,
optional: true,
@@ -62,6 +68,7 @@ pub mod zfs;
pub fn list_disks(
skipsmart: bool,
include_partitions: bool,
+ exclude_used: bool,
usage_type: Option<DiskUsageType>,
) -> Result<Vec<DiskUsageInfo>, Error> {
let mut list = Vec::new();
@@ -69,6 +76,7 @@ pub fn list_disks(
for (_, info) in DiskUsageQuery::new()
.smart(!skipsmart)
.partitions(include_partitions)
+ .exclude_used(exclude_used)
.query()?
{
if let Some(ref usage_type) = usage_type {
diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
index 93c1a882..8d84e312 100644
--- a/src/tools/disks/mod.rs
+++ b/src/tools/disks/mod.rs
@@ -30,6 +30,7 @@ pub use zpool_list::*;
mod lvm;
pub use lvm::*;
mod smart;
+use crate::api2::node::disks::directory::list_datastore_mounts;
pub use smart::*;
static ISCSI_PATH_REGEX: LazyLock<regex::Regex> =
@@ -824,6 +825,7 @@ fn scan_partitions(
pub struct DiskUsageQuery {
smart: bool,
partitions: bool,
+ exclude_used: bool,
}
impl DiskUsageQuery {
@@ -831,6 +833,7 @@ impl DiskUsageQuery {
Self {
smart: true,
partitions: false,
+ exclude_used: false,
}
}
@@ -844,12 +847,22 @@ impl DiskUsageQuery {
self
}
+ pub fn exclude_used(&mut self, exclude_used: bool) -> &mut Self {
+ self.exclude_used = exclude_used;
+ self
+ }
+
pub fn query(&self) -> Result<HashMap<String, DiskUsageInfo>, Error> {
- get_disks(None, !self.smart, self.partitions)
+ get_disks(None, !self.smart, self.partitions, self.exclude_used)
}
pub fn find(&self, disk: &str) -> Result<DiskUsageInfo, Error> {
- let mut map = get_disks(Some(vec![disk.to_string()]), !self.smart, self.partitions)?;
+ let mut map = get_disks(
+ Some(vec![disk.to_string()]),
+ !self.smart,
+ self.partitions,
+ self.exclude_used,
+ )?;
if let Some(info) = map.remove(disk) {
Ok(info)
} else {
@@ -858,7 +871,7 @@ impl DiskUsageQuery {
}
pub fn find_all(&self, disks: Vec<String>) -> Result<HashMap<String, DiskUsageInfo>, Error> {
- get_disks(Some(disks), !self.smart, self.partitions)
+ get_disks(Some(disks), !self.smart, self.partitions, self.exclude_used)
}
}
@@ -931,6 +944,8 @@ fn get_disks(
no_smart: bool,
// include partitions
include_partitions: bool,
+ // skip partitions which are in use
+ exclude_used: bool,
) -> Result<HashMap<String, DiskUsageInfo>, Error> {
let disk_manager = DiskManage::new();
@@ -948,6 +963,31 @@ fn get_disks(
// fixme: ceph journals/volumes
+ let uuids_in_use = if exclude_used && include_partitions {
+ let (config, _digest) = pbs_config::datastore::config()?;
+
+ let uuids_from_datastores: Vec<String> = config
+ .sections
+ .iter()
+ .filter_map(|(_, (_, data))| {
+ data.as_object()
+ .and_then(|cfg| cfg.get("backing-device"))
+ .and_then(serde_json::Value::as_str)
+ .map(String::from)
+ })
+ .collect();
+
+ let uuids_from_mounts: Vec<String> = list_datastore_mounts()?
+ .into_iter()
+ // FIXME: include UUID in DatastoreMountInfo
+ .filter_map(|mount| mount.device.split('/').last().map(String::from))
+ .collect();
+
+ [&uuids_from_datastores[..], &uuids_from_mounts[..]].concat()
+ } else {
+ Vec::new()
+ };
+
let mut result = HashMap::new();
for item in proxmox_sys::fs::scan_subdir(libc::AT_FDCWD, "/sys/block", &BLOCKDEVICE_NAME_REGEX)?
@@ -1020,12 +1060,14 @@ fn get_disks(
let partitions: Option<Vec<PartitionInfo>> = if include_partitions {
disk.partitions().map_or(None, |parts| {
- Some(get_partitions_info(
- parts,
- &lvm_devices,
- &zfs_devices,
- &file_system_devices,
- ))
+ Some(
+ get_partitions_info(parts, &lvm_devices, &zfs_devices, &file_system_devices)
+ .into_iter()
+ .filter(|part| {
+ !part.uuid.as_ref().is_some_and(|u| uuids_in_use.contains(u))
+ })
+ .collect(),
+ )
})
} else {
None
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 15:17 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> ... used by the partition selector for removable datastore creation.
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/api2/node/disks/mod.rs | 8 +++++
> src/tools/disks/mod.rs | 60 ++++++++++++++++++++++++++++++++------
> 2 files changed, 59 insertions(+), 9 deletions(-)
>
> diff --git a/src/api2/node/disks/mod.rs b/src/api2/node/disks/mod.rs
> index 4ef4ee2b..5bb7eb94 100644
> --- a/src/api2/node/disks/mod.rs
> +++ b/src/api2/node/disks/mod.rs
> @@ -41,6 +41,12 @@ pub mod zfs;
> optional: true,
> default: false,
> },
> + "exclude-used": {
> + description: "Exclude partitions already used for removable datastores or mounted directories.",
see comment below..
> + type: bool,
> + optional: true,
> + default: false,
> + },
> "usage-type": {
> type: DiskUsageType,
> optional: true,
> @@ -62,6 +68,7 @@ pub mod zfs;
> pub fn list_disks(
> skipsmart: bool,
> include_partitions: bool,
> + exclude_used: bool,
> usage_type: Option<DiskUsageType>,
> ) -> Result<Vec<DiskUsageInfo>, Error> {
> let mut list = Vec::new();
> @@ -69,6 +76,7 @@ pub fn list_disks(
> for (_, info) in DiskUsageQuery::new()
> .smart(!skipsmart)
> .partitions(include_partitions)
> + .exclude_used(exclude_used)
see comment below..
> .query()?
> {
> if let Some(ref usage_type) = usage_type {
> diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
> index 93c1a882..8d84e312 100644
> --- a/src/tools/disks/mod.rs
> +++ b/src/tools/disks/mod.rs
> @@ -30,6 +30,7 @@ pub use zpool_list::*;
> mod lvm;
> pub use lvm::*;
> mod smart;
> +use crate::api2::node::disks::directory::list_datastore_mounts;
nit: wrong sorting of use statements, but also wrong direction of usage?
tools shouldn't use api code, but the other way round..
it would probably be cleaner to instead pass in a list of excluded
devices/mountpoints/..?
> pub use smart::*;
>
> static ISCSI_PATH_REGEX: LazyLock<regex::Regex> =
> @@ -824,6 +825,7 @@ fn scan_partitions(
> pub struct DiskUsageQuery {
> smart: bool,
> partitions: bool,
> + exclude_used: bool,
> }
>
> impl DiskUsageQuery {
> @@ -831,6 +833,7 @@ impl DiskUsageQuery {
> Self {
> smart: true,
> partitions: false,
> + exclude_used: false,
> }
> }
>
> @@ -844,12 +847,22 @@ impl DiskUsageQuery {
> self
> }
>
> + pub fn exclude_used(&mut self, exclude_used: bool) -> &mut Self {
> + self.exclude_used = exclude_used;
> + self
> + }
> +
> pub fn query(&self) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> - get_disks(None, !self.smart, self.partitions)
> + get_disks(None, !self.smart, self.partitions, self.exclude_used)
> }
>
> pub fn find(&self, disk: &str) -> Result<DiskUsageInfo, Error> {
> - let mut map = get_disks(Some(vec![disk.to_string()]), !self.smart, self.partitions)?;
> + let mut map = get_disks(
> + Some(vec![disk.to_string()]),
> + !self.smart,
> + self.partitions,
> + self.exclude_used,
> + )?;
> if let Some(info) = map.remove(disk) {
> Ok(info)
> } else {
> @@ -858,7 +871,7 @@ impl DiskUsageQuery {
> }
>
> pub fn find_all(&self, disks: Vec<String>) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> - get_disks(Some(disks), !self.smart, self.partitions)
> + get_disks(Some(disks), !self.smart, self.partitions, self.exclude_used)
> }
> }
>
> @@ -931,6 +944,8 @@ fn get_disks(
> no_smart: bool,
> // include partitions
> include_partitions: bool,
> + // skip partitions which are in use
> + exclude_used: bool,
> ) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> let disk_manager = DiskManage::new();
>
> @@ -948,6 +963,31 @@ fn get_disks(
>
> // fixme: ceph journals/volumes
>
> + let uuids_in_use = if exclude_used && include_partitions {
this here is tricky, if exclusion only matters for partitions that
should be mentioned somewhere.. but then again, why is that the case?
couldn't a removable datastore be directly on a disk without partitions?
> + let (config, _digest) = pbs_config::datastore::config()?;
> +
> + let uuids_from_datastores: Vec<String> = config
> + .sections
> + .iter()
> + .filter_map(|(_, (_, data))| {
> + data.as_object()
> + .and_then(|cfg| cfg.get("backing-device"))
> + .and_then(serde_json::Value::as_str)
> + .map(String::from)
> + })
> + .collect();
> +
> + let uuids_from_mounts: Vec<String> = list_datastore_mounts()?
> + .into_iter()
> + // FIXME: include UUID in DatastoreMountInfo
> + .filter_map(|mount| mount.device.split('/').last().map(String::from))
> + .collect();
> +
> + [&uuids_from_datastores[..], &uuids_from_mounts[..]].concat()
> + } else {
> + Vec::new()
> + };
i.e., pass in this Vec/slice..
> +
> let mut result = HashMap::new();
>
> for item in proxmox_sys::fs::scan_subdir(libc::AT_FDCWD, "/sys/block", &BLOCKDEVICE_NAME_REGEX)?
> @@ -1020,12 +1060,14 @@ fn get_disks(
>
> let partitions: Option<Vec<PartitionInfo>> = if include_partitions {
> disk.partitions().map_or(None, |parts| {
> - Some(get_partitions_info(
> - parts,
> - &lvm_devices,
> - &zfs_devices,
> - &file_system_devices,
> - ))
> + Some(
> + get_partitions_info(parts, &lvm_devices, &zfs_devices, &file_system_devices)
> + .into_iter()
> + .filter(|part| {
> + !part.uuid.as_ref().is_some_and(|u| uuids_in_use.contains(u))
> + })
> + .collect(),
> + )
> })
> } else {
> None
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 15:17 ` Hannes Laimer
0 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 15:17 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
I think we could even drop this completely, given we allow multiple
datastores on the same device, there isn't really any other reason for
this patch.
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > ... used by the partition selector for removable datastore creation.
> >
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > src/api2/node/disks/mod.rs | 8 +++++
> > src/tools/disks/mod.rs | 60 ++++++++++++++++++++++++++++++++------
> > 2 files changed, 59 insertions(+), 9 deletions(-)
> >
> > diff --git a/src/api2/node/disks/mod.rs b/src/api2/node/disks/mod.rs
> > index 4ef4ee2b..5bb7eb94 100644
> > --- a/src/api2/node/disks/mod.rs
> > +++ b/src/api2/node/disks/mod.rs
> > @@ -41,6 +41,12 @@ pub mod zfs;
> > optional: true,
> > default: false,
> > },
> > + "exclude-used": {
> > + description: "Exclude partitions already used for removable datastores or mounted directories.",
>
> see comment below..
>
> > + type: bool,
> > + optional: true,
> > + default: false,
> > + },
> > "usage-type": {
> > type: DiskUsageType,
> > optional: true,
> > @@ -62,6 +68,7 @@ pub mod zfs;
> > pub fn list_disks(
> > skipsmart: bool,
> > include_partitions: bool,
> > + exclude_used: bool,
> > usage_type: Option<DiskUsageType>,
> > ) -> Result<Vec<DiskUsageInfo>, Error> {
> > let mut list = Vec::new();
> > @@ -69,6 +76,7 @@ pub fn list_disks(
> > for (_, info) in DiskUsageQuery::new()
> > .smart(!skipsmart)
> > .partitions(include_partitions)
> > + .exclude_used(exclude_used)
>
> see comment below..
>
> > .query()?
> > {
> > if let Some(ref usage_type) = usage_type {
> > diff --git a/src/tools/disks/mod.rs b/src/tools/disks/mod.rs
> > index 93c1a882..8d84e312 100644
> > --- a/src/tools/disks/mod.rs
> > +++ b/src/tools/disks/mod.rs
> > @@ -30,6 +30,7 @@ pub use zpool_list::*;
> > mod lvm;
> > pub use lvm::*;
> > mod smart;
> > +use crate::api2::node::disks::directory::list_datastore_mounts;
>
> nit: wrong sorting of use statements, but also wrong direction of usage?
> tools shouldn't use api code, but the other way round..
>
> it would probably be cleaner to instead pass in a list of excluded
> devices/mountpoints/..?
>
> > pub use smart::*;
> >
> > static ISCSI_PATH_REGEX: LazyLock<regex::Regex> =
> > @@ -824,6 +825,7 @@ fn scan_partitions(
> > pub struct DiskUsageQuery {
> > smart: bool,
> > partitions: bool,
> > + exclude_used: bool,
> > }
> >
> > impl DiskUsageQuery {
> > @@ -831,6 +833,7 @@ impl DiskUsageQuery {
> > Self {
> > smart: true,
> > partitions: false,
> > + exclude_used: false,
> > }
> > }
> >
> > @@ -844,12 +847,22 @@ impl DiskUsageQuery {
> > self
> > }
> >
> > + pub fn exclude_used(&mut self, exclude_used: bool) -> &mut Self {
> > + self.exclude_used = exclude_used;
> > + self
> > + }
> > +
> > pub fn query(&self) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> > - get_disks(None, !self.smart, self.partitions)
> > + get_disks(None, !self.smart, self.partitions, self.exclude_used)
> > }
> >
> > pub fn find(&self, disk: &str) -> Result<DiskUsageInfo, Error> {
> > - let mut map = get_disks(Some(vec![disk.to_string()]), !self.smart, self.partitions)?;
> > + let mut map = get_disks(
> > + Some(vec![disk.to_string()]),
> > + !self.smart,
> > + self.partitions,
> > + self.exclude_used,
> > + )?;
> > if let Some(info) = map.remove(disk) {
> > Ok(info)
> > } else {
> > @@ -858,7 +871,7 @@ impl DiskUsageQuery {
> > }
> >
> > pub fn find_all(&self, disks: Vec<String>) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> > - get_disks(Some(disks), !self.smart, self.partitions)
> > + get_disks(Some(disks), !self.smart, self.partitions, self.exclude_used)
> > }
> > }
> >
> > @@ -931,6 +944,8 @@ fn get_disks(
> > no_smart: bool,
> > // include partitions
> > include_partitions: bool,
> > + // skip partitions which are in use
> > + exclude_used: bool,
> > ) -> Result<HashMap<String, DiskUsageInfo>, Error> {
> > let disk_manager = DiskManage::new();
> >
> > @@ -948,6 +963,31 @@ fn get_disks(
> >
> > // fixme: ceph journals/volumes
> >
> > + let uuids_in_use = if exclude_used && include_partitions {
>
> this here is tricky, if exclusion only matters for partitions that
> should be mentioned somewhere.. but then again, why is that the case?
> couldn't a removable datastore be directly on a disk without partitions?
>
> > + let (config, _digest) = pbs_config::datastore::config()?;
> > +
> > + let uuids_from_datastores: Vec<String> = config
> > + .sections
> > + .iter()
> > + .filter_map(|(_, (_, data))| {
> > + data.as_object()
> > + .and_then(|cfg| cfg.get("backing-device"))
> > + .and_then(serde_json::Value::as_str)
> > + .map(String::from)
> > + })
> > + .collect();
> > +
> > + let uuids_from_mounts: Vec<String> = list_datastore_mounts()?
> > + .into_iter()
> > + // FIXME: include UUID in DatastoreMountInfo
> > + .filter_map(|mount| mount.device.split('/').last().map(String::from))
> > + .collect();
> > +
> > + [&uuids_from_datastores[..], &uuids_from_mounts[..]].concat()
> > + } else {
> > + Vec::new()
> > + };
>
> i.e., pass in this Vec/slice..
>
> > +
> > let mut result = HashMap::new();
> >
> > for item in proxmox_sys::fs::scan_subdir(libc::AT_FDCWD, "/sys/block", &BLOCKDEVICE_NAME_REGEX)?
> > @@ -1020,12 +1060,14 @@ fn get_disks(
> >
> > let partitions: Option<Vec<PartitionInfo>> = if include_partitions {
> > disk.partitions().map_or(None, |parts| {
> > - Some(get_partitions_info(
> > - parts,
> > - &lvm_devices,
> > - &zfs_devices,
> > - &file_system_devices,
> > - ))
> > + Some(
> > + get_partitions_info(parts, &lvm_devices, &zfs_devices, &file_system_devices)
> > + .into_iter()
> > + .filter(|part| {
> > + !part.uuid.as_ref().is_some_and(|u| uuids_in_use.contains(u))
> > + })
> > + .collect(),
> > + )
> > })
> > } else {
> > None
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (8 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 09/26] api: disks list: add exclude-used flag Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command Hannes Laimer
` (16 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/datastore.rs | 9 ++++++++-
src/api2/admin/datastore.rs | 17 +++++++++--------
src/api2/status.rs | 18 +++++++++++++++---
3 files changed, 32 insertions(+), 12 deletions(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index d6e2f97e..6a999372 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -454,6 +454,10 @@ impl DataStoreConfig {
pub struct DataStoreListItem {
pub store: String,
pub comment: Option<String>,
+ /// Datastore is removable
+ pub removable: bool,
+ /// Datastore is available
+ pub available: bool,
/// If the datastore is in maintenance mode, information about it
#[serde(skip_serializing_if = "Option::is_none")]
pub maintenance: Option<String>,
@@ -1453,6 +1457,8 @@ pub struct DataStoreStatusListItem {
/// The available bytes of the underlying storage. (-1 on error)
#[serde(skip_serializing_if = "Option::is_none")]
pub avail: Option<u64>,
+ /// The datastore is available, relevant if removable
+ pub is_available: bool,
/// A list of usages of the past (last Month).
#[serde(skip_serializing_if = "Option::is_none")]
pub history: Option<Vec<Option<f64>>>,
@@ -1477,12 +1483,13 @@ pub struct DataStoreStatusListItem {
}
impl DataStoreStatusListItem {
- pub fn empty(store: &str, err: Option<String>) -> Self {
+ pub fn empty(store: &str, err: Option<String>, is_available: bool) -> Self {
DataStoreStatusListItem {
store: store.to_owned(),
total: None,
used: None,
avail: None,
+ is_available,
history: None,
history_start: None,
history_delta: None,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 3c95888d..0b5dede7 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -1310,8 +1310,8 @@ pub fn get_datastore_list(
let mut list = Vec::new();
- for (store, (_, data)) in &config.sections {
- let acl_path = &["datastore", store];
+ for (store, (_, data)) in config.sections {
+ let acl_path = &["datastore", &store];
let user_privs = user_info.lookup_privs(&auth_id, acl_path);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
@@ -1322,15 +1322,16 @@ pub fn get_datastore_list(
}
}
+ let store_config: DataStoreConfig = serde_json::from_value(data)?;
+ let is_available = pbs_datastore::is_datastore_available(&store_config);
+
if allowed || allow_id {
list.push(DataStoreListItem {
store: store.clone(),
- comment: if !allowed {
- None
- } else {
- data["comment"].as_str().map(String::from)
- },
- maintenance: data["maintenance-mode"].as_str().map(String::from),
+ comment: store_config.comment.filter(|_| allowed),
+ removable: store_config.backing_device.is_some(),
+ available: is_available,
+ maintenance: store_config.maintenance_mode,
});
}
}
diff --git a/src/api2/status.rs b/src/api2/status.rs
index f1ae0ef5..a0dd46b6 100644
--- a/src/api2/status.rs
+++ b/src/api2/status.rs
@@ -13,7 +13,7 @@ use pbs_api_types::{
};
use pbs_config::CachedUserInfo;
-use pbs_datastore::DataStore;
+use pbs_datastore::{is_datastore_available, DataStore};
use crate::rrd_cache::extract_rrd_data;
use crate::tools::statistics::linear_regression;
@@ -48,10 +48,17 @@ pub async fn datastore_status(
for (store, (_, _)) in &config.sections {
let user_privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
+
+ let store_config = config.lookup("datastore", store)?;
+ if !is_datastore_available(&store_config) {
+ list.push(DataStoreStatusListItem::empty(store, None, false));
+ continue;
+ }
+
if !allowed {
if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
if can_access_any_namespace(datastore, &auth_id, &user_info) {
- list.push(DataStoreStatusListItem::empty(store, None));
+ list.push(DataStoreStatusListItem::empty(store, None, true));
}
}
continue;
@@ -60,7 +67,11 @@ pub async fn datastore_status(
let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
Ok(datastore) => datastore,
Err(err) => {
- list.push(DataStoreStatusListItem::empty(store, Some(err.to_string())));
+ list.push(DataStoreStatusListItem::empty(
+ store,
+ Some(err.to_string()),
+ true,
+ ));
continue;
}
};
@@ -71,6 +82,7 @@ pub async fn datastore_status(
total: Some(status.total),
used: Some(status.used),
avail: Some(status.available),
+ is_available: true,
history: None,
history_start: None,
history_delta: None,
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 15:21 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-api-types/src/datastore.rs | 9 ++++++++-
> src/api2/admin/datastore.rs | 17 +++++++++--------
> src/api2/status.rs | 18 +++++++++++++++---
> 3 files changed, 32 insertions(+), 12 deletions(-)
>
> diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> index d6e2f97e..6a999372 100644
> --- a/pbs-api-types/src/datastore.rs
> +++ b/pbs-api-types/src/datastore.rs
> @@ -454,6 +454,10 @@ impl DataStoreConfig {
> pub struct DataStoreListItem {
> pub store: String,
> pub comment: Option<String>,
> + /// Datastore is removable
> + pub removable: bool,
> + /// Datastore is available
> + pub available: bool,
> /// If the datastore is in maintenance mode, information about it
> #[serde(skip_serializing_if = "Option::is_none")]
> pub maintenance: Option<String>,
> @@ -1453,6 +1457,8 @@ pub struct DataStoreStatusListItem {
> /// The available bytes of the underlying storage. (-1 on error)
> #[serde(skip_serializing_if = "Option::is_none")]
> pub avail: Option<u64>,
> + /// The datastore is available, relevant if removable
> + pub is_available: bool,
but it doesn't explain what available is.. why is it only relevant if
removable? and wouldn't it then make more sense to make removable an
enum that encodes whether it's available or not?
> /// A list of usages of the past (last Month).
> #[serde(skip_serializing_if = "Option::is_none")]
> pub history: Option<Vec<Option<f64>>>,
> @@ -1477,12 +1483,13 @@ pub struct DataStoreStatusListItem {
> }
>
> impl DataStoreStatusListItem {
> - pub fn empty(store: &str, err: Option<String>) -> Self {
> + pub fn empty(store: &str, err: Option<String>, is_available: bool) -> Self {
> DataStoreStatusListItem {
> store: store.to_owned(),
> total: None,
> used: None,
> avail: None,
> + is_available,
> history: None,
> history_start: None,
> history_delta: None,
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 3c95888d..0b5dede7 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -1310,8 +1310,8 @@ pub fn get_datastore_list(
>
> let mut list = Vec::new();
>
> - for (store, (_, data)) in &config.sections {
> - let acl_path = &["datastore", store];
> + for (store, (_, data)) in config.sections {
> + let acl_path = &["datastore", &store];
> let user_privs = user_info.lookup_privs(&auth_id, acl_path);
> let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
>
> @@ -1322,15 +1322,16 @@ pub fn get_datastore_list(
> }
> }
>
> + let store_config: DataStoreConfig = serde_json::from_value(data)?;
> + let is_available = pbs_datastore::is_datastore_available(&store_config);
> +
> if allowed || allow_id {
> list.push(DataStoreListItem {
> store: store.clone(),
> - comment: if !allowed {
> - None
> - } else {
> - data["comment"].as_str().map(String::from)
> - },
> - maintenance: data["maintenance-mode"].as_str().map(String::from),
> + comment: store_config.comment.filter(|_| allowed),
> + removable: store_config.backing_device.is_some(),
> + available: is_available,
> + maintenance: store_config.maintenance_mode,
> });
> }
> }
> diff --git a/src/api2/status.rs b/src/api2/status.rs
> index f1ae0ef5..a0dd46b6 100644
> --- a/src/api2/status.rs
> +++ b/src/api2/status.rs
> @@ -13,7 +13,7 @@ use pbs_api_types::{
> };
>
> use pbs_config::CachedUserInfo;
> -use pbs_datastore::DataStore;
> +use pbs_datastore::{is_datastore_available, DataStore};
>
> use crate::rrd_cache::extract_rrd_data;
> use crate::tools::statistics::linear_regression;
> @@ -48,10 +48,17 @@ pub async fn datastore_status(
> for (store, (_, _)) in &config.sections {
> let user_privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
> let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
> +
> + let store_config = config.lookup("datastore", store)?;
> + if !is_datastore_available(&store_config) {
> + list.push(DataStoreStatusListItem::empty(store, None, false));
> + continue;
> + }
> +
> if !allowed {
> if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
> if can_access_any_namespace(datastore, &auth_id, &user_info) {
> - list.push(DataStoreStatusListItem::empty(store, None));
> + list.push(DataStoreStatusListItem::empty(store, None, true));
> }
> }
> continue;
> @@ -60,7 +67,11 @@ pub async fn datastore_status(
> let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
> Ok(datastore) => datastore,
> Err(err) => {
> - list.push(DataStoreStatusListItem::empty(store, Some(err.to_string())));
> + list.push(DataStoreStatusListItem::empty(
> + store,
> + Some(err.to_string()),
> + true,
> + ));
> continue;
> }
> };
> @@ -71,6 +82,7 @@ pub async fn datastore_status(
> total: Some(status.total),
> used: Some(status.used),
> avail: Some(status.available),
> + is_available: true,
> history: None,
> history_start: None,
> history_delta: None,
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 15:21 ` Hannes Laimer
0 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 15:21 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > pbs-api-types/src/datastore.rs | 9 ++++++++-
> > src/api2/admin/datastore.rs | 17 +++++++++--------
> > src/api2/status.rs | 18 +++++++++++++++---
> > 3 files changed, 32 insertions(+), 12 deletions(-)
> >
> > diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
> > index d6e2f97e..6a999372 100644
> > --- a/pbs-api-types/src/datastore.rs
> > +++ b/pbs-api-types/src/datastore.rs
> > @@ -454,6 +454,10 @@ impl DataStoreConfig {
> > pub struct DataStoreListItem {
> > pub store: String,
> > pub comment: Option<String>,
> > + /// Datastore is removable
> > + pub removable: bool,
> > + /// Datastore is available
> > + pub available: bool,
> > /// If the datastore is in maintenance mode, information about it
> > #[serde(skip_serializing_if = "Option::is_none")]
> > pub maintenance: Option<String>,
> > @@ -1453,6 +1457,8 @@ pub struct DataStoreStatusListItem {
> > /// The available bytes of the underlying storage. (-1 on error)
> > #[serde(skip_serializing_if = "Option::is_none")]
> > pub avail: Option<u64>,
> > + /// The datastore is available, relevant if removable
> > + pub is_available: bool,
>
> but it doesn't explain what available is.. why is it only relevant if
> removable? and wouldn't it then make more sense to make removable an
> enum that encodes whether it's available or not?
>
good point, that actually would make sense
> > /// A list of usages of the past (last Month).
> > #[serde(skip_serializing_if = "Option::is_none")]
> > pub history: Option<Vec<Option<f64>>>,
> > @@ -1477,12 +1483,13 @@ pub struct DataStoreStatusListItem {
> > }
> >
> > impl DataStoreStatusListItem {
> > - pub fn empty(store: &str, err: Option<String>) -> Self {
> > + pub fn empty(store: &str, err: Option<String>, is_available: bool) -> Self {
> > DataStoreStatusListItem {
> > store: store.to_owned(),
> > total: None,
> > used: None,
> > avail: None,
> > + is_available,
> > history: None,
> > history_start: None,
> > history_delta: None,
> > diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> > index 3c95888d..0b5dede7 100644
> > --- a/src/api2/admin/datastore.rs
> > +++ b/src/api2/admin/datastore.rs
> > @@ -1310,8 +1310,8 @@ pub fn get_datastore_list(
> >
> > let mut list = Vec::new();
> >
> > - for (store, (_, data)) in &config.sections {
> > - let acl_path = &["datastore", store];
> > + for (store, (_, data)) in config.sections {
> > + let acl_path = &["datastore", &store];
> > let user_privs = user_info.lookup_privs(&auth_id, acl_path);
> > let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
> >
> > @@ -1322,15 +1322,16 @@ pub fn get_datastore_list(
> > }
> > }
> >
> > + let store_config: DataStoreConfig = serde_json::from_value(data)?;
> > + let is_available = pbs_datastore::is_datastore_available(&store_config);
> > +
> > if allowed || allow_id {
> > list.push(DataStoreListItem {
> > store: store.clone(),
> > - comment: if !allowed {
> > - None
> > - } else {
> > - data["comment"].as_str().map(String::from)
> > - },
> > - maintenance: data["maintenance-mode"].as_str().map(String::from),
> > + comment: store_config.comment.filter(|_| allowed),
> > + removable: store_config.backing_device.is_some(),
> > + available: is_available,
> > + maintenance: store_config.maintenance_mode,
> > });
> > }
> > }
> > diff --git a/src/api2/status.rs b/src/api2/status.rs
> > index f1ae0ef5..a0dd46b6 100644
> > --- a/src/api2/status.rs
> > +++ b/src/api2/status.rs
> > @@ -13,7 +13,7 @@ use pbs_api_types::{
> > };
> >
> > use pbs_config::CachedUserInfo;
> > -use pbs_datastore::DataStore;
> > +use pbs_datastore::{is_datastore_available, DataStore};
> >
> > use crate::rrd_cache::extract_rrd_data;
> > use crate::tools::statistics::linear_regression;
> > @@ -48,10 +48,17 @@ pub async fn datastore_status(
> > for (store, (_, _)) in &config.sections {
> > let user_privs = user_info.lookup_privs(&auth_id, &["datastore", store]);
> > let allowed = (user_privs & (PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_BACKUP)) != 0;
> > +
> > + let store_config = config.lookup("datastore", store)?;
> > + if !is_datastore_available(&store_config) {
> > + list.push(DataStoreStatusListItem::empty(store, None, false));
> > + continue;
> > + }
> > +
> > if !allowed {
> > if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
> > if can_access_any_namespace(datastore, &auth_id, &user_info) {
> > - list.push(DataStoreStatusListItem::empty(store, None));
> > + list.push(DataStoreStatusListItem::empty(store, None, true));
> > }
> > }
> > continue;
> > @@ -60,7 +67,11 @@ pub async fn datastore_status(
> > let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
> > Ok(datastore) => datastore,
> > Err(err) => {
> > - list.push(DataStoreStatusListItem::empty(store, Some(err.to_string())));
> > + list.push(DataStoreStatusListItem::empty(
> > + store,
> > + Some(err.to_string()),
> > + true,
> > + ));
> > continue;
> > }
> > };
> > @@ -71,6 +82,7 @@ pub async fn datastore_status(
> > total: Some(status.total),
> > used: Some(status.used),
> > avail: Some(status.available),
> > + is_available: true,
> > history: None,
> > history_start: None,
> > history_delta: None,
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (9 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 10/26] pbs-api-types: add removable/is-available flag to DataStoreListItem Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores Hannes Laimer
` (15 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-config/src/datastore.rs | 14 ++++
src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
2 files changed, 89 insertions(+), 1 deletion(-)
diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
index dc5bb3da..396dcb37 100644
--- a/pbs-config/src/datastore.rs
+++ b/pbs-config/src/datastore.rs
@@ -62,6 +62,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
}
}
+pub fn complete_removable_datastore_name(
+ _arg: &str,
+ _param: &HashMap<String, String>,
+) -> Vec<String> {
+ match config() {
+ Ok((data, _digest)) => data
+ .sections
+ .into_iter()
+ .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
+ .collect(),
+ Err(_) => Vec::new(),
+ }
+}
+
pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
let mut list = vec![
String::from("/"),
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 383bcd24..f2795b39 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::Error;
+use anyhow::{format_err, Error};
use serde_json::Value;
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
Ok(Value::Null)
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Mount a removable datastore.
+async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+ param["node"] = "localhost".into();
+
+ let info = &api2::admin::datastore::API_METHOD_MOUNT;
+ let result = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
+ _ => unreachable!(),
+ };
+
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ Ok(())
+}
+
#[api(
input: {
properties: {
@@ -99,6 +127,34 @@ async fn create_datastore(mut param: Value) -> Result<Value, Error> {
Ok(Value::Null)
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ store: {
+ schema: DATASTORE_SCHEMA,
+ },
+ digest: {
+ optional: true,
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+)]
+/// Unmount a removable datastore.
+async fn unmount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
+ param["node"] = "localhost".into();
+
+ let info = &api2::admin::datastore::API_METHOD_UNMOUNT;
+ let result = match info.handler {
+ ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
+ _ => unreachable!(),
+ };
+
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ Ok(())
+}
+
#[api(
protected: true,
input: {
@@ -142,6 +198,15 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
pub fn datastore_commands() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
.insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
+ .insert(
+ "mount",
+ CliCommand::new(&API_METHOD_MOUNT_DATASTORE)
+ .arg_param(&["store"])
+ .completion_cb(
+ "store",
+ pbs_config::datastore::complete_removable_datastore_name,
+ ),
+ )
.insert(
"show",
CliCommand::new(&API_METHOD_SHOW_DATASTORE)
@@ -152,6 +217,15 @@ pub fn datastore_commands() -> CommandLineInterface {
"create",
CliCommand::new(&API_METHOD_CREATE_DATASTORE).arg_param(&["name", "path"]),
)
+ .insert(
+ "unmount",
+ CliCommand::new(&API_METHOD_UNMOUNT_DATASTORE)
+ .arg_param(&["store"])
+ .completion_cb(
+ "store",
+ pbs_config::datastore::complete_removable_datastore_name,
+ ),
+ )
.insert(
"update",
CliCommand::new(&api2::config::datastore::API_METHOD_UPDATE_DATASTORE)
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 15:40 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-config/src/datastore.rs | 14 ++++
> src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
> 2 files changed, 89 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
> index dc5bb3da..396dcb37 100644
> --- a/pbs-config/src/datastore.rs
> +++ b/pbs-config/src/datastore.rs
> @@ -62,6 +62,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
> }
> }
>
> +pub fn complete_removable_datastore_name(
> + _arg: &str,
> + _param: &HashMap<String, String>,
> +) -> Vec<String> {
> + match config() {
> + Ok((data, _digest)) => data
> + .sections
> + .into_iter()
> + .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
nit: I'd prefer `.and(Some(name))`` instead of mapping just to throw the
value away
> + .collect(),
> + Err(_) => Vec::new(),
> + }
> +}
> +
> pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> let mut list = vec![
> String::from("/"),
> diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> index 383bcd24..f2795b39 100644
> --- a/src/bin/proxmox_backup_manager/datastore.rs
> +++ b/src/bin/proxmox_backup_manager/datastore.rs
> @@ -1,4 +1,4 @@
> -use anyhow::Error;
> +use anyhow::{format_err, Error};
> use serde_json::Value;
>
> use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> @@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
> Ok(Value::Null)
> }
>
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + digest: {
> + optional: true,
> + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> + },
> + },
> + },
> +)]
> +/// Mount a removable datastore.
> +async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
why can't this just delegate directly to the API endpoint? (if there is
a reason, it would be nice to mention such things in the commit
message..)
> + param["node"] = "localhost".into();
> +
> + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> + let result = match info.handler {
> + ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
> + _ => unreachable!(),
> + };
> +
> + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> + Ok(())
> +}
> +
> #[api(
> input: {
> properties: {
> @@ -99,6 +127,34 @@ async fn create_datastore(mut param: Value) -> Result<Value, Error> {
> Ok(Value::Null)
> }
>
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + store: {
> + schema: DATASTORE_SCHEMA,
> + },
> + digest: {
> + optional: true,
> + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> + },
> + },
> + },
> +)]
> +/// Unmount a removable datastore.
> +async fn unmount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
same question here?
> + param["node"] = "localhost".into();
> +
> + let info = &api2::admin::datastore::API_METHOD_UNMOUNT;
> + let result = match info.handler {
> + ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
> + _ => unreachable!(),
> + };
> +
> + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> + Ok(())
> +}
> +
> #[api(
> protected: true,
> input: {
> @@ -142,6 +198,15 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> pub fn datastore_commands() -> CommandLineInterface {
> let cmd_def = CliCommandMap::new()
> .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> + .insert(
> + "mount",
> + CliCommand::new(&API_METHOD_MOUNT_DATASTORE)
> + .arg_param(&["store"])
> + .completion_cb(
> + "store",
> + pbs_config::datastore::complete_removable_datastore_name,
> + ),
> + )
> .insert(
> "show",
> CliCommand::new(&API_METHOD_SHOW_DATASTORE)
> @@ -152,6 +217,15 @@ pub fn datastore_commands() -> CommandLineInterface {
> "create",
> CliCommand::new(&API_METHOD_CREATE_DATASTORE).arg_param(&["name", "path"]),
> )
> + .insert(
> + "unmount",
> + CliCommand::new(&API_METHOD_UNMOUNT_DATASTORE)
> + .arg_param(&["store"])
> + .completion_cb(
> + "store",
> + pbs_config::datastore::complete_removable_datastore_name,
> + ),
> + )
> .insert(
> "update",
> CliCommand::new(&api2::config::datastore::API_METHOD_UPDATE_DATASTORE)
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 15:40 ` Hannes Laimer
2024-10-30 10:13 ` Fabian Grünbichler
0 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 15:40 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > pbs-config/src/datastore.rs | 14 ++++
> > src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
> > 2 files changed, 89 insertions(+), 1 deletion(-)
> >
> > diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
> > index dc5bb3da..396dcb37 100644
> > --- a/pbs-config/src/datastore.rs
> > +++ b/pbs-config/src/datastore.rs
> > @@ -62,6 +62,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
> > }
> > }
> >
> > +pub fn complete_removable_datastore_name(
> > + _arg: &str,
> > + _param: &HashMap<String, String>,
> > +) -> Vec<String> {
> > + match config() {
> > + Ok((data, _digest)) => data
> > + .sections
> > + .into_iter()
> > + .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
>
> nit: I'd prefer `.and(Some(name))`` instead of mapping just to throw the
> value away
>
> > + .collect(),
> > + Err(_) => Vec::new(),
> > + }
> > +}
> > +
> > pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> > let mut list = vec![
> > String::from("/"),
> > diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> > index 383bcd24..f2795b39 100644
> > --- a/src/bin/proxmox_backup_manager/datastore.rs
> > +++ b/src/bin/proxmox_backup_manager/datastore.rs
> > @@ -1,4 +1,4 @@
> > -use anyhow::Error;
> > +use anyhow::{format_err, Error};
> > use serde_json::Value;
> >
> > use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> > @@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
> > Ok(Value::Null)
> > }
> >
> > +#[api(
> > + protected: true,
> > + input: {
> > + properties: {
> > + store: {
> > + schema: DATASTORE_SCHEMA,
> > + },
> > + digest: {
> > + optional: true,
> > + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> > + },
> > + },
> > + },
> > +)]
> > +/// Mount a removable datastore.
> > +async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
>
> why can't this just delegate directly to the API endpoint? (if there is
> a reason, it would be nice to mention such things in the commit
> message..)
>
The mounting is done by a worker, and this worker has its parent (the CLI bin)
killed once it started the worker and finished. [1]
[1] https://lore.proxmox.com/pbs-devel/2bcb9be3-b4cf-450b-8cf1-1e74b49a58f2@proxmox.com/#t
> > + param["node"] = "localhost".into();
> > +
> > + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> > + let result = match info.handler {
> > + ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
> > + _ => unreachable!(),
> > + };
> > +
> > + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> > + Ok(())
> > +}
> > +
> > #[api(
> > input: {
> > properties: {
> > @@ -99,6 +127,34 @@ async fn create_datastore(mut param: Value) -> Result<Value, Error> {
> > Ok(Value::Null)
> > }
> >
> > +#[api(
> > + protected: true,
> > + input: {
> > + properties: {
> > + store: {
> > + schema: DATASTORE_SCHEMA,
> > + },
> > + digest: {
> > + optional: true,
> > + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> > + },
> > + },
> > + },
> > +)]
> > +/// Unmount a removable datastore.
> > +async fn unmount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
>
> same question here?
>
> > + param["node"] = "localhost".into();
> > +
> > + let info = &api2::admin::datastore::API_METHOD_UNMOUNT;
> > + let result = match info.handler {
> > + ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
> > + _ => unreachable!(),
> > + };
> > +
> > + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> > + Ok(())
> > +}
> > +
> > #[api(
> > protected: true,
> > input: {
> > @@ -142,6 +198,15 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> > pub fn datastore_commands() -> CommandLineInterface {
> > let cmd_def = CliCommandMap::new()
> > .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> > + .insert(
> > + "mount",
> > + CliCommand::new(&API_METHOD_MOUNT_DATASTORE)
> > + .arg_param(&["store"])
> > + .completion_cb(
> > + "store",
> > + pbs_config::datastore::complete_removable_datastore_name,
> > + ),
> > + )
> > .insert(
> > "show",
> > CliCommand::new(&API_METHOD_SHOW_DATASTORE)
> > @@ -152,6 +217,15 @@ pub fn datastore_commands() -> CommandLineInterface {
> > "create",
> > CliCommand::new(&API_METHOD_CREATE_DATASTORE).arg_param(&["name", "path"]),
> > )
> > + .insert(
> > + "unmount",
> > + CliCommand::new(&API_METHOD_UNMOUNT_DATASTORE)
> > + .arg_param(&["store"])
> > + .completion_cb(
> > + "store",
> > + pbs_config::datastore::complete_removable_datastore_name,
> > + ),
> > + )
> > .insert(
> > "update",
> > CliCommand::new(&api2::config::datastore::API_METHOD_UPDATE_DATASTORE)
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command
2024-10-29 15:40 ` Hannes Laimer
@ 2024-10-30 10:13 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-30 10:13 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
Quoting Hannes Laimer (2024-10-29 16:40:22)
> On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> > On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > > ---
> > > pbs-config/src/datastore.rs | 14 ++++
> > > src/bin/proxmox_backup_manager/datastore.rs | 76 ++++++++++++++++++++-
> > > 2 files changed, 89 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/pbs-config/src/datastore.rs b/pbs-config/src/datastore.rs
> > > index dc5bb3da..396dcb37 100644
> > > --- a/pbs-config/src/datastore.rs
> > > +++ b/pbs-config/src/datastore.rs
> > > @@ -62,6 +62,20 @@ pub fn complete_datastore_name(_arg: &str, _param: &HashMap<String, String>) ->
> > > }
> > > }
> > >
> > > +pub fn complete_removable_datastore_name(
> > > + _arg: &str,
> > > + _param: &HashMap<String, String>,
> > > +) -> Vec<String> {
> > > + match config() {
> > > + Ok((data, _digest)) => data
> > > + .sections
> > > + .into_iter()
> > > + .filter_map(|(name, (_, c))| c.get("backing-device").map(|_| name))
> >
> > nit: I'd prefer `.and(Some(name))`` instead of mapping just to throw the
> > value away
> >
> > > + .collect(),
> > > + Err(_) => Vec::new(),
> > > + }
> > > +}
> > > +
> > > pub fn complete_acl_path(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> > > let mut list = vec![
> > > String::from("/"),
> > > diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> > > index 383bcd24..f2795b39 100644
> > > --- a/src/bin/proxmox_backup_manager/datastore.rs
> > > +++ b/src/bin/proxmox_backup_manager/datastore.rs
> > > @@ -1,4 +1,4 @@
> > > -use anyhow::Error;
> > > +use anyhow::{format_err, Error};
> > > use serde_json::Value;
> > >
> > > use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> > > @@ -40,6 +40,34 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
> > > Ok(Value::Null)
> > > }
> > >
> > > +#[api(
> > > + protected: true,
> > > + input: {
> > > + properties: {
> > > + store: {
> > > + schema: DATASTORE_SCHEMA,
> > > + },
> > > + digest: {
> > > + optional: true,
> > > + schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
> > > + },
> > > + },
> > > + },
> > > +)]
> > > +/// Mount a removable datastore.
> > > +async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
> >
> > why can't this just delegate directly to the API endpoint? (if there is
> > a reason, it would be nice to mention such things in the commit
> > message..)
> >
>
> The mounting is done by a worker, and this worker has its parent (the CLI bin)
> killed once it started the worker and finished. [1]
>
> [1] https://lore.proxmox.com/pbs-devel/2bcb9be3-b4cf-450b-8cf1-1e74b49a58f2@proxmox.com/#t
ah, we ran into that recently with some other datastore maintenance related
task.
seems like this could/should be fixed in the CLI handler? but in the meantime,
if you need to workaround such limitations note them somewhere please..
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (10 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 11/26] bin: manager: add (un)mount command Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
` (14 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
debian/proxmox-backup-server.install | 1 +
debian/proxmox-backup-server.udev | 3 ++
etc/Makefile | 3 +-
etc/removable-device-attach@.service.in | 8 +++
src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
5 files changed, 68 insertions(+), 2 deletions(-)
create mode 100644 etc/removable-device-attach@.service.in
diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
index ef1e9ba1..34bfc454 100644
--- a/debian/proxmox-backup-server.install
+++ b/debian/proxmox-backup-server.install
@@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
etc/proxmox-backup-daily-update.timer /lib/systemd/system/
etc/proxmox-backup-proxy.service /lib/systemd/system/
etc/proxmox-backup.service /lib/systemd/system/
+etc/removable-device-attach@.service /lib/systemd/system/
usr/bin/pmt
usr/bin/pmtx
usr/bin/proxmox-tape
diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
index afdfb2bc..e21b8bc7 100644
--- a/debian/proxmox-backup-server.udev
+++ b/debian/proxmox-backup-server.udev
@@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
LABEL="persistent_storage_tape_end"
+
+# triggers the mounting of a removable device
+ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
\ No newline at end of file
diff --git a/etc/Makefile b/etc/Makefile
index 42f639f6..730de4f8 100644
--- a/etc/Makefile
+++ b/etc/Makefile
@@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
proxmox-backup-banner.service \
proxmox-backup-daily-update.service \
proxmox-backup.service \
- proxmox-backup-proxy.service
+ proxmox-backup-proxy.service \
+ removable-device-attach@.service
all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
new file mode 100644
index 00000000..e10d1ea3
--- /dev/null
+++ b/etc/removable-device-attach@.service.in
@@ -0,0 +1,8 @@
+[Unit]
+Description=Try to mount the removable device of a datastore with uuid '%i'.
+After=proxmox-backup-proxy.service
+Requires=proxmox-backup-proxy.service
+
+[Service]
+Type=simple
+ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index f2795b39..e74c8537 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -1,4 +1,4 @@
-use anyhow::{format_err, Error};
+use anyhow::{bail, format_err, Error};
use serde_json::Value;
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
@@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
Ok(())
}
+#[api(
+ protected: true,
+ input: {
+ properties: {
+ uuid: {
+ type: String,
+ description: "The UUID of the device that should be mounted",
+ },
+ "output-format": {
+ schema: OUTPUT_FORMAT,
+ optional: true,
+ },
+ },
+ },
+)]
+/// Try mounting a removable datastore given the UUID.
+async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let uuid = param["uuid"]
+ .as_str()
+ .ok_or_else(|| format_err!("uuid has to be specified"))?;
+
+ let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
+ let data: Value = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
+ _ => unreachable!(),
+ };
+
+ let store_name = data.as_array().and_then(|list| {
+ list.iter()
+ .filter_map(Value::as_object)
+ .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
+ .and_then(|s| s.get("name").and_then(Value::as_str))
+ });
+
+ if let Some(store_name) = store_name {
+ let info = &api2::admin::datastore::API_METHOD_MOUNT;
+ let mount_param = serde_json::json!({
+ "store": store_name,
+ });
+ let result = match info.handler {
+ ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
+ _ => unreachable!(),
+ };
+ crate::wait_for_local_worker(result.as_str().unwrap()).await?;
+ return Ok(Value::Null);
+ }
+ bail!("'{uuid}' is not associated with any datastore")
+}
+
pub fn datastore_commands() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
.insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
@@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
pbs_config::datastore::complete_calendar_event,
),
)
+ .insert(
+ "uuid-mount",
+ CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
+ )
.insert(
"remove",
CliCommand::new(&API_METHOD_DELETE_DATASTORE)
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 15:52 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> debian/proxmox-backup-server.install | 1 +
> debian/proxmox-backup-server.udev | 3 ++
> etc/Makefile | 3 +-
> etc/removable-device-attach@.service.in | 8 +++
> src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
> 5 files changed, 68 insertions(+), 2 deletions(-)
> create mode 100644 etc/removable-device-attach@.service.in
>
> diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
> index ef1e9ba1..34bfc454 100644
> --- a/debian/proxmox-backup-server.install
> +++ b/debian/proxmox-backup-server.install
> @@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
> etc/proxmox-backup-daily-update.timer /lib/systemd/system/
> etc/proxmox-backup-proxy.service /lib/systemd/system/
> etc/proxmox-backup.service /lib/systemd/system/
> +etc/removable-device-attach@.service /lib/systemd/system/
> usr/bin/pmt
> usr/bin/pmtx
> usr/bin/proxmox-tape
> diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
> index afdfb2bc..e21b8bc7 100644
> --- a/debian/proxmox-backup-server.udev
> +++ b/debian/proxmox-backup-server.udev
> @@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
> SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
>
> LABEL="persistent_storage_tape_end"
> +
> +# triggers the mounting of a removable device
> +ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
so this adds a Wants on our new unit for all block devices with an FS
UUID set?
> \ No newline at end of file
> diff --git a/etc/Makefile b/etc/Makefile
> index 42f639f6..730de4f8 100644
> --- a/etc/Makefile
> +++ b/etc/Makefile
> @@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
> proxmox-backup-banner.service \
> proxmox-backup-daily-update.service \
> proxmox-backup.service \
> - proxmox-backup-proxy.service
> + proxmox-backup-proxy.service \
> + removable-device-attach@.service
>
> all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
>
> diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
> new file mode 100644
> index 00000000..e10d1ea3
> --- /dev/null
> +++ b/etc/removable-device-attach@.service.in
why is this `.in` / a dynamic unit?
> @@ -0,0 +1,8 @@
> +[Unit]
> +Description=Try to mount the removable device of a datastore with uuid '%i'.
> +After=proxmox-backup-proxy.service
> +Requires=proxmox-backup-proxy.service
> +
> +[Service]
> +Type=simple
> +ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
does this really need to be a custom command?
> diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> index f2795b39..e74c8537 100644
> --- a/src/bin/proxmox_backup_manager/datastore.rs
> +++ b/src/bin/proxmox_backup_manager/datastore.rs
> @@ -1,4 +1,4 @@
> -use anyhow::{format_err, Error};
> +use anyhow::{bail, format_err, Error};
> use serde_json::Value;
>
> use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> @@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> Ok(())
> }
>
> +#[api(
> + protected: true,
> + input: {
> + properties: {
> + uuid: {
> + type: String,
> + description: "The UUID of the device that should be mounted",
> + },
> + "output-format": {
> + schema: OUTPUT_FORMAT,
> + optional: true,
> + },
> + },
> + },
> +)]
> +/// Try mounting a removable datastore given the UUID.
> +async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> + let uuid = param["uuid"]
> + .as_str()
> + .ok_or_else(|| format_err!("uuid has to be specified"))?;
> +
> + let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
> + let data: Value = match info.handler {
> + ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
> + _ => unreachable!(),
> + };
> +
> + let store_name = data.as_array().and_then(|list| {
> + list.iter()
> + .filter_map(Value::as_object)
> + .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
> + .and_then(|s| s.get("name").and_then(Value::as_str))
> + });
wouldn't it be easier to just call
let config = pbs_config::datastore::config()?;
if let Some(store) = config.convert_to_typed_array("datastore")?.into_iter().find(...) {
... store.name
}
if we stay with this approach?
> +
> + if let Some(store_name) = store_name {
> + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> + let mount_param = serde_json::json!({
> + "store": store_name,
> + });
> + let result = match info.handler {
> + ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
> + _ => unreachable!(),
> + };
> + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> + return Ok(Value::Null);
> + }
> + bail!("'{uuid}' is not associated with any datastore")
if this triggers for all block devices, isn't it kinda noisy? or am I
missing some point where additional filtering takes place..
> +}
> +
> pub fn datastore_commands() -> CommandLineInterface {
> let cmd_def = CliCommandMap::new()
> .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> @@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
> pbs_config::datastore::complete_calendar_event,
> ),
> )
> + .insert(
> + "uuid-mount",
> + CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
> + )
> .insert(
> "remove",
> CliCommand::new(&API_METHOD_DELETE_DATASTORE)
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 15:52 ` Hannes Laimer
2024-10-30 10:21 ` Fabian Grünbichler
0 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 15:52 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > debian/proxmox-backup-server.install | 1 +
> > debian/proxmox-backup-server.udev | 3 ++
> > etc/Makefile | 3 +-
> > etc/removable-device-attach@.service.in | 8 +++
> > src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
> > 5 files changed, 68 insertions(+), 2 deletions(-)
> > create mode 100644 etc/removable-device-attach@.service.in
> >
> > diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
> > index ef1e9ba1..34bfc454 100644
> > --- a/debian/proxmox-backup-server.install
> > +++ b/debian/proxmox-backup-server.install
> > @@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
> > etc/proxmox-backup-daily-update.timer /lib/systemd/system/
> > etc/proxmox-backup-proxy.service /lib/systemd/system/
> > etc/proxmox-backup.service /lib/systemd/system/
> > +etc/removable-device-attach@.service /lib/systemd/system/
> > usr/bin/pmt
> > usr/bin/pmtx
> > usr/bin/proxmox-tape
> > diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
> > index afdfb2bc..e21b8bc7 100644
> > --- a/debian/proxmox-backup-server.udev
> > +++ b/debian/proxmox-backup-server.udev
> > @@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
> > SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
> >
> > LABEL="persistent_storage_tape_end"
> > +
> > +# triggers the mounting of a removable device
> > +ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
>
> so this adds a Wants on our new unit for all block devices with an FS
> UUID set?
>
yes, there is not really another way to do this. At least I couldn't
come up with one, I'm open to suggestions tough
> > \ No newline at end of file
> > diff --git a/etc/Makefile b/etc/Makefile
> > index 42f639f6..730de4f8 100644
> > --- a/etc/Makefile
> > +++ b/etc/Makefile
> > @@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
> > proxmox-backup-banner.service \
> > proxmox-backup-daily-update.service \
> > proxmox-backup.service \
> > - proxmox-backup-proxy.service
> > + proxmox-backup-proxy.service \
> > + removable-device-attach@.service
> >
> > all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
> >
> > diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
> > new file mode 100644
> > index 00000000..e10d1ea3
> > --- /dev/null
> > +++ b/etc/removable-device-attach@.service.in
>
> why is this `.in` / a dynamic unit?
>
This is mostly because I wasn't sure where else to put it, where should
it be/would you put it?
> > @@ -0,0 +1,8 @@
> > +[Unit]
> > +Description=Try to mount the removable device of a datastore with uuid '%i'.
> > +After=proxmox-backup-proxy.service
> > +Requires=proxmox-backup-proxy.service
> > +
> > +[Service]
> > +Type=simple
> > +ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
>
> does this really need to be a custom command?
>
Is there a good alternative? It does use the normal mount endpoint after
finding tge datastore(s), the UUID belongs to.
Actually, it only mounts the first, I feel like this is not what we
want. Either none or all? I'm actually not sure what we want here.
> > diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> > index f2795b39..e74c8537 100644
> > --- a/src/bin/proxmox_backup_manager/datastore.rs
> > +++ b/src/bin/proxmox_backup_manager/datastore.rs
> > @@ -1,4 +1,4 @@
> > -use anyhow::{format_err, Error};
> > +use anyhow::{bail, format_err, Error};
> > use serde_json::Value;
> >
> > use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> > @@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> > Ok(())
> > }
> >
> > +#[api(
> > + protected: true,
> > + input: {
> > + properties: {
> > + uuid: {
> > + type: String,
> > + description: "The UUID of the device that should be mounted",
> > + },
> > + "output-format": {
> > + schema: OUTPUT_FORMAT,
> > + optional: true,
> > + },
> > + },
> > + },
> > +)]
> > +/// Try mounting a removable datastore given the UUID.
> > +async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> > + let uuid = param["uuid"]
> > + .as_str()
> > + .ok_or_else(|| format_err!("uuid has to be specified"))?;
> > +
> > + let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
> > + let data: Value = match info.handler {
> > + ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
> > + _ => unreachable!(),
> > + };
> > +
> > + let store_name = data.as_array().and_then(|list| {
> > + list.iter()
> > + .filter_map(Value::as_object)
> > + .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
> > + .and_then(|s| s.get("name").and_then(Value::as_str))
> > + });
>
> wouldn't it be easier to just call
>
> let config = pbs_config::datastore::config()?;
> if let Some(store) = config.convert_to_typed_array("datastore")?.into_iter().find(...) {
> ... store.name
well, yes :)
> }
>
> if we stay with this approach?
>
> > +
> > + if let Some(store_name) = store_name {
> > + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> > + let mount_param = serde_json::json!({
> > + "store": store_name,
> > + });
> > + let result = match info.handler {
> > + ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
> > + _ => unreachable!(),
> > + };
> > + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> > + return Ok(Value::Null);
> > + }
> > + bail!("'{uuid}' is not associated with any datastore")
>
> if this triggers for all block devices, isn't it kinda noisy? or am I
> missing some point where additional filtering takes place..
>
it is, relatively, depending on how often things are plugged in. It might
make sense to drop this logging (before having found a matching ds) all together?
> > +}
> > +
> > pub fn datastore_commands() -> CommandLineInterface {
> > let cmd_def = CliCommandMap::new()
> > .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> > @@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
> > pbs_config::datastore::complete_calendar_event,
> > ),
> > )
> > + .insert(
> > + "uuid-mount",
> > + CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
> > + )
> > .insert(
> > "remove",
> > CliCommand::new(&API_METHOD_DELETE_DATASTORE)
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores
2024-10-29 15:52 ` Hannes Laimer
@ 2024-10-30 10:21 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-30 10:21 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
Quoting Hannes Laimer (2024-10-29 16:52:41)
> On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> > On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > > ---
> > > debian/proxmox-backup-server.install | 1 +
> > > debian/proxmox-backup-server.udev | 3 ++
> > > etc/Makefile | 3 +-
> > > etc/removable-device-attach@.service.in | 8 +++
> > > src/bin/proxmox_backup_manager/datastore.rs | 55 ++++++++++++++++++++-
> > > 5 files changed, 68 insertions(+), 2 deletions(-)
> > > create mode 100644 etc/removable-device-attach@.service.in
> > >
> > > diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
> > > index ef1e9ba1..34bfc454 100644
> > > --- a/debian/proxmox-backup-server.install
> > > +++ b/debian/proxmox-backup-server.install
> > > @@ -4,6 +4,7 @@ etc/proxmox-backup-daily-update.service /lib/systemd/system/
> > > etc/proxmox-backup-daily-update.timer /lib/systemd/system/
> > > etc/proxmox-backup-proxy.service /lib/systemd/system/
> > > etc/proxmox-backup.service /lib/systemd/system/
> > > +etc/removable-device-attach@.service /lib/systemd/system/
> > > usr/bin/pmt
> > > usr/bin/pmtx
> > > usr/bin/proxmox-tape
> > > diff --git a/debian/proxmox-backup-server.udev b/debian/proxmox-backup-server.udev
> > > index afdfb2bc..e21b8bc7 100644
> > > --- a/debian/proxmox-backup-server.udev
> > > +++ b/debian/proxmox-backup-server.udev
> > > @@ -16,3 +16,6 @@ SUBSYSTEM=="scsi_generic", SUBSYSTEMS=="scsi", ATTRS{type}=="1", ENV{ID_SCSI_SER
> > > SYMLINK+="tape/by-id/scsi-$env{ID_SCSI_SERIAL}-sg"
> > >
> > > LABEL="persistent_storage_tape_end"
> > > +
> > > +# triggers the mounting of a removable device
> > > +ACTION=="add", SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", TAG+="systemd", ENV{SYSTEMD_WANTS}="removable-device-attach@$env{ID_FS_UUID}"
> >
> > so this adds a Wants on our new unit for all block devices with an FS
> > UUID set?
> >
>
> yes, there is not really another way to do this. At least I couldn't
> come up with one, I'm open to suggestions tough
could execute a tiny helper binary that maps the UUID to a datastore name..
there's plenty of precedent for this (e.g., it's how zvols and rbd volumes get
their readable names/links).
could even create a symlink in /dev/disk/by-id then ;)
pbs-datastore-name-$NAME
> > > \ No newline at end of file
> > > diff --git a/etc/Makefile b/etc/Makefile
> > > index 42f639f6..730de4f8 100644
> > > --- a/etc/Makefile
> > > +++ b/etc/Makefile
> > > @@ -7,7 +7,8 @@ DYNAMIC_UNITS := \
> > > proxmox-backup-banner.service \
> > > proxmox-backup-daily-update.service \
> > > proxmox-backup.service \
> > > - proxmox-backup-proxy.service
> > > + proxmox-backup-proxy.service \
> > > + removable-device-attach@.service
> > >
> > > all: $(UNITS) $(DYNAMIC_UNITS) pbs-enterprise.list
> > >
> > > diff --git a/etc/removable-device-attach@.service.in b/etc/removable-device-attach@.service.in
> > > new file mode 100644
> > > index 00000000..e10d1ea3
> > > --- /dev/null
> > > +++ b/etc/removable-device-attach@.service.in
> >
> > why is this `.in` / a dynamic unit?
> >
>
> This is mostly because I wasn't sure where else to put it, where should
> it be/would you put it?
just ship the file as it is, without making it a "dynamic unit" that doesn't
contain anything that is replaced and is 100% static? ;)
>
> > > @@ -0,0 +1,8 @@
> > > +[Unit]
> > > +Description=Try to mount the removable device of a datastore with uuid '%i'.
> > > +After=proxmox-backup-proxy.service
> > > +Requires=proxmox-backup-proxy.service
> > > +
> > > +[Service]
> > > +Type=simple
> > > +ExecStart=/usr/sbin/proxmox-backup-manager datastore uuid-mount %i
> >
> > does this really need to be a custom command?
> >
>
> Is there a good alternative? It does use the normal mount endpoint after
> finding tge datastore(s), the UUID belongs to.
>
> Actually, it only mounts the first, I feel like this is not what we
> want. Either none or all? I'm actually not sure what we want here.
if the unit is templated via the datastore name, then this issue would disappear as well..
>
> > > diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> > > index f2795b39..e74c8537 100644
> > > --- a/src/bin/proxmox_backup_manager/datastore.rs
> > > +++ b/src/bin/proxmox_backup_manager/datastore.rs
> > > @@ -1,4 +1,4 @@
> > > -use anyhow::{format_err, Error};
> > > +use anyhow::{bail, format_err, Error};
> > > use serde_json::Value;
> > >
> > > use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> > > @@ -195,6 +195,55 @@ async fn delete_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) ->
> > > Ok(())
> > > }
> > >
> > > +#[api(
> > > + protected: true,
> > > + input: {
> > > + properties: {
> > > + uuid: {
> > > + type: String,
> > > + description: "The UUID of the device that should be mounted",
> > > + },
> > > + "output-format": {
> > > + schema: OUTPUT_FORMAT,
> > > + optional: true,
> > > + },
> > > + },
> > > + },
> > > +)]
> > > +/// Try mounting a removable datastore given the UUID.
> > > +async fn uuid_mount(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
> > > + let uuid = param["uuid"]
> > > + .as_str()
> > > + .ok_or_else(|| format_err!("uuid has to be specified"))?;
> > > +
> > > + let info = &api2::config::datastore::API_METHOD_LIST_DATASTORES;
> > > + let data: Value = match info.handler {
> > > + ApiHandler::Sync(handler) => (handler)(serde_json::json!({}), info, rpcenv)?,
> > > + _ => unreachable!(),
> > > + };
> > > +
> > > + let store_name = data.as_array().and_then(|list| {
> > > + list.iter()
> > > + .filter_map(Value::as_object)
> > > + .find(|store| store.get("backing-device").map_or(false, |d| d.eq(&uuid)))
> > > + .and_then(|s| s.get("name").and_then(Value::as_str))
> > > + });
> >
> > wouldn't it be easier to just call
> >
> > let config = pbs_config::datastore::config()?;
> > if let Some(store) = config.convert_to_typed_array("datastore")?.into_iter().find(...) {
> > ... store.name
>
> well, yes :)
>
> > }
> >
> > if we stay with this approach?
> >
> > > +
> > > + if let Some(store_name) = store_name {
> > > + let info = &api2::admin::datastore::API_METHOD_MOUNT;
> > > + let mount_param = serde_json::json!({
> > > + "store": store_name,
> > > + });
> > > + let result = match info.handler {
> > > + ApiHandler::Sync(handler) => (handler)(mount_param, info, rpcenv)?,
> > > + _ => unreachable!(),
> > > + };
> > > + crate::wait_for_local_worker(result.as_str().unwrap()).await?;
> > > + return Ok(Value::Null);
> > > + }
> > > + bail!("'{uuid}' is not associated with any datastore")
> >
> > if this triggers for all block devices, isn't it kinda noisy? or am I
> > missing some point where additional filtering takes place..
> >
>
> it is, relatively, depending on how often things are plugged in. It might
> make sense to drop this logging (before having found a matching ds) all together?
see above
>
> > > +}
> > > +
> > > pub fn datastore_commands() -> CommandLineInterface {
> > > let cmd_def = CliCommandMap::new()
> > > .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
> > > @@ -240,6 +289,10 @@ pub fn datastore_commands() -> CommandLineInterface {
> > > pbs_config::datastore::complete_calendar_event,
> > > ),
> > > )
> > > + .insert(
> > > + "uuid-mount",
> > > + CliCommand::new(&API_METHOD_UUID_MOUNT).arg_param(&["uuid"]),
> > > + )
> > > .insert(
> > > "remove",
> > > CliCommand::new(&API_METHOD_DELETE_DATASTORE)
> > > --
> > > 2.39.2
> > >
> > >
> > >
> > > _______________________________________________
> > > pbs-devel mailing list
> > > pbs-devel@lists.proxmox.com
> > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> > >
> > >
> > >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (11 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 12/26] add auto-mounting for removable datastores Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section Hannes Laimer
` (13 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-datastore/src/datastore.rs | 4 +++-
src/api2/config/datastore.rs | 32 ++++++++++++++++++++++++++++++++
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 29f98b37..df038d62 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1510,7 +1510,9 @@ impl DataStore {
// weird, but ok
}
Err(err) if err.is_errno(nix::errno::Errno::EBUSY) => {
- warn!("Cannot delete datastore directory (is it a mount point?).")
+ if datastore_config.backing_device.is_none() {
+ warn!("Cannot delete datastore directory (is it a mount point?).")
+ }
}
Err(err) if err.is_errno(nix::errno::Errno::ENOTEMPTY) => {
warn!("Datastore directory not empty, not deleting.")
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index c24b6e9d..c7d59c44 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -29,9 +29,11 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
use crate::api2::config::verify::delete_verification_job;
use pbs_config::CachedUserInfo;
+use pbs_datastore::is_datastore_available;
use proxmox_rest_server::WorkerTask;
use crate::server::jobstate;
+use crate::tools::disks::unmount_by_mountpoint;
#[api(
input: {
@@ -523,6 +525,14 @@ pub async fn delete_datastore(
http_bail!(NOT_FOUND, "datastore '{}' does not exist.", name);
}
+ let store_config: DataStoreConfig = config.lookup("datastore", &name)?;
+ if destroy_data && !is_datastore_available(&store_config) {
+ http_bail!(
+ BAD_REQUEST,
+ "cannot destroy data on '{name}' unless the datastore is mounted"
+ );
+ }
+
if !keep_job_configs {
for job in list_verification_jobs(Some(name.clone()), Value::Null, rpcenv)? {
delete_verification_job(job.config.id, None, rpcenv)?
@@ -549,6 +559,22 @@ pub async fn delete_datastore(
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
+ let name_copy = name.clone();
+ tokio::spawn(async move {
+ if let Ok(proxy_pid) =
+ proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+ {
+ let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
+ let _ = proxmox_daemon::command_socket::send_raw(
+ sock,
+ &format!(
+ "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
+ name_copy
+ ),
+ )
+ .await;
+ }
+ });
let upid = WorkerTask::new_thread(
"delete-datastore",
@@ -561,6 +587,12 @@ pub async fn delete_datastore(
// ignore errors
let _ = jobstate::remove_state_file("prune", &name);
let _ = jobstate::remove_state_file("garbage_collection", &name);
+ if destroy_data {
+ if let Some(mount_point) = store_config.get_mount_point() {
+ let _ = unmount_by_mountpoint(&mount_point);
+ let _ = std::fs::remove_dir(&mount_point);
+ }
+ }
if let Err(err) =
proxmox_async::runtime::block_on(crate::server::notify_datastore_removed())
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
2024-10-29 16:15 ` Hannes Laimer
0 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> pbs-datastore/src/datastore.rs | 4 +++-
> src/api2/config/datastore.rs | 32 ++++++++++++++++++++++++++++++++
> 2 files changed, 35 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 29f98b37..df038d62 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -1510,7 +1510,9 @@ impl DataStore {
> // weird, but ok
> }
> Err(err) if err.is_errno(nix::errno::Errno::EBUSY) => {
> - warn!("Cannot delete datastore directory (is it a mount point?).")
> + if datastore_config.backing_device.is_none() {
> + warn!("Cannot delete datastore directory (is it a mount point?).")
> + }
> }
> Err(err) if err.is_errno(nix::errno::Errno::ENOTEMPTY) => {
> warn!("Datastore directory not empty, not deleting.")
> diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
> index c24b6e9d..c7d59c44 100644
> --- a/src/api2/config/datastore.rs
> +++ b/src/api2/config/datastore.rs
> @@ -29,9 +29,11 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
> use crate::api2::config::verify::delete_verification_job;
> use pbs_config::CachedUserInfo;
>
> +use pbs_datastore::is_datastore_available;
> use proxmox_rest_server::WorkerTask;
>
> use crate::server::jobstate;
> +use crate::tools::disks::unmount_by_mountpoint;
>
> #[api(
> input: {
> @@ -523,6 +525,14 @@ pub async fn delete_datastore(
> http_bail!(NOT_FOUND, "datastore '{}' does not exist.", name);
> }
>
> + let store_config: DataStoreConfig = config.lookup("datastore", &name)?;
> + if destroy_data && !is_datastore_available(&store_config) {
> + http_bail!(
> + BAD_REQUEST,
> + "cannot destroy data on '{name}' unless the datastore is mounted"
> + );
> + }
> +
> if !keep_job_configs {
> for job in list_verification_jobs(Some(name.clone()), Value::Null, rpcenv)? {
> delete_verification_job(job.config.id, None, rpcenv)?
> @@ -549,6 +559,22 @@ pub async fn delete_datastore(
>
> let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> + let name_copy = name.clone();
> + tokio::spawn(async move {
> + if let Ok(proxy_pid) =
> + proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
> + {
> + let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
> + let _ = proxmox_daemon::command_socket::send_raw(
> + sock,
> + &format!(
> + "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
> + name_copy
> + ),
> + )
> + .await;
> + }
> + });
why is this tokio::spawned, but the one in unmount introduced in the
same series is not?
since we now have three call sites with basically the same code, should
we have a helper to do this?
>
> let upid = WorkerTask::new_thread(
> "delete-datastore",
> @@ -561,6 +587,12 @@ pub async fn delete_datastore(
> // ignore errors
> let _ = jobstate::remove_state_file("prune", &name);
> let _ = jobstate::remove_state_file("garbage_collection", &name);
> + if destroy_data {
> + if let Some(mount_point) = store_config.get_mount_point() {
> + let _ = unmount_by_mountpoint(&mount_point);
> + let _ = std::fs::remove_dir(&mount_point);
> + }
> + }
>
> if let Err(err) =
> proxmox_async::runtime::block_on(crate::server::notify_datastore_removed())
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly
2024-10-14 13:42 ` Fabian Grünbichler
@ 2024-10-29 16:15 ` Hannes Laimer
0 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 16:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Mon Oct 14, 2024 at 3:42 PM CEST, Fabian Grünbichler wrote:
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> > ---
> > pbs-datastore/src/datastore.rs | 4 +++-
> > src/api2/config/datastore.rs | 32 ++++++++++++++++++++++++++++++++
> > 2 files changed, 35 insertions(+), 1 deletion(-)
> >
> > diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> > index 29f98b37..df038d62 100644
> > --- a/pbs-datastore/src/datastore.rs
> > +++ b/pbs-datastore/src/datastore.rs
> > @@ -1510,7 +1510,9 @@ impl DataStore {
> > // weird, but ok
> > }
> > Err(err) if err.is_errno(nix::errno::Errno::EBUSY) => {
> > - warn!("Cannot delete datastore directory (is it a mount point?).")
> > + if datastore_config.backing_device.is_none() {
> > + warn!("Cannot delete datastore directory (is it a mount point?).")
> > + }
> > }
> > Err(err) if err.is_errno(nix::errno::Errno::ENOTEMPTY) => {
> > warn!("Datastore directory not empty, not deleting.")
> > diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
> > index c24b6e9d..c7d59c44 100644
> > --- a/src/api2/config/datastore.rs
> > +++ b/src/api2/config/datastore.rs
> > @@ -29,9 +29,11 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
> > use crate::api2::config::verify::delete_verification_job;
> > use pbs_config::CachedUserInfo;
> >
> > +use pbs_datastore::is_datastore_available;
> > use proxmox_rest_server::WorkerTask;
> >
> > use crate::server::jobstate;
> > +use crate::tools::disks::unmount_by_mountpoint;
> >
> > #[api(
> > input: {
> > @@ -523,6 +525,14 @@ pub async fn delete_datastore(
> > http_bail!(NOT_FOUND, "datastore '{}' does not exist.", name);
> > }
> >
> > + let store_config: DataStoreConfig = config.lookup("datastore", &name)?;
> > + if destroy_data && !is_datastore_available(&store_config) {
> > + http_bail!(
> > + BAD_REQUEST,
> > + "cannot destroy data on '{name}' unless the datastore is mounted"
> > + );
> > + }
> > +
> > if !keep_job_configs {
> > for job in list_verification_jobs(Some(name.clone()), Value::Null, rpcenv)? {
> > delete_verification_job(job.config.id, None, rpcenv)?
> > @@ -549,6 +559,22 @@ pub async fn delete_datastore(
> >
> > let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> > let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
> > + let name_copy = name.clone();
> > + tokio::spawn(async move {
> > + if let Ok(proxy_pid) =
> > + proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
> > + {
> > + let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
> > + let _ = proxmox_daemon::command_socket::send_raw(
> > + sock,
> > + &format!(
> > + "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
> > + name_copy
> > + ),
> > + )
> > + .await;
> > + }
> > + });
>
> why is this tokio::spawned, but the one in unmount introduced in the
> same series is not?
>
I took it from update_datastore which is not async, you're right we don't need
spawn here.
> since we now have three call sites with basically the same code, should
> we have a helper to do this?
>
yes
> >
> > let upid = WorkerTask::new_thread(
> > "delete-datastore",
> > @@ -561,6 +587,12 @@ pub async fn delete_datastore(
> > // ignore errors
> > let _ = jobstate::remove_state_file("prune", &name);
> > let _ = jobstate::remove_state_file("garbage_collection", &name);
> > + if destroy_data {
> > + if let Some(mount_point) = store_config.get_mount_point() {
> > + let _ = unmount_by_mountpoint(&mount_point);
> > + let _ = std::fs::remove_dir(&mount_point);
> > + }
> > + }
> >
> > if let Err(err) =
> > proxmox_async::runtime::block_on(crate::server::notify_datastore_removed())
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (12 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 13/26] datastore: handle deletion of removable datastore properly Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:42 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 15/26] ui: add partition selector form Hannes Laimer
` (12 subsequent siblings)
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
docs/storage.rst | 29 +++++++++++++++++++++++++++++
1 file changed, 29 insertions(+)
diff --git a/docs/storage.rst b/docs/storage.rst
index f1e15d52..8713352a 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -165,6 +165,35 @@ following command creates a new datastore called ``store1`` on
# proxmox-backup-manager datastore create store1 /backup/disk1/store1
+Removable Datastores
+^^^^^^^^^^^^^^^^^^^^
+Removable datastores have a ``backing-device`` associated with them, they can be
+mounted and unmounted. Other than that they behave the same way a normal datastore
+would.
+
+They can be created on already correctly formatted partitions, which, as with normal
+datastores, should be either ``ext4`` or ``xfs``. It is also possible to create them
+on completely unused disks through "Admnistartion" > "Disks / Storage" > "Directory",
+using this method the disk will be partitioned and formatted automatically for the datastore.
+
+They are mounted automatically when plugged in. It is possible to create a
+removable datastore on one PBS and use it on multiple, the device just has to be added
+on all as a removable datastore. This can be done using the normal `Create Datastore`
+form, by checking **Removable datastore**, selecting the device and setting the correct path.
+If the device already contains a datastore at the specified path it'll just be added as
+a new datastore to the PBS instance and will be mounted whenever plugged in. Unmounting has
+to be done through the UI by clicking "Unmount" on the summary page or using the CLI
+
+.. code-block:: console
+
+ # proxmox-backup-manager datastore unmount store1
+
+both will wait for any running tasks to finish and unmount the device.
+
+All removable datastores are mounted under /mnt/datastore/<name>, and the specified path
+refers to the path on the device.
+
+
Managing Datastores
^^^^^^^^^^^^^^^^^^^
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section Hannes Laimer
@ 2024-10-14 13:42 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:42 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> docs/storage.rst | 29 +++++++++++++++++++++++++++++
> 1 file changed, 29 insertions(+)
>
> diff --git a/docs/storage.rst b/docs/storage.rst
> index f1e15d52..8713352a 100644
> --- a/docs/storage.rst
> +++ b/docs/storage.rst
> @@ -165,6 +165,35 @@ following command creates a new datastore called ``store1`` on
> # proxmox-backup-manager datastore create store1 /backup/disk1/store1
>
>
> +Removable Datastores
> +^^^^^^^^^^^^^^^^^^^^
> +Removable datastores have a ``backing-device`` associated with them, they can be
> +mounted and unmounted. Other than that they behave the same way a normal datastore
> +would.
> +
> +They can be created on already correctly formatted partitions, which, as with normal
> +datastores, should be either ``ext4`` or ``xfs``. It is also possible to create them
> +on completely unused disks through "Admnistartion" > "Disks / Storage" > "Directory",
typo Admnistartion
> +using this method the disk will be partitioned and formatted automatically for the datastore.
> +
> +They are mounted automatically when plugged in. It is possible to create a
> +removable datastore on one PBS and use it on multiple, the device just has to be added
multiple instances
> +on all as a removable datastore. This can be done using the normal `Create Datastore`
s/all/each instance/
> +form, by checking **Removable datastore**, selecting the device and setting the correct path.
form?
> +If the device already contains a datastore at the specified path it'll just be added as
> +a new datastore to the PBS instance and will be mounted whenever plugged in. Unmounting has
> +to be done through the UI by clicking "Unmount" on the summary page or using the CLI
> +
> +.. code-block:: console
> +
> + # proxmox-backup-manager datastore unmount store1
> +
> +both will wait for any running tasks to finish and unmount the device.
> +
> +All removable datastores are mounted under /mnt/datastore/<name>, and the specified path
> +refers to the path on the device.
> +
> +
> Managing Datastores
> ^^^^^^^^^^^^^^^^^^^
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 15/26] ui: add partition selector form
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (13 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 14/26] docs: add removable datastores section Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 16/26] ui: add removable datastore creation support Hannes Laimer
` (11 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Makefile | 1 +
www/form/PartitionSelector.js | 81 +++++++++++++++++++++++++++++++++++
2 files changed, 82 insertions(+)
create mode 100644 www/form/PartitionSelector.js
diff --git a/www/Makefile b/www/Makefile
index 609a0ba6..45adfc54 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -49,6 +49,7 @@ JSSRC= \
form/NamespaceMaxDepth.js \
form/CalendarEvent.js \
form/PermissionPathSelector.js \
+ form/PartitionSelector.js \
form/GroupSelector.js \
form/GroupFilter.js \
form/VerifyOutdatedAfter.js \
diff --git a/www/form/PartitionSelector.js b/www/form/PartitionSelector.js
new file mode 100644
index 00000000..3d6eadd5
--- /dev/null
+++ b/www/form/PartitionSelector.js
@@ -0,0 +1,81 @@
+Ext.define('pbs-partition-list', {
+ extend: 'Ext.data.Model',
+ fields: ['name', 'uuid', 'filesystem', 'devpath', 'size', 'model'],
+ proxy: {
+ type: 'proxmox',
+ url: "/api2/json/nodes/localhost/disks/list?skipsmart=1&include-partitions=1&exclude-used=1",
+ reader: {
+ transform: (rawData) => rawData.data
+ .flatMap(disk => (disk.partitions
+ .map(part => ({ ...part, model: disk.model })) ?? [])
+ .filter(partition => partition.used === 'filesystem')),
+ },
+ },
+ idProperty: 'devpath',
+
+});
+
+Ext.define('PBS.form.PartitionSelector', {
+ extend: 'Proxmox.form.ComboGrid',
+ alias: 'widget.pbsPartitionSelector',
+
+ allowBlank: false,
+ autoSelect: false,
+ submitEmpty: false,
+ valueField: 'uuid',
+ displayField: 'devpath',
+
+ store: {
+ model: 'pbs-partition-list',
+ autoLoad: true,
+ sorters: 'devpath',
+ },
+ getSubmitData: function() {
+ let me = this;
+ let data = null;
+ if (!me.disabled && me.submitValue && !me.isFileUpload()) {
+ let val = me.getSubmitValue();
+ if (val !== undefined && val !== null && val !== '') {
+ data = {};
+ data[me.getName()] = val;
+ } else if (me.getDeleteEmpty()) {
+ data = {};
+ data.delete = me.getName();
+ }
+ }
+ return data;
+ },
+ listConfig: {
+ columns: [
+ {
+ header: gettext('Path'),
+ sortable: true,
+ dataIndex: 'devpath',
+ renderer: (v, metaData, rec) => Ext.String.htmlEncode(v),
+ flex: 1,
+ },
+ {
+ header: gettext('Filesystem'),
+ sortable: true,
+ dataIndex: 'filesystem',
+ flex: 1,
+ },
+ {
+ header: gettext('Size'),
+ sortable: true,
+ dataIndex: 'size',
+ renderer: Proxmox.Utils.format_size,
+ flex: 1,
+ },
+ {
+ header: gettext('Model'),
+ sortable: true,
+ dataIndex: 'model',
+ flex: 1,
+ },
+ ],
+ viewConfig: {
+ emptyText: 'No usable partitions present',
+ },
+ },
+});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 16/26] ui: add removable datastore creation support
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (14 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 15/26] ui: add partition selector form Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 17/26] ui: add (un)mount button to summary Hannes Laimer
` (10 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/window/DataStoreEdit.js | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index b6115460..54929382 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -62,6 +62,20 @@ Ext.define('PBS.DataStoreEdit', {
fieldLabel: gettext('Backing Path'),
emptyText: gettext('An absolute path'),
},
+ {
+ xtype: 'pmxDisplayEditField',
+ fieldLabel: gettext('Device'),
+ name: 'backing-device',
+ disabled: true,
+ cbind: {
+ editable: '{isCreate}',
+ },
+ editConfig: {
+ xtype: 'pbsPartitionSelector',
+ allowBlank: true,
+ },
+ emptyText: gettext('Device path'),
+ },
],
column2: [
{
@@ -87,6 +101,29 @@ Ext.define('PBS.DataStoreEdit', {
},
],
columnB: [
+ {
+ xtype: 'checkbox',
+ boxLabel: gettext('Removable datastore'),
+ submitValue: false,
+ listeners: {
+ change: function(checkbox, isRemovable) {
+ let inputPanel = checkbox.up('inputpanel');
+ let pathField = inputPanel.down('[name=path]');
+ let uuidField = inputPanel.down('pbsPartitionSelector[name=backing-device]');
+ let uuidEditField = inputPanel.down('[name=backing-device]');
+
+ uuidField.allowBlank = !isRemovable;
+ uuidEditField.setDisabled(!isRemovable);
+ uuidField.setDisabled(!isRemovable);
+ uuidField.setValue('');
+ if (isRemovable) {
+ pathField.setFieldLabel(gettext('On device path'));
+ } else {
+ pathField.setFieldLabel(gettext('Backing Path'));
+ }
+ },
+ },
+ },
{
xtype: 'textfield',
name: 'comment',
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 17/26] ui: add (un)mount button to summary
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (15 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 16/26] ui: add removable datastore creation support Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 18/26] ui: tree: render unmounted datastores correctly Hannes Laimer
` (9 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
And only try to load datastore information if the datastore is
available.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/datastore/Summary.js | 92 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 90 insertions(+), 2 deletions(-)
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index a932b4e0..49aa3b3c 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -309,7 +309,82 @@ Ext.define('PBS.DataStoreSummary', {
model: 'pve-rrd-datastore',
});
- me.callParent();
+ me.statusStore = Ext.create('Proxmox.data.ObjectStore', {
+ url: `/api2/json/admin/datastore/${me.datastore}/status`,
+ interval: 1000,
+ });
+
+ let unmountBtn = Ext.create('Ext.Button', {
+ text: gettext('Unmount'),
+ hidden: true,
+ handler: () => {
+ Proxmox.Utils.API2Request({
+ url: `/admin/datastore/${me.datastore}/unmount`,
+ method: 'POST',
+ failure: function(response) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, options) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ }).show();
+ },
+ });
+ },
+ });
+
+ let mountBtn = Ext.create('Ext.Button', {
+ text: gettext('Mount'),
+ hidden: true,
+ handler: () => {
+ Proxmox.Utils.API2Request({
+ url: `/admin/datastore/${me.datastore}/mount`,
+ method: 'POST',
+ failure: function(response) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, options) {
+ Ext.create('Proxmox.window.TaskViewer', {
+ upid: response.result.data,
+ }).show();
+ },
+ });
+ },
+ });
+
+ Ext.apply(me, {
+ tbar: [unmountBtn, mountBtn, '->', { xtype: 'proxmoxRRDTypeSelector' }],
+ });
+
+ me.mon(me.statusStore, 'load', (s, records, success) => {
+ if (!success) {
+ me.down('pbsDataStoreInfo').fireEvent('deactivate');
+ Proxmox.Utils.API2Request({
+ url: `/config/datastore/${me.datastore}`,
+ success: response => {
+ let mode = response.result.data['maintenance-mode'];
+ let [type, _message] = PBS.Utils.parseMaintenanceMode(mode);
+ if (!response.result.data['backing-device']) {
+ return;
+ }
+ if (!type || type === 'read-only') {
+ unmountBtn.setDisabled(true);
+ mountBtn.setDisabled(false);
+ } else if (type === 'unmount') {
+ unmountBtn.setDisabled(true);
+ mountBtn.setDisabled(true);
+ } else {
+ unmountBtn.setDisabled(false);
+ mountBtn.setDisabled(false);
+ }
+ },
+ });
+ } else {
+ me.down('pbsDataStoreInfo').fireEvent('activate');
+ unmountBtn.setDisabled(false);
+ mountBtn.setDisabled(true);
+ }
+ });
let sp = Ext.state.Manager.getProvider();
me.mon(sp, 'statechange', function(provider, key, value) {
@@ -322,11 +397,17 @@ Ext.define('PBS.DataStoreSummary', {
Proxmox.Utils.updateColumns(me);
});
+ me.callParent();
+
Proxmox.Utils.API2Request({
url: `/config/datastore/${me.datastore}`,
waitMsgTarget: me.down('pbsDataStoreInfo'),
success: function(response) {
- let path = Ext.htmlEncode(response.result.data.path);
+ let data = response.result.data;
+ let path = Ext.htmlEncode(data.path);
+ const removable = Object.prototype.hasOwnProperty.call(data, "backing-device");
+ unmountBtn.setHidden(!removable);
+ mountBtn.setHidden(!removable);
me.down('pbsDataStoreInfo').setTitle(`${me.datastore} (${path})`);
me.down('pbsDataStoreNotes').setNotes(response.result.data.comment);
},
@@ -344,6 +425,13 @@ Ext.define('PBS.DataStoreSummary', {
let hasIoTicks = records?.some((rec) => rec?.data?.io_ticks !== undefined);
me.down('#ioDelayChart').setVisible(!success || hasIoTicks);
}, undefined, { single: true });
+ me.on('afterrender', () => {
+ me.statusStore.startUpdate();
+ });
+
+ me.on('destroy', () => {
+ me.statusStore.stopUpdate();
+ });
me.query('proxmoxRRDChart').forEach((chart) => {
chart.setStore(me.rrdstore);
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 18/26] ui: tree: render unmounted datastores correctly
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (16 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 17/26] ui: add (un)mount button to summary Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 19/26] ui: utils: make parseMaintenanceMode more robust Hannes Laimer
` (8 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/NavigationTree.js | 15 ++++++++++++---
www/css/ext6-pbs.css | 8 ++++++++
www/datastore/DataStoreListSummary.js | 1 +
3 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index 53c8daff..4f1a8fd5 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -267,13 +267,22 @@ Ext.define('PBS.view.main.NavigationTree', {
j++;
}
- let [qtip, iconCls] = ['', 'fa fa-database'];
+ let mainIcon = `fa fa-${records[i].data.removable ? 'plug' : 'database'}`;
+ let [qtip, iconCls] = ['', mainIcon];
const maintenance = records[i].data.maintenance;
+ const removable_not_available = records[i].data.removable && !records[i].data.available;
+ if (removable_not_available) {
+ iconCls = `${mainIcon} pmx-tree-icon-custom unplugged`;
+ qtip = gettext('Removable datastore not mounted');
+ }
if (maintenance) {
const [type, message] = PBS.Utils.parseMaintenanceMode(maintenance);
qtip = `${type}${message ? ': ' + message : ''}`;
- let maintenanceTypeCls = type === 'delete' ? 'destroying' : 'maintenance';
- iconCls = `fa fa-database pmx-tree-icon-custom ${maintenanceTypeCls}`;
+ let maintenanceTypeCls = 'maintenance';
+ if (type === 'delete') {
+ maintenanceTypeCls = 'destroying';
+ }
+ iconCls = `${mainIcon} pmx-tree-icon-custom ${maintenanceTypeCls}`;
}
if (getChildTextAt(j).localeCompare(name) !== 0) {
diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index c33ce684..706e681e 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -271,6 +271,10 @@ span.snapshot-comment-column {
content: "\ ";
}
+.x-treelist-item-icon.fa-plug, .pmx-tree-icon-custom.fa-plug {
+ font-size: 12px;
+}
+
/* datastore maintenance */
.pmx-tree-icon-custom.maintenance:after {
content: "\f0ad";
@@ -290,6 +294,10 @@ span.snapshot-comment-column {
color: #888;
}
+.pmx-tree-icon-custom.unplugged:before {
+ color: #888;
+}
+
/*' PBS specific icons */
.pbs-icon-tape {
diff --git a/www/datastore/DataStoreListSummary.js b/www/datastore/DataStoreListSummary.js
index b908034d..f7ea83e7 100644
--- a/www/datastore/DataStoreListSummary.js
+++ b/www/datastore/DataStoreListSummary.js
@@ -22,6 +22,7 @@ Ext.define('PBS.datastore.DataStoreListSummary', {
stillbad: 0,
deduplication: 1.0,
error: "",
+ removable: false,
maintenance: '',
},
},
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 19/26] ui: utils: make parseMaintenanceMode more robust
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (17 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 18/26] ui: tree: render unmounted datastores correctly Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 20/26] ui: add datastore status mask for unmounted removable datastores Hannes Laimer
` (7 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Utils.js | 29 ++++++++++++++++++++++-------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/www/Utils.js b/www/Utils.js
index 4853be36..7756e9b5 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -740,14 +740,29 @@ Ext.define('PBS.Utils', {
return `${icon} ${value}`;
},
- // FIXME: this "parser" is brittle and relies on the order the arguments will appear in
+ /**
+ * Parses maintenance mode property string.
+ * Examples:
+ * "offline,message=foo" -> ["offline", "foo"]
+ * "offline" -> ["offline", null]
+ * "message=foo,offline" -> ["offline", "foo"]
+ * null/undefined -> [null, null]
+ *
+ * @param {string|null} mode - Maintenance mode string to parse.
+ * @return {Array<string|null>} - Parsed maintenance mode values.
+ */
parseMaintenanceMode: function(mode) {
- let [type, message] = mode.split(/,(.+)/);
- type = type.split("=").pop();
- message = message ? message.split("=")[1]
- .replace(/^"(.*)"$/, '$1')
- .replaceAll('\\"', '"') : null;
- return [type, message];
+ if (!mode) {
+ return [null, null];
+ }
+ return mode.split(',').reduce(([m, msg], pair) => {
+ const [key, value] = pair.split('=');
+ if (key === 'message') {
+ return [m, value.replace(/^"(.*)"$/, '$1').replace(/\\"/g, '"')];
+ } else {
+ return [value ?? key, msg];
+ }
+ }, [null, null]);
},
renderMaintenance: function(mode, activeTasks) {
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 20/26] ui: add datastore status mask for unmounted removable datastores
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (18 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 19/26] ui: utils: make parseMaintenanceMode more robust Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 21/26] ui: maintenance: fix disable msg field if no type is selected Hannes Laimer
` (6 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/css/ext6-pbs.css | 12 ++++++++++++
www/datastore/Summary.js | 21 +++++++++++++--------
2 files changed, 25 insertions(+), 8 deletions(-)
diff --git a/www/css/ext6-pbs.css b/www/css/ext6-pbs.css
index 706e681e..891189ae 100644
--- a/www/css/ext6-pbs.css
+++ b/www/css/ext6-pbs.css
@@ -261,6 +261,18 @@ span.snapshot-comment-column {
content: "\f0ad";
}
+.pbs-unplugged-mask div.x-mask-msg-text {
+ background: None;
+ padding: 12px 0 0;
+}
+
+.pbs-unplugged-mask:before {
+ font-size: 3em;
+ display: flex;
+ justify-content: center;
+ content: "\f1e6";
+}
+
/* the small icons TODO move to proxmox-widget-toolkit */
.pmx-tree-icon-custom:after {
position: relative;
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index 49aa3b3c..33b4e18a 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -61,16 +61,21 @@ Ext.define('PBS.DataStoreInfo', {
Proxmox.Utils.API2Request({
url: `/config/datastore/${me.view.datastore}`,
success: function(response) {
- const config = response.result.data;
- if (config['maintenance-mode']) {
- const [_type, msg] = PBS.Utils.parseMaintenanceMode(config['maintenance-mode']);
- me.view.el.mask(
- `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`,
- 'fa pbs-maintenance-mask',
- );
- } else {
+ let maintenanceString = response.result.data['maintenance-mode'];
+ let removable = !!response.result.data['backing-device'];
+ if (!maintenanceString && !removable) {
me.view.el.mask(gettext('Datastore is not available'));
+ return;
}
+
+ let [_type, msg] = PBS.Utils.parseMaintenanceMode(maintenanceString);
+ let isUnplugged = !maintenanceString && removable;
+ let maskMessage = isUnplugged
+ ? gettext('Datastore is not mounted')
+ : `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`;
+
+ let maskIcon = isUnplugged ? 'fa pbs-unplugged-mask' : 'fa pbs-maintenance-mask';
+ me.view.el.mask(maskMessage, maskIcon);
},
});
return;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 21/26] ui: maintenance: fix disable msg field if no type is selected
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (19 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 20/26] ui: add datastore status mask for unmounted removable datastores Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 22/26] ui: render 'unmount' maintenance mode correctly Hannes Laimer
` (5 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/window/MaintenanceOptions.js | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 1ee92542..527c3698 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -56,12 +56,17 @@ Ext.define('PBS.window.MaintenanceOptions', {
fieldLabel: gettext('Maintenance Type'),
value: '__default__',
deleteEmpty: true,
+ listeners: {
+ change: (field, newValue) => {
+ Ext.getCmp('message-field').setDisabled(newValue === '__default__');
+ },
+ },
},
{
xtype: 'proxmoxtextfield',
+ id: 'message-field',
name: 'maintenance-msg',
fieldLabel: gettext('Description'),
- // FIXME: disable if maintenance type is none
},
],
},
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 22/26] ui: render 'unmount' maintenance mode correctly
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (20 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 21/26] ui: maintenance: fix disable msg field if no type is selected Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 23/26] api: node: allow creation of removable datastore through directory endpoint Hannes Laimer
` (4 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
www/Utils.js | 4 +++-
www/window/MaintenanceOptions.js | 10 ++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/www/Utils.js b/www/Utils.js
index 7756e9b5..6bae9b70 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -775,7 +775,7 @@ Ext.define('PBS.Utils', {
let extra = '';
if (activeTasks !== undefined) {
- const conflictingTasks = activeTasks.write + (type === 'offline' ? activeTasks.read : 0);
+ const conflictingTasks = activeTasks.write + (type === 'offline' || type === 'unmount' ? activeTasks.read : 0);
if (conflictingTasks > 0) {
extra += '| <i class="fa fa-spinner fa-pulse fa-fw"></i> ';
@@ -795,6 +795,8 @@ Ext.define('PBS.Utils', {
break;
case 'offline': modeText = gettext("Offline");
break;
+ case 'unmount': modeText = gettext("Unmounting");
+ break;
}
return `${modeText} ${extra}`;
},
diff --git a/www/window/MaintenanceOptions.js b/www/window/MaintenanceOptions.js
index 527c3698..d7348cb4 100644
--- a/www/window/MaintenanceOptions.js
+++ b/www/window/MaintenanceOptions.js
@@ -52,6 +52,7 @@ Ext.define('PBS.window.MaintenanceOptions', {
items: [
{
xtype: 'pbsMaintenanceType',
+ id: 'type-field',
name: 'maintenance-type',
fieldLabel: gettext('Maintenance Type'),
value: '__default__',
@@ -85,6 +86,15 @@ Ext.define('PBS.window.MaintenanceOptions', {
};
}
+ let unmounting = options['maintenance-type'] === 'unmount';
+ let defaultType = options['maintenance-type'] === '__default__';
+ if (unmounting) {
+ options['maintenance-type'] = '';
+ }
+
me.callParent([options]);
+
+ Ext.ComponentManager.get('type-field').setDisabled(unmounting);
+ Ext.ComponentManager.get('message-field').setDisabled(unmounting || defaultType);
},
});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 23/26] api: node: allow creation of removable datastore through directory endpoint
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (21 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 22/26] ui: render 'unmount' maintenance mode correctly Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 24/26] api: node: include removable datastores in directory list Hannes Laimer
` (3 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 57 +++++++++++++++++++++++++++++---
1 file changed, 52 insertions(+), 5 deletions(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index c0c53d84..cd816414 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -123,6 +123,11 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
description: "Configure a datastore using the directory.",
type: bool,
optional: true,
+ default: false,
+ },
+ "removable-datastore": {
+ description: "The added datastore is removable.",
+ type: bool,
},
filesystem: {
type: FileSystemType,
@@ -141,7 +146,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
pub fn create_datastore_disk(
name: String,
disk: String,
- add_datastore: Option<bool>,
+ add_datastore: bool,
+ removable_datastore: bool,
filesystem: Option<FileSystemType>,
rpcenv: &mut dyn RpcEnvironment,
) -> Result<String, Error> {
@@ -155,8 +161,49 @@ pub fn create_datastore_disk(
bail!("disk '{}' is already in use.", disk);
}
- let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+ if add_datastore && removable_datastore {
+ let upid_str = WorkerTask::new_thread(
+ "dircreate",
+ Some(name.clone()),
+ auth_id,
+ to_stdout,
+ move |_worker| {
+ info!("create removable datastore '{name}' on disk {disk}");
+
+ let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
+
+ let manager = DiskManage::new();
+
+ let disk = manager.disk_by_name(&disk)?;
+
+ let partition = create_single_linux_partition(&disk)?;
+ create_file_system(&partition, filesystem)?;
+
+ let uuid = get_fs_uuid(&partition)?;
+
+ let lock = pbs_config::datastore::lock_config()?;
+ let datastore: DataStoreConfig = serde_json::from_value(
+ json!({ "name": name, "path": name, "backing-device": uuid }),
+ )?;
+
+ let (config, _digest) = pbs_config::datastore::config()?;
+
+ if config.sections.get(&datastore.name).is_some() {
+ bail!("datastore '{}' already exists.", datastore.name);
+ }
+
+ // we don't have to check if the UUID is already in use since we just created the
+ // fs ourself
+ crate::api2::config::datastore::do_create_datastore(lock, config, datastore)?;
+
+ Ok(())
+ },
+ )?;
+ return Ok(upid_str);
+ };
+
+ let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
// check if the default path exists already.
// bail if it is not empty or another filesystem mounted on top
let default_path = std::path::PathBuf::from(&mount_point);
@@ -183,7 +230,6 @@ pub fn create_datastore_disk(
move |_worker| {
info!("create datastore '{name}' on disk {disk}");
- let add_datastore = add_datastore.unwrap_or(false);
let filesystem = filesystem.unwrap_or(FileSystemType::Ext4);
let manager = DiskManage::new();
@@ -246,8 +292,9 @@ pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
// path of datastore cannot be changed
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
- let conflicting_datastore: Option<DataStoreConfig> =
- datastores.into_iter().find(|ds| ds.absolute_path() == path);
+ let conflicting_datastore: Option<DataStoreConfig> = datastores.into_iter().find(|ds| {
+ ds.absolute_path() == path || ds.get_mount_point().map_or(false, |mp| mp == path)
+ });
if let Some(conflicting_datastore) = conflicting_datastore {
bail!(
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 24/26] api: node: include removable datastores in directory list
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (22 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 23/26] api: node: allow creation of removable datastore through directory endpoint Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
` (2 subsequent siblings)
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 29 ++++++++++++++++++++++++++++-
1 file changed, 28 insertions(+), 1 deletion(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index cd816414..ed0bd03d 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -45,6 +45,8 @@ pub struct DatastoreMountInfo {
pub path: String,
/// The mounted device.
pub device: String,
+ /// This is removable
+ pub removable: bool,
/// File system type
pub filesystem: Option<String>,
/// Mount options
@@ -61,7 +63,7 @@ pub struct DatastoreMountInfo {
}
},
returns: {
- description: "List of systemd datastore mount units.",
+ description: "List of removable-datastore devices and systemd datastore mount units.",
type: Array,
items: {
type: DatastoreMountInfo,
@@ -100,6 +102,31 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
path: data.Where,
filesystem: data.Type,
options: data.Options,
+ removable: false,
+ });
+ }
+
+ let (config, _digest) = pbs_config::datastore::config()?;
+ let store_list: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
+
+ for item in store_list
+ .into_iter()
+ .filter(|store| store.backing_device.is_some())
+ {
+ let Some(backing_device) = item.backing_device.as_deref() else {
+ continue;
+ };
+ let Some(mount_point) = item.get_mount_point() else {
+ continue;
+ };
+ list.push(DatastoreMountInfo {
+ unitfile: "datastore config".to_string(),
+ name: item.name.clone(),
+ device: format!("/dev/disk/by-uuid/{backing_device}"),
+ path: mount_point,
+ filesystem: None,
+ options: None,
+ removable: true,
});
}
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* [pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (23 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 24/26] api: node: include removable datastores in directory list Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:43 ` Fabian Grünbichler
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 26/26] ui: support create removable datastore through directory creation Hannes Laimer
2024-10-14 13:43 ` [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Fabian Grünbichler
26 siblings, 1 reply; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 14 ++++++--------
1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index ed0bd03d..2525642d 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -11,8 +11,8 @@ use proxmox_schema::api;
use proxmox_section_config::SectionConfigData;
use pbs_api_types::{
- DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_SCHEMA, NODE_SCHEMA, PRIV_SYS_AUDIT,
- PRIV_SYS_MODIFY, UPID_SCHEMA,
+ DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_MOUNT_DIR, DATASTORE_SCHEMA, NODE_SCHEMA,
+ PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, UPID_SCHEMA,
};
use crate::tools::disks::{
@@ -23,8 +23,6 @@ use crate::tools::systemd::{self, types::*};
use proxmox_rest_server::WorkerTask;
-const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
-
#[api(
properties: {
"filesystem": {
@@ -91,7 +89,7 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
let name = data
.Where
- .strip_prefix(BASE_MOUNT_DIR)
+ .strip_prefix(DATASTORE_MOUNT_DIR)
.unwrap_or(&data.Where)
.to_string();
@@ -230,7 +228,7 @@ pub fn create_datastore_disk(
return Ok(upid_str);
};
- let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
+ let mount_point = format!("{}/{}", DATASTORE_MOUNT_DIR, &name);
// check if the default path exists already.
// bail if it is not empty or another filesystem mounted on top
let default_path = std::path::PathBuf::from(&mount_point);
@@ -238,7 +236,7 @@ pub fn create_datastore_disk(
match std::fs::metadata(&default_path) {
Err(_) => {} // path does not exist
Ok(stat) => {
- let basedir_dev = std::fs::metadata(BASE_MOUNT_DIR)?.st_dev();
+ let basedir_dev = std::fs::metadata(DATASTORE_MOUNT_DIR)?.st_dev();
if stat.st_dev() != basedir_dev {
bail!("path {default_path:?} already exists and is mountpoint");
}
@@ -315,7 +313,7 @@ pub fn create_datastore_disk(
)]
/// Remove a Filesystem mounted under `/mnt/datastore/<name>`.
pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
- let path = format!("{}{}", BASE_MOUNT_DIR, name);
+ let path = format!("{}/{}", DATASTORE_MOUNT_DIR, name);
// path of datastore cannot be changed
let (config, _) = pbs_config::datastore::config()?;
let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
@ 2024-10-14 13:43 ` Fabian Grünbichler
0 siblings, 0 replies; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:43 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
since they have the same value (this is important information!)
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> src/api2/node/disks/directory.rs | 14 ++++++--------
> 1 file changed, 6 insertions(+), 8 deletions(-)
>
> diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
> index ed0bd03d..2525642d 100644
> --- a/src/api2/node/disks/directory.rs
> +++ b/src/api2/node/disks/directory.rs
> @@ -11,8 +11,8 @@ use proxmox_schema::api;
> use proxmox_section_config::SectionConfigData;
>
> use pbs_api_types::{
> - DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_SCHEMA, NODE_SCHEMA, PRIV_SYS_AUDIT,
> - PRIV_SYS_MODIFY, UPID_SCHEMA,
> + DataStoreConfig, BLOCKDEVICE_NAME_SCHEMA, DATASTORE_MOUNT_DIR, DATASTORE_SCHEMA, NODE_SCHEMA,
> + PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, UPID_SCHEMA,
> };
>
> use crate::tools::disks::{
> @@ -23,8 +23,6 @@ use crate::tools::systemd::{self, types::*};
>
> use proxmox_rest_server::WorkerTask;
>
> -const BASE_MOUNT_DIR: &str = "/mnt/datastore/";
> -
> #[api(
> properties: {
> "filesystem": {
> @@ -91,7 +89,7 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
>
> let name = data
> .Where
> - .strip_prefix(BASE_MOUNT_DIR)
> + .strip_prefix(DATASTORE_MOUNT_DIR)
> .unwrap_or(&data.Where)
> .to_string();
>
> @@ -230,7 +228,7 @@ pub fn create_datastore_disk(
> return Ok(upid_str);
> };
>
> - let mount_point = format!("{}{}", BASE_MOUNT_DIR, &name);
> + let mount_point = format!("{}/{}", DATASTORE_MOUNT_DIR, &name);
> // check if the default path exists already.
> // bail if it is not empty or another filesystem mounted on top
> let default_path = std::path::PathBuf::from(&mount_point);
> @@ -238,7 +236,7 @@ pub fn create_datastore_disk(
> match std::fs::metadata(&default_path) {
> Err(_) => {} // path does not exist
> Ok(stat) => {
> - let basedir_dev = std::fs::metadata(BASE_MOUNT_DIR)?.st_dev();
> + let basedir_dev = std::fs::metadata(DATASTORE_MOUNT_DIR)?.st_dev();
> if stat.st_dev() != basedir_dev {
> bail!("path {default_path:?} already exists and is mountpoint");
> }
> @@ -315,7 +313,7 @@ pub fn create_datastore_disk(
> )]
> /// Remove a Filesystem mounted under `/mnt/datastore/<name>`.
> pub fn delete_datastore_disk(name: String) -> Result<(), Error> {
> - let path = format!("{}{}", BASE_MOUNT_DIR, name);
> + let path = format!("{}/{}", DATASTORE_MOUNT_DIR, name);
> // path of datastore cannot be changed
> let (config, _) = pbs_config::datastore::config()?;
> let datastores: Vec<DataStoreConfig> = config.convert_to_typed_array("datastore")?;
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v12 26/26] ui: support create removable datastore through directory creation
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (24 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 25/26] node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR Hannes Laimer
@ 2024-09-04 14:11 ` Hannes Laimer
2024-10-14 13:43 ` [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Fabian Grünbichler
26 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-09-04 14:11 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/node/disks/directory.rs | 2 ++
www/DirectoryList.js | 13 +++++++++++++
www/window/CreateDirectory.js | 14 ++++++++++++++
3 files changed, 29 insertions(+)
diff --git a/src/api2/node/disks/directory.rs b/src/api2/node/disks/directory.rs
index 2525642d..356e102f 100644
--- a/src/api2/node/disks/directory.rs
+++ b/src/api2/node/disks/directory.rs
@@ -153,6 +153,8 @@ pub fn list_datastore_mounts() -> Result<Vec<DatastoreMountInfo>, Error> {
"removable-datastore": {
description: "The added datastore is removable.",
type: bool,
+ optional: true,
+ default: false,
},
filesystem: {
type: FileSystemType,
diff --git a/www/DirectoryList.js b/www/DirectoryList.js
index adefa9ab..25921a62 100644
--- a/www/DirectoryList.js
+++ b/www/DirectoryList.js
@@ -121,6 +121,19 @@ Ext.define('PBS.admin.Directorylist', {
],
columns: [
+ {
+ text: '<span class="fa fa-plug"/>',
+ flex: 0,
+ width: 35,
+ dataIndex: 'removable',
+ renderer: function(_text, _, row) {
+ if (row.data.removable) {
+ return `<i class="fa fa-check"/>`;
+ } else {
+ return '';
+ }
+ },
+ },
{
text: gettext('Path'),
dataIndex: 'path',
diff --git a/www/window/CreateDirectory.js b/www/window/CreateDirectory.js
index 6aabe21a..38d6979d 100644
--- a/www/window/CreateDirectory.js
+++ b/www/window/CreateDirectory.js
@@ -43,6 +43,20 @@ Ext.define('PBS.window.CreateDirectory', {
name: 'add-datastore',
fieldLabel: gettext('Add as Datastore'),
value: '1',
+ listeners: {
+ change(field, newValue, _oldValue) {
+ let form = field.up('form');
+ let rmBox = form.down('[name=removable-datastore]');
+
+ rmBox.setDisabled(!newValue);
+ rmBox.setValue(false);
+ },
+ },
+ },
+ {
+ xtype: 'proxmoxcheckbox',
+ name: 'removable-datastore',
+ fieldLabel: gettext('is removable'),
},
],
});
--
2.39.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores
2024-09-04 14:11 [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Hannes Laimer
` (25 preceding siblings ...)
2024-09-04 14:11 ` [pbs-devel] [PATCH proxmox-backup v12 26/26] ui: support create removable datastore through directory creation Hannes Laimer
@ 2024-10-14 13:43 ` Fabian Grünbichler
2024-10-29 16:18 ` Hannes Laimer
26 siblings, 1 reply; 55+ messages in thread
From: Fabian Grünbichler @ 2024-10-14 13:43 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
41 files changed, 1756 insertions(+), 733 deletions(-)
26 patches
3(!) of them have a commit message, all of them very terse and not on
the patches that implement the meat of this series.. no notes on patches
either calling out history of the patches themselves.
NACKed in its current form on that alone, you should know better!
On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
>
> The datastore path specified is relative to the root of the used device.
> Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> If we decide to allow multiple datastores on the same device in the future
> they can be mounted/unmounted individually.
see comment on patch #7 regarding this..
and other nits and comments on individual patches as well..
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in.
>
> v12: thanks @Wolfgang
> * use bind mounts, so now
> <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
> this is a bit cleaner and allows for multiple datastores
> on a single device to be mounted individually, if we
> want to allow that in the future
> * small code improvements
>
>
> v11:
> * rebase onto master
>
> v10: thanks @Gabriel and @Wolfgang
> * make is_datastore_available more robust
> * fix a lot of wording
> * drop format on uuid_mount command for UUID
> * only gather_disk_stats if datastore is available
> * overall code improvements
> * ui: include model in partition selector
> * rebased onto master
>
> v9:
> * change mount point to `/mnt/datastore/<name>`
> * update "Directory" list UI
> * add `absolute_path()` from Dietmar's RFC
> * update docs
>
> v8:
> * still depends on [1]
> * paths for removable datastores are now relative to
> `/mnt/removable_datastore/<UUID>`
> * add support for creation of removable datastore through the
> "create directory" endpoint (last 3 patches)
> * update datastore creation UI
> * update docs
>
> v7:
> * depends on [1]
> * improve logging when waiting for tasks
> * drop `update-datatore-cache` refactoring
> * fix some commit messages
>
> [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
>
> v6:
> * remove 'drop' flag in datastore cache
> * use maintenance-mode 'unmount' for unmounting process, only for the
> unmounting not for being unmounted
> * rename/simplify update-datastore-cache command
> * ui: integrate new unmounting maintenance mode
> * basically a mix of v3 and v4
>
> v5: thanks @Dietmar and @Christian
> * drop --force for unmount since it'll always fail if tasks are still running, and if
> there are not normal unount will work
> * improve several commit messages
> * improve error message wording
> * add removable datastore section to docs
> * add documentation for is_datastore_available
>
> v4: thanks a lot @Dietmar and @Christian
> * make check if mounted wayyy faster
> * don't keep track of mounting state
> * drop Unplugged maintenance mode
> * use UUID_FORMAT for uuid field
> * a lot of small things, like use of bail!, inline format!, ...
> * include improvement to cache handling
>
> v3:
> * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
> * fix CLI (un)mount command, thanks @Gabriel
> * add removable datastore CLI autocomplete helper
> * rebase onto master
> * move ui patches to the end
>
> thanks @Lukas and @Thomas for the feedback
> v2:
> * fix datastore 'add' button in the UI
> * some format!("{}", a) -> format!("{a}")
> * replace `const` with `let` in js code
> * change icon `fa-usb` -> `fa-plug`
> * add some docs
> * add JDoc for parseMaintenanceMode
> * proxmox-schema dep bump
>
> Dietmar Maurer (2):
> config: factor out method to get the absolute datastore path
> maintenance: add 'Unmount' maintenance type
>
> Hannes Laimer (24):
> tools: add disks utility functions
> pbs-api-types: add backing-device to DataStoreConfig
> disks: add UUID to partition info
> datastore: add helper for checking if a removable datastore is
> available
> api: admin: add (un)mount endpoint for removable datastores
> api: removable datastore creation
> api: disks list: add exclude-used flag
> pbs-api-types: add removable/is-available flag to DataStoreListItem
> bin: manager: add (un)mount command
> add auto-mounting for removable datastores
> datastore: handle deletion of removable datastore properly
> docs: add removable datastores section
> ui: add partition selector form
> ui: add removable datastore creation support
> ui: add (un)mount button to summary
> ui: tree: render unmounted datastores correctly
> ui: utils: make parseMaintenanceMode more robust
> ui: add datastore status mask for unmounted removable datastores
> ui: maintenance: fix disable msg field if no type is selected
> ui: render 'unmount' maintenance mode correctly
> api: node: allow creation of removable datastore through directory
> endpoint
> api: node: include removable datastores in directory list
> node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
> ui: support create removable datastore through directory creation
>
> debian/proxmox-backup-server.install | 1 +
> debian/proxmox-backup-server.udev | 3 +
> docs/storage.rst | 29 +++
> etc/Makefile | 3 +-
> etc/removable-device-attach@.service.in | 8 +
> pbs-api-types/src/datastore.rs | 42 +++-
> pbs-api-types/src/maintenance.rs | 11 +-
> pbs-config/src/datastore.rs | 14 ++
> pbs-datastore/src/datastore.rs | 73 +++++-
> pbs-datastore/src/lib.rs | 2 +-
> src/api2/admin/datastore.rs | 260 ++++++++++++++++++--
> src/api2/config/datastore.rs | 69 +++++-
> src/api2/node/disks/directory.rs | 102 ++++++--
> src/api2/node/disks/mod.rs | 8 +
> src/api2/status.rs | 18 +-
> src/bin/proxmox-backup-proxy.rs | 9 +-
> src/bin/proxmox_backup_manager/datastore.rs | 129 +++++++++-
> src/tools/disks/mod.rs | 99 +++++++-
> www/DirectoryList.js | 13 +
> www/Makefile | 1 +
> www/NavigationTree.js | 15 +-
> www/Utils.js | 33 ++-
> www/css/ext6-pbs.css | 20 ++
> www/datastore/DataStoreListSummary.js | 1 +
> www/datastore/Summary.js | 113 ++++++++-
> www/form/PartitionSelector.js | 81 ++++++
> www/window/CreateDirectory.js | 14 ++
> www/window/DataStoreEdit.js | 37 +++
> www/window/MaintenanceOptions.js | 17 +-
> 29 files changed, 1141 insertions(+), 84 deletions(-)
> create mode 100644 etc/removable-device-attach@.service.in
> create mode 100644 www/form/PartitionSelector.js
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores
2024-10-14 13:43 ` [pbs-devel] [PATCH proxmox-backup v12 00/26] add removable datastores Fabian Grünbichler
@ 2024-10-29 16:18 ` Hannes Laimer
0 siblings, 0 replies; 55+ messages in thread
From: Hannes Laimer @ 2024-10-29 16:18 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
Thanks a lot for taking the time to review this series, I added some
comments to things you mentioned. And I'll fix the obviouse
shortcommings of this version in the next one.
On Mon Oct 14, 2024 at 3:43 PM CEST, Fabian Grünbichler wrote:
> 41 files changed, 1756 insertions(+), 733 deletions(-)
>
> 26 patches
>
> 3(!) of them have a commit message, all of them very terse and not on
> the patches that implement the meat of this series.. no notes on patches
> either calling out history of the patches themselves.
>
> NACKed in its current form on that alone, you should know better!
>
> On September 4, 2024 4:11 pm, Hannes Laimer wrote:
> > These patches add support for removable datastores. All removable
> > datastores have a backing-device(a UUID) associated with them. Removable
> > datastores work like normal ones, just that they can be unplugged. It is
> > possible to create a removable datastore, sync backups onto it, unplug
> > it and use it on a different PBS.
> >
> > The datastore path specified is relative to the root of the used device.
> > Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> > If we decide to allow multiple datastores on the same device in the future
> > they can be mounted/unmounted individually.
>
> see comment on patch #7 regarding this..
>
> and other nits and comments on individual patches as well..
>
> > When a removable datastore is deleted and 'destroy-data' is set, the
> > device has to be mounted. If 'destroy-data' is not set the datastore
> > can be deleted even if the device is not present. Removable datastores
> > are automatically mounted when plugged in.
> >
> > v12: thanks @Wolfgang
> > * use bind mounts, so now
> > <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
> > this is a bit cleaner and allows for multiple datastores
> > on a single device to be mounted individually, if we
> > want to allow that in the future
> > * small code improvements
> >
> >
> > v11:
> > * rebase onto master
> >
> > v10: thanks @Gabriel and @Wolfgang
> > * make is_datastore_available more robust
> > * fix a lot of wording
> > * drop format on uuid_mount command for UUID
> > * only gather_disk_stats if datastore is available
> > * overall code improvements
> > * ui: include model in partition selector
> > * rebased onto master
> >
> > v9:
> > * change mount point to `/mnt/datastore/<name>`
> > * update "Directory" list UI
> > * add `absolute_path()` from Dietmar's RFC
> > * update docs
> >
> > v8:
> > * still depends on [1]
> > * paths for removable datastores are now relative to
> > `/mnt/removable_datastore/<UUID>`
> > * add support for creation of removable datastore through the
> > "create directory" endpoint (last 3 patches)
> > * update datastore creation UI
> > * update docs
> >
> > v7:
> > * depends on [1]
> > * improve logging when waiting for tasks
> > * drop `update-datatore-cache` refactoring
> > * fix some commit messages
> >
> > [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
> >
> > v6:
> > * remove 'drop' flag in datastore cache
> > * use maintenance-mode 'unmount' for unmounting process, only for the
> > unmounting not for being unmounted
> > * rename/simplify update-datastore-cache command
> > * ui: integrate new unmounting maintenance mode
> > * basically a mix of v3 and v4
> >
> > v5: thanks @Dietmar and @Christian
> > * drop --force for unmount since it'll always fail if tasks are still running, and if
> > there are not normal unount will work
> > * improve several commit messages
> > * improve error message wording
> > * add removable datastore section to docs
> > * add documentation for is_datastore_available
> >
> > v4: thanks a lot @Dietmar and @Christian
> > * make check if mounted wayyy faster
> > * don't keep track of mounting state
> > * drop Unplugged maintenance mode
> > * use UUID_FORMAT for uuid field
> > * a lot of small things, like use of bail!, inline format!, ...
> > * include improvement to cache handling
> >
> > v3:
> > * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
> > * fix CLI (un)mount command, thanks @Gabriel
> > * add removable datastore CLI autocomplete helper
> > * rebase onto master
> > * move ui patches to the end
> >
> > thanks @Lukas and @Thomas for the feedback
> > v2:
> > * fix datastore 'add' button in the UI
> > * some format!("{}", a) -> format!("{a}")
> > * replace `const` with `let` in js code
> > * change icon `fa-usb` -> `fa-plug`
> > * add some docs
> > * add JDoc for parseMaintenanceMode
> > * proxmox-schema dep bump
> >
> > Dietmar Maurer (2):
> > config: factor out method to get the absolute datastore path
> > maintenance: add 'Unmount' maintenance type
> >
> > Hannes Laimer (24):
> > tools: add disks utility functions
> > pbs-api-types: add backing-device to DataStoreConfig
> > disks: add UUID to partition info
> > datastore: add helper for checking if a removable datastore is
> > available
> > api: admin: add (un)mount endpoint for removable datastores
> > api: removable datastore creation
> > api: disks list: add exclude-used flag
> > pbs-api-types: add removable/is-available flag to DataStoreListItem
> > bin: manager: add (un)mount command
> > add auto-mounting for removable datastores
> > datastore: handle deletion of removable datastore properly
> > docs: add removable datastores section
> > ui: add partition selector form
> > ui: add removable datastore creation support
> > ui: add (un)mount button to summary
> > ui: tree: render unmounted datastores correctly
> > ui: utils: make parseMaintenanceMode more robust
> > ui: add datastore status mask for unmounted removable datastores
> > ui: maintenance: fix disable msg field if no type is selected
> > ui: render 'unmount' maintenance mode correctly
> > api: node: allow creation of removable datastore through directory
> > endpoint
> > api: node: include removable datastores in directory list
> > node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
> > ui: support create removable datastore through directory creation
> >
> > debian/proxmox-backup-server.install | 1 +
> > debian/proxmox-backup-server.udev | 3 +
> > docs/storage.rst | 29 +++
> > etc/Makefile | 3 +-
> > etc/removable-device-attach@.service.in | 8 +
> > pbs-api-types/src/datastore.rs | 42 +++-
> > pbs-api-types/src/maintenance.rs | 11 +-
> > pbs-config/src/datastore.rs | 14 ++
> > pbs-datastore/src/datastore.rs | 73 +++++-
> > pbs-datastore/src/lib.rs | 2 +-
> > src/api2/admin/datastore.rs | 260 ++++++++++++++++++--
> > src/api2/config/datastore.rs | 69 +++++-
> > src/api2/node/disks/directory.rs | 102 ++++++--
> > src/api2/node/disks/mod.rs | 8 +
> > src/api2/status.rs | 18 +-
> > src/bin/proxmox-backup-proxy.rs | 9 +-
> > src/bin/proxmox_backup_manager/datastore.rs | 129 +++++++++-
> > src/tools/disks/mod.rs | 99 +++++++-
> > www/DirectoryList.js | 13 +
> > www/Makefile | 1 +
> > www/NavigationTree.js | 15 +-
> > www/Utils.js | 33 ++-
> > www/css/ext6-pbs.css | 20 ++
> > www/datastore/DataStoreListSummary.js | 1 +
> > www/datastore/Summary.js | 113 ++++++++-
> > www/form/PartitionSelector.js | 81 ++++++
> > www/window/CreateDirectory.js | 14 ++
> > www/window/DataStoreEdit.js | 37 +++
> > www/window/MaintenanceOptions.js | 17 +-
> > 29 files changed, 1141 insertions(+), 84 deletions(-)
> > create mode 100644 etc/removable-device-attach@.service.in
> > create mode 100644 www/form/PartitionSelector.js
> >
> > --
> > 2.39.2
> >
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
> >
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 55+ messages in thread