* [pbs-devel] [PATCH proxmox v4 1/1] pbs-api-types: add run-on-mount flag to SyncJobConfig
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:27 ` Christian Ebner
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 1/7] api: config: sync: update run-on-mount correctly Hannes Laimer
` (8 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
pbs-api-types/src/jobs.rs | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/pbs-api-types/src/jobs.rs b/pbs-api-types/src/jobs.rs
index 6ef13dc2..3eb61cde 100644
--- a/pbs-api-types/src/jobs.rs
+++ b/pbs-api-types/src/jobs.rs
@@ -536,6 +536,8 @@ pub const SYNC_ENCRYPTED_ONLY_SCHEMA: Schema =
BooleanSchema::new("Only synchronize encrypted backup snapshots, exclude others.").schema();
pub const SYNC_VERIFIED_ONLY_SCHEMA: Schema =
BooleanSchema::new("Only synchronize verified backup snapshots, exclude others.").schema();
+pub const RUN_SYNC_ON_MOUNT_SCHEMA: Schema =
+ BooleanSchema::new("Run this job when a relevant datastore is mounted.").schema();
#[api(
properties: {
@@ -603,6 +605,10 @@ pub const SYNC_VERIFIED_ONLY_SCHEMA: Schema =
schema: SYNC_VERIFIED_ONLY_SCHEMA,
optional: true,
},
+ "run-on-mount": {
+ schema: RUN_SYNC_ON_MOUNT_SCHEMA,
+ optional: true,
+ },
"sync-direction": {
type: SyncDirection,
optional: true,
@@ -647,6 +653,8 @@ pub struct SyncJobConfig {
#[serde(skip_serializing_if = "Option::is_none")]
pub verified_only: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
+ pub run_on_mount: Option<bool>,
+ #[serde(skip_serializing_if = "Option::is_none")]
pub sync_direction: Option<SyncDirection>,
}
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 1/7] api: config: sync: update run-on-mount correctly
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox v4 1/1] pbs-api-types: add run-on-mount flag to SyncJobConfig Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:27 ` Christian Ebner
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 2/7] api: admin: run configured sync jobs when a datastore is mounted Hannes Laimer
` (7 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/api2/config/sync.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/api2/config/sync.rs b/src/api2/config/sync.rs
index 6194d865..358409b5 100644
--- a/src/api2/config/sync.rs
+++ b/src/api2/config/sync.rs
@@ -339,6 +339,8 @@ pub enum DeletableProperty {
EncryptedOnly,
/// Delete the verified_only property,
VerifiedOnly,
+ /// Delete the run_on_mount property,
+ RunOnMount,
/// Delete the sync_direction property,
SyncDirection,
}
@@ -458,6 +460,9 @@ pub fn update_sync_job(
DeletableProperty::VerifiedOnly => {
data.verified_only = None;
}
+ DeletableProperty::RunOnMount => {
+ data.run_on_mount = None;
+ }
DeletableProperty::SyncDirection => {
data.sync_direction = None;
}
@@ -507,6 +512,9 @@ pub fn update_sync_job(
if let Some(verified_only) = update.verified_only {
data.verified_only = Some(verified_only);
}
+ if let Some(run_on_mount) = update.run_on_mount {
+ data.run_on_mount = Some(run_on_mount);
+ }
if let Some(sync_direction) = update.sync_direction {
data.sync_direction = Some(sync_direction);
}
@@ -683,6 +691,7 @@ acl:1:/remote/remote1/remotestore1:write@pbs:RemoteSyncOperator
transfer_last: None,
encrypted_only: None,
verified_only: None,
+ run_on_mount: None,
sync_direction: None, // use default
};
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 2/7] api: admin: run configured sync jobs when a datastore is mounted
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox v4 1/1] pbs-api-types: add run-on-mount flag to SyncJobConfig Hannes Laimer
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 1/7] api: config: sync: update run-on-mount correctly Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:27 ` Christian Ebner
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 3/7] api: admin: trigger sync jobs only on datastore mount Hannes Laimer
` (6 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
When a datastore is mounted, spawn a new task to run all sync jobs
marked with `run-on-mount`. These jobs run sequentially and include
any job for which the mounted datastore is:
- The source or target in a local push/pull job
- The source in a push job to a remote datastore
- The target in a pull job from a remote datastore
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
new in v4:
- warp do_mount_device in .spawn_blocking(|| ...)
src/api2/admin/datastore.rs | 115 ++++++++++++++++++++++++++++++++++--
1 file changed, 109 insertions(+), 6 deletions(-)
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index e24bc1c1..753772f9 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -44,8 +44,8 @@ use pbs_api_types::{
DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
- BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
- BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
+ SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+ BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
@@ -68,7 +68,7 @@ use pbs_datastore::{
DataStore, LocalChunkReader, StoreProgress,
};
use pbs_tools::json::required_string_param;
-use proxmox_rest_server::{formatter, WorkerTask};
+use proxmox_rest_server::{formatter, worker_is_active, WorkerTask};
use crate::api2::backup::optional_ns_param;
use crate::api2::node::rrd::create_value_from_rrd;
@@ -2495,6 +2495,63 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
Ok(())
}
+async fn do_sync_jobs(
+ jobs_to_run: Vec<SyncJobConfig>,
+ worker: Arc<WorkerTask>,
+) -> Result<(), Error> {
+ let count = jobs_to_run.len();
+ info!(
+ "will run {count} sync jobs: {}",
+ jobs_to_run
+ .iter()
+ .map(|job| job.id.as_str())
+ .collect::<Vec<&str>>()
+ .join(", ")
+ );
+
+ let client = crate::client_helpers::connect_to_localhost()
+ .with_context(|| format!("Failed to connect to localhost for starting sync jobs"))?;
+ for (i, job_config) in jobs_to_run.into_iter().enumerate() {
+ if worker.abort_requested() {
+ bail!("aborted due to user request");
+ }
+ let job_id = &job_config.id;
+ let Ok(result) = client
+ .post(format!("api2/json/admin/sync/{job_id}/run").as_str(), None)
+ .await
+ else {
+ warn!("unable to start sync job {job_id}");
+ continue;
+ };
+ info!("[{}/{count}] starting '{job_id}'...", i + 1);
+ let Some(upid_str) = result["data"].as_str() else {
+ warn!(
+ "could not receive UPID of started job (may be running, just can't track it here)"
+ );
+ continue;
+ };
+ let upid: UPID = upid_str.parse()?;
+
+ let sleep_duration = core::time::Duration::from_secs(1);
+ let mut status_retries = 1;
+ loop {
+ if worker.abort_requested() {
+ bail!("aborted due to user request, already started job will finish");
+ }
+ match worker_is_active(&upid).await {
+ Ok(true) => tokio::time::sleep(sleep_duration).await,
+ Ok(false) => break,
+ Err(_) if status_retries > 3 => break,
+ Err(err) => {
+ warn!("could not get job status: {err} ({status_retries}/3)");
+ status_retries += 1;
+ }
+ }
+ }
+ }
+ Ok(())
+}
+
#[api(
protected: true,
input: {
@@ -2526,12 +2583,58 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
- let upid = WorkerTask::new_thread(
+ let upid = WorkerTask::spawn(
"mount-device",
- Some(store),
+ Some(store.clone()),
auth_id.to_string(),
to_stdout,
- move |_worker| do_mount_device(datastore),
+ move |_worker| async move {
+ let name = datastore.name.clone();
+ let log_context = LogContext::current();
+ tokio::task::spawn_blocking(|| {
+ if let Some(log_context) = log_context {
+ log_context.sync_scope(|| do_mount_device(datastore))
+ } else {
+ do_mount_device(datastore)
+ }
+ })
+ .await??;
+ let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
+ warn!("unable to read sync job config, won't run any sync jobs");
+ return Ok(());
+ };
+ let Ok(list) = sync_config.convert_to_typed_array("sync") else {
+ warn!("unable to parse sync job config, won't run any sync jobs");
+ return Ok(());
+ };
+ let mut jobs_to_run: Vec<SyncJobConfig> = list
+ .into_iter()
+ .filter(|job: &SyncJobConfig| {
+ // add job iff (running on mount is enabled and) any of these apply
+ // - the jobs is local and we are source or target
+ // - we are the source of a push to a remote
+ // - we are the target of a pull from a remote
+ //
+ // `job.store == datastore.name` iff we are the target for pull from remote or we
+ // are the source for push to remote, therefore we don't have to check for the
+ // direction of the job.
+ job.run_on_mount.unwrap_or(false)
+ && (job.remote.is_none() && job.remote_store == name || job.store == name)
+ })
+ .collect();
+ jobs_to_run.sort_by(|j1, j2| j1.id.cmp(&j2.id));
+ if !jobs_to_run.is_empty() {
+ info!("starting {} sync jobs", jobs_to_run.len());
+ let _ = WorkerTask::spawn(
+ "mount-sync-jobs",
+ Some(store),
+ auth_id.to_string(),
+ false,
+ move |worker| async move { do_sync_jobs(jobs_to_run, worker).await },
+ );
+ }
+ Ok(())
+ },
)?;
Ok(json!(upid))
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [pbs-devel] [PATCH proxmox-backup v4 2/7] api: admin: run configured sync jobs when a datastore is mounted
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 2/7] api: admin: run configured sync jobs when a datastore is mounted Hannes Laimer
@ 2025-07-17 13:27 ` Christian Ebner
0 siblings, 0 replies; 17+ messages in thread
From: Christian Ebner @ 2025-07-17 13:27 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Hannes Laimer
On 7/16/25 4:52 PM, Hannes Laimer wrote:
> When a datastore is mounted, spawn a new task to run all sync jobs
> marked with `run-on-mount`. These jobs run sequentially and include
> any job for which the mounted datastore is:
>
> - The source or target in a local push/pull job
just noticed this now, but there are no local push jobs, so this should
state pull only.
> - The source in a push job to a remote datastore
> - The target in a pull job from a remote datastore
>
> Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
> ---
> new in v4:
> - warp do_mount_device in .spawn_blocking(|| ...)
>
> src/api2/admin/datastore.rs | 115 ++++++++++++++++++++++++++++++++++--
> 1 file changed, 109 insertions(+), 6 deletions(-)
>
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index e24bc1c1..753772f9 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -44,8 +44,8 @@ use pbs_api_types::{
> DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
> GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
> MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
> - BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
> - BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
> + SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
> + BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
> IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
> PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
> PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
> @@ -68,7 +68,7 @@ use pbs_datastore::{
> DataStore, LocalChunkReader, StoreProgress,
> };
> use pbs_tools::json::required_string_param;
> -use proxmox_rest_server::{formatter, WorkerTask};
> +use proxmox_rest_server::{formatter, worker_is_active, WorkerTask};
>
> use crate::api2::backup::optional_ns_param;
> use crate::api2::node::rrd::create_value_from_rrd;
> @@ -2495,6 +2495,63 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
> Ok(())
> }
>
> +async fn do_sync_jobs(
> + jobs_to_run: Vec<SyncJobConfig>,
> + worker: Arc<WorkerTask>,
> +) -> Result<(), Error> {
> + let count = jobs_to_run.len();
> + info!(
> + "will run {count} sync jobs: {}",
> + jobs_to_run
> + .iter()
> + .map(|job| job.id.as_str())
> + .collect::<Vec<&str>>()
> + .join(", ")
> + );
> +
> + let client = crate::client_helpers::connect_to_localhost()
> + .with_context(|| format!("Failed to connect to localhost for starting sync jobs"))?;
nit: this contains a 'static &str only, so no need for the string
instantiation via format! and the with_context.
Should be a `.context("Failed to connect to localhost for starting sync
jobs")?;`
> + for (i, job_config) in jobs_to_run.into_iter().enumerate() {
> + if worker.abort_requested() {
> + bail!("aborted due to user request");
> + }
> + let job_id = &job_config.id;
> + let Ok(result) = client
> + .post(format!("api2/json/admin/sync/{job_id}/run").as_str(), None)
> + .await
> + else {
> + warn!("unable to start sync job {job_id}");
nit: this could include the actual error message with context for ease
of debugging!
> + continue;
> + };
> + info!("[{}/{count}] starting '{job_id}'...", i + 1);
nit: moving above log output to right before the api call to start the
job makes more sense I think.
so all in all:
```
let job_id = &job_config.id;
info!("[{}/{count}] starting '{job_id}'...", i + 1);
let result = match client
.post(format!("api2/json/admin/sync/{job_id}/run").as_str(), None)
.await
{
Ok(result) => result,
Err(err) => {
warn!("unable to start sync job {job_id}: {err:#}");
continue;
}
};
let Some(upid_str) = result["data"].as_str() else {
warn!(
"could not receive UPID of started job (may be running,
just can't track it here)"
);
continue;
};
```
> + let Some(upid_str) = result["data"].as_str() else {
> + warn!(
> + "could not receive UPID of started job (may be running, just can't track it here)"
> + );
> + continue;
> + };
> + let upid: UPID = upid_str.parse()?;
> +
> + let sleep_duration = core::time::Duration::from_secs(1);
> + let mut status_retries = 1;
> + loop {
> + if worker.abort_requested() {
> + bail!("aborted due to user request, already started job will finish");
> + }
> + match worker_is_active(&upid).await {
> + Ok(true) => tokio::time::sleep(sleep_duration).await,
> + Ok(false) => break,
> + Err(_) if status_retries > 3 => break,
> + Err(err) => {
> + warn!("could not get job status: {err} ({status_retries}/3)");
> + status_retries += 1;
> + }
> + }
> + }
> + }
> + Ok(())
> +}
> +
> #[api(
> protected: true,
> input: {
> @@ -2526,12 +2583,58 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
> let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>
> - let upid = WorkerTask::new_thread(
> + let upid = WorkerTask::spawn(
> "mount-device",
> - Some(store),
> + Some(store.clone()),
> auth_id.to_string(),
> to_stdout,
> - move |_worker| do_mount_device(datastore),
> + move |_worker| async move {
> + let name = datastore.name.clone();
> + let log_context = LogContext::current();
> + tokio::task::spawn_blocking(|| {
> + if let Some(log_context) = log_context {
> + log_context.sync_scope(|| do_mount_device(datastore))
> + } else {
> + do_mount_device(datastore)
> + }
> + })
> + .await??;
> + let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
> + warn!("unable to read sync job config, won't run any sync jobs");
> + return Ok(());
> + };
> + let Ok(list) = sync_config.convert_to_typed_array("sync") else {
> + warn!("unable to parse sync job config, won't run any sync jobs");
> + return Ok(());
> + };
> + let mut jobs_to_run: Vec<SyncJobConfig> = list
> + .into_iter()
> + .filter(|job: &SyncJobConfig| {
> + // add job iff (running on mount is enabled and) any of these apply
> + // - the jobs is local and we are source or target
> + // - we are the source of a push to a remote
> + // - we are the target of a pull from a remote
> + //
> + // `job.store == datastore.name` iff we are the target for pull from remote or we
> + // are the source for push to remote, therefore we don't have to check for the
> + // direction of the job.
> + job.run_on_mount.unwrap_or(false)
> + && (job.remote.is_none() && job.remote_store == name || job.store == name)
> + })
> + .collect();
> + jobs_to_run.sort_by(|j1, j2| j1.id.cmp(&j2.id));
> + if !jobs_to_run.is_empty() {
> + info!("starting {} sync jobs", jobs_to_run.len());
> + let _ = WorkerTask::spawn(
> + "mount-sync-jobs",
> + Some(store),
> + auth_id.to_string(),
> + false,
> + move |worker| async move { do_sync_jobs(jobs_to_run, worker).await },
> + );
> + }
> + Ok(())
> + },
> )?;
>
> Ok(json!(upid))
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v4 3/7] api: admin: trigger sync jobs only on datastore mount
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (2 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 2/7] api: admin: run configured sync jobs when a datastore is mounted Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 4/7] bin: manager: run uuid_mount/mount tasks on the proxy Hannes Laimer
` (5 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Ensure sync jobs are triggered only when the datastore is actually
mounted. If the datastore is already mounted, we don't fail,
but sync jobs should not be re-triggered unnecessarily. This change
prevents redundant sync job execution.
---
src/api2/admin/datastore.rs | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 753772f9..9fed22cc 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -2440,14 +2440,14 @@ fn setup_mounted_device(datastore: &DataStoreConfig, tmp_mount_path: &str) -> Re
/// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
/// the same path, this is *very* unlikely since the device is only mounted really shortly, but
/// technically possible.
-pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
+pub fn do_mount_device(datastore: DataStoreConfig) -> Result<bool, Error> {
if let Some(uuid) = datastore.backing_device.as_ref() {
if pbs_datastore::get_datastore_mount_status(&datastore) == Some(true) {
info!(
"device is already mounted at '{}'",
datastore.absolute_path()
);
- return Ok(());
+ return Ok(false);
}
let tmp_mount_path = format!(
"{}/{:x}",
@@ -2492,7 +2492,7 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
datastore.name
)
}
- Ok(())
+ Ok(true)
}
async fn do_sync_jobs(
@@ -2591,14 +2591,17 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
move |_worker| async move {
let name = datastore.name.clone();
let log_context = LogContext::current();
- tokio::task::spawn_blocking(|| {
+ if !tokio::task::spawn_blocking(|| {
if let Some(log_context) = log_context {
log_context.sync_scope(|| do_mount_device(datastore))
} else {
do_mount_device(datastore)
}
})
- .await??;
+ .await??
+ {
+ return Ok(());
+ }
let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
warn!("unable to read sync job config, won't run any sync jobs");
return Ok(());
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 4/7] bin: manager: run uuid_mount/mount tasks on the proxy
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (3 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 3/7] api: admin: trigger sync jobs only on datastore mount Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 5/7] ui: add run-on-mount checkbox to SyncJob form Hannes Laimer
` (4 subsequent siblings)
9 siblings, 0 replies; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Use the API instead of running uuid_mount/mount directly in the CLI binary.
This ensures that all triggered tasks are handled by the proxy process.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
src/bin/proxmox_backup_manager/datastore.rs | 41 ++++++++++++++++-----
1 file changed, 31 insertions(+), 10 deletions(-)
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 1922a55a..3fbb5fe5 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -49,6 +49,10 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
store: {
schema: DATASTORE_SCHEMA,
},
+ "output-format": {
+ schema: OUTPUT_FORMAT,
+ optional: true,
+ },
digest: {
optional: true,
schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
@@ -57,16 +61,23 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
},
)]
/// Mount a removable datastore.
-async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
- param["node"] = "localhost".into();
+async fn mount_datastore(
+ store: String,
+ mut param: Value,
+ _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+ let output_format = extract_output_format(&mut param);
- let info = &api2::admin::datastore::API_METHOD_MOUNT;
- let result = match info.handler {
- ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
- _ => unreachable!(),
- };
+ let client = connect_to_localhost()?;
+ let result = client
+ .post(
+ format!("api2/json/admin/datastore/{store}/mount").as_str(),
+ None,
+ )
+ .await?;
+
+ view_task_result(&client, result, &output_format).await?;
- crate::wait_for_local_worker(result.as_str().unwrap()).await?;
Ok(())
}
@@ -260,7 +271,8 @@ async fn update_datastore(name: String, mut param: Value) -> Result<(), Error> {
},
)]
/// Try mounting a removable datastore given the UUID.
-async fn uuid_mount(param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+async fn uuid_mount(mut param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+ let output_format = extract_output_format(&mut param);
let uuid = param["uuid"]
.as_str()
.ok_or_else(|| format_err!("uuid has to be specified"))?;
@@ -282,7 +294,16 @@ async fn uuid_mount(param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Va
}
if let Some(store) = matching_stores.first() {
- api2::admin::datastore::do_mount_device(store.clone())?;
+ let client = connect_to_localhost()?;
+ let result = client
+ .post(
+ format!("api2/json/admin/datastore/{}/mount", store.name).as_str(),
+ None,
+ )
+ .await?;
+
+ view_task_result(&client, result, &output_format).await?;
+ return Ok(Value::Null);
}
// we don't want to fail for UUIDs that are not associated with datastores, as that produces
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 5/7] ui: add run-on-mount checkbox to SyncJob form
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (4 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 4/7] bin: manager: run uuid_mount/mount tasks on the proxy Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:33 ` Christian Ebner
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 6/7] ui: add task title for triggering sync jobs Hannes Laimer
` (3 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
new in v4:
- delete iff `false`
www/window/SyncJobEdit.js | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/www/window/SyncJobEdit.js b/www/window/SyncJobEdit.js
index 0daae9ba..f430db0a 100644
--- a/www/window/SyncJobEdit.js
+++ b/www/window/SyncJobEdit.js
@@ -121,6 +121,7 @@ Ext.define('PBS.window.SyncJobEdit', {
values.id = 's-' + Ext.data.identifier.Uuid.Global.generate().slice(0, 13);
}
if (!me.isCreate) {
+ PBS.Utils.delete_if_default(values, 'run-on-mount', false);
PBS.Utils.delete_if_default(values, 'rate-in');
PBS.Utils.delete_if_default(values, 'rate-out');
PBS.Utils.delete_if_default(values, 'remote');
@@ -205,7 +206,7 @@ Ext.define('PBS.window.SyncJobEdit', {
xtype: 'pbsCalendarEvent',
name: 'schedule',
fieldLabel: gettext('Sync Schedule'),
- emptyText: gettext('none (disabled)'),
+ emptyText: gettext('none'),
cbind: {
deleteEmpty: '{!isCreate}',
value: '{scheduleValue}',
@@ -488,6 +489,19 @@ Ext.define('PBS.window.SyncJobEdit', {
uncheckedValue: false,
value: false,
},
+ {
+ xtype: 'proxmoxcheckbox',
+ name: 'run-on-mount',
+ fieldLabel: gettext('Run when mounted'),
+ autoEl: {
+ tag: 'div',
+ 'data-qtip': gettext(
+ 'Run this job when a relevant removable datastore is mounted.',
+ ),
+ },
+ uncheckedValue: false,
+ value: false,
+ },
],
},
{
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 6/7] ui: add task title for triggering sync jobs
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (5 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 5/7] ui: add run-on-mount checkbox to SyncJob form Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:30 ` Christian Ebner
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 7/7] docs: add section about `run-on-mount` sync job flag Hannes Laimer
` (2 subsequent siblings)
9 siblings, 1 reply; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
new in v4:
- updated title
www/Utils.js | 1 +
1 file changed, 1 insertion(+)
diff --git a/www/Utils.js b/www/Utils.js
index 30b4a6e7..ef430636 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -441,6 +441,7 @@ Ext.define('PBS.Utils', {
prunejob: (type, id) => PBS.Utils.render_prune_job_worker_id(id, gettext('Prune Job')),
reader: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Read Objects')),
'rewind-media': [gettext('Drive'), gettext('Rewind Media')],
+ 'mount-sync-jobs': [gettext('Datastore'), gettext('sync jobs handler triggered by mount')],
sync: ['Datastore', gettext('Remote Sync')],
syncjob: [gettext('Sync Job'), gettext('Remote Sync')],
'tape-backup': (type, id) =>
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* [pbs-devel] [PATCH proxmox-backup v4 7/7] docs: add section about `run-on-mount` sync job flag
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (6 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 6/7] ui: add task title for triggering sync jobs Hannes Laimer
@ 2025-07-16 14:52 ` Hannes Laimer
2025-07-17 13:33 ` [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Christian Ebner
2025-07-21 14:38 ` [pbs-devel] superseded: " Hannes Laimer
9 siblings, 0 replies; 17+ messages in thread
From: Hannes Laimer @ 2025-07-16 14:52 UTC (permalink / raw)
To: pbs-devel
---
new, did not exists in v3
docs/managing-remotes.rst | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/docs/managing-remotes.rst b/docs/managing-remotes.rst
index ddbe3e9b..b6ebda18 100644
--- a/docs/managing-remotes.rst
+++ b/docs/managing-remotes.rst
@@ -145,6 +145,11 @@ job needs to be run before a sync job with 'resync-corrupt' can be carried out.
that a 'resync-corrupt'-job needs to check the manifests of all snapshots in a datastore
and might take much longer than regular sync jobs.
+If the ``run-on-mount`` flag is set, the sync job will be automatically started whenever a
+relevant removable datastore is mounted. If mounting a removable datastore would start
+multiple sync jobs, these jobs will be run sequentially in alphabetical order based on
+their ID.
+
Namespace Support
^^^^^^^^^^^^^^^^^
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (7 preceding siblings ...)
2025-07-16 14:52 ` [pbs-devel] [PATCH proxmox-backup v4 7/7] docs: add section about `run-on-mount` sync job flag Hannes Laimer
@ 2025-07-17 13:33 ` Christian Ebner
2025-07-21 14:38 ` [pbs-devel] superseded: " Hannes Laimer
9 siblings, 0 replies; 17+ messages in thread
From: Christian Ebner @ 2025-07-17 13:33 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Hannes Laimer
On 7/16/25 4:52 PM, Hannes Laimer wrote:
> Sync jobs now have a run-on-mount flag, that, if set, runs the job whenever
> a relevant removable datastore is mounted.
>
> changes since v3, thanks @Chris:
> - wrap potentially blocking call to mount function in .spawn_blocking(||
> ...)
> - ui: improve task title, delete `run-on-mount` flag iff `false`
> - docs: add section about `run-on-mount` flag
> - some minor things(typos, move stuff out of loop)
> - rebased onto master
> - ui: fixed formatting with `proxmox-biome`
A few more comments on some individual patches, but nothing big.
Other than that consider:
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Tested the following:
- Sync jobs are executed when the config flag is set
- The order of the executed sync jobs is alphabetical
- The run-on-mount flag is cleared from the config when uncheck
- The mount command via cli behaves the same as mounting via ui/api
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread* Re: [pbs-devel] superseded: [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount
2025-07-16 14:52 [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Hannes Laimer
` (8 preceding siblings ...)
2025-07-17 13:33 ` [pbs-devel] [PATCH proxmox{, -backup} v4 0/8] trigger sync jobs on mount Christian Ebner
@ 2025-07-21 14:38 ` Hannes Laimer
9 siblings, 0 replies; 17+ messages in thread
From: Hannes Laimer @ 2025-07-21 14:38 UTC (permalink / raw)
To: pbs-devel
superseded-by v5:
https://lore.proxmox.com/pbs-devel/20250721113314.59342-1-h.laimer@proxmox.com/T/#t
On 16.07.25 16:52, Hannes Laimer wrote:
> Sync jobs now have a run-on-mount flag, that, if set, runs the job whenever
> a relevant removable datastore is mounted.
>
> changes since v3, thanks @Chris:
> - wrap potentially blocking call to mount function in .spawn_blocking(||
> ...)
> - ui: improve task title, delete `run-on-mount` flag iff `false`
> - docs: add section about `run-on-mount` flag
> - some minor things(typos, move stuff out of loop)
> - rebased onto master
> - ui: fixed formatting with `proxmox-biome`
>
>
> proxmox:
>
> Hannes Laimer (1):
> pbs-api-types: add run-on-mount flag to SyncJobConfig
>
> pbs-api-types/src/jobs.rs | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
>
> proxmox-backup:
>
> Hannes Laimer (7):
> api: config: sync: update run-on-mount correctly
> api: admin: run configured sync jobs when a datastore is mounted
> api: admin: trigger sync jobs only on datastore mount
> bin: manager: run uuid_mount/mount tasks on the proxy
> ui: add run-on-mount checkbox to SyncJob form
> ui: add task title for triggering sync jobs
> docs: add section about `run-on-mount` sync job flag
>
> docs/managing-remotes.rst | 5 +
> src/api2/admin/datastore.rs | 122 ++++++++++++++++++--
> src/api2/config/sync.rs | 9 ++
> src/bin/proxmox_backup_manager/datastore.rs | 41 +++++--
> www/Utils.js | 1 +
> www/window/SyncJobEdit.js | 16 ++-
> 6 files changed, 175 insertions(+), 19 deletions(-)
>
>
> Summary over all repositories:
> 7 files changed, 183 insertions(+), 19 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 17+ messages in thread