[pdm-devel] [PATCH datacenter-manager 0/7] PBS remotes: task API and task cache support

[pdm-devel] [PATCH datacenter-manager 0/7] PBS remotes: task API and task cache support

[Lukas Wagner]

NOTE: This one is based of my series which changes the RemoteUpid to include
the remote type [0]. If that RFC is rejected, I will rebase onto master (and
try to solve the problem that the RFC is trying to fixin some other way). If
the RFC is accepted, I will probably merge both series into one for easier
handling.

This series adds the necessary bindings to the PBS client for basic task
retrieval as required by the task cache and the remote task overview in the UI.

[0] https://lore.proxmox.com/pdm-devel/20251111105059.148997-1-l.wagner@proxmox.com/T/#t


proxmox-datacenter-manager:

Lukas Wagner (7):
  pdm-api-types: remote upid: add helpers for getting native UPID type
  pbs-client: add bindings for task list, task status, task log
  pdm-api-types: api: factor out schema definitions for task log params
  api: pbs tasks: add PBS task API
  api: pve tasks: use shared helpers for RemoteUpid handling
  remote tasks: fetch/track PBS tasks
  remote updates: re-enable PBS update fetching

 lib/pdm-api-types/src/lib.rs                  |  21 +++
 lib/pdm-api-types/src/remote_upid.rs          |  22 +++
 server/Cargo.toml                             |   1 +
 server/src/api/mod.rs                         |  22 ++-
 server/src/api/nodes/tasks.rs                 |  30 +---
 server/src/api/pbs/mod.rs                     |   4 +-
 server/src/api/pbs/tasks.rs                   | 170 ++++++++++++++++++
 server/src/api/pve/tasks.rs                   |  97 +++-------
 .../tasks/remote_tasks.rs                     |  98 +++++++---
 server/src/pbs_client.rs                      | 120 ++++++++++++-
 server/src/remote_updates.rs                  |  23 ++-
 11 files changed, 468 insertions(+), 140 deletions(-)
 create mode 100644 server/src/api/pbs/tasks.rs


Summary over all repositories:
  11 files changed, 468 insertions(+), 140 deletions(-)

-- 
Generated by murpp 0.9.0


_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 1/7] pdm-api-types: remote upid: add helpers for getting native UPID type

[Lukas Wagner]

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 lib/pdm-api-types/src/remote_upid.rs | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/lib/pdm-api-types/src/remote_upid.rs b/lib/pdm-api-types/src/remote_upid.rs
index 10d3c1dc..080837db 100644
--- a/lib/pdm-api-types/src/remote_upid.rs
+++ b/lib/pdm-api-types/src/remote_upid.rs
@@ -70,6 +70,28 @@ impl RemoteUpid {
         })
     }
 
+    /// Get the parsed PVE UPID.
+    ///
+    /// If the UPID could not be parsed, or has an unexpected format (PBS),
+    /// an error is returned.
+    pub fn pve_upid(&self) -> Result<pve_api_types::PveUpid, Error> {
+        match self.native_upid()? {
+            NativeUpid::PveUpid(pve_upid) => Ok(pve_upid),
+            NativeUpid::PbsUpid(_) => bail!("got a PBS UPID when expecting a PVE UPID"),
+        }
+    }
+
+    /// Get the parsed PBS UPID.
+    ///
+    /// If the UPID could not be parsed, or has an unexpected format (PVE),
+    /// an error is returned.
+    pub fn pbs_upid(&self) -> Result<pbs_api_types::UPID, Error> {
+        match self.native_upid()? {
+            NativeUpid::PveUpid(_) => bail!("got a PVE UPID when expecting a PBS UPID"),
+            NativeUpid::PbsUpid(pbs_upid) => Ok(pbs_upid),
+        }
+    }
+
     fn deduce_type(raw_upid: &str) -> Result<RemoteType, Error> {
         if raw_upid.parse::<pve_api_types::PveUpid>().is_ok() {
             Ok(RemoteType::Pve)
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 2/7] pbs-client: add bindings for task list, task status, task log

[Lukas Wagner]

Add the necessary bindings for remote task management for PBS remotes.
Some of the parameter types / return types should be moved to a shared
crate, but this involves some refactoring in PBS, which is not the
priority now. The current implementation is just enough to get the
remote task fetching and tracking working for PBS remotes.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/Cargo.toml        |   1 +
 server/src/pbs_client.rs | 120 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 119 insertions(+), 2 deletions(-)

diff --git a/server/Cargo.toml b/server/Cargo.toml
index a215aaf2..b336f1a0 100644
--- a/server/Cargo.toml
+++ b/server/Cargo.toml
@@ -27,6 +27,7 @@ openssl.workspace = true
 percent-encoding.workspace = true
 serde.workspace = true
 serde_json.workspace = true
+serde_plain.workspace = true
 syslog.workspace = true
 tokio = { workspace = true, features = [ "fs", "io-util", "io-std", "macros", "net", "parking_lot", "process", "rt", "rt-multi-thread", "signal", "time" ] }
 tokio-stream.workspace = true
diff --git a/server/src/pbs_client.rs b/server/src/pbs_client.rs
index 49087360..4285f402 100644
--- a/server/src/pbs_client.rs
+++ b/server/src/pbs_client.rs
@@ -6,9 +6,9 @@
 
 use anyhow::bail; // don't import Error as default error in here
 use http_body_util::BodyExt;
-use serde::Deserialize;
+use serde::{Deserialize, Serialize};
 
-use proxmox_client::{ApiPathBuilder, Error, HttpApiClient};
+use proxmox_client::{ApiPathBuilder, ApiResponseData, Error, HttpApiClient};
 use proxmox_router::stream::JsonRecords;
 use proxmox_schema::api;
 use proxmox_section_config::typed::SectionConfigData;
@@ -125,6 +125,75 @@ pub struct AptUpdateParams {
     pub quiet: Option<bool>,
 }
 
+// TODO: This is incomplete, it only contains the parameters needed for remote task fetching.
+// Ideally, the task list API in PBS would use a parameter struct defined in pbs-api-types, which
+// is then also used here.
+#[derive(Debug, Default, Deserialize, Serialize)]
+pub struct ListTasks {
+    /// Only list this number of tasks.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub limit: Option<u64>,
+
+    /// Only list tasks since this UNIX epoch.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub since: Option<i64>,
+}
+
+// TODO: The task-status APIs in PBS as well as PDM don't have a
+// proper type defined anywhere. This should be moved to a shared crate
+// and then the API handlers adapted.
+#[derive(Debug, Deserialize, Serialize)]
+pub struct TaskStatus {
+    pub exitstatus: Option<String>,
+
+    pub id: Option<String>,
+
+    pub node: String,
+
+    pub pid: i64,
+
+    pub pstart: i64,
+
+    pub starttime: i64,
+
+    pub status: IsRunning,
+
+    #[serde(rename = "type")]
+    pub ty: String,
+
+    pub upid: String,
+
+    pub user: String,
+}
+
+#[derive(Debug, Deserialize, Serialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+pub enum IsRunning {
+    Running,
+    Stopped,
+}
+
+impl TaskStatus {
+    /// Checks if the task is currently running.
+    pub fn is_running(&self) -> bool {
+        self.status == IsRunning::Running
+    }
+}
+
+#[api]
+// TODO: The task-status APIs in PBS as well as PDM don't have a
+// proper type defined anywhere. This should be moved to a shared crate
+// and then the API handlers adapted.
+/// One line in the task log.
+#[derive(Debug, serde::Deserialize, serde::Serialize)]
+pub struct TaskLogLine {
+    /// Line number
+    pub n: i64,
+
+    /// Line text
+    pub t: String,
+}
+
 impl PbsClient {
     /// API version details, including some parts of the global datacenter config.
     pub async fn version(&self) -> Result<pve_api_types::VersionResponse, Error> {
@@ -318,6 +387,53 @@ impl PbsClient {
 
         Ok(self.0.get(&path).await?.expect_json()?.data)
     }
+
+    /// Get list of tasks.
+    ///
+    /// `params`: Filters specifying which tasks to get.
+    pub async fn get_task_list(
+        &self,
+        params: ListTasks,
+    ) -> Result<Vec<pbs_api_types::TaskListItem>, Error> {
+        let ListTasks { limit, since } = params;
+
+        let url = ApiPathBuilder::new("/api2/extjs/nodes/localhost/tasks".to_string())
+            .maybe_arg("limit", &limit)
+            .maybe_arg("since", &since)
+            .build();
+
+        Ok(self.0.get(&url).await?.expect_json()?.data)
+    }
+
+    /// Read task log.
+    pub async fn get_task_log(
+        &self,
+        upid: &str,
+        download: Option<bool>,
+        limit: Option<u64>,
+        start: Option<u64>,
+    ) -> Result<ApiResponseData<Vec<TaskLogLine>>, Error> {
+        let url = ApiPathBuilder::new(format!("/api2/extjs/nodes/localhost/tasks/{upid}/log"))
+            .maybe_bool_arg("download", download)
+            .maybe_arg("limit", &limit)
+            .maybe_arg("start", &start)
+            .build();
+
+        self.0.get(&url).await?.expect_json()
+    }
+
+    /// Read task status.
+    pub async fn get_task_status(&self, upid: &str) -> Result<TaskStatus, Error> {
+        let url = format!("/api2/extjs/nodes/localhost/tasks/{upid}/status");
+        let response = self.0.get(&url).await?;
+        Ok(response.expect_json()?.data)
+    }
+
+    /// Stop a task.
+    pub async fn stop_task(&self, upid: &str) -> Result<(), Error> {
+        let url = format!("/api2/extjs/nodes/localhost/tasks/{upid}");
+        self.0.delete(&url).await?.nodata()
+    }
 }
 
 #[derive(Deserialize)]
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 3/7] pdm-api-types: api: factor out schema definitions for task log params

[Lukas Wagner]

The PBS tasks API will also need them, so it makes sense to move them to
a shared crate now.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 lib/pdm-api-types/src/lib.rs  | 21 +++++++++++++++++++++
 server/src/api/nodes/tasks.rs | 30 +++++-------------------------
 server/src/api/pve/tasks.rs   | 35 ++++++++---------------------------
 3 files changed, 34 insertions(+), 52 deletions(-)

diff --git a/lib/pdm-api-types/src/lib.rs b/lib/pdm-api-types/src/lib.rs
index ad52372e..5ac99aa1 100644
--- a/lib/pdm-api-types/src/lib.rs
+++ b/lib/pdm-api-types/src/lib.rs
@@ -494,3 +494,24 @@ pub struct TaskFilters {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub statusfilter: Option<Vec<TaskStateType>>,
 }
+
+pub const TASKLOG_START_PARAM_SCHEMA: Schema =
+    proxmox_schema::IntegerSchema::new("Start at this line when reading the tasklog")
+        .minimum(0)
+        .default(0)
+        .schema();
+
+pub const TASKLOG_LIMIT_PARAM_SCHEMA: Schema = proxmox_schema::IntegerSchema::new(
+    "The amount of lines to read from the tasklog. \
+         Setting this parameter to 0 will return all lines until the end of the file.",
+)
+.minimum(0)
+.default(50)
+.schema();
+
+pub const TASKLOG_DOWNLOAD_PARAM_SCHEMA: Schema = proxmox_schema::BooleanSchema::new(
+    "Whether the tasklog file should be downloaded. \
+        This parameter can't be used in conjunction with other parameters",
+)
+.default(false)
+.schema();
diff --git a/server/src/api/nodes/tasks.rs b/server/src/api/nodes/tasks.rs
index 2ab1284f..9725f022 100644
--- a/server/src/api/nodes/tasks.rs
+++ b/server/src/api/nodes/tasks.rs
@@ -13,7 +13,8 @@ use proxmox_sortable_macro::sortable;
 
 use pdm_api_types::{
     Authid, TaskFilters, TaskListItem, TaskStateType, Tokenname, Userid, NODE_SCHEMA,
-    PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
+    PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, TASKLOG_DOWNLOAD_PARAM_SCHEMA, TASKLOG_LIMIT_PARAM_SCHEMA,
+    TASKLOG_START_PARAM_SCHEMA, UPID, UPID_SCHEMA,
 };
 
 pub const ROUTER: Router = Router::new()
@@ -305,27 +306,6 @@ async fn get_task_status(upid: UPID, rpcenv: &mut dyn RpcEnvironment) -> Result<
     Ok(result)
 }
 
-const START_PARAM_SCHEMA: Schema =
-    proxmox_schema::IntegerSchema::new("Start at this line when reading the tasklog")
-        .minimum(0)
-        .default(0)
-        .schema();
-
-const LIMIT_PARAM_SCHEMA: Schema = proxmox_schema::IntegerSchema::new(
-    "The amount of lines to read from the tasklog. \
-         Setting this parameter to 0 will return all lines until the end of the file.",
-)
-.minimum(0)
-.default(50)
-.schema();
-
-const DOWNLOAD_PARAM_SCHEMA: Schema = proxmox_schema::BooleanSchema::new(
-    "Whether the tasklog file should be downloaded. \
-        This parameter can't be used in conjunction with other parameters",
-)
-.default(false)
-.schema();
-
 const TEST_STATUS_PARAM_SCHEMA: Schema = proxmox_schema::BooleanSchema::new(
     "Test task status, and set result attribute \"active\" accordingly.",
 )
@@ -339,9 +319,9 @@ pub const API_METHOD_READ_TASK_LOG: proxmox_router::ApiMethod = proxmox_router::
         &sorted!([
             ("node", false, &NODE_SCHEMA),
             ("upid", false, &UPID_SCHEMA),
-            ("start", true, &START_PARAM_SCHEMA),
-            ("limit", true, &LIMIT_PARAM_SCHEMA),
-            ("download", true, &DOWNLOAD_PARAM_SCHEMA),
+            ("start", true, &TASKLOG_START_PARAM_SCHEMA),
+            ("limit", true, &TASKLOG_LIMIT_PARAM_SCHEMA),
+            ("download", true, &TASKLOG_DOWNLOAD_PARAM_SCHEMA),
             ("test-status", true, &TEST_STATUS_PARAM_SCHEMA)
         ]),
     ),
diff --git a/server/src/api/pve/tasks.rs b/server/src/api/pve/tasks.rs
index 9f04a448..e66712ac 100644
--- a/server/src/api/pve/tasks.rs
+++ b/server/src/api/pve/tasks.rs
@@ -3,11 +3,14 @@
 use anyhow::{bail, format_err, Error};
 
 use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
-use proxmox_schema::{api, Schema};
+use proxmox_schema::api;
 use proxmox_sortable_macro::sortable;
 
 use pdm_api_types::remotes::REMOTE_ID_SCHEMA;
-use pdm_api_types::{RemoteUpid, NODE_SCHEMA, PRIV_RESOURCE_AUDIT, PRIV_RESOURCE_MANAGE};
+use pdm_api_types::{
+    RemoteUpid, NODE_SCHEMA, PRIV_RESOURCE_AUDIT, PRIV_RESOURCE_MANAGE,
+    TASKLOG_DOWNLOAD_PARAM_SCHEMA, TASKLOG_LIMIT_PARAM_SCHEMA, TASKLOG_START_PARAM_SCHEMA,
+};
 use pve_api_types::PveUpid;
 
 use super::{connect, connect_to_remote, get_remote};
@@ -149,28 +152,6 @@ pub async fn get_task_status(
     }
 }
 
-// FIXME: Deduplicate these into pdm_api_types:
-const START_PARAM_SCHEMA: Schema =
-    proxmox_schema::IntegerSchema::new("Start at this line when reading the tasklog")
-        .minimum(0)
-        .default(0)
-        .schema();
-
-const LIMIT_PARAM_SCHEMA: Schema = proxmox_schema::IntegerSchema::new(
-    "The amount of lines to read from the tasklog. \
-         Setting this parameter to 0 will return all lines until the end of the file.",
-)
-.minimum(0)
-.default(50)
-.schema();
-
-const DOWNLOAD_PARAM_SCHEMA: Schema = proxmox_schema::BooleanSchema::new(
-    "Whether the tasklog file should be downloaded. \
-        This parameter can't be used in conjunction with other parameters",
-)
-.default(false)
-.schema();
-
 // FIXME: make *actually* streaming with router support!
 #[api(
     input: {
@@ -178,15 +159,15 @@ const DOWNLOAD_PARAM_SCHEMA: Schema = proxmox_schema::BooleanSchema::new(
             remote: { schema: REMOTE_ID_SCHEMA },
             upid: { type: RemoteUpid },
             start: {
-                schema: START_PARAM_SCHEMA,
+                schema: TASKLOG_START_PARAM_SCHEMA,
                 optional: true,
             },
             limit: {
-                schema: LIMIT_PARAM_SCHEMA,
+                schema: TASKLOG_LIMIT_PARAM_SCHEMA,
                 optional: true,
             },
             download: {
-                schema: DOWNLOAD_PARAM_SCHEMA,
+                schema: TASKLOG_DOWNLOAD_PARAM_SCHEMA,
                 optional: true,
             }
         },
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 4/7] api: pbs tasks: add PBS task API

[Lukas Wagner]

Mostly based off of the PVE task API with the necessary changes for PBS.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/src/api/mod.rs       |  22 ++++-
 server/src/api/pbs/mod.rs   |   4 +-
 server/src/api/pbs/tasks.rs | 170 ++++++++++++++++++++++++++++++++++++
 3 files changed, 194 insertions(+), 2 deletions(-)
 create mode 100644 server/src/api/pbs/tasks.rs

diff --git a/server/src/api/mod.rs b/server/src/api/mod.rs
index 6a7a65a2..119d57b7 100644
--- a/server/src/api/mod.rs
+++ b/server/src/api/mod.rs
@@ -1,6 +1,7 @@
 //! Common API endpoints
 
-use anyhow::Error;
+use anyhow::{bail, Error};
+use pdm_api_types::{remotes::RemoteType, RemoteUpid};
 use serde_json::{json, Value};
 
 use proxmox_router::{list_subdirs_api_method, Permission, Router, SubdirMap};
@@ -67,3 +68,22 @@ fn version() -> Result<Value, Error> {
         "repoid": pdm_buildcfg::PROXMOX_PKG_REPOID
     }))
 }
+
+/// Check a [`RemoteUpid`] matches the expected remote name and type.
+pub(crate) fn verify_upid(
+    remote: &str,
+    remote_type: RemoteType,
+    upid: &RemoteUpid,
+) -> Result<(), Error> {
+    if upid.remote() != remote {
+        bail!(
+            "remote '{remote}' does not match remote in upid ('{}')",
+            upid.remote()
+        );
+    }
+    if upid.remote_type() != remote_type {
+        bail!("upid does not belong to a {remote_type} remote");
+    }
+
+    Ok(())
+}
diff --git a/server/src/api/pbs/mod.rs b/server/src/api/pbs/mod.rs
index a37fadb5..15e42727 100644
--- a/server/src/api/pbs/mod.rs
+++ b/server/src/api/pbs/mod.rs
@@ -22,6 +22,7 @@ use crate::remote_tasks;
 
 mod node;
 mod rrddata;
+pub mod tasks;
 
 pub const ROUTER: Router = Router::new()
     .get(&list_subdirs_api_method!(SUBDIRS))
@@ -49,7 +50,8 @@ const REMOTE_SUBDIRS: SubdirMap = &sorted!([
     ("nodes", &NODES_ROUTER),
     ("status", &Router::new().get(&API_METHOD_GET_STATUS)),
     ("rrddata", &rrddata::PBS_NODE_RRD_ROUTER),
-    ("datastore", &DATASTORE_ROUTER)
+    ("datastore", &DATASTORE_ROUTER),
+    ("tasks", &tasks::ROUTER),
 ]);
 
 const DATASTORE_ROUTER: Router = Router::new()
diff --git a/server/src/api/pbs/tasks.rs b/server/src/api/pbs/tasks.rs
new file mode 100644
index 00000000..5c86bff8
--- /dev/null
+++ b/server/src/api/pbs/tasks.rs
@@ -0,0 +1,170 @@
+//! Access to PBS tasks.
+
+use anyhow::Error;
+
+use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
+use proxmox_schema::api;
+use proxmox_sortable_macro::sortable;
+
+use pdm_api_types::remotes::{RemoteType, REMOTE_ID_SCHEMA};
+use pdm_api_types::{
+    RemoteUpid, PRIV_RESOURCE_AUDIT, PRIV_RESOURCE_MANAGE, TASKLOG_DOWNLOAD_PARAM_SCHEMA,
+    TASKLOG_LIMIT_PARAM_SCHEMA, TASKLOG_START_PARAM_SCHEMA,
+};
+
+use crate::pbs_client;
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_TASKS)
+    .match_all("upid", &UPID_API_ROUTER);
+
+pub const UPID_API_ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(UPID_API_SUBDIRS))
+    .delete(&API_METHOD_STOP_TASK)
+    .subdirs(UPID_API_SUBDIRS);
+
+#[sortable]
+const UPID_API_SUBDIRS: SubdirMap = &sorted!([
+    ("log", &Router::new().get(&API_METHOD_READ_TASK_LOG)),
+    ("status", &Router::new().get(&API_METHOD_GET_TASK_STATUS)),
+]);
+
+#[api(
+    input: {
+        properties: {
+            remote: { schema: REMOTE_ID_SCHEMA },
+        },
+    },
+    access: {
+        // FIXME: fine-grained task filtering?
+        permission: &Permission::Privilege(&["resource", "{remote}"], PRIV_RESOURCE_AUDIT, false),
+    },
+    returns: { type: pve_api_types::TaskStatus },
+)]
+/// Get the list of tasks either for a specific node, or query all at once.
+async fn list_tasks(remote: String) -> Result<Vec<pbs_api_types::TaskListItem>, Error> {
+    let (remotes, _) = pdm_config::remotes::config()?;
+    let client = pbs_client::connect_to_remote(&remotes, &remote)?;
+
+    Ok(client.get_task_list(Default::default()).await?)
+}
+
+#[api(
+    input: {
+        properties: {
+            remote: { schema: REMOTE_ID_SCHEMA },
+            upid: { type: RemoteUpid },
+        },
+    },
+    access: {
+        // FIXME: fine-grained task filtering?
+        permission: &Permission::Privilege(&["resource", "{remote}"], PRIV_RESOURCE_MANAGE, false),
+    },
+)]
+/// Get the status of a task from a Proxmox VE instance.
+async fn stop_task(remote: String, upid: RemoteUpid) -> Result<(), Error> {
+    let (remotes, _) = pdm_config::remotes::config()?;
+
+    crate::api::verify_upid(&remote, RemoteType::Pbs, &upid)?;
+
+    let client = pbs_client::connect_to_remote(&remotes, upid.remote())?;
+
+    Ok(client.stop_task(upid.upid()).await?)
+}
+
+#[api(
+    input: {
+        properties: {
+            remote: { schema: REMOTE_ID_SCHEMA },
+            upid: { type: RemoteUpid },
+            wait: {
+                description: "wait for the task to finish before returning its result",
+                type: Boolean,
+                optional: true,
+                default: false,
+            },
+        },
+    },
+    access: {
+        // FIXME: fine-grained task filtering?
+        permission: &Permission::Privilege(&["resource", "{remote}"], PRIV_RESOURCE_AUDIT, false),
+    },
+    returns: { type: pve_api_types::TaskStatus },
+)]
+/// Get the status of a task from a Proxmox VE instance.
+pub async fn get_task_status(
+    remote: String,
+    upid: RemoteUpid,
+    wait: bool,
+) -> Result<pbs_client::TaskStatus, Error> {
+    let (remotes, _) = pdm_config::remotes::config()?;
+
+    crate::api::verify_upid(&remote, RemoteType::Pbs, &upid)?;
+
+    let client = pbs_client::connect_to_remote(&remotes, upid.remote())?;
+
+    loop {
+        let status = client.get_task_status(upid.upid()).await?;
+        if !wait || !status.is_running() {
+            break Ok(status);
+        }
+    }
+}
+
+// FIXME: make *actually* streaming with router support!
+#[api(
+    input: {
+        properties: {
+            remote: { schema: REMOTE_ID_SCHEMA },
+            upid: { type: RemoteUpid },
+            start: {
+                schema: TASKLOG_START_PARAM_SCHEMA,
+                optional: true,
+            },
+            limit: {
+                schema: TASKLOG_LIMIT_PARAM_SCHEMA,
+                optional: true,
+            },
+            download: {
+                schema: TASKLOG_DOWNLOAD_PARAM_SCHEMA,
+                optional: true,
+            }
+        },
+    },
+    access: {
+        // FIXME: fine-grained task filtering?
+        permission: &Permission::Privilege(&["resource", "{remote}"], PRIV_RESOURCE_AUDIT, false),
+    },
+    returns: {
+        type: Array,
+        items: {
+            type: pbs_client::TaskLogLine,
+        },
+        description: "Array of task log lines",
+    },
+)]
+/// Read a task log.
+async fn read_task_log(
+    remote: String,
+    upid: RemoteUpid,
+    download: Option<bool>,
+    start: Option<u64>,
+    limit: Option<u64>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<pbs_client::TaskLogLine>, Error> {
+    let (remotes, _) = pdm_config::remotes::config()?;
+
+    crate::api::verify_upid(&remote, RemoteType::Pbs, &upid)?;
+
+    let client = pbs_client::connect_to_remote(&remotes, &remote)?;
+
+    let response = client
+        .get_task_log(upid.upid(), download, limit, start)
+        .await?;
+
+    for (key, value) in response.attribs {
+        rpcenv[&key] = value;
+    }
+
+    Ok(response.data)
+}
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 5/7] api: pve tasks: use shared helpers for RemoteUpid handling

[Lukas Wagner]

The helpers defined in RemoteUpid, as well as the verify_upid helper
make the code much shorter and easy to read.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/src/api/pve/tasks.rs | 62 ++++++++++---------------------------
 1 file changed, 17 insertions(+), 45 deletions(-)

diff --git a/server/src/api/pve/tasks.rs b/server/src/api/pve/tasks.rs
index e66712ac..5edce149 100644
--- a/server/src/api/pve/tasks.rs
+++ b/server/src/api/pve/tasks.rs
@@ -1,17 +1,16 @@
 //! Access to PVE tasks.
 
-use anyhow::{bail, format_err, Error};
+use anyhow::Error;
 
 use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
 use proxmox_schema::api;
 use proxmox_sortable_macro::sortable;
 
-use pdm_api_types::remotes::REMOTE_ID_SCHEMA;
+use pdm_api_types::remotes::{RemoteType, REMOTE_ID_SCHEMA};
 use pdm_api_types::{
     RemoteUpid, NODE_SCHEMA, PRIV_RESOURCE_AUDIT, PRIV_RESOURCE_MANAGE,
     TASKLOG_DOWNLOAD_PARAM_SCHEMA, TASKLOG_LIMIT_PARAM_SCHEMA, TASKLOG_START_PARAM_SCHEMA,
 };
-use pve_api_types::PveUpid;
 
 use super::{connect, connect_to_remote, get_remote};
 
@@ -82,23 +81,14 @@ async fn list_tasks(
 async fn stop_task(remote: String, upid: RemoteUpid) -> Result<(), Error> {
     let (remotes, _) = pdm_config::remotes::config()?;
 
-    if upid.remote() != remote {
-        bail!(
-            "remote '{remote}' does not match remote in upid ('{}')",
-            upid.remote()
-        );
-    }
+    crate::api::verify_upid(&remote, RemoteType::Pve, &upid)?;
 
-    let pve = get_remote(&remotes, upid.remote())?;
+    let pve_upid = upid.pve_upid()?;
 
-    let pve_upid: PveUpid = upid
-        .upid()
-        .parse()
-        .map_err(|err| format_err!("invalid upid for PVE: {} - {err}", upid.upid()))?;
+    let remote = get_remote(&remotes, upid.remote())?;
+    let client = connect(remote)?;
 
-    let pve = connect(pve)?;
-
-    Ok(pve.stop_task(&pve_upid.node, upid.upid()).await?)
+    Ok(client.stop_task(&pve_upid.node, upid.upid()).await?)
 }
 
 #[api(
@@ -128,24 +118,15 @@ pub async fn get_task_status(
 ) -> Result<pve_api_types::TaskStatus, Error> {
     let (remotes, _) = pdm_config::remotes::config()?;
 
-    if upid.remote() != remote {
-        bail!(
-            "remote '{remote}' does not match remote in upid ('{}')",
-            upid.remote()
-        );
-    }
+    crate::api::verify_upid(&remote, RemoteType::Pve, &upid)?;
 
-    let pve = get_remote(&remotes, upid.remote())?;
+    let pve_upid = upid.pve_upid()?;
 
-    let pve_upid: PveUpid = upid
-        .upid()
-        .parse()
-        .map_err(|err| format_err!("invalid upid for PVE: {} - {err}", upid.upid()))?;
-
-    let pve = connect(pve)?;
+    let remote = get_remote(&remotes, upid.remote())?;
+    let client = connect(remote)?;
 
     loop {
-        let status = pve.get_task_status(&pve_upid.node, upid.upid()).await?;
+        let status = client.get_task_status(&pve_upid.node, upid.upid()).await?;
         if !wait || !status.is_running() {
             break Ok(status);
         }
@@ -189,23 +170,14 @@ async fn read_task_log(
 ) -> Result<Vec<pve_api_types::TaskLogLine>, Error> {
     let (remotes, _) = pdm_config::remotes::config()?;
 
-    if upid.remote() != remote {
-        bail!(
-            "remote '{remote}' does not match remote in upid ('{}')",
-            upid.remote()
-        );
-    }
+    crate::api::verify_upid(&remote, RemoteType::Pve, &upid)?;
 
-    let pve = get_remote(&remotes, upid.remote())?;
+    let pve_upid = upid.pve_upid()?;
 
-    let pve_upid: PveUpid = upid
-        .upid()
-        .parse()
-        .map_err(|err| format_err!("invalid upid for PVE: {} - {err}", upid.upid()))?;
+    let remote = get_remote(&remotes, upid.remote())?;
+    let client = connect(remote)?;
 
-    let pve = connect(pve)?;
-
-    let response = pve
+    let response = client
         .get_task_log(&pve_upid.node, upid.upid(), download, limit, start)
         .await?;
 
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 6/7] remote tasks: fetch/track PBS tasks

[Lukas Wagner]

Use the new API method bindings in pbs_client to fetch PBS task lists
as well as to track the status of a task started by PDM.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 .../tasks/remote_tasks.rs                     | 98 ++++++++++++++-----
 1 file changed, 72 insertions(+), 26 deletions(-)

diff --git a/server/src/bin/proxmox-datacenter-api/tasks/remote_tasks.rs b/server/src/bin/proxmox-datacenter-api/tasks/remote_tasks.rs
index e97f2a08..92f0f241 100644
--- a/server/src/bin/proxmox-datacenter-api/tasks/remote_tasks.rs
+++ b/server/src/bin/proxmox-datacenter-api/tasks/remote_tasks.rs
@@ -13,11 +13,11 @@ use pdm_api_types::{
     RemoteUpid,
 };
 use proxmox_section_config::typed::SectionConfigData;
-use pve_api_types::{ListTasks, ListTasksResponse, ListTasksSource};
 
 use server::{
-    api::pve,
+    connection,
     parallel_fetcher::{NodeResults, ParallelFetcher},
+    pbs_client,
     remote_tasks::{
         self,
         task_cache::{NodeFetchSuccessMap, State, TaskCache, TaskCacheItem},
@@ -260,32 +260,51 @@ async fn fetch_tasks_from_single_node(
     remote: Remote,
     node: String,
 ) -> Result<Vec<TaskCacheItem>, Error> {
+    let since = context
+        .cutoff_timestamp(&remote.id, &node)
+        .unwrap_or_else(|| {
+            proxmox_time::epoch_i64() - (KEEP_OLD_FILES as u64 * ROTATE_AFTER) as i64
+        });
+
     match remote.ty {
         RemoteType::Pve => {
-            let since = context
-                .cutoff_timestamp(&remote.id, &node)
-                .unwrap_or_else(|| {
-                    proxmox_time::epoch_i64() - (KEEP_OLD_FILES as u64 * ROTATE_AFTER) as i64
-                });
-
-            let params = ListTasks {
-                source: Some(ListTasksSource::Archive),
+            let params = pve_api_types::ListTasks {
+                source: Some(pve_api_types::ListTasksSource::Archive),
                 since: Some(since),
                 // If `limit` is not provided, we only receive 50 tasks
                 limit: Some(MAX_TASKS_TO_FETCH),
                 ..Default::default()
             };
 
-            let client = pve::connect(&remote)?;
+            let client = connection::make_pve_client(&remote)?;
 
             let task_list = client
                 .get_task_list(&node, params)
                 .await?
                 .into_iter()
-                .filter_map(|task| match map_pve_task(task, &remote.id) {
-                    Ok(task) => Some(task),
-                    Err(err) => {
-                        log::error!("could not map PVE task: {err:#}");
+                .map(|task| map_pve_task(task, remote.id.clone()))
+                .collect();
+
+            Ok(task_list)
+        }
+        RemoteType::Pbs => {
+            let params = pbs_client::ListTasks {
+                since: Some(since),
+                // If `limit` is not provided, we only receive 50 tasks
+                limit: Some(MAX_TASKS_TO_FETCH),
+            };
+
+            let client = connection::make_pbs_client(&remote)?;
+
+            let task_list = client
+                .get_task_list(params)
+                .await?
+                .into_iter()
+                .filter_map(|task| {
+                    if task.endtime.is_none() {
+                        // We only care about finished tasks.
+                        Some(map_pbs_task(task, remote.id.clone()))
+                    } else {
                         None
                     }
                 })
@@ -293,10 +312,6 @@ async fn fetch_tasks_from_single_node(
 
             Ok(task_list)
         }
-        RemoteType::Pbs => {
-            // TODO: Support PBS.
-            Ok(vec![])
-        }
     }
 }
 
@@ -426,22 +441,53 @@ async fn poll_single_tracked_task(remote: Remote, task: RemoteUpid) -> (RemoteUp
             (task, result)
         }
         RemoteType::Pbs => {
-            // TODO: Implement for PBS
-            (task, PollResult::RequestError)
+            let status = match server::api::pbs::tasks::get_task_status(
+                remote.id.clone(),
+                task.clone(),
+                false,
+            )
+            .await
+            {
+                Ok(status) => status,
+                Err(err) => {
+                    log::error!("could not get status from remote: {err:#}");
+                    return (task, PollResult::RequestError);
+                }
+            };
+
+            let result = if status.exitstatus.is_some() {
+                PollResult::Finished
+            } else {
+                PollResult::Running
+            };
+
+            (task, result)
         }
     }
 }
 
-/// Map a `ListTasksResponse` to `TaskCacheItem`
-fn map_pve_task(task: ListTasksResponse, remote: &str) -> Result<TaskCacheItem, Error> {
-    let remote_upid: RemoteUpid = (remote.to_string(), task.upid.to_string()).try_into()?;
+/// Map a `pve_api_types::ListTasksResponse` to `TaskCacheItem`
+fn map_pve_task(task: pve_api_types::ListTasksResponse, remote: String) -> TaskCacheItem {
+    let remote_upid = RemoteUpid::new(remote, RemoteType::Pve, task.upid);
 
-    Ok(TaskCacheItem {
+    TaskCacheItem {
         upid: remote_upid,
         starttime: task.starttime,
         endtime: task.endtime,
         status: task.status,
-    })
+    }
+}
+
+/// Map a `pbs_api_types::TaskListItem` to `TaskCacheItem`
+fn map_pbs_task(task: pbs_api_types::TaskListItem, remote: String) -> TaskCacheItem {
+    let remote_upid = RemoteUpid::new(remote, RemoteType::Pbs, task.upid);
+
+    TaskCacheItem {
+        upid: remote_upid,
+        starttime: task.starttime,
+        endtime: task.endtime,
+        status: task.status,
+    }
 }
 
 /// Update task cache with results from tracked task polling & regular task fetching.
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] [PATCH datacenter-manager 7/7] remote updates: re-enable PBS update fetching

[Lukas Wagner]

The task subsystem does not handle PBS tasks correctly, so we can start
tasks and track them until they are completed.

Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
---
 server/src/remote_updates.rs | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/server/src/remote_updates.rs b/server/src/remote_updates.rs
index cd9b1aec..b1db3b81 100644
--- a/server/src/remote_updates.rs
+++ b/server/src/remote_updates.rs
@@ -1,7 +1,7 @@
 use std::fs::File;
 use std::io::ErrorKind;
 
-use anyhow::{bail, Error};
+use anyhow::Error;
 use serde::{Deserialize, Serialize};
 
 use proxmox_apt_api_types::APTUpdateInfo;
@@ -66,18 +66,15 @@ pub async fn update_apt_database(remote: &Remote, node: &str) -> Result<RemoteUp
             crate::api::pve::new_remote_upid(remote.id.clone(), upid).await
         }
         RemoteType::Pbs => {
-            // let client = connection::make_pbs_client(remote)?;
-            //
-            // let params = crate::pbs_client::AptUpdateParams {
-            //     notify: Some(false),
-            //     quiet: Some(false),
-            // };
-            // let upid = client.update_apt_database(params).await?;
-            //
-            // crate::api::pbs::new_remote_upid(remote.id.clone(), upid).await
-            // TODO: task infrastructure for PBS not finished yet, uncomment once
-            // this is done.
-            bail!("PBS is not supported yet");
+            let client = connection::make_pbs_client(remote)?;
+
+            let params = crate::pbs_client::AptUpdateParams {
+                notify: Some(false),
+                quiet: Some(false),
+            };
+            let upid = client.update_apt_database(params).await?;
+
+            crate::api::pbs::new_remote_upid(remote.id.clone(), upid).await
         }
     }
 }
-- 
2.47.3



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

Re: [pdm-devel] [PATCH datacenter-manager 2/7] pbs-client: add bindings for task list, task status, task log

[Thomas Lamprecht]

Am 12.11.25 um 10:42 schrieb Lukas Wagner:

> diff --git a/server/src/pbs_client.rs b/server/src/pbs_client.rs
> index 49087360..4285f402 100644
> --- a/server/src/pbs_client.rs
> +++ b/server/src/pbs_client.rs

> @@ -125,6 +125,75 @@ pub struct AptUpdateParams {
>      pub quiet: Option<bool>,
>  }
>  
> +// TODO: This is incomplete, it only contains the parameters needed for remote task fetching.
> +// Ideally, the task list API in PBS would use a parameter struct defined in pbs-api-types, which
> +// is then also used here.

As we have tasks in all projects it certainly can be something very common,
and some thought that crossed my mind more than once was splitting out upid
from the proxmox-schema crate, while I can relate to why it got placed there
initially, it just feels like the wrong place. In a dedicated proxmox-task
or proxmox-upid or the like crate these types here might also fit well in.

> +#[derive(Debug, Default, Deserialize, Serialize)]
> +pub struct ListTasks {
> +    /// Only list this number of tasks.
> +    #[serde(default, skip_serializing_if = "Option::is_none")]
> +    pub limit: Option<u64>,
> +
> +    /// Only list tasks since this UNIX epoch.
> +    #[serde(default, skip_serializing_if = "Option::is_none")]
> +    pub since: Option<i64>,
> +}
> +
> +// TODO: The task-status APIs in PBS as well as PDM don't have a
> +// proper type defined anywhere. This should be moved to a shared crate
> +// and then the API handlers adapted.
> +#[derive(Debug, Deserialize, Serialize)]
> +pub struct TaskStatus {
> +    pub exitstatus: Option<String>,
> +
> +    pub id: Option<String>,
> +
> +    pub node: String,
> +
> +    pub pid: i64,
> +
> +    pub pstart: i64,
> +
> +    pub starttime: i64,
> +
> +    pub status: IsRunning,
> +
> +    #[serde(rename = "type")]
> +    pub ty: String,
> +
> +    pub upid: String,
> +
> +    pub user: String,
> +}
> +
> +#[derive(Debug, Deserialize, Serialize, PartialEq)]
> +#[serde(rename_all = "kebab-case")]
> +pub enum IsRunning {
> +    Running,
> +    Stopped,
> +}
> +
> +impl TaskStatus {
> +    /// Checks if the task is currently running.
> +    pub fn is_running(&self) -> bool {
> +        self.status == IsRunning::Running
> +    }
> +}
> +



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

Re: [pdm-devel] [PATCH datacenter-manager 7/7] remote updates: re-enable PBS update fetching

[Thomas Lamprecht]

Am 12.11.25 um 10:42 schrieb Lukas Wagner:
> The task subsystem does not handle PBS tasks correctly, so we can start

s/not/now/ I'd figure?

> tasks and track them until they are completed.



_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel

[pdm-devel] applied-series: [PATCH datacenter-manager 0/7] PBS remotes: task API and task cache support

[Thomas Lamprecht]

On Wed, 12 Nov 2025 10:41:56 +0100, Lukas Wagner wrote:
> NOTE: This one is based of my series which changes the RemoteUpid to include
> the remote type [0]. If that RFC is rejected, I will rebase onto master (and
> try to solve the problem that the RFC is trying to fixin some other way). If
> the RFC is accepted, I will probably merge both series into one for easier
> handling.
> 
> This series adds the necessary bindings to the PBS client for basic task
> retrieval as required by the task cache and the remote task overview in the UI.
> 
> [...]

Applied, thanks!

[1/7] pdm-api-types: remote upid: add helpers for getting native UPID type
      commit: 34d20db572541c1d82107ed2b761be328c92b45b
[2/7] pbs-client: add bindings for task list, task status, task log
      commit: e40f8b80f0166f5dcf660a80d69f9f8e9aa932f9
[3/7] pdm-api-types: api: factor out schema definitions for task log params
      commit: 93620611f33ef2ddcf89a9a42bccfd478b754272
[4/7] api: pbs tasks: add PBS task API
      commit: 131787fe678a29329894e515c91cbe257a7228a1
[5/7] api: pve tasks: use shared helpers for RemoteUpid handling
      commit: 1584001ce3581011c84b7c6c29aa8bb39a06330a
[6/7] remote tasks: fetch/track PBS tasks
      commit: f3e6f48dfc9c8ba9fd45d7c6b0ac0bf26a41b95c
[7/7] remote updates: re-enable PBS update fetching
      commit: 5e75686cbe2b2faaaf60450347dfa42e74051960


_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel