[pdm-devel] [PATCH proxmox-datacenter-manager 01/13] server: add locked sdn client and helper methods

[Stefan Hanreich <s.hanreich@proxmox.com>]

Add a new client that represents a remote with a locked SDN
configuration. It works by creating a new PveClient and then locking
the SDN configuration via the client. It ensures that, while the lock
is held, all methods are called with the proper lock secret.

There are also helpers included that make writing code that tries to
connect to multiple remotes simultaneously easier. This will be
particularly useful for the API methods that are manipulating the SDN
configuration across multiple remotes.

For more information on how they work, please consult the
documentation of the struct, its methods and the helper methods.

Signed-off-by: Stefan Hanreich <s.hanreich@proxmox.com>
---
 server/src/lib.rs        |   1 +
 server/src/sdn_client.rs | 234 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 235 insertions(+)
 create mode 100644 server/src/sdn_client.rs

diff --git a/server/src/lib.rs b/server/src/lib.rs
index 12dc912..45eee84 100644
--- a/server/src/lib.rs
+++ b/server/src/lib.rs
@@ -12,6 +12,7 @@ pub mod task_utils;
 
 pub mod connection;
 pub mod pbs_client;
+pub mod sdn_client;
 
 #[cfg(remote_config = "faked")]
 pub mod test_support;
diff --git a/server/src/sdn_client.rs b/server/src/sdn_client.rs
new file mode 100644
index 0000000..fd77305
--- /dev/null
+++ b/server/src/sdn_client.rs
@@ -0,0 +1,234 @@
+use std::{collections::HashMap, time::Duration};
+
+use anyhow::{self, bail};
+
+use futures::{stream::FuturesUnordered, StreamExt, TryFutureExt};
+use pdm_api_types::{remotes::Remote, RemoteUpid};
+use pve_api_types::{
+    client::PveClient, CreateSdnLock, CreateVnet, CreateZone, PveUpid, ReleaseSdnLock, ReloadSdn,
+};
+
+use crate::api::pve::{connect, get_remote};
+
+/// Wrapper for [`PveClient`] for representing a locked SDN configuration.
+///
+/// It stores the client that has been locked, as well as the lock_secret that is required for
+/// making changes to the SDN configuration. It provides methods that proxy the respective SDN
+/// endpoints, where it adds the lock_secret when making the proxied calls.
+pub struct LockedSdnClient {
+    secret: String,
+    client: Box<dyn PveClient + Send + Sync>,
+}
+
+impl LockedSdnClient {
+    /// Consumes a [`PveClient`] and locks the remote instance. On success, returns a new
+    /// [`LockedSdnClient`] where the remotes' SDN configuration has been locked.
+    ///
+    /// # Errors
+    ///
+    /// This function will return an error if locking the remote fails.
+    pub async fn new(
+        remote: &Remote,
+        allow_pending: impl Into<Option<bool>>,
+    ) -> Result<Self, anyhow::Error> {
+        let client = connect(remote)?;
+
+        let params = CreateSdnLock {
+            allow_pending: allow_pending.into(),
+        };
+
+        let secret = client.acquire_sdn_lock(params).await?;
+
+        Ok(Self { secret, client })
+    }
+
+    /// proxies [`PveClient::create_vnet`] and adds lock_secret to the passed parameters before
+    /// making the call.
+    pub async fn create_vnet(&self, mut params: CreateVnet) -> Result<(), proxmox_client::Error> {
+        params.lock_secret = Some(self.secret.clone());
+
+        self.client.create_vnet(params).await
+    }
+
+    /// proxies [`PveClient::create_zone`] and adds lock_secret to the passed parameters before
+    /// making the call.
+    pub async fn create_zone(&self, mut params: CreateZone) -> Result<(), proxmox_client::Error> {
+        params.lock_secret = Some(self.secret.clone());
+
+        self.client.create_zone(params).await
+    }
+
+    /// applies the changes made while the client was locked and returns the original [`PveClient`] if the
+    /// changes have been applied successfully.
+    pub async fn apply_and_release(
+        self,
+    ) -> Result<(PveUpid, Box<dyn PveClient + Send + Sync>), proxmox_client::Error> {
+        let params = ReloadSdn {
+            lock_secret: Some(self.secret.clone()),
+            release_lock: Some(true),
+        };
+
+        self.client
+            .sdn_apply(params)
+            .await
+            .map(|upid| (upid, self.client))
+    }
+
+    /// releases the lock on the [`PveClient`] without applying pending changes.
+    pub async fn release(
+        self,
+        force: impl Into<Option<bool>>,
+    ) -> Result<Box<dyn PveClient + Send + Sync>, proxmox_client::Error> {
+        let params = ReleaseSdnLock {
+            force: force.into(),
+            lock_secret: Some(self.secret),
+        };
+
+        self.client.release_sdn_lock(params).await?;
+        Ok(self.client)
+    }
+}
+
+/// Releases all clients found in the [`clients`] parameter.
+///
+/// Any errors occuring during this process will get loggged, but the function will still try to
+/// release all other clients before returning.
+async fn release_clients(clients: HashMap<String, LockedSdnClient>) {
+    for (remote, client) in clients {
+        proxmox_log::info!("releasing lock for remote {remote}");
+
+        if let Err(error) = client.release(false).await {
+            proxmox_log::error!("could not release lock for remote {remote}: {error:#}",)
+        }
+    }
+}
+
+/// A convenience function for creating locked clients for multiple remotes.
+///
+/// # Errors
+///
+/// This function will return an error if:
+/// * the remote configuration cannot be read
+/// * any of the supplied remotes is not contained in the configuration
+/// * any of the supplied remotes cannot be successfully locked
+///
+/// In any of those cases all remotes that have already been locked will get unlocked before the
+/// error gets returned.
+pub(crate) async fn create_locked_clients(
+    remotes: impl Iterator<Item = String>,
+) -> Result<HashMap<String, LockedSdnClient>, anyhow::Error> {
+    let (remote_config, _) = pdm_config::remotes::config()?;
+    let mut locked_clients = HashMap::new();
+
+    for remote in remotes {
+        proxmox_log::info!("obtaining lock for remote {remote}");
+
+        let Ok(remote_config) = get_remote(&remote_config, &remote) else {
+            release_clients(locked_clients).await;
+            bail!("remote {remote} does not exist in configuration");
+        };
+
+        let Ok(client) = LockedSdnClient::new(remote_config, false).await else {
+            release_clients(locked_clients).await;
+            bail!("could not lock sdn configuration for remote {remote}",);
+        };
+
+        locked_clients.insert(remote, client);
+    }
+
+    Ok(locked_clients)
+}
+
+// pve-http-server TCP connection timeout is 5 seconds, use a lower amount with some margin for
+// latency in order to avoid re-opening TCP connections for every polling request.
+const POLLING_INTERVAL: Duration = Duration::from_secs(3);
+
+/// Convenience function for polling a running task on a PVE remote.
+///
+/// It polls a given task on a given node, waiting for the task to finish successfully.
+///
+/// # Errors
+///
+/// This function will return an error if:
+/// * There was a problem querying the task status (this does not necessarily mean the task failed).
+/// * The task finished unsuccessfully.
+async fn poll_task(
+    node: String,
+    upid: RemoteUpid,
+    client: Box<dyn PveClient + Send + Sync>,
+) -> Result<(RemoteUpid, Box<dyn PveClient + Send + Sync>), anyhow::Error> {
+    loop {
+        tokio::time::sleep(POLLING_INTERVAL).await;
+
+        let status = client.get_task_status(&node, &upid.upid).await?;
+
+        if !status.is_running() {
+            if status.finished_successfully() == Some(true) {
+                return Ok((upid, client));
+            } else {
+                bail!(
+                    "task did not finish successfully on remote {}",
+                    upid.remote()
+                );
+            }
+        }
+    }
+}
+
+/// Applies the SDN configuration for multiple locked clients.
+///
+/// This function tries to apply the SDN configuration for all supplied locked clients. It logs
+/// success and error messages via proxmox_log.
+///
+/// # Errors
+/// This function returns an error if applying the configuration on one of the remotes failed. It
+/// will always wait for all futures to finish and only return an error afterwards.
+pub(crate) async fn apply_sdn_configuration(
+    locked_clients: HashMap<String, LockedSdnClient>,
+) -> Result<(), anyhow::Error> {
+    let mut futures = FuturesUnordered::new();
+
+    for (id, client) in locked_clients.into_iter() {
+        proxmox_log::info!("applying sdn config on remote {id}");
+
+        let remote_id = id.clone();
+
+        let future = client
+            .apply_and_release()
+            .map_err(anyhow::Error::msg)
+            .and_then(move |(upid, client)| {
+                proxmox_log::info!("reloading SDN configuration on remote {}", remote_id);
+
+                let remote_upid =
+                    RemoteUpid::try_from((remote_id, upid.to_string())).expect("valid upid");
+
+                poll_task(upid.node.clone(), remote_upid, client)
+            });
+
+        futures.push(future);
+    }
+
+    proxmox_log::info!("Waiting for reload tasks to finish on all remotes, this can take awhile");
+
+    let mut errors = false;
+    while let Some(result) = futures.next().await {
+        match result {
+            Ok((upid, _)) => {
+                proxmox_log::info!(
+                    "successfully applied configuration on remote {}",
+                    upid.remote()
+                );
+            }
+            Err(error) => {
+                proxmox_log::error!("{error:#}",);
+                errors = true;
+            }
+        }
+    }
+
+    if errors {
+        bail!("failed to apply configuration on at least one remote");
+    }
+
+    Ok(())
+}
-- 
2.39.5


_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel