From: Lukas Wagner <l.wagner@proxmox.com>
To: pdm-devel@lists.proxmox.com
Subject: [pdm-devel] [PATCH proxmox-datacenter-manager v3 15/26] metric collection: save time needed for collection run to RRD
Date: Wed, 16 Apr 2025 14:56:31 +0200 [thread overview]
Message-ID: <20250416125642.291552-16-l.wagner@proxmox.com> (raw)
In-Reply-To: <20250416125642.291552-1-l.wagner@proxmox.com>
For large setups, it might be useful to know how much time was needed to
collect metrics for *all* remotes together, e.g. for making sure that
the collection interval is not exceeded.
Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
Reviewed-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
---
.../src/metric_collection/collection_task.rs | 14 +++++
server/src/metric_collection/rrd_task.rs | 53 ++++++++++++++-----
2 files changed, 55 insertions(+), 12 deletions(-)
diff --git a/server/src/metric_collection/collection_task.rs b/server/src/metric_collection/collection_task.rs
index a6c16608..978e0da5 100644
--- a/server/src/metric_collection/collection_task.rs
+++ b/server/src/metric_collection/collection_task.rs
@@ -21,6 +21,7 @@ use pdm_api_types::{
};
use pdm_config::metric_collection::COLLECTION_SETTINGS_TYPE;
+use crate::metric_collection::rrd_task::CollectionStats;
use crate::{connection, task_utils};
use super::{
@@ -93,8 +94,21 @@ impl MetricCollectionTask {
log::debug!("starting metric collection from all remotes - triggered by timer");
if let Some(remotes) = Self::load_remote_config() {
+ let now = Instant::now();
let to_fetch = remotes.order.as_slice();
self.fetch_remotes(&remotes, to_fetch).await;
+ let elapsed = now.elapsed();
+
+ if let Err(err) = self.metric_data_tx.send(
+ RrdStoreRequest::CollectionStats {
+ timestamp: proxmox_time::epoch_i64(),
+ stats: CollectionStats {
+ // TODO: use as_millis_f64 once stabilized
+ total_time: elapsed.as_secs_f64() * 1000.
+ }
+ }).await {
+ log::error!("could not send collection stats to rrd task: {err}");
+ }
}
}
diff --git a/server/src/metric_collection/rrd_task.rs b/server/src/metric_collection/rrd_task.rs
index a8e48e89..7d0b95b2 100644
--- a/server/src/metric_collection/rrd_task.rs
+++ b/server/src/metric_collection/rrd_task.rs
@@ -38,6 +38,13 @@ pub(super) enum RrdStoreRequest {
/// Timestamp at which the request was done (UNIX epoch).
request_at: i64,
},
+ /// Store collection stats.
+ CollectionStats {
+ /// Timestamp at which the collection took place (UNIX epoch).
+ timestamp: i64,
+ /// Statistics.
+ stats: CollectionStats,
+ },
}
/// Result for a [`RrdStoreRequest`].
@@ -46,6 +53,12 @@ pub(super) struct RrdStoreResult {
pub(super) most_recent_timestamp: i64,
}
+/// Statistics for a (full) metric collection run.
+pub(super) struct CollectionStats {
+ /// Total time in ms.
+ pub(super) total_time: f64,
+}
+
/// Task which stores received metrics in the RRD. Metric data is fed into
/// this task via a MPSC channel.
pub(super) async fn store_in_rrd_task(
@@ -57,7 +70,8 @@ pub(super) async fn store_in_rrd_task(
// Involves some blocking file IO
let res = tokio::task::spawn_blocking(move || {
let mut most_recent_timestamp = 0;
- let channel = match msg {
+
+ match msg {
RrdStoreRequest::Pve {
remote,
metrics,
@@ -71,7 +85,13 @@ pub(super) async fn store_in_rrd_task(
}
store_response_time(&cache_clone, &remote, response_time, request_at);
- channel
+ let result = RrdStoreResult {
+ most_recent_timestamp,
+ };
+
+ if channel.send(result).is_err() {
+ log::error!("could not send RrdStoreStoreResult to metric collection task");
+ };
}
RrdStoreRequest::Pbs {
remote,
@@ -86,17 +106,17 @@ pub(super) async fn store_in_rrd_task(
}
store_response_time(&cache_clone, &remote, response_time, request_at);
- channel
- }
- };
+ let result = RrdStoreResult {
+ most_recent_timestamp,
+ };
- if channel
- .send(RrdStoreResult {
- most_recent_timestamp,
- })
- .is_err()
- {
- log::error!("could not send RrdStoreStoreResult to metric collection task");
+ if channel.send(result).is_err() {
+ log::error!("could not send RrdStoreStoreResult to metric collection task");
+ };
+ }
+ RrdStoreRequest::CollectionStats { timestamp, stats } => {
+ store_stats(&cache_clone, &stats, timestamp)
+ }
};
})
.await;
@@ -157,6 +177,15 @@ fn store_response_time(cache: &RrdCache, remote_name: &str, response_time: f64,
cache.update_value(&name, response_time, timestamp, DataSourceType::Gauge);
}
+fn store_stats(cache: &RrdCache, stats: &CollectionStats, timestamp: i64) {
+ cache.update_value(
+ "local/metric-collection/total-time",
+ stats.total_time,
+ timestamp,
+ DataSourceType::Gauge,
+ );
+}
+
#[cfg(test)]
mod tests {
use proxmox_rrd_api_types::{RrdMode, RrdTimeframe};
--
2.39.5
_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel
next prev parent reply other threads:[~2025-04-16 12:56 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-04-16 12:56 [pdm-devel] [PATCH proxmox-datacenter-manager v3 00/26] metric collection improvements (concurrency, config, API, CLI) Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 01/26] pdm-api-types: add CollectionSettings type Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 02/26] pdm-config: add functions for reading/writing metric collection settings Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 03/26] metric collection: split top_entities split into separate module Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 04/26] metric collection: save metric data to RRD in separate task Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 05/26] metric collection: rework metric poll task Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 06/26] metric collection: persist state after metric collection Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 07/26] metric collection: skip if last_collection < MIN_COLLECTION_INTERVAL Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 08/26] metric collection: collect overdue metrics on startup/timer change Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 09/26] metric collection: add tests for the fetch_remotes function Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 10/26] metric collection: add test for fetch_overdue Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 11/26] metric collection: pass rrd cache instance as function parameter Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 12/26] metric collection: add test for rrd task Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 13/26] metric collection: wrap rrd_cache::Cache in a struct Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 14/26] metric collection: record remote response time in metric database Lukas Wagner
2025-04-16 12:56 ` Lukas Wagner [this message]
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 16/26] metric collection: periodically clean removed remotes from statefile Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 17/26] api: add endpoint for updating metric collection settings Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 18/26] api: add endpoint to trigger metric collection Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 19/26] api: remotes: trigger immediate metric collection for newly added nodes Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 20/26] api: add api for querying metric collection RRD data Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 21/26] api: metric-collection: add status endpoint Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 22/26] pdm-client: add metric collection API methods Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 23/26] cli: add commands for metric-collection settings, trigger, status Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 24/26] metric collection: factor out handle_tick and handle_control_message fns Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 25/26] metric collection: skip missed timer ticks Lukas Wagner
2025-04-16 12:56 ` [pdm-devel] [PATCH proxmox-datacenter-manager v3 26/26] metric collection: use JoinSet instead of joining from handles in a Vec Lukas Wagner
2025-05-12 13:38 ` [pdm-devel] superseded: [PATCH proxmox-datacenter-manager v3 00/26] metric collection improvements (concurrency, config, API, CLI) Lukas Wagner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250416125642.291552-16-l.wagner@proxmox.com \
--to=l.wagner@proxmox.com \
--cc=pdm-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal