From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id BEC8E1FF165 for ; Thu, 14 Aug 2025 15:30:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 34580133CB; Thu, 14 Aug 2025 15:32:24 +0200 (CEST) From: Lukas Wagner To: pdm-devel@lists.proxmox.com Date: Thu, 14 Aug 2025 15:31:25 +0200 Message-ID: <20250814133142.386650-7-l.wagner@proxmox.com> X-Mailer: git-send-email 2.47.2 In-Reply-To: <20250814133142.386650-1-l.wagner@proxmox.com> References: <20250814133142.386650-1-l.wagner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1755178276694 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.022 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pdm-devel] [PATCH proxmox-datacenter-manager v5 06/23] metric collection: collect overdue metrics on startup/timer change X-BeenThere: pdm-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Datacenter Manager development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pdm-devel-bounces@lists.proxmox.com Sender: "pdm-devel" Due to the fact that the timer fires at aligned points in time and might now fire right away after being set up, it could happen that we get gaps in the data if we change the timer interval or at daemon startup. To mitigate this, on daemon startup and also if the collection interval changes, we - check if the time until the next scheduled regular collection plus the time *since* the last successful collection exceeds the collection interval - if yes, we collect immediately - if no, we do nothing and let the remote be collected at the next timer tick Signed-off-by: Lukas Wagner Reviewed-by: Maximiliano Sandoval --- Notes: Changes since v1: - Document return values of `setup_timer` .../src/metric_collection/collection_task.rs | 57 +++++++++++++++++-- 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/server/src/metric_collection/collection_task.rs b/server/src/metric_collection/collection_task.rs index 27e097f7..1821f138 100644 --- a/server/src/metric_collection/collection_task.rs +++ b/server/src/metric_collection/collection_task.rs @@ -1,4 +1,7 @@ -use std::{sync::Arc, time::Duration}; +use std::{ + sync::Arc, + time::{Duration, Instant}, +}; use anyhow::Error; use tokio::{ @@ -68,12 +71,17 @@ impl MetricCollectionTask { /// This function never returns. #[tracing::instrument(skip_all, name = "metric_collection_task")] pub(super) async fn run(&mut self) { - let mut timer = Self::setup_timer(DEFAULT_COLLECTION_INTERVAL); + let (mut timer, first_tick) = Self::setup_timer(DEFAULT_COLLECTION_INTERVAL); log::debug!( "metric collection starting up. Collection interval set to {} seconds.", DEFAULT_COLLECTION_INTERVAL, ); + // Check and fetch any remote which would be overdue by the time the + // timer first fires. + if let Some(remote_config) = Self::load_remote_config() { + self.fetch_overdue(&remote_config, first_tick).await; + } loop { tokio::select! { @@ -116,12 +124,16 @@ impl MetricCollectionTask { /// Set up a [`tokio::time::Interval`] instance with the provided interval. /// The timer will be aligned, e.g. an interval of `60` will let the timer /// fire at minute boundaries. - fn setup_timer(interval: u64) -> Interval { + /// + /// The return values are a tuple of the [`tokio::time::Interval`] timer instance + /// and the [`std::time::Instant`] at which the timer first fires. + fn setup_timer(interval: u64) -> (Interval, Instant) { + log::debug!("setting metric collection interval timer to {interval} seconds.",); let mut timer = tokio::time::interval(Duration::from_secs(interval)); - let first_run = task_utils::next_aligned_instant(interval).into(); - timer.reset_at(first_run); + let first_run = task_utils::next_aligned_instant(interval); + timer.reset_at(first_run.into()); - timer + (timer, first_run) } /// Convenience helper to load `remote.cfg`, logging the error @@ -197,6 +209,39 @@ impl MetricCollectionTask { } } + /// Fetch metric data from remotes which are overdue for collection. + /// + /// Use this on startup of the metric collection loop as well as + /// when the collection interval changes. + async fn fetch_overdue( + &mut self, + remote_config: &SectionConfigData, + next_run: Instant, + ) { + let left_until_scheduled = next_run - Instant::now(); + let now = proxmox_time::epoch_i64(); + + let mut overdue = Vec::new(); + + for (remote, _) in remote_config.iter() { + let last_collection = self + .state + .get_status(remote) + .and_then(|s| s.last_collection) + .unwrap_or(0); + + let diff = now - last_collection; + + if diff + left_until_scheduled.as_secs() as i64 > DEFAULT_COLLECTION_INTERVAL as i64 { + log::debug!( + "starting metric collection for remote '{remote}' - triggered because collection is overdue" + ); + overdue.push(remote.into()); + } + } + self.fetch_remotes(remote_config, &overdue).await; + } + /// Fetch a single remote. #[tracing::instrument(skip_all, fields(remote = remote.id), name = "metric_collection_task")] async fn fetch_single_remote( -- 2.47.2 _______________________________________________ pdm-devel mailing list pdm-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel