public inbox for pdm-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Wolfgang Bumiller <w.bumiller@proxmox.com>
To: Lukas Wagner <l.wagner@proxmox.com>
Cc: pdm-devel@lists.proxmox.com
Subject: Re: [pdm-devel] [PATCH proxmox-datacenter-manager v2 03/28] pdm-api-types: add CollectionSettings type
Date: Tue, 18 Feb 2025 16:26:09 +0100	[thread overview]
Message-ID: <aguxeubkyqnqrfsrmmo4suxmap34ep34bhs5l3rhhqrjr33hlo@vw7u7jhojkka> (raw)
In-Reply-To: <20250214130653.283012-4-l.wagner@proxmox.com>

On Fri, Feb 14, 2025 at 02:06:28PM +0100, Lukas Wagner wrote:
> This commit adds the CollectionSettings type which holds settings for
> the metric collection system. Included are collection interval, max
> concurrency and upper/lower bounds for the metric collection loop.
> 
> Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
> ---
>  lib/pdm-api-types/src/lib.rs               |   3 +
>  lib/pdm-api-types/src/metric_collection.rs | 188 +++++++++++++++++++++
>  2 files changed, 191 insertions(+)
>  create mode 100644 lib/pdm-api-types/src/metric_collection.rs
> 
> diff --git a/lib/pdm-api-types/src/lib.rs b/lib/pdm-api-types/src/lib.rs
> index 38449071..6115e41c 100644
> --- a/lib/pdm-api-types/src/lib.rs
> +++ b/lib/pdm-api-types/src/lib.rs
> @@ -19,6 +19,9 @@ pub use acl::*;
>  mod node_config;
>  pub use node_config::*;
>  
> +mod metric_collection;
> +pub use metric_collection::*;
> +
>  mod proxy;
>  pub use proxy::HTTP_PROXY_SCHEMA;
>  
> diff --git a/lib/pdm-api-types/src/metric_collection.rs b/lib/pdm-api-types/src/metric_collection.rs
> new file mode 100644
> index 00000000..92487d6c
> --- /dev/null
> +++ b/lib/pdm-api-types/src/metric_collection.rs
> @@ -0,0 +1,188 @@
> +//! API types for metric collection settings.
> +
> +use serde::{Deserialize, Serialize};
> +
> +use proxmox_schema::{api, Updater};
> +
> +/// Default metric collection interval.
> +pub const DEFAULT_COLLECTION_INTERVAL: u64 = 600;
> +
> +/// Minimum metric collection interval.
> +pub const MIN_COLLECTION_INTERVAL: u64 = 10;
> +
> +/// Maximum metric collection interval.
> +/// PVE and PBS keep 30 minutes of metric history,
> +/// maximum is set to 25 minutes to leave some headroom.
> +pub const MAX_COLLECTION_INTERVAL: u64 = 1500;
> +
> +/// Default number of concurrent connections.
> +pub const DEFAULT_CONCURRENCY: usize = 10;
> +/// Maximum number of concurrent connections.
> +pub const MAX_CONCURRENCY: u64 = 50;
> +/// Minimum number of concurrent connections (no concurrency).
> +pub const MIN_CONCURRENCY: u64 = 1;
> +
> +/// Default upper bound for the random delay for the
> +/// main metric collection loop.
> +pub const DEFAULT_MAX_OFFSET: u64 = 10;
> +/// Default lower bound for the random delay for the
> +/// main metric collection loop.
> +pub const DEFAULT_MIN_OFFSET: u64 = 0;
> +
> +/// Highest configurable upper bound for the random interval offset for the main loop.
> +pub const MAX_INTERVAL_OFFSET: u64 = 300;
> +/// Lowest configureable lower bound for the random interval offset for the main loop.
> +pub const MIN_INTERVAL_OFFSET: u64 = 0;
> +
> +/// Default upper bound for the random individual connection delay.
> +pub const DEFAULT_MAX_CONNECTION_DELAY: u64 = 100;
> +/// Default lower bound for the random individual connection delay.
> +pub const DEFAULT_MIN_CONNECTION_DELAY: u64 = 0;
> +
> +/// Highest configurable upper bound for the random individual connection delay.
> +pub const MAX_CONNECTION_DELAY: u64 = 1000;
> +/// Lowest configurable lower bound for the random individual connection delay.
> +pub const MIN_CONNECTION_DELAY: u64 = 0;
> +
> +#[api(
> +    properties: {
> +        "collection-interval" : {
> +            optional: true,
> +            default: DEFAULT_COLLECTION_INTERVAL as isize,
> +            minimum: MIN_COLLECTION_INTERVAL as isize,
> +            maximum: MAX_COLLECTION_INTERVAL as isize,

I thought about this the first time around but then forgot again:

Given that it is possible we might change these types (see Stefan's
patch for proxmox-schema) and because it is much more convenient
anyway you could use `as _` for these casts.

While I'm usually not a fan of `_` casts - I think this might be okay
for schema declarations.

(No need for a v3 if there's nothing else...)

> +        },
> +        "max-concurrent-connections" : {
> +            optional: true,
> +            default: DEFAULT_CONCURRENCY as isize,
> +            minimum: MIN_CONCURRENCY as isize,
> +            maximum: MAX_CONCURRENCY as isize,
> +        },
> +        "max-interval-offset" : {
> +            optional: true,
> +            default: DEFAULT_MAX_OFFSET as isize,
> +            minimum: MIN_INTERVAL_OFFSET as isize,
> +            maximum: MAX_INTERVAL_OFFSET as isize,
> +        },
> +        "min-interval-offset" : {
> +            optional: true,
> +            default: DEFAULT_MIN_OFFSET as isize,
> +            minimum: MIN_INTERVAL_OFFSET as isize,
> +            maximum: MAX_INTERVAL_OFFSET as isize,
> +        },
> +        "max-connection-delay" : {
> +            optional: true,
> +            default: DEFAULT_MAX_CONNECTION_DELAY as isize,
> +            minimum: MIN_CONNECTION_DELAY as isize,
> +            maximum: MAX_CONNECTION_DELAY as isize,
> +        },
> +        "min-connection-delay" : {
> +            optional: true,
> +            default: DEFAULT_MIN_CONNECTION_DELAY as isize,
> +            minimum: MIN_CONNECTION_DELAY as isize,
> +            maximum: MAX_CONNECTION_DELAY as isize,
> +        },
> +    },
> +)]
> +#[derive(Clone, Default, Deserialize, Serialize, Updater)]
> +#[serde(rename_all = "kebab-case")]
> +/// Settings for the metric collection system.
> +pub struct CollectionSettings {
> +    /// Collection settings ID
> +    #[updater(skip)]
> +    pub id: String,
> +
> +    /// Interval in seconds at which to collect metrics.
> +    /// The point in time at which metrics are collected
> +    /// are aligned based on the collection interval. For instance,
> +    /// a collection interval of 300 (5 minutes) would schedule metric collection
> +    /// at 11:00:00, 11:05:00 (without accounting for the random offset).
> +    ///
> +    /// To avoid load spikes, metric collection runs are offeset by
> +    /// a random number of seconds between min_interval_offset and max_interval_offset.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub collection_interval: Option<u64>,
> +
> +    /// Maximum number of concurrent connections while collecting metrics.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub max_concurrent_connections: Option<usize>,
> +
> +    /// Maximum offset in seconds for the metric collection interval.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub max_interval_offset: Option<u64>,
> +
> +    /// Minimum offset in seconds for the metric collection interval.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub min_interval_offset: Option<u64>,
> +
> +    /// Maximum random delay in milliseconds for each outgoing connection.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub max_connection_delay: Option<u64>,
> +
> +    /// Minimum random delay in milliseconds for each outgoing connection.
> +    #[serde(skip_serializing_if = "Option::is_none")]
> +    pub min_connection_delay: Option<u64>,
> +}
> +
> +#[api]
> +#[derive(Copy, Clone, Deserialize, Serialize)]
> +#[serde(rename_all = "kebab-case")]
> +/// Deletable property for [`CollectionSettings`].
> +pub enum DeletableCollectionSettingsProperty {
> +    /// Delete 'collection-interval'.
> +    CollectionInterval,
> +    /// Delete 'max-concurrent-connections'.
> +    MaxConcurrentConnections,
> +    /// Delete 'max-interval-offset'.
> +    MaxIntervalOffset,
> +    /// Delete 'min-interval-offset'.
> +    MinIntervalOffset,
> +    /// Delete 'min-connection-delay'.
> +    MaxConnectionDelay,
> +    /// Delete 'min-connection-delay'.
> +    MinConnectionDelay,
> +}
> +
> +impl CollectionSettings {
> +    /// Create a new settings instance with a given `id`.
> +    pub fn new(id: &str) -> Self {
> +        Self {
> +            id: id.into(),
> +            ..Default::default()
> +        }
> +    }
> +
> +    /// Return the collection interval or the default if not configured.
> +    pub fn collection_interval_or_default(&self) -> u64 {
> +        self.collection_interval
> +            .unwrap_or(DEFAULT_COLLECTION_INTERVAL)
> +    }
> +
> +    /// Return the number of allowed concurrent connections or the default if not configured.
> +    pub fn max_concurrent_connections_or_default(&self) -> usize {
> +        self.max_concurrent_connections
> +            .unwrap_or(DEFAULT_CONCURRENCY)
> +    }
> +
> +    /// Return the upper bound for the main loop delay or the default if not configured.
> +    pub fn max_interval_offset_or_default(&self) -> u64 {
> +        self.max_interval_offset.unwrap_or(DEFAULT_MAX_OFFSET)
> +    }
> +
> +    /// Return the lower bound for the main loop delay or the default if not configured.
> +    pub fn min_interval_offset_or_default(&self) -> u64 {
> +        self.min_interval_offset.unwrap_or(DEFAULT_MIN_OFFSET)
> +    }
> +
> +    /// Return the upper bound for the connection delay or the default if not configured.
> +    pub fn max_connection_delay_or_default(&self) -> u64 {
> +        self.max_connection_delay
> +            .unwrap_or(DEFAULT_MAX_CONNECTION_DELAY)
> +    }
> +
> +    /// Return the lower bound for the connection delay or the default if not configured.
> +    pub fn min_connection_delay_or_default(&self) -> u64 {
> +        self.min_connection_delay
> +            .unwrap_or(DEFAULT_MIN_CONNECTION_DELAY)
> +    }
> +}
> -- 
> 2.39.5


_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel


  reply	other threads:[~2025-02-18 15:26 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-02-14 13:06 [pdm-devel] [PATCH proxmox-datacenter-manager v2 00/28] metric collection improvements (concurrency, config, API, CLI) Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 01/28] test support: add NamedTempFile helper Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 02/28] test support: add NamedTempDir helper Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 03/28] pdm-api-types: add CollectionSettings type Lukas Wagner
2025-02-18 15:26   ` Wolfgang Bumiller [this message]
2025-02-18 15:31     ` Stefan Hanreich
2025-02-21  8:27     ` Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 04/28] pdm-config: add functions for reading/writing metric collection settings Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 05/28] metric collection: split top_entities split into separate module Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 06/28] metric collection: save metric data to RRD in separate task Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 07/28] metric collection: rework metric poll task Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 08/28] metric collection: persist state after metric collection Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 09/28] metric collection: skip if last_collection < MIN_COLLECTION_INTERVAL Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 10/28] metric collection: collect overdue metrics on startup/timer change Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 11/28] metric collection: add tests for the fetch_remotes function Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 12/28] metric collection: add test for fetch_overdue Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 13/28] metric collection: pass rrd cache instance as function parameter Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 14/28] metric collection: add test for rrd task Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 15/28] metric collection: wrap rrd_cache::Cache in a struct Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 16/28] metric collection: record remote response time in metric database Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 17/28] metric collection: save time needed for collection run to RRD Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 18/28] metric collection: periodically clean removed remotes from statefile Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 19/28] api: add endpoint for updating metric collection settings Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 20/28] api: add endpoint to trigger metric collection Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 21/28] api: remotes: trigger immediate metric collection for newly added nodes Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 22/28] api: add api for querying metric collection RRD data Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 23/28] api: metric-collection: add status endpoint Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 24/28] pdm-client: add metric collection API methods Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 25/28] cli: add commands for metric-collection settings, trigger, status Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 26/28] metric collection: factor out handle_tick and handle_control_message fns Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 27/28] metric collection: skip missed timer ticks Lukas Wagner
2025-02-14 13:06 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 28/28] metric collection: use JoinSet instead of joining from handles in a Vec Lukas Wagner
2025-02-21 13:19 ` [pdm-devel] [PATCH proxmox-datacenter-manager v2 00/28] metric collection improvements (concurrency, config, API, CLI) Maximiliano Sandoval
2025-03-14 14:10 ` Lukas Wagner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aguxeubkyqnqrfsrmmo4suxmap34ep34bhs5l3rhhqrjr33hlo@vw7u7jhojkka \
    --to=w.bumiller@proxmox.com \
    --cc=l.wagner@proxmox.com \
    --cc=pdm-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal