public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options
Date: Thu, 19 Nov 2026 13:45:03 +0100	[thread overview]
Message-ID: <DMZ743TVBU4A.1S9VS6988B1F6@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-10-d.rusovac@proxmox.com>

On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> allow passing options to the constructor of the static and dynamic
> scheduler, which specify the weights of the CPU load and memory usage
> applied to every load balancing computation. these options are a single
> optional parameter with only optional fields, so the signatures of the
> existing methods stay the same
>
> with specified weights, every rebalancing method uses them for the load
> imbalance and the TOPSIS criteria alike, so that for example 1:1 means
> 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS
> criteria weights
>
> without specified weights, every method keeps its previous behavior: the
> load imbalance and the bruteforce method use equal weights (1:1); the
> TOPSIS methods use weights of 1:5 for their criteria
>
> NOTE: weights are not validated. if both are 0, there is simply no load
> imbalance and the TOPSIS methods fail, so callers must not pass such
> weights
>
> NOTE: to decide whether a migration improves the balance enough, the HA
> manager compares the load imbalance reported for the migrations ranked
> by TOPSIS with the current load imbalance. without specified weights,
> the TOPSIS ranking would report an imbalance weighted 1:5, while the
> load imbalance is unweighted. so, without specified weights, recompute
> the reported imbalance of the ranked migrations with equal weights, as
> before. while this requires reiterating all migrations candidates, it is
> required to keep older versions of the HA manager working as before by
> preventing them from running into this undesirable load imbalance
> mismatch
>
> add test cases checking that:
> - without options or with unspecified options, the load imbalance and
>   the node scores stay the same as before
> - equal weights keep the unweighted load imbalance, regardless of their
>   magnitude
> - weights of 1:5 keep the previous node scores, but change the load
>   imbalance, as it's also being weighted 1:5
> - weighting only the CPU load or only the memory usage results in a
>   different load imbalance
> - the TOPSIS ranking reports the same load imbalance as the bruteforce
>   method, both without specified options and with weights of 1:5, as the
>   HA manager compares it with the current load imbalance
> - without specified options, the TOPSIS ranking is the same as with
>   weights of 1:5
> - the dynamic scheduler accepts options as well
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
>     changes since v1:
>     * reword commit
>     * pass the weights once as optional scheduler options to the constructor
>       instead of adding weight parameters to every method, as suggested in
>       the review of v1, so the signatures of the existing methods stay the
>       same and older HA manager versions can still call them
>     * options are a single optional hash with only optional fields, a
>       missing weight defaults to 1
>     * without options, every method keeps its previous behavior, including
>       the TOPSIS ranking with weights of 1:5
>     * without options, recompute the imbalance reported with the TOPSIS
>       ranking with equal weights, so that it stays comparable to the node
>       imbalance for older HA manager version
>     * weights also apply to score_nodes_to_start_service()
>     * drop the adjustment of the existing test, as calling the methods
>       without weights keeps working
>     * add test cases for the scheduler options
>     
>     requires the proxmox-resource-scheduling patches of this series
>
>  .../src/bindings/resource_scheduling/mod.rs   |   1 +
>  .../bindings/resource_scheduling/options.rs   |  59 ++++++++
>  .../resource_scheduling/pve_dynamic.rs        |  71 +++++++--
>  .../resource_scheduling/pve_static.rs         |  64 ++++++--
>  pve-rs/test/resource_scheduling.pl            | 141 ++++++++++++++++++
>  5 files changed, 304 insertions(+), 32 deletions(-)
>  create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/bindings/resource_scheduling/mod.rs
> index 87b4a03..dc94f21 100644
> --- a/pve-rs/src/bindings/resource_scheduling/mod.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/mod.rs
> @@ -1,5 +1,6 @@
>  //! Resource scheduling related bindings.
>  
> +mod options;
>  mod resource;
>  mod usage;
>  
> diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/src/bindings/resource_scheduling/options.rs
> new file mode 100644
> index 0000000..6759cc3
> --- /dev/null
> +++ b/pve-rs/src/bindings/resource_scheduling/options.rs
> @@ -0,0 +1,59 @@
> +//! Options for the resource scheduling bindings.
> +
> +use serde::Deserialize;
> +
> +use proxmox_resource_scheduling::node::NodeStatsWeights;
> +use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredMigration};
> +use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
> +
> +/// Options for a scheduler instance.
> +#[derive(Clone, Copy, Debug, Default, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct SchedulerOptions {
> +    /// The weight of the CPU load of nodes. Defaults to 1 if only the other weight is specified.
> +    pub cpu_weight: Option<usize>,
> +    /// The weight of the memory usage of nodes. Defaults to 1 if only the other weight is
> +    /// specified.
> +    pub mem_weight: Option<usize>,
> +}
> +
> +impl SchedulerOptions {
> +    /// Returns the weights or [`None`] if no weight is set.
> +    pub fn weights(&self) -> Option<NodeStatsWeights> {
> +        if self.cpu_weight.is_none() && self.mem_weight.is_none() {
> +            return None;
> +        }
> +
> +        let mut weights = NodeStatsWeights::default();
> +        // unspecified weight defaults to 1
> +        weights.cpu_weight = self.cpu_weight.unwrap_or(1) as f64;
> +        weights.mem_weight = self.mem_weight.unwrap_or(1) as f64;
> +
> +        Some(weights)
> +    }
> +}
> +
> +/// Replaces the imbalance of each of the `scored_migrations` with the unweighted imbalance of the
> +/// respective migration in `candidates`, for the cluster usage `usage` aggregated by `F`.
> +pub fn with_unweighted_imbalance<F: UsageAggregator>(
> +    usage: &Usage,
> +    candidates: &[MigrationCandidate],
> +    scored_migrations: Vec<ScoredMigration>,
> +) -> Vec<ScoredMigration> {
> +    let scheduler = usage.to_scheduler::<F>();
> +
> +    scored_migrations
> +        .into_iter()
> +        // every scored migration should belong to some candidate
> +        .flat_map(|scored_migration| {
> +            candidates
> +                .iter()
> +                .find(|candidate| candidate.migration == scored_migration.migration)
> +                .map(|candidate| {
> +                    let imbalance = scheduler.node_imbalance_with_migration_candidate(candidate);
> +
> +                    ScoredMigration::new(scored_migration.migration, imbalance)
> +                })
> +        })
> +        .collect()
> +}
> diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> index 8e066e0..62d6426 100644
> --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
>      use serde::{Deserialize, Serialize};
>  
>      use perlmod::Value;
> -    use proxmox_resource_scheduling::node::NodeStats;
> +    use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
>      use proxmox_resource_scheduling::resource::ResourceStats;
> -    use proxmox_resource_scheduling::scheduler::ScoredMigration;
> -    use proxmox_resource_scheduling::usage::Usage;
> +    use proxmox_resource_scheduling::scheduler::{
> +        LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
> +    };
> +    use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
>  
> +    use crate::bindings::resource_scheduling::options::{
> +        SchedulerOptions, with_unweighted_imbalance,
> +    };
>      use crate::bindings::resource_scheduling::resource::{
>          CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
>      };
> @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic {
>      /// A scheduler instance contains the cluster usage.
>      pub struct Scheduler {
>          inner: Mutex<Usage>,
> +        weights: Option<NodeStatsWeights>,
> +    }
> +
> +    impl Scheduler {
> +        /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
> +        /// this instance or `default_weights` if this instance has none.
> +        fn to_weighted_scheduler<F: UsageAggregator>(
> +            &self,
> +            usage: &Usage,
> +            default_weights: NodeStatsWeights,
> +        ) -> CoreScheduler {
> +            usage
> +                .to_scheduler::<F>()
> +                .with_weights(self.weights.unwrap_or(default_weights))
> +        }

Hm, not a big one, but this could also just live in
proxmox_resource_scheduling itself, where Usage implements a

    pub fn to_weighted_scheduler<F: UsageAggregator>(&self, weights: NodeStatsWeights) -> Scheduler {
        let node_usages = F::aggregate(self);

        Scheduler::from_nodes(node_usages).with_weights(weights)
    }

and then the unwrap_or() could be made explicitly where it's called to
it's clear that it's a default fallback in there.

No hard feelings though.

>      }
>  
>      #[derive(Clone, Copy, Debug, Serialize, Deserialize)]
> @@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
>  
>      type DynamicResource = PveResource<DynamicResourceStats>;
>  
> -    /// Class method: Create a new [`Scheduler`] instance.
> +    /// Class method: Create a new [`Scheduler`] instance with the optional `options`.
>      ///
> -    /// See [`proxmox_resource_scheduling::usage::Usage::new`].
> +    /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`].
>      #[export(raw_return)]
> -    pub fn new(#[raw] class: Value) -> Result<Value, Error> {
> +    pub fn new(#[raw] class: Value, options: Option<SchedulerOptions>) -> Result<Value, Error> {
>          let inner = Usage::new();
> +        let weights = options.unwrap_or_default().weights();
>  
>          Ok(perlmod::instantiate_magic!(
> -            &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) })
> +            &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights })
>          ))
>      }
>  
> @@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
>      pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
>          let usage = this.inner.lock().unwrap();
>  
> -        usage.to_scheduler::<IdentityAggregator>().node_imbalance()
> +        this.to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
> +            .node_imbalance()
>      }
>  
>      /// Method: Scores the given migration `candidates` by the best node imbalance improvement with
> @@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
>  
>          let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>  
> -        Ok(usage
> -            .to_scheduler::<IdentityAggregator>()
> +        Ok(this
> +            .to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
>              .score_best_balancing_migration_candidates(candidates, limit))
>      }
>  
> @@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic {
>  
>          let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>  
> -        usage
> -            .to_scheduler::<IdentityAggregator>()
> -            .score_best_balancing_migration_candidates_topsis(&candidates, limit)
> +        let scored_migrations = this
> +            .to_weighted_scheduler::<IdentityAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
> +            .score_best_balancing_migration_candidates_topsis(&candidates, limit)?;
> +
> +        if this.weights.is_some() {
> +            return Ok(scored_migrations);
> +        }
> +
> +        // callers not passing any weights would compare the 1:5 weighted imbalance reported for the
> +        // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted
> +        // imbalance
> +        //
> +        // to avoid this mismatch, report the unweighted imbalance instead

good comment!

> +        Ok(with_unweighted_imbalance::<IdentityAggregator>(
> +            &usage,
> +            &candidates,
> +            scored_migrations,
> +        ))
>      }
>  
>      /// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
> @@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic {
>      ) -> Result<Vec<(String, f64)>, Error> {
>          let usage = this.inner.lock().unwrap();
>  
> -        usage
> -            .to_scheduler::<StartingAsStartedResourceAggregator>()
> -            .score_nodes_to_start_resource(resource_stats)
> +        this.to_weighted_scheduler::<StartingAsStartedResourceAggregator>(
> +            &usage,
> +            LEGACY_TOPSIS_WEIGHTS,
> +        )
> +        .score_nodes_to_start_resource(resource_stats)
>      }
>  }




  reply	other threads:[~2026-11-19 12:45 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-12  9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:50     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41   ` Daniel Kral
2026-11-19 12:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40   ` Daniel Kral
2026-11-19 13:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45   ` Daniel Kral [this message]
2026-11-19 13:28     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZ743TVBU4A.1S9VS6988B1F6@proxmox.com \
    --to=d.kral@proxmox.com \
    --cc=d.rusovac@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal