public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
Date: Thu, 19 Nov 2026 13:40:00 +0100	[thread overview]
Message-ID: <DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-7-d.rusovac@proxmox.com>

On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> CRS did not rebalance a cluster that was clearly imbalanced in CPU load
> or memory usage alone, because the combined node load looked balanced
> enough [0]
>
> add weights to the scheduler, so that callers can prioritize balancing
> one resource over the other
>
> load imbalance computations and the bruteforce method use the weighted
> node load
>
> for the TOPSIS methods, the weights replace the hardcoded criteria
> weights from commit 2b325aa5 ("add pve_static module"). those favor
> memory usage 5 times over CPU load. Scaling them with the weights
> instead would make the weights relative to that built-in emphasis, so
> 1:1 would effectively mean 1:5, which is undesirable. only the factor of
> 2 between the criteria for the highest and the average usage of each
> resource is kept, so custom weights of 1:5 reproduce the previous
> criteria weights of 1:5 (average) and 2:10 (highest)
>
> the same weights are used for the TOPSIS criteria and the node load, so
> the imbalance reported for the migrations ranked by TOPSIS is weighted
> like the criteria that ranked them
>
> the weights default to 1:1. This changes the default ranking of the
> TOPSIS methods (expected result of one affected test was adjusted), so
> callers have to pass hardcoded legacy topsis weights to keep their
> previous rankings. with these, the node load is weighted 1:5 as well and
> the previous rankings are preserved, which is also being tested
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
>     new with changes regarding patch [0] in v1:
>     * add weights as field to Scheduler struct
>     * weigh the TOPSIS criteria directly with the weights, keeping the factor
>       of 2 between the highest and the average usage criteria of each resource
>       - weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
>       - add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
>     * fall back to previous TOPSIS ranking of migration candidates
>     * no longer expose score and keep ScoredMigration as is
>     * adjust one scheduler test's expected result
>     * test that legacy topsis weights preserve previous rankings
>     
>     [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
>  proxmox-resource-scheduling/src/pve_static.rs |   4 +-
>  proxmox-resource-scheduling/src/scheduler.rs  | 105 ++++++++++++++----
>  .../tests/scheduler.rs                        |  64 ++++++++++-
>  3 files changed, 149 insertions(+), 24 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
> index 40b76cbc..25439a0e 100644
> --- a/proxmox-resource-scheduling/src/pve_static.rs
> +++ b/proxmox-resource-scheduling/src/pve_static.rs
> @@ -1,7 +1,7 @@
>  use anyhow::Error;
>  use serde::{Deserialize, Serialize};
>  
> -use crate::scheduler::{NodeUsage, Scheduler};
> +use crate::scheduler::{LEGACY_TOPSIS_WEIGHTS, NodeUsage, Scheduler};
>  use crate::{node::NodeStats, resource::ResourceStats};
>  
>  #[derive(Clone, Serialize, Deserialize)]
> @@ -94,7 +94,7 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
>          .map(|node| node.as_ref().clone().into())
>          .collect::<Vec<NodeUsage>>();
>  
> -    let scheduler = Scheduler::from_nodes(nodes);
> +    let scheduler = Scheduler::from_nodes(nodes).with_weights(LEGACY_TOPSIS_WEIGHTS);
>  
>      scheduler.score_nodes_to_start_resource(*service)
>  }
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index a9390357..75929a14 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -1,6 +1,10 @@
>  use anyhow::Error;
>  
> -use crate::{node::NodeStats, resource::ResourceStats, topsis};
> +use crate::{
> +    node::{NodeStats, NodeStatsWeights},
> +    resource::ResourceStats,
> +    topsis,
> +};
>  
>  use serde::{Deserialize, Serialize};
>  use std::{
> @@ -69,26 +73,69 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
>      new_stats
>  }
>  
> -criteria_struct! {
> -    /// A given alternative.
> -    struct PveTopsisAlternative {
> -        #[criterion("average CPU", -1.0)]
> -        average_cpu: f64,
> -        #[criterion("highest CPU", -2.0)]
> -        highest_cpu: f64,
> -        #[criterion("average memory", -5.0)]
> -        average_memory: f64,
> -        #[criterion("highest memory", -10.0)]
> -        highest_memory: f64,
> +/// A given alternative.
> +struct PveTopsisAlternative {
> +    average_cpu: f64,
> +    highest_cpu: f64,
> +    average_memory: f64,
> +    highest_memory: f64,
> +}
> +
> +const N_CRITERIA: usize = 4;
> +
> +impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
> +    fn from(alternative: PveTopsisAlternative) -> Self {
> +        [
> +            alternative.average_cpu,
> +            alternative.highest_cpu,
> +            alternative.average_memory,
> +            alternative.highest_memory,
> +        ]
>      }
> +}
>  
> -    const N_CRITERIA;
> -    static PVE_HA_TOPSIS_CRITERIA;
> +/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
> +/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
> +/// weights introduced in commit 2b325aa5.

nit: include the commit title here, i.e.

2b325aa5 ("add pve_static module")

> +///
> +/// For more details see [`topsis_criteria`].
> +const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
> +
> +/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with

here as well

> +/// [`topsis_criteria`].
> +pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
> +    cpu_weight: 1.0,
> +    mem_weight: 5.0,
> +};
> +
> +/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
> +///
> +/// The criteria for the average usage of each resource get the respective weight in `weights`,
> +/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
> +fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {

nit: topsis_criteria_from_weights() or make this a method of Scheduler,
     which is its only user

> +    let NodeStatsWeights {
> +        cpu_weight,
> +        mem_weight,
> +    } = *weights;
> +
> +    topsis::Criteria::new([
> +        topsis::Criterion::new("average CPU".to_owned(), -cpu_weight),
> +        topsis::Criterion::new(
> +            "highest CPU".to_owned(),
> +            -TOPSIS_HIGHEST_USAGE_FACTOR * cpu_weight,
> +        ),
> +        topsis::Criterion::new("average memory".to_owned(), -mem_weight),
> +        topsis::Criterion::new(
> +            "highest memory".to_owned(),
> +            -TOPSIS_HIGHEST_USAGE_FACTOR * mem_weight,
> +        ),
> +    ])
>  }
>  
>  #[derive(Clone, Debug)]
>  pub struct Scheduler {
>      nodes: Vec<NodeUsage>,
> +    weights: NodeStatsWeights,
>  }
>  
>  /// A possible migration.
> @@ -176,9 +223,28 @@ impl Scheduler {
>      {
>          Self {
>              nodes: nodes.into_iter().map(|node| node.into()).collect(),
> +            weights: NodeStatsWeights::EQUAL,
>          }
>      }
>  
> +    /// Sets the `weights` controlling how much balancing the CPU load and the memory usage of the
> +    /// nodes is prioritized relative to each other.
> +    ///
> +    /// For [`Self::node_imbalance`], [`Self::node_imbalance_with_migration_candidate`], the
> +    /// bruteforce method [`Self::score_best_balancing_migration_candidates`] and the imbalance
> +    /// reported by [`Self::score_best_balancing_migration_candidates_topsis`], the `weights`
> +    /// determine the [`NodeStats::weighted_load`] of each node.
> +    ///
> +    /// For the TOPSIS-based methods, [`Self::score_best_balancing_migration_candidates_topsis`] and
> +    /// [`Self::score_nodes_to_start_resource`], the `weights` also weigh the average CPU and
> +    /// average memory criteria, see [`topsis_criteria`]. Use [`LEGACY_TOPSIS_WEIGHTS`] to keep the
> +    /// criteria weights introduced in commit 2b325aa5.
> +    pub fn with_weights(mut self, weights: NodeStatsWeights) -> Self {
> +        self.weights = weights;
> +
> +        self
> +    }
> +
>      /// Maps the current node usages to a [`PveTopsisAlternative`].
>      ///
>      /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
> @@ -221,7 +287,7 @@ impl Scheduler {
>      ///
>      /// See [`calculate_node_imbalance`] for more information.
>      pub fn node_imbalance(&self) -> f64 {
> -        calculate_node_imbalance(&self.nodes, |node| node.stats.load())
> +        calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(&self.weights))
>      }
>  
>      /// Returns the load imbalance among the nodes as if a specific resource was moved.
> @@ -229,7 +295,7 @@ impl Scheduler {
>      /// See [`calculate_node_imbalance`] for more information.
>      pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
>          calculate_node_imbalance(&self.nodes, |node| {
> -            node_stats_after_migration(candidate, node).load()
> +            node_stats_after_migration(candidate, node).weighted_load(&self.weights)
>          })
>      }
>  
> @@ -297,8 +363,9 @@ impl Scheduler {
>              })
>              .collect::<Vec<_>>();
>  
> +        let criteria = topsis_criteria(&self.weights)?;
>          let best_alternatives =
> -            topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> +            topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>  
>          Ok(best_alternatives
>              .into_iter()
> @@ -343,8 +410,8 @@ impl Scheduler {
>              })
>              .collect::<Vec<_>>();
>  
> -        let scores =
> -            topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> +        let criteria = topsis_criteria(&self.weights)?;
> +        let scores = topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>  
>          Ok(scores
>              .into_iter()
> diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
> index 61fdd840..603a9418 100644
> --- a/proxmox-resource-scheduling/tests/scheduler.rs
> +++ b/proxmox-resource-scheduling/tests/scheduler.rs
> @@ -2,7 +2,7 @@ use anyhow::Error;
>  use proxmox_resource_scheduling::{
>      node::NodeStats,
>      resource::ResourceStats,
> -    scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
> +    scheduler::{LEGACY_TOPSIS_WEIGHTS, Migration, MigrationCandidate, NodeUsage, Scheduler},
>  };
>  
>  fn new_empty_cluster_scheduler() -> Scheduler {
> @@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {

is the scheduler here intentionally not augmented with

.with_weights(LEGACY_TOPSIS_WEIGHTS) ?

>  
>      assert_eq!(
>          rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
> -        vec!["node2", "node3", "node1"]
> +        vec!["node1", "node3", "node2"]
>      );
>  
>      Ok(())
> @@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>  
>  #[test]
>  fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
> -    let scheduler = new_heterogeneous_cluster_scheduler();
> +    let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
>  
>      let heavy_memory_resource_stats = ResourceStats {
>          cpu: 0.0,
> @@ -377,3 +377,61 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
>  
>      Ok(())
>  }
> +
> +#[test]
> +fn test_legacy_topsis_weights_keep_unweighted_rankings() -> Result<(), Error> {
> +    let resource_stats =
> +        [(1.0, 12 << 30), (12.0, 0), (0.0, 0), (12.0, 12 << 30)].map(|(maxcpu, maxmem)| {
> +            ResourceStats {
> +                cpu: 0.0,
> +                maxcpu,
> +                mem: 0,
> +                maxmem,
> +            }
> +        });
> +
> +    // rankings with the TOPSIS criteria weights from before the weights became adjustable
> +    let expected_homogeneous_rankings = [
> +        ["node2", "node3", "node1"],
> +        ["node1", "node3", "node2"],
> +        ["node1", "node3", "node2"],
> +        ["node2", "node3", "node1"],
> +    ];
> +    let expected_heterogeneous_rankings = [
> +        ["node2", "node1", "node3"],
> +        ["node3", "node2", "node1"],
> +        ["node1", "node3", "node2"],
> +        ["node2", "node1", "node3"],
> +    ];
> +
> +    let scenarios: [(fn() -> Scheduler, _); 2] = [
> +        (
> +            new_homogeneous_cluster_scheduler,
> +            expected_homogeneous_rankings,
> +        ),
> +        (
> +            new_heterogeneous_cluster_scheduler,
> +            expected_heterogeneous_rankings,
> +        ),
> +    ];
> +
> +    for (new_scheduler, expected_rankings) in scenarios {
> +        let legacy = new_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
> +
> +        for (resource_stats, expected_ranking) in resource_stats.iter().zip(expected_rankings) {
> +            assert_eq!(
> +                rank_nodes_to_start_resource(&legacy, *resource_stats)?,
> +                expected_ranking
> +            );
> +        }
> +
> +        let (candidates, migration1, migration2) = new_simple_migration_candidates();
> +
> +        assert_eq!(
> +            rank_best_balancing_migration_candidates_topsis(&legacy, &candidates, 2)?,
> +            vec![migration1, migration2]
> +        );
> +    }
> +
> +    Ok(())
> +}






  reply	other threads:[~2026-11-19 12:40 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-12  9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:50     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41   ` Daniel Kral
2026-11-19 12:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40   ` Daniel Kral [this message]
2026-11-19 13:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45   ` Daniel Kral
2026-11-19 13:28     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com \
    --to=d.kral@proxmox.com \
    --cc=d.rusovac@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal