From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
Date: Thu, 19 Nov 2026 13:40:00 +0100 [thread overview]
Message-ID: <DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-7-d.rusovac@proxmox.com>
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> CRS did not rebalance a cluster that was clearly imbalanced in CPU load
> or memory usage alone, because the combined node load looked balanced
> enough [0]
>
> add weights to the scheduler, so that callers can prioritize balancing
> one resource over the other
>
> load imbalance computations and the bruteforce method use the weighted
> node load
>
> for the TOPSIS methods, the weights replace the hardcoded criteria
> weights from commit 2b325aa5 ("add pve_static module"). those favor
> memory usage 5 times over CPU load. Scaling them with the weights
> instead would make the weights relative to that built-in emphasis, so
> 1:1 would effectively mean 1:5, which is undesirable. only the factor of
> 2 between the criteria for the highest and the average usage of each
> resource is kept, so custom weights of 1:5 reproduce the previous
> criteria weights of 1:5 (average) and 2:10 (highest)
>
> the same weights are used for the TOPSIS criteria and the node load, so
> the imbalance reported for the migrations ranked by TOPSIS is weighted
> like the criteria that ranked them
>
> the weights default to 1:1. This changes the default ranking of the
> TOPSIS methods (expected result of one affected test was adjusted), so
> callers have to pass hardcoded legacy topsis weights to keep their
> previous rankings. with these, the node load is weighted 1:5 as well and
> the previous rankings are preserved, which is also being tested
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> new with changes regarding patch [0] in v1:
> * add weights as field to Scheduler struct
> * weigh the TOPSIS criteria directly with the weights, keeping the factor
> of 2 between the highest and the average usage criteria of each resource
> - weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
> - add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
> * fall back to previous TOPSIS ranking of migration candidates
> * no longer expose score and keep ScoredMigration as is
> * adjust one scheduler test's expected result
> * test that legacy topsis weights preserve previous rankings
>
> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
> proxmox-resource-scheduling/src/pve_static.rs | 4 +-
> proxmox-resource-scheduling/src/scheduler.rs | 105 ++++++++++++++----
> .../tests/scheduler.rs | 64 ++++++++++-
> 3 files changed, 149 insertions(+), 24 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
> index 40b76cbc..25439a0e 100644
> --- a/proxmox-resource-scheduling/src/pve_static.rs
> +++ b/proxmox-resource-scheduling/src/pve_static.rs
> @@ -1,7 +1,7 @@
> use anyhow::Error;
> use serde::{Deserialize, Serialize};
>
> -use crate::scheduler::{NodeUsage, Scheduler};
> +use crate::scheduler::{LEGACY_TOPSIS_WEIGHTS, NodeUsage, Scheduler};
> use crate::{node::NodeStats, resource::ResourceStats};
>
> #[derive(Clone, Serialize, Deserialize)]
> @@ -94,7 +94,7 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
> .map(|node| node.as_ref().clone().into())
> .collect::<Vec<NodeUsage>>();
>
> - let scheduler = Scheduler::from_nodes(nodes);
> + let scheduler = Scheduler::from_nodes(nodes).with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> scheduler.score_nodes_to_start_resource(*service)
> }
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index a9390357..75929a14 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -1,6 +1,10 @@
> use anyhow::Error;
>
> -use crate::{node::NodeStats, resource::ResourceStats, topsis};
> +use crate::{
> + node::{NodeStats, NodeStatsWeights},
> + resource::ResourceStats,
> + topsis,
> +};
>
> use serde::{Deserialize, Serialize};
> use std::{
> @@ -69,26 +73,69 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
> new_stats
> }
>
> -criteria_struct! {
> - /// A given alternative.
> - struct PveTopsisAlternative {
> - #[criterion("average CPU", -1.0)]
> - average_cpu: f64,
> - #[criterion("highest CPU", -2.0)]
> - highest_cpu: f64,
> - #[criterion("average memory", -5.0)]
> - average_memory: f64,
> - #[criterion("highest memory", -10.0)]
> - highest_memory: f64,
> +/// A given alternative.
> +struct PveTopsisAlternative {
> + average_cpu: f64,
> + highest_cpu: f64,
> + average_memory: f64,
> + highest_memory: f64,
> +}
> +
> +const N_CRITERIA: usize = 4;
> +
> +impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
> + fn from(alternative: PveTopsisAlternative) -> Self {
> + [
> + alternative.average_cpu,
> + alternative.highest_cpu,
> + alternative.average_memory,
> + alternative.highest_memory,
> + ]
> }
> +}
>
> - const N_CRITERIA;
> - static PVE_HA_TOPSIS_CRITERIA;
> +/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
> +/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
> +/// weights introduced in commit 2b325aa5.
nit: include the commit title here, i.e.
2b325aa5 ("add pve_static module")
> +///
> +/// For more details see [`topsis_criteria`].
> +const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
> +
> +/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with
here as well
> +/// [`topsis_criteria`].
> +pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
> + cpu_weight: 1.0,
> + mem_weight: 5.0,
> +};
> +
> +/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
> +///
> +/// The criteria for the average usage of each resource get the respective weight in `weights`,
> +/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
> +fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {
nit: topsis_criteria_from_weights() or make this a method of Scheduler,
which is its only user
> + let NodeStatsWeights {
> + cpu_weight,
> + mem_weight,
> + } = *weights;
> +
> + topsis::Criteria::new([
> + topsis::Criterion::new("average CPU".to_owned(), -cpu_weight),
> + topsis::Criterion::new(
> + "highest CPU".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * cpu_weight,
> + ),
> + topsis::Criterion::new("average memory".to_owned(), -mem_weight),
> + topsis::Criterion::new(
> + "highest memory".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * mem_weight,
> + ),
> + ])
> }
>
> #[derive(Clone, Debug)]
> pub struct Scheduler {
> nodes: Vec<NodeUsage>,
> + weights: NodeStatsWeights,
> }
>
> /// A possible migration.
> @@ -176,9 +223,28 @@ impl Scheduler {
> {
> Self {
> nodes: nodes.into_iter().map(|node| node.into()).collect(),
> + weights: NodeStatsWeights::EQUAL,
> }
> }
>
> + /// Sets the `weights` controlling how much balancing the CPU load and the memory usage of the
> + /// nodes is prioritized relative to each other.
> + ///
> + /// For [`Self::node_imbalance`], [`Self::node_imbalance_with_migration_candidate`], the
> + /// bruteforce method [`Self::score_best_balancing_migration_candidates`] and the imbalance
> + /// reported by [`Self::score_best_balancing_migration_candidates_topsis`], the `weights`
> + /// determine the [`NodeStats::weighted_load`] of each node.
> + ///
> + /// For the TOPSIS-based methods, [`Self::score_best_balancing_migration_candidates_topsis`] and
> + /// [`Self::score_nodes_to_start_resource`], the `weights` also weigh the average CPU and
> + /// average memory criteria, see [`topsis_criteria`]. Use [`LEGACY_TOPSIS_WEIGHTS`] to keep the
> + /// criteria weights introduced in commit 2b325aa5.
> + pub fn with_weights(mut self, weights: NodeStatsWeights) -> Self {
> + self.weights = weights;
> +
> + self
> + }
> +
> /// Maps the current node usages to a [`PveTopsisAlternative`].
> ///
> /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
> @@ -221,7 +287,7 @@ impl Scheduler {
> ///
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance(&self) -> f64 {
> - calculate_node_imbalance(&self.nodes, |node| node.stats.load())
> + calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(&self.weights))
> }
>
> /// Returns the load imbalance among the nodes as if a specific resource was moved.
> @@ -229,7 +295,7 @@ impl Scheduler {
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - node_stats_after_migration(candidate, node).load()
> + node_stats_after_migration(candidate, node).weighted_load(&self.weights)
> })
> }
>
> @@ -297,8 +363,9 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> + let criteria = topsis_criteria(&self.weights)?;
> let best_alternatives =
> - topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(best_alternatives
> .into_iter()
> @@ -343,8 +410,8 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> - let scores =
> - topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + let criteria = topsis_criteria(&self.weights)?;
> + let scores = topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(scores
> .into_iter()
> diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
> index 61fdd840..603a9418 100644
> --- a/proxmox-resource-scheduling/tests/scheduler.rs
> +++ b/proxmox-resource-scheduling/tests/scheduler.rs
> @@ -2,7 +2,7 @@ use anyhow::Error;
> use proxmox_resource_scheduling::{
> node::NodeStats,
> resource::ResourceStats,
> - scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
> + scheduler::{LEGACY_TOPSIS_WEIGHTS, Migration, MigrationCandidate, NodeUsage, Scheduler},
> };
>
> fn new_empty_cluster_scheduler() -> Scheduler {
> @@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
is the scheduler here intentionally not augmented with
.with_weights(LEGACY_TOPSIS_WEIGHTS) ?
>
> assert_eq!(
> rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
> - vec!["node2", "node3", "node1"]
> + vec!["node1", "node3", "node2"]
> );
>
> Ok(())
> @@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> #[test]
> fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
> - let scheduler = new_heterogeneous_cluster_scheduler();
> + let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> let heavy_memory_resource_stats = ResourceStats {
> cpu: 0.0,
> @@ -377,3 +377,61 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> Ok(())
> }
> +
> +#[test]
> +fn test_legacy_topsis_weights_keep_unweighted_rankings() -> Result<(), Error> {
> + let resource_stats =
> + [(1.0, 12 << 30), (12.0, 0), (0.0, 0), (12.0, 12 << 30)].map(|(maxcpu, maxmem)| {
> + ResourceStats {
> + cpu: 0.0,
> + maxcpu,
> + mem: 0,
> + maxmem,
> + }
> + });
> +
> + // rankings with the TOPSIS criteria weights from before the weights became adjustable
> + let expected_homogeneous_rankings = [
> + ["node2", "node3", "node1"],
> + ["node1", "node3", "node2"],
> + ["node1", "node3", "node2"],
> + ["node2", "node3", "node1"],
> + ];
> + let expected_heterogeneous_rankings = [
> + ["node2", "node1", "node3"],
> + ["node3", "node2", "node1"],
> + ["node1", "node3", "node2"],
> + ["node2", "node1", "node3"],
> + ];
> +
> + let scenarios: [(fn() -> Scheduler, _); 2] = [
> + (
> + new_homogeneous_cluster_scheduler,
> + expected_homogeneous_rankings,
> + ),
> + (
> + new_heterogeneous_cluster_scheduler,
> + expected_heterogeneous_rankings,
> + ),
> + ];
> +
> + for (new_scheduler, expected_rankings) in scenarios {
> + let legacy = new_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
> +
> + for (resource_stats, expected_ranking) in resource_stats.iter().zip(expected_rankings) {
> + assert_eq!(
> + rank_nodes_to_start_resource(&legacy, *resource_stats)?,
> + expected_ranking
> + );
> + }
> +
> + let (candidates, migration1, migration2) = new_simple_migration_candidates();
> +
> + assert_eq!(
> + rank_best_balancing_migration_candidates_topsis(&legacy, &candidates, 2)?,
> + vec![migration1, migration2]
> + );
> + }
> +
> + Ok(())
> +}
next prev parent reply other threads:[~2026-11-19 12:40 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:50 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41 ` Daniel Kral
2026-11-19 12:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40 ` Daniel Kral [this message]
2026-11-19 13:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45 ` Daniel Kral
2026-11-19 13:28 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com \
--to=d.kral@proxmox.com \
--cc=d.rusovac@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox