From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
Date: Thu, 19 Nov 2026 13:40:00 +0100 [thread overview]
Message-ID: <DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-7-d.rusovac@proxmox.com>
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> CRS did not rebalance a cluster that was clearly imbalanced in CPU load
> or memory usage alone, because the combined node load looked balanced
> enough [0]
>
> add weights to the scheduler, so that callers can prioritize balancing
> one resource over the other
>
> load imbalance computations and the bruteforce method use the weighted
> node load
>
> for the TOPSIS methods, the weights replace the hardcoded criteria
> weights from commit 2b325aa5 ("add pve_static module"). those favor
> memory usage 5 times over CPU load. Scaling them with the weights
> instead would make the weights relative to that built-in emphasis, so
> 1:1 would effectively mean 1:5, which is undesirable. only the factor of
> 2 between the criteria for the highest and the average usage of each
> resource is kept, so custom weights of 1:5 reproduce the previous
> criteria weights of 1:5 (average) and 2:10 (highest)
>
> the same weights are used for the TOPSIS criteria and the node load, so
> the imbalance reported for the migrations ranked by TOPSIS is weighted
> like the criteria that ranked them
>
> the weights default to 1:1. This changes the default ranking of the
> TOPSIS methods (expected result of one affected test was adjusted), so
> callers have to pass hardcoded legacy topsis weights to keep their
> previous rankings. with these, the node load is weighted 1:5 as well and
> the previous rankings are preserved, which is also being tested
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> new with changes regarding patch [0] in v1:
> * add weights as field to Scheduler struct
> * weigh the TOPSIS criteria directly with the weights, keeping the factor
> of 2 between the highest and the average usage criteria of each resource
> - weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
> - add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
> * fall back to previous TOPSIS ranking of migration candidates
> * no longer expose score and keep ScoredMigration as is
> * adjust one scheduler test's expected result
> * test that legacy topsis weights preserve previous rankings
>
> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
> proxmox-resource-scheduling/src/pve_static.rs | 4 +-
> proxmox-resource-scheduling/src/scheduler.rs | 105 ++++++++++++++----
> .../tests/scheduler.rs | 64 ++++++++++-
> 3 files changed, 149 insertions(+), 24 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
> index 40b76cbc..25439a0e 100644
> --- a/proxmox-resource-scheduling/src/pve_static.rs
> +++ b/proxmox-resource-scheduling/src/pve_static.rs
> @@ -1,7 +1,7 @@
> use anyhow::Error;
> use serde::{Deserialize, Serialize};
>
> -use crate::scheduler::{NodeUsage, Scheduler};
> +use crate::scheduler::{LEGACY_TOPSIS_WEIGHTS, NodeUsage, Scheduler};
> use crate::{node::NodeStats, resource::ResourceStats};
>
> #[derive(Clone, Serialize, Deserialize)]
> @@ -94,7 +94,7 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
> .map(|node| node.as_ref().clone().into())
> .collect::<Vec<NodeUsage>>();
>
> - let scheduler = Scheduler::from_nodes(nodes);
> + let scheduler = Scheduler::from_nodes(nodes).with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> scheduler.score_nodes_to_start_resource(*service)
> }
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index a9390357..75929a14 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -1,6 +1,10 @@
> use anyhow::Error;
>
> -use crate::{node::NodeStats, resource::ResourceStats, topsis};
> +use crate::{
> + node::{NodeStats, NodeStatsWeights},
> + resource::ResourceStats,
> + topsis,
> +};
>
> use serde::{Deserialize, Serialize};
> use std::{
> @@ -69,26 +73,69 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
> new_stats
> }
>
> -criteria_struct! {
> - /// A given alternative.
> - struct PveTopsisAlternative {
> - #[criterion("average CPU", -1.0)]
> - average_cpu: f64,
> - #[criterion("highest CPU", -2.0)]
> - highest_cpu: f64,
> - #[criterion("average memory", -5.0)]
> - average_memory: f64,
> - #[criterion("highest memory", -10.0)]
> - highest_memory: f64,
> +/// A given alternative.
> +struct PveTopsisAlternative {
> + average_cpu: f64,
> + highest_cpu: f64,
> + average_memory: f64,
> + highest_memory: f64,
> +}
> +
> +const N_CRITERIA: usize = 4;
> +
> +impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
> + fn from(alternative: PveTopsisAlternative) -> Self {
> + [
> + alternative.average_cpu,
> + alternative.highest_cpu,
> + alternative.average_memory,
> + alternative.highest_memory,
> + ]
> }
> +}
>
> - const N_CRITERIA;
> - static PVE_HA_TOPSIS_CRITERIA;
> +/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
> +/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
> +/// weights introduced in commit 2b325aa5.
nit: include the commit title here, i.e.
2b325aa5 ("add pve_static module")
> +///
> +/// For more details see [`topsis_criteria`].
> +const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
> +
> +/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with
here as well
> +/// [`topsis_criteria`].
> +pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
> + cpu_weight: 1.0,
> + mem_weight: 5.0,
> +};
> +
> +/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
> +///
> +/// The criteria for the average usage of each resource get the respective weight in `weights`,
> +/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
> +fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {
nit: topsis_criteria_from_weights() or make this a method of Scheduler,
which is its only user
> + let NodeStatsWeights {
> + cpu_weight,
> + mem_weight,
> + } = *weights;
> +
> + topsis::Criteria::new([
> + topsis::Criterion::new("average CPU".to_owned(), -cpu_weight),
> + topsis::Criterion::new(
> + "highest CPU".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * cpu_weight,
> + ),
> + topsis::Criterion::new("average memory".to_owned(), -mem_weight),
> + topsis::Criterion::new(
> + "highest memory".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * mem_weight,
> + ),
> + ])
> }
>
> #[derive(Clone, Debug)]
> pub struct Scheduler {
> nodes: Vec<NodeUsage>,
> + weights: NodeStatsWeights,
> }
>
> /// A possible migration.
> @@ -176,9 +223,28 @@ impl Scheduler {
> {
> Self {
> nodes: nodes.into_iter().map(|node| node.into()).collect(),
> + weights: NodeStatsWeights::EQUAL,
> }
> }
>
> + /// Sets the `weights` controlling how much balancing the CPU load and the memory usage of the
> + /// nodes is prioritized relative to each other.
> + ///
> + /// For [`Self::node_imbalance`], [`Self::node_imbalance_with_migration_candidate`], the
> + /// bruteforce method [`Self::score_best_balancing_migration_candidates`] and the imbalance
> + /// reported by [`Self::score_best_balancing_migration_candidates_topsis`], the `weights`
> + /// determine the [`NodeStats::weighted_load`] of each node.
> + ///
> + /// For the TOPSIS-based methods, [`Self::score_best_balancing_migration_candidates_topsis`] and
> + /// [`Self::score_nodes_to_start_resource`], the `weights` also weigh the average CPU and
> + /// average memory criteria, see [`topsis_criteria`]. Use [`LEGACY_TOPSIS_WEIGHTS`] to keep the
> + /// criteria weights introduced in commit 2b325aa5.
> + pub fn with_weights(mut self, weights: NodeStatsWeights) -> Self {
> + self.weights = weights;
> +
> + self
> + }
> +
> /// Maps the current node usages to a [`PveTopsisAlternative`].
> ///
> /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
> @@ -221,7 +287,7 @@ impl Scheduler {
> ///
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance(&self) -> f64 {
> - calculate_node_imbalance(&self.nodes, |node| node.stats.load())
> + calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(&self.weights))
> }
>
> /// Returns the load imbalance among the nodes as if a specific resource was moved.
> @@ -229,7 +295,7 @@ impl Scheduler {
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - node_stats_after_migration(candidate, node).load()
> + node_stats_after_migration(candidate, node).weighted_load(&self.weights)
> })
> }
>
> @@ -297,8 +363,9 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> + let criteria = topsis_criteria(&self.weights)?;
> let best_alternatives =
> - topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(best_alternatives
> .into_iter()
> @@ -343,8 +410,8 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> - let scores =
> - topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + let criteria = topsis_criteria(&self.weights)?;
> + let scores = topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(scores
> .into_iter()
> diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
> index 61fdd840..603a9418 100644
> --- a/proxmox-resource-scheduling/tests/scheduler.rs
> +++ b/proxmox-resource-scheduling/tests/scheduler.rs
> @@ -2,7 +2,7 @@ use anyhow::Error;
> use proxmox_resource_scheduling::{
> node::NodeStats,
> resource::ResourceStats,
> - scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
> + scheduler::{LEGACY_TOPSIS_WEIGHTS, Migration, MigrationCandidate, NodeUsage, Scheduler},
> };
>
> fn new_empty_cluster_scheduler() -> Scheduler {
> @@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
is the scheduler here intentionally not augmented with
.with_weights(LEGACY_TOPSIS_WEIGHTS) ?
>
> assert_eq!(
> rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
> - vec!["node2", "node3", "node1"]
> + vec!["node1", "node3", "node2"]
> );
>
> Ok(())
> @@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> #[test]
> fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
> - let scheduler = new_heterogeneous_cluster_scheduler();
> + let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> let heavy_memory_resource_stats = ResourceStats {
> cpu: 0.0,
> @@ -377,3 +377,61 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> Ok(())
> }
> +
> +#[test]
> +fn test_legacy_topsis_weights_keep_unweighted_rankings() -> Result<(), Error> {
> + let resource_stats =
> + [(1.0, 12 << 30), (12.0, 0), (0.0, 0), (12.0, 12 << 30)].map(|(maxcpu, maxmem)| {
> + ResourceStats {
> + cpu: 0.0,
> + maxcpu,
> + mem: 0,
> + maxmem,
> + }
> + });
> +
> + // rankings with the TOPSIS criteria weights from before the weights became adjustable
> + let expected_homogeneous_rankings = [
> + ["node2", "node3", "node1"],
> + ["node1", "node3", "node2"],
> + ["node1", "node3", "node2"],
> + ["node2", "node3", "node1"],
> + ];
> + let expected_heterogeneous_rankings = [
> + ["node2", "node1", "node3"],
> + ["node3", "node2", "node1"],
> + ["node1", "node3", "node2"],
> + ["node2", "node1", "node3"],
> + ];
> +
> + let scenarios: [(fn() -> Scheduler, _); 2] = [
> + (
> + new_homogeneous_cluster_scheduler,
> + expected_homogeneous_rankings,
> + ),
> + (
> + new_heterogeneous_cluster_scheduler,
> + expected_heterogeneous_rankings,
> + ),
> + ];
> +
> + for (new_scheduler, expected_rankings) in scenarios {
> + let legacy = new_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
> +
> + for (resource_stats, expected_ranking) in resource_stats.iter().zip(expected_rankings) {
> + assert_eq!(
> + rank_nodes_to_start_resource(&legacy, *resource_stats)?,
> + expected_ranking
> + );
> + }
> +
> + let (candidates, migration1, migration2) = new_simple_migration_candidates();
> +
> + assert_eq!(
> + rank_best_balancing_migration_candidates_topsis(&legacy, &candidates, 2)?,
> + vec![migration1, migration2]
> + );
> + }
> +
> + Ok(())
> +}
next prev parent reply other threads:[~2026-11-19 12:40 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:50 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41 ` Daniel Kral
2026-11-19 12:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40 ` Daniel Kral [this message]
2026-11-19 13:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45 ` Daniel Kral
2026-11-19 13:28 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ708HZS6D0.26KL5P0JR38QU@proxmox.com \
--to=d.kral@proxmox.com \
--cc=d.rusovac@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.