From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options
Date: Thu, 19 Nov 2026 13:45:03 +0100 [thread overview]
Message-ID: <DMZ743TVBU4A.1S9VS6988B1F6@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-10-d.rusovac@proxmox.com>
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> allow passing options to the constructor of the static and dynamic
> scheduler, which specify the weights of the CPU load and memory usage
> applied to every load balancing computation. these options are a single
> optional parameter with only optional fields, so the signatures of the
> existing methods stay the same
>
> with specified weights, every rebalancing method uses them for the load
> imbalance and the TOPSIS criteria alike, so that for example 1:1 means
> 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS
> criteria weights
>
> without specified weights, every method keeps its previous behavior: the
> load imbalance and the bruteforce method use equal weights (1:1); the
> TOPSIS methods use weights of 1:5 for their criteria
>
> NOTE: weights are not validated. if both are 0, there is simply no load
> imbalance and the TOPSIS methods fail, so callers must not pass such
> weights
>
> NOTE: to decide whether a migration improves the balance enough, the HA
> manager compares the load imbalance reported for the migrations ranked
> by TOPSIS with the current load imbalance. without specified weights,
> the TOPSIS ranking would report an imbalance weighted 1:5, while the
> load imbalance is unweighted. so, without specified weights, recompute
> the reported imbalance of the ranked migrations with equal weights, as
> before. while this requires reiterating all migrations candidates, it is
> required to keep older versions of the HA manager working as before by
> preventing them from running into this undesirable load imbalance
> mismatch
>
> add test cases checking that:
> - without options or with unspecified options, the load imbalance and
> the node scores stay the same as before
> - equal weights keep the unweighted load imbalance, regardless of their
> magnitude
> - weights of 1:5 keep the previous node scores, but change the load
> imbalance, as it's also being weighted 1:5
> - weighting only the CPU load or only the memory usage results in a
> different load imbalance
> - the TOPSIS ranking reports the same load imbalance as the bruteforce
> method, both without specified options and with weights of 1:5, as the
> HA manager compares it with the current load imbalance
> - without specified options, the TOPSIS ranking is the same as with
> weights of 1:5
> - the dynamic scheduler accepts options as well
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> changes since v1:
> * reword commit
> * pass the weights once as optional scheduler options to the constructor
> instead of adding weight parameters to every method, as suggested in
> the review of v1, so the signatures of the existing methods stay the
> same and older HA manager versions can still call them
> * options are a single optional hash with only optional fields, a
> missing weight defaults to 1
> * without options, every method keeps its previous behavior, including
> the TOPSIS ranking with weights of 1:5
> * without options, recompute the imbalance reported with the TOPSIS
> ranking with equal weights, so that it stays comparable to the node
> imbalance for older HA manager version
> * weights also apply to score_nodes_to_start_service()
> * drop the adjustment of the existing test, as calling the methods
> without weights keeps working
> * add test cases for the scheduler options
>
> requires the proxmox-resource-scheduling patches of this series
>
> .../src/bindings/resource_scheduling/mod.rs | 1 +
> .../bindings/resource_scheduling/options.rs | 59 ++++++++
> .../resource_scheduling/pve_dynamic.rs | 71 +++++++--
> .../resource_scheduling/pve_static.rs | 64 ++++++--
> pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++
> 5 files changed, 304 insertions(+), 32 deletions(-)
> create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/bindings/resource_scheduling/mod.rs
> index 87b4a03..dc94f21 100644
> --- a/pve-rs/src/bindings/resource_scheduling/mod.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/mod.rs
> @@ -1,5 +1,6 @@
> //! Resource scheduling related bindings.
>
> +mod options;
> mod resource;
> mod usage;
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/src/bindings/resource_scheduling/options.rs
> new file mode 100644
> index 0000000..6759cc3
> --- /dev/null
> +++ b/pve-rs/src/bindings/resource_scheduling/options.rs
> @@ -0,0 +1,59 @@
> +//! Options for the resource scheduling bindings.
> +
> +use serde::Deserialize;
> +
> +use proxmox_resource_scheduling::node::NodeStatsWeights;
> +use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredMigration};
> +use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
> +
> +/// Options for a scheduler instance.
> +#[derive(Clone, Copy, Debug, Default, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct SchedulerOptions {
> + /// The weight of the CPU load of nodes. Defaults to 1 if only the other weight is specified.
> + pub cpu_weight: Option<usize>,
> + /// The weight of the memory usage of nodes. Defaults to 1 if only the other weight is
> + /// specified.
> + pub mem_weight: Option<usize>,
> +}
> +
> +impl SchedulerOptions {
> + /// Returns the weights or [`None`] if no weight is set.
> + pub fn weights(&self) -> Option<NodeStatsWeights> {
> + if self.cpu_weight.is_none() && self.mem_weight.is_none() {
> + return None;
> + }
> +
> + let mut weights = NodeStatsWeights::default();
> + // unspecified weight defaults to 1
> + weights.cpu_weight = self.cpu_weight.unwrap_or(1) as f64;
> + weights.mem_weight = self.mem_weight.unwrap_or(1) as f64;
> +
> + Some(weights)
> + }
> +}
> +
> +/// Replaces the imbalance of each of the `scored_migrations` with the unweighted imbalance of the
> +/// respective migration in `candidates`, for the cluster usage `usage` aggregated by `F`.
> +pub fn with_unweighted_imbalance<F: UsageAggregator>(
> + usage: &Usage,
> + candidates: &[MigrationCandidate],
> + scored_migrations: Vec<ScoredMigration>,
> +) -> Vec<ScoredMigration> {
> + let scheduler = usage.to_scheduler::<F>();
> +
> + scored_migrations
> + .into_iter()
> + // every scored migration should belong to some candidate
> + .flat_map(|scored_migration| {
> + candidates
> + .iter()
> + .find(|candidate| candidate.migration == scored_migration.migration)
> + .map(|candidate| {
> + let imbalance = scheduler.node_imbalance_with_migration_candidate(candidate);
> +
> + ScoredMigration::new(scored_migration.migration, imbalance)
> + })
> + })
> + .collect()
> +}
> diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> index 8e066e0..62d6426 100644
> --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
> use serde::{Deserialize, Serialize};
>
> use perlmod::Value;
> - use proxmox_resource_scheduling::node::NodeStats;
> + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
> use proxmox_resource_scheduling::resource::ResourceStats;
> - use proxmox_resource_scheduling::scheduler::ScoredMigration;
> - use proxmox_resource_scheduling::usage::Usage;
> + use proxmox_resource_scheduling::scheduler::{
> + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
> + };
> + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
>
> + use crate::bindings::resource_scheduling::options::{
> + SchedulerOptions, with_unweighted_imbalance,
> + };
> use crate::bindings::resource_scheduling::resource::{
> CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
> };
> @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic {
> /// A scheduler instance contains the cluster usage.
> pub struct Scheduler {
> inner: Mutex<Usage>,
> + weights: Option<NodeStatsWeights>,
> + }
> +
> + impl Scheduler {
> + /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
> + /// this instance or `default_weights` if this instance has none.
> + fn to_weighted_scheduler<F: UsageAggregator>(
> + &self,
> + usage: &Usage,
> + default_weights: NodeStatsWeights,
> + ) -> CoreScheduler {
> + usage
> + .to_scheduler::<F>()
> + .with_weights(self.weights.unwrap_or(default_weights))
> + }
Hm, not a big one, but this could also just live in
proxmox_resource_scheduling itself, where Usage implements a
pub fn to_weighted_scheduler<F: UsageAggregator>(&self, weights: NodeStatsWeights) -> Scheduler {
let node_usages = F::aggregate(self);
Scheduler::from_nodes(node_usages).with_weights(weights)
}
and then the unwrap_or() could be made explicitly where it's called to
it's clear that it's a default fallback in there.
No hard feelings though.
> }
>
> #[derive(Clone, Copy, Debug, Serialize, Deserialize)]
> @@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> type DynamicResource = PveResource<DynamicResourceStats>;
>
> - /// Class method: Create a new [`Scheduler`] instance.
> + /// Class method: Create a new [`Scheduler`] instance with the optional `options`.
> ///
> - /// See [`proxmox_resource_scheduling::usage::Usage::new`].
> + /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`].
> #[export(raw_return)]
> - pub fn new(#[raw] class: Value) -> Result<Value, Error> {
> + pub fn new(#[raw] class: Value, options: Option<SchedulerOptions>) -> Result<Value, Error> {
> let inner = Usage::new();
> + let weights = options.unwrap_or_default().weights();
>
> Ok(perlmod::instantiate_magic!(
> - &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) })
> + &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights })
> ))
> }
>
> @@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
> pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
> let usage = this.inner.lock().unwrap();
>
> - usage.to_scheduler::<IdentityAggregator>().node_imbalance()
> + this.to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
> + .node_imbalance()
> }
>
> /// Method: Scores the given migration `candidates` by the best node imbalance improvement with
> @@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>
> - Ok(usage
> - .to_scheduler::<IdentityAggregator>()
> + Ok(this
> + .to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
> .score_best_balancing_migration_candidates(candidates, limit))
> }
>
> @@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>
> - usage
> - .to_scheduler::<IdentityAggregator>()
> - .score_best_balancing_migration_candidates_topsis(&candidates, limit)
> + let scored_migrations = this
> + .to_weighted_scheduler::<IdentityAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
> + .score_best_balancing_migration_candidates_topsis(&candidates, limit)?;
> +
> + if this.weights.is_some() {
> + return Ok(scored_migrations);
> + }
> +
> + // callers not passing any weights would compare the 1:5 weighted imbalance reported for the
> + // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted
> + // imbalance
> + //
> + // to avoid this mismatch, report the unweighted imbalance instead
good comment!
> + Ok(with_unweighted_imbalance::<IdentityAggregator>(
> + &usage,
> + &candidates,
> + scored_migrations,
> + ))
> }
>
> /// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
> @@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic {
> ) -> Result<Vec<(String, f64)>, Error> {
> let usage = this.inner.lock().unwrap();
>
> - usage
> - .to_scheduler::<StartingAsStartedResourceAggregator>()
> - .score_nodes_to_start_resource(resource_stats)
> + this.to_weighted_scheduler::<StartingAsStartedResourceAggregator>(
> + &usage,
> + LEGACY_TOPSIS_WEIGHTS,
> + )
> + .score_nodes_to_start_resource(resource_stats)
> }
> }
next prev parent reply other threads:[~2026-11-19 12:45 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:50 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41 ` Daniel Kral
2026-11-19 12:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40 ` Daniel Kral
2026-11-19 13:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45 ` Daniel Kral [this message]
2026-11-19 13:28 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMZ743TVBU4A.1S9VS6988B1F6@proxmox.com \
--to=d.kral@proxmox.com \
--cc=d.rusovac@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox