From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 022D81FF0DE for ; Thu, 19 Nov 2026 13:45:10 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 51F29216CD; Thu, 19 Nov 2026 13:45:09 +0100 (CET) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Nov 2026 13:45:03 +0100 Message-Id: Subject: Re: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options From: "Daniel Kral" To: "Dominik Rusovac" , X-Mailer: aerc 0.22.0-45-gc30c872fde7b-dirty References: <20261112094437.1951271-1-d.rusovac@proxmox.com> <20261112094437.1951271-10-d.rusovac@proxmox.com> In-Reply-To: <20261112094437.1951271-10-d.rusovac@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795092303743 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.434 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TYBIABSRF2RPDJO2NYGNVK4CZ7QRDEKO X-Message-ID-Hash: TYBIABSRF2RPDJO2NYGNVK4CZ7QRDEKO X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote: > allow passing options to the constructor of the static and dynamic > scheduler, which specify the weights of the CPU load and memory usage > applied to every load balancing computation. these options are a single > optional parameter with only optional fields, so the signatures of the > existing methods stay the same > > with specified weights, every rebalancing method uses them for the load > imbalance and the TOPSIS criteria alike, so that for example 1:1 means > 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS > criteria weights > > without specified weights, every method keeps its previous behavior: the > load imbalance and the bruteforce method use equal weights (1:1); the > TOPSIS methods use weights of 1:5 for their criteria > > NOTE: weights are not validated. if both are 0, there is simply no load > imbalance and the TOPSIS methods fail, so callers must not pass such > weights > > NOTE: to decide whether a migration improves the balance enough, the HA > manager compares the load imbalance reported for the migrations ranked > by TOPSIS with the current load imbalance. without specified weights, > the TOPSIS ranking would report an imbalance weighted 1:5, while the > load imbalance is unweighted. so, without specified weights, recompute > the reported imbalance of the ranked migrations with equal weights, as > before. while this requires reiterating all migrations candidates, it is > required to keep older versions of the HA manager working as before by > preventing them from running into this undesirable load imbalance > mismatch > > add test cases checking that: > - without options or with unspecified options, the load imbalance and > the node scores stay the same as before > - equal weights keep the unweighted load imbalance, regardless of their > magnitude > - weights of 1:5 keep the previous node scores, but change the load > imbalance, as it's also being weighted 1:5 > - weighting only the CPU load or only the memory usage results in a > different load imbalance > - the TOPSIS ranking reports the same load imbalance as the bruteforce > method, both without specified options and with weights of 1:5, as the > HA manager compares it with the current load imbalance > - without specified options, the TOPSIS ranking is the same as with > weights of 1:5 > - the dynamic scheduler accepts options as well > > Signed-off-by: Dominik Rusovac > --- > > Notes: > changes since v1: > * reword commit > * pass the weights once as optional scheduler options to the construc= tor > instead of adding weight parameters to every method, as suggested i= n > the review of v1, so the signatures of the existing methods stay th= e > same and older HA manager versions can still call them > * options are a single optional hash with only optional fields, a > missing weight defaults to 1 > * without options, every method keeps its previous behavior, includin= g > the TOPSIS ranking with weights of 1:5 > * without options, recompute the imbalance reported with the TOPSIS > ranking with equal weights, so that it stays comparable to the node > imbalance for older HA manager version > * weights also apply to score_nodes_to_start_service() > * drop the adjustment of the existing test, as calling the methods > without weights keeps working > * add test cases for the scheduler options > =20 > requires the proxmox-resource-scheduling patches of this series > > .../src/bindings/resource_scheduling/mod.rs | 1 + > .../bindings/resource_scheduling/options.rs | 59 ++++++++ > .../resource_scheduling/pve_dynamic.rs | 71 +++++++-- > .../resource_scheduling/pve_static.rs | 64 ++++++-- > pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++ > 5 files changed, 304 insertions(+), 32 deletions(-) > create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs > > diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/= bindings/resource_scheduling/mod.rs > index 87b4a03..dc94f21 100644 > --- a/pve-rs/src/bindings/resource_scheduling/mod.rs > +++ b/pve-rs/src/bindings/resource_scheduling/mod.rs > @@ -1,5 +1,6 @@ > //! Resource scheduling related bindings. > =20 > +mod options; > mod resource; > mod usage; > =20 > diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/= src/bindings/resource_scheduling/options.rs > new file mode 100644 > index 0000000..6759cc3 > --- /dev/null > +++ b/pve-rs/src/bindings/resource_scheduling/options.rs > @@ -0,0 +1,59 @@ > +//! Options for the resource scheduling bindings. > + > +use serde::Deserialize; > + > +use proxmox_resource_scheduling::node::NodeStatsWeights; > +use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredM= igration}; > +use proxmox_resource_scheduling::usage::{Usage, UsageAggregator}; > + > +/// Options for a scheduler instance. > +#[derive(Clone, Copy, Debug, Default, Deserialize)] > +#[serde(rename_all =3D "kebab-case")] > +pub struct SchedulerOptions { > + /// The weight of the CPU load of nodes. Defaults to 1 if only the o= ther weight is specified. > + pub cpu_weight: Option, > + /// The weight of the memory usage of nodes. Defaults to 1 if only t= he other weight is > + /// specified. > + pub mem_weight: Option, > +} > + > +impl SchedulerOptions { > + /// Returns the weights or [`None`] if no weight is set. > + pub fn weights(&self) -> Option { > + if self.cpu_weight.is_none() && self.mem_weight.is_none() { > + return None; > + } > + > + let mut weights =3D NodeStatsWeights::default(); > + // unspecified weight defaults to 1 > + weights.cpu_weight =3D self.cpu_weight.unwrap_or(1) as f64; > + weights.mem_weight =3D self.mem_weight.unwrap_or(1) as f64; > + > + Some(weights) > + } > +} > + > +/// Replaces the imbalance of each of the `scored_migrations` with the u= nweighted imbalance of the > +/// respective migration in `candidates`, for the cluster usage `usage` = aggregated by `F`. > +pub fn with_unweighted_imbalance( > + usage: &Usage, > + candidates: &[MigrationCandidate], > + scored_migrations: Vec, > +) -> Vec { > + let scheduler =3D usage.to_scheduler::(); > + > + scored_migrations > + .into_iter() > + // every scored migration should belong to some candidate > + .flat_map(|scored_migration| { > + candidates > + .iter() > + .find(|candidate| candidate.migration =3D=3D scored_migr= ation.migration) > + .map(|candidate| { > + let imbalance =3D scheduler.node_imbalance_with_migr= ation_candidate(candidate); > + > + ScoredMigration::new(scored_migration.migration, imb= alance) > + }) > + }) > + .collect() > +} > diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve= -rs/src/bindings/resource_scheduling/pve_dynamic.rs > index 8e066e0..62d6426 100644 > --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs > +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs > @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic { > use serde::{Deserialize, Serialize}; > =20 > use perlmod::Value; > - use proxmox_resource_scheduling::node::NodeStats; > + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights}= ; > use proxmox_resource_scheduling::resource::ResourceStats; > - use proxmox_resource_scheduling::scheduler::ScoredMigration; > - use proxmox_resource_scheduling::usage::Usage; > + use proxmox_resource_scheduling::scheduler::{ > + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigrati= on, > + }; > + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator}; > =20 > + use crate::bindings::resource_scheduling::options::{ > + SchedulerOptions, with_unweighted_imbalance, > + }; > use crate::bindings::resource_scheduling::resource::{ > CompactMigrationCandidate, PveResource, decompose_compact_migrat= ion_candidates, > }; > @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic { > /// A scheduler instance contains the cluster usage. > pub struct Scheduler { > inner: Mutex, > + weights: Option, > + } > + > + impl Scheduler { > + /// Returns a scheduler for the current `usage`, aggregated by `= F`, with the weights of > + /// this instance or `default_weights` if this instance has none= . > + fn to_weighted_scheduler( > + &self, > + usage: &Usage, > + default_weights: NodeStatsWeights, > + ) -> CoreScheduler { > + usage > + .to_scheduler::() > + .with_weights(self.weights.unwrap_or(default_weights)) > + } Hm, not a big one, but this could also just live in proxmox_resource_scheduling itself, where Usage implements a pub fn to_weighted_scheduler(&self, weights: NodeSt= atsWeights) -> Scheduler { let node_usages =3D F::aggregate(self); Scheduler::from_nodes(node_usages).with_weights(weights) } and then the unwrap_or() could be made explicitly where it's called to it's clear that it's a default fallback in there. No hard feelings though. > } > =20 > #[derive(Clone, Copy, Debug, Serialize, Deserialize)] > @@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic { > =20 > type DynamicResource =3D PveResource; > =20 > - /// Class method: Create a new [`Scheduler`] instance. > + /// Class method: Create a new [`Scheduler`] instance with the optio= nal `options`. > /// > - /// See [`proxmox_resource_scheduling::usage::Usage::new`]. > + /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`Sch= edulerOptions`]. > #[export(raw_return)] > - pub fn new(#[raw] class: Value) -> Result { > + pub fn new(#[raw] class: Value, options: Option) -= > Result { > let inner =3D Usage::new(); > + let weights =3D options.unwrap_or_default().weights(); > =20 > Ok(perlmod::instantiate_magic!( > - &class, MAGIC =3D> Box::new(Scheduler { inner: Mutex::new(in= ner) }) > + &class, MAGIC =3D> Box::new(Scheduler { inner: Mutex::new(in= ner), weights }) > )) > } > =20 > @@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic { > pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) ->= f64 { > let usage =3D this.inner.lock().unwrap(); > =20 > - usage.to_scheduler::().node_imbalance() > + this.to_weighted_scheduler::(&usage, NodeSta= tsWeights::EQUAL) > + .node_imbalance() > } > =20 > /// Method: Scores the given migration `candidates` by the best node= imbalance improvement with > @@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic { > =20 > let candidates =3D decompose_compact_migration_candidates(&usage= , candidates)?; > =20 > - Ok(usage > - .to_scheduler::() > + Ok(this > + .to_weighted_scheduler::(&usage, NodeSta= tsWeights::EQUAL) > .score_best_balancing_migration_candidates(candidates, limit= )) > } > =20 > @@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic { > =20 > let candidates =3D decompose_compact_migration_candidates(&usage= , candidates)?; > =20 > - usage > - .to_scheduler::() > - .score_best_balancing_migration_candidates_topsis(&candidate= s, limit) > + let scored_migrations =3D this > + .to_weighted_scheduler::(&usage, LEGACY_= TOPSIS_WEIGHTS) > + .score_best_balancing_migration_candidates_topsis(&candidate= s, limit)?; > + > + if this.weights.is_some() { > + return Ok(scored_migrations); > + } > + > + // callers not passing any weights would compare the 1:5 weighte= d imbalance reported for the > + // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS= method to the unweighted > + // imbalance > + // > + // to avoid this mismatch, report the unweighted imbalance inste= ad good comment! > + Ok(with_unweighted_imbalance::( > + &usage, > + &candidates, > + scored_migrations, > + )) > } > =20 > /// Method: Scores nodes to start a resource with the usage statisti= cs `resource_stats` on. > @@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic { > ) -> Result, Error> { > let usage =3D this.inner.lock().unwrap(); > =20 > - usage > - .to_scheduler::() > - .score_nodes_to_start_resource(resource_stats) > + this.to_weighted_scheduler::( > + &usage, > + LEGACY_TOPSIS_WEIGHTS, > + ) > + .score_nodes_to_start_resource(resource_stats) > } > }