From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id D63601FF0B6 for ; Thu, 12 Nov 2026 10:47:07 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3FCA021703; Thu, 12 Nov 2026 10:46:21 +0100 (CET) From: Dominik Rusovac To: pve-devel@lists.proxmox.com Subject: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Date: Thu, 12 Nov 2026 10:44:27 +0100 Message-ID: <20261112094437.1951271-10-d.rusovac@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261112094437.1951271-1-d.rusovac@proxmox.com> References: <20261112094437.1951271-1-d.rusovac@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794476685832 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.032 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: HSB3GZFA62FBW3DTGIBUOT5JOOAOXEQA X-Message-ID-Hash: HSB3GZFA62FBW3DTGIBUOT5JOOAOXEQA X-MailFrom: d.rusovac@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: allow passing options to the constructor of the static and dynamic scheduler, which specify the weights of the CPU load and memory usage applied to every load balancing computation. these options are a single optional parameter with only optional fields, so the signatures of the existing methods stay the same with specified weights, every rebalancing method uses them for the load imbalance and the TOPSIS criteria alike, so that for example 1:1 means 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS criteria weights without specified weights, every method keeps its previous behavior: the load imbalance and the bruteforce method use equal weights (1:1); the TOPSIS methods use weights of 1:5 for their criteria NOTE: weights are not validated. if both are 0, there is simply no load imbalance and the TOPSIS methods fail, so callers must not pass such weights NOTE: to decide whether a migration improves the balance enough, the HA manager compares the load imbalance reported for the migrations ranked by TOPSIS with the current load imbalance. without specified weights, the TOPSIS ranking would report an imbalance weighted 1:5, while the load imbalance is unweighted. so, without specified weights, recompute the reported imbalance of the ranked migrations with equal weights, as before. while this requires reiterating all migrations candidates, it is required to keep older versions of the HA manager working as before by preventing them from running into this undesirable load imbalance mismatch add test cases checking that: - without options or with unspecified options, the load imbalance and the node scores stay the same as before - equal weights keep the unweighted load imbalance, regardless of their magnitude - weights of 1:5 keep the previous node scores, but change the load imbalance, as it's also being weighted 1:5 - weighting only the CPU load or only the memory usage results in a different load imbalance - the TOPSIS ranking reports the same load imbalance as the bruteforce method, both without specified options and with weights of 1:5, as the HA manager compares it with the current load imbalance - without specified options, the TOPSIS ranking is the same as with weights of 1:5 - the dynamic scheduler accepts options as well Signed-off-by: Dominik Rusovac --- Notes: changes since v1: * reword commit * pass the weights once as optional scheduler options to the constructor instead of adding weight parameters to every method, as suggested in the review of v1, so the signatures of the existing methods stay the same and older HA manager versions can still call them * options are a single optional hash with only optional fields, a missing weight defaults to 1 * without options, every method keeps its previous behavior, including the TOPSIS ranking with weights of 1:5 * without options, recompute the imbalance reported with the TOPSIS ranking with equal weights, so that it stays comparable to the node imbalance for older HA manager version * weights also apply to score_nodes_to_start_service() * drop the adjustment of the existing test, as calling the methods without weights keeps working * add test cases for the scheduler options requires the proxmox-resource-scheduling patches of this series .../src/bindings/resource_scheduling/mod.rs | 1 + .../bindings/resource_scheduling/options.rs | 59 ++++++++ .../resource_scheduling/pve_dynamic.rs | 71 +++++++-- .../resource_scheduling/pve_static.rs | 64 ++++++-- pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++ 5 files changed, 304 insertions(+), 32 deletions(-) create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/bindings/resource_scheduling/mod.rs index 87b4a03..dc94f21 100644 --- a/pve-rs/src/bindings/resource_scheduling/mod.rs +++ b/pve-rs/src/bindings/resource_scheduling/mod.rs @@ -1,5 +1,6 @@ //! Resource scheduling related bindings. +mod options; mod resource; mod usage; diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/src/bindings/resource_scheduling/options.rs new file mode 100644 index 0000000..6759cc3 --- /dev/null +++ b/pve-rs/src/bindings/resource_scheduling/options.rs @@ -0,0 +1,59 @@ +//! Options for the resource scheduling bindings. + +use serde::Deserialize; + +use proxmox_resource_scheduling::node::NodeStatsWeights; +use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredMigration}; +use proxmox_resource_scheduling::usage::{Usage, UsageAggregator}; + +/// Options for a scheduler instance. +#[derive(Clone, Copy, Debug, Default, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub struct SchedulerOptions { + /// The weight of the CPU load of nodes. Defaults to 1 if only the other weight is specified. + pub cpu_weight: Option, + /// The weight of the memory usage of nodes. Defaults to 1 if only the other weight is + /// specified. + pub mem_weight: Option, +} + +impl SchedulerOptions { + /// Returns the weights or [`None`] if no weight is set. + pub fn weights(&self) -> Option { + if self.cpu_weight.is_none() && self.mem_weight.is_none() { + return None; + } + + let mut weights = NodeStatsWeights::default(); + // unspecified weight defaults to 1 + weights.cpu_weight = self.cpu_weight.unwrap_or(1) as f64; + weights.mem_weight = self.mem_weight.unwrap_or(1) as f64; + + Some(weights) + } +} + +/// Replaces the imbalance of each of the `scored_migrations` with the unweighted imbalance of the +/// respective migration in `candidates`, for the cluster usage `usage` aggregated by `F`. +pub fn with_unweighted_imbalance( + usage: &Usage, + candidates: &[MigrationCandidate], + scored_migrations: Vec, +) -> Vec { + let scheduler = usage.to_scheduler::(); + + scored_migrations + .into_iter() + // every scored migration should belong to some candidate + .flat_map(|scored_migration| { + candidates + .iter() + .find(|candidate| candidate.migration == scored_migration.migration) + .map(|candidate| { + let imbalance = scheduler.node_imbalance_with_migration_candidate(candidate); + + ScoredMigration::new(scored_migration.migration, imbalance) + }) + }) + .collect() +} diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs index 8e066e0..62d6426 100644 --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic { use serde::{Deserialize, Serialize}; use perlmod::Value; - use proxmox_resource_scheduling::node::NodeStats; + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights}; use proxmox_resource_scheduling::resource::ResourceStats; - use proxmox_resource_scheduling::scheduler::ScoredMigration; - use proxmox_resource_scheduling::usage::Usage; + use proxmox_resource_scheduling::scheduler::{ + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration, + }; + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator}; + use crate::bindings::resource_scheduling::options::{ + SchedulerOptions, with_unweighted_imbalance, + }; use crate::bindings::resource_scheduling::resource::{ CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates, }; @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic { /// A scheduler instance contains the cluster usage. pub struct Scheduler { inner: Mutex, + weights: Option, + } + + impl Scheduler { + /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of + /// this instance or `default_weights` if this instance has none. + fn to_weighted_scheduler( + &self, + usage: &Usage, + default_weights: NodeStatsWeights, + ) -> CoreScheduler { + usage + .to_scheduler::() + .with_weights(self.weights.unwrap_or(default_weights)) + } } #[derive(Clone, Copy, Debug, Serialize, Deserialize)] @@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic { type DynamicResource = PveResource; - /// Class method: Create a new [`Scheduler`] instance. + /// Class method: Create a new [`Scheduler`] instance with the optional `options`. /// - /// See [`proxmox_resource_scheduling::usage::Usage::new`]. + /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`]. #[export(raw_return)] - pub fn new(#[raw] class: Value) -> Result { + pub fn new(#[raw] class: Value, options: Option) -> Result { let inner = Usage::new(); + let weights = options.unwrap_or_default().weights(); Ok(perlmod::instantiate_magic!( - &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) }) + &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights }) )) } @@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic { pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 { let usage = this.inner.lock().unwrap(); - usage.to_scheduler::().node_imbalance() + this.to_weighted_scheduler::(&usage, NodeStatsWeights::EQUAL) + .node_imbalance() } /// Method: Scores the given migration `candidates` by the best node imbalance improvement with @@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic { let candidates = decompose_compact_migration_candidates(&usage, candidates)?; - Ok(usage - .to_scheduler::() + Ok(this + .to_weighted_scheduler::(&usage, NodeStatsWeights::EQUAL) .score_best_balancing_migration_candidates(candidates, limit)) } @@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic { let candidates = decompose_compact_migration_candidates(&usage, candidates)?; - usage - .to_scheduler::() - .score_best_balancing_migration_candidates_topsis(&candidates, limit) + let scored_migrations = this + .to_weighted_scheduler::(&usage, LEGACY_TOPSIS_WEIGHTS) + .score_best_balancing_migration_candidates_topsis(&candidates, limit)?; + + if this.weights.is_some() { + return Ok(scored_migrations); + } + + // callers not passing any weights would compare the 1:5 weighted imbalance reported for the + // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted + // imbalance + // + // to avoid this mismatch, report the unweighted imbalance instead + Ok(with_unweighted_imbalance::( + &usage, + &candidates, + scored_migrations, + )) } /// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on. @@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic { ) -> Result, Error> { let usage = this.inner.lock().unwrap(); - usage - .to_scheduler::() - .score_nodes_to_start_resource(resource_stats) + this.to_weighted_scheduler::( + &usage, + LEGACY_TOPSIS_WEIGHTS, + ) + .score_nodes_to_start_resource(resource_stats) } } diff --git a/pve-rs/src/bindings/resource_scheduling/pve_static.rs b/pve-rs/src/bindings/resource_scheduling/pve_static.rs index d83aa38..2399f71 100644 --- a/pve-rs/src/bindings/resource_scheduling/pve_static.rs +++ b/pve-rs/src/bindings/resource_scheduling/pve_static.rs @@ -12,12 +12,15 @@ pub mod pve_rs_resource_scheduling_static { use serde::{Deserialize, Serialize}; use perlmod::Value; - use proxmox_resource_scheduling::node::NodeStats; + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights}; use proxmox_resource_scheduling::resource::ResourceStats; - use proxmox_resource_scheduling::scheduler::ScoredMigration; - use proxmox_resource_scheduling::usage::Usage; + use proxmox_resource_scheduling::scheduler::{ + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration, + }; + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator}; use crate::bindings::resource_scheduling::{ + options::{SchedulerOptions, with_unweighted_imbalance}, resource::{ CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates, }, @@ -29,6 +32,21 @@ pub mod pve_rs_resource_scheduling_static { /// A scheduler instance contains the cluster usage. pub struct Scheduler { inner: Mutex, + weights: Option, + } + + impl Scheduler { + /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of + /// this instance or `default_weights` if this instance has none. + fn to_weighted_scheduler( + &self, + usage: &Usage, + default_weights: NodeStatsWeights, + ) -> CoreScheduler { + usage + .to_scheduler::() + .with_weights(self.weights.unwrap_or(default_weights)) + } } #[derive(Clone, Copy, Debug, Serialize, Deserialize)] @@ -54,15 +72,16 @@ pub mod pve_rs_resource_scheduling_static { type StaticResource = PveResource; - /// Class method: Create a new [`Scheduler`] instance. + /// Class method: Create a new [`Scheduler`] instance with the optional `options`. /// - /// See [`proxmox_resource_scheduling::usage::Usage::new`]. + /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`]. #[export(raw_return)] - pub fn new(#[raw] class: Value) -> Result { + pub fn new(#[raw] class: Value, options: Option) -> Result { let inner = Usage::new(); + let weights = options.unwrap_or_default().weights(); Ok(perlmod::instantiate_magic!( - &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) }) + &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights }) )) } @@ -165,8 +184,7 @@ pub mod pve_rs_resource_scheduling_static { pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 { let usage = this.inner.lock().unwrap(); - usage - .to_scheduler::() + this.to_weighted_scheduler::(&usage, NodeStatsWeights::EQUAL) .node_imbalance() } @@ -184,8 +202,8 @@ pub mod pve_rs_resource_scheduling_static { let candidates = decompose_compact_migration_candidates(&usage, candidates)?; - Ok(usage - .to_scheduler::() + Ok(this + .to_weighted_scheduler::(&usage, NodeStatsWeights::EQUAL) .score_best_balancing_migration_candidates(candidates, limit)) } @@ -203,9 +221,24 @@ pub mod pve_rs_resource_scheduling_static { let candidates = decompose_compact_migration_candidates(&usage, candidates)?; - usage - .to_scheduler::() - .score_best_balancing_migration_candidates_topsis(&candidates, limit) + let scored_migrations = this + .to_weighted_scheduler::(&usage, LEGACY_TOPSIS_WEIGHTS) + .score_best_balancing_migration_candidates_topsis(&candidates, limit)?; + + if this.weights.is_some() { + return Ok(scored_migrations); + } + + // callers not passing any weights would compare the 1:5 weighted imbalance reported for the + // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted + // imbalance + // + // to avoid this mismatch, report the unweighted imbalance instead + Ok(with_unweighted_imbalance::( + &usage, + &candidates, + scored_migrations, + )) } /// Method: Scores nodes to start a service with the usage statistics `service_stats` on. @@ -218,8 +251,7 @@ pub mod pve_rs_resource_scheduling_static { ) -> Result, Error> { let usage = this.inner.lock().unwrap(); - usage - .to_scheduler::() + this.to_weighted_scheduler::(&usage, LEGACY_TOPSIS_WEIGHTS) .score_nodes_to_start_resource(service_stats) } } diff --git a/pve-rs/test/resource_scheduling.pl b/pve-rs/test/resource_scheduling.pl index 3775242..f8ed89a 100755 --- a/pve-rs/test/resource_scheduling.pl +++ b/pve-rs/test/resource_scheduling.pl @@ -179,11 +179,152 @@ sub test_balance_small_memory_difference { } } +sub test_scheduler_options { + my $new_static = sub { + my $static = PVE::RS::ResourceScheduling::Static->new(@_); + $static->add_node("A", 8, 10_000_000_000); + $static->add_node("B", 4, 9_000_000_000); + $static->add_node("C", 4, 8_000_000_000); + $static->add_service_usage_to_node("A", "vm:100", { maxcpu => 4, maxmem => 1_000_000_000 }); + $static->add_service_usage_to_node("B", "vm:101", { maxcpu => 1, maxmem => 6_000_000_000 }); + $static->add_service_usage_to_node("C", "vm:102", { maxcpu => 3, maxmem => 1_000_000_000 }); + return $static; + }; + + my $service = { maxcpu => 2, maxmem => 2_000_000_000 }; + + my sub scores { + my ($static) = @_; + return { map { $_->[0] => $_->[1] } $static->score_nodes_to_start_service($service)->@* }; + } + + my $default = $new_static->(); + + # without weights, the previous behavior is kept + for my $entry ([undef, 'undef'], [{}, 'empty']) { + my ($options, $desc) = $entry->@*; + my $static = $new_static->($options); + + is( + $static->calculate_node_imbalance(), + $default->calculate_node_imbalance(), + "imbalance with $desc options equals default", + ); + is_deeply(scores($static), scores($default), "scores with $desc options equal default"); + } + + # equal weights keep the previous unweighted imbalance + for my $options ( + { 'cpu-weight' => 1 }, + { 'cpu-weight' => 1, 'mem-weight' => 1 }, + { 'cpu-weight' => 3, 'mem-weight' => 3 }, + ) { + my $desc = join(', ', map { "$_ $options->{$_}" } sort keys $options->%*); + + is( + $new_static->($options)->calculate_node_imbalance(), + $default->calculate_node_imbalance(), + "imbalance with $desc equals default", + ); + } + + # weights of 1:5 keep the previous TOPSIS criteria weights, but also weight the imbalance + my $legacy_ratio = $new_static->({ 'cpu-weight' => 1, 'mem-weight' => 5 }); + + is_deeply(scores($legacy_ratio), scores($default), 'scores with weights 1:5 equal default'); + isnt( + $legacy_ratio->calculate_node_imbalance(), + $default->calculate_node_imbalance(), + 'weights 1:5 change imbalance', + ); + + my $cpu_only = $new_static->({ 'cpu-weight' => 1, 'mem-weight' => 0 }); + my $mem_only = $new_static->({ 'cpu-weight' => 0, 'mem-weight' => 1 }); + + isnt( + $cpu_only->calculate_node_imbalance(), + $mem_only->calculate_node_imbalance(), + 'weights change imbalance', + ); + + my $dynamic = eval { PVE::RS::ResourceScheduling::Dynamic->new({ 'cpu-weight' => 3 }) }; + ok(defined($dynamic) && !$@, 'dynamic scheduler accepts options'); +} + +sub test_topsis_imbalance_with_scheduler_options { + my $new_static = sub { + my $static = PVE::RS::ResourceScheduling::Static->new(@_); + $static->add_node("A", 8, 10_000_000_000); + $static->add_node("B", 4, 9_000_000_000); + $static->add_node("C", 4, 8_000_000_000); + + my $services = { + "vm:100" => ["A", 4, 1_000_000_000], + "vm:101" => ["A", 2, 4_000_000_000], + "vm:102" => ["B", 1, 6_000_000_000], + "vm:103" => ["C", 3, 1_000_000_000], + }; + for my $sid (sort keys $services->%*) { + my ($node, $maxcpu, $maxmem) = $services->{$sid}->@*; + my $stats = { maxcpu => $maxcpu, maxmem => $maxmem }; + $static->add_service($sid, { stats => $stats, running => 1, 'current-node' => $node }); + } + + return $static; + }; + + # 2^2 candidates + my $candidates = [ + map { { leader => $_, resources => [$_], nodes => ["B", "C"] } } ("vm:100", "vm:101") + ]; + + my sub imbalances { + my ($migrations) = @_; + + return { + map { + my $m = $_->{migration}; + ("$m->{sid} $m->{'source-node'} $m->{'target-node'}" => $_->{imbalance}) + } $migrations->@* + }; + } + + my $rankings = {}; + + for my $entry ([undef, 'without options'], [{ 'cpu-weight' => 1, 'mem-weight' => 5 }, '1:5']) { + my ($options, $desc) = $entry->@*; + my $static = $new_static->($options); + + my $topsis = $static->score_best_balancing_migration_candidates_topsis($candidates, 4); + my $bruteforce = $static->score_best_balancing_migration_candidates($candidates, 4); + + # the TOPSIS ranking must report the same imbalance as the current imbalance computation + is_deeply( + imbalances($topsis), + imbalances($bruteforce), + "TOPSIS ranking reports the same imbalance as bruteforce $desc", + ); + + # only keep the ranking, as the reported imbalance differs between both cases + $rankings->{$desc} = + [map { "$_->{migration}->{sid} $_->{migration}->{'target-node'}" } $topsis->@*]; + } + + # without options, the TOPSIS ranking uses weights of 1:5 + is_deeply( + $rankings->{'without options'}, + $rankings->{'1:5'}, + 'TOPSIS ranking without options equals ranking with weights 1:5', + ); +} + test_basic(); test_balance(); test_balance_removal(); test_overcommitted(); test_balance_small_memory_difference(1); test_balance_small_memory_difference(0); +test_scheduler_options(); +test_topsis_imbalance_with_scheduler_options(); done_testing(); -- 2.47.3