* [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
` (13 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 2 +-
proxmox-resource-scheduling/src/topsis.rs | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index eb5c6142..9078fa72 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -166,7 +166,7 @@ impl Scheduler {
}
}
- /// Map the current node usages to a [`PveTopsisAlternative`].
+ /// Maps the current node usages to a [`PveTopsisAlternative`].
///
/// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
/// square (RMS) and maximum value of each stat in the node usages.
diff --git a/proxmox-resource-scheduling/src/topsis.rs b/proxmox-resource-scheduling/src/topsis.rs
index 3d42ff30..89f4ea40 100644
--- a/proxmox-resource-scheduling/src/topsis.rs
+++ b/proxmox-resource-scheduling/src/topsis.rs
@@ -161,7 +161,7 @@ impl<const N: usize> IdealAlternatives<N> {
}
/// Scores the alternatives in `matrix` according to their similarity to the ideal worst
-/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closness to the ideal worst
+/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closeness to the ideal worst
/// and/or distance to the ideal best alternatives.
pub fn score_alternatives<const N: usize>(
matrix: &Matrix<N>,
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 12:49 ` Daniel Kral
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
` (12 subsequent siblings)
14 siblings, 1 reply; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
DRY and hide repeatedly used closure inside reuasble function.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 41 ++++++++------------
1 file changed, 16 insertions(+), 25 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 9078fa72..a12cd519 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -56,6 +56,19 @@ fn calculate_node_imbalance(nodes: &[NodeUsage], to_load: impl Fn(&NodeUsage) ->
cv / max_cv
}
+/// Returns the hypothetical [`NodeStats`] of `node` with respect to the migration `candidate`.
+fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage) -> NodeStats {
+ let mut new_stats = node.stats;
+
+ if node.name == candidate.migration.source_node {
+ new_stats.remove_running_resource(&candidate.stats);
+ } else if node.name == candidate.migration.target_node {
+ new_stats.add_running_resource(&candidate.stats);
+ }
+
+ new_stats
+}
+
criteria_struct! {
/// A given alternative.
struct PveTopsisAlternative {
@@ -216,15 +229,7 @@ impl Scheduler {
/// See [`calculate_node_imbalance`] for more information.
fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- let mut new_stats = node.stats;
-
- if node.name == candidate.migration.source_node {
- new_stats.remove_running_resource(&candidate.stats);
- } else if node.name == candidate.migration.target_node {
- new_stats.add_running_resource(&candidate.stats);
- }
-
- new_stats.load()
+ node_stats_after_migration(candidate, node).load()
})
}
@@ -288,22 +293,8 @@ impl Scheduler {
let matrix = candidates
.iter()
.map(|candidate| {
- let resource_stats = &candidate.stats;
- let source_node = &candidate.migration.source_node;
- let target_node = &candidate.migration.target_node;
-
- self.topsis_alternative_with(|node| {
- let mut new_stats = node.stats;
-
- if &node.name == source_node {
- new_stats.remove_running_resource(resource_stats);
- } else if &node.name == target_node {
- new_stats.add_running_resource(resource_stats);
- }
-
- new_stats
- })
- .into()
+ self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
+ .into()
})
.collect::<Vec<_>>();
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* Re: [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-10-21 12:49 ` Daniel Kral
0 siblings, 0 replies; 17+ messages in thread
From: Daniel Kral @ 2026-10-21 12:49 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Nice!
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> DRY and hide repeatedly used closure inside reuasble function.
nit: s/reuasble/reusable/
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> proxmox-resource-scheduling/src/scheduler.rs | 41 ++++++++------------
> 1 file changed, 16 insertions(+), 25 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index 9078fa72..a12cd519 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -56,6 +56,19 @@ fn calculate_node_imbalance(nodes: &[NodeUsage], to_load: impl Fn(&NodeUsage) ->
> cv / max_cv
> }
>
> +/// Returns the hypothetical [`NodeStats`] of `node` with respect to the migration `candidate`.
> +fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage) -> NodeStats {
> + let mut new_stats = node.stats;
> +
> + if node.name == candidate.migration.source_node {
> + new_stats.remove_running_resource(&candidate.stats);
> + } else if node.name == candidate.migration.target_node {
> + new_stats.add_running_resource(&candidate.stats);
> + }
> +
> + new_stats
> +}
> +
> criteria_struct! {
> /// A given alternative.
> struct PveTopsisAlternative {
> @@ -216,15 +229,7 @@ impl Scheduler {
> /// See [`calculate_node_imbalance`] for more information.
> fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - let mut new_stats = node.stats;
> -
> - if node.name == candidate.migration.source_node {
> - new_stats.remove_running_resource(&candidate.stats);
> - } else if node.name == candidate.migration.target_node {
> - new_stats.add_running_resource(&candidate.stats);
> - }
> -
> - new_stats.load()
> + node_stats_after_migration(candidate, node).load()
> })
nit: if it fits on a line, could drop the braces {} here
> }
>
> @@ -288,22 +293,8 @@ impl Scheduler {
> let matrix = candidates
> .iter()
> .map(|candidate| {
> - let resource_stats = &candidate.stats;
> - let source_node = &candidate.migration.source_node;
> - let target_node = &candidate.migration.target_node;
> -
> - self.topsis_alternative_with(|node| {
> - let mut new_stats = node.stats;
> -
> - if &node.name == source_node {
> - new_stats.remove_running_resource(resource_stats);
> - } else if &node.name == target_node {
> - new_stats.add_running_resource(resource_stats);
> - }
> -
> - new_stats
> - })
> - .into()
> + self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
> + .into()
> })
> .collect::<Vec<_>>();
>
Otherwise, consider as:
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate Dominik Rusovac
` (11 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Blend weights into node load computation, effectively replacing
arithmetic mean [0] of the node's CPU load and memory usage with
weighted arithmetic mean [1] thereof.
Expose customizable weights for bruteforce method. For TOPSIS method
expose customizable weights for average cpu and average memory only. The
weights of highest cpu and highest memory are not independently
configurable; each is derived as twice as much as its average
counterpart, that is, the weight of highest cpu/memory is twice as much
as the weight of average cpu/memory. This makes it possible to
prioritize balancing a specific computational resource while also
preserving the ratio between a resource's two criteria, as introduced in
[2].
Apart from the achieved imbalance, expose the score of a scored
migration. Moreover, primarily order scored migrations by their score
(lower is better), falling back to imbalance when scores tie, and
finally to the migration itself if both tie -- ensuring a total,
deterministic order even when several candidates achieve the exact same
score. Add basic unit test for this.
Refactor existing tests.
[0] https://en.wikipedia.org/wiki/Arithmetic_mean
[1] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
[2] https://lore.proxmox.com/all/20221110143800.98047-3-f.ebner@proxmox.com/
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/node.rs | 27 +-
proxmox-resource-scheduling/src/pve_static.rs | 3 +-
proxmox-resource-scheduling/src/scheduler.rs | 248 +++++++++++++-----
.../tests/scheduler.rs | 136 ++++++++--
4 files changed, 327 insertions(+), 87 deletions(-)
diff --git a/proxmox-resource-scheduling/src/node.rs b/proxmox-resource-scheduling/src/node.rs
index 3c7ed393..f9dc5b16 100644
--- a/proxmox-resource-scheduling/src/node.rs
+++ b/proxmox-resource-scheduling/src/node.rs
@@ -51,9 +51,30 @@ impl NodeStats {
self.mem as f64 / self.maxmem as f64
}
- /// Returns a combined node usage as a percentage.
- pub fn load(&self) -> f64 {
- (self.cpu_load() + self.mem_load()) / 2.0
+ /// Returns the node load as the [weighted arithmetic mean] of [`Self::cpu_load`] and
+ /// [`Self::mem_load`].
+ ///
+ /// `cpu_weight` controls how much [`Self::cpu_load`] contributes to the load, analogously,
+ /// `mem_weight` controls how much [`Self::mem_load`] contributes to the load; the higher the
+ /// weight, the more that resource dominates the result.
+ ///
+ /// NOTE:
+ /// - the weights have to be positive
+ /// - only the weights' ratio matters, not their absolute magnitude
+ /// - a weight of 0.0 means no contribution to the load
+ /// - [weighted arithmetic mean] is the same as [arithmetic mean] if the weights are equal
+ ///
+ /// [weighted arithmetic mean]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
+ /// [arithmetic mean]: https://en.wikipedia.org/wiki/Arithmetic_mean
+ pub fn load(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
+ let weights_sum = cpu_weight + mem_weight;
+
+ if weights_sum == 0.0 {
+ // both, cpu and mem, should not contribute to the load, so there is no load
+ return 0.0;
+ }
+
+ (self.cpu_load() * cpu_weight + self.mem_load() * mem_weight) / weights_sum
}
}
diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
index 40b76cbc..98d64907 100644
--- a/proxmox-resource-scheduling/src/pve_static.rs
+++ b/proxmox-resource-scheduling/src/pve_static.rs
@@ -96,5 +96,6 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
let scheduler = Scheduler::from_nodes(nodes);
- scheduler.score_nodes_to_start_resource(*service)
+ // uses original empirically determined TOPSIS weights
+ scheduler.score_nodes_to_start_resource(*service, 1, 5)
}
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index a12cd519..0690385d 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -8,6 +8,9 @@ use std::{
collections::BinaryHeap,
};
+/// Constant to control the ratio of the weights of a resource's average and highest criteria.
+pub const TOPSIS_AVERAGE_VS_HIGHEST_FACTOR: f64 = 2.0;
+
/// The scheduler view of a node.
#[derive(Clone, Debug)]
pub struct NodeUsage {
@@ -69,23 +72,6 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
new_stats
}
-criteria_struct! {
- /// A given alternative.
- struct PveTopsisAlternative {
- #[criterion("average CPU", -1.0)]
- average_cpu: f64,
- #[criterion("highest CPU", -2.0)]
- highest_cpu: f64,
- #[criterion("average memory", -5.0)]
- average_memory: f64,
- #[criterion("highest memory", -10.0)]
- highest_memory: f64,
- }
-
- const N_CRITERIA;
- static PVE_HA_TOPSIS_CRITERIA;
-}
-
#[derive(Clone, Debug)]
pub struct Scheduler {
nodes: Vec<NodeUsage>,
@@ -111,12 +97,20 @@ pub struct ScoredMigration {
pub migration: Migration,
/// The expected node imbalance after the migration.
pub imbalance: f64,
+ /// The score.
+ ///
+ /// - corresponds to imbalance of weighted loads for
+ /// [`Scheduler::score_best_balancing_migration_candidates`]
+ /// - corresponds to distance to ideal best alternatives for
+ /// [`Scheduler::score_best_balancing_migration_candidates_topsis`]
+ pub score: f64,
}
impl Ord for ScoredMigration {
fn cmp(&self, other: &Self) -> Ordering {
- self.imbalance
- .total_cmp(&other.imbalance)
+ self.score
+ .total_cmp(&other.score)
+ .then(self.imbalance.total_cmp(&other.imbalance))
.then(self.migration.cmp(&other.migration))
}
}
@@ -136,19 +130,21 @@ impl PartialEq for ScoredMigration {
impl Eq for ScoredMigration {}
impl ScoredMigration {
- pub fn new<T: Into<Migration>>(migration: T, imbalance: f64) -> Self {
- // Depending on how the imbalance is calculated, it can contain minor approximation errors.
- // As this struct implements the Ord trait, users of the struct's cmp() can run into cases,
- // where the imbalance is the same up to the significant digits in base 10, but treated as
- // different values.
+ pub fn new<T: Into<Migration>>(migration: T, imbalance: f64, score: f64) -> Self {
+ // Depending on how the imbalance or score is calculated, it can contain minor approximation
+ // errors. As this struct implements the Ord trait, users of the struct's cmp() can run into
+ // cases, where the imbalance or score is the same up to the significant digits in base 10,
+ // but treated as different values.
//
// Therefore, truncate any non-significant digits to prevent these cases.
let factor = 10_f64.powi(f64::DIGITS as i32);
let truncated_imbalance = f64::trunc(factor * imbalance) / factor;
+ let truncated_score = f64::trunc(factor * score) / factor;
Self {
migration: migration.into(),
imbalance: truncated_imbalance,
+ score: truncated_score,
}
}
}
@@ -179,15 +175,15 @@ impl Scheduler {
}
}
- /// Maps the current node usages to a [`PveTopsisAlternative`].
+ /// Maps the current node usages to a TOPSIS alternative.
///
- /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
- /// square (RMS) and maximum value of each stat in the node usages.
+ /// The alternative is derived by calculating a modified version of the root mean square (RMS)
+ /// and maximum value of each stat in the node usages.
fn topsis_alternative_with(
&self,
map_node_stats: impl Fn(&NodeUsage) -> NodeStats,
- ) -> PveTopsisAlternative {
- let len = self.nodes.len();
+ ) -> [f64; 4] {
+ let len = self.nodes.len() as f64;
// Base values on percentages to allow comparing nodes with different stats.
let mut highest_cpu = 0.0;
@@ -209,55 +205,79 @@ impl Scheduler {
// Add 1.0 to avoid boosting tiny differences: e.g. 0.004 is twice as much as 0.002, but
// 1.004 is only slightly more than 1.002.
- PveTopsisAlternative {
- average_cpu: 1.0 + (squares_cpu / len as f64).sqrt(),
- highest_cpu: 1.0 + highest_cpu,
- average_memory: 1.0 + (squares_mem / len as f64).sqrt(),
- highest_memory: 1.0 + highest_mem,
- }
+ [
+ 1.0 + (squares_cpu / len).sqrt(),
+ 1.0 + highest_cpu,
+ 1.0 + (squares_mem / len).sqrt(),
+ 1.0 + highest_mem,
+ ]
}
/// Returns the load imbalance among the nodes.
///
+ /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
+ /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
+ /// [`NodeStats::load`].
+ ///
/// See [`calculate_node_imbalance`] for more information.
- pub fn node_imbalance(&self) -> f64 {
- calculate_node_imbalance(&self.nodes, |node| node.stats.load())
+ pub fn node_imbalance(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
+ calculate_node_imbalance(&self.nodes, |node| node.stats.load(cpu_weight, mem_weight))
}
/// Returns the load imbalance among the nodes as if a specific resource was moved.
///
+ /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
+ /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
+ /// [`NodeStats::load`].
+ ///
/// See [`calculate_node_imbalance`] for more information.
- fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
+ fn node_imbalance_with_migration_candidate(
+ &self,
+ candidate: &MigrationCandidate,
+ cpu_weight: f64,
+ mem_weight: f64,
+ ) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- node_stats_after_migration(candidate, node).load()
+ node_stats_after_migration(candidate, node).load(cpu_weight, mem_weight)
})
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with
- /// exhaustive search.
+ /// Uses brute-force search to score how much either of the given migration `candidates`
+ /// improves node imbalance and ranks them accordingly.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
///
/// The scoring is done as if each resource migration has already been done. This assumes that
/// the already migrated resource consumes the same amount of each stat as on the previous node
- /// according to its `stats`.
+ /// according to its `stats`. The lower the score, the better.
+ ///
+ /// `cpu_weight` and `mem_weight` are blended into [`NodeStats::load`]. The higher the weight of
+ /// a computational resource, the more the search will tend to prioritize balancing out that
+ /// resource, e.g., a high `mem_weight` will steer migrations toward evening out memory usage,
+ /// even at the cost of leaving CPU load less balanced. This is a general tendency, not a
+ /// guarantee.
///
/// Returns up to `limit` of the best scored migrations.
pub fn score_best_balancing_migration_candidates<I>(
&self,
candidates: I,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Vec<ScoredMigration>
where
I: IntoIterator<Item = MigrationCandidate>,
{
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
let mut scored_migrations = candidates
.into_iter()
.map(|candidate| {
- let imbalance = self.node_imbalance_with_migration_candidate(&candidate);
+ let imbalance = self.node_imbalance_with_migration_candidate(&candidate, 1.0, 1.0);
+ let score = self.node_imbalance_with_migration_candidate(&candidate, w1, w2);
- Reverse(ScoredMigration::new(candidate, imbalance))
+ Reverse(ScoredMigration::new(candidate, imbalance, score))
})
.collect::<BinaryHeap<_>>();
@@ -274,56 +294,106 @@ impl Scheduler {
best_migrations
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with the
- /// TOPSIS method.
+ /// Scores the given migration `candidates` using the TOPSIS method.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
///
/// The scoring is done as if each resource migration has already been done. This assumes that
/// the already migrated resource consumes the same amount of each stat as on the previous node
- /// according to its `stats`.
+ /// according to its `stats`. The lower the score, the better.
+ ///
+ /// The weights are assigned to the respective criteria of the TOPSIS method.
///
/// Returns up to `limit` of the best scored migrations.
pub fn score_best_balancing_migration_candidates_topsis(
&self,
candidates: &[MigrationCandidate],
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
+ let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
+ let highest_cpu = topsis::Criterion::new(
+ "highest_cpu".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
+ );
+ let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
+ let highest_mem = topsis::Criterion::new(
+ "highest_mem".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
+ );
+
let matrix = candidates
.iter()
.map(|candidate| {
self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
- .into()
})
.collect::<Vec<_>>();
- let best_alternatives =
- topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ let mut scored_migrations = topsis::score_alternatives(
+ &topsis::Matrix::new(matrix)?,
+ &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
+ )?
+ .iter()
+ .zip(candidates.iter())
+ .map(|(topsis_score, candidate)| {
+ let imbalance = self.node_imbalance_with_migration_candidate(candidate, w1, w2);
+
+ Reverse(ScoredMigration::new(
+ candidate.clone(),
+ imbalance,
+ // invert score for consistent sorting
+ 1.0 - topsis_score,
+ ))
+ })
+ .collect::<BinaryHeap<_>>();
- Ok(best_alternatives
- .into_iter()
- .take(limit)
- .map(|i| {
- let imbalance = self.node_imbalance_with_migration_candidate(&candidates[i]);
+ let mut best_migrations = Vec::with_capacity(limit);
- ScoredMigration::new(candidates[i].clone(), imbalance)
- })
- .collect())
+ // BinaryHeap::into_iter_sorted() is still in nightly unfortunately
+ while best_migrations.len() < limit {
+ match scored_migrations.pop() {
+ Some(Reverse(alternative)) => best_migrations.push(alternative),
+ None => break,
+ }
+ }
+
+ Ok(best_migrations)
}
- /// Scores nodes to start a resource with the usage statistics `resource_stats` on.
+ /// Scores nodes to start a resource with the usage statistics `resource_stats` on, using the
+ /// TOPSIS method.
///
/// The scoring is done as if the resource is already started on each node. This assumes that
/// the already started resource consumes the maximum amount of each stat according to its
/// `resource_stats`.
///
+ /// The weights are assigned to the respective criteria of the TOPSIS method.
+ ///
/// Returns a vector of (nodename, score) pairs. Scores are between 0.0 and 1.0 and a higher
/// score is better.
pub fn score_nodes_to_start_resource<T: Into<ResourceStats>>(
&self,
resource_stats: T,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
+ let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
+ let highest_cpu = topsis::Criterion::new(
+ "highest_cpu".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
+ );
+ let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
+ let highest_mem = topsis::Criterion::new(
+ "highest_mem".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
+ );
+
let resource_stats = resource_stats.into();
let matrix = self
@@ -339,12 +409,13 @@ impl Scheduler {
new_stats
})
- .into()
})
.collect::<Vec<_>>();
- let scores =
- topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ let scores = topsis::score_alternatives(
+ &topsis::Matrix::new(matrix)?,
+ &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
+ )?;
Ok(scores
.into_iter()
@@ -367,6 +438,7 @@ mod tests {
target_node: String::from("node2"),
},
0.7231749488916931,
+ 0.7231749488916931,
);
let migration2 = ScoredMigration::new(
Migration {
@@ -375,6 +447,7 @@ mod tests {
target_node: String::from("node3"),
},
0.723174948891693,
+ 0.723174948891693,
);
let migration3 = ScoredMigration::new(
Migration {
@@ -383,6 +456,57 @@ mod tests {
target_node: String::from("node2"),
},
0.723174948891693 + 1e-15,
+ 0.723174948891693 + 1e-15,
+ );
+
+ let mut migrations = vec![migration2.clone(), migration3.clone(), migration1.clone()];
+
+ migrations.sort();
+
+ assert_eq!(
+ vec![migration1.clone(), migration2.clone(), migration3.clone()],
+ migrations
+ );
+
+ let mut heap = BinaryHeap::from(vec![
+ Reverse(migration2.clone()),
+ Reverse(migration3.clone()),
+ Reverse(migration1.clone()),
+ ]);
+
+ assert_eq!(heap.pop(), Some(Reverse(migration1)));
+ assert_eq!(heap.pop(), Some(Reverse(migration2)));
+ assert_eq!(heap.pop(), Some(Reverse(migration3)));
+ }
+
+ #[test]
+ fn test_score_tie_breaks_on_imbalance() {
+ let migration1 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:102"),
+ source_node: String::from("node1"),
+ target_node: String::from("node2"),
+ },
+ 0.0,
+ 0.1,
+ );
+ let migration2 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:102"),
+ source_node: String::from("node1"),
+ target_node: String::from("node3"),
+ },
+ 0.0,
+ 0.2,
+ );
+ let migration3 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:101"),
+ source_node: String::from("node1"),
+ target_node: String::from("node2"),
+ },
+ 0.3,
+ 0.3,
);
let mut migrations = vec![migration2.clone(), migration3.clone(), migration1.clone()];
diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
index 61fdd840..19e2c353 100644
--- a/proxmox-resource-scheduling/tests/scheduler.rs
+++ b/proxmox-resource-scheduling/tests/scheduler.rs
@@ -5,6 +5,11 @@ use proxmox_resource_scheduling::{
scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
};
+const DEFAULT_CPU_WEIGHT: usize = 1;
+const DEFAULT_MEM_WEIGHT: usize = 1;
+
+const TOPSIS_CPU_VS_MEM_FACTOR: usize = 5;
+
fn new_empty_cluster_scheduler() -> Scheduler {
Scheduler::from_nodes(Vec::<NodeUsage>::new())
}
@@ -83,7 +88,10 @@ fn new_heterogeneous_cluster_scheduler() -> Scheduler {
fn test_node_imbalance_with_empty_cluster() {
let scheduler = new_empty_cluster_scheduler();
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
}
#[test]
@@ -100,11 +108,17 @@ fn test_node_imbalance_with_perfectly_balanced_cluster() {
let scheduler = Scheduler::from_nodes(vec![node.clone()]);
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
let scheduler = Scheduler::from_nodes(vec![node.clone(), node.clone(), node]);
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
}
fn new_simple_migration_candidates() -> (Vec<MigrationCandidate>, Migration, Migration) {
@@ -150,9 +164,11 @@ fn rank_best_balancing_migration_candidates(
scheduler: &Scheduler,
candidates: Vec<MigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Vec<Migration> {
scheduler
- .score_best_balancing_migration_candidates(candidates, limit)
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight)
.into_iter()
.map(|entry| entry.migration)
.collect()
@@ -163,7 +179,13 @@ fn test_score_best_balancing_migration_candidates_with_no_candidates() {
let scheduler = new_homogeneous_cluster_scheduler();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, vec![], 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ vec![],
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![]
);
}
@@ -172,12 +194,21 @@ fn test_score_best_balancing_migration_candidates_with_no_candidates() {
fn test_score_best_balancing_migration_candidates_in_homogeneous_cluster() {
let scheduler = new_homogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.3460548572604576);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.3460548572604576,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, candidates, 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ candidates,
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![migration2, migration1]
);
}
@@ -186,12 +217,21 @@ fn test_score_best_balancing_migration_candidates_in_homogeneous_cluster() {
fn test_score_best_balancing_migration_candidates_in_heterogeneous_cluster() {
let scheduler = new_heterogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.23352917788066363);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.23352917788066363,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, candidates, 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ candidates,
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![migration2, migration1]
);
}
@@ -202,7 +242,12 @@ fn rank_best_balancing_migration_candidates_topsis(
limit: usize,
) -> Result<Vec<Migration>, Error> {
Ok(scheduler
- .score_best_balancing_migration_candidates_topsis(candidates, limit)?
+ .score_best_balancing_migration_candidates_topsis(
+ candidates,
+ limit,
+ DEFAULT_CPU_WEIGHT,
+ 2 * DEFAULT_MEM_WEIGHT,
+ )?
.into_iter()
.map(|entry| entry.migration)
.collect())
@@ -225,7 +270,10 @@ fn test_score_best_balancing_migration_candidates_topsis_in_homogeneous_cluster(
-> Result<(), Error> {
let scheduler = new_homogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.3460548572604576);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.3460548572604576,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
@@ -242,7 +290,10 @@ fn test_score_best_balancing_migration_candidates_topsis_in_heterogeneous_cluste
-> Result<(), Error> {
let scheduler = new_heterogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.23352917788066363);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.23352917788066363,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
@@ -257,8 +308,11 @@ fn test_score_best_balancing_migration_candidates_topsis_in_heterogeneous_cluste
fn rank_nodes_to_start_resource(
scheduler: &Scheduler,
resource_stats: ResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<String>, Error> {
- let mut alternatives = scheduler.score_nodes_to_start_resource(resource_stats)?;
+ let mut alternatives =
+ scheduler.score_nodes_to_start_resource(resource_stats, cpu_weight, mem_weight)?;
alternatives.sort_by(|a, b| b.1.total_cmp(&a.1));
@@ -280,7 +334,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_memory_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_memory_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node3", "node1"]
);
@@ -292,7 +351,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -304,7 +368,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, unlimited_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ unlimited_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -316,7 +385,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ combined_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node3", "node1"]
);
@@ -335,7 +409,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_memory_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_memory_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node1", "node3"]
);
@@ -347,7 +426,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node3", "node2", "node1"]
);
@@ -359,7 +443,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, unlimited_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ unlimited_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -371,7 +460,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ combined_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node1", "node3"]
);
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (2 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights Dominik Rusovac
` (10 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
proptest 1.11 exists, but version 1.6 is fine for now.
afaict updating to 1.11 at some point shouldn't come with breaking
changes.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Cargo.toml | 1 +
proxmox-resource-scheduling/Cargo.toml | 3 +++
2 files changed, 4 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index 09ef21b4..72c18c54 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -155,6 +155,7 @@ udev = "0.9"
url = "2.2"
walkdir = "2"
zstd = "0.13"
+proptest = "1.6.0"
# workspace dependencies
proxmox-access-control = { version = "1.3.0", path = "proxmox-access-control" }
diff --git a/proxmox-resource-scheduling/Cargo.toml b/proxmox-resource-scheduling/Cargo.toml
index 2bfb7871..27d0f3f6 100644
--- a/proxmox-resource-scheduling/Cargo.toml
+++ b/proxmox-resource-scheduling/Cargo.toml
@@ -11,3 +11,6 @@ exclude.workspace = true
[dependencies]
anyhow.workspace = true
serde = { workspace = true, features = [ "derive" ] }
+
+[dev-dependencies]
+proptest.workspace = true
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (3 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
` (9 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Introduce proptests with constants to control scheduler proptests and
functions implementing useful proptest strategies.
Use per-method `mod`s to make deliberate testing easier, e.g., executing
`cargo test bruteforce` to run bruteforce (prop)tests only.
For each method (bruteforce, TOPSIS), respectively, add proptests:
- test_weight_scale_invariance: test that only weight ratio matters, not
their absolute magnitudes; and
- test_zero_cpu_weight_discards_cpu_load: cpu weight of 0 discards CPU
load (symmetric test discarding memory usage is omitted).
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 343 +++++++++++++++++++
1 file changed, 343 insertions(+)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 0690385d..3c4c8d87 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -429,6 +429,27 @@ impl Scheduler {
mod tests {
use super::*;
+ use proptest::prelude::*;
+ use std::ops::Range;
+
+ const MIN_NODES: usize = 3;
+ const MAX_NODES: usize = 36;
+
+ const MIN_CPU: usize = 2;
+ const MIN_MEM: usize = 2;
+ const MAX_CPU: usize = 128;
+ const MAX_MEM: usize = 256;
+
+ const MIN_WEIGHT: usize = 1;
+ const MAX_WEIGHT: usize = 1_000;
+
+ const MIN_CONSTANT: usize = 1;
+ const MAX_CONSTANT: usize = 1_000;
+
+ const REPEAT_PROPTEST_N_TIMES: u32 = 256; // proptest's default
+
+ const EPSILON: f64 = 1e-9;
+
#[test]
fn test_scored_migration_order() {
let migration1 = ScoredMigration::new(
@@ -528,4 +549,326 @@ mod tests {
assert_eq!(heap.pop(), Some(Reverse(migration2)));
assert_eq!(heap.pop(), Some(Reverse(migration3)));
}
+
+ /// Strategy to generate `n` random nodes with stats bound by `max_cpu` and `max_mem`.
+ pub fn nodes_strategy(
+ n: usize,
+ max_cpu: usize,
+ max_mem: usize,
+ ) -> impl Strategy<Value = Vec<NodeUsage>> {
+ prop::collection::vec((0..max_cpu + 1, 0..max_mem + 1), n).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(|(i, (cpu, mem))| NodeUsage {
+ name: format!("node{i}"),
+ stats: NodeStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu,
+ mem,
+ maxmem: max_mem,
+ },
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate a random number in `n_range` of migration candidates between `nodes`.
+ /// Resource stats are bound by `max_cpu` and `max_mem`.
+ fn candidates_strategy(
+ n_range: impl Into<proptest::collection::SizeRange>,
+ max_cpu: usize,
+ max_mem: usize,
+ nodes: Vec<NodeUsage>,
+ ) -> impl Strategy<Value = Vec<MigrationCandidate>> {
+ let k = nodes.len();
+
+ prop::collection::vec((0..max_cpu, 0..max_mem, 0..k, 0..k), n_range).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(
+ |(i, (cpu, mem, source_idx, target_idx))| MigrationCandidate {
+ migration: Migration {
+ sid: format!("vm:10{i}"),
+ source_node: nodes[source_idx].name.clone(),
+ target_node: nodes[target_idx].name.clone(),
+ },
+ stats: ResourceStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu as f64,
+ mem,
+ maxmem: max_mem,
+ },
+ },
+ )
+ .filter(|candidate| {
+ candidate.migration.source_node != candidate.migration.target_node
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate:
+ /// - a random number n in `n_nodes_range` of nodes
+ /// - a random number between 1 and n * (n - 1) of migration candidates between the generated
+ /// nodes
+ ///
+ /// The stats of nodes and migration candidates are bound by random numbers in `cpu_range` and
+ /// `mem_range`, respectively.
+ fn scenario_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<Value = (Vec<NodeUsage>, Vec<MigrationCandidate>)> {
+ (cpu_range, mem_range).prop_flat_map(move |(max_cpu, max_mem)| {
+ n_nodes_range
+ .clone()
+ .prop_filter("cluster size must be odd", |n| n % 2 == 1)
+ .prop_flat_map(move |n_nodes| {
+ nodes_strategy(n_nodes, max_cpu, max_mem).prop_flat_map(move |nodes| {
+ let max_candidates = n_nodes * (n_nodes - 1);
+
+ candidates_strategy(1..=max_candidates, max_cpu, max_mem, nodes.clone())
+ .prop_map(move |candidates| (nodes.clone(), candidates))
+ })
+ })
+ })
+ }
+
+ mod bruteforce {
+ use super::*;
+
+ mod weights {
+ use super::*;
+
+ proptest! {
+ #![proptest_config(ProptestConfig {
+ cases: REPEAT_PROPTEST_N_TIMES,
+ ..ProptestConfig::default()
+ })]
+
+ /// Test that the achievable scores are completely unaffected by uniformly scaling
+ /// both weights; only the weights' ratio matters.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let scheduler = Scheduler::from_nodes(nodes);
+ let n = candidates.len();
+
+ scheduler
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .iter()
+ .zip(
+ scheduler
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ n,
+ k * cpu_weight,
+ k * mem_weight,
+ )
+ .iter(),
+ )
+ .for_each(|(lhs, rhs)| {
+ assert!(
+ (lhs.score - rhs.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ lhs,
+ rhs,
+ );
+ });
+ }
+
+ /// Test that with a `cpu_weight` of 0 the score for each scored migration should
+ /// match an imbalance computed directly from [`NodeStats::mem_load()`] alone.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let cpu_weight = 0;
+
+ Scheduler::from_nodes(nodes.clone())
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ candidates.len(),
+ cpu_weight,
+ mem_weight,
+ )
+ .iter()
+ .filter_map(|scored_migration| {
+ candidates
+ .iter()
+ .find(|c| c.migration.sid == scored_migration.migration.sid)
+ .map(|c| (scored_migration, c))
+ })
+ .for_each(|(scored_migration, candidate)| {
+ let reference_imbalance = calculate_node_imbalance(&nodes, |node| {
+ node_stats_after_migration(candidate, node).mem_load()
+ });
+
+ assert!(
+ (scored_migration.score - reference_imbalance).abs() < EPSILON
+ );
+ })
+ }
+ }
+ }
+ }
+
+ mod topsis {
+ use super::*;
+
+ /// Strategy producing two scenarios using [`super::scenario_strategy`] that are identical
+ /// in every memory-related field and in topology but have varying CPU fields.
+ #[allow(clippy::type_complexity)]
+ fn two_scenarios_with_varying_cpu_stats_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<
+ Value = (
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ ),
+ > {
+ let max_possible_nodes = n_nodes_range.end - 1; // Range<T> is exclusive
+ let max_possible_candidates = max_possible_nodes * (max_possible_nodes - 1);
+ let cpu_pool_bound = cpu_range.end;
+
+ (
+ scenario_strategy(cpu_range, mem_range, n_nodes_range),
+ prop::collection::vec(0usize..cpu_pool_bound + 1, max_possible_nodes),
+ prop::collection::vec(0usize..cpu_pool_bound, max_possible_candidates),
+ )
+ .prop_map(
+ move |((nodes, candidates), alt_node_cpu_pool, alt_candidate_cpu_pool)| {
+ let mut alt_nodes = nodes.clone();
+ alt_nodes.iter_mut().zip(alt_node_cpu_pool.iter()).for_each(
+ |(node, &cpu)| {
+ node.stats.cpu = cpu as f64;
+ },
+ );
+
+ let mut alt_candidates = candidates.clone();
+ alt_candidates
+ .iter_mut()
+ .zip(alt_candidate_cpu_pool.iter())
+ .for_each(|(candidate, &cpu)| {
+ candidate.stats.cpu = cpu as f64;
+ });
+
+ ((nodes, candidates), (alt_nodes, alt_candidates))
+ },
+ )
+ }
+
+ proptest! {
+ #![proptest_config(ProptestConfig {
+ cases: REPEAT_PROPTEST_N_TIMES,
+ ..ProptestConfig::default()
+ })]
+
+ /// Test that the achievable scores are completely unaffected by uniformly scaling both
+ /// weights; only the weights' ratio matters.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let scheduler = Scheduler::from_nodes(nodes);
+ let n = candidates.len();
+
+ scheduler
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter()
+ .zip(
+ scheduler
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ n,
+ k * cpu_weight,
+ k * mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter(),
+ )
+ .for_each(|(lhs, rhs)| {
+ assert!(
+ (lhs.score - rhs.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ lhs,
+ rhs,
+ );
+ });
+ }
+
+ /// Test that varying CPU stats between two otherwise identical scenarios do not change
+ /// the achievable scores if CPU weight is 0.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ ((nodes_a, candidates_a), (nodes_b, candidates_b)) in
+ two_scenarios_with_varying_cpu_stats_strategy(
+ MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES
+ ),
+ ) {
+ let cpu_weight = 0;
+ let n = candidates_a.len();
+
+ Scheduler::from_nodes(nodes_a)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates_a,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter()
+ .zip(
+ Scheduler::from_nodes(nodes_b)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates_b,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter(),
+ )
+ .for_each(|(a, b)| {
+ assert!(
+ (a.score - b.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ a,
+ b,
+ );
+ });
+ }
+ }
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (4 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights Dominik Rusovac
` (8 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add weights for weighted load balancing.
Adjust existing resource scheduling test to explicitly use same weights
as before.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
.../resource_scheduling/pve_dynamic.rs | 27 +++++++++++++++----
.../resource_scheduling/pve_static.rs | 25 +++++++++++++----
pve-rs/test/resource_scheduling.pl | 3 ++-
3 files changed, 44 insertions(+), 11 deletions(-)
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
index 8e066e0..d6b6f9b 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
@@ -166,10 +166,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
///
/// See [`proxmox_resource_scheduling::scheduler::Scheduler::node_imbalance`].
#[export]
- pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
+ pub fn calculate_node_imbalance(
+ #[try_from_ref] this: &Scheduler,
+ cpu_weight: usize,
+ mem_weight: usize,
+ ) -> f64 {
let usage = this.inner.lock().unwrap();
- usage.to_scheduler::<IdentityAggregator>().node_imbalance()
+ usage
+ .to_scheduler::<IdentityAggregator>()
+ .node_imbalance(cpu_weight as f64, mem_weight as f64)
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -181,6 +187,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -188,7 +196,7 @@ pub mod pve_rs_resource_scheduling_dynamic {
Ok(usage
.to_scheduler::<IdentityAggregator>()
- .score_best_balancing_migration_candidates(candidates, limit))
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight))
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -200,6 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -207,7 +217,12 @@ pub mod pve_rs_resource_scheduling_dynamic {
usage
.to_scheduler::<IdentityAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ limit,
+ cpu_weight,
+ mem_weight,
+ )
}
/// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
@@ -217,11 +232,13 @@ pub mod pve_rs_resource_scheduling_dynamic {
pub fn score_nodes_to_start_resource(
#[try_from_ref] this: &Scheduler,
resource_stats: DynamicResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartingAsStartedResourceAggregator>()
- .score_nodes_to_start_resource(resource_stats)
+ .score_nodes_to_start_resource(resource_stats, cpu_weight, mem_weight)
}
}
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_static.rs b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
index d83aa38..2fa8d8c 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_static.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
@@ -162,12 +162,16 @@ pub mod pve_rs_resource_scheduling_static {
///
/// See [`proxmox_resource_scheduling::scheduler::Scheduler::node_imbalance`].
#[export]
- pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
+ pub fn calculate_node_imbalance(
+ #[try_from_ref] this: &Scheduler,
+ cpu_weight: usize,
+ mem_weight: usize,
+ ) -> f64 {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartedResourceAggregator>()
- .node_imbalance()
+ .node_imbalance(cpu_weight as f64, mem_weight as f64)
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -179,6 +183,8 @@ pub mod pve_rs_resource_scheduling_static {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -186,7 +192,7 @@ pub mod pve_rs_resource_scheduling_static {
Ok(usage
.to_scheduler::<StartedResourceAggregator>()
- .score_best_balancing_migration_candidates(candidates, limit))
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight))
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -198,6 +204,8 @@ pub mod pve_rs_resource_scheduling_static {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -205,7 +213,12 @@ pub mod pve_rs_resource_scheduling_static {
usage
.to_scheduler::<StartedResourceAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ limit,
+ cpu_weight,
+ mem_weight,
+ )
}
/// Method: Scores nodes to start a service with the usage statistics `service_stats` on.
@@ -215,11 +228,13 @@ pub mod pve_rs_resource_scheduling_static {
pub fn score_nodes_to_start_service(
#[try_from_ref] this: &Scheduler,
service_stats: StaticResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartedResourceAggregator>()
- .score_nodes_to_start_resource(service_stats)
+ .score_nodes_to_start_resource(service_stats, cpu_weight, mem_weight)
}
}
diff --git a/pve-rs/test/resource_scheduling.pl b/pve-rs/test/resource_scheduling.pl
index 3775242..31f2622 100755
--- a/pve-rs/test/resource_scheduling.pl
+++ b/pve-rs/test/resource_scheduling.pl
@@ -11,7 +11,8 @@ use PVE::RS::ResourceScheduling::Dynamic;
my sub score_nodes {
my ($static, $service) = @_;
- my $score_list = $static->score_nodes_to_start_service($service);
+ # uses original empirically determined weights
+ my $score_list = $static->score_nodes_to_start_service($service, 1.0, 5.0);
# imitate HA manager
my $scores = { map { $_->[0] => -$_->[1] } $score_list->@* };
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (5 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-manager 08/15] ui: form: crs options: make options advanced Dominik Rusovac
` (7 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Default 'Memory Weight' changes with selected method; value should be 5
for 'TOPSIS' and 1 otherwise.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
www/manager6/form/CRSOptions.js | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index 84f6d506..cde389ee 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -20,6 +20,7 @@ Ext.define('PVE.form.CRSOptions', {
viewModel: {
data: {
crsMode: '__default__',
+ rebalanceMethod: '__default__',
autoRebalancing: 0,
},
formulas: {
@@ -27,6 +28,9 @@ Ext.define('PVE.form.CRSOptions', {
get('crsMode') === 'static' || get('crsMode') === 'dynamic',
autoRebalancingDisabled: (get) =>
!get('autoRebalancing') || !get('canUseAutoRebalancing'),
+ isTopsis: (get) => get('rebalanceMethod') === 'topsis',
+ memWeightEmptyText: (get) =>
+ Proxmox.Utils.defaultText + (get('isTopsis') ? ' (5)' : ' (1)'),
},
},
@@ -100,6 +104,7 @@ Ext.define('PVE.form.CRSOptions', {
],
defaultValue: '__default__',
bind: {
+ value: '{rebalanceMethod}',
disabled: '{autoRebalancingDisabled}',
},
},
@@ -126,6 +131,28 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ {
+ xtype: 'numberfield',
+ name: 'ha-auto-rebalance-cpu-weight',
+ fieldLabel: gettext('CPU Weight'),
+ emptyText: Proxmox.Utils.defaultText + ' (1)',
+ minValue: 0,
+ step: 1,
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ },
+ },
+ {
+ xtype: 'numberfield',
+ name: 'ha-auto-rebalance-mem-weight',
+ fieldLabel: gettext('Memory Weight'),
+ minValue: 0,
+ step: 1,
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ emptyText: '{memWeightEmptyText}',
+ },
+ },
],
},
],
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-manager 08/15] ui: form: crs options: make options advanced
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (6 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
` (6 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
With the addition of auto-rebalance weights the CRS settings in the UI
have become even more cluttered.
Make 'Hold Duration', 'Minimal Imbalance Improvement', 'CPU Weight' and
'Memory Weight' advanced options.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
www/manager6/form/CRSOptions.js | 2 ++
1 file changed, 2 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index cde389ee..8419a805 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -108,6 +108,8 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ ],
+ advancedItems: [
{
xtype: 'numberfield',
name: 'ha-auto-rebalance-hold-duration',
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 09/15] manager: remove obsolete argument
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (7 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-manager 08/15] ui: form: crs options: make options advanced Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
` (5 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Respective parameter was removed by [0].
[0] https://lore.proxmox.com/all/20260319164047.642936-4-d.kral@proxmox.com/
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..7a5a685 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -385,7 +385,7 @@ sub select_service_node {
return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
- my $scores = $online_node_usage->score_nodes_to_start_service($sid, $current_node);
+ my $scores = $online_node_usage->score_nodes_to_start_service($sid);
my @nodes = sort {
$scores->{$a} <=> $scores->{$b} || $a cmp $b
} keys %$pri_nodes;
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (8 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime Dominik Rusovac
` (4 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Include ha-auto-rebalance-cpu-weight and ha-auto-rebalance-mem-weight in
load imbalance calculation and decision making.
The weights only play into decisions in the context of automatic
rebalancing. The bruteforce method optimizes the weighted load
imbalance. For the TOPSIS method ha-auto-rebalance-cpu-weight and
ha-auto-rebalance-mem-weight are assigned to the respective average cpu
and average memory criteria. The weights of the highest cpu and highest
memory criteria of TOPSIS are not independently configurable. Nodes to
start service are scored with the same fixed (TOPSIS) weights as before.
The weights also shape how sensitively the load balancer reacts to CPU
load changes relative to memory usage changes. This is to address: "In
our case CRS has not moved any VM around because the imbalance
percentage is low (7%). Eventhough the cluster is very imbalance [sic]
CPU wise, or memory wise alone." [0]
Skip load balancing with warning if both weights are 0.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 47 ++++++++++++++++---
src/PVE/HA/Usage.pm | 21 +++++----
src/PVE/HA/Usage/Basic.pm | 14 ++++--
src/PVE/HA/Usage/Dynamic.pm | 27 +++++++----
src/PVE/HA/Usage/Static.pm | 27 +++++++----
.../log.expect | 13 +----
6 files changed, 102 insertions(+), 47 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 7a5a685..c6818e2 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -119,6 +119,12 @@ sub update_crs_scheduler_mode {
$self->{crs}->{auto_rebalance}->{margin} =
($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ my $default_weight = 1;
+ $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
+ # by default TOPSIS weights mem higher than cpu
+ $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
+ $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
+
my $old_mode = $self->{crs}->{scheduler};
my $new_mode = $crs_cfg->{ha} || 'basic';
@@ -236,8 +242,20 @@ sub load_balance {
return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
return if $self->any_resource_motion_queued_or_running();
- my ($threshold, $method, $hold_duration, $margin) =
- $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
+ my (
+ $threshold, $method, $hold_duration, $margin, $cpu_weight, $mem_weight,
+ ) =
+ $auto_rebalance_opts->@{qw(threshold method hold_duration margin cpu_weight mem_weight)};
+
+ if ($cpu_weight + $mem_weight eq 0) {
+ $haenv->log(
+ 'warning',
+ "auto-rebalance weights are both set to 0, skipping load balancing -"
+ . " set at least one to a positive value to avoid constant load"
+ . " imbalance of 0",
+ );
+ return;
+ }
# with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
# migration, churning the cluster forever - skip and warn instead of silently looping.
@@ -250,7 +268,7 @@ sub load_balance {
return;
}
- my $imbalance = $online_node_usage->calculate_node_imbalance();
+ my $imbalance = $online_node_usage->calculate_node_imbalance($cpu_weight, $mem_weight);
# do not load balance unless imbalance threshold has been exceeded
# consecutively for $hold_duration calls to load_balance();
@@ -268,15 +286,24 @@ sub load_balance {
my $result;
if ($method eq 'bruteforce') {
- $result = $online_node_usage->select_best_balancing_migration($candidates);
+ $result = $online_node_usage->select_best_balancing_migration(
+ $candidates,
+ $cpu_weight,
+ $mem_weight,
+ );
} elsif ($method eq 'topsis') {
- $result = $online_node_usage->select_best_balancing_migration_topsis($candidates);
+ $result = $online_node_usage->select_best_balancing_migration_topsis(
+ $candidates,
+ $cpu_weight,
+ $mem_weight,
+ );
}
# happens if $candidates is empty or $method isn't handled above
return if !$result;
- my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
+ my ($migration, $target_imbalance, $score) = $result->@{qw(migration imbalance score)};
+ $target_imbalance = $score if $method eq 'bruteforce';
my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
return if $relative_change < $margin;
@@ -311,7 +338,13 @@ sub flush_master_status {
$ms->{node_status} = $ns->{status};
$ms->{service_status} = $ss;
$ms->{timestamp} = $haenv->get_time();
- $ms->{imbalance} = $self->{online_node_usage}->calculate_node_imbalance();
+
+ my ($cpu_weight, $mem_weight) = (
+ $self->{crs}->{auto_rebalance}->{cpu_weight},
+ $self->{crs}->{auto_rebalance}->{mem_weight},
+ );
+ $ms->{imbalance} =
+ $self->{online_node_usage}->calculate_node_imbalance($cpu_weight, $mem_weight);
$haenv->write_manager_status($ms);
}
diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
index 659ab30..fbd8800 100644
--- a/src/PVE/HA/Usage.pm
+++ b/src/PVE/HA/Usage.pm
@@ -61,42 +61,47 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
sub select_best_balancing_migration {
- my ($self, $migration_candidates) = @_;
+ my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
- my $migrations = $self->score_best_balancing_migrations($migration_candidates, 1);
+ my $migrations =
+ $self->score_best_balancing_migrations($migration_candidates, 1, $cpu_weight, $mem_weight);
return $migrations->[0];
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
die "implement in subclass";
}
sub select_best_balancing_migration_topsis {
- my ($self, $migration_candidates) = @_;
+ my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
- my $migrations = $self->score_best_balancing_migrations_topsis($migration_candidates, 1);
+ my $migrations = $self->score_best_balancing_migrations_topsis(
+ $migration_candidates, 1, $cpu_weight, $mem_weight,
+ );
return $migrations->[0];
}
# Returns a hash with $nodename => $score pairs. A lower $score is better.
sub score_nodes_to_start_service {
- my ($self, $sid) = @_;
+ my ($self, $sid, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
index 4dce9e1..57eecd3 100644
--- a/src/PVE/HA/Usage/Basic.pm
+++ b/src/PVE/HA/Usage/Basic.pm
@@ -67,19 +67,27 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
return 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
return [];
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self,
+ $migration_candidates,
+ $limit,
+ $cpu_weight,
+ $max_cpu_weight,
+ $mem_weight,
+ $max_mem_weight,
+ ) = @_;
return [];
}
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..bd03e89 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -105,20 +105,22 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
- my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
+ my $node_imbalance =
+ eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
$self->{haenv}->log('warning', "unable to calculate dynamic node imbalance - $@") if $@;
return $node_imbalance // 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -126,11 +128,14 @@ sub score_best_balancing_migrations {
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -140,9 +145,13 @@ sub score_best_balancing_migrations_topsis {
sub score_nodes_to_start_service {
my ($self, $sid) = @_;
+ my ($cpu_weight, $mem_weight) = (1, 5);
+
my $score_list = eval {
my $service_usage = get_service_usage($self, $sid);
- $self->{scheduler}->score_nodes_to_start_resource($service_usage);
+ $self->{scheduler}->score_nodes_to_start_resource(
+ $service_usage, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}
->log('err', "unable to score nodes according to dynamic usage for service '$sid' - $@")
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index e67d5f5..39308ff 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -112,20 +112,22 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
- my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
+ my $node_imbalance =
+ eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
$self->{haenv}->log('warning', "unable to calculate static node imbalance - $@") if $@;
return $node_imbalance // 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -133,11 +135,14 @@ sub score_best_balancing_migrations {
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -147,9 +152,13 @@ sub score_best_balancing_migrations_topsis {
sub score_nodes_to_start_service {
my ($self, $sid) = @_;
+ my ($cpu_weight, $mem_weight) = (1, 5);
+
my $score_list = eval {
my $service_usage = get_service_usage($self, $sid);
- $self->{scheduler}->score_nodes_to_start_service($service_usage);
+ $self->{scheduler}->score_nodes_to_start_service(
+ $service_usage, $cpu_weight, $mem_weight,
+ );
};
if (my $err = $@) {
$self->{haenv}->log(
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 2f19c72..95163d4 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -53,7 +53,7 @@ info 25 node3/lrm: service status vm:107 started
info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912
info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192
info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011
-info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%)
+info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%)
info 160 node1/crm: got crm command: migrate vm:105 node2
info 160 node1/crm: migrate service 'vm:105' to node 'node2'
info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2)
@@ -68,7 +68,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8
info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201
info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211
info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191
-info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%)
+info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%)
info 260 node1/crm: got crm command: migrate vm:103 node3
info 260 node1/crm: migrate service 'vm:103' to node 'node3'
info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3)
@@ -77,13 +77,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
info 285 node3/lrm: starting service vm:103
info 285 node3/lrm: service status vm:103 started
-info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
-info 340 node1/crm: got crm command: migrate vm:105 node1
-info 340 node1/crm: migrate service 'vm:105' to node 'node1'
-info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
-info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
-info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
-info 361 node1/lrm: starting service vm:105
-info 361 node1/lrm: service status vm:105 started
info 820 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (9 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
` (3 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
This adds commands to change CRS settings or shutdown policy at test
runtime.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Sim/Hardware.pm | 62 ++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index e1a9ddd..e75229c 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -59,6 +59,7 @@ my $default_node_maxmem = 131072 * 1024**2;
# $testdir/status/service_config Service configuration
# $testdir/status/service_status_<node> Service status
# $testdir/status/groups HA groups configuration
+# $testdir/status/datacenter.cfg Datacenter wide HA configuration
sub read_lrm_status {
my ($self, $node) = @_;
@@ -699,6 +700,58 @@ sub write_datacenter_conf {
PVE::HA::Tools::write_json_to_file($filename, $conf);
}
+my $datacenter_settings = {
+ ha => {
+ shutdown_policy => qr/^(?:conditional|freeze|failover||migrate)$/,
+ },
+ crs => {
+ ha => qr/^(?:basic|static|dynamic)$/,
+ 'ha-rebalance-on-start' => qr/^[01]$/,
+ 'ha-auto-rebalance' => qr/^[01]$/,
+ 'ha-auto-rebalance-threshold' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-auto-rebalance-method' => qr/^(?:bruteforce|topsis)$/,
+ 'ha-auto-rebalance-hold-duration' => qr/^\d+$/,
+ 'ha-auto-rebalance-margin' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-auto-rebalance-cpu-weight' => qr/^\d+$/,
+ 'ha-auto-rebalance-mem-weight' => qr/^\d+$/,
+ },
+};
+
+sub update_datacenter_setting {
+ my ($self, $section, $action, $key, $value) = @_;
+
+ my $section_props = $datacenter_settings->{$section}
+ or die "sim_hardware_cmd: unknown datacenter section '$section'\n";
+
+ die "sim_hardware_cmd: missing property for 'datacenter $section $action' command\n"
+ if !defined($key);
+
+ my $verifier = $section_props->{$key}
+ or die "sim_hardware_cmd: unknown datacenter property '$key' in section '$section'\n";
+
+ my $conf = $self->read_datacenter_conf();
+
+ if ($action eq 'set') {
+ die "sim_hardware_cmd: missing value for 'datacenter $section set $key' command\n"
+ if !defined($value);
+ die "sim_hardware_cmd: invalid value '$value' for datacenter property '$section.$key'\n"
+ if $value !~ $verifier;
+
+ $value = $value + 0 if $value =~ m/^\d+(?:\.\d+)?$/;
+
+ $conf->{$section}->{$key} = $value;
+ } elsif ($action eq 'delete') {
+ delete $conf->{$section}->{$key} if $conf->{$section};
+ delete $conf->{$section} if $conf->{$section} && !%{ $conf->{$section} };
+ } else {
+ die "sim_hardware_cmd: unknown datacenter action '$action'\n";
+ }
+
+ $self->write_datacenter_conf($conf);
+
+ return $conf;
+}
+
sub global_lock {
my ($self, $code, @param) = @_;
@@ -802,6 +855,8 @@ sub get_cfs_state {
# service <sid> set-static-stats [maxcpu <cores>] [maxmem <MiB>]
# service <sid> set-dynamic-stats [cpu <cores>] [mem <MiB>]
# service <sid> delete
+# datacenter <ha|crs> set <property> <value>
+# datacenter <ha|crs> delete <property>
sub sim_hardware_cmd {
my ($self, $cmdstr, $logid) = @_;
@@ -820,6 +875,8 @@ sub sim_hardware_cmd {
if ($cmd eq 'service') {
$sid = PVE::HA::Tools::pve_verify_ha_resource_id($objid);
+ } elsif ($cmd eq 'datacenter') {
+ # cluster-wide, neither node nor service specific
} else {
$node = $objid;
$d = $self->{nodes}->{$node}
@@ -1028,6 +1085,11 @@ sub sim_hardware_cmd {
die "sim_hardware_cmd: unknown service action '$action' "
. "- not implemented\n";
}
+ } elsif ($cmd eq 'datacenter') {
+ die "sim_hardware_cmd: missing action for 'datacenter $objid' command\n"
+ if !defined($action);
+
+ $self->update_datacenter_setting($objid, $action, @params);
} else {
die "sim_hardware_cmd: unknown command '$cmdstr'\n";
}
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 12/15] manager: log changes to crs settings
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (10 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
` (2 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Especially with the addition of weights, it is important to be able to
trace how the load balancer is configured.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 89 +++++++++++++++++++++++++++++++++----------
1 file changed, 68 insertions(+), 21 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index c6818e2..4d0114c 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -99,6 +99,20 @@ sub new {
return $self;
}
+sub update_and_log_change {
+ my ($self, $curr, $next, $msg) = @_;
+
+ my $haenv = $self->{haenv};
+
+ if (defined($$curr) && $$curr ne $next) {
+ $haenv->log('info', "$msg from '$$curr' to '$next'");
+ }
+
+ $$curr = $next;
+
+ return;
+}
+
sub update_crs_scheduler_mode {
my ($self) = @_;
@@ -106,37 +120,70 @@ sub update_crs_scheduler_mode {
my $dc_cfg = $haenv->get_datacenter_settings();
my $crs_cfg = $dc_cfg->{crs} // {};
- $self->{crs}->{rebalance_on_request_start} = !!$crs_cfg->{'ha-rebalance-on-start'};
- $self->{crs}->{auto_rebalance}->{enable} = !!$crs_cfg->{'ha-auto-rebalance'};
+ my $log_prefix = "crs settings:";
+
+ $self->update_and_log_change(
+ \$self->{crs}->{rebalance_on_request_start},
+ !!$crs_cfg->{'ha-rebalance-on-start'},
+ "$log_prefix 'ha-rebalance-on-start' changed",
+ );
+
+ my $auto_rebalance = $self->{crs}->{auto_rebalance};
+
+ $self->update_and_log_change(
+ \$auto_rebalance->{enable},
+ !!$crs_cfg->{'ha-auto-rebalance'},
+ "$log_prefix 'ha-auto-rebalance' changed",
+ );
+
# configured as percent for UI/config consistency, scaled to [0, 1] to match the
# internal imbalance value that the comparison gates against.
- $self->{crs}->{auto_rebalance}->{threshold} =
- ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0;
- $self->{crs}->{auto_rebalance}->{method} = $crs_cfg->{'ha-auto-rebalance-method'}
- // 'bruteforce';
- $self->{crs}->{auto_rebalance}->{hold_duration} = $crs_cfg->{'ha-auto-rebalance-hold-duration'}
- // 3;
- $self->{crs}->{auto_rebalance}->{margin} =
- ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ $self->update_and_log_change(
+ \$auto_rebalance->{threshold},
+ ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0,
+ "$log_prefix 'ha-auto-rebalance-threshold' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{method},
+ $crs_cfg->{'ha-auto-rebalance-method'} // 'bruteforce',
+ "$log_prefix 'ha-auto-rebalance-method' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{hold_duration},
+ $crs_cfg->{'ha-auto-rebalance-hold-duration'} // 3,
+ "$log_prefix 'ha-auto-rebalance-hold-duration' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{margin},
+ ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0,
+ "$log_prefix 'ha-auto-rebalance-margin' changed",
+ );
my $default_weight = 1;
- $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
+ $self->update_and_log_change(
+ \$auto_rebalance->{cpu_weight},
+ $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight,
+ "$log_prefix 'ha-auto-rebalance-cpu-weight' changed",
+ );
# by default TOPSIS weights mem higher than cpu
- $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
- $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
+ $default_weight = 5 if ($auto_rebalance->{method} eq 'topsis');
+ $self->update_and_log_change(
+ \$auto_rebalance->{mem_weight},
+ $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight,
+ "$log_prefix 'ha-auto-rebalance-mem-weight' changed",
+ );
- my $old_mode = $self->{crs}->{scheduler};
+ my $old_mode = \$self->{crs}->{scheduler};
my $new_mode = $crs_cfg->{ha} || 'basic';
- if (!defined($old_mode)) {
+ if (!defined($$old_mode)) {
$haenv->log('info', "using scheduler mode '$new_mode'") if $new_mode ne 'basic';
- } elsif ($new_mode eq $old_mode) {
- return; # nothing to do
- } else {
- $haenv->log('info', "switching scheduler mode from '$old_mode' to '$new_mode'");
}
-
- $self->{crs}->{scheduler} = $new_mode;
+ $self->update_and_log_change(
+ $old_mode,
+ $new_mode,
+ "$log_prefix scheduler mode changed"
+ );
return;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (11 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight Dominik Rusovac
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add tests documenting basic efficacy of auto rebalancing with weights.
Overview:
- test-crs-dynamic-auto-rebalance-with-increasing-bias: no rebalancing,
increment mem-weight until rebalancing is triggered
- test-crs-dynamic-auto-rebalance-with-{cpu,mem}-bias: trigger migration
that rebalances {cpu,mem} by incrementing {cpu,mem}-weight once
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
.../README | 5 ++
.../cmdlist | 6 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 62 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 4 ++
.../cmdlist | 8 +++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 66 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 5 ++
.../cmdlist | 6 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 62 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
27 files changed, 332 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
new file mode 100644
index 0000000..f67978e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
@@ -0,0 +1,5 @@
+At equal weights (=1) load imbalance sits at 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing the
+ha-auto-rebalance-cpu-weight will increase load imbalance and trigger a
+rebalancing migration from cpu-heavy node3 to node2, which sits low on either
+resource.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
new file mode 100644
index 0000000..c29b4b6
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
new file mode 100644
index 0000000..352aa1c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
@@ -0,0 +1,62 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-cpu-weight' changed from '1' to '2'
+info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%)
+info 360 node1/crm: got crm command: migrate vm:106 node2
+info 360 node1/crm: migrate service 'vm:106' to node 'node2'
+info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 365 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 365 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:106
+info 383 node2/lrm: service status vm:106 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
new file mode 100644
index 0000000..a9974a7
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
@@ -0,0 +1,4 @@
+At equal weights (=1) load imbalance sits at 18.7%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that incrementing
+ha-auto-rebalance-mem-weight will eventually trigger a rebalancing migration
+from mem-heavy node3.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
new file mode 100644
index 0000000..643ab0b
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
@@ -0,0 +1,8 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 3" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 4" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
new file mode 100644
index 0000000..14740f1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 10.0, "mem": 3221225472 },
+ "vm:102": { "cpu": 5.0, "mem": 1073741824 },
+ "vm:103": { "cpu": 5.0, "mem": 3221225472 },
+ "vm:104": { "cpu": 3.0, "mem": 1073741824 },
+ "vm:105": { "cpu": 0.5, "mem": 32212254720 },
+ "vm:106": { "cpu": 0.5, "mem": 6442450944 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
new file mode 100644
index 0000000..596ef29
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
@@ -0,0 +1,66 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
+info 420 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 3
+info 420 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '2' to '3'
+info 520 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 4
+info 520 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '3' to '4'
+info 560 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%)
+info 560 node1/crm: got crm command: migrate vm:106 node2
+info 560 node1/crm: migrate service 'vm:106' to node 'node2'
+info 560 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 565 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 565 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 580 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 583 node2/lrm: starting service vm:106
+info 583 node2/lrm: service status vm:106 started
+info 1120 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
new file mode 100644
index 0000000..19cebf9
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
@@ -0,0 +1,5 @@
+At equal weights (=1) load imbalance sits at 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing the
+ha-auto-rebalance-mem-weight will increase load imbalance and trigger a
+rebalancing migration from mem-heavy node1 to node2, which sits low on either
+resource.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
new file mode 100644
index 0000000..2144ffb
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
new file mode 100644
index 0000000..5fc901f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
@@ -0,0 +1,62 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
+info 360 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%)
+info 360 node1/crm: got crm command: migrate vm:102 node2
+info 360 node1/crm: migrate service 'vm:102' to node 'node2'
+info 360 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 361 node1/lrm: service vm:102 - start migrate to node 'node2'
+info 361 node1/lrm: service vm:102 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:102
+info 383 node2/lrm: service status vm:102 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (12 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight Dominik Rusovac
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Rename to 'update_crs_settings' to be more precise and avoid confusion
with CRS 'Scheduling Mode'.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 4d0114c..78d4952 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -94,7 +94,7 @@ sub new {
# preserve disarm state across CRM master changes
$self->{ms}->{disarm} = $old_ms->{disarm} if defined($old_ms->{disarm});
- $self->update_crs_scheduler_mode(); # initial set, we update it once every loop
+ $self->update_crs_settings(); # initial set, we update it once every loop
return $self;
}
@@ -113,7 +113,7 @@ sub update_and_log_change {
return;
}
-sub update_crs_scheduler_mode {
+sub update_crs_settings {
my ($self) = @_;
my $haenv = $self->{haenv};
@@ -1108,7 +1108,7 @@ sub manage {
return;
}
- $self->update_crs_scheduler_mode();
+ $self->update_crs_settings();
$self->try_persistent_group_migration();
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread* [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (13 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
14 siblings, 0 replies; 17+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add weights for controlling how much balancing out the CPU load,
respectively, memory usage, of nodes should be prioritized.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/DataCenterConfig.pm | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 004122e..19dec7c 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -78,6 +78,22 @@ EODESC
description => "The minimum relative improvement in cluster node imbalance,"
. " in percent, to commit to a resource balancing migration.",
},
+ 'ha-auto-rebalance-cpu-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ requires => 'ha-auto-rebalance',
+ description => "The weight controlling how much balancing out"
+ . " the CPU load of nodes should be prioritized.",
+ },
+ 'ha-auto-rebalance-mem-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ requires => 'ha-auto-rebalance',
+ description => "The weight controlling how much balancing out"
+ . " the memory usage of nodes should be prioritized",
+ },
};
my $migration_format = {
--
2.47.3
^ permalink raw reply related [flat|nested] 17+ messages in thread