* [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
` (13 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 2 +-
proxmox-resource-scheduling/src/topsis.rs | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index eb5c6142..9078fa72 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -166,7 +166,7 @@ impl Scheduler {
}
}
- /// Map the current node usages to a [`PveTopsisAlternative`].
+ /// Maps the current node usages to a [`PveTopsisAlternative`].
///
/// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
/// square (RMS) and maximum value of each stat in the node usages.
diff --git a/proxmox-resource-scheduling/src/topsis.rs b/proxmox-resource-scheduling/src/topsis.rs
index 3d42ff30..89f4ea40 100644
--- a/proxmox-resource-scheduling/src/topsis.rs
+++ b/proxmox-resource-scheduling/src/topsis.rs
@@ -161,7 +161,7 @@ impl<const N: usize> IdealAlternatives<N> {
}
/// Scores the alternatives in `matrix` according to their similarity to the ideal worst
-/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closness to the ideal worst
+/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closeness to the ideal worst
/// and/or distance to the ideal best alternatives.
pub fn score_alternatives<const N: usize>(
matrix: &Matrix<N>,
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 12:49 ` Daniel Kral
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
` (12 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
DRY and hide repeatedly used closure inside reuasble function.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 41 ++++++++------------
1 file changed, 16 insertions(+), 25 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 9078fa72..a12cd519 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -56,6 +56,19 @@ fn calculate_node_imbalance(nodes: &[NodeUsage], to_load: impl Fn(&NodeUsage) ->
cv / max_cv
}
+/// Returns the hypothetical [`NodeStats`] of `node` with respect to the migration `candidate`.
+fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage) -> NodeStats {
+ let mut new_stats = node.stats;
+
+ if node.name == candidate.migration.source_node {
+ new_stats.remove_running_resource(&candidate.stats);
+ } else if node.name == candidate.migration.target_node {
+ new_stats.add_running_resource(&candidate.stats);
+ }
+
+ new_stats
+}
+
criteria_struct! {
/// A given alternative.
struct PveTopsisAlternative {
@@ -216,15 +229,7 @@ impl Scheduler {
/// See [`calculate_node_imbalance`] for more information.
fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- let mut new_stats = node.stats;
-
- if node.name == candidate.migration.source_node {
- new_stats.remove_running_resource(&candidate.stats);
- } else if node.name == candidate.migration.target_node {
- new_stats.add_running_resource(&candidate.stats);
- }
-
- new_stats.load()
+ node_stats_after_migration(candidate, node).load()
})
}
@@ -288,22 +293,8 @@ impl Scheduler {
let matrix = candidates
.iter()
.map(|candidate| {
- let resource_stats = &candidate.stats;
- let source_node = &candidate.migration.source_node;
- let target_node = &candidate.migration.target_node;
-
- self.topsis_alternative_with(|node| {
- let mut new_stats = node.stats;
-
- if &node.name == source_node {
- new_stats.remove_running_resource(resource_stats);
- } else if &node.name == target_node {
- new_stats.add_running_resource(resource_stats);
- }
-
- new_stats
- })
- .into()
+ self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
+ .into()
})
.collect::<Vec<_>>();
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-10-21 12:49 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-21 12:49 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Nice!
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> DRY and hide repeatedly used closure inside reuasble function.
nit: s/reuasble/reusable/
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> proxmox-resource-scheduling/src/scheduler.rs | 41 ++++++++------------
> 1 file changed, 16 insertions(+), 25 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index 9078fa72..a12cd519 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -56,6 +56,19 @@ fn calculate_node_imbalance(nodes: &[NodeUsage], to_load: impl Fn(&NodeUsage) ->
> cv / max_cv
> }
>
> +/// Returns the hypothetical [`NodeStats`] of `node` with respect to the migration `candidate`.
> +fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage) -> NodeStats {
> + let mut new_stats = node.stats;
> +
> + if node.name == candidate.migration.source_node {
> + new_stats.remove_running_resource(&candidate.stats);
> + } else if node.name == candidate.migration.target_node {
> + new_stats.add_running_resource(&candidate.stats);
> + }
> +
> + new_stats
> +}
> +
> criteria_struct! {
> /// A given alternative.
> struct PveTopsisAlternative {
> @@ -216,15 +229,7 @@ impl Scheduler {
> /// See [`calculate_node_imbalance`] for more information.
> fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - let mut new_stats = node.stats;
> -
> - if node.name == candidate.migration.source_node {
> - new_stats.remove_running_resource(&candidate.stats);
> - } else if node.name == candidate.migration.target_node {
> - new_stats.add_running_resource(&candidate.stats);
> - }
> -
> - new_stats.load()
> + node_stats_after_migration(candidate, node).load()
> })
nit: if it fits on a line, could drop the braces {} here
> }
>
> @@ -288,22 +293,8 @@ impl Scheduler {
> let matrix = candidates
> .iter()
> .map(|candidate| {
> - let resource_stats = &candidate.stats;
> - let source_node = &candidate.migration.source_node;
> - let target_node = &candidate.migration.target_node;
> -
> - self.topsis_alternative_with(|node| {
> - let mut new_stats = node.stats;
> -
> - if &node.name == source_node {
> - new_stats.remove_running_resource(resource_stats);
> - } else if &node.name == target_node {
> - new_stats.add_running_resource(resource_stats);
> - }
> -
> - new_stats
> - })
> - .into()
> + self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
> + .into()
> })
> .collect::<Vec<_>>();
>
Otherwise, consider as:
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 9:15 ` Daniel Kral
2026-10-21 8:59 ` [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate Dominik Rusovac
` (11 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Blend weights into node load computation, effectively replacing
arithmetic mean [0] of the node's CPU load and memory usage with
weighted arithmetic mean [1] thereof.
Expose customizable weights for bruteforce method. For TOPSIS method
expose customizable weights for average cpu and average memory only. The
weights of highest cpu and highest memory are not independently
configurable; each is derived as twice as much as its average
counterpart, that is, the weight of highest cpu/memory is twice as much
as the weight of average cpu/memory. This makes it possible to
prioritize balancing a specific computational resource while also
preserving the ratio between a resource's two criteria, as introduced in
[2].
Apart from the achieved imbalance, expose the score of a scored
migration. Moreover, primarily order scored migrations by their score
(lower is better), falling back to imbalance when scores tie, and
finally to the migration itself if both tie -- ensuring a total,
deterministic order even when several candidates achieve the exact same
score. Add basic unit test for this.
Refactor existing tests.
[0] https://en.wikipedia.org/wiki/Arithmetic_mean
[1] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
[2] https://lore.proxmox.com/all/20221110143800.98047-3-f.ebner@proxmox.com/
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/node.rs | 27 +-
proxmox-resource-scheduling/src/pve_static.rs | 3 +-
proxmox-resource-scheduling/src/scheduler.rs | 248 +++++++++++++-----
.../tests/scheduler.rs | 136 ++++++++--
4 files changed, 327 insertions(+), 87 deletions(-)
diff --git a/proxmox-resource-scheduling/src/node.rs b/proxmox-resource-scheduling/src/node.rs
index 3c7ed393..f9dc5b16 100644
--- a/proxmox-resource-scheduling/src/node.rs
+++ b/proxmox-resource-scheduling/src/node.rs
@@ -51,9 +51,30 @@ impl NodeStats {
self.mem as f64 / self.maxmem as f64
}
- /// Returns a combined node usage as a percentage.
- pub fn load(&self) -> f64 {
- (self.cpu_load() + self.mem_load()) / 2.0
+ /// Returns the node load as the [weighted arithmetic mean] of [`Self::cpu_load`] and
+ /// [`Self::mem_load`].
+ ///
+ /// `cpu_weight` controls how much [`Self::cpu_load`] contributes to the load, analogously,
+ /// `mem_weight` controls how much [`Self::mem_load`] contributes to the load; the higher the
+ /// weight, the more that resource dominates the result.
+ ///
+ /// NOTE:
+ /// - the weights have to be positive
+ /// - only the weights' ratio matters, not their absolute magnitude
+ /// - a weight of 0.0 means no contribution to the load
+ /// - [weighted arithmetic mean] is the same as [arithmetic mean] if the weights are equal
+ ///
+ /// [weighted arithmetic mean]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
+ /// [arithmetic mean]: https://en.wikipedia.org/wiki/Arithmetic_mean
+ pub fn load(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
+ let weights_sum = cpu_weight + mem_weight;
+
+ if weights_sum == 0.0 {
+ // both, cpu and mem, should not contribute to the load, so there is no load
+ return 0.0;
+ }
+
+ (self.cpu_load() * cpu_weight + self.mem_load() * mem_weight) / weights_sum
}
}
diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
index 40b76cbc..98d64907 100644
--- a/proxmox-resource-scheduling/src/pve_static.rs
+++ b/proxmox-resource-scheduling/src/pve_static.rs
@@ -96,5 +96,6 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
let scheduler = Scheduler::from_nodes(nodes);
- scheduler.score_nodes_to_start_resource(*service)
+ // uses original empirically determined TOPSIS weights
+ scheduler.score_nodes_to_start_resource(*service, 1, 5)
}
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index a12cd519..0690385d 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -8,6 +8,9 @@ use std::{
collections::BinaryHeap,
};
+/// Constant to control the ratio of the weights of a resource's average and highest criteria.
+pub const TOPSIS_AVERAGE_VS_HIGHEST_FACTOR: f64 = 2.0;
+
/// The scheduler view of a node.
#[derive(Clone, Debug)]
pub struct NodeUsage {
@@ -69,23 +72,6 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
new_stats
}
-criteria_struct! {
- /// A given alternative.
- struct PveTopsisAlternative {
- #[criterion("average CPU", -1.0)]
- average_cpu: f64,
- #[criterion("highest CPU", -2.0)]
- highest_cpu: f64,
- #[criterion("average memory", -5.0)]
- average_memory: f64,
- #[criterion("highest memory", -10.0)]
- highest_memory: f64,
- }
-
- const N_CRITERIA;
- static PVE_HA_TOPSIS_CRITERIA;
-}
-
#[derive(Clone, Debug)]
pub struct Scheduler {
nodes: Vec<NodeUsage>,
@@ -111,12 +97,20 @@ pub struct ScoredMigration {
pub migration: Migration,
/// The expected node imbalance after the migration.
pub imbalance: f64,
+ /// The score.
+ ///
+ /// - corresponds to imbalance of weighted loads for
+ /// [`Scheduler::score_best_balancing_migration_candidates`]
+ /// - corresponds to distance to ideal best alternatives for
+ /// [`Scheduler::score_best_balancing_migration_candidates_topsis`]
+ pub score: f64,
}
impl Ord for ScoredMigration {
fn cmp(&self, other: &Self) -> Ordering {
- self.imbalance
- .total_cmp(&other.imbalance)
+ self.score
+ .total_cmp(&other.score)
+ .then(self.imbalance.total_cmp(&other.imbalance))
.then(self.migration.cmp(&other.migration))
}
}
@@ -136,19 +130,21 @@ impl PartialEq for ScoredMigration {
impl Eq for ScoredMigration {}
impl ScoredMigration {
- pub fn new<T: Into<Migration>>(migration: T, imbalance: f64) -> Self {
- // Depending on how the imbalance is calculated, it can contain minor approximation errors.
- // As this struct implements the Ord trait, users of the struct's cmp() can run into cases,
- // where the imbalance is the same up to the significant digits in base 10, but treated as
- // different values.
+ pub fn new<T: Into<Migration>>(migration: T, imbalance: f64, score: f64) -> Self {
+ // Depending on how the imbalance or score is calculated, it can contain minor approximation
+ // errors. As this struct implements the Ord trait, users of the struct's cmp() can run into
+ // cases, where the imbalance or score is the same up to the significant digits in base 10,
+ // but treated as different values.
//
// Therefore, truncate any non-significant digits to prevent these cases.
let factor = 10_f64.powi(f64::DIGITS as i32);
let truncated_imbalance = f64::trunc(factor * imbalance) / factor;
+ let truncated_score = f64::trunc(factor * score) / factor;
Self {
migration: migration.into(),
imbalance: truncated_imbalance,
+ score: truncated_score,
}
}
}
@@ -179,15 +175,15 @@ impl Scheduler {
}
}
- /// Maps the current node usages to a [`PveTopsisAlternative`].
+ /// Maps the current node usages to a TOPSIS alternative.
///
- /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
- /// square (RMS) and maximum value of each stat in the node usages.
+ /// The alternative is derived by calculating a modified version of the root mean square (RMS)
+ /// and maximum value of each stat in the node usages.
fn topsis_alternative_with(
&self,
map_node_stats: impl Fn(&NodeUsage) -> NodeStats,
- ) -> PveTopsisAlternative {
- let len = self.nodes.len();
+ ) -> [f64; 4] {
+ let len = self.nodes.len() as f64;
// Base values on percentages to allow comparing nodes with different stats.
let mut highest_cpu = 0.0;
@@ -209,55 +205,79 @@ impl Scheduler {
// Add 1.0 to avoid boosting tiny differences: e.g. 0.004 is twice as much as 0.002, but
// 1.004 is only slightly more than 1.002.
- PveTopsisAlternative {
- average_cpu: 1.0 + (squares_cpu / len as f64).sqrt(),
- highest_cpu: 1.0 + highest_cpu,
- average_memory: 1.0 + (squares_mem / len as f64).sqrt(),
- highest_memory: 1.0 + highest_mem,
- }
+ [
+ 1.0 + (squares_cpu / len).sqrt(),
+ 1.0 + highest_cpu,
+ 1.0 + (squares_mem / len).sqrt(),
+ 1.0 + highest_mem,
+ ]
}
/// Returns the load imbalance among the nodes.
///
+ /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
+ /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
+ /// [`NodeStats::load`].
+ ///
/// See [`calculate_node_imbalance`] for more information.
- pub fn node_imbalance(&self) -> f64 {
- calculate_node_imbalance(&self.nodes, |node| node.stats.load())
+ pub fn node_imbalance(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
+ calculate_node_imbalance(&self.nodes, |node| node.stats.load(cpu_weight, mem_weight))
}
/// Returns the load imbalance among the nodes as if a specific resource was moved.
///
+ /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
+ /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
+ /// [`NodeStats::load`].
+ ///
/// See [`calculate_node_imbalance`] for more information.
- fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
+ fn node_imbalance_with_migration_candidate(
+ &self,
+ candidate: &MigrationCandidate,
+ cpu_weight: f64,
+ mem_weight: f64,
+ ) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- node_stats_after_migration(candidate, node).load()
+ node_stats_after_migration(candidate, node).load(cpu_weight, mem_weight)
})
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with
- /// exhaustive search.
+ /// Uses brute-force search to score how much either of the given migration `candidates`
+ /// improves node imbalance and ranks them accordingly.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
///
/// The scoring is done as if each resource migration has already been done. This assumes that
/// the already migrated resource consumes the same amount of each stat as on the previous node
- /// according to its `stats`.
+ /// according to its `stats`. The lower the score, the better.
+ ///
+ /// `cpu_weight` and `mem_weight` are blended into [`NodeStats::load`]. The higher the weight of
+ /// a computational resource, the more the search will tend to prioritize balancing out that
+ /// resource, e.g., a high `mem_weight` will steer migrations toward evening out memory usage,
+ /// even at the cost of leaving CPU load less balanced. This is a general tendency, not a
+ /// guarantee.
///
/// Returns up to `limit` of the best scored migrations.
pub fn score_best_balancing_migration_candidates<I>(
&self,
candidates: I,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Vec<ScoredMigration>
where
I: IntoIterator<Item = MigrationCandidate>,
{
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
let mut scored_migrations = candidates
.into_iter()
.map(|candidate| {
- let imbalance = self.node_imbalance_with_migration_candidate(&candidate);
+ let imbalance = self.node_imbalance_with_migration_candidate(&candidate, 1.0, 1.0);
+ let score = self.node_imbalance_with_migration_candidate(&candidate, w1, w2);
- Reverse(ScoredMigration::new(candidate, imbalance))
+ Reverse(ScoredMigration::new(candidate, imbalance, score))
})
.collect::<BinaryHeap<_>>();
@@ -274,56 +294,106 @@ impl Scheduler {
best_migrations
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with the
- /// TOPSIS method.
+ /// Scores the given migration `candidates` using the TOPSIS method.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
///
/// The scoring is done as if each resource migration has already been done. This assumes that
/// the already migrated resource consumes the same amount of each stat as on the previous node
- /// according to its `stats`.
+ /// according to its `stats`. The lower the score, the better.
+ ///
+ /// The weights are assigned to the respective criteria of the TOPSIS method.
///
/// Returns up to `limit` of the best scored migrations.
pub fn score_best_balancing_migration_candidates_topsis(
&self,
candidates: &[MigrationCandidate],
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
+ let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
+ let highest_cpu = topsis::Criterion::new(
+ "highest_cpu".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
+ );
+ let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
+ let highest_mem = topsis::Criterion::new(
+ "highest_mem".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
+ );
+
let matrix = candidates
.iter()
.map(|candidate| {
self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
- .into()
})
.collect::<Vec<_>>();
- let best_alternatives =
- topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ let mut scored_migrations = topsis::score_alternatives(
+ &topsis::Matrix::new(matrix)?,
+ &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
+ )?
+ .iter()
+ .zip(candidates.iter())
+ .map(|(topsis_score, candidate)| {
+ let imbalance = self.node_imbalance_with_migration_candidate(candidate, w1, w2);
+
+ Reverse(ScoredMigration::new(
+ candidate.clone(),
+ imbalance,
+ // invert score for consistent sorting
+ 1.0 - topsis_score,
+ ))
+ })
+ .collect::<BinaryHeap<_>>();
- Ok(best_alternatives
- .into_iter()
- .take(limit)
- .map(|i| {
- let imbalance = self.node_imbalance_with_migration_candidate(&candidates[i]);
+ let mut best_migrations = Vec::with_capacity(limit);
- ScoredMigration::new(candidates[i].clone(), imbalance)
- })
- .collect())
+ // BinaryHeap::into_iter_sorted() is still in nightly unfortunately
+ while best_migrations.len() < limit {
+ match scored_migrations.pop() {
+ Some(Reverse(alternative)) => best_migrations.push(alternative),
+ None => break,
+ }
+ }
+
+ Ok(best_migrations)
}
- /// Scores nodes to start a resource with the usage statistics `resource_stats` on.
+ /// Scores nodes to start a resource with the usage statistics `resource_stats` on, using the
+ /// TOPSIS method.
///
/// The scoring is done as if the resource is already started on each node. This assumes that
/// the already started resource consumes the maximum amount of each stat according to its
/// `resource_stats`.
///
+ /// The weights are assigned to the respective criteria of the TOPSIS method.
+ ///
/// Returns a vector of (nodename, score) pairs. Scores are between 0.0 and 1.0 and a higher
/// score is better.
pub fn score_nodes_to_start_resource<T: Into<ResourceStats>>(
&self,
resource_stats: T,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
+ let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
+
+ let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
+ let highest_cpu = topsis::Criterion::new(
+ "highest_cpu".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
+ );
+ let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
+ let highest_mem = topsis::Criterion::new(
+ "highest_mem".to_owned(),
+ -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
+ );
+
let resource_stats = resource_stats.into();
let matrix = self
@@ -339,12 +409,13 @@ impl Scheduler {
new_stats
})
- .into()
})
.collect::<Vec<_>>();
- let scores =
- topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ let scores = topsis::score_alternatives(
+ &topsis::Matrix::new(matrix)?,
+ &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
+ )?;
Ok(scores
.into_iter()
@@ -367,6 +438,7 @@ mod tests {
target_node: String::from("node2"),
},
0.7231749488916931,
+ 0.7231749488916931,
);
let migration2 = ScoredMigration::new(
Migration {
@@ -375,6 +447,7 @@ mod tests {
target_node: String::from("node3"),
},
0.723174948891693,
+ 0.723174948891693,
);
let migration3 = ScoredMigration::new(
Migration {
@@ -383,6 +456,57 @@ mod tests {
target_node: String::from("node2"),
},
0.723174948891693 + 1e-15,
+ 0.723174948891693 + 1e-15,
+ );
+
+ let mut migrations = vec![migration2.clone(), migration3.clone(), migration1.clone()];
+
+ migrations.sort();
+
+ assert_eq!(
+ vec![migration1.clone(), migration2.clone(), migration3.clone()],
+ migrations
+ );
+
+ let mut heap = BinaryHeap::from(vec![
+ Reverse(migration2.clone()),
+ Reverse(migration3.clone()),
+ Reverse(migration1.clone()),
+ ]);
+
+ assert_eq!(heap.pop(), Some(Reverse(migration1)));
+ assert_eq!(heap.pop(), Some(Reverse(migration2)));
+ assert_eq!(heap.pop(), Some(Reverse(migration3)));
+ }
+
+ #[test]
+ fn test_score_tie_breaks_on_imbalance() {
+ let migration1 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:102"),
+ source_node: String::from("node1"),
+ target_node: String::from("node2"),
+ },
+ 0.0,
+ 0.1,
+ );
+ let migration2 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:102"),
+ source_node: String::from("node1"),
+ target_node: String::from("node3"),
+ },
+ 0.0,
+ 0.2,
+ );
+ let migration3 = ScoredMigration::new(
+ Migration {
+ sid: String::from("vm:101"),
+ source_node: String::from("node1"),
+ target_node: String::from("node2"),
+ },
+ 0.3,
+ 0.3,
);
let mut migrations = vec![migration2.clone(), migration3.clone(), migration1.clone()];
diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
index 61fdd840..19e2c353 100644
--- a/proxmox-resource-scheduling/tests/scheduler.rs
+++ b/proxmox-resource-scheduling/tests/scheduler.rs
@@ -5,6 +5,11 @@ use proxmox_resource_scheduling::{
scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
};
+const DEFAULT_CPU_WEIGHT: usize = 1;
+const DEFAULT_MEM_WEIGHT: usize = 1;
+
+const TOPSIS_CPU_VS_MEM_FACTOR: usize = 5;
+
fn new_empty_cluster_scheduler() -> Scheduler {
Scheduler::from_nodes(Vec::<NodeUsage>::new())
}
@@ -83,7 +88,10 @@ fn new_heterogeneous_cluster_scheduler() -> Scheduler {
fn test_node_imbalance_with_empty_cluster() {
let scheduler = new_empty_cluster_scheduler();
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
}
#[test]
@@ -100,11 +108,17 @@ fn test_node_imbalance_with_perfectly_balanced_cluster() {
let scheduler = Scheduler::from_nodes(vec![node.clone()]);
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
let scheduler = Scheduler::from_nodes(vec![node.clone(), node.clone(), node]);
- assert_eq!(scheduler.node_imbalance(), 0.0);
+ assert_eq!(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.0
+ );
}
fn new_simple_migration_candidates() -> (Vec<MigrationCandidate>, Migration, Migration) {
@@ -150,9 +164,11 @@ fn rank_best_balancing_migration_candidates(
scheduler: &Scheduler,
candidates: Vec<MigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Vec<Migration> {
scheduler
- .score_best_balancing_migration_candidates(candidates, limit)
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight)
.into_iter()
.map(|entry| entry.migration)
.collect()
@@ -163,7 +179,13 @@ fn test_score_best_balancing_migration_candidates_with_no_candidates() {
let scheduler = new_homogeneous_cluster_scheduler();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, vec![], 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ vec![],
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![]
);
}
@@ -172,12 +194,21 @@ fn test_score_best_balancing_migration_candidates_with_no_candidates() {
fn test_score_best_balancing_migration_candidates_in_homogeneous_cluster() {
let scheduler = new_homogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.3460548572604576);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.3460548572604576,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, candidates, 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ candidates,
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![migration2, migration1]
);
}
@@ -186,12 +217,21 @@ fn test_score_best_balancing_migration_candidates_in_homogeneous_cluster() {
fn test_score_best_balancing_migration_candidates_in_heterogeneous_cluster() {
let scheduler = new_heterogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.23352917788066363);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.23352917788066363,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
assert_eq!(
- rank_best_balancing_migration_candidates(&scheduler, candidates, 2),
+ rank_best_balancing_migration_candidates(
+ &scheduler,
+ candidates,
+ 2,
+ DEFAULT_CPU_WEIGHT,
+ DEFAULT_MEM_WEIGHT
+ ),
vec![migration2, migration1]
);
}
@@ -202,7 +242,12 @@ fn rank_best_balancing_migration_candidates_topsis(
limit: usize,
) -> Result<Vec<Migration>, Error> {
Ok(scheduler
- .score_best_balancing_migration_candidates_topsis(candidates, limit)?
+ .score_best_balancing_migration_candidates_topsis(
+ candidates,
+ limit,
+ DEFAULT_CPU_WEIGHT,
+ 2 * DEFAULT_MEM_WEIGHT,
+ )?
.into_iter()
.map(|entry| entry.migration)
.collect())
@@ -225,7 +270,10 @@ fn test_score_best_balancing_migration_candidates_topsis_in_homogeneous_cluster(
-> Result<(), Error> {
let scheduler = new_homogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.3460548572604576);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.3460548572604576,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
@@ -242,7 +290,10 @@ fn test_score_best_balancing_migration_candidates_topsis_in_heterogeneous_cluste
-> Result<(), Error> {
let scheduler = new_heterogeneous_cluster_scheduler();
- assert_imbalance(scheduler.node_imbalance(), 0.23352917788066363);
+ assert_imbalance(
+ scheduler.node_imbalance(DEFAULT_CPU_WEIGHT as f64, DEFAULT_MEM_WEIGHT as f64),
+ 0.23352917788066363,
+ );
let (candidates, migration1, migration2) = new_simple_migration_candidates();
@@ -257,8 +308,11 @@ fn test_score_best_balancing_migration_candidates_topsis_in_heterogeneous_cluste
fn rank_nodes_to_start_resource(
scheduler: &Scheduler,
resource_stats: ResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<String>, Error> {
- let mut alternatives = scheduler.score_nodes_to_start_resource(resource_stats)?;
+ let mut alternatives =
+ scheduler.score_nodes_to_start_resource(resource_stats, cpu_weight, mem_weight)?;
alternatives.sort_by(|a, b| b.1.total_cmp(&a.1));
@@ -280,7 +334,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_memory_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_memory_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node3", "node1"]
);
@@ -292,7 +351,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -304,7 +368,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, unlimited_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ unlimited_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -316,7 +385,12 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ combined_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node3", "node1"]
);
@@ -335,7 +409,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_memory_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_memory_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node1", "node3"]
);
@@ -347,7 +426,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, heavy_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ heavy_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node3", "node2", "node1"]
);
@@ -359,7 +443,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, unlimited_cpu_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ unlimited_cpu_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node1", "node3", "node2"]
);
@@ -371,7 +460,12 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
};
assert_eq!(
- rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
+ rank_nodes_to_start_resource(
+ &scheduler,
+ combined_resource_stats,
+ DEFAULT_CPU_WEIGHT,
+ TOPSIS_CPU_VS_MEM_FACTOR * DEFAULT_MEM_WEIGHT
+ )?,
vec!["node2", "node1", "node3"]
);
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
@ 2026-10-22 9:15 ` Daniel Kral
2026-10-22 11:34 ` Daniel Kral
0 siblings, 1 reply; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 9:15 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Thanks a lot for working on this, nice work! Especially the cover letter
is a very interesting analysis and read!
Looks generally sound, though there are some comments inline on having
backwards and forwards compatibility and some patch cleanup.
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Blend weights into node load computation, effectively replacing
> arithmetic mean [0] of the node's CPU load and memory usage with
> weighted arithmetic mean [1] thereof.
>
> Expose customizable weights for bruteforce method. For TOPSIS method
> expose customizable weights for average cpu and average memory only. The
> weights of highest cpu and highest memory are not independently
> configurable; each is derived as twice as much as its average
> counterpart, that is, the weight of highest cpu/memory is twice as much
> as the weight of average cpu/memory. This makes it possible to
> prioritize balancing a specific computational resource while also
> preserving the ratio between a resource's two criteria, as introduced in
> [2].
>
> Apart from the achieved imbalance, expose the score of a scored
> migration. Moreover, primarily order scored migrations by their score
> (lower is better), falling back to imbalance when scores tie, and
> finally to the migration itself if both tie -- ensuring a total,
> deterministic order even when several candidates achieve the exact same
> score. Add basic unit test for this.
>
> Refactor existing tests.
>
> [0] https://en.wikipedia.org/wiki/Arithmetic_mean
> [1] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
> [2] https://lore.proxmox.com/all/20221110143800.98047-3-f.ebner@proxmox.com/
Changes which were applied should be referenced as commit refs rather
than the patch on the mailing list, i.e. here
commit 2b325aa5 ("add pve_static module")
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> proxmox-resource-scheduling/src/node.rs | 27 +-
> proxmox-resource-scheduling/src/pve_static.rs | 3 +-
> proxmox-resource-scheduling/src/scheduler.rs | 248 +++++++++++++-----
> .../tests/scheduler.rs | 136 ++++++++--
> 4 files changed, 327 insertions(+), 87 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/node.rs b/proxmox-resource-scheduling/src/node.rs
> index 3c7ed393..f9dc5b16 100644
> --- a/proxmox-resource-scheduling/src/node.rs
> +++ b/proxmox-resource-scheduling/src/node.rs
> @@ -51,9 +51,30 @@ impl NodeStats {
> self.mem as f64 / self.maxmem as f64
> }
>
> - /// Returns a combined node usage as a percentage.
> - pub fn load(&self) -> f64 {
> - (self.cpu_load() + self.mem_load()) / 2.0
> + /// Returns the node load as the [weighted arithmetic mean] of [`Self::cpu_load`] and
> + /// [`Self::mem_load`].
> + ///
> + /// `cpu_weight` controls how much [`Self::cpu_load`] contributes to the load, analogously,
> + /// `mem_weight` controls how much [`Self::mem_load`] contributes to the load; the higher the
> + /// weight, the more that resource dominates the result.
nit: can be documented in struct NodeStatsWeights, see below for more
> + ///
> + /// NOTE:
> + /// - the weights have to be positive
> + /// - only the weights' ratio matters, not their absolute magnitude
> + /// - a weight of 0.0 means no contribution to the load
> + /// - [weighted arithmetic mean] is the same as [arithmetic mean] if the weights are equal
> + ///
> + /// [weighted arithmetic mean]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
> + /// [arithmetic mean]: https://en.wikipedia.org/wiki/Arithmetic_mean
> + pub fn load(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
> + let weights_sum = cpu_weight + mem_weight;
> +
> + if weights_sum == 0.0 {
> + // both, cpu and mem, should not contribute to the load, so there is no load
> + return 0.0;
> + }
> +
> + (self.cpu_load() * cpu_weight + self.mem_load() * mem_weight) / weights_sum
> }
Would be nice to have a separate weighted_load() next to the existing
load() for backwards compatibility.
And for some type of forwards compatibility, maybe something like
#[derive(Copy, Clone, Debug)]
pub struct NodeStatsWeights {
pub cpu_weight: f64,
pub mem_weight: f64,
}
impl Default for NodeStatsWeights {
fn default() -> Self {
Self {
cpu_weight: 1,
mem_weight: 1,
}
}
}
pub fn weighted_load(&self, weights: &NodeStatsWeights) -> f64 {}
?
The NodeStatsWeights struct can then be used in all callers of
weighted_load() and therefore the members and their documentation do not
need to be repeated there.
Though we'd probably need a default for TOPSIS as well, because there it
should be 1 : 5 with cpu : mem for backwards compatibility...
On another note, adding a new field to NodeStatsWeights then would
unfortunately break external callers, except if every external caller
allocates it with:
let weights = NodeStatsWeights {
cpu_weight: 2,
mem_weight: 3,
..Default::default()
}
#[non_exhaustive] would be nice here, but unfortunately that prevents
external callers to use the literal struct syntax with functional update
syntax as above [0] and we would need to build a proper interface to set
the weights with methods... We can also do that, but adds some code just
for that.
[0] https://doc.rust-lang.org/reference/attributes/type_system.html#r-attributes.type-system.non_exhaustive.construction
> }
>
> diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
> index 40b76cbc..98d64907 100644
> --- a/proxmox-resource-scheduling/src/pve_static.rs
> +++ b/proxmox-resource-scheduling/src/pve_static.rs
> @@ -96,5 +96,6 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
>
> let scheduler = Scheduler::from_nodes(nodes);
>
> - scheduler.score_nodes_to_start_resource(*service)
> + // uses original empirically determined TOPSIS weights
> + scheduler.score_nodes_to_start_resource(*service, 1, 5)
> }
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index a12cd519..0690385d 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -8,6 +8,9 @@ use std::{
> collections::BinaryHeap,
> };
>
> +/// Constant to control the ratio of the weights of a resource's average and highest criteria.
> +pub const TOPSIS_AVERAGE_VS_HIGHEST_FACTOR: f64 = 2.0;
> +
> /// The scheduler view of a node.
> #[derive(Clone, Debug)]
> pub struct NodeUsage {
> @@ -69,23 +72,6 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
> new_stats
> }
>
> -criteria_struct! {
> - /// A given alternative.
> - struct PveTopsisAlternative {
> - #[criterion("average CPU", -1.0)]
> - average_cpu: f64,
> - #[criterion("highest CPU", -2.0)]
> - highest_cpu: f64,
> - #[criterion("average memory", -5.0)]
> - average_memory: f64,
> - #[criterion("highest memory", -10.0)]
> - highest_memory: f64,
> - }
> -
> - const N_CRITERIA;
> - static PVE_HA_TOPSIS_CRITERIA;
> -}
> -
would prefer keeping the struct, so there's still a type giving the
numbers some name.
struct PveTopsisAlternative {
average_cpu: f64,
highest_cpu: f64,
average_memory: f64,
highest_memory: f64,
}
const N_CRITERIA = 4;
impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
fn from(alternative: PveTopsisAlternative) -> Self {
[
alternative.average_cpu,
alternative.highest_cpu,
alternative.average_memory,
alternative.highest_memory,
]
}
}
> #[derive(Clone, Debug)]
> pub struct Scheduler {
> nodes: Vec<NodeUsage>,
> @@ -111,12 +97,20 @@ pub struct ScoredMigration {
> pub migration: Migration,
> /// The expected node imbalance after the migration.
> pub imbalance: f64,
> + /// The score.
> + ///
> + /// - corresponds to imbalance of weighted loads for
> + /// [`Scheduler::score_best_balancing_migration_candidates`]
> + /// - corresponds to distance to ideal best alternatives for
> + /// [`Scheduler::score_best_balancing_migration_candidates_topsis`]
> + pub score: f64,
As discussed off-list, I'm not sure whether we should expose the 'score'
as it isn't relevant to the feedback loop now.
> }
>
> impl Ord for ScoredMigration {
> fn cmp(&self, other: &Self) -> Ordering {
> - self.imbalance
> - .total_cmp(&other.imbalance)
> + self.score
> + .total_cmp(&other.score)
> + .then(self.imbalance.total_cmp(&other.imbalance))
> .then(self.migration.cmp(&other.migration))
AFAICT this could lead to unwanted load balancing decisions at least for
the brute force case:
If a user configures cpu_weight = 0, mem_weight = 1, i.e., do not
include CPU usage in load balancing decisions, then if two migration
candidates would have the equal change in memory distribution but
different ones in CPU (i.e. equal imbalance values), it would then still
include CPU usage in the load balancing decision with the unweighted
'imbalance' value.
Of course, this wouldn't even be really noticable, so might not be an
issue.
Though I wonder whether calculating both the unweighted and weighted
node imbalance values for all migration candidates at least in the
bruteforce case does increase the computational load for a relatively
small special case (i.e. equal weighted imbalance values).
> }
> }
> @@ -136,19 +130,21 @@ impl PartialEq for ScoredMigration {
> impl Eq for ScoredMigration {}
>
> impl ScoredMigration {
> - pub fn new<T: Into<Migration>>(migration: T, imbalance: f64) -> Self {
> - // Depending on how the imbalance is calculated, it can contain minor approximation errors.
> - // As this struct implements the Ord trait, users of the struct's cmp() can run into cases,
> - // where the imbalance is the same up to the significant digits in base 10, but treated as
> - // different values.
> + pub fn new<T: Into<Migration>>(migration: T, imbalance: f64, score: f64) -> Self {
> + // Depending on how the imbalance or score is calculated, it can contain minor approximation
> + // errors. As this struct implements the Ord trait, users of the struct's cmp() can run into
> + // cases, where the imbalance or score is the same up to the significant digits in base 10,
> + // but treated as different values.
> //
> // Therefore, truncate any non-significant digits to prevent these cases.
> let factor = 10_f64.powi(f64::DIGITS as i32);
> let truncated_imbalance = f64::trunc(factor * imbalance) / factor;
> + let truncated_score = f64::trunc(factor * score) / factor;
>
> Self {
> migration: migration.into(),
> imbalance: truncated_imbalance,
> + score: truncated_score,
> }
> }
> }
> @@ -179,15 +175,15 @@ impl Scheduler {
> }
> }
>
> - /// Maps the current node usages to a [`PveTopsisAlternative`].
> + /// Maps the current node usages to a TOPSIS alternative.
> ///
> - /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
> - /// square (RMS) and maximum value of each stat in the node usages.
> + /// The alternative is derived by calculating a modified version of the root mean square (RMS)
> + /// and maximum value of each stat in the node usages.
> fn topsis_alternative_with(
> &self,
> map_node_stats: impl Fn(&NodeUsage) -> NodeStats,
> - ) -> PveTopsisAlternative {
> - let len = self.nodes.len();
> + ) -> [f64; 4] {
> + let len = self.nodes.len() as f64;
>
> // Base values on percentages to allow comparing nodes with different stats.
> let mut highest_cpu = 0.0;
> @@ -209,55 +205,79 @@ impl Scheduler {
>
> // Add 1.0 to avoid boosting tiny differences: e.g. 0.004 is twice as much as 0.002, but
> // 1.004 is only slightly more than 1.002.
> - PveTopsisAlternative {
> - average_cpu: 1.0 + (squares_cpu / len as f64).sqrt(),
> - highest_cpu: 1.0 + highest_cpu,
> - average_memory: 1.0 + (squares_mem / len as f64).sqrt(),
> - highest_memory: 1.0 + highest_mem,
> - }
> + [
> + 1.0 + (squares_cpu / len).sqrt(),
> + 1.0 + highest_cpu,
> + 1.0 + (squares_mem / len).sqrt(),
> + 1.0 + highest_mem,
> + ]
> }
>
> /// Returns the load imbalance among the nodes.
> ///
> + /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
> + /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
> + /// [`NodeStats::load`].
nit: can be documented in struct NodeStatsWeights
> + ///
> /// See [`calculate_node_imbalance`] for more information.
> - pub fn node_imbalance(&self) -> f64 {
> - calculate_node_imbalance(&self.nodes, |node| node.stats.load())
> + pub fn node_imbalance(&self, cpu_weight: f64, mem_weight: f64) -> f64 {
> + calculate_node_imbalance(&self.nodes, |node| node.stats.load(cpu_weight, mem_weight))
> }
If we go the NodeStatsWeights route, then
pub fn weighted_node_imbalance(&self, weights: &NodeStatsWeights) -> f64 {
calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(weights))
}
pub fn node_imbalance(&self) -> f64 {
self.weighted_node_imbalance(NodeStatsWeights::default())
}
So always using the idea that being able to set weights is the
generalized form and the old equal weights version of these methods is
a specialized variant.
>
> /// Returns the load imbalance among the nodes as if a specific resource was moved.
> ///
> + /// `cpu_weight`, respectively, `mem_weight`, control how much [`NodeStats::cpu_load`],
> + /// respectively, [`NodeStats::mem_load`], contributes to the load imbalance via
> + /// [`NodeStats::load`].
nit: can be documented in struct NodeStatsWeights
> + ///
> /// See [`calculate_node_imbalance`] for more information.
> - fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> + fn node_imbalance_with_migration_candidate(
> + &self,
> + candidate: &MigrationCandidate,
> + cpu_weight: f64,
> + mem_weight: f64,
> + ) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - node_stats_after_migration(candidate, node).load()
> + node_stats_after_migration(candidate, node).load(cpu_weight, mem_weight)
> })
> }
same idea as above with weighted_node_imbalance()
>
> - /// Scores the given migration `candidates` by the best node imbalance improvement with
> - /// exhaustive search.
> + /// Uses brute-force search to score how much either of the given migration `candidates`
> + /// improves node imbalance and ranks them accordingly.
Good catch!
As noted off-list, the "score" name is unfortunately a bit of a
misnomer, because this doesn't actually score them (the imbalance part
is only information for the feedback loop and log output outside), but
ranks them, therefore making the limit parameter reasonable.
However, as this is pre-existing and isn't relevant to this specific
patch, this should be done in a separate patch to not clutter this one,
which already changes quite a lot on its own.
This makes understanding past changes much easier in the future and
reviewing them too.
> ///
> /// The `candidates` are assumed to be consistent with the scheduler. No further validation is
> /// done whether the given nodenames actually exist in the scheduler.
> ///
> /// The scoring is done as if each resource migration has already been done. This assumes that
> /// the already migrated resource consumes the same amount of each stat as on the previous node
> - /// according to its `stats`.
> + /// according to its `stats`. The lower the score, the better.
> + ///
> + /// `cpu_weight` and `mem_weight` are blended into [`NodeStats::load`]. The higher the weight of
> + /// a computational resource, the more the search will tend to prioritize balancing out that
> + /// resource, e.g., a high `mem_weight` will steer migrations toward evening out memory usage,
> + /// even at the cost of leaving CPU load less balanced. This is a general tendency, not a
> + /// guarantee.
> ///
> /// Returns up to `limit` of the best scored migrations.
> pub fn score_best_balancing_migration_candidates<I>(
same idea here with a
weighted_score_best_balancing_migration_candidates()
> &self,
> candidates: I,
> limit: usize,
> + cpu_weight: usize,
> + mem_weight: usize,
I'd be fine to use the NodeStatsWeights here as well for the weighted
version of this.
Even though it doesn't guarantee the non-negative integer property
anymore on the type level, that's up to the caller. However, we can
enforce this on the pve-rs side on the type level or with assertions
here and notes in the doc comment.
> ) -> Vec<ScoredMigration>
> where
> I: IntoIterator<Item = MigrationCandidate>,
> {
> + let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
nit: variable shadowing would probably be better here just for casting,
but if you go for using the NodeStatsWeights struct here as well,
this isn't needed anymore
> +
> let mut scored_migrations = candidates
> .into_iter()
> .map(|candidate| {
> - let imbalance = self.node_imbalance_with_migration_candidate(&candidate);
> + let imbalance = self.node_imbalance_with_migration_candidate(&candidate, 1.0, 1.0);
> + let score = self.node_imbalance_with_migration_candidate(&candidate, w1, w2);
>
> - Reverse(ScoredMigration::new(candidate, imbalance))
> + Reverse(ScoredMigration::new(candidate, imbalance, score))
> })
> .collect::<BinaryHeap<_>>();
>
> @@ -274,56 +294,106 @@ impl Scheduler {
> best_migrations
> }
>
> - /// Scores the given migration `candidates` by the best node imbalance improvement with the
> - /// TOPSIS method.
> + /// Scores the given migration `candidates` using the TOPSIS method.
Same here, should be in a separate patch as this was already true
beforehand. Both documentation changes for the score methods can be done
in one patch, of course.
> ///
> /// The `candidates` are assumed to be consistent with the scheduler. No further validation is
> /// done whether the given nodenames actually exist in the scheduler.
> ///
> /// The scoring is done as if each resource migration has already been done. This assumes that
> /// the already migrated resource consumes the same amount of each stat as on the previous node
> - /// according to its `stats`.
> + /// according to its `stats`. The lower the score, the better.
> + ///
> + /// The weights are assigned to the respective criteria of the TOPSIS method.
> ///
> /// Returns up to `limit` of the best scored migrations.
> pub fn score_best_balancing_migration_candidates_topsis(
same for this method as above
> &self,
> candidates: &[MigrationCandidate],
> limit: usize,
> + cpu_weight: usize,
> + mem_weight: usize,
> ) -> Result<Vec<ScoredMigration>, Error> {
> + let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
> +
> + let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
> + let highest_cpu = topsis::Criterion::new(
> + "highest_cpu".to_owned(),
> + -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
> + );
> + let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
> + let highest_mem = topsis::Criterion::new(
> + "highest_mem".to_owned(),
> + -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
> + );
> +
> let matrix = candidates
> .iter()
> .map(|candidate| {
> self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
> - .into()
> })
> .collect::<Vec<_>>();
>
> - let best_alternatives =
> - topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + let mut scored_migrations = topsis::score_alternatives(
> + &topsis::Matrix::new(matrix)?,
> + &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
> + )?
> + .iter()
> + .zip(candidates.iter())
> + .map(|(topsis_score, candidate)| {
> + let imbalance = self.node_imbalance_with_migration_candidate(candidate, w1, w2);
So the meaning of the 'imbalance' field in ScoredMigration would also
differ in meaning between brute force and TOPSIS?
> +
> + Reverse(ScoredMigration::new(
> + candidate.clone(),
> + imbalance,
> + // invert score for consistent sorting
> + 1.0 - topsis_score,
> + ))
> + })
> + .collect::<BinaryHeap<_>>();
>
> - Ok(best_alternatives
> - .into_iter()
> - .take(limit)
> - .map(|i| {
> - let imbalance = self.node_imbalance_with_migration_candidate(&candidates[i]);
> + let mut best_migrations = Vec::with_capacity(limit);
>
> - ScoredMigration::new(candidates[i].clone(), imbalance)
> - })
> - .collect())
> + // BinaryHeap::into_iter_sorted() is still in nightly unfortunately
> + while best_migrations.len() < limit {
> + match scored_migrations.pop() {
> + Some(Reverse(alternative)) => best_migrations.push(alternative),
> + None => break,
> + }
> + }
> +
> + Ok(best_migrations)
> }
Hm, I think we should leave
score_best_balancing_migration_candidates_topsis() to only rank by
TOPSIS and not do the same procedure as is done with brute force.
Essentially this boils down to bruteforce and TOPSIS having different
guarantees in their return values, i.e., bruteforce will return a
Vec<ScoredMigration> with a monotonically increasing 'imbalance' value
while TOPSIS doesn't guarantee that but ranks after its own method.
The 'imbalance' value is then only used by external callers (i.e. the HA
Manager) to give users information and use it to decide whether the
migration is worth it with the 'margin' parameter.
>
> - /// Scores nodes to start a resource with the usage statistics `resource_stats` on.
> + /// Scores nodes to start a resource with the usage statistics `resource_stats` on, using the
> + /// TOPSIS method.
nit: ideally a separate patch
> ///
> /// The scoring is done as if the resource is already started on each node. This assumes that
> /// the already started resource consumes the maximum amount of each stat according to its
> /// `resource_stats`.
> ///
> + /// The weights are assigned to the respective criteria of the TOPSIS method.
> + ///
> /// Returns a vector of (nodename, score) pairs. Scores are between 0.0 and 1.0 and a higher
> /// score is better.
> pub fn score_nodes_to_start_resource<T: Into<ResourceStats>>(
> &self,
> resource_stats: T,
> + cpu_weight: usize,
> + mem_weight: usize,
> ) -> Result<Vec<(String, f64)>, Error> {
Unrelated to this patch itself, but the parameters are exposed as
'ha-auto-rebalance-{cpu,mem}-weight', but here they are also used for
other kinds of load balancing, i.e., initial placement balancing.
So maybe we should change the name of those?
Otherwise, should get a weighted_ version as well.
> + let (w1, w2) = (cpu_weight as f64, mem_weight as f64);
> +
> + let average_cpu = topsis::Criterion::new("average_cpu".to_owned(), -w1);
> + let highest_cpu = topsis::Criterion::new(
> + "highest_cpu".to_owned(),
> + -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w1),
> + );
> + let average_mem = topsis::Criterion::new("average_mem".to_owned(), -w2);
> + let highest_mem = topsis::Criterion::new(
> + "highest_mem".to_owned(),
> + -(TOPSIS_AVERAGE_VS_HIGHEST_FACTOR * w2),
> + );
> +
> let resource_stats = resource_stats.into();
>
> let matrix = self
> @@ -339,12 +409,13 @@ impl Scheduler {
>
> new_stats
> })
> - .into()
> })
> .collect::<Vec<_>>();
>
> - let scores =
> - topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + let scores = topsis::score_alternatives(
> + &topsis::Matrix::new(matrix)?,
> + &topsis::Criteria::new([average_cpu, highest_cpu, average_mem, highest_mem])?,
> + )?;
>
> Ok(scores
> .into_iter()
> @@ -367,6 +438,7 @@ mod tests {
Haven't reviewed the tests yet
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing
2026-10-22 9:15 ` Daniel Kral
@ 2026-10-22 11:34 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 11:34 UTC (permalink / raw)
To: Daniel Kral, Dominik Rusovac, pve-devel
As discussed off-list, another idea to solve the issue with backwards
compatibility is to make the Scheduler struct configurable with the
weights, so that the weighted_ versions of the function calls aren't
needed anymore.
As already noted in the pve-rs review part, an options struct with only
optional fields at the end of the bindings should be treated as optional
alltogether so that it doesn't break any callers from Perl, but should
definitely be checked (maybe even with test cases).
On Thu Oct 22, 2026 at 11:15 AM CEST, Daniel Kral wrote:
> Thanks a lot for working on this, nice work! Especially the cover letter
> is a very interesting analysis and read!
>
> Looks generally sound, though there are some comments inline on having
> backwards and forwards compatibility and some patch cleanup.
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (2 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights Dominik Rusovac
` (10 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
proptest 1.11 exists, but version 1.6 is fine for now.
afaict updating to 1.11 at some point shouldn't come with breaking
changes.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Cargo.toml | 1 +
proxmox-resource-scheduling/Cargo.toml | 3 +++
2 files changed, 4 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index 09ef21b4..72c18c54 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -155,6 +155,7 @@ udev = "0.9"
url = "2.2"
walkdir = "2"
zstd = "0.13"
+proptest = "1.6.0"
# workspace dependencies
proxmox-access-control = { version = "1.3.0", path = "proxmox-access-control" }
diff --git a/proxmox-resource-scheduling/Cargo.toml b/proxmox-resource-scheduling/Cargo.toml
index 2bfb7871..27d0f3f6 100644
--- a/proxmox-resource-scheduling/Cargo.toml
+++ b/proxmox-resource-scheduling/Cargo.toml
@@ -11,3 +11,6 @@ exclude.workspace = true
[dependencies]
anyhow.workspace = true
serde = { workspace = true, features = [ "derive" ] }
+
+[dev-dependencies]
+proptest.workspace = true
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (3 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
` (9 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Introduce proptests with constants to control scheduler proptests and
functions implementing useful proptest strategies.
Use per-method `mod`s to make deliberate testing easier, e.g., executing
`cargo test bruteforce` to run bruteforce (prop)tests only.
For each method (bruteforce, TOPSIS), respectively, add proptests:
- test_weight_scale_invariance: test that only weight ratio matters, not
their absolute magnitudes; and
- test_zero_cpu_weight_discards_cpu_load: cpu weight of 0 discards CPU
load (symmetric test discarding memory usage is omitted).
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
proxmox-resource-scheduling/src/scheduler.rs | 343 +++++++++++++++++++
1 file changed, 343 insertions(+)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 0690385d..3c4c8d87 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -429,6 +429,27 @@ impl Scheduler {
mod tests {
use super::*;
+ use proptest::prelude::*;
+ use std::ops::Range;
+
+ const MIN_NODES: usize = 3;
+ const MAX_NODES: usize = 36;
+
+ const MIN_CPU: usize = 2;
+ const MIN_MEM: usize = 2;
+ const MAX_CPU: usize = 128;
+ const MAX_MEM: usize = 256;
+
+ const MIN_WEIGHT: usize = 1;
+ const MAX_WEIGHT: usize = 1_000;
+
+ const MIN_CONSTANT: usize = 1;
+ const MAX_CONSTANT: usize = 1_000;
+
+ const REPEAT_PROPTEST_N_TIMES: u32 = 256; // proptest's default
+
+ const EPSILON: f64 = 1e-9;
+
#[test]
fn test_scored_migration_order() {
let migration1 = ScoredMigration::new(
@@ -528,4 +549,326 @@ mod tests {
assert_eq!(heap.pop(), Some(Reverse(migration2)));
assert_eq!(heap.pop(), Some(Reverse(migration3)));
}
+
+ /// Strategy to generate `n` random nodes with stats bound by `max_cpu` and `max_mem`.
+ pub fn nodes_strategy(
+ n: usize,
+ max_cpu: usize,
+ max_mem: usize,
+ ) -> impl Strategy<Value = Vec<NodeUsage>> {
+ prop::collection::vec((0..max_cpu + 1, 0..max_mem + 1), n).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(|(i, (cpu, mem))| NodeUsage {
+ name: format!("node{i}"),
+ stats: NodeStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu,
+ mem,
+ maxmem: max_mem,
+ },
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate a random number in `n_range` of migration candidates between `nodes`.
+ /// Resource stats are bound by `max_cpu` and `max_mem`.
+ fn candidates_strategy(
+ n_range: impl Into<proptest::collection::SizeRange>,
+ max_cpu: usize,
+ max_mem: usize,
+ nodes: Vec<NodeUsage>,
+ ) -> impl Strategy<Value = Vec<MigrationCandidate>> {
+ let k = nodes.len();
+
+ prop::collection::vec((0..max_cpu, 0..max_mem, 0..k, 0..k), n_range).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(
+ |(i, (cpu, mem, source_idx, target_idx))| MigrationCandidate {
+ migration: Migration {
+ sid: format!("vm:10{i}"),
+ source_node: nodes[source_idx].name.clone(),
+ target_node: nodes[target_idx].name.clone(),
+ },
+ stats: ResourceStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu as f64,
+ mem,
+ maxmem: max_mem,
+ },
+ },
+ )
+ .filter(|candidate| {
+ candidate.migration.source_node != candidate.migration.target_node
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate:
+ /// - a random number n in `n_nodes_range` of nodes
+ /// - a random number between 1 and n * (n - 1) of migration candidates between the generated
+ /// nodes
+ ///
+ /// The stats of nodes and migration candidates are bound by random numbers in `cpu_range` and
+ /// `mem_range`, respectively.
+ fn scenario_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<Value = (Vec<NodeUsage>, Vec<MigrationCandidate>)> {
+ (cpu_range, mem_range).prop_flat_map(move |(max_cpu, max_mem)| {
+ n_nodes_range
+ .clone()
+ .prop_filter("cluster size must be odd", |n| n % 2 == 1)
+ .prop_flat_map(move |n_nodes| {
+ nodes_strategy(n_nodes, max_cpu, max_mem).prop_flat_map(move |nodes| {
+ let max_candidates = n_nodes * (n_nodes - 1);
+
+ candidates_strategy(1..=max_candidates, max_cpu, max_mem, nodes.clone())
+ .prop_map(move |candidates| (nodes.clone(), candidates))
+ })
+ })
+ })
+ }
+
+ mod bruteforce {
+ use super::*;
+
+ mod weights {
+ use super::*;
+
+ proptest! {
+ #![proptest_config(ProptestConfig {
+ cases: REPEAT_PROPTEST_N_TIMES,
+ ..ProptestConfig::default()
+ })]
+
+ /// Test that the achievable scores are completely unaffected by uniformly scaling
+ /// both weights; only the weights' ratio matters.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let scheduler = Scheduler::from_nodes(nodes);
+ let n = candidates.len();
+
+ scheduler
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .iter()
+ .zip(
+ scheduler
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ n,
+ k * cpu_weight,
+ k * mem_weight,
+ )
+ .iter(),
+ )
+ .for_each(|(lhs, rhs)| {
+ assert!(
+ (lhs.score - rhs.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ lhs,
+ rhs,
+ );
+ });
+ }
+
+ /// Test that with a `cpu_weight` of 0 the score for each scored migration should
+ /// match an imbalance computed directly from [`NodeStats::mem_load()`] alone.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let cpu_weight = 0;
+
+ Scheduler::from_nodes(nodes.clone())
+ .score_best_balancing_migration_candidates(
+ candidates.clone(),
+ candidates.len(),
+ cpu_weight,
+ mem_weight,
+ )
+ .iter()
+ .filter_map(|scored_migration| {
+ candidates
+ .iter()
+ .find(|c| c.migration.sid == scored_migration.migration.sid)
+ .map(|c| (scored_migration, c))
+ })
+ .for_each(|(scored_migration, candidate)| {
+ let reference_imbalance = calculate_node_imbalance(&nodes, |node| {
+ node_stats_after_migration(candidate, node).mem_load()
+ });
+
+ assert!(
+ (scored_migration.score - reference_imbalance).abs() < EPSILON
+ );
+ })
+ }
+ }
+ }
+ }
+
+ mod topsis {
+ use super::*;
+
+ /// Strategy producing two scenarios using [`super::scenario_strategy`] that are identical
+ /// in every memory-related field and in topology but have varying CPU fields.
+ #[allow(clippy::type_complexity)]
+ fn two_scenarios_with_varying_cpu_stats_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<
+ Value = (
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ ),
+ > {
+ let max_possible_nodes = n_nodes_range.end - 1; // Range<T> is exclusive
+ let max_possible_candidates = max_possible_nodes * (max_possible_nodes - 1);
+ let cpu_pool_bound = cpu_range.end;
+
+ (
+ scenario_strategy(cpu_range, mem_range, n_nodes_range),
+ prop::collection::vec(0usize..cpu_pool_bound + 1, max_possible_nodes),
+ prop::collection::vec(0usize..cpu_pool_bound, max_possible_candidates),
+ )
+ .prop_map(
+ move |((nodes, candidates), alt_node_cpu_pool, alt_candidate_cpu_pool)| {
+ let mut alt_nodes = nodes.clone();
+ alt_nodes.iter_mut().zip(alt_node_cpu_pool.iter()).for_each(
+ |(node, &cpu)| {
+ node.stats.cpu = cpu as f64;
+ },
+ );
+
+ let mut alt_candidates = candidates.clone();
+ alt_candidates
+ .iter_mut()
+ .zip(alt_candidate_cpu_pool.iter())
+ .for_each(|(candidate, &cpu)| {
+ candidate.stats.cpu = cpu as f64;
+ });
+
+ ((nodes, candidates), (alt_nodes, alt_candidates))
+ },
+ )
+ }
+
+ proptest! {
+ #![proptest_config(ProptestConfig {
+ cases: REPEAT_PROPTEST_N_TIMES,
+ ..ProptestConfig::default()
+ })]
+
+ /// Test that the achievable scores are completely unaffected by uniformly scaling both
+ /// weights; only the weights' ratio matters.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let scheduler = Scheduler::from_nodes(nodes);
+ let n = candidates.len();
+
+ scheduler
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter()
+ .zip(
+ scheduler
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ n,
+ k * cpu_weight,
+ k * mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter(),
+ )
+ .for_each(|(lhs, rhs)| {
+ assert!(
+ (lhs.score - rhs.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ lhs,
+ rhs,
+ );
+ });
+ }
+
+ /// Test that varying CPU stats between two otherwise identical scenarios do not change
+ /// the achievable scores if CPU weight is 0.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ ((nodes_a, candidates_a), (nodes_b, candidates_b)) in
+ two_scenarios_with_varying_cpu_stats_strategy(
+ MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES
+ ),
+ ) {
+ let cpu_weight = 0;
+ let n = candidates_a.len();
+
+ Scheduler::from_nodes(nodes_a)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates_a,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter()
+ .zip(
+ Scheduler::from_nodes(nodes_b)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates_b,
+ n,
+ cpu_weight,
+ mem_weight,
+ )
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"))
+ .iter(),
+ )
+ .for_each(|(a, b)| {
+ assert!(
+ (a.score - b.score).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same score",
+ a,
+ b,
+ );
+ });
+ }
+ }
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (4 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 9:26 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights Dominik Rusovac
` (8 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add weights for weighted load balancing.
Adjust existing resource scheduling test to explicitly use same weights
as before.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
.../resource_scheduling/pve_dynamic.rs | 27 +++++++++++++++----
.../resource_scheduling/pve_static.rs | 25 +++++++++++++----
pve-rs/test/resource_scheduling.pl | 3 ++-
3 files changed, 44 insertions(+), 11 deletions(-)
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
index 8e066e0..d6b6f9b 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
@@ -166,10 +166,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
///
/// See [`proxmox_resource_scheduling::scheduler::Scheduler::node_imbalance`].
#[export]
- pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
+ pub fn calculate_node_imbalance(
+ #[try_from_ref] this: &Scheduler,
+ cpu_weight: usize,
+ mem_weight: usize,
+ ) -> f64 {
let usage = this.inner.lock().unwrap();
- usage.to_scheduler::<IdentityAggregator>().node_imbalance()
+ usage
+ .to_scheduler::<IdentityAggregator>()
+ .node_imbalance(cpu_weight as f64, mem_weight as f64)
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -181,6 +187,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -188,7 +196,7 @@ pub mod pve_rs_resource_scheduling_dynamic {
Ok(usage
.to_scheduler::<IdentityAggregator>()
- .score_best_balancing_migration_candidates(candidates, limit))
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight))
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -200,6 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -207,7 +217,12 @@ pub mod pve_rs_resource_scheduling_dynamic {
usage
.to_scheduler::<IdentityAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ limit,
+ cpu_weight,
+ mem_weight,
+ )
}
/// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
@@ -217,11 +232,13 @@ pub mod pve_rs_resource_scheduling_dynamic {
pub fn score_nodes_to_start_resource(
#[try_from_ref] this: &Scheduler,
resource_stats: DynamicResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartingAsStartedResourceAggregator>()
- .score_nodes_to_start_resource(resource_stats)
+ .score_nodes_to_start_resource(resource_stats, cpu_weight, mem_weight)
}
}
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_static.rs b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
index d83aa38..2fa8d8c 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_static.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
@@ -162,12 +162,16 @@ pub mod pve_rs_resource_scheduling_static {
///
/// See [`proxmox_resource_scheduling::scheduler::Scheduler::node_imbalance`].
#[export]
- pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
+ pub fn calculate_node_imbalance(
+ #[try_from_ref] this: &Scheduler,
+ cpu_weight: usize,
+ mem_weight: usize,
+ ) -> f64 {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartedResourceAggregator>()
- .node_imbalance()
+ .node_imbalance(cpu_weight as f64, mem_weight as f64)
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -179,6 +183,8 @@ pub mod pve_rs_resource_scheduling_static {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -186,7 +192,7 @@ pub mod pve_rs_resource_scheduling_static {
Ok(usage
.to_scheduler::<StartedResourceAggregator>()
- .score_best_balancing_migration_candidates(candidates, limit))
+ .score_best_balancing_migration_candidates(candidates, limit, cpu_weight, mem_weight))
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -198,6 +204,8 @@ pub mod pve_rs_resource_scheduling_static {
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<ScoredMigration>, Error> {
let usage = this.inner.lock().unwrap();
@@ -205,7 +213,12 @@ pub mod pve_rs_resource_scheduling_static {
usage
.to_scheduler::<StartedResourceAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ .score_best_balancing_migration_candidates_topsis(
+ &candidates,
+ limit,
+ cpu_weight,
+ mem_weight,
+ )
}
/// Method: Scores nodes to start a service with the usage statistics `service_stats` on.
@@ -215,11 +228,13 @@ pub mod pve_rs_resource_scheduling_static {
pub fn score_nodes_to_start_service(
#[try_from_ref] this: &Scheduler,
service_stats: StaticResourceStats,
+ cpu_weight: usize,
+ mem_weight: usize,
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
usage
.to_scheduler::<StartedResourceAggregator>()
- .score_nodes_to_start_resource(service_stats)
+ .score_nodes_to_start_resource(service_stats, cpu_weight, mem_weight)
}
}
diff --git a/pve-rs/test/resource_scheduling.pl b/pve-rs/test/resource_scheduling.pl
index 3775242..31f2622 100755
--- a/pve-rs/test/resource_scheduling.pl
+++ b/pve-rs/test/resource_scheduling.pl
@@ -11,7 +11,8 @@ use PVE::RS::ResourceScheduling::Dynamic;
my sub score_nodes {
my ($static, $service) = @_;
- my $score_list = $static->score_nodes_to_start_service($service);
+ # uses original empirically determined weights
+ my $score_list = $static->score_nodes_to_start_service($service, 1.0, 5.0);
# imitate HA manager
my $scores = { map { $_->[0] => -$_->[1] } $score_list->@* };
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
@ 2026-10-22 9:26 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 9:26 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
The same applies here as in the changes for proxmox_resource_scheduling,
this would need a versioned break, e.g., if users have a newer libpve-rs
but an older ha-manager, the HA Manager would fail calling these as the
function signature was changed.
Though here, we could probably add options structs at the end of the
function parameter list instead of having extra weighted_ versions,
e.g.:
#[derive(Deserialize, Debug)]
struct BruteforceOptions {
#[serde(default = "1")]
cpu_weight: usize,
#[serde(default = "1")]
mem_weight: usize,
}
#[export]
pub fn score_best_balancing_migration_candidates(
#[try_from_ref] this: &Scheduler,
candidates: Vec<CompactMigrationCandidate>,
limit: usize,
options: BruteforceOptions,
) -> Result<Vec<ScoredMigration>, Error> {}
AFAIK this shouldn't cause old pve-ha-manager versions failing to call
this as there's a default value for each parameter. Could also be a
struct with all fields being some Option<T> and then doing the default
thing in the function body itself.
Haven't tested this yet though.
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Add weights for weighted load balancing.
>
> Adjust existing resource scheduling test to explicitly use same weights
> as before.
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> .../resource_scheduling/pve_dynamic.rs | 27 +++++++++++++++----
> .../resource_scheduling/pve_static.rs | 25 +++++++++++++----
> pve-rs/test/resource_scheduling.pl | 3 ++-
> 3 files changed, 44 insertions(+), 11 deletions(-)
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> index 8e066e0..d6b6f9b 100644
> --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> @@ -166,10 +166,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
> ///
> /// See [`proxmox_resource_scheduling::scheduler::Scheduler::node_imbalance`].
> #[export]
> - pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
> + pub fn calculate_node_imbalance(
> + #[try_from_ref] this: &Scheduler,
> + cpu_weight: usize,
> + mem_weight: usize,
> + ) -> f64 {
> let usage = this.inner.lock().unwrap();
>
> - usage.to_scheduler::<IdentityAggregator>().node_imbalance()
> + usage
> + .to_scheduler::<IdentityAggregator>()
> + .node_imbalance(cpu_weight as f64, mem_weight as f64)
> }
>
> /// Method: Scores the given migration `candidates` by the best node imbalance improvement with
> @@ -181,6 +187,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
> #[try_from_ref] this: &Scheduler,
> candidates: Vec<CompactMigrationCandidate>,
> limit: usize,
> + cpu_weight: usize,
> + mem_weight: usize,
> ) -> Result<Vec<ScoredMigration>, Error> {
> let usage = this.inner.lock().unwrap();
>
[ -->8- snip -8<-- ]
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (5 preceding siblings ...)
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-manager 08/15] ui: form: crs options: make options advanced Dominik Rusovac
` (7 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Default 'Memory Weight' changes with selected method; value should be 5
for 'TOPSIS' and 1 otherwise.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
www/manager6/form/CRSOptions.js | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index 84f6d506..cde389ee 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -20,6 +20,7 @@ Ext.define('PVE.form.CRSOptions', {
viewModel: {
data: {
crsMode: '__default__',
+ rebalanceMethod: '__default__',
autoRebalancing: 0,
},
formulas: {
@@ -27,6 +28,9 @@ Ext.define('PVE.form.CRSOptions', {
get('crsMode') === 'static' || get('crsMode') === 'dynamic',
autoRebalancingDisabled: (get) =>
!get('autoRebalancing') || !get('canUseAutoRebalancing'),
+ isTopsis: (get) => get('rebalanceMethod') === 'topsis',
+ memWeightEmptyText: (get) =>
+ Proxmox.Utils.defaultText + (get('isTopsis') ? ' (5)' : ' (1)'),
},
},
@@ -100,6 +104,7 @@ Ext.define('PVE.form.CRSOptions', {
],
defaultValue: '__default__',
bind: {
+ value: '{rebalanceMethod}',
disabled: '{autoRebalancingDisabled}',
},
},
@@ -126,6 +131,28 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ {
+ xtype: 'numberfield',
+ name: 'ha-auto-rebalance-cpu-weight',
+ fieldLabel: gettext('CPU Weight'),
+ emptyText: Proxmox.Utils.defaultText + ' (1)',
+ minValue: 0,
+ step: 1,
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ },
+ },
+ {
+ xtype: 'numberfield',
+ name: 'ha-auto-rebalance-mem-weight',
+ fieldLabel: gettext('Memory Weight'),
+ minValue: 0,
+ step: 1,
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ emptyText: '{memWeightEmptyText}',
+ },
+ },
],
},
],
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH pve-manager 08/15] ui: form: crs options: make options advanced
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (6 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
` (6 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
With the addition of auto-rebalance weights the CRS settings in the UI
have become even more cluttered.
Make 'Hold Duration', 'Minimal Imbalance Improvement', 'CPU Weight' and
'Memory Weight' advanced options.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
www/manager6/form/CRSOptions.js | 2 ++
1 file changed, 2 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index cde389ee..8419a805 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -108,6 +108,8 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ ],
+ advancedItems: [
{
xtype: 'numberfield',
name: 'ha-auto-rebalance-hold-duration',
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH pve-ha-manager 09/15] manager: remove obsolete argument
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (7 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-manager 08/15] ui: form: crs options: make options advanced Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 9:29 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
` (5 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Respective parameter was removed by [0].
[0] https://lore.proxmox.com/all/20260319164047.642936-4-d.kral@proxmox.com/
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..7a5a685 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -385,7 +385,7 @@ sub select_service_node {
return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
- my $scores = $online_node_usage->score_nodes_to_start_service($sid, $current_node);
+ my $scores = $online_node_usage->score_nodes_to_start_service($sid);
my @nodes = sort {
$scores->{$a} <=> $scores->{$b} || $a cmp $b
} keys %$pri_nodes;
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH pve-ha-manager 09/15] manager: remove obsolete argument
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
@ 2026-10-22 9:29 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 9:29 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Good catch, thanks!
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Respective parameter was removed by [0].
>
> [0] https://lore.proxmox.com/all/20260319164047.642936-4-d.kral@proxmox.com/
Though same here, this should in fact be a Fixes: <commitref>
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> src/PVE/HA/Manager.pm | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 5840a76..7a5a685 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -385,7 +385,7 @@ sub select_service_node {
>
> return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
>
> - my $scores = $online_node_usage->score_nodes_to_start_service($sid, $current_node);
> + my $scores = $online_node_usage->score_nodes_to_start_service($sid);
> my @nodes = sort {
> $scores->{$a} <=> $scores->{$b} || $a cmp $b
> } keys %$pri_nodes;
Otherwise, consider as:
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (8 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 9:58 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime Dominik Rusovac
` (4 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Include ha-auto-rebalance-cpu-weight and ha-auto-rebalance-mem-weight in
load imbalance calculation and decision making.
The weights only play into decisions in the context of automatic
rebalancing. The bruteforce method optimizes the weighted load
imbalance. For the TOPSIS method ha-auto-rebalance-cpu-weight and
ha-auto-rebalance-mem-weight are assigned to the respective average cpu
and average memory criteria. The weights of the highest cpu and highest
memory criteria of TOPSIS are not independently configurable. Nodes to
start service are scored with the same fixed (TOPSIS) weights as before.
The weights also shape how sensitively the load balancer reacts to CPU
load changes relative to memory usage changes. This is to address: "In
our case CRS has not moved any VM around because the imbalance
percentage is low (7%). Eventhough the cluster is very imbalance [sic]
CPU wise, or memory wise alone." [0]
Skip load balancing with warning if both weights are 0.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 47 ++++++++++++++++---
src/PVE/HA/Usage.pm | 21 +++++----
src/PVE/HA/Usage/Basic.pm | 14 ++++--
src/PVE/HA/Usage/Dynamic.pm | 27 +++++++----
src/PVE/HA/Usage/Static.pm | 27 +++++++----
.../log.expect | 13 +----
6 files changed, 102 insertions(+), 47 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 7a5a685..c6818e2 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -119,6 +119,12 @@ sub update_crs_scheduler_mode {
$self->{crs}->{auto_rebalance}->{margin} =
($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ my $default_weight = 1;
+ $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
+ # by default TOPSIS weights mem higher than cpu
+ $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
+ $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
+
my $old_mode = $self->{crs}->{scheduler};
my $new_mode = $crs_cfg->{ha} || 'basic';
@@ -236,8 +242,20 @@ sub load_balance {
return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
return if $self->any_resource_motion_queued_or_running();
- my ($threshold, $method, $hold_duration, $margin) =
- $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
+ my (
+ $threshold, $method, $hold_duration, $margin, $cpu_weight, $mem_weight,
+ ) =
+ $auto_rebalance_opts->@{qw(threshold method hold_duration margin cpu_weight mem_weight)};
+
+ if ($cpu_weight + $mem_weight eq 0) {
+ $haenv->log(
+ 'warning',
+ "auto-rebalance weights are both set to 0, skipping load balancing -"
+ . " set at least one to a positive value to avoid constant load"
+ . " imbalance of 0",
+ );
+ return;
+ }
# with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
# migration, churning the cluster forever - skip and warn instead of silently looping.
@@ -250,7 +268,7 @@ sub load_balance {
return;
}
- my $imbalance = $online_node_usage->calculate_node_imbalance();
+ my $imbalance = $online_node_usage->calculate_node_imbalance($cpu_weight, $mem_weight);
# do not load balance unless imbalance threshold has been exceeded
# consecutively for $hold_duration calls to load_balance();
@@ -268,15 +286,24 @@ sub load_balance {
my $result;
if ($method eq 'bruteforce') {
- $result = $online_node_usage->select_best_balancing_migration($candidates);
+ $result = $online_node_usage->select_best_balancing_migration(
+ $candidates,
+ $cpu_weight,
+ $mem_weight,
+ );
} elsif ($method eq 'topsis') {
- $result = $online_node_usage->select_best_balancing_migration_topsis($candidates);
+ $result = $online_node_usage->select_best_balancing_migration_topsis(
+ $candidates,
+ $cpu_weight,
+ $mem_weight,
+ );
}
# happens if $candidates is empty or $method isn't handled above
return if !$result;
- my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
+ my ($migration, $target_imbalance, $score) = $result->@{qw(migration imbalance score)};
+ $target_imbalance = $score if $method eq 'bruteforce';
my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
return if $relative_change < $margin;
@@ -311,7 +338,13 @@ sub flush_master_status {
$ms->{node_status} = $ns->{status};
$ms->{service_status} = $ss;
$ms->{timestamp} = $haenv->get_time();
- $ms->{imbalance} = $self->{online_node_usage}->calculate_node_imbalance();
+
+ my ($cpu_weight, $mem_weight) = (
+ $self->{crs}->{auto_rebalance}->{cpu_weight},
+ $self->{crs}->{auto_rebalance}->{mem_weight},
+ );
+ $ms->{imbalance} =
+ $self->{online_node_usage}->calculate_node_imbalance($cpu_weight, $mem_weight);
$haenv->write_manager_status($ms);
}
diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
index 659ab30..fbd8800 100644
--- a/src/PVE/HA/Usage.pm
+++ b/src/PVE/HA/Usage.pm
@@ -61,42 +61,47 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
sub select_best_balancing_migration {
- my ($self, $migration_candidates) = @_;
+ my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
- my $migrations = $self->score_best_balancing_migrations($migration_candidates, 1);
+ my $migrations =
+ $self->score_best_balancing_migrations($migration_candidates, 1, $cpu_weight, $mem_weight);
return $migrations->[0];
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
die "implement in subclass";
}
sub select_best_balancing_migration_topsis {
- my ($self, $migration_candidates) = @_;
+ my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
- my $migrations = $self->score_best_balancing_migrations_topsis($migration_candidates, 1);
+ my $migrations = $self->score_best_balancing_migrations_topsis(
+ $migration_candidates, 1, $cpu_weight, $mem_weight,
+ );
return $migrations->[0];
}
# Returns a hash with $nodename => $score pairs. A lower $score is better.
sub score_nodes_to_start_service {
- my ($self, $sid) = @_;
+ my ($self, $sid, $cpu_weight, $mem_weight) = @_;
die "implement in subclass";
}
diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
index 4dce9e1..57eecd3 100644
--- a/src/PVE/HA/Usage/Basic.pm
+++ b/src/PVE/HA/Usage/Basic.pm
@@ -67,19 +67,27 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
return 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
return [];
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self,
+ $migration_candidates,
+ $limit,
+ $cpu_weight,
+ $max_cpu_weight,
+ $mem_weight,
+ $max_mem_weight,
+ ) = @_;
return [];
}
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..bd03e89 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -105,20 +105,22 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
- my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
+ my $node_imbalance =
+ eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
$self->{haenv}->log('warning', "unable to calculate dynamic node imbalance - $@") if $@;
return $node_imbalance // 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -126,11 +128,14 @@ sub score_best_balancing_migrations {
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -140,9 +145,13 @@ sub score_best_balancing_migrations_topsis {
sub score_nodes_to_start_service {
my ($self, $sid) = @_;
+ my ($cpu_weight, $mem_weight) = (1, 5);
+
my $score_list = eval {
my $service_usage = get_service_usage($self, $sid);
- $self->{scheduler}->score_nodes_to_start_resource($service_usage);
+ $self->{scheduler}->score_nodes_to_start_resource(
+ $service_usage, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}
->log('err', "unable to score nodes according to dynamic usage for service '$sid' - $@")
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index e67d5f5..39308ff 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -112,20 +112,22 @@ sub remove_service_usage {
}
sub calculate_node_imbalance {
- my ($self) = @_;
+ my ($self, $cpu_weight, $mem_weight) = @_;
- my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
+ my $node_imbalance =
+ eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
$self->{haenv}->log('warning', "unable to calculate static node imbalance - $@") if $@;
return $node_imbalance // 0.0;
}
sub score_best_balancing_migrations {
- my ($self, $migration_candidates, $limit) = @_;
+ my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -133,11 +135,14 @@ sub score_best_balancing_migrations {
}
sub score_best_balancing_migrations_topsis {
- my ($self, $migration_candidates, $limit) = @_;
+ my (
+ $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ ) = @_;
my $migrations = eval {
- $self->{scheduler}
- ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
+ $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
+ $migration_candidates, $limit, $cpu_weight, $mem_weight,
+ );
};
$self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
@@ -147,9 +152,13 @@ sub score_best_balancing_migrations_topsis {
sub score_nodes_to_start_service {
my ($self, $sid) = @_;
+ my ($cpu_weight, $mem_weight) = (1, 5);
+
my $score_list = eval {
my $service_usage = get_service_usage($self, $sid);
- $self->{scheduler}->score_nodes_to_start_service($service_usage);
+ $self->{scheduler}->score_nodes_to_start_service(
+ $service_usage, $cpu_weight, $mem_weight,
+ );
};
if (my $err = $@) {
$self->{haenv}->log(
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 2f19c72..95163d4 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -53,7 +53,7 @@ info 25 node3/lrm: service status vm:107 started
info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912
info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192
info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011
-info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%)
+info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%)
info 160 node1/crm: got crm command: migrate vm:105 node2
info 160 node1/crm: migrate service 'vm:105' to node 'node2'
info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2)
@@ -68,7 +68,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8
info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201
info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211
info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191
-info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%)
+info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%)
info 260 node1/crm: got crm command: migrate vm:103 node3
info 260 node1/crm: migrate service 'vm:103' to node 'node3'
info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3)
@@ -77,13 +77,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
info 285 node3/lrm: starting service vm:103
info 285 node3/lrm: service status vm:103 started
-info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
-info 340 node1/crm: got crm command: migrate vm:105 node1
-info 340 node1/crm: migrate service 'vm:105' to node 'node1'
-info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
-info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
-info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
-info 361 node1/lrm: starting service vm:105
-info 361 node1/lrm: service status vm:105 started
info 820 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
@ 2026-10-22 9:58 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 9:58 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Include ha-auto-rebalance-cpu-weight and ha-auto-rebalance-mem-weight in
> load imbalance calculation and decision making.
As noted, these options control the decision making for intial placement
(rebalance-on-start) as well, so should probably drop the 'auto-' part.
>
> The weights only play into decisions in the context of automatic
> rebalancing. The bruteforce method optimizes the weighted load
> imbalance. For the TOPSIS method ha-auto-rebalance-cpu-weight and
> ha-auto-rebalance-mem-weight are assigned to the respective average cpu
> and average memory criteria. The weights of the highest cpu and highest
> memory criteria of TOPSIS are not independently configurable. Nodes to
> start service are scored with the same fixed (TOPSIS) weights as before.
Oh, so the weights don't influence score_nodes_to_start_resource()...
Hm, but maybe we could do that as well? What do you think?
>
> The weights also shape how sensitively the load balancer reacts to CPU
> load changes relative to memory usage changes. This is to address: "In
> our case CRS has not moved any VM around because the imbalance
> percentage is low (7%). Eventhough the cluster is very imbalance [sic]
> CPU wise, or memory wise alone." [0]
>
> Skip load balancing with warning if both weights are 0.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> src/PVE/HA/Manager.pm | 47 ++++++++++++++++---
> src/PVE/HA/Usage.pm | 21 +++++----
> src/PVE/HA/Usage/Basic.pm | 14 ++++--
> src/PVE/HA/Usage/Dynamic.pm | 27 +++++++----
> src/PVE/HA/Usage/Static.pm | 27 +++++++----
> .../log.expect | 13 +----
> 6 files changed, 102 insertions(+), 47 deletions(-)
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 7a5a685..c6818e2 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -119,6 +119,12 @@ sub update_crs_scheduler_mode {
> $self->{crs}->{auto_rebalance}->{margin} =
> ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
>
> + my $default_weight = 1;
> + $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
> + # by default TOPSIS weights mem higher than cpu
> + $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
> + $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
nit: might be slightly nicer to read with a
my ($cpu_weight, $mem_weight) = $crs_cfg->@{qw(ha-rebalance-cpu-weight ha-rebalance-mem-weight)};
if ($method eq 'topsis') {
$self->{crs}->{cpu_weight} = $cpu_weight // 1;
$self->{crs}->{mem_weight} = $mem_weight // 5;
} else {
$self->{crs}->{cpu_weight} = $cpu_weight // 1;
$self->{crs}->{mem_weight} = $mem_weight // 1;
}
but no hard feelings.
Put the fields outside of auto_rebalance as these are also used for
rebalance-on-start, but no hard feelings for that either.
Another idea, that would be a bit more refactoring beforehand, would be
to actually make 'method' a intialization parameter for
$online_node_usage on the pve-ha-manager side.
Then $cpu_weight and $mem_weight could also become initialization
parameters, and we wouldn't need to pass them to
calculate_node_imbalance() and the score_best_balancing_migrations()
methods anymore.
Then we would also need only one score_best_balancing_migrations(),
which under its hood would then either call the bruteforce or topsis
pve-rs version.
This works because $online_node_usage is reconstructed in every HA
Manager round anyway.
> +
> my $old_mode = $self->{crs}->{scheduler};
> my $new_mode = $crs_cfg->{ha} || 'basic';
>
> @@ -236,8 +242,20 @@ sub load_balance {
> return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
> return if $self->any_resource_motion_queued_or_running();
>
> - my ($threshold, $method, $hold_duration, $margin) =
> - $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
> + my (
> + $threshold, $method, $hold_duration, $margin, $cpu_weight, $mem_weight,
> + ) =
> + $auto_rebalance_opts->@{qw(threshold method hold_duration margin cpu_weight mem_weight)};
> +
> + if ($cpu_weight + $mem_weight eq 0) {
> + $haenv->log(
> + 'warning',
> + "auto-rebalance weights are both set to 0, skipping load balancing -"
nit: all resource weights set to 0, skipping load balancing -
> + . " set at least one to a positive value to avoid constant load"
> + . " imbalance of 0",
> + );
> + return;
> + }
Though maybe this should come after the threshold + duration check, so
it is only skips an unnecessary call to
score_best_balancing_migrations() and so it doesn't get logged in each
call to load_balance()?
No hard feelings though
>
> # with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
> # migration, churning the cluster forever - skip and warn instead of silently looping.
> @@ -250,7 +268,7 @@ sub load_balance {
> return;
> }
>
> - my $imbalance = $online_node_usage->calculate_node_imbalance();
> + my $imbalance = $online_node_usage->calculate_node_imbalance($cpu_weight, $mem_weight);
>
> # do not load balance unless imbalance threshold has been exceeded
> # consecutively for $hold_duration calls to load_balance();
> @@ -268,15 +286,24 @@ sub load_balance {
>
> my $result;
> if ($method eq 'bruteforce') {
> - $result = $online_node_usage->select_best_balancing_migration($candidates);
> + $result = $online_node_usage->select_best_balancing_migration(
> + $candidates,
> + $cpu_weight,
> + $mem_weight,
> + );
> } elsif ($method eq 'topsis') {
> - $result = $online_node_usage->select_best_balancing_migration_topsis($candidates);
> + $result = $online_node_usage->select_best_balancing_migration_topsis(
> + $candidates,
> + $cpu_weight,
> + $mem_weight,
> + );
> }
>
> # happens if $candidates is empty or $method isn't handled above
> return if !$result;
>
> - my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
> + my ($migration, $target_imbalance, $score) = $result->@{qw(migration imbalance score)};
> + $target_imbalance = $score if $method eq 'bruteforce';
>
> my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
> return if $relative_change < $margin;
> @@ -311,7 +338,13 @@ sub flush_master_status {
> $ms->{node_status} = $ns->{status};
> $ms->{service_status} = $ss;
> $ms->{timestamp} = $haenv->get_time();
> - $ms->{imbalance} = $self->{online_node_usage}->calculate_node_imbalance();
> +
> + my ($cpu_weight, $mem_weight) = (
> + $self->{crs}->{auto_rebalance}->{cpu_weight},
> + $self->{crs}->{auto_rebalance}->{mem_weight},
> + );
nit: can be
my ($cpu_weight, $mem_weight) = $self->{crs}->@{qw(cpu_weight mem_weight)};
> + $ms->{imbalance} =
> + $self->{online_node_usage}->calculate_node_imbalance($cpu_weight, $mem_weight);
>
> $haenv->write_manager_status($ms);
> }
> diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
> index 659ab30..fbd8800 100644
> --- a/src/PVE/HA/Usage.pm
> +++ b/src/PVE/HA/Usage.pm
> @@ -61,42 +61,47 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
>
> sub select_best_balancing_migration {
> - my ($self, $migration_candidates) = @_;
> + my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
>
> - my $migrations = $self->score_best_balancing_migrations($migration_candidates, 1);
> + my $migrations =
> + $self->score_best_balancing_migrations($migration_candidates, 1, $cpu_weight, $mem_weight);
>
> return $migrations->[0];
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> die "implement in subclass";
> }
>
> sub select_best_balancing_migration_topsis {
> - my ($self, $migration_candidates) = @_;
> + my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
>
> - my $migrations = $self->score_best_balancing_migrations_topsis($migration_candidates, 1);
> + my $migrations = $self->score_best_balancing_migrations_topsis(
> + $migration_candidates, 1, $cpu_weight, $mem_weight,
> + );
>
> return $migrations->[0];
> }
>
> # Returns a hash with $nodename => $score pairs. A lower $score is better.
> sub score_nodes_to_start_service {
> - my ($self, $sid) = @_;
> + my ($self, $sid, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
> diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
> index 4dce9e1..57eecd3 100644
> --- a/src/PVE/HA/Usage/Basic.pm
> +++ b/src/PVE/HA/Usage/Basic.pm
> @@ -67,19 +67,27 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> return 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> return [];
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self,
> + $migration_candidates,
> + $limit,
> + $cpu_weight,
> + $max_cpu_weight,
> + $mem_weight,
> + $max_mem_weight,
$max_{cpu,mem}_weight seem to be from a former design of this series :-)
> + ) = @_;
>
> return [];
> }
> diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
> index 76d0fea..bd03e89 100644
> --- a/src/PVE/HA/Usage/Dynamic.pm
> +++ b/src/PVE/HA/Usage/Dynamic.pm
> @@ -105,20 +105,22 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> - my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
> + my $node_imbalance =
> + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
> $self->{haenv}->log('warning', "unable to calculate dynamic node imbalance - $@") if $@;
>
> return $node_imbalance // 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -126,11 +128,14 @@ sub score_best_balancing_migrations {
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -140,9 +145,13 @@ sub score_best_balancing_migrations_topsis {
> sub score_nodes_to_start_service {
> my ($self, $sid) = @_;
>
> + my ($cpu_weight, $mem_weight) = (1, 5);
> +
> my $score_list = eval {
> my $service_usage = get_service_usage($self, $sid);
> - $self->{scheduler}->score_nodes_to_start_resource($service_usage);
> + $self->{scheduler}->score_nodes_to_start_resource(
> + $service_usage, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}
> ->log('err', "unable to score nodes according to dynamic usage for service '$sid' - $@")
> diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
> index e67d5f5..39308ff 100644
> --- a/src/PVE/HA/Usage/Static.pm
> +++ b/src/PVE/HA/Usage/Static.pm
> @@ -112,20 +112,22 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> - my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
> + my $node_imbalance =
> + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
> $self->{haenv}->log('warning', "unable to calculate static node imbalance - $@") if $@;
>
> return $node_imbalance // 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -133,11 +135,14 @@ sub score_best_balancing_migrations {
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -147,9 +152,13 @@ sub score_best_balancing_migrations_topsis {
> sub score_nodes_to_start_service {
> my ($self, $sid) = @_;
>
> + my ($cpu_weight, $mem_weight) = (1, 5);
> +
> my $score_list = eval {
> my $service_usage = get_service_usage($self, $sid);
> - $self->{scheduler}->score_nodes_to_start_service($service_usage);
> + $self->{scheduler}->score_nodes_to_start_service(
> + $service_usage, $cpu_weight, $mem_weight,
> + );
> };
> if (my $err = $@) {
> $self->{haenv}->log(
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> index 2f19c72..95163d4 100644
> --- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> +++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> @@ -53,7 +53,7 @@ info 25 node3/lrm: service status vm:107 started
> info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912
> info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192
> info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011
> -info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%)
> +info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%)
> info 160 node1/crm: got crm command: migrate vm:105 node2
> info 160 node1/crm: migrate service 'vm:105' to node 'node2'
> info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2)
> @@ -68,7 +68,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8
> info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201
> info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211
> info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191
> -info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%)
> +info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%)
> info 260 node1/crm: got crm command: migrate vm:103 node3
> info 260 node1/crm: migrate service 'vm:103' to node 'node3'
> info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3)
> @@ -77,13 +77,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
> info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
> info 285 node3/lrm: starting service vm:103
> info 285 node3/lrm: service status vm:103 started
> -info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
> -info 340 node1/crm: got crm command: migrate vm:105 node1
> -info 340 node1/crm: migrate service 'vm:105' to node 'node1'
> -info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
> -info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
> -info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
> -info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
> -info 361 node1/lrm: starting service vm:105
> -info 361 node1/lrm: service status vm:105 started
> info 820 hardware: exit simulation - done
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (9 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
` (3 subsequent siblings)
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
This adds commands to change CRS settings or shutdown policy at test
runtime.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Sim/Hardware.pm | 62 ++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index e1a9ddd..e75229c 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -59,6 +59,7 @@ my $default_node_maxmem = 131072 * 1024**2;
# $testdir/status/service_config Service configuration
# $testdir/status/service_status_<node> Service status
# $testdir/status/groups HA groups configuration
+# $testdir/status/datacenter.cfg Datacenter wide HA configuration
sub read_lrm_status {
my ($self, $node) = @_;
@@ -699,6 +700,58 @@ sub write_datacenter_conf {
PVE::HA::Tools::write_json_to_file($filename, $conf);
}
+my $datacenter_settings = {
+ ha => {
+ shutdown_policy => qr/^(?:conditional|freeze|failover||migrate)$/,
+ },
+ crs => {
+ ha => qr/^(?:basic|static|dynamic)$/,
+ 'ha-rebalance-on-start' => qr/^[01]$/,
+ 'ha-auto-rebalance' => qr/^[01]$/,
+ 'ha-auto-rebalance-threshold' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-auto-rebalance-method' => qr/^(?:bruteforce|topsis)$/,
+ 'ha-auto-rebalance-hold-duration' => qr/^\d+$/,
+ 'ha-auto-rebalance-margin' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-auto-rebalance-cpu-weight' => qr/^\d+$/,
+ 'ha-auto-rebalance-mem-weight' => qr/^\d+$/,
+ },
+};
+
+sub update_datacenter_setting {
+ my ($self, $section, $action, $key, $value) = @_;
+
+ my $section_props = $datacenter_settings->{$section}
+ or die "sim_hardware_cmd: unknown datacenter section '$section'\n";
+
+ die "sim_hardware_cmd: missing property for 'datacenter $section $action' command\n"
+ if !defined($key);
+
+ my $verifier = $section_props->{$key}
+ or die "sim_hardware_cmd: unknown datacenter property '$key' in section '$section'\n";
+
+ my $conf = $self->read_datacenter_conf();
+
+ if ($action eq 'set') {
+ die "sim_hardware_cmd: missing value for 'datacenter $section set $key' command\n"
+ if !defined($value);
+ die "sim_hardware_cmd: invalid value '$value' for datacenter property '$section.$key'\n"
+ if $value !~ $verifier;
+
+ $value = $value + 0 if $value =~ m/^\d+(?:\.\d+)?$/;
+
+ $conf->{$section}->{$key} = $value;
+ } elsif ($action eq 'delete') {
+ delete $conf->{$section}->{$key} if $conf->{$section};
+ delete $conf->{$section} if $conf->{$section} && !%{ $conf->{$section} };
+ } else {
+ die "sim_hardware_cmd: unknown datacenter action '$action'\n";
+ }
+
+ $self->write_datacenter_conf($conf);
+
+ return $conf;
+}
+
sub global_lock {
my ($self, $code, @param) = @_;
@@ -802,6 +855,8 @@ sub get_cfs_state {
# service <sid> set-static-stats [maxcpu <cores>] [maxmem <MiB>]
# service <sid> set-dynamic-stats [cpu <cores>] [mem <MiB>]
# service <sid> delete
+# datacenter <ha|crs> set <property> <value>
+# datacenter <ha|crs> delete <property>
sub sim_hardware_cmd {
my ($self, $cmdstr, $logid) = @_;
@@ -820,6 +875,8 @@ sub sim_hardware_cmd {
if ($cmd eq 'service') {
$sid = PVE::HA::Tools::pve_verify_ha_resource_id($objid);
+ } elsif ($cmd eq 'datacenter') {
+ # cluster-wide, neither node nor service specific
} else {
$node = $objid;
$d = $self->{nodes}->{$node}
@@ -1028,6 +1085,11 @@ sub sim_hardware_cmd {
die "sim_hardware_cmd: unknown service action '$action' "
. "- not implemented\n";
}
+ } elsif ($cmd eq 'datacenter') {
+ die "sim_hardware_cmd: missing action for 'datacenter $objid' command\n"
+ if !defined($action);
+
+ $self->update_datacenter_setting($objid, $action, @params);
} else {
die "sim_hardware_cmd: unknown command '$cmdstr'\n";
}
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH pve-ha-manager 12/15] manager: log changes to crs settings
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (10 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 12:05 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
` (2 subsequent siblings)
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Especially with the addition of weights, it is important to be able to
trace how the load balancer is configured.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 89 +++++++++++++++++++++++++++++++++----------
1 file changed, 68 insertions(+), 21 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index c6818e2..4d0114c 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -99,6 +99,20 @@ sub new {
return $self;
}
+sub update_and_log_change {
+ my ($self, $curr, $next, $msg) = @_;
+
+ my $haenv = $self->{haenv};
+
+ if (defined($$curr) && $$curr ne $next) {
+ $haenv->log('info', "$msg from '$$curr' to '$next'");
+ }
+
+ $$curr = $next;
+
+ return;
+}
+
sub update_crs_scheduler_mode {
my ($self) = @_;
@@ -106,37 +120,70 @@ sub update_crs_scheduler_mode {
my $dc_cfg = $haenv->get_datacenter_settings();
my $crs_cfg = $dc_cfg->{crs} // {};
- $self->{crs}->{rebalance_on_request_start} = !!$crs_cfg->{'ha-rebalance-on-start'};
- $self->{crs}->{auto_rebalance}->{enable} = !!$crs_cfg->{'ha-auto-rebalance'};
+ my $log_prefix = "crs settings:";
+
+ $self->update_and_log_change(
+ \$self->{crs}->{rebalance_on_request_start},
+ !!$crs_cfg->{'ha-rebalance-on-start'},
+ "$log_prefix 'ha-rebalance-on-start' changed",
+ );
+
+ my $auto_rebalance = $self->{crs}->{auto_rebalance};
+
+ $self->update_and_log_change(
+ \$auto_rebalance->{enable},
+ !!$crs_cfg->{'ha-auto-rebalance'},
+ "$log_prefix 'ha-auto-rebalance' changed",
+ );
+
# configured as percent for UI/config consistency, scaled to [0, 1] to match the
# internal imbalance value that the comparison gates against.
- $self->{crs}->{auto_rebalance}->{threshold} =
- ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0;
- $self->{crs}->{auto_rebalance}->{method} = $crs_cfg->{'ha-auto-rebalance-method'}
- // 'bruteforce';
- $self->{crs}->{auto_rebalance}->{hold_duration} = $crs_cfg->{'ha-auto-rebalance-hold-duration'}
- // 3;
- $self->{crs}->{auto_rebalance}->{margin} =
- ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ $self->update_and_log_change(
+ \$auto_rebalance->{threshold},
+ ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0,
+ "$log_prefix 'ha-auto-rebalance-threshold' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{method},
+ $crs_cfg->{'ha-auto-rebalance-method'} // 'bruteforce',
+ "$log_prefix 'ha-auto-rebalance-method' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{hold_duration},
+ $crs_cfg->{'ha-auto-rebalance-hold-duration'} // 3,
+ "$log_prefix 'ha-auto-rebalance-hold-duration' changed",
+ );
+ $self->update_and_log_change(
+ \$auto_rebalance->{margin},
+ ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0,
+ "$log_prefix 'ha-auto-rebalance-margin' changed",
+ );
my $default_weight = 1;
- $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
+ $self->update_and_log_change(
+ \$auto_rebalance->{cpu_weight},
+ $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight,
+ "$log_prefix 'ha-auto-rebalance-cpu-weight' changed",
+ );
# by default TOPSIS weights mem higher than cpu
- $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
- $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
+ $default_weight = 5 if ($auto_rebalance->{method} eq 'topsis');
+ $self->update_and_log_change(
+ \$auto_rebalance->{mem_weight},
+ $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight,
+ "$log_prefix 'ha-auto-rebalance-mem-weight' changed",
+ );
- my $old_mode = $self->{crs}->{scheduler};
+ my $old_mode = \$self->{crs}->{scheduler};
my $new_mode = $crs_cfg->{ha} || 'basic';
- if (!defined($old_mode)) {
+ if (!defined($$old_mode)) {
$haenv->log('info', "using scheduler mode '$new_mode'") if $new_mode ne 'basic';
- } elsif ($new_mode eq $old_mode) {
- return; # nothing to do
- } else {
- $haenv->log('info', "switching scheduler mode from '$old_mode' to '$new_mode'");
}
-
- $self->{crs}->{scheduler} = $new_mode;
+ $self->update_and_log_change(
+ $old_mode,
+ $new_mode,
+ "$log_prefix scheduler mode changed"
+ );
return;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH pve-ha-manager 12/15] manager: log changes to crs settings
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
@ 2026-10-22 12:05 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 12:05 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Nice, thanks for adding this!
This is definitely helpful to have in the log so changes in the config
have a timestamp if one wants to triage a bug or follow what happened in
the past.
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Especially with the addition of weights, it is important to be able to
> trace how the load balancer is configured.
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> src/PVE/HA/Manager.pm | 89 +++++++++++++++++++++++++++++++++----------
> 1 file changed, 68 insertions(+), 21 deletions(-)
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index c6818e2..4d0114c 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -99,6 +99,20 @@ sub new {
> return $self;
> }
>
> +sub update_and_log_change {
> + my ($self, $curr, $next, $msg) = @_;
> +
> + my $haenv = $self->{haenv};
> +
> + if (defined($$curr) && $$curr ne $next) {
> + $haenv->log('info', "$msg from '$$curr' to '$next'");
> + }
> +
> + $$curr = $next;
> +
> + return;
> +}
It would be nice that this also logs what the value is set to at the
start when it is first set, i.e., when defined($$curr) is true and isn't
the default value, just as it's already done for the scheduler mode.
Would be nice to see this first factor out the logic already present for
the scheduler mode and then in a second patch use it on the other
options.
Furthermore, I think we can specialize the helper to crs settings for
now as long as we don't have any other parts to configure in the near
future.
Maybe we could go with a helper like
sub update_crs_option {
my ($self, $name, $optionref, $value, $default) = @_;
my $haenv = $self->{haenv};
my $old_value = $$optionref;
my $new_value = $value // $default;
if (!defined($old_value)) {
$haenv->log('info', "using $name '$new_value'") if $new_value ne $default;
} elsif ($old_value ne $new_value) {
$haenv->log('info', "switching $name from '$old_value' to '$new_value'");
$$optionref = $new_value;
}
}
and then
sub update_crs_scheduler_mode {
my ($self) = @_;
my $dc_cfg = $haenv->get_datacenter_settings();
my $crs_cfg = $dc_cfg->{crs} // {};
$self->update_crs_option(
'scheduler mode',
\$self->{crs}->{scheduler},
$crs_cfg->{ha},
'basic',
);
# ... and so on ...
}
s/switching/changing/ can be done in a separate patch if that makes it
more general for all kinds of options.
> +
> sub update_crs_scheduler_mode {
> my ($self) = @_;
>
> @@ -106,37 +120,70 @@ sub update_crs_scheduler_mode {
> my $dc_cfg = $haenv->get_datacenter_settings();
> my $crs_cfg = $dc_cfg->{crs} // {};
>
> - $self->{crs}->{rebalance_on_request_start} = !!$crs_cfg->{'ha-rebalance-on-start'};
> - $self->{crs}->{auto_rebalance}->{enable} = !!$crs_cfg->{'ha-auto-rebalance'};
> + my $log_prefix = "crs settings:";
> +
> + $self->update_and_log_change(
> + \$self->{crs}->{rebalance_on_request_start},
> + !!$crs_cfg->{'ha-rebalance-on-start'},
> + "$log_prefix 'ha-rebalance-on-start' changed",
> + );
> +
> + my $auto_rebalance = $self->{crs}->{auto_rebalance};
> +
> + $self->update_and_log_change(
> + \$auto_rebalance->{enable},
> + !!$crs_cfg->{'ha-auto-rebalance'},
> + "$log_prefix 'ha-auto-rebalance' changed",
> + );
> +
> # configured as percent for UI/config consistency, scaled to [0, 1] to match the
> # internal imbalance value that the comparison gates against.
> - $self->{crs}->{auto_rebalance}->{threshold} =
> - ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0;
> - $self->{crs}->{auto_rebalance}->{method} = $crs_cfg->{'ha-auto-rebalance-method'}
> - // 'bruteforce';
> - $self->{crs}->{auto_rebalance}->{hold_duration} = $crs_cfg->{'ha-auto-rebalance-hold-duration'}
> - // 3;
> - $self->{crs}->{auto_rebalance}->{margin} =
> - ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
> + $self->update_and_log_change(
> + \$auto_rebalance->{threshold},
> + ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0,
> + "$log_prefix 'ha-auto-rebalance-threshold' changed",
> + );
> + $self->update_and_log_change(
> + \$auto_rebalance->{method},
> + $crs_cfg->{'ha-auto-rebalance-method'} // 'bruteforce',
> + "$log_prefix 'ha-auto-rebalance-method' changed",
> + );
> + $self->update_and_log_change(
> + \$auto_rebalance->{hold_duration},
> + $crs_cfg->{'ha-auto-rebalance-hold-duration'} // 3,
> + "$log_prefix 'ha-auto-rebalance-hold-duration' changed",
> + );
> + $self->update_and_log_change(
> + \$auto_rebalance->{margin},
> + ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0,
> + "$log_prefix 'ha-auto-rebalance-margin' changed",
> + );
>
> my $default_weight = 1;
> - $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
> + $self->update_and_log_change(
> + \$auto_rebalance->{cpu_weight},
> + $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight,
> + "$log_prefix 'ha-auto-rebalance-cpu-weight' changed",
> + );
> # by default TOPSIS weights mem higher than cpu
> - $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
> - $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
> + $default_weight = 5 if ($auto_rebalance->{method} eq 'topsis');
> + $self->update_and_log_change(
> + \$auto_rebalance->{mem_weight},
> + $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight,
> + "$log_prefix 'ha-auto-rebalance-mem-weight' changed",
> + );
>
> - my $old_mode = $self->{crs}->{scheduler};
> + my $old_mode = \$self->{crs}->{scheduler};
> my $new_mode = $crs_cfg->{ha} || 'basic';
>
> - if (!defined($old_mode)) {
> + if (!defined($$old_mode)) {
> $haenv->log('info', "using scheduler mode '$new_mode'") if $new_mode ne 'basic';
> - } elsif ($new_mode eq $old_mode) {
> - return; # nothing to do
> - } else {
> - $haenv->log('info', "switching scheduler mode from '$old_mode' to '$new_mode'");
> }
> -
> - $self->{crs}->{scheduler} = $new_mode;
> + $self->update_and_log_change(
> + $old_mode,
> + $new_mode,
> + "$log_prefix scheduler mode changed"
Even though we don't have any test cases for this yet, which would catch
that the log messages for switching the scheduler mode changed, we
should make sure to not change them as long as feasible.
Otherwise, the scheduler mode should probably be logged first before all
of the other options.
> + );
>
> return;
> }
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (11 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-22 12:20 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight Dominik Rusovac
14 siblings, 1 reply; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add tests documenting basic efficacy of auto rebalancing with weights.
Overview:
- test-crs-dynamic-auto-rebalance-with-increasing-bias: no rebalancing,
increment mem-weight until rebalancing is triggered
- test-crs-dynamic-auto-rebalance-with-{cpu,mem}-bias: trigger migration
that rebalances {cpu,mem} by incrementing {cpu,mem}-weight once
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
.../README | 5 ++
.../cmdlist | 6 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 62 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 4 ++
.../cmdlist | 8 +++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 66 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 5 ++
.../cmdlist | 6 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 62 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
27 files changed, 332 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
new file mode 100644
index 0000000..f67978e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
@@ -0,0 +1,5 @@
+At equal weights (=1) load imbalance sits at 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing the
+ha-auto-rebalance-cpu-weight will increase load imbalance and trigger a
+rebalancing migration from cpu-heavy node3 to node2, which sits low on either
+resource.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
new file mode 100644
index 0000000..c29b4b6
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
new file mode 100644
index 0000000..352aa1c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
@@ -0,0 +1,62 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-cpu-weight' changed from '1' to '2'
+info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%)
+info 360 node1/crm: got crm command: migrate vm:106 node2
+info 360 node1/crm: migrate service 'vm:106' to node 'node2'
+info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 365 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 365 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:106
+info 383 node2/lrm: service status vm:106 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
new file mode 100644
index 0000000..a9974a7
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
@@ -0,0 +1,4 @@
+At equal weights (=1) load imbalance sits at 18.7%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that incrementing
+ha-auto-rebalance-mem-weight will eventually trigger a rebalancing migration
+from mem-heavy node3.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
new file mode 100644
index 0000000..643ab0b
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
@@ -0,0 +1,8 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 3" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 4" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
new file mode 100644
index 0000000..14740f1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 10.0, "mem": 3221225472 },
+ "vm:102": { "cpu": 5.0, "mem": 1073741824 },
+ "vm:103": { "cpu": 5.0, "mem": 3221225472 },
+ "vm:104": { "cpu": 3.0, "mem": 1073741824 },
+ "vm:105": { "cpu": 0.5, "mem": 32212254720 },
+ "vm:106": { "cpu": 0.5, "mem": 6442450944 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
new file mode 100644
index 0000000..596ef29
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
@@ -0,0 +1,66 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
+info 420 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 3
+info 420 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '2' to '3'
+info 520 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 4
+info 520 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '3' to '4'
+info 560 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%)
+info 560 node1/crm: got crm command: migrate vm:106 node2
+info 560 node1/crm: migrate service 'vm:106' to node 'node2'
+info 560 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 565 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 565 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 580 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 583 node2/lrm: starting service vm:106
+info 583 node2/lrm: service status vm:106 started
+info 1120 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
new file mode 100644
index 0000000..19cebf9
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
@@ -0,0 +1,5 @@
+At equal weights (=1) load imbalance sits at 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing the
+ha-auto-rebalance-mem-weight will increase load imbalance and trigger a
+rebalancing migration from mem-heavy node1 to node2, which sits low on either
+resource.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
new file mode 100644
index 0000000..2144ffb
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
+ [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
new file mode 100644
index 0000000..5fc901f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
@@ -0,0 +1,62 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
+info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
+info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
+info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
+info 360 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%)
+info 360 node1/crm: got crm command: migrate vm:102 node2
+info 360 node1/crm: migrate service 'vm:102' to node 'node2'
+info 360 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 361 node1/lrm: service vm:102 - start migrate to node 'node2'
+info 361 node1/lrm: service vm:102 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:102
+info 383 node2/lrm: service status vm:102 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
@ 2026-10-22 12:20 ` Daniel Kral
0 siblings, 0 replies; 24+ messages in thread
From: Daniel Kral @ 2026-10-22 12:20 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
Nice test cases!
Would be nice to also have test cases for the case where one of them or
even both of them are set to 0, but these are already nice to have.
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Add tests documenting basic efficacy of auto rebalancing with weights.
>
> Overview:
> - test-crs-dynamic-auto-rebalance-with-increasing-bias: no rebalancing,
> increment mem-weight until rebalancing is triggered
> - test-crs-dynamic-auto-rebalance-with-{cpu,mem}-bias: trigger migration
> that rebalances {cpu,mem} by incrementing {cpu,mem}-weight once
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> .../README | 5 ++
> .../cmdlist | 6 ++
> .../datacenter.cfg | 6 ++
> .../dynamic_service_stats | 8 +++
> .../hardware_status | 5 ++
> .../log.expect | 62 +++++++++++++++++
> .../manager_status | 1 +
> .../service_config | 8 +++
> .../static_service_stats | 8 +++
> .../README | 4 ++
> .../cmdlist | 8 +++
> .../datacenter.cfg | 6 ++
> .../dynamic_service_stats | 8 +++
> .../hardware_status | 5 ++
> .../log.expect | 66 +++++++++++++++++++
> .../manager_status | 1 +
> .../service_config | 8 +++
> .../static_service_stats | 8 +++
> .../README | 5 ++
> .../cmdlist | 6 ++
> .../datacenter.cfg | 6 ++
> .../dynamic_service_stats | 8 +++
> .../hardware_status | 5 ++
> .../log.expect | 62 +++++++++++++++++
> .../manager_status | 1 +
> .../service_config | 8 +++
> .../static_service_stats | 8 +++
> 27 files changed, 332 insertions(+)
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
> create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
>
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
> new file mode 100644
> index 0000000..f67978e
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
> @@ -0,0 +1,5 @@
> +At equal weights (=1) load imbalance sits at 27.5%, which is below the
> +ha-auto-rebalance-threshold of 30%. Test that increasing the
> +ha-auto-rebalance-cpu-weight will increase load imbalance and trigger a
> +rebalancing migration from cpu-heavy node3 to node2, which sits low on either
> +resource.
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
> new file mode 100644
> index 0000000..c29b4b6
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
> @@ -0,0 +1,6 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on" ],
> + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
> + [ "datacenter crs set ha-auto-rebalance-cpu-weight 2" ]
Wouldn't it suffice to only issue the last command?
Each command list is always executed 5 HA Manager rounds apart, so
with the default hold duration of 3 rounds it would already have done
something before setting the {cpu,mem}-weight to 1 explicitly.
If we also log the first set value as noted in the crs option log patch,
then it would also show up in the log here if it's set to something
different than the default value.
> +]
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
> new file mode 100644
> index 0000000..01c8114
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
> @@ -0,0 +1,6 @@
> +{
> + "crs": {
> + "ha": "dynamic",
> + "ha-auto-rebalance": 1
> + }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
> new file mode 100644
> index 0000000..0d0b3b2
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "cpu": 1.5, "mem": 42949672960 },
> + "vm:102": { "cpu": 0.5, "mem": 21474836480 },
> + "vm:103": { "cpu": 2.0, "mem": 8589934592 },
> + "vm:104": { "cpu": 1.0, "mem": 4294967296 },
> + "vm:105": { "cpu": 9.0, "mem": 5368709120 },
> + "vm:106": { "cpu": 6.0, "mem": 3221225472 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
> new file mode 100644
> index 0000000..4395b83
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
> @@ -0,0 +1,5 @@
> +{
> + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
> new file mode 100644
> index 0000000..352aa1c
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
> @@ -0,0 +1,62 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: using scheduler mode 'dynamic'
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:104' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:105' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:106' on node 'node3'
> +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
> +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
> +info 21 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 21 node1/lrm: status change wait_for_agent_lock => active
> +info 21 node1/lrm: starting service vm:101
> +info 21 node1/lrm: service status vm:101 started
> +info 21 node1/lrm: starting service vm:102
> +info 21 node1/lrm: service status vm:102 started
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 23 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 23 node2/lrm: status change wait_for_agent_lock => active
> +info 23 node2/lrm: starting service vm:103
> +info 23 node2/lrm: service status vm:103 started
> +info 23 node2/lrm: starting service vm:104
> +info 23 node2/lrm: service status vm:104 started
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:105
> +info 25 node3/lrm: service status vm:105 started
> +info 25 node3/lrm: starting service vm:106
> +info 25 node3/lrm: service status vm:106 started
> +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
> +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
> +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 2
> +info 320 node1/crm: crs settings: 'ha-auto-rebalance-cpu-weight' changed from '1' to '2'
> +info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%)
> +info 360 node1/crm: got crm command: migrate vm:106 node2
> +info 360 node1/crm: migrate service 'vm:106' to node 'node2'
> +info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
> +info 365 node3/lrm: service vm:106 - start migrate to node 'node2'
> +info 365 node3/lrm: service vm:106 - end migrate to node 'node2'
> +info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
> +info 383 node2/lrm: starting service vm:106
> +info 383 node2/lrm: service status vm:106 started
> +info 920 hardware: exit simulation - done
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
> new file mode 100644
> index 0000000..60cadca
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "node": "node1", "state": "started" },
> + "vm:102": { "node": "node1", "state": "started" },
> + "vm:103": { "node": "node2", "state": "started" },
> + "vm:104": { "node": "node2", "state": "started" },
> + "vm:105": { "node": "node3", "state": "started" },
> + "vm:106": { "node": "node3", "state": "started" }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
> new file mode 100644
> index 0000000..9fcd31f
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
> new file mode 100644
> index 0000000..a9974a7
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
> @@ -0,0 +1,4 @@
> +At equal weights (=1) load imbalance sits at 18.7%, which is below the
> +ha-auto-rebalance-threshold of 30%. Test that incrementing
> +ha-auto-rebalance-mem-weight will eventually trigger a rebalancing migration
> +from mem-heavy node3.
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
> new file mode 100644
> index 0000000..643ab0b
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
> @@ -0,0 +1,8 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on" ],
> + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
Same here, these seem redundant for the test case alone
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ],
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 3" ],
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 4" ]
> +]
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
> new file mode 100644
> index 0000000..01c8114
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
> @@ -0,0 +1,6 @@
> +{
> + "crs": {
> + "ha": "dynamic",
> + "ha-auto-rebalance": 1
> + }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
> new file mode 100644
> index 0000000..14740f1
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "cpu": 10.0, "mem": 3221225472 },
> + "vm:102": { "cpu": 5.0, "mem": 1073741824 },
> + "vm:103": { "cpu": 5.0, "mem": 3221225472 },
> + "vm:104": { "cpu": 3.0, "mem": 1073741824 },
> + "vm:105": { "cpu": 0.5, "mem": 32212254720 },
> + "vm:106": { "cpu": 0.5, "mem": 6442450944 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
> new file mode 100644
> index 0000000..4395b83
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
> @@ -0,0 +1,5 @@
> +{
> + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
> new file mode 100644
> index 0000000..596ef29
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
> @@ -0,0 +1,66 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: using scheduler mode 'dynamic'
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:104' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:105' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:106' on node 'node3'
> +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
> +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
> +info 21 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 21 node1/lrm: status change wait_for_agent_lock => active
> +info 21 node1/lrm: starting service vm:101
> +info 21 node1/lrm: service status vm:101 started
> +info 21 node1/lrm: starting service vm:102
> +info 21 node1/lrm: service status vm:102 started
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 23 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 23 node2/lrm: status change wait_for_agent_lock => active
> +info 23 node2/lrm: starting service vm:103
> +info 23 node2/lrm: service status vm:103 started
> +info 23 node2/lrm: starting service vm:104
> +info 23 node2/lrm: service status vm:104 started
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:105
> +info 25 node3/lrm: service status vm:105 started
> +info 25 node3/lrm: starting service vm:106
> +info 25 node3/lrm: service status vm:106 started
> +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
> +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
> +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
> +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
> +info 420 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 3
> +info 420 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '2' to '3'
> +info 520 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 4
> +info 520 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '3' to '4'
> +info 560 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%)
> +info 560 node1/crm: got crm command: migrate vm:106 node2
> +info 560 node1/crm: migrate service 'vm:106' to node 'node2'
> +info 560 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
> +info 565 node3/lrm: service vm:106 - start migrate to node 'node2'
> +info 565 node3/lrm: service vm:106 - end migrate to node 'node2'
> +info 580 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
> +info 583 node2/lrm: starting service vm:106
> +info 583 node2/lrm: service status vm:106 started
> +info 1120 hardware: exit simulation - done
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
> new file mode 100644
> index 0000000..60cadca
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "node": "node1", "state": "started" },
> + "vm:102": { "node": "node1", "state": "started" },
> + "vm:103": { "node": "node2", "state": "started" },
> + "vm:104": { "node": "node2", "state": "started" },
> + "vm:105": { "node": "node3", "state": "started" },
> + "vm:106": { "node": "node3", "state": "started" }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
> new file mode 100644
> index 0000000..9fcd31f
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
> new file mode 100644
> index 0000000..19cebf9
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
> @@ -0,0 +1,5 @@
> +At equal weights (=1) load imbalance sits at 27.5%, which is below the
> +ha-auto-rebalance-threshold of 30%. Test that increasing the
> +ha-auto-rebalance-mem-weight will increase load imbalance and trigger a
> +rebalancing migration from mem-heavy node1 to node2, which sits low on either
> +resource.
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
> new file mode 100644
> index 0000000..2144ffb
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
> @@ -0,0 +1,6 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on" ],
> + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ],
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ],
same here
> + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ]
> +]
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
> new file mode 100644
> index 0000000..01c8114
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
> @@ -0,0 +1,6 @@
> +{
> + "crs": {
> + "ha": "dynamic",
> + "ha-auto-rebalance": 1
> + }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
> new file mode 100644
> index 0000000..0d0b3b2
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "cpu": 1.5, "mem": 42949672960 },
> + "vm:102": { "cpu": 0.5, "mem": 21474836480 },
> + "vm:103": { "cpu": 2.0, "mem": 8589934592 },
> + "vm:104": { "cpu": 1.0, "mem": 4294967296 },
> + "vm:105": { "cpu": 9.0, "mem": 5368709120 },
> + "vm:106": { "cpu": 6.0, "mem": 3221225472 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
> new file mode 100644
> index 0000000..4395b83
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
> @@ -0,0 +1,5 @@
> +{
> + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
> + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
> new file mode 100644
> index 0000000..5fc901f
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
> @@ -0,0 +1,62 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: using scheduler mode 'dynamic'
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:104' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:105' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:106' on node 'node3'
> +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
> +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
> +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
> +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
> +info 21 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 21 node1/lrm: status change wait_for_agent_lock => active
> +info 21 node1/lrm: starting service vm:101
> +info 21 node1/lrm: service status vm:101 started
> +info 21 node1/lrm: starting service vm:102
> +info 21 node1/lrm: service status vm:102 started
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 23 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 23 node2/lrm: status change wait_for_agent_lock => active
> +info 23 node2/lrm: starting service vm:103
> +info 23 node2/lrm: service status vm:103 started
> +info 23 node2/lrm: starting service vm:104
> +info 23 node2/lrm: service status vm:104 started
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:105
> +info 25 node3/lrm: service status vm:105 started
> +info 25 node3/lrm: starting service vm:106
> +info 25 node3/lrm: service status vm:106 started
> +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1
> +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1
> +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2
> +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2'
> +info 360 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%)
> +info 360 node1/crm: got crm command: migrate vm:102 node2
> +info 360 node1/crm: migrate service 'vm:102' to node 'node2'
> +info 360 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2)
> +info 361 node1/lrm: service vm:102 - start migrate to node 'node2'
> +info 361 node1/lrm: service vm:102 - end migrate to node 'node2'
> +info 380 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2)
> +info 383 node2/lrm: starting service vm:102
> +info 383 node2/lrm: service status vm:102 started
> +info 920 hardware: exit simulation - done
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
> new file mode 100644
> index 0000000..60cadca
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "node": "node1", "state": "started" },
> + "vm:102": { "node": "node1", "state": "started" },
> + "vm:103": { "node": "node2", "state": "started" },
> + "vm:104": { "node": "node2", "state": "started" },
> + "vm:105": { "node": "node3", "state": "started" },
> + "vm:106": { "node": "node3", "state": "started" }
> +}
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
> new file mode 100644
> index 0000000..9fcd31f
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
> @@ -0,0 +1,8 @@
> +{
> + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
> + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
> +}
^ permalink raw reply [flat|nested] 24+ messages in thread
* [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (12 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight Dominik Rusovac
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Rename to 'update_crs_settings' to be more precise and avoid confusion
with CRS 'Scheduling Mode'.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/HA/Manager.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 4d0114c..78d4952 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -94,7 +94,7 @@ sub new {
# preserve disarm state across CRM master changes
$self->{ms}->{disarm} = $old_ms->{disarm} if defined($old_ms->{disarm});
- $self->update_crs_scheduler_mode(); # initial set, we update it once every loop
+ $self->update_crs_settings(); # initial set, we update it once every loop
return $self;
}
@@ -113,7 +113,7 @@ sub update_and_log_change {
return;
}
-sub update_crs_scheduler_mode {
+sub update_crs_settings {
my ($self) = @_;
my $haenv = $self->{haenv};
@@ -1108,7 +1108,7 @@ sub manage {
return;
}
- $self->update_crs_scheduler_mode();
+ $self->update_crs_settings();
$self->try_persistent_group_migration();
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
` (13 preceding siblings ...)
2026-10-21 8:59 ` [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode Dominik Rusovac
@ 2026-10-21 8:59 ` Dominik Rusovac
14 siblings, 0 replies; 24+ messages in thread
From: Dominik Rusovac @ 2026-10-21 8:59 UTC (permalink / raw)
To: pve-devel
Add weights for controlling how much balancing out the CPU load,
respectively, memory usage, of nodes should be prioritized.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
src/PVE/DataCenterConfig.pm | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 004122e..19dec7c 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -78,6 +78,22 @@ EODESC
description => "The minimum relative improvement in cluster node imbalance,"
. " in percent, to commit to a resource balancing migration.",
},
+ 'ha-auto-rebalance-cpu-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ requires => 'ha-auto-rebalance',
+ description => "The weight controlling how much balancing out"
+ . " the CPU load of nodes should be prioritized.",
+ },
+ 'ha-auto-rebalance-mem-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ requires => 'ha-auto-rebalance',
+ description => "The weight controlling how much balancing out"
+ . " the memory usage of nodes should be prioritized",
+ },
};
my $migration_format = {
--
2.47.3
^ permalink raw reply related [flat|nested] 24+ messages in thread