* [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing
@ 2026-11-12 9:44 Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
` (18 more replies)
0 siblings, 19 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
Let users express a preference for prioritizing CPU load balancing over memory
usage balancing, or vice versa, by means of weights.
Tested basic functionality in virtualized 3-node cluster. Added property tests,
unit tests, and test cases documenting the effect of the weights on automatic
rebalancing.
Will follow this up with a separate series updating CRS docs accordingly, once
this one is considered.
# TL;DR
This series enables weighting CPU load against memory usage when computing node
load. The node load becomes the weighted arithmetic mean of both (see Details).
Only the weight ratio matters, and equal weights reproduce the old plain mean.
This addresses cases [0] where the load balancer didn't act because a blended
load imbalance looked fine even though the cluster was rather imbalanced in CPU
load or memory usage alone.
The weights are configured with the new `ha-cpu-weight` and `ha-mem-weight` CRS
options. They apply to the static and dynamic CRS modes, that is, to the load
imbalance, both automatic rebalancing methods, and the selection of a node to
start or recover a resource on.
## Bruteforce
- a higher weight on a resource steers migrations toward balancing that
resource; this is a tendency, not a guarantee
- a migration that leaves the variance of both resources higher can still look
better balanced, because CPU and memory deviations can cancel out (negative
covariance); weights don't disable this
- load imbalance values computed with different weights, and the thresholds
they are compared against, aren't directly comparable
## TOPSIS
- the weights replace the fixed criteria weights; weights of 1:5 reproduce
the previous criteria weights
- the weight of the highest usage criterion of a resource stays twice the
weight of its average usage criterion, which keeps the ratio introduced in
commit 2b325aa5 ("add pve_static module")
## Defaults and compatibility
- without configured weights, the bruteforce method and the load imbalance
keep using equal weights, and TOPSIS keeps its previous ranking (1:5)
- with TOPSIS as automatic rebalancing method and no configured weights, the
HA manager weighs the load imbalance 1:5 as well, so that it reflects what
TOPSIS balances; for these setups, the imbalance is dominated by memory
usage after upgrading
- HA manager logs the weights it passes to the scheduler, so the implicit
1:5 shows up in the log once weights are configured or the method changes
- older HA managers don't pass weights and get the same results as before from
the new bindings (see Compatibility)
# Details
A weight is any non-negative integer. Weights are blended into the computation
of node loads. The load of a node is currently the arithmetic mean of the
node's CPU load `cpu(node)` and memory usage `mem(node)`:
(F1) (cpu(node) + mem(node)) / 2
This series replaces F1 with the weighted arithmetic mean [1]:
(F2) (cpu(node) * cpu_w + mem(node) * mem_w) / (cpu_w + mem_w)
`cpu_w` controls how much `cpu(node)` contributes to F2, analogously, `mem_w`
controls how much `mem(node)` contributes to F2; the higher the weight, the
more that resource dominates the result of F2.
NOTE:
- a weight of 0 means no contribution to F2;
- only the weights' ratio matters, not their absolute magnitude; and
- F2 reduces exactly to the plain arithmetic mean F1 whenever `cpu_w` and
`mem_w` are equal (not just when both equal 1).
Let `X` be the population of loads according to F2 with weights `cpu_w` and
`mem_w` of all nodes in a cluster consisting of `n` nodes. Since weights
`cpu_w` and `mem_w` affect the result of F2, they also shape how sensitively
the load imbalance computation:
(F3) coefficient_of_variation(X) / sqrt(n - 1)
reacts to CPU load changes relative to memory usage changes. Accordingly, this
series enables users to control how much a given change in CPU load or memory
usage moves the imbalance metric by means of weights. This is to address: "In
our case CRS has not moved any VM around because the imbalance percentage is
low (7%). Eventhough the cluster is very imbalance [sic] CPU wise, or memory
wise alone." [0]
Weights of course also factor into the load balancer's decision, the 'how'
depends on which method is used.
## Bruteforce Method
The higher the weight of a computational resource, the more the search will
tend to prioritize balancing out that resource, e.g., a much greater 'cpu_w'
than 'mem_w' will steer migrations toward evening out CPU load, even at the
cost of leaving memory usage less balanced. This is a general tendency, not a
guarantee. The following deep(er) dive into the load imbalance formula gives
more insights.
Recall F3. Note that
(F4) coefficient_of_variation(X) := standard_deviation(X) / mean(X)
(F5) standard_deviation(X) := sqrt(variance(X))
Hence, `variance(X)` is the part of F3 that captures how dispersed the node
loads are. Define `C := [cpu(x) | x in cluster]` and `M := [mem(x) | x in cluster]`.
In fact, one can derive that `variance(X)` can be rewritten as:
(F6) (cpu_t + mem_t + cov_t) / (cpu_w + mem_w)^2
where
cpu_t := cpu_w^2 * variance(C)
mem_t := mem_w^2 * variance(M)
cov_t := 2 * cpu_w * mem_w * covariance(M,C)
F6 dissects how the weights shape the variance part of the load imbalance.
`cpu_t` and `mem_t` relate to the respective resource's dispersion weighted by
the square of its weight (note that doubling a weight roughly quadruples that
term's pull on the total). Note that `variance(C)` as part of `cpu_t` and
`variance(M)` as part of `mem_t` are each an average of squared deviations [3],
so they can never be negative, and neither can the squared weights; hence
`cpu_t` and `mem_t` can only ever add to `variance(X)`, never subtract from it.
They are also where each weight acts on its own resource's dispersion, and this
is the mechanism behind letting users express a preference for balancing one
computational resource over the other. Because `cpu_t` scales with `cpu_w^2`
and `mem_t` scales with `mem_w^2`, raising `cpu_w` relative to `mem_w` makes
`variance(C)`'s own dispersion count for a disproportionately larger share of
the numerator of F6, which steers the bruteforce method toward candidates that
keep CPU load tight, even at the cost of leaving memory usage more spread out.
This is the sense in which a higher `cpu_w` means "prioritize balancing CPU
load": not that memory usage is ignored, but that `mem_t`'s pull on the
numerator shrinks relative to `cpu_t`'s as the weight ratio skews toward CPU
load.
`cov_t` is the only one of the three terms that can be negative, which makes it
the only lever capable of pulling `variance(X)` below what the two resources'
own dispersions would otherwise force it to be. A rebalancing migration
can exploit this by arranging `cpu(·)` and `mem(·)` to deviate from their means
in opposite directions across nodes. A node that ends up above-average on
`mem(·)` tends to end up below-average on `cpu(·)`, and vice versa. Averaged
into F2 per node, those two errors partially cancel out, so `variance(X)` (and
hence the reported load imbalance) can come out looking well-balanced even when
neither `variance(C)` nor `variance(M)` individually is. One caveat that
follows directly: weighting a resource more heavily increases its pull on
`variance(X)`, but it doesn't disable `cov_t`. A candidate with a sufficiently
favorable (negative) covariance can still beat one that's honestly better on
the heavily-weighted resource's own dispersion. In short, weighting changes how
much a resource's balance matters to the outcome, but it does not make bruteforce
immune to the cancellation mechanism described above.
Besides `variance(X)`, the weights also enter `mean(X)`, the denominator of F4.
By definition, `mean(X)` is the plain (unweighted) arithmetic mean over the
nodes of the weighted node loads, that is, the sum of the loads according to F2
divided by the number of nodes `n`. The weights act across the two resources
within each node, not across nodes. In fact, `mean(X)` can be rewritten as the
weighted arithmetic mean of the two resources' mean loads:
(F7) (cpu_w * mean(C) + mem_w * mean(M)) / (cpu_w + mem_w)
The result of F7 lies between `mean(C)` and `mean(M)` and moves toward the mean
of the more heavily weighted resource. Scaling both weights by the same factor
leaves it unchanged.
Note that a single rebalancing migration only redistributes computational
resources between nodes, it never changes the cluster-wide total CPU load or
memory usage. Hence, for nodes of equal capacity (homogeneous cluster),
`mean(C)`, `mean(M)` and thus `mean(X)` are the same constant for every
migration candidate at fixed weights: the mean scales the load imbalance (F3)
but does not influence which candidate has the lowest one, which is why the
weights' steering effect on the ranking is dissected by F6. Two caveats follow:
- Different weights: Both `variance(X)` (F6) and `mean(X)` (F7) change with the
choice of weights, so the same state of the cluster yields different load
imbalance values for different weights, in particular if `mean(C)` and
`mean(M)` or `variance(C)` and `variance(M)` differ considerably. Values
computed with different weights, and thresholds they are compared against, are
therefore not directly comparable.
- Nodes of different capacity (heterogeneous cluster): The same amount of a
resource corresponds to a different fraction of the capacity of different
nodes, so migrating it to different targets can change the sum of node loads.
Then `mean(C)`, `mean(M)` and `mean(X)` can differ between migration
candidates, and the mean influences which candidate has the lowest load
imbalance in addition to `variance(X)`. F6 remains valid but in this case only
explains the variance part of the ranking.
## TOPSIS Method
Weights have always been part of the TOPSIS [2] method, weighing the criteria
of the decision process. Previously, all of these weights were fixed and
favored memory usage 5 times over CPU load. With this series, the configured
weights replace them directly: `cpu_w` is the weight of the average CPU
criterion and `mem_w` the weight of the average memory criterion. The weights
of highest CPU and highest memory are not independently configurable; each is
twice the weight of its average counterpart. This makes it possible for users
to prioritize balancing a specific computational resource while also preserving
the ratio between a resource's two criteria, as introduced in commit 2b325aa5
("add pve_static module"). Configured weights of 1:5 reproduce the previous
criteria weights 1, 2, 5, 10.
Scaling the fixed weights with the configured ones instead would have made the
configured weights relative to the built-in emphasis on memory, so that 1:1
would effectively mean 1:5.
The same weights are used for the TOPSIS criteria and the node load, so the
imbalance reported for the migrations ranked by TOPSIS is weighted like the
criteria that ranked them, and it is compared to a load imbalance weighted the
same way.
Unlike for the bruteforce method, TOPSIS' per-resource criteria never combine
CPU load and memory usage into a single quantity, so there is no known
cancellation-style lever to be aware of here.
# Compatibility
As suggested in the v1 review, the weights are part of the scheduler struct in
proxmox-resource-scheduling and are passed once to the constructor of the
pve-rs bindings as optional scheduler options, instead of to every method. So,
signatures of the existing methods stay the same.
The crate defaults to equal weights, which changes the default TOPSIS ranking.
Callers keep the previous ranking with hardcoded legacy topsis weights.
The bindings keep the previous behavior of every method if no weights are
passed: the load imbalance and the bruteforce method use equal weights, TOPSIS
uses 1:5. The HA manager compares the imbalance reported for the migrations
ranked by TOPSIS with the current load imbalance. To keep these comparable for
older HA managers, which don't pass weights, the bindings recompute the
imbalance of the ranked migrations with equal weights in that case, as before.
Hence, the new bindings shouldn't break older HA manager versions.
Newer HA managers need a versioned dependency on the new bindings, as older
bindings reject the options, which makes the HA manager fall back to the
'basic' CRS mode.
If both weights are set to 0, no resource contributes to the node load. The HA
manager then warns once, skips automatic rebalancing, uses the weights for
selecting nodes to start or recover resources on, and computes the load
imbalance using 1:1, as before.
# Summary of main changes since v1
- weights are part of the scheduler instead of being passed to each of its
methods
- the weights apply to selecting a node to start or recover a resource on as
well, so the options were renamed to `ha-cpu-weight` and `ha-mem-weight`
- weights are passed as optional scheduler options on construction, so older HA
managers keep working
- factor out the CRS option logging into a helper, first for the scheduler
mode, keeping its log messages, then for the other options; values differing
from the defaults are logged on startup
- log the implicit 1:5 weights for TOPSIS
- skip automatic rebalancing and warn once if both weights are 0
- in the ui the weights are enabled for the static and dynamic modes regardless
of automatic rebalancing, and are advanced options together with the hold
duration and the minimum imbalance improvement
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
[1] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
[2] https://en.wikipedia.org/wiki/TOPSIS
[3] https://en.wikipedia.org/wiki/Variance
proxmox:
Dominik Rusovac (8):
resource-scheduling: fix typos
resource-scheduling: scheduler: refine score_* docs
resource-scheduling: scheduler: abstract post-migration stats logic
fix #7993: resource-scheduling: node: add weighted node load
resource-scheduling: scheduler: make imbalance with candidate public
fix #7993: resource-scheduling: scheduler: add weights
workspace,resource-scheduling: add proptest as dev-dependency
resource-scheduling: scheduler: add property tests for weights
Cargo.toml | 1 +
proxmox-resource-scheduling/Cargo.toml | 3 +
proxmox-resource-scheduling/src/node.rs | 114 ++++
proxmox-resource-scheduling/src/pve_static.rs | 4 +-
proxmox-resource-scheduling/src/scheduler.rs | 485 ++++++++++++++++--
proxmox-resource-scheduling/src/topsis.rs | 2 +-
.../tests/scheduler.rs | 64 ++-
7 files changed, 616 insertions(+), 57 deletions(-)
proxmox-perl-rs:
Dominik Rusovac (1):
fix #7993: pve-rs: resource-scheduling: add scheduler options
.../src/bindings/resource_scheduling/mod.rs | 1 +
.../bindings/resource_scheduling/options.rs | 59 ++++++++
.../resource_scheduling/pve_dynamic.rs | 71 +++++++--
.../resource_scheduling/pve_static.rs | 64 ++++++--
pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++
5 files changed, 304 insertions(+), 32 deletions(-)
create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
pve-cluster:
Dominik Rusovac (1):
fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options
src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
pve-manager:
Dominik Rusovac (2):
ui: form: crs options: make fine-tuning options advanced
fix #7993: ui: form: crs options: add weights
www/manager6/form/CRSOptions.js | 47 +++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
pve-ha-manager:
Dominik Rusovac (7):
manager: remove obsolete argument
manager: rename update_crs_scheduler_mod to update_crs_settings
manager: factor out crs option update helper
manager: log changes to crs settings
sim: hardware: allow changing datacenter settings at runtime
fix #7993: manager,usage: use weights for static and dynamic mode
test: add automatic rebalancing test cases with weights
src/PVE/HA/Manager.pm | 140 ++++++++++++++----
src/PVE/HA/Sim/Hardware.pm | 62 ++++++++
src/PVE/HA/Usage/Dynamic.pm | 6 +-
src/PVE/HA/Usage/Static.pm | 6 +-
.../log.expect | 2 +
.../log.expect | 2 +
.../log.expect | 15 +-
.../log.expect | 3 +
.../README | 4 +
.../cmdlist | 4 +
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 8 +
.../hardware_status | 5 +
.../log.expect | 61 ++++++++
.../manager_status | 1 +
.../service_config | 8 +
.../static_service_stats | 8 +
.../README | 4 +
.../cmdlist | 6 +
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 8 +
.../hardware_status | 5 +
.../log.expect | 65 ++++++++
.../manager_status | 1 +
.../service_config | 8 +
.../static_service_stats | 8 +
.../README | 5 +
.../cmdlist | 4 +
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 8 +
.../hardware_status | 5 +
.../log.expect | 61 ++++++++
.../manager_status | 1 +
.../service_config | 8 +
.../static_service_stats | 8 +
.../README | 5 +
.../cmdlist | 4 +
.../datacenter.cfg | 7 +
.../dynamic_service_stats | 8 +
.../hardware_status | 5 +
.../log.expect | 62 ++++++++
.../manager_status | 1 +
.../service_config | 8 +
.../static_service_stats | 8 +
.../README | 5 +
.../cmdlist | 4 +
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 8 +
.../hardware_status | 5 +
.../log.expect | 61 ++++++++
.../manager_status | 1 +
.../service_config | 8 +
.../static_service_stats | 8 +
.../README | 6 +
.../cmdlist | 3 +
.../datacenter.cfg | 8 +
.../dynamic_service_stats | 6 +
.../hardware_status | 5 +
.../log.expect | 41 +++++
.../manager_status | 1 +
.../service_config | 6 +
.../static_service_stats | 6 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 2 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 2 +
.../log.expect | 1 +
.../test-crs-dynamic-rebalance1/log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 4 +
.../test-crs-static-rebalance1/log.expect | 1 +
.../test-crs-static-rebalance2/log.expect | 1 +
79 files changed, 833 insertions(+), 43 deletions(-)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats
Summary over all repositories:
93 files changed, 1828 insertions(+), 132 deletions(-)
--
Generated by murpp 0.11.0
^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 01/19] resource-scheduling: fix typos
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
` (17 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
proxmox-resource-scheduling/src/scheduler.rs | 2 +-
proxmox-resource-scheduling/src/topsis.rs | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index eb5c6142..9078fa72 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -166,7 +166,7 @@ impl Scheduler {
}
}
- /// Map the current node usages to a [`PveTopsisAlternative`].
+ /// Maps the current node usages to a [`PveTopsisAlternative`].
///
/// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
/// square (RMS) and maximum value of each stat in the node usages.
diff --git a/proxmox-resource-scheduling/src/topsis.rs b/proxmox-resource-scheduling/src/topsis.rs
index 3d42ff30..89f4ea40 100644
--- a/proxmox-resource-scheduling/src/topsis.rs
+++ b/proxmox-resource-scheduling/src/topsis.rs
@@ -161,7 +161,7 @@ impl<const N: usize> IdealAlternatives<N> {
}
/// Scores the alternatives in `matrix` according to their similarity to the ideal worst
-/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closness to the ideal worst
+/// alternative. Scores range from 0.0 to 1.0 and a low score indicates closeness to the ideal worst
/// and/or distance to the ideal best alternatives.
pub fn score_alternatives<const N: usize>(
matrix: &Matrix<N>,
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
` (16 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
to be more precise, refine parts of docs of functions responsible for
scoring migration candidates and selecting node to start service on,
respectively
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
new, split out of patch [0] in v1
respective changes:
* slightly reword docs
[0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
proxmox-resource-scheduling/src/scheduler.rs | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 9078fa72..60e60d3c 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -228,15 +228,15 @@ impl Scheduler {
})
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with
- /// exhaustive search.
+ /// Uses bruteforce search to score how much each of the given migration `candidates` improves
+ /// node imbalance and ranks them accordingly.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
///
/// The scoring is done as if each resource migration has already been done. This assumes that
/// the already migrated resource consumes the same amount of each stat as on the previous node
- /// according to its `stats`.
+ /// according to its `stats`. The lower the score, the better.
///
/// Returns up to `limit` of the best scored migrations.
pub fn score_best_balancing_migration_candidates<I>(
@@ -269,8 +269,7 @@ impl Scheduler {
best_migrations
}
- /// Scores the given migration `candidates` by the best node imbalance improvement with the
- /// TOPSIS method.
+ /// Scores the given migration `candidates` using the TOPSIS method.
///
/// The `candidates` are assumed to be consistent with the scheduler. No further validation is
/// done whether the given nodenames actually exist in the scheduler.
@@ -321,7 +320,8 @@ impl Scheduler {
.collect())
}
- /// Scores nodes to start a resource with the usage statistics `resource_stats` on.
+ /// Scores nodes to start a resource with the usage statistics `resource_stats` on, using the
+ /// TOPSIS method.
///
/// The scoring is done as if the resource is already started on each node. This assumes that
/// the already started resource consumes the maximum amount of each stat according to its
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
` (15 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
DRY and hide repeatedly used closure inside reusable function
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* fix commit typo
* add daniel's r-b
proxmox-resource-scheduling/src/scheduler.rs | 41 ++++++++------------
1 file changed, 16 insertions(+), 25 deletions(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 60e60d3c..d6508c15 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -56,6 +56,19 @@ fn calculate_node_imbalance(nodes: &[NodeUsage], to_load: impl Fn(&NodeUsage) ->
cv / max_cv
}
+/// Returns the hypothetical [`NodeStats`] of `node` with respect to the migration `candidate`.
+fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage) -> NodeStats {
+ let mut new_stats = node.stats;
+
+ if node.name == candidate.migration.source_node {
+ new_stats.remove_running_resource(&candidate.stats);
+ } else if node.name == candidate.migration.target_node {
+ new_stats.add_running_resource(&candidate.stats);
+ }
+
+ new_stats
+}
+
criteria_struct! {
/// A given alternative.
struct PveTopsisAlternative {
@@ -216,15 +229,7 @@ impl Scheduler {
/// See [`calculate_node_imbalance`] for more information.
fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- let mut new_stats = node.stats;
-
- if node.name == candidate.migration.source_node {
- new_stats.remove_running_resource(&candidate.stats);
- } else if node.name == candidate.migration.target_node {
- new_stats.add_running_resource(&candidate.stats);
- }
-
- new_stats.load()
+ node_stats_after_migration(candidate, node).load()
})
}
@@ -287,22 +292,8 @@ impl Scheduler {
let matrix = candidates
.iter()
.map(|candidate| {
- let resource_stats = &candidate.stats;
- let source_node = &candidate.migration.source_node;
- let target_node = &candidate.migration.target_node;
-
- self.topsis_alternative_with(|node| {
- let mut new_stats = node.stats;
-
- if &node.name == source_node {
- new_stats.remove_running_resource(resource_stats);
- } else if &node.name == target_node {
- new_stats.add_running_resource(resource_stats);
- }
-
- new_stats
- })
- .into()
+ self.topsis_alternative_with(|node| node_stats_after_migration(candidate, node))
+ .into()
})
.collect::<Vec<_>>();
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (2 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
` (14 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
add weighted node load, which computes the node load as the weighted
arithmetic mean [0] of CPU load and memory usage, alongside basic unit
tests
this is in preparation for letting users prioritize balancing one
resource over the other
[0] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
new with changes regarding patch [0] in v1:
* add weighted_load function
* add NodeStatsWeights struct
- make NodeStatsWeights #[non_exhaustive]
- callers ought to start constructing weights via default function and
then set the public fields, if required
- exhaustive deconstructuring in weighted_load function will force
compile error for new field, which is desirable
- a new field must default to 0.0, otherwise a caller starting from
default function might silently get a different weighted_load
result
dispatched changes into this new patch to make the series more organized
[0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
proxmox-resource-scheduling/src/node.rs | 114 ++++++++++++++++++++++++
1 file changed, 114 insertions(+)
diff --git a/proxmox-resource-scheduling/src/node.rs b/proxmox-resource-scheduling/src/node.rs
index 3c7ed393..98568bb7 100644
--- a/proxmox-resource-scheduling/src/node.rs
+++ b/proxmox-resource-scheduling/src/node.rs
@@ -2,6 +2,39 @@ use std::collections::HashSet;
use crate::resource::ResourceStats;
+/// Weights controlling how much the CPU load and the memory usage of a node contribute to its
+/// [`NodeStats::weighted_load`].
+///
+/// Only the ratio of the weights matters, not their absolute magnitude. A weight of 0.0 means that
+/// the respective resource does not contribute at all.
+///
+/// Callers outside this crate start from [`NodeStatsWeights::default`] and set the weights they
+/// need.
+#[non_exhaustive]
+#[derive(Copy, Clone, PartialEq, Debug)]
+pub struct NodeStatsWeights {
+ /// Weight of [`NodeStats::cpu_load`]. Must be finite and non-negative.
+ pub cpu_weight: f64,
+ /// Weight of [`NodeStats::mem_load`]. Must be finite and non-negative.
+ pub mem_weight: f64,
+}
+
+impl NodeStatsWeights {
+ /// Equal weights, making [`NodeStats::weighted_load`] the same as [`NodeStats::load`].
+ ///
+ /// NOTE: any pair of equal weights bigger than 0.0 works here
+ pub const EQUAL: Self = Self {
+ cpu_weight: 1.0,
+ mem_weight: 1.0,
+ };
+}
+
+impl Default for NodeStatsWeights {
+ fn default() -> Self {
+ Self::EQUAL
+ }
+}
+
/// Usage statistics of a node.
#[derive(Copy, Clone, PartialEq, PartialOrd, Debug, Default)]
pub struct NodeStats {
@@ -55,6 +88,28 @@ impl NodeStats {
pub fn load(&self) -> f64 {
(self.cpu_load() + self.mem_load()) / 2.0
}
+
+ /// Returns a combined node usage as a percentage, which is the [weighted arithmetic mean] of
+ /// [`Self::cpu_load`] and [`Self::mem_load`] with respect to `weights`.
+ ///
+ /// With equal weights, this is the same as [`Self::load`].
+ ///
+ /// [weighted arithmetic mean]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
+ pub fn weighted_load(&self, weights: &NodeStatsWeights) -> f64 {
+ let NodeStatsWeights {
+ cpu_weight,
+ mem_weight,
+ } = *weights;
+
+ let weights_sum = cpu_weight + mem_weight;
+
+ if weights_sum == 0.0 {
+ // neither should contribute to load, so there is no load
+ return 0.0;
+ }
+
+ (self.cpu_load() * cpu_weight + self.mem_load() * mem_weight) / weights_sum
+ }
}
/// A node in the cluster context.
@@ -94,3 +149,62 @@ impl Node {
self.resources.contains(sid)
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn stats() -> NodeStats {
+ NodeStats {
+ cpu: 4.0,
+ maxcpu: 16,
+ mem: 48 << 30,
+ maxmem: 64 << 30,
+ }
+ }
+
+ #[test]
+ fn test_weighted_load_with_equal_weights_is_load() {
+ let stats = stats();
+
+ assert_eq!(stats.weighted_load(&NodeStatsWeights::EQUAL), stats.load());
+ assert_eq!(
+ stats.weighted_load(&NodeStatsWeights {
+ cpu_weight: 3.0,
+ mem_weight: 3.0
+ }),
+ stats.load()
+ );
+ }
+
+ #[test]
+ fn test_weighted_load() {
+ let stats = stats();
+
+ assert_eq!(
+ stats.weighted_load(&NodeStatsWeights {
+ cpu_weight: 1.0,
+ mem_weight: 0.0
+ }),
+ stats.cpu_load()
+ );
+ assert_eq!(
+ stats.weighted_load(&NodeStatsWeights {
+ cpu_weight: 0.0,
+ mem_weight: 1.0
+ }),
+ stats.mem_load()
+ );
+ }
+
+ #[test]
+ fn test_weighted_load_with_zero_weights_is_zero() {
+ assert_eq!(
+ stats().weighted_load(&NodeStatsWeights {
+ cpu_weight: 0.0,
+ mem_weight: 0.0
+ }),
+ 0.0
+ );
+ }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (3 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:41 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
` (13 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
this is made public, so callers like pve-rs bindings can compute load
imbalance with migration candidates
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
no changes since v1, new
ultimately, required to avoid mismatch in imbalance comparison in older
HA manager versions
proxmox-resource-scheduling/src/scheduler.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index d6508c15..a9390357 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -227,7 +227,7 @@ impl Scheduler {
/// Returns the load imbalance among the nodes as if a specific resource was moved.
///
/// See [`calculate_node_imbalance`] for more information.
- fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
+ pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
node_stats_after_migration(candidate, node).load()
})
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (4 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:40 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
` (12 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
CRS did not rebalance a cluster that was clearly imbalanced in CPU load
or memory usage alone, because the combined node load looked balanced
enough [0]
add weights to the scheduler, so that callers can prioritize balancing
one resource over the other
load imbalance computations and the bruteforce method use the weighted
node load
for the TOPSIS methods, the weights replace the hardcoded criteria
weights from commit 2b325aa5 ("add pve_static module"). those favor
memory usage 5 times over CPU load. Scaling them with the weights
instead would make the weights relative to that built-in emphasis, so
1:1 would effectively mean 1:5, which is undesirable. only the factor of
2 between the criteria for the highest and the average usage of each
resource is kept, so custom weights of 1:5 reproduce the previous
criteria weights of 1:5 (average) and 2:10 (highest)
the same weights are used for the TOPSIS criteria and the node load, so
the imbalance reported for the migrations ranked by TOPSIS is weighted
like the criteria that ranked them
the weights default to 1:1. This changes the default ranking of the
TOPSIS methods (expected result of one affected test was adjusted), so
callers have to pass hardcoded legacy topsis weights to keep their
previous rankings. with these, the node load is weighted 1:5 as well and
the previous rankings are preserved, which is also being tested
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
new with changes regarding patch [0] in v1:
* add weights as field to Scheduler struct
* weigh the TOPSIS criteria directly with the weights, keeping the factor
of 2 between the highest and the average usage criteria of each resource
- weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
- add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
* fall back to previous TOPSIS ranking of migration candidates
* no longer expose score and keep ScoredMigration as is
* adjust one scheduler test's expected result
* test that legacy topsis weights preserve previous rankings
[0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
proxmox-resource-scheduling/src/pve_static.rs | 4 +-
proxmox-resource-scheduling/src/scheduler.rs | 105 ++++++++++++++----
.../tests/scheduler.rs | 64 ++++++++++-
3 files changed, 149 insertions(+), 24 deletions(-)
diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
index 40b76cbc..25439a0e 100644
--- a/proxmox-resource-scheduling/src/pve_static.rs
+++ b/proxmox-resource-scheduling/src/pve_static.rs
@@ -1,7 +1,7 @@
use anyhow::Error;
use serde::{Deserialize, Serialize};
-use crate::scheduler::{NodeUsage, Scheduler};
+use crate::scheduler::{LEGACY_TOPSIS_WEIGHTS, NodeUsage, Scheduler};
use crate::{node::NodeStats, resource::ResourceStats};
#[derive(Clone, Serialize, Deserialize)]
@@ -94,7 +94,7 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
.map(|node| node.as_ref().clone().into())
.collect::<Vec<NodeUsage>>();
- let scheduler = Scheduler::from_nodes(nodes);
+ let scheduler = Scheduler::from_nodes(nodes).with_weights(LEGACY_TOPSIS_WEIGHTS);
scheduler.score_nodes_to_start_resource(*service)
}
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index a9390357..75929a14 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -1,6 +1,10 @@
use anyhow::Error;
-use crate::{node::NodeStats, resource::ResourceStats, topsis};
+use crate::{
+ node::{NodeStats, NodeStatsWeights},
+ resource::ResourceStats,
+ topsis,
+};
use serde::{Deserialize, Serialize};
use std::{
@@ -69,26 +73,69 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
new_stats
}
-criteria_struct! {
- /// A given alternative.
- struct PveTopsisAlternative {
- #[criterion("average CPU", -1.0)]
- average_cpu: f64,
- #[criterion("highest CPU", -2.0)]
- highest_cpu: f64,
- #[criterion("average memory", -5.0)]
- average_memory: f64,
- #[criterion("highest memory", -10.0)]
- highest_memory: f64,
+/// A given alternative.
+struct PveTopsisAlternative {
+ average_cpu: f64,
+ highest_cpu: f64,
+ average_memory: f64,
+ highest_memory: f64,
+}
+
+const N_CRITERIA: usize = 4;
+
+impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
+ fn from(alternative: PveTopsisAlternative) -> Self {
+ [
+ alternative.average_cpu,
+ alternative.highest_cpu,
+ alternative.average_memory,
+ alternative.highest_memory,
+ ]
}
+}
- const N_CRITERIA;
- static PVE_HA_TOPSIS_CRITERIA;
+/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
+/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
+/// weights introduced in commit 2b325aa5.
+///
+/// For more details see [`topsis_criteria`].
+const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
+
+/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with
+/// [`topsis_criteria`].
+pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
+ cpu_weight: 1.0,
+ mem_weight: 5.0,
+};
+
+/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
+///
+/// The criteria for the average usage of each resource get the respective weight in `weights`,
+/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
+fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {
+ let NodeStatsWeights {
+ cpu_weight,
+ mem_weight,
+ } = *weights;
+
+ topsis::Criteria::new([
+ topsis::Criterion::new("average CPU".to_owned(), -cpu_weight),
+ topsis::Criterion::new(
+ "highest CPU".to_owned(),
+ -TOPSIS_HIGHEST_USAGE_FACTOR * cpu_weight,
+ ),
+ topsis::Criterion::new("average memory".to_owned(), -mem_weight),
+ topsis::Criterion::new(
+ "highest memory".to_owned(),
+ -TOPSIS_HIGHEST_USAGE_FACTOR * mem_weight,
+ ),
+ ])
}
#[derive(Clone, Debug)]
pub struct Scheduler {
nodes: Vec<NodeUsage>,
+ weights: NodeStatsWeights,
}
/// A possible migration.
@@ -176,9 +223,28 @@ impl Scheduler {
{
Self {
nodes: nodes.into_iter().map(|node| node.into()).collect(),
+ weights: NodeStatsWeights::EQUAL,
}
}
+ /// Sets the `weights` controlling how much balancing the CPU load and the memory usage of the
+ /// nodes is prioritized relative to each other.
+ ///
+ /// For [`Self::node_imbalance`], [`Self::node_imbalance_with_migration_candidate`], the
+ /// bruteforce method [`Self::score_best_balancing_migration_candidates`] and the imbalance
+ /// reported by [`Self::score_best_balancing_migration_candidates_topsis`], the `weights`
+ /// determine the [`NodeStats::weighted_load`] of each node.
+ ///
+ /// For the TOPSIS-based methods, [`Self::score_best_balancing_migration_candidates_topsis`] and
+ /// [`Self::score_nodes_to_start_resource`], the `weights` also weigh the average CPU and
+ /// average memory criteria, see [`topsis_criteria`]. Use [`LEGACY_TOPSIS_WEIGHTS`] to keep the
+ /// criteria weights introduced in commit 2b325aa5.
+ pub fn with_weights(mut self, weights: NodeStatsWeights) -> Self {
+ self.weights = weights;
+
+ self
+ }
+
/// Maps the current node usages to a [`PveTopsisAlternative`].
///
/// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
@@ -221,7 +287,7 @@ impl Scheduler {
///
/// See [`calculate_node_imbalance`] for more information.
pub fn node_imbalance(&self) -> f64 {
- calculate_node_imbalance(&self.nodes, |node| node.stats.load())
+ calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(&self.weights))
}
/// Returns the load imbalance among the nodes as if a specific resource was moved.
@@ -229,7 +295,7 @@ impl Scheduler {
/// See [`calculate_node_imbalance`] for more information.
pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
calculate_node_imbalance(&self.nodes, |node| {
- node_stats_after_migration(candidate, node).load()
+ node_stats_after_migration(candidate, node).weighted_load(&self.weights)
})
}
@@ -297,8 +363,9 @@ impl Scheduler {
})
.collect::<Vec<_>>();
+ let criteria = topsis_criteria(&self.weights)?;
let best_alternatives =
- topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
Ok(best_alternatives
.into_iter()
@@ -343,8 +410,8 @@ impl Scheduler {
})
.collect::<Vec<_>>();
- let scores =
- topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
+ let criteria = topsis_criteria(&self.weights)?;
+ let scores = topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
Ok(scores
.into_iter()
diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
index 61fdd840..603a9418 100644
--- a/proxmox-resource-scheduling/tests/scheduler.rs
+++ b/proxmox-resource-scheduling/tests/scheduler.rs
@@ -2,7 +2,7 @@ use anyhow::Error;
use proxmox_resource_scheduling::{
node::NodeStats,
resource::ResourceStats,
- scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
+ scheduler::{LEGACY_TOPSIS_WEIGHTS, Migration, MigrationCandidate, NodeUsage, Scheduler},
};
fn new_empty_cluster_scheduler() -> Scheduler {
@@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
assert_eq!(
rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
- vec!["node2", "node3", "node1"]
+ vec!["node1", "node3", "node2"]
);
Ok(())
@@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
#[test]
fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
- let scheduler = new_heterogeneous_cluster_scheduler();
+ let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
let heavy_memory_resource_stats = ResourceStats {
cpu: 0.0,
@@ -377,3 +377,61 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
Ok(())
}
+
+#[test]
+fn test_legacy_topsis_weights_keep_unweighted_rankings() -> Result<(), Error> {
+ let resource_stats =
+ [(1.0, 12 << 30), (12.0, 0), (0.0, 0), (12.0, 12 << 30)].map(|(maxcpu, maxmem)| {
+ ResourceStats {
+ cpu: 0.0,
+ maxcpu,
+ mem: 0,
+ maxmem,
+ }
+ });
+
+ // rankings with the TOPSIS criteria weights from before the weights became adjustable
+ let expected_homogeneous_rankings = [
+ ["node2", "node3", "node1"],
+ ["node1", "node3", "node2"],
+ ["node1", "node3", "node2"],
+ ["node2", "node3", "node1"],
+ ];
+ let expected_heterogeneous_rankings = [
+ ["node2", "node1", "node3"],
+ ["node3", "node2", "node1"],
+ ["node1", "node3", "node2"],
+ ["node2", "node1", "node3"],
+ ];
+
+ let scenarios: [(fn() -> Scheduler, _); 2] = [
+ (
+ new_homogeneous_cluster_scheduler,
+ expected_homogeneous_rankings,
+ ),
+ (
+ new_heterogeneous_cluster_scheduler,
+ expected_heterogeneous_rankings,
+ ),
+ ];
+
+ for (new_scheduler, expected_rankings) in scenarios {
+ let legacy = new_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
+
+ for (resource_stats, expected_ranking) in resource_stats.iter().zip(expected_rankings) {
+ assert_eq!(
+ rank_nodes_to_start_resource(&legacy, *resource_stats)?,
+ expected_ranking
+ );
+ }
+
+ let (candidates, migration1, migration2) = new_simple_migration_candidates();
+
+ assert_eq!(
+ rank_best_balancing_migration_candidates_topsis(&legacy, &candidates, 2)?,
+ vec![migration1, migration2]
+ );
+ }
+
+ Ok(())
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (5 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
` (11 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
property-based tests let the resource-scheduling methods be checked
against invariants over a broad range of, for example, generated
clusters, instead of only a few hand-picked examples
proptest 1.11 crate exists, but packaged version 1.6 is fine for now
afaict updating to 1.11 at some point shouldn't come with breaking
changes
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* place proptest in alphabetical order in the workspace Cargo.toml
Cargo.toml | 1 +
proxmox-resource-scheduling/Cargo.toml | 3 +++
2 files changed, 4 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index ba62378e..361dd1cb 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -130,6 +130,7 @@ percent-encoding = "2.1"
pin-utils = "0.1.0"
pretty_assertions = "1.4"
proc-macro2 = "1.0"
+proptest = "1.6"
quick-xml = "0.36.1"
quote = "1.0"
regex = "1.5"
diff --git a/proxmox-resource-scheduling/Cargo.toml b/proxmox-resource-scheduling/Cargo.toml
index 2bfb7871..27d0f3f6 100644
--- a/proxmox-resource-scheduling/Cargo.toml
+++ b/proxmox-resource-scheduling/Cargo.toml
@@ -11,3 +11,6 @@ exclude.workspace = true
[dependencies]
anyhow.workspace = true
serde = { workspace = true, features = [ "derive" ] }
+
+[dev-dependencies]
+proptest.workspace = true
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (6 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
` (10 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
only the ratio of the weights should matter, and a weight of 0 should
make the results independent of the respective resource, for any
cluster. add property tests that check this on randomly generated
clusters and migration candidates, namely that:
- for the bruteforce method, uniformly scaling both weights changes
neither the number of scored migrations nor the imbalance at each
rank, which is compared instead of the migrations themselves, as
migrations with nearly equal imbalance may swap places due to
rounding
- for the bruteforce method, with a CPU weight of 0, the imbalance of
each scored migration equals the imbalance computed from the memory
load of the nodes alone
- for the TOPSIS method, uniformly scaling both weights does not change
the node scores for starting a resource
- for the TOPSIS method, with a CPU weight of 0, two clusters that only
differ in CPU usage result in the same ranked migrations and the same
node scores for starting a resource
for the TOPSIS method, scale invariance is checked on the node scores
for starting a resource. ranked migrations only carry the resulting
imbalance, so a swap of two migrations with nearly equal TOPSIS scores
due to rounding cannot be told apart from an actual ranking change
the tests are grouped in per-method modules, so that, for example,
`cargo test bruteforce` runs only the tests of the bruteforce method
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* adapt to weights being a field of Scheduler
* compare the imbalance of scored migrations instead of the removed
score
* check TOPSIS scale invariance on the node scores for starting a
resource
* check that a TOPSIS CPU weight of 0 results in exactly equal ranked
migrations and node scores for starting a resource
* reword commit
proxmox-resource-scheduling/src/scheduler.rs | 325 +++++++++++++++++++
1 file changed, 325 insertions(+)
diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
index 75929a14..ce0262c6 100644
--- a/proxmox-resource-scheduling/src/scheduler.rs
+++ b/proxmox-resource-scheduling/src/scheduler.rs
@@ -425,6 +425,27 @@ impl Scheduler {
mod tests {
use super::*;
+ use proptest::prelude::*;
+ use std::ops::Range;
+
+ const MIN_NODES: usize = 3;
+ const MAX_NODES: usize = 36;
+
+ const MIN_CPU: usize = 2;
+ const MIN_MEM: usize = 2;
+ const MAX_CPU: usize = 128;
+ const MAX_MEM: usize = 256;
+
+ const MIN_WEIGHT: usize = 1;
+ const MAX_WEIGHT: usize = 1_000;
+
+ const MIN_CONSTANT: usize = 1;
+ const MAX_CONSTANT: usize = 1_000;
+
+ const PROPTEST_CASES: u32 = 256; // proptest's default
+
+ const EPSILON: f64 = 1e-9;
+
#[test]
fn test_scored_migration_order() {
let migration1 = ScoredMigration::new(
@@ -471,4 +492,308 @@ mod tests {
assert_eq!(heap.pop(), Some(Reverse(migration2)));
assert_eq!(heap.pop(), Some(Reverse(migration3)));
}
+
+ fn weights(cpu_weight: usize, mem_weight: usize) -> NodeStatsWeights {
+ let mut weights = NodeStatsWeights::default();
+ weights.cpu_weight = cpu_weight as f64;
+ weights.mem_weight = mem_weight as f64;
+
+ weights
+ }
+
+ /// Strategy to generate `n` random nodes with stats bound by `max_cpu` and `max_mem`.
+ fn nodes_strategy(
+ n: usize,
+ max_cpu: usize,
+ max_mem: usize,
+ ) -> impl Strategy<Value = Vec<NodeUsage>> {
+ prop::collection::vec((0..max_cpu + 1, 0..max_mem + 1), n).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(|(i, (cpu, mem))| NodeUsage {
+ name: format!("node{i}"),
+ stats: NodeStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu,
+ mem,
+ maxmem: max_mem,
+ },
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate a random number in `n_range` of migration candidates between `nodes`.
+ /// Resource stats are bound by `max_cpu` and `max_mem`.
+ fn candidates_strategy(
+ n_range: impl Into<prop::collection::SizeRange>,
+ max_cpu: usize,
+ max_mem: usize,
+ nodes: Vec<NodeUsage>,
+ ) -> impl Strategy<Value = Vec<MigrationCandidate>> {
+ let k = nodes.len();
+
+ prop::collection::vec((0..max_cpu, 0..max_mem, 0..k, 0..k), n_range).prop_map(
+ move |computational_resources| {
+ computational_resources
+ .into_iter()
+ .enumerate()
+ .map(
+ |(i, (cpu, mem, source_idx, target_idx))| MigrationCandidate {
+ migration: Migration {
+ sid: format!("vm:10{i}"),
+ source_node: nodes[source_idx].name.clone(),
+ target_node: nodes[target_idx].name.clone(),
+ },
+ stats: ResourceStats {
+ cpu: cpu as f64,
+ maxcpu: max_cpu as f64,
+ mem,
+ maxmem: max_mem,
+ },
+ },
+ )
+ .filter(|candidate| {
+ candidate.migration.source_node != candidate.migration.target_node
+ })
+ .collect()
+ },
+ )
+ }
+
+ /// Strategy to generate:
+ /// - a random odd number n in `n_nodes_range` of nodes
+ /// - up to n * (n - 1) migration candidates between the generated nodes
+ ///
+ /// The stats of nodes and migration candidates are bound by random numbers in `cpu_range` and
+ /// `mem_range`, respectively.
+ fn scenario_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<Value = (Vec<NodeUsage>, Vec<MigrationCandidate>)> {
+ (cpu_range, mem_range).prop_flat_map(move |(max_cpu, max_mem)| {
+ n_nodes_range
+ .clone()
+ .prop_filter("cluster size must be odd", |n| n % 2 == 1)
+ .prop_flat_map(move |n_nodes| {
+ nodes_strategy(n_nodes, max_cpu, max_mem).prop_flat_map(move |nodes| {
+ let max_candidates = n_nodes * (n_nodes - 1);
+
+ candidates_strategy(1..=max_candidates, max_cpu, max_mem, nodes.clone())
+ .prop_map(move |candidates| (nodes.clone(), candidates))
+ })
+ })
+ })
+ }
+
+ /// Strategy to generate two scenarios with [`scenario_strategy`], which are identical in
+ /// topology and in every memory-related field, but differ in their CPU usage.
+ #[allow(clippy::type_complexity)]
+ fn two_scenarios_with_varying_cpu_usage_strategy(
+ cpu_range: Range<usize>,
+ mem_range: Range<usize>,
+ n_nodes_range: Range<usize>,
+ ) -> impl Strategy<
+ Value = (
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ (Vec<NodeUsage>, Vec<MigrationCandidate>),
+ ),
+ > {
+ let max_possible_nodes = n_nodes_range.end - 1; // Range<T> is exclusive
+ let max_possible_candidates = max_possible_nodes * (max_possible_nodes - 1);
+ let cpu_pool_bound = cpu_range.end;
+
+ (
+ scenario_strategy(cpu_range, mem_range, n_nodes_range),
+ prop::collection::vec(0..cpu_pool_bound + 1, max_possible_nodes),
+ prop::collection::vec(0..cpu_pool_bound, max_possible_candidates),
+ )
+ .prop_map(
+ move |((nodes, candidates), alt_node_cpu_pool, alt_candidate_cpu_pool)| {
+ let mut alt_nodes = nodes.clone();
+ alt_nodes
+ .iter_mut()
+ .zip(alt_node_cpu_pool)
+ .for_each(|(node, cpu)| node.stats.cpu = cpu as f64);
+
+ let mut alt_candidates = candidates.clone();
+ alt_candidates
+ .iter_mut()
+ .zip(alt_candidate_cpu_pool)
+ .for_each(|(candidate, cpu)| candidate.stats.cpu = cpu as f64);
+
+ ((nodes, candidates), (alt_nodes, alt_candidates))
+ },
+ )
+ }
+
+ mod bruteforce {
+ use super::*;
+
+ proptest! {
+ #![proptest_config(ProptestConfig::with_cases(PROPTEST_CASES))]
+
+ /// Test that uniformly scaling both weights does not change the scored migrations;
+ /// only the ratio of the weights matters.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let n = candidates.len();
+
+ let scored = Scheduler::from_nodes(nodes.clone())
+ .with_weights(weights(cpu_weight, mem_weight))
+ .score_best_balancing_migration_candidates(candidates.clone(), n);
+ let scaled = Scheduler::from_nodes(nodes)
+ .with_weights(weights(k * cpu_weight, k * mem_weight))
+ .score_best_balancing_migration_candidates(candidates, n);
+
+ prop_assert_eq!(scored.len(), scaled.len());
+
+ // compare by rank, as near-equal imbalances may swap places due to rounding
+ for (lhs, rhs) in scored.iter().zip(scaled.iter()) {
+ prop_assert!(
+ (lhs.imbalance - rhs.imbalance).abs() < EPSILON,
+ "equally ranked migrations {:?} and {:?} must have the same imbalance",
+ lhs,
+ rhs,
+ );
+ }
+ }
+
+ /// Test that with a CPU weight of 0, the imbalance of each scored migration matches
+ /// the imbalance computed from [`NodeStats::mem_load`] alone.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let n = candidates.len();
+
+ let scored = Scheduler::from_nodes(nodes.clone())
+ .with_weights(weights(0, mem_weight))
+ .score_best_balancing_migration_candidates(candidates.clone(), n);
+
+ prop_assert_eq!(scored.len(), n);
+
+ for scored_migration in scored {
+ let candidate = candidates
+ .iter()
+ .find(|candidate| candidate.migration == scored_migration.migration)
+ .expect("scored migration must be one of the candidates");
+
+ let reference_imbalance = calculate_node_imbalance(&nodes, |node| {
+ node_stats_after_migration(candidate, node).mem_load()
+ });
+
+ prop_assert!(
+ (scored_migration.imbalance - reference_imbalance).abs() < EPSILON,
+ "migration {:?} must have imbalance {}",
+ scored_migration,
+ reference_imbalance,
+ );
+ }
+ }
+ }
+ }
+
+ mod topsis {
+ use super::*;
+
+ proptest! {
+ #![proptest_config(ProptestConfig::with_cases(PROPTEST_CASES))]
+
+ /// Test that uniformly scaling both weights does not change the node scores to start
+ /// a resource; only the ratio of the weights matters.
+ ///
+ /// The node scores are compared instead of the ranked migrations, because the latter
+ /// only expose the imbalance, so near-equal TOPSIS scores that swap places due to
+ /// rounding could not be told apart from an actual change in the ranking.
+ #[test]
+ fn test_weight_scale_invariance(
+ cpu_weight in MIN_WEIGHT..MAX_WEIGHT,
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ k in MIN_CONSTANT..MAX_CONSTANT,
+ (nodes, candidates) in
+ scenario_strategy(MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES)
+ ) {
+ let scheduler = Scheduler::from_nodes(nodes.clone())
+ .with_weights(weights(cpu_weight, mem_weight));
+ let scaled_scheduler = Scheduler::from_nodes(nodes)
+ .with_weights(weights(k * cpu_weight, k * mem_weight));
+
+ // use the stats of the generated candidates as resources to start
+ for candidate in candidates {
+ let scores = scheduler
+ .score_nodes_to_start_resource(candidate.stats)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"));
+ let scaled_scores = scaled_scheduler
+ .score_nodes_to_start_resource(candidate.stats)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}"));
+
+ for ((node, score), (scaled_node, scaled_score)) in
+ scores.iter().zip(scaled_scores.iter())
+ {
+ prop_assert_eq!(node, scaled_node);
+ prop_assert!(
+ (score - scaled_score).abs() < EPSILON,
+ "node {} must have the same score, got {} and {}",
+ node,
+ score,
+ scaled_score,
+ );
+ }
+ }
+ }
+
+ /// Test that with a CPU weight of 0, varying the CPU usage between two otherwise
+ /// identical scenarios changes neither the ranked migrations nor the node scores to
+ /// start a resource.
+ #[test]
+ fn test_zero_cpu_weight_discards_cpu_load(
+ mem_weight in MIN_WEIGHT..MAX_WEIGHT,
+ ((nodes_a, candidates_a), (nodes_b, candidates_b)) in
+ two_scenarios_with_varying_cpu_usage_strategy(
+ MIN_CPU..MAX_CPU, MIN_MEM..MAX_MEM, MIN_NODES..MAX_NODES
+ ),
+ ) {
+ let n = candidates_a.len();
+
+ let scheduler_a =
+ Scheduler::from_nodes(nodes_a).with_weights(weights(0, mem_weight));
+ let scheduler_b =
+ Scheduler::from_nodes(nodes_b).with_weights(weights(0, mem_weight));
+
+ // a CPU weight of 0 zeroes out every CPU criterion, so the results are identical
+ prop_assert_eq!(
+ scheduler_a
+ .score_best_balancing_migration_candidates_topsis(&candidates_a, n)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}")),
+ scheduler_b
+ .score_best_balancing_migration_candidates_topsis(&candidates_b, n)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}")),
+ );
+
+ for candidate in candidates_a {
+ prop_assert_eq!(
+ scheduler_a
+ .score_nodes_to_start_resource(candidate.stats)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}")),
+ scheduler_b
+ .score_nodes_to_start_resource(candidate.stats)
+ .unwrap_or_else(|err| panic!("topsis failed with: {err}")),
+ );
+ }
+ }
+ }
+ }
}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (7 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-19 12:45 ` Daniel Kral
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
` (9 subsequent siblings)
18 siblings, 1 reply; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
allow passing options to the constructor of the static and dynamic
scheduler, which specify the weights of the CPU load and memory usage
applied to every load balancing computation. these options are a single
optional parameter with only optional fields, so the signatures of the
existing methods stay the same
with specified weights, every rebalancing method uses them for the load
imbalance and the TOPSIS criteria alike, so that for example 1:1 means
1:1 for every method. weights of 1:5 reproduce the previous TOPSIS
criteria weights
without specified weights, every method keeps its previous behavior: the
load imbalance and the bruteforce method use equal weights (1:1); the
TOPSIS methods use weights of 1:5 for their criteria
NOTE: weights are not validated. if both are 0, there is simply no load
imbalance and the TOPSIS methods fail, so callers must not pass such
weights
NOTE: to decide whether a migration improves the balance enough, the HA
manager compares the load imbalance reported for the migrations ranked
by TOPSIS with the current load imbalance. without specified weights,
the TOPSIS ranking would report an imbalance weighted 1:5, while the
load imbalance is unweighted. so, without specified weights, recompute
the reported imbalance of the ranked migrations with equal weights, as
before. while this requires reiterating all migrations candidates, it is
required to keep older versions of the HA manager working as before by
preventing them from running into this undesirable load imbalance
mismatch
add test cases checking that:
- without options or with unspecified options, the load imbalance and
the node scores stay the same as before
- equal weights keep the unweighted load imbalance, regardless of their
magnitude
- weights of 1:5 keep the previous node scores, but change the load
imbalance, as it's also being weighted 1:5
- weighting only the CPU load or only the memory usage results in a
different load imbalance
- the TOPSIS ranking reports the same load imbalance as the bruteforce
method, both without specified options and with weights of 1:5, as the
HA manager compares it with the current load imbalance
- without specified options, the TOPSIS ranking is the same as with
weights of 1:5
- the dynamic scheduler accepts options as well
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* pass the weights once as optional scheduler options to the constructor
instead of adding weight parameters to every method, as suggested in
the review of v1, so the signatures of the existing methods stay the
same and older HA manager versions can still call them
* options are a single optional hash with only optional fields, a
missing weight defaults to 1
* without options, every method keeps its previous behavior, including
the TOPSIS ranking with weights of 1:5
* without options, recompute the imbalance reported with the TOPSIS
ranking with equal weights, so that it stays comparable to the node
imbalance for older HA manager version
* weights also apply to score_nodes_to_start_service()
* drop the adjustment of the existing test, as calling the methods
without weights keeps working
* add test cases for the scheduler options
requires the proxmox-resource-scheduling patches of this series
.../src/bindings/resource_scheduling/mod.rs | 1 +
.../bindings/resource_scheduling/options.rs | 59 ++++++++
.../resource_scheduling/pve_dynamic.rs | 71 +++++++--
.../resource_scheduling/pve_static.rs | 64 ++++++--
pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++
5 files changed, 304 insertions(+), 32 deletions(-)
create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/bindings/resource_scheduling/mod.rs
index 87b4a03..dc94f21 100644
--- a/pve-rs/src/bindings/resource_scheduling/mod.rs
+++ b/pve-rs/src/bindings/resource_scheduling/mod.rs
@@ -1,5 +1,6 @@
//! Resource scheduling related bindings.
+mod options;
mod resource;
mod usage;
diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/src/bindings/resource_scheduling/options.rs
new file mode 100644
index 0000000..6759cc3
--- /dev/null
+++ b/pve-rs/src/bindings/resource_scheduling/options.rs
@@ -0,0 +1,59 @@
+//! Options for the resource scheduling bindings.
+
+use serde::Deserialize;
+
+use proxmox_resource_scheduling::node::NodeStatsWeights;
+use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredMigration};
+use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
+
+/// Options for a scheduler instance.
+#[derive(Clone, Copy, Debug, Default, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct SchedulerOptions {
+ /// The weight of the CPU load of nodes. Defaults to 1 if only the other weight is specified.
+ pub cpu_weight: Option<usize>,
+ /// The weight of the memory usage of nodes. Defaults to 1 if only the other weight is
+ /// specified.
+ pub mem_weight: Option<usize>,
+}
+
+impl SchedulerOptions {
+ /// Returns the weights or [`None`] if no weight is set.
+ pub fn weights(&self) -> Option<NodeStatsWeights> {
+ if self.cpu_weight.is_none() && self.mem_weight.is_none() {
+ return None;
+ }
+
+ let mut weights = NodeStatsWeights::default();
+ // unspecified weight defaults to 1
+ weights.cpu_weight = self.cpu_weight.unwrap_or(1) as f64;
+ weights.mem_weight = self.mem_weight.unwrap_or(1) as f64;
+
+ Some(weights)
+ }
+}
+
+/// Replaces the imbalance of each of the `scored_migrations` with the unweighted imbalance of the
+/// respective migration in `candidates`, for the cluster usage `usage` aggregated by `F`.
+pub fn with_unweighted_imbalance<F: UsageAggregator>(
+ usage: &Usage,
+ candidates: &[MigrationCandidate],
+ scored_migrations: Vec<ScoredMigration>,
+) -> Vec<ScoredMigration> {
+ let scheduler = usage.to_scheduler::<F>();
+
+ scored_migrations
+ .into_iter()
+ // every scored migration should belong to some candidate
+ .flat_map(|scored_migration| {
+ candidates
+ .iter()
+ .find(|candidate| candidate.migration == scored_migration.migration)
+ .map(|candidate| {
+ let imbalance = scheduler.node_imbalance_with_migration_candidate(candidate);
+
+ ScoredMigration::new(scored_migration.migration, imbalance)
+ })
+ })
+ .collect()
+}
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
index 8e066e0..62d6426 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
@@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
use serde::{Deserialize, Serialize};
use perlmod::Value;
- use proxmox_resource_scheduling::node::NodeStats;
+ use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
use proxmox_resource_scheduling::resource::ResourceStats;
- use proxmox_resource_scheduling::scheduler::ScoredMigration;
- use proxmox_resource_scheduling::usage::Usage;
+ use proxmox_resource_scheduling::scheduler::{
+ LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
+ };
+ use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
+ use crate::bindings::resource_scheduling::options::{
+ SchedulerOptions, with_unweighted_imbalance,
+ };
use crate::bindings::resource_scheduling::resource::{
CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
};
@@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic {
/// A scheduler instance contains the cluster usage.
pub struct Scheduler {
inner: Mutex<Usage>,
+ weights: Option<NodeStatsWeights>,
+ }
+
+ impl Scheduler {
+ /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
+ /// this instance or `default_weights` if this instance has none.
+ fn to_weighted_scheduler<F: UsageAggregator>(
+ &self,
+ usage: &Usage,
+ default_weights: NodeStatsWeights,
+ ) -> CoreScheduler {
+ usage
+ .to_scheduler::<F>()
+ .with_weights(self.weights.unwrap_or(default_weights))
+ }
}
#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
@@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
type DynamicResource = PveResource<DynamicResourceStats>;
- /// Class method: Create a new [`Scheduler`] instance.
+ /// Class method: Create a new [`Scheduler`] instance with the optional `options`.
///
- /// See [`proxmox_resource_scheduling::usage::Usage::new`].
+ /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`].
#[export(raw_return)]
- pub fn new(#[raw] class: Value) -> Result<Value, Error> {
+ pub fn new(#[raw] class: Value, options: Option<SchedulerOptions>) -> Result<Value, Error> {
let inner = Usage::new();
+ let weights = options.unwrap_or_default().weights();
Ok(perlmod::instantiate_magic!(
- &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) })
+ &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights })
))
}
@@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
let usage = this.inner.lock().unwrap();
- usage.to_scheduler::<IdentityAggregator>().node_imbalance()
+ this.to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
+ .node_imbalance()
}
/// Method: Scores the given migration `candidates` by the best node imbalance improvement with
@@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
- Ok(usage
- .to_scheduler::<IdentityAggregator>()
+ Ok(this
+ .to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
.score_best_balancing_migration_candidates(candidates, limit))
}
@@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic {
let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
- usage
- .to_scheduler::<IdentityAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ let scored_migrations = this
+ .to_weighted_scheduler::<IdentityAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
+ .score_best_balancing_migration_candidates_topsis(&candidates, limit)?;
+
+ if this.weights.is_some() {
+ return Ok(scored_migrations);
+ }
+
+ // callers not passing any weights would compare the 1:5 weighted imbalance reported for the
+ // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted
+ // imbalance
+ //
+ // to avoid this mismatch, report the unweighted imbalance instead
+ Ok(with_unweighted_imbalance::<IdentityAggregator>(
+ &usage,
+ &candidates,
+ scored_migrations,
+ ))
}
/// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
@@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic {
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
- usage
- .to_scheduler::<StartingAsStartedResourceAggregator>()
- .score_nodes_to_start_resource(resource_stats)
+ this.to_weighted_scheduler::<StartingAsStartedResourceAggregator>(
+ &usage,
+ LEGACY_TOPSIS_WEIGHTS,
+ )
+ .score_nodes_to_start_resource(resource_stats)
}
}
diff --git a/pve-rs/src/bindings/resource_scheduling/pve_static.rs b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
index d83aa38..2399f71 100644
--- a/pve-rs/src/bindings/resource_scheduling/pve_static.rs
+++ b/pve-rs/src/bindings/resource_scheduling/pve_static.rs
@@ -12,12 +12,15 @@ pub mod pve_rs_resource_scheduling_static {
use serde::{Deserialize, Serialize};
use perlmod::Value;
- use proxmox_resource_scheduling::node::NodeStats;
+ use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
use proxmox_resource_scheduling::resource::ResourceStats;
- use proxmox_resource_scheduling::scheduler::ScoredMigration;
- use proxmox_resource_scheduling::usage::Usage;
+ use proxmox_resource_scheduling::scheduler::{
+ LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
+ };
+ use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
use crate::bindings::resource_scheduling::{
+ options::{SchedulerOptions, with_unweighted_imbalance},
resource::{
CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
},
@@ -29,6 +32,21 @@ pub mod pve_rs_resource_scheduling_static {
/// A scheduler instance contains the cluster usage.
pub struct Scheduler {
inner: Mutex<Usage>,
+ weights: Option<NodeStatsWeights>,
+ }
+
+ impl Scheduler {
+ /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
+ /// this instance or `default_weights` if this instance has none.
+ fn to_weighted_scheduler<F: UsageAggregator>(
+ &self,
+ usage: &Usage,
+ default_weights: NodeStatsWeights,
+ ) -> CoreScheduler {
+ usage
+ .to_scheduler::<F>()
+ .with_weights(self.weights.unwrap_or(default_weights))
+ }
}
#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
@@ -54,15 +72,16 @@ pub mod pve_rs_resource_scheduling_static {
type StaticResource = PveResource<StaticResourceStats>;
- /// Class method: Create a new [`Scheduler`] instance.
+ /// Class method: Create a new [`Scheduler`] instance with the optional `options`.
///
- /// See [`proxmox_resource_scheduling::usage::Usage::new`].
+ /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`].
#[export(raw_return)]
- pub fn new(#[raw] class: Value) -> Result<Value, Error> {
+ pub fn new(#[raw] class: Value, options: Option<SchedulerOptions>) -> Result<Value, Error> {
let inner = Usage::new();
+ let weights = options.unwrap_or_default().weights();
Ok(perlmod::instantiate_magic!(
- &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) })
+ &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights })
))
}
@@ -165,8 +184,7 @@ pub mod pve_rs_resource_scheduling_static {
pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
let usage = this.inner.lock().unwrap();
- usage
- .to_scheduler::<StartedResourceAggregator>()
+ this.to_weighted_scheduler::<StartedResourceAggregator>(&usage, NodeStatsWeights::EQUAL)
.node_imbalance()
}
@@ -184,8 +202,8 @@ pub mod pve_rs_resource_scheduling_static {
let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
- Ok(usage
- .to_scheduler::<StartedResourceAggregator>()
+ Ok(this
+ .to_weighted_scheduler::<StartedResourceAggregator>(&usage, NodeStatsWeights::EQUAL)
.score_best_balancing_migration_candidates(candidates, limit))
}
@@ -203,9 +221,24 @@ pub mod pve_rs_resource_scheduling_static {
let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
- usage
- .to_scheduler::<StartedResourceAggregator>()
- .score_best_balancing_migration_candidates_topsis(&candidates, limit)
+ let scored_migrations = this
+ .to_weighted_scheduler::<StartedResourceAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
+ .score_best_balancing_migration_candidates_topsis(&candidates, limit)?;
+
+ if this.weights.is_some() {
+ return Ok(scored_migrations);
+ }
+
+ // callers not passing any weights would compare the 1:5 weighted imbalance reported for the
+ // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted
+ // imbalance
+ //
+ // to avoid this mismatch, report the unweighted imbalance instead
+ Ok(with_unweighted_imbalance::<StartedResourceAggregator>(
+ &usage,
+ &candidates,
+ scored_migrations,
+ ))
}
/// Method: Scores nodes to start a service with the usage statistics `service_stats` on.
@@ -218,8 +251,7 @@ pub mod pve_rs_resource_scheduling_static {
) -> Result<Vec<(String, f64)>, Error> {
let usage = this.inner.lock().unwrap();
- usage
- .to_scheduler::<StartedResourceAggregator>()
+ this.to_weighted_scheduler::<StartedResourceAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
.score_nodes_to_start_resource(service_stats)
}
}
diff --git a/pve-rs/test/resource_scheduling.pl b/pve-rs/test/resource_scheduling.pl
index 3775242..f8ed89a 100755
--- a/pve-rs/test/resource_scheduling.pl
+++ b/pve-rs/test/resource_scheduling.pl
@@ -179,11 +179,152 @@ sub test_balance_small_memory_difference {
}
}
+sub test_scheduler_options {
+ my $new_static = sub {
+ my $static = PVE::RS::ResourceScheduling::Static->new(@_);
+ $static->add_node("A", 8, 10_000_000_000);
+ $static->add_node("B", 4, 9_000_000_000);
+ $static->add_node("C", 4, 8_000_000_000);
+ $static->add_service_usage_to_node("A", "vm:100", { maxcpu => 4, maxmem => 1_000_000_000 });
+ $static->add_service_usage_to_node("B", "vm:101", { maxcpu => 1, maxmem => 6_000_000_000 });
+ $static->add_service_usage_to_node("C", "vm:102", { maxcpu => 3, maxmem => 1_000_000_000 });
+ return $static;
+ };
+
+ my $service = { maxcpu => 2, maxmem => 2_000_000_000 };
+
+ my sub scores {
+ my ($static) = @_;
+ return { map { $_->[0] => $_->[1] } $static->score_nodes_to_start_service($service)->@* };
+ }
+
+ my $default = $new_static->();
+
+ # without weights, the previous behavior is kept
+ for my $entry ([undef, 'undef'], [{}, 'empty']) {
+ my ($options, $desc) = $entry->@*;
+ my $static = $new_static->($options);
+
+ is(
+ $static->calculate_node_imbalance(),
+ $default->calculate_node_imbalance(),
+ "imbalance with $desc options equals default",
+ );
+ is_deeply(scores($static), scores($default), "scores with $desc options equal default");
+ }
+
+ # equal weights keep the previous unweighted imbalance
+ for my $options (
+ { 'cpu-weight' => 1 },
+ { 'cpu-weight' => 1, 'mem-weight' => 1 },
+ { 'cpu-weight' => 3, 'mem-weight' => 3 },
+ ) {
+ my $desc = join(', ', map { "$_ $options->{$_}" } sort keys $options->%*);
+
+ is(
+ $new_static->($options)->calculate_node_imbalance(),
+ $default->calculate_node_imbalance(),
+ "imbalance with $desc equals default",
+ );
+ }
+
+ # weights of 1:5 keep the previous TOPSIS criteria weights, but also weight the imbalance
+ my $legacy_ratio = $new_static->({ 'cpu-weight' => 1, 'mem-weight' => 5 });
+
+ is_deeply(scores($legacy_ratio), scores($default), 'scores with weights 1:5 equal default');
+ isnt(
+ $legacy_ratio->calculate_node_imbalance(),
+ $default->calculate_node_imbalance(),
+ 'weights 1:5 change imbalance',
+ );
+
+ my $cpu_only = $new_static->({ 'cpu-weight' => 1, 'mem-weight' => 0 });
+ my $mem_only = $new_static->({ 'cpu-weight' => 0, 'mem-weight' => 1 });
+
+ isnt(
+ $cpu_only->calculate_node_imbalance(),
+ $mem_only->calculate_node_imbalance(),
+ 'weights change imbalance',
+ );
+
+ my $dynamic = eval { PVE::RS::ResourceScheduling::Dynamic->new({ 'cpu-weight' => 3 }) };
+ ok(defined($dynamic) && !$@, 'dynamic scheduler accepts options');
+}
+
+sub test_topsis_imbalance_with_scheduler_options {
+ my $new_static = sub {
+ my $static = PVE::RS::ResourceScheduling::Static->new(@_);
+ $static->add_node("A", 8, 10_000_000_000);
+ $static->add_node("B", 4, 9_000_000_000);
+ $static->add_node("C", 4, 8_000_000_000);
+
+ my $services = {
+ "vm:100" => ["A", 4, 1_000_000_000],
+ "vm:101" => ["A", 2, 4_000_000_000],
+ "vm:102" => ["B", 1, 6_000_000_000],
+ "vm:103" => ["C", 3, 1_000_000_000],
+ };
+ for my $sid (sort keys $services->%*) {
+ my ($node, $maxcpu, $maxmem) = $services->{$sid}->@*;
+ my $stats = { maxcpu => $maxcpu, maxmem => $maxmem };
+ $static->add_service($sid, { stats => $stats, running => 1, 'current-node' => $node });
+ }
+
+ return $static;
+ };
+
+ # 2^2 candidates
+ my $candidates = [
+ map { { leader => $_, resources => [$_], nodes => ["B", "C"] } } ("vm:100", "vm:101")
+ ];
+
+ my sub imbalances {
+ my ($migrations) = @_;
+
+ return {
+ map {
+ my $m = $_->{migration};
+ ("$m->{sid} $m->{'source-node'} $m->{'target-node'}" => $_->{imbalance})
+ } $migrations->@*
+ };
+ }
+
+ my $rankings = {};
+
+ for my $entry ([undef, 'without options'], [{ 'cpu-weight' => 1, 'mem-weight' => 5 }, '1:5']) {
+ my ($options, $desc) = $entry->@*;
+ my $static = $new_static->($options);
+
+ my $topsis = $static->score_best_balancing_migration_candidates_topsis($candidates, 4);
+ my $bruteforce = $static->score_best_balancing_migration_candidates($candidates, 4);
+
+ # the TOPSIS ranking must report the same imbalance as the current imbalance computation
+ is_deeply(
+ imbalances($topsis),
+ imbalances($bruteforce),
+ "TOPSIS ranking reports the same imbalance as bruteforce $desc",
+ );
+
+ # only keep the ranking, as the reported imbalance differs between both cases
+ $rankings->{$desc} =
+ [map { "$_->{migration}->{sid} $_->{migration}->{'target-node'}" } $topsis->@*];
+ }
+
+ # without options, the TOPSIS ranking uses weights of 1:5
+ is_deeply(
+ $rankings->{'without options'},
+ $rankings->{'1:5'},
+ 'TOPSIS ranking without options equals ranking with weights 1:5',
+ );
+}
+
test_basic();
test_balance();
test_balance_removal();
test_overcommitted();
test_balance_small_memory_difference(1);
test_balance_small_memory_difference(0);
+test_scheduler_options();
+test_topsis_imbalance_with_scheduler_options();
done_testing();
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (8 preceding siblings ...)
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
` (8 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
add weights controlling how much balancing the CPU load, respectively
the memory usage, of nodes is prioritized relative to each other by the
static and dynamic scheduling modes
the weights are not specific to automatic rebalancing, as they are also
used when selecting a node to start or recover a resource on, so they do
not require 'ha-auto-rebalance'
if neither weight is set, the previous behavior is kept: selecting a
node and the TOPSIS method favor memory usage 5 times over CPU load,
while the bruteforce method weighs both equally. if only one weight is
set, the other one defaults to 1
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* rename ha-auto-rebalance-{cpu,mem}-weight, as the weights are not
auto-rebalance specific
* add verbose_description
* reword commit
src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 004122e..a678838 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -78,6 +78,34 @@ EODESC
description => "The minimum relative improvement in cluster node imbalance,"
. " in percent, to commit to a resource balancing migration.",
},
+ 'ha-cpu-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ default => 1,
+ description => "How much balancing the CPU load of nodes is prioritized, relative to"
+ . " 'ha-mem-weight'. Used for the 'static' and 'dynamic' scheduler modes.",
+ verbose_description => "How much balancing the CPU load of nodes is prioritized,"
+ . " relative to 'ha-mem-weight'. Used for the 'static' and 'dynamic' scheduler"
+ . " modes, both for selecting a node to start or recover a resource on and for"
+ . " the automatic rebalancing. Only the ratio of the weights matters. If"
+ . " neither weight is set, selecting a node to start or recover a resource on and"
+ . " the 'topsis' rebalancing method prioritize memory usage 5 times over CPU load.",
+ },
+ 'ha-mem-weight' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ default => 1,
+ description => "How much balancing the memory usage of nodes is prioritized, relative"
+ . " to 'ha-cpu-weight'. Used for the 'static' and 'dynamic' scheduler modes.",
+ verbose_description => "How much balancing the memory usage of nodes is prioritized,"
+ . " relative to 'ha-cpu-weight'. Used for the 'static' and 'dynamic' scheduler"
+ . " modes, both for selecting a node to start or recover a resource on and for"
+ . " the automatic rebalancing. Only the ratio of the weights matters. If"
+ . " neither weight is set, selecting a node to start or recover a resource on and"
+ . " the 'topsis' rebalancing method prioritize memory usage 5 times over CPU load.",
+ },
};
my $migration_format = {
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (9 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
` (7 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
the CRS settings window has become rather crowded since the addition of
the automatic rebalancing options
show the hold duration and the minimum imbalance improvement only as
advanced options, as they fine-tune the automatic rebalancing
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* make weights advanced options in the patch adding them
www/manager6/form/CRSOptions.js | 2 ++
1 file changed, 2 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index 84f6d506..05c5e222 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -103,6 +103,8 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ ],
+ advancedItems: [
{
xtype: 'numberfield',
name: 'ha-auto-rebalance-hold-duration',
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (10 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
` (6 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
add the ha-cpu-weight and ha-mem-weight options as advanced options.
they are enabled for the static and dynamic scheduling modes, also
without automatic rebalancing, as they are used for selecting a node to
start or recover a resource on as well
if neither weight is set, the previous behavior is kept, where the
TOPSIS method and selecting a node favor memory usage over CPU load,
while the bruteforce method weighs both equally. as there is no single
default then, only show the default value of a weight once the other one
is set
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* use the new option names
* enable weights for the static and dynamic modes, independent of
automatic rebalancing
* add weights as advanced options directly
* show the default of a weight only if the other one is set, as the
effective weights depend on the method if neither is set
* add a hint that only the ratio of the weights matters
www/manager6/form/CRSOptions.js | 45 +++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index 05c5e222..4713bbb3 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -21,12 +21,21 @@ Ext.define('PVE.form.CRSOptions', {
data: {
crsMode: '__default__',
autoRebalancing: 0,
+ cpuWeight: null,
+ memWeight: null,
},
formulas: {
canUseAutoRebalancing: (get) =>
get('crsMode') === 'static' || get('crsMode') === 'dynamic',
autoRebalancingDisabled: (get) =>
!get('autoRebalancing') || !get('canUseAutoRebalancing'),
+ // with neither weight set, the effective weights depend on the rebalancing method and
+ // differ between selecting a node and automatic rebalancing, so only show a default
+ // value once the other weight is set
+ cpuWeightEmptyText: (get) =>
+ Proxmox.Utils.defaultText + (Ext.isNumber(get('memWeight')) ? ' (1)' : ''),
+ memWeightEmptyText: (get) =>
+ Proxmox.Utils.defaultText + (Ext.isNumber(get('cpuWeight')) ? ' (1)' : ''),
},
},
@@ -128,6 +137,42 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ {
+ xtype: 'numberfield',
+ name: 'ha-cpu-weight',
+ fieldLabel: gettext('CPU Weight'),
+ minValue: 0,
+ step: 1,
+ allowDecimals: false,
+ bind: {
+ value: '{cpuWeight}',
+ emptyText: '{cpuWeightEmptyText}',
+ disabled: '{!canUseAutoRebalancing}',
+ },
+ },
+ {
+ xtype: 'numberfield',
+ name: 'ha-mem-weight',
+ fieldLabel: gettext('Memory Weight'),
+ minValue: 0,
+ step: 1,
+ allowDecimals: false,
+ bind: {
+ value: '{memWeight}',
+ emptyText: '{memWeightEmptyText}',
+ disabled: '{!canUseAutoRebalancing}',
+ },
+ },
+ {
+ xtype: 'displayfield',
+ userCls: 'pmx-hint',
+ value: gettext(
+ 'Weights prioritize balancing the CPU load or the memory usage of nodes. Only their ratio matters.',
+ ),
+ bind: {
+ hidden: '{!canUseAutoRebalancing}',
+ },
+ },
],
},
],
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (11 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
` (5 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
Respective parameter was removed.
Fixes: c087c06 ("remove redundant service_node and migration_target parameter")
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* reference the commit with a Fixes: trailer instead of the lore link
* add daniel's r-b
src/PVE/HA/Manager.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..7a5a685 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -385,7 +385,7 @@ sub select_service_node {
return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
- my $scores = $online_node_usage->score_nodes_to_start_service($sid, $current_node);
+ my $scores = $online_node_usage->score_nodes_to_start_service($sid);
my @nodes = sort {
$scores->{$a} <=> $scores->{$b} || $a cmp $b
} keys %$pri_nodes;
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (12 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
` (4 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
the method updates all CRS settings, including the automatic rebalancing
options, not just the scheduler mode. rename it to avoid confusion with
the CRS scheduler mode setting
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* moved before the logging patches
src/PVE/HA/Manager.pm | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 7a5a685..b206719 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -94,12 +94,12 @@ sub new {
# preserve disarm state across CRM master changes
$self->{ms}->{disarm} = $old_ms->{disarm} if defined($old_ms->{disarm});
- $self->update_crs_scheduler_mode(); # initial set, we update it once every loop
+ $self->update_crs_settings(); # initial set, we update it once every loop
return $self;
}
-sub update_crs_scheduler_mode {
+sub update_crs_settings {
my ($self) = @_;
my $haenv = $self->{haenv};
@@ -1028,7 +1028,7 @@ sub manage {
return;
}
- $self->update_crs_scheduler_mode();
+ $self->update_crs_settings();
$self->try_persistent_group_migration();
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (13 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
` (3 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
factor out logic that updates the scheduler mode and logs its changes
into a helper, so that it can be reused for the other CRS options. the
logged messages for the scheduler mode stay the same
scheduler mode is now updated first, so that it is logged before any
other CRS option once these are logged too
Suggested-by: Daniel Kral <d.kral@proxmox.com>
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
new with changes regarding patch [0] in v1:
* add helper suggested by daniel
* as suggested, split out of patch [0] in v1
[0] https://lore.proxmox.com/pve-devel/20261021085932.1196302-13-d.rusovac@proxmox.com/
src/PVE/HA/Manager.pm | 39 ++++++++++++++++++++++++++-------------
1 file changed, 26 insertions(+), 13 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index b206719..a91eec3 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -99,6 +99,25 @@ sub new {
return $self;
}
+sub update_crs_option {
+ my ($self, $name, $option_ref, $value, $default) = @_;
+
+ my $haenv = $self->{haenv};
+
+ my $old_value = $$option_ref;
+ my $new_value = $value // $default;
+
+ if (!defined($old_value)) {
+ $haenv->log('info', "using $name '$new_value'") if $new_value ne $default;
+ } elsif ($old_value ne $new_value) {
+ $haenv->log('info', "switching $name from '$old_value' to '$new_value'");
+ }
+
+ $$option_ref = $new_value;
+
+ return;
+}
+
sub update_crs_settings {
my ($self) = @_;
@@ -106,6 +125,13 @@ sub update_crs_settings {
my $dc_cfg = $haenv->get_datacenter_settings();
my $crs_cfg = $dc_cfg->{crs} // {};
+ $self->update_crs_option(
+ 'scheduler mode',
+ \$self->{crs}->{scheduler},
+ $crs_cfg->{ha} || 'basic',
+ 'basic',
+ );
+
$self->{crs}->{rebalance_on_request_start} = !!$crs_cfg->{'ha-rebalance-on-start'};
$self->{crs}->{auto_rebalance}->{enable} = !!$crs_cfg->{'ha-auto-rebalance'};
# configured as percent for UI/config consistency, scaled to [0, 1] to match the
@@ -119,19 +145,6 @@ sub update_crs_settings {
$self->{crs}->{auto_rebalance}->{margin} =
($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
- my $old_mode = $self->{crs}->{scheduler};
- my $new_mode = $crs_cfg->{ha} || 'basic';
-
- if (!defined($old_mode)) {
- $haenv->log('info', "using scheduler mode '$new_mode'") if $new_mode ne 'basic';
- } elsif ($new_mode eq $old_mode) {
- return; # nothing to do
- } else {
- $haenv->log('info', "switching scheduler mode from '$old_mode' to '$new_mode'");
- }
-
- $self->{crs}->{scheduler} = $new_mode;
-
return;
}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (14 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
` (2 subsequent siblings)
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
log the CRS settings when they are first read with non-default values
and whenever they change, so that it can be traced how the load balancer
is configured
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* use the update_crs_option() helper, which also logs non-default
values when they are first read
* keep the log messages for the scheduler mode
* adjust test logs
* reword commit
src/PVE/HA/Manager.pm | 49 +++++++++++++++----
.../log.expect | 2 +
.../log.expect | 2 +
.../log.expect | 2 +
.../log.expect | 3 ++
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 2 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 2 +
.../log.expect | 1 +
.../test-crs-dynamic-rebalance1/log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 1 +
.../log.expect | 4 ++
.../test-crs-static-rebalance1/log.expect | 1 +
.../test-crs-static-rebalance2/log.expect | 1 +
22 files changed, 70 insertions(+), 10 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index a91eec3..7de8781 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -132,18 +132,47 @@ sub update_crs_settings {
'basic',
);
- $self->{crs}->{rebalance_on_request_start} = !!$crs_cfg->{'ha-rebalance-on-start'};
- $self->{crs}->{auto_rebalance}->{enable} = !!$crs_cfg->{'ha-auto-rebalance'};
+ my $crs = $self->{crs};
+ my $auto_rebalance = $crs->{auto_rebalance};
+
+ $self->update_crs_option(
+ 'ha-rebalance-on-start',
+ \$crs->{rebalance_on_request_start},
+ $crs_cfg->{'ha-rebalance-on-start'},
+ 0,
+ );
+ $self->update_crs_option(
+ 'ha-auto-rebalance',
+ \$auto_rebalance->{enable},
+ $crs_cfg->{'ha-auto-rebalance'},
+ 0,
+ );
# configured as percent for UI/config consistency, scaled to [0, 1] to match the
# internal imbalance value that the comparison gates against.
- $self->{crs}->{auto_rebalance}->{threshold} =
- ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0;
- $self->{crs}->{auto_rebalance}->{method} = $crs_cfg->{'ha-auto-rebalance-method'}
- // 'bruteforce';
- $self->{crs}->{auto_rebalance}->{hold_duration} = $crs_cfg->{'ha-auto-rebalance-hold-duration'}
- // 3;
- $self->{crs}->{auto_rebalance}->{margin} =
- ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ $self->update_crs_option(
+ 'ha-auto-rebalance-threshold',
+ \$auto_rebalance->{threshold},
+ ($crs_cfg->{'ha-auto-rebalance-threshold'} // 30) / 100.0,
+ 0.3,
+ );
+ $self->update_crs_option(
+ 'ha-auto-rebalance-method',
+ \$auto_rebalance->{method},
+ $crs_cfg->{'ha-auto-rebalance-method'},
+ 'bruteforce',
+ );
+ $self->update_crs_option(
+ 'ha-auto-rebalance-hold-duration',
+ \$auto_rebalance->{hold_duration},
+ $crs_cfg->{'ha-auto-rebalance-hold-duration'},
+ 3,
+ );
+ $self->update_crs_option(
+ 'ha-auto-rebalance-margin',
+ \$auto_rebalance->{margin},
+ ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0,
+ 0.1,
+ );
return;
}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect
index e6ee440..d684c89 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect
@@ -11,6 +11,8 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-method 'topsis'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect
index a09cb5e..d95785b 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect
@@ -11,6 +11,8 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-method 'topsis'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 2f19c72..8779848 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -11,6 +11,8 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-method 'topsis'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect
index 30898f1..4fc46a9 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect
@@ -11,6 +11,9 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-method 'topsis'
+info 20 node1/crm: using ha-auto-rebalance-hold-duration '6'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance1/log.expect b/src/test/test-crs-dynamic-auto-rebalance1/log.expect
index e6ee440..c2bd61f 100644
--- a/src/test/test-crs-dynamic-auto-rebalance1/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance2/log.expect b/src/test/test-crs-dynamic-auto-rebalance2/log.expect
index a09cb5e..807d480 100644
--- a/src/test/test-crs-dynamic-auto-rebalance2/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance2/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance3/log.expect b/src/test/test-crs-dynamic-auto-rebalance3/log.expect
index 8ef7514..888d83e 100644
--- a/src/test/test-crs-dynamic-auto-rebalance3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance3/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-auto-rebalance4/log.expect b/src/test/test-crs-dynamic-auto-rebalance4/log.expect
index 30898f1..57223fb 100644
--- a/src/test/test-crs-dynamic-auto-rebalance4/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance4/log.expect
@@ -11,6 +11,8 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-hold-duration '6'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect b/src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect
index 88a2441..2d4e019 100644
--- a/src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect
+++ b/src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect b/src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect
index ec0aed1..e96fabf 100644
--- a/src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect
+++ b/src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect b/src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect
index 0d299a4..715ccfe 100644
--- a/src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect
+++ b/src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect
@@ -11,6 +11,8 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-hold-duration '0'
info 21 node1/lrm: got lock 'ha_agent_node1_lock'
info 21 node1/lrm: status change wait_for_agent_lock => active
info 21 node1/lrm: starting service vm:102
diff --git a/src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect b/src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect
index fa5f59a..cefcc04 100644
--- a/src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect
+++ b/src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-dynamic-rebalance1/log.expect b/src/test/test-crs-dynamic-rebalance1/log.expect
index 5c8b050..27d2c8f 100644
--- a/src/test/test-crs-dynamic-rebalance1/log.expect
+++ b/src/test/test-crs-dynamic-rebalance1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-rebalance-on-start '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-auto-rebalance1/log.expect b/src/test/test-crs-static-auto-rebalance1/log.expect
index d2c27be..33b3023 100644
--- a/src/test/test-crs-static-auto-rebalance1/log.expect
+++ b/src/test/test-crs-static-auto-rebalance1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-auto-rebalance2/log.expect b/src/test/test-crs-static-auto-rebalance2/log.expect
index a533d81..da95a37 100644
--- a/src/test/test-crs-static-auto-rebalance2/log.expect
+++ b/src/test/test-crs-static-auto-rebalance2/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-auto-rebalance3/log.expect b/src/test/test-crs-static-auto-rebalance3/log.expect
index b804508..e0dc32d 100644
--- a/src/test/test-crs-static-auto-rebalance3/log.expect
+++ b/src/test/test-crs-static-auto-rebalance3/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-auto-rebalance '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-rebalance-resource-affinity1/log.expect b/src/test/test-crs-static-rebalance-resource-affinity1/log.expect
index cdd2497..bd5961b 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity1/log.expect
+++ b/src/test/test-crs-static-rebalance-resource-affinity1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-rebalance-on-start '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-rebalance-resource-affinity2/log.expect b/src/test/test-crs-static-rebalance-resource-affinity2/log.expect
index eb795a2..83e3c9c 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity2/log.expect
+++ b/src/test/test-crs-static-rebalance-resource-affinity2/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-rebalance-on-start '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-rebalance-resource-affinity3/log.expect b/src/test/test-crs-static-rebalance-resource-affinity3/log.expect
index f692ab3..b4da09a 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity3/log.expect
+++ b/src/test/test-crs-static-rebalance-resource-affinity3/log.expect
@@ -17,6 +17,7 @@ info 20 node5/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-rebalance-on-start '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
@@ -96,6 +97,7 @@ info 220 cmdlist: execute delay 100
info 222 node3/crm: got lock 'ha_manager_lock'
info 222 node3/crm: status change slave => master
info 222 node3/crm: using scheduler mode 'static'
+info 222 node3/crm: using ha-rebalance-on-start '1'
info 222 node3/crm: node 'node1': state changed from 'online' => 'unknown'
info 282 node3/crm: service 'vm:200': state changed from 'started' to 'fence'
info 282 node3/crm: node 'node1': state changed from 'unknown' => 'fence'
@@ -163,6 +165,7 @@ info 1340 cmdlist: execute delay 100
info 1346 node5/crm: got lock 'ha_manager_lock'
info 1346 node5/crm: status change slave => master
info 1346 node5/crm: using scheduler mode 'static'
+info 1346 node5/crm: using ha-rebalance-on-start '1'
info 1346 node5/crm: node 'node3': state changed from 'online' => 'unknown'
info 1406 node5/crm: service 'vm:400': state changed from 'started' to 'fence'
info 1406 node5/crm: node 'node3': state changed from 'unknown' => 'fence'
@@ -237,6 +240,7 @@ info 2460 cmdlist: execute delay 100
info 2480 node1/crm: got lock 'ha_manager_lock'
info 2480 node1/crm: status change slave => master
info 2480 node1/crm: using scheduler mode 'static'
+info 2480 node1/crm: using ha-rebalance-on-start '1'
info 2480 node1/crm: node 'node5': state changed from 'online' => 'unknown'
info 2540 node1/crm: service 'vm:201': state changed from 'started' to 'fence'
info 2540 node1/crm: service 'vm:500': state changed from 'started' to 'fence'
diff --git a/src/test/test-crs-static-rebalance1/log.expect b/src/test/test-crs-static-rebalance1/log.expect
index da3ed5b..cf33a53 100644
--- a/src/test/test-crs-static-rebalance1/log.expect
+++ b/src/test/test-crs-static-rebalance1/log.expect
@@ -11,6 +11,7 @@ info 20 node3/lrm: status change startup => wait_for_agent_lock
info 20 node1/crm: got lock 'ha_manager_lock'
info 20 node1/crm: status change wait_for_quorum => master
info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: using ha-rebalance-on-start '1'
info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
diff --git a/src/test/test-crs-static-rebalance2/log.expect b/src/test/test-crs-static-rebalance2/log.expect
index 1be2e17..781cdc8 100644
--- a/src/test/test-crs-static-rebalance2/log.expect
+++ b/src/test/test-crs-static-rebalance2/log.expect
@@ -12,6 +12,7 @@ info 120 cmdlist: execute service vm:100 add node2 started 1
info 120 node1/crm: got lock 'ha_manager_lock'
info 120 node1/crm: status change wait_for_quorum => master
info 120 node1/crm: using scheduler mode 'static'
+info 120 node1/crm: using ha-rebalance-on-start '1'
info 120 node1/crm: node 'node1': state changed from 'unknown' => 'online'
info 120 node1/crm: node 'node2': state changed from 'unknown' => 'online'
info 120 node1/crm: node 'node3': state changed from 'unknown' => 'online'
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (15 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
This adds commands to change CRS settings or shutdown policy at test
runtime.
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* fix accidentally validated empty alternative in the shutdown_policy regex
* weight options are added with the patch introducing them
src/PVE/HA/Sim/Hardware.pm | 60 ++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index e1a9ddd..f3d8301 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -59,6 +59,7 @@ my $default_node_maxmem = 131072 * 1024**2;
# $testdir/status/service_config Service configuration
# $testdir/status/service_status_<node> Service status
# $testdir/status/groups HA groups configuration
+# $testdir/status/datacenter.cfg Datacenter wide HA configuration
sub read_lrm_status {
my ($self, $node) = @_;
@@ -699,6 +700,56 @@ sub write_datacenter_conf {
PVE::HA::Tools::write_json_to_file($filename, $conf);
}
+my $datacenter_settings = {
+ ha => {
+ shutdown_policy => qr/^(?:conditional|freeze|failover|migrate)$/,
+ },
+ crs => {
+ ha => qr/^(?:basic|static|dynamic)$/,
+ 'ha-rebalance-on-start' => qr/^[01]$/,
+ 'ha-auto-rebalance' => qr/^[01]$/,
+ 'ha-auto-rebalance-threshold' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-auto-rebalance-method' => qr/^(?:bruteforce|topsis)$/,
+ 'ha-auto-rebalance-hold-duration' => qr/^\d+$/,
+ 'ha-auto-rebalance-margin' => qr/^\d+(?:\.\d+)?$/,
+ },
+};
+
+sub update_datacenter_setting {
+ my ($self, $section, $action, $key, $value) = @_;
+
+ my $section_props = $datacenter_settings->{$section}
+ or die "sim_hardware_cmd: unknown datacenter section '$section'\n";
+
+ die "sim_hardware_cmd: missing property for 'datacenter $section $action' command\n"
+ if !defined($key);
+
+ my $verifier = $section_props->{$key}
+ or die "sim_hardware_cmd: unknown datacenter property '$key' in section '$section'\n";
+
+ my $conf = $self->read_datacenter_conf();
+
+ if ($action eq 'set') {
+ die "sim_hardware_cmd: missing value for 'datacenter $section set $key' command\n"
+ if !defined($value);
+ die "sim_hardware_cmd: invalid value '$value' for datacenter property '$section.$key'\n"
+ if $value !~ $verifier;
+
+ $value = $value + 0 if $value =~ m/^\d+(?:\.\d+)?$/;
+
+ $conf->{$section}->{$key} = $value;
+ } elsif ($action eq 'delete') {
+ delete $conf->{$section}->{$key} if $conf->{$section};
+ delete $conf->{$section} if $conf->{$section} && !%{ $conf->{$section} };
+ } else {
+ die "sim_hardware_cmd: unknown datacenter action '$action'\n";
+ }
+
+ $self->write_datacenter_conf($conf);
+
+ return $conf;
+}
+
sub global_lock {
my ($self, $code, @param) = @_;
@@ -802,6 +853,8 @@ sub get_cfs_state {
# service <sid> set-static-stats [maxcpu <cores>] [maxmem <MiB>]
# service <sid> set-dynamic-stats [cpu <cores>] [mem <MiB>]
# service <sid> delete
+# datacenter <ha|crs> set <property> <value>
+# datacenter <ha|crs> delete <property>
sub sim_hardware_cmd {
my ($self, $cmdstr, $logid) = @_;
@@ -820,6 +873,8 @@ sub sim_hardware_cmd {
if ($cmd eq 'service') {
$sid = PVE::HA::Tools::pve_verify_ha_resource_id($objid);
+ } elsif ($cmd eq 'datacenter') {
+ # cluster-wide, neither node nor service specific
} else {
$node = $objid;
$d = $self->{nodes}->{$node}
@@ -1028,6 +1083,11 @@ sub sim_hardware_cmd {
die "sim_hardware_cmd: unknown service action '$action' "
. "- not implemented\n";
}
+ } elsif ($cmd eq 'datacenter') {
+ die "sim_hardware_cmd: missing action for 'datacenter $objid' command\n"
+ if !defined($action);
+
+ $self->update_datacenter_setting($objid, $action, @params);
} else {
die "sim_hardware_cmd: unknown command '$cmdstr'\n";
}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (16 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
pass the ha-cpu-weight and ha-mem-weight CRS options to the static and
dynamic scheduler, so that users can prioritize balancing the CPU load
or the memory usage of nodes. weights apply to the load imbalance, both
automatic rebalancing methods and the selection of a node to start or
recover a resource on
without configured weights, the TOPSIS method keeps favoring memory
usage 5 times over CPU load. if TOPSIS is the automatic rebalancing
method, pass weights of 1:5 in that case, so that the node imbalance is
weighted the same way and reflects what the TOPSIS method balances. this
changes the behavior for existing setups using the TOPSIS method: after
upgrading, their load imbalance is dominated by memory usage. setups
using the bruteforce method keep the unweighted load imbalance
log changes to the weights as they are passed to the scheduler, so that
the implicit memory weight of 5 shows up in the log when configuring or
removing weights, or when switching the automatic rebalancing method
if both weights are set to 0, no resource contributes to the node load,
so there is nothing to balance. skip automatic rebalancing and log a
warning once in that case. the weights are not passed to the scheduler
then, so selecting a node to start or recover a resource on keeps
working with the default weights instead of failing to score
allow changing the weights in the simulator at runtime, and add a test
case with both weights set to 0, which neither rebalances nor fails to
start the resources. Update the expected log of the existing TOPSIS test
case whose node imbalance changes with the weights of 1:5: the reported
imbalance values differ, and the last rebalancing migration is not done
anymore
passing weights requires a version of libpve-rs-perl that supports
scheduler options. older versions reject the options, so the scheduler
fails to initialize and falls back to the 'basic' mode. this happens
whenever weights are configured, and also without configured weights if
TOPSIS is the rebalancing method, as weights of 1:5 are passed then
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* name options ha-cpu-weight and ha-mem-weight, as they also apply
to placing resources on start and recovery
* weights are passed once on scheduler construction instead of to each
method, see the proxmox-perl-rs patch
* weights are kept outside of auto_rebalance
* skip rebalancing and warn once if both weights are 0
* log the weights as passed to the scheduler, including the implicit
memory weight of 5 for TOPSIS
* add zero-weights test case
* adjust existing topsis test
using weights requires a libpve-rs-perl version with scheduler options support
src/PVE/HA/Manager.pm | 46 ++++++++++++++++++-
src/PVE/HA/Sim/Hardware.pm | 2 +
src/PVE/HA/Usage/Dynamic.pm | 6 ++-
src/PVE/HA/Usage/Static.pm | 6 ++-
.../log.expect | 13 +-----
.../README | 6 +++
.../cmdlist | 3 ++
.../datacenter.cfg | 8 ++++
.../dynamic_service_stats | 6 +++
.../hardware_status | 5 ++
.../log.expect | 41 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 6 +++
.../static_service_stats | 6 +++
14 files changed, 138 insertions(+), 17 deletions(-)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 7de8781..3db5402 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -174,6 +174,39 @@ sub update_crs_settings {
0.1,
);
+ my ($cpu_weight, $mem_weight) = $crs_cfg->@{qw(ha-cpu-weight ha-mem-weight)};
+ my $weights_configured = defined($cpu_weight) || defined($mem_weight);
+ # the legacy TOPSIS criteria favor memory usage 5 times over CPU load by default, weigh the node
+ # load the same way, so that the imbalance reflects what the TOPSIS method balances
+ my $use_legacy_topsis_weights = !$weights_configured && $auto_rebalance->{method} eq 'topsis';
+
+ $self->update_crs_option('ha-cpu-weight', \$crs->{cpu_weight}, $cpu_weight, 1);
+ $self->update_crs_option(
+ 'ha-mem-weight',
+ \$crs->{mem_weight},
+ $mem_weight,
+ $use_legacy_topsis_weights ? 5 : 1,
+ );
+
+ my $scheduler_options;
+ my $all_weights_zero = 0;
+ if (!$crs->{cpu_weight} && !$crs->{mem_weight}) {
+ # no resource would contribute to the node load, so there is nothing to balance; keep
+ # the default weights for selecting nodes to start or recover resources on
+ $haenv->log(
+ 'warning',
+ "all CRS weights are set to 0, ignoring them and skipping automatic rebalancing",
+ ) if !$crs->{all_weights_zero};
+ $all_weights_zero = 1;
+ } elsif ($weights_configured || $use_legacy_topsis_weights) {
+ $scheduler_options = {
+ 'cpu-weight' => $crs->{cpu_weight},
+ 'mem-weight' => $crs->{mem_weight},
+ };
+ }
+ $crs->{scheduler_options} = $scheduler_options;
+ $crs->{all_weights_zero} = $all_weights_zero;
+
return;
}
@@ -276,6 +309,7 @@ sub load_balance {
return if !$auto_rebalance_opts->{enable};
return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
+ return if $crs->{all_weights_zero};
return if $self->any_resource_motion_queued_or_running();
my ($threshold, $method, $hold_duration, $margin) =
@@ -489,7 +523,11 @@ sub recompute_online_node_usage {
if ($have_static_scheduling) {
$online_node_usage = eval {
$service_stats = $haenv->get_static_service_stats();
- my $scheduler = PVE::HA::Usage::Static->new($haenv, $service_stats);
+ my $scheduler = PVE::HA::Usage::Static->new(
+ $haenv,
+ $service_stats,
+ $self->{crs}->{scheduler_options},
+ );
$scheduler->add_node($_) for $online_nodes->@*;
return $scheduler;
};
@@ -504,7 +542,11 @@ sub recompute_online_node_usage {
if ($have_dynamic_scheduling) {
$online_node_usage = eval {
$service_stats = $haenv->get_dynamic_service_stats();
- my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats);
+ my $scheduler = PVE::HA::Usage::Dynamic->new(
+ $haenv,
+ $service_stats,
+ $self->{crs}->{scheduler_options},
+ );
$scheduler->add_node($_) for $online_nodes->@*;
return $scheduler;
};
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index f3d8301..957548a 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -712,6 +712,8 @@ my $datacenter_settings = {
'ha-auto-rebalance-method' => qr/^(?:bruteforce|topsis)$/,
'ha-auto-rebalance-hold-duration' => qr/^\d+$/,
'ha-auto-rebalance-margin' => qr/^\d+(?:\.\d+)?$/,
+ 'ha-cpu-weight' => qr/^\d+$/,
+ 'ha-mem-weight' => qr/^\d+$/,
},
};
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..0da6115 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -9,12 +9,14 @@ use PVE::RS::ResourceScheduling::Dynamic;
use base qw(PVE::HA::Usage);
sub new {
- my ($class, $haenv, $service_stats) = @_;
+ my ($class, $haenv, $service_stats, $scheduler_options) = @_;
my $node_stats = eval { $haenv->get_dynamic_node_stats() };
die "did not get dynamic node usage information - $@" if $@;
- my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() };
+ # only pass options if there are any, so that older versions of the bindings still work
+ my @args = defined($scheduler_options) ? ($scheduler_options) : ();
+ my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new(@args) };
die "unable to initialize dynamic scheduling - $@" if $@;
return bless {
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index e67d5f5..9b91ba6 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -9,12 +9,14 @@ use PVE::RS::ResourceScheduling::Static;
use base qw(PVE::HA::Usage);
sub new {
- my ($class, $haenv, $service_stats) = @_;
+ my ($class, $haenv, $service_stats, $scheduler_options) = @_;
my $node_stats = eval { $haenv->get_static_node_stats() };
die "did not get static node usage information - $@" if $@;
- my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(); };
+ # only pass options if there are any, so that older versions of the bindings still work
+ my @args = defined($scheduler_options) ? ($scheduler_options) : ();
+ my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(@args) };
die "unable to initialize static scheduling - $@" if $@;
return bless {
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 8779848..0847a04 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -55,7 +55,7 @@ info 25 node3/lrm: service status vm:107 started
info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912
info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192
info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011
-info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%)
+info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%)
info 160 node1/crm: got crm command: migrate vm:105 node2
info 160 node1/crm: migrate service 'vm:105' to node 'node2'
info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2)
@@ -70,7 +70,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8
info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201
info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211
info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191
-info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%)
+info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%)
info 260 node1/crm: got crm command: migrate vm:103 node3
info 260 node1/crm: migrate service 'vm:103' to node 'node3'
info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3)
@@ -79,13 +79,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
info 285 node3/lrm: starting service vm:103
info 285 node3/lrm: service status vm:103 started
-info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
-info 340 node1/crm: got crm command: migrate vm:105 node1
-info 340 node1/crm: migrate service 'vm:105' to node 'node1'
-info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
-info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
-info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
-info 361 node1/lrm: starting service vm:105
-info 361 node1/lrm: service status vm:105 started
info 820 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README
new file mode 100644
index 0000000..8108448
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README
@@ -0,0 +1,6 @@
+Test that the auto rebalance system with dynamic usage information does not
+rebalance if all CRS weights are set to 0, as no resource contributes to the
+node load then.
+
+The same setup without weights causes two rebalancing migrations, see
+test-crs-dynamic-auto-rebalance2. A warning is logged once instead.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist
new file mode 100644
index 0000000..13f90cd
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg
new file mode 100644
index 0000000..c395bc8
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg
@@ -0,0 +1,8 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1,
+ "ha-cpu-weight": 0,
+ "ha-mem-weight": 0
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats
new file mode 100644
index 0000000..f01fd76
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats
@@ -0,0 +1,6 @@
+{
+ "vm:101": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:102": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:103": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status
new file mode 100644
index 0000000..ce8cf0e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect
new file mode 100644
index 0000000..792a3a2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect
@@ -0,0 +1,41 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-cpu-weight '0'
+info 20 node1/crm: using ha-mem-weight '0'
+warn 20 node1/crm: all CRS weights are set to 0, ignoring them and skipping automatic rebalancing
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 20 node1/crm: adding new service 'vm:104' on node 'node1'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 21 node1/lrm: starting service vm:104
+info 21 node1/lrm: service status vm:104 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config
new file mode 100644
index 0000000..b5960cb
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config
@@ -0,0 +1,6 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node1", "state": "started" },
+ "vm:104": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats
new file mode 100644
index 0000000..6cf8c10
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats
@@ -0,0 +1,6 @@
+{
+ "vm:101": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
` (17 preceding siblings ...)
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
@ 2026-11-12 9:44 ` Dominik Rusovac
18 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-12 9:44 UTC (permalink / raw)
To: pve-devel
add test cases which document that the crs weights steer the automatic
rebalancing towards balancing the CPU load or the memory usage of the
nodes; increasing a weight lets the load imbalance exceed the threshold
for a cluster that is balanced with equal weights; and a weight of 0
balances the other resource only
Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---
Notes:
changes since v1:
* reword commit
* use the new option names
* drop the redundant commands setting the weights to 1
* add test cases with one of the weights set to 0
.../README | 4 ++
.../cmdlist | 4 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 61 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 4 ++
.../cmdlist | 6 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 65 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 5 ++
.../cmdlist | 4 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 61 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 5 ++
.../cmdlist | 4 ++
.../datacenter.cfg | 7 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 62 ++++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
.../README | 5 ++
.../cmdlist | 4 ++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 8 +++
.../hardware_status | 5 ++
.../log.expect | 61 +++++++++++++++++
.../manager_status | 1 +
.../service_config | 8 +++
.../static_service_stats | 8 +++
45 files changed, 536 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
new file mode 100644
index 0000000..05cb5df
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
@@ -0,0 +1,4 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing ha-cpu-weight
+increases the node imbalance and triggers a rebalancing migration from
+the CPU-heavy node3 to node2, which has a low CPU load and memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
new file mode 100644
index 0000000..869d8e8
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-cpu-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
new file mode 100644
index 0000000..cf770d5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
@@ -0,0 +1,61 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-cpu-weight 2
+info 120 node1/crm: switching ha-cpu-weight from '1' to '2'
+info 160 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%)
+info 160 node1/crm: got crm command: migrate vm:106 node2
+info 160 node1/crm: migrate service 'vm:106' to node 'node2'
+info 160 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 165 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 165 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 180 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 183 node2/lrm: starting service vm:106
+info 183 node2/lrm: service status vm:106 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
new file mode 100644
index 0000000..e2568fb
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
@@ -0,0 +1,4 @@
+With equal weights, the node imbalance is 18.7%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that incrementing
+ha-mem-weight eventually triggers a rebalancing migration from the
+memory-heavy node3.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
new file mode 100644
index 0000000..375ab2f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-mem-weight 2" ],
+ [ "datacenter crs set ha-mem-weight 3" ],
+ [ "datacenter crs set ha-mem-weight 4" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
new file mode 100644
index 0000000..14740f1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 10.0, "mem": 3221225472 },
+ "vm:102": { "cpu": 5.0, "mem": 1073741824 },
+ "vm:103": { "cpu": 5.0, "mem": 3221225472 },
+ "vm:104": { "cpu": 3.0, "mem": 1073741824 },
+ "vm:105": { "cpu": 0.5, "mem": 32212254720 },
+ "vm:106": { "cpu": 0.5, "mem": 6442450944 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
new file mode 100644
index 0000000..6dd1901
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
@@ -0,0 +1,65 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-mem-weight 2
+info 120 node1/crm: switching ha-mem-weight from '1' to '2'
+info 220 cmdlist: execute datacenter crs set ha-mem-weight 3
+info 220 node1/crm: switching ha-mem-weight from '2' to '3'
+info 320 cmdlist: execute datacenter crs set ha-mem-weight 4
+info 320 node1/crm: switching ha-mem-weight from '3' to '4'
+info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%)
+info 360 node1/crm: got crm command: migrate vm:106 node2
+info 360 node1/crm: migrate service 'vm:106' to node 'node2'
+info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 365 node3/lrm: service vm:106 - start migrate to node 'node2'
+info 365 node3/lrm: service vm:106 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:106
+info 383 node2/lrm: service status vm:106 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
new file mode 100644
index 0000000..e9ea92c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing ha-mem-weight
+increases the node imbalance and triggers a rebalancing migration from
+the memory-heavy node1 to node2, which has a low CPU load and memory
+usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
new file mode 100644
index 0000000..ed207f5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-mem-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
new file mode 100644
index 0000000..91f6695
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
@@ -0,0 +1,61 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-mem-weight 2
+info 120 node1/crm: switching ha-mem-weight from '1' to '2'
+info 160 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%)
+info 160 node1/crm: got crm command: migrate vm:102 node2
+info 160 node1/crm: migrate service 'vm:102' to node 'node2'
+info 160 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 161 node1/lrm: service vm:102 - start migrate to node 'node2'
+info 161 node1/lrm: service vm:102 - end migrate to node 'node2'
+info 180 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2)
+info 183 node2/lrm: starting service vm:102
+info 183 node2/lrm: service status vm:102 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
new file mode 100644
index 0000000..82e46f8
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 35%. Test that setting ha-cpu-weight to
+0 balances the memory usage only, which triggers a rebalancing
+migration from the memory-heavy node1 to node3, which has the highest
+CPU load but a low memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
new file mode 100644
index 0000000..863befe
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-cpu-weight 0" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
new file mode 100644
index 0000000..7fcaf00
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
@@ -0,0 +1,7 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1,
+ "ha-auto-rebalance-threshold": 35
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
new file mode 100644
index 0000000..b0cdaaf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
@@ -0,0 +1,62 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: using ha-auto-rebalance-threshold '0.35'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-cpu-weight 0
+info 120 node1/crm: switching ha-cpu-weight from '1' to '0'
+info 160 node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 62.6% to 30.4%)
+info 160 node1/crm: got crm command: migrate vm:102 node3
+info 160 node1/crm: migrate service 'vm:102' to node 'node3'
+info 160 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 161 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 161 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 180 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 185 node3/lrm: starting service vm:102
+info 185 node3/lrm: service status vm:102 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
new file mode 100644
index 0000000..8ede76c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that setting ha-mem-weight to
+0 balances the CPU load only, which triggers a rebalancing migration
+from the CPU-heavy node3 to node1, which has the lowest CPU load but
+the highest memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
new file mode 100644
index 0000000..12b01a7
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "datacenter crs set ha-mem-weight 0" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+ "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+ "vm:103": { "cpu": 2.0, "mem": 8589934592 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:105": { "cpu": 9.0, "mem": 5368709120 },
+ "vm:106": { "cpu": 6.0, "mem": 3221225472 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
new file mode 100644
index 0000000..1a7e746
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
@@ -0,0 +1,61 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: using ha-auto-rebalance '1'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node2'
+info 20 node1/crm: adding new service 'vm:104' on node 'node2'
+info 20 node1/crm: adding new service 'vm:105' on node 'node3'
+info 20 node1/crm: adding new service 'vm:106' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:103
+info 23 node2/lrm: service status vm:103 started
+info 23 node2/lrm: starting service vm:104
+info 23 node2/lrm: service status vm:104 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:105
+info 25 node3/lrm: service status vm:105 started
+info 25 node3/lrm: starting service vm:106
+info 25 node3/lrm: service status vm:106 started
+info 120 cmdlist: execute datacenter crs set ha-mem-weight 0
+info 120 node1/crm: switching ha-mem-weight from '1' to '0'
+info 160 node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 62.6% to 27.8%)
+info 160 node1/crm: got crm command: migrate vm:106 node1
+info 160 node1/crm: migrate service 'vm:106' to node 'node1'
+info 160 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 165 node3/lrm: service vm:106 - start migrate to node 'node1'
+info 165 node3/lrm: service vm:106 - end migrate to node 'node1'
+info 180 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node1)
+info 181 node1/lrm: starting service vm:106
+info 181 node1/lrm: service status vm:106 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node2", "state": "started" },
+ "vm:104": { "node": "node2", "state": "started" },
+ "vm:105": { "node": "node3", "state": "started" },
+ "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
@@ -0,0 +1,8 @@
+{
+ "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+ "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 01/19] resource-scheduling: fix typos
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
@ 2026-11-19 12:27 ` Daniel Kral
0 siblings, 0 replies; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:27 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
@ 2026-11-19 12:27 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:27 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> to be more precise, refine parts of docs of functions responsible for
> scoring migration candidates and selecting node to start service on,
> respectively
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> new, split out of patch [0] in v1
>
> respective changes:
> * slightly reword docs
>
> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
> proxmox-resource-scheduling/src/scheduler.rs | 12 ++++++------
> 1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index 9078fa72..60e60d3c 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -228,15 +228,15 @@ impl Scheduler {
> })
> }
>
> - /// Scores the given migration `candidates` by the best node imbalance improvement with
> - /// exhaustive search.
> + /// Uses bruteforce search to score how much each of the given migration `candidates` improves
> + /// node imbalance and ranks them accordingly.
nit: if possible start this with 'Scores'/'Ranks' here as well just to
be consistent with the TOPSIS doc
> ///
> /// The `candidates` are assumed to be consistent with the scheduler. No further validation is
> /// done whether the given nodenames actually exist in the scheduler.
> ///
> /// The scoring is done as if each resource migration has already been done. This assumes that
> /// the already migrated resource consumes the same amount of each stat as on the previous node
> - /// according to its `stats`.
> + /// according to its `stats`. The lower the score, the better.
> ///
> /// Returns up to `limit` of the best scored migrations.
> pub fn score_best_balancing_migration_candidates<I>(
> @@ -269,8 +269,7 @@ impl Scheduler {
> best_migrations
> }
>
> - /// Scores the given migration `candidates` by the best node imbalance improvement with the
> - /// TOPSIS method.
> + /// Scores the given migration `candidates` using the TOPSIS method.
nit: maybe use 'Ranks' here as well and clarify that the 'score_' is a
misnomer in the commit message (I'm sorry ;-))
> ///
> /// The `candidates` are assumed to be consistent with the scheduler. No further validation is
> /// done whether the given nodenames actually exist in the scheduler.
> @@ -321,7 +320,8 @@ impl Scheduler {
> .collect())
> }
>
> - /// Scores nodes to start a resource with the usage statistics `resource_stats` on.
> + /// Scores nodes to start a resource with the usage statistics `resource_stats` on, using the
> + /// TOPSIS method.
Here scoring is the accurate description of course, because it does not
sort/rank them, just scoring.
> ///
> /// The scoring is done as if the resource is already started on each node. This assumes that
> /// the already started resource consumes the maximum amount of each stat according to its
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
@ 2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:28 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> DRY and hide repeatedly used closure inside reusable function
>
> Reviewed-by: Daniel Kral <d.kral@proxmox.com>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
The Git trailers should be ordered causally, so you Signed-off-by first
and then the Reviews/Tests/etc. follow.
Patch LGTM, nice refactoring! Anyway:
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
@ 2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:50 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:28 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
nit: drop the 'fix #7993' as this is only prepatory for adding weights
later, which is the actual fix
With or without the nits, consider as:
Reviewed-by: Daniel Kral <d.kral@proxmox.com>
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> add weighted node load, which computes the node load as the weighted
> arithmetic mean [0] of CPU load and memory usage, alongside basic unit
> tests
>
> this is in preparation for letting users prioritize balancing one
> resource over the other
>
> [0] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> new with changes regarding patch [0] in v1:
> * add weighted_load function
> * add NodeStatsWeights struct
> - make NodeStatsWeights #[non_exhaustive]
> - callers ought to start constructing weights via default function and
> then set the public fields, if required
> - exhaustive deconstructuring in weighted_load function will force
> compile error for new field, which is desirable
> - a new field must default to 0.0, otherwise a caller starting from
> default function might silently get a different weighted_load
> result
>
> dispatched changes into this new patch to make the series more organized
>
> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
> proxmox-resource-scheduling/src/node.rs | 114 ++++++++++++++++++++++++
> 1 file changed, 114 insertions(+)
>
> diff --git a/proxmox-resource-scheduling/src/node.rs b/proxmox-resource-scheduling/src/node.rs
> index 3c7ed393..98568bb7 100644
> --- a/proxmox-resource-scheduling/src/node.rs
> +++ b/proxmox-resource-scheduling/src/node.rs
> @@ -2,6 +2,39 @@ use std::collections::HashSet;
>
> use crate::resource::ResourceStats;
>
> +/// Weights controlling how much the CPU load and the memory usage of a node contribute to its
> +/// [`NodeStats::weighted_load`].
> +///
> +/// Only the ratio of the weights matters, not their absolute magnitude. A weight of 0.0 means that
> +/// the respective resource does not contribute at all.
> +///
> +/// Callers outside this crate start from [`NodeStatsWeights::default`] and set the weights they
> +/// need.
> +#[non_exhaustive]
> +#[derive(Copy, Clone, PartialEq, Debug)]
> +pub struct NodeStatsWeights {
> + /// Weight of [`NodeStats::cpu_load`]. Must be finite and non-negative.
> + pub cpu_weight: f64,
> + /// Weight of [`NodeStats::mem_load`]. Must be finite and non-negative.
> + pub mem_weight: f64,
> +}
> +
> +impl NodeStatsWeights {
> + /// Equal weights, making [`NodeStats::weighted_load`] the same as [`NodeStats::load`].
> + ///
> + /// NOTE: any pair of equal weights bigger than 0.0 works here
> + pub const EQUAL: Self = Self {
> + cpu_weight: 1.0,
> + mem_weight: 1.0,
> + };
> +}
> +
> +impl Default for NodeStatsWeights {
> + fn default() -> Self {
> + Self::EQUAL
> + }
> +}
> +
> /// Usage statistics of a node.
> #[derive(Copy, Clone, PartialEq, PartialOrd, Debug, Default)]
> pub struct NodeStats {
> @@ -55,6 +88,28 @@ impl NodeStats {
> pub fn load(&self) -> f64 {
> (self.cpu_load() + self.mem_load()) / 2.0
> }
> +
> + /// Returns a combined node usage as a percentage, which is the [weighted arithmetic mean] of
> + /// [`Self::cpu_load`] and [`Self::mem_load`] with respect to `weights`.
nit: Returns the [weighted] combined node usage as a percentage with
respect to `weights`.
Just to not expose too much implementation detail in the docs, which
needs to be changed if node stat and weight fields should ever be added
in the future, but this is just a matter of taste, so can also be
ignored.
> + ///
> + /// With equal weights, this is the same as [`Self::load`].
> + ///
> + /// [weighted arithmetic mean]: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean
> + pub fn weighted_load(&self, weights: &NodeStatsWeights) -> f64 {
> + let NodeStatsWeights {
> + cpu_weight,
> + mem_weight,
> + } = *weights;
> +
> + let weights_sum = cpu_weight + mem_weight;
> +
> + if weights_sum == 0.0 {
> + // neither should contribute to load, so there is no load
> + return 0.0;
> + }
> +
> + (self.cpu_load() * cpu_weight + self.mem_load() * mem_weight) / weights_sum
> + }
> }
>
> /// A node in the cluster context.
> @@ -94,3 +149,62 @@ impl Node {
> self.resources.contains(sid)
> }
> }
> +
> +#[cfg(test)]
> +mod tests {
> + use super::*;
> +
> + fn stats() -> NodeStats {
> + NodeStats {
> + cpu: 4.0,
> + maxcpu: 16,
> + mem: 48 << 30,
> + maxmem: 64 << 30,
> + }
> + }
> +
> + #[test]
> + fn test_weighted_load_with_equal_weights_is_load() {
> + let stats = stats();
> +
> + assert_eq!(stats.weighted_load(&NodeStatsWeights::EQUAL), stats.load());
> + assert_eq!(
> + stats.weighted_load(&NodeStatsWeights {
> + cpu_weight: 3.0,
> + mem_weight: 3.0
> + }),
> + stats.load()
> + );
> + }
> +
> + #[test]
> + fn test_weighted_load() {
> + let stats = stats();
> +
> + assert_eq!(
> + stats.weighted_load(&NodeStatsWeights {
> + cpu_weight: 1.0,
> + mem_weight: 0.0
> + }),
> + stats.cpu_load()
> + );
> + assert_eq!(
> + stats.weighted_load(&NodeStatsWeights {
> + cpu_weight: 0.0,
> + mem_weight: 1.0
> + }),
> + stats.mem_load()
> + );
> + }
> +
> + #[test]
> + fn test_weighted_load_with_zero_weights_is_zero() {
> + assert_eq!(
> + stats().weighted_load(&NodeStatsWeights {
> + cpu_weight: 0.0,
> + mem_weight: 0.0
> + }),
> + 0.0
> + );
> + }
> +}
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
@ 2026-11-19 12:40 ` Daniel Kral
2026-11-19 13:52 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:40 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> CRS did not rebalance a cluster that was clearly imbalanced in CPU load
> or memory usage alone, because the combined node load looked balanced
> enough [0]
>
> add weights to the scheduler, so that callers can prioritize balancing
> one resource over the other
>
> load imbalance computations and the bruteforce method use the weighted
> node load
>
> for the TOPSIS methods, the weights replace the hardcoded criteria
> weights from commit 2b325aa5 ("add pve_static module"). those favor
> memory usage 5 times over CPU load. Scaling them with the weights
> instead would make the weights relative to that built-in emphasis, so
> 1:1 would effectively mean 1:5, which is undesirable. only the factor of
> 2 between the criteria for the highest and the average usage of each
> resource is kept, so custom weights of 1:5 reproduce the previous
> criteria weights of 1:5 (average) and 2:10 (highest)
>
> the same weights are used for the TOPSIS criteria and the node load, so
> the imbalance reported for the migrations ranked by TOPSIS is weighted
> like the criteria that ranked them
>
> the weights default to 1:1. This changes the default ranking of the
> TOPSIS methods (expected result of one affected test was adjusted), so
> callers have to pass hardcoded legacy topsis weights to keep their
> previous rankings. with these, the node load is weighted 1:5 as well and
> the previous rankings are preserved, which is also being tested
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> new with changes regarding patch [0] in v1:
> * add weights as field to Scheduler struct
> * weigh the TOPSIS criteria directly with the weights, keeping the factor
> of 2 between the highest and the average usage criteria of each resource
> - weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
> - add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
> * fall back to previous TOPSIS ranking of migration candidates
> * no longer expose score and keep ScoredMigration as is
> * adjust one scheduler test's expected result
> * test that legacy topsis weights preserve previous rankings
>
> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>
> proxmox-resource-scheduling/src/pve_static.rs | 4 +-
> proxmox-resource-scheduling/src/scheduler.rs | 105 ++++++++++++++----
> .../tests/scheduler.rs | 64 ++++++++++-
> 3 files changed, 149 insertions(+), 24 deletions(-)
>
> diff --git a/proxmox-resource-scheduling/src/pve_static.rs b/proxmox-resource-scheduling/src/pve_static.rs
> index 40b76cbc..25439a0e 100644
> --- a/proxmox-resource-scheduling/src/pve_static.rs
> +++ b/proxmox-resource-scheduling/src/pve_static.rs
> @@ -1,7 +1,7 @@
> use anyhow::Error;
> use serde::{Deserialize, Serialize};
>
> -use crate::scheduler::{NodeUsage, Scheduler};
> +use crate::scheduler::{LEGACY_TOPSIS_WEIGHTS, NodeUsage, Scheduler};
> use crate::{node::NodeStats, resource::ResourceStats};
>
> #[derive(Clone, Serialize, Deserialize)]
> @@ -94,7 +94,7 @@ pub fn score_nodes_to_start_service<T: AsRef<StaticNodeUsage>>(
> .map(|node| node.as_ref().clone().into())
> .collect::<Vec<NodeUsage>>();
>
> - let scheduler = Scheduler::from_nodes(nodes);
> + let scheduler = Scheduler::from_nodes(nodes).with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> scheduler.score_nodes_to_start_resource(*service)
> }
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index a9390357..75929a14 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -1,6 +1,10 @@
> use anyhow::Error;
>
> -use crate::{node::NodeStats, resource::ResourceStats, topsis};
> +use crate::{
> + node::{NodeStats, NodeStatsWeights},
> + resource::ResourceStats,
> + topsis,
> +};
>
> use serde::{Deserialize, Serialize};
> use std::{
> @@ -69,26 +73,69 @@ fn node_stats_after_migration(candidate: &MigrationCandidate, node: &NodeUsage)
> new_stats
> }
>
> -criteria_struct! {
> - /// A given alternative.
> - struct PveTopsisAlternative {
> - #[criterion("average CPU", -1.0)]
> - average_cpu: f64,
> - #[criterion("highest CPU", -2.0)]
> - highest_cpu: f64,
> - #[criterion("average memory", -5.0)]
> - average_memory: f64,
> - #[criterion("highest memory", -10.0)]
> - highest_memory: f64,
> +/// A given alternative.
> +struct PveTopsisAlternative {
> + average_cpu: f64,
> + highest_cpu: f64,
> + average_memory: f64,
> + highest_memory: f64,
> +}
> +
> +const N_CRITERIA: usize = 4;
> +
> +impl From<PveTopsisAlternative> for [f64; N_CRITERIA] {
> + fn from(alternative: PveTopsisAlternative) -> Self {
> + [
> + alternative.average_cpu,
> + alternative.highest_cpu,
> + alternative.average_memory,
> + alternative.highest_memory,
> + ]
> }
> +}
>
> - const N_CRITERIA;
> - static PVE_HA_TOPSIS_CRITERIA;
> +/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
> +/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
> +/// weights introduced in commit 2b325aa5.
nit: include the commit title here, i.e.
2b325aa5 ("add pve_static module")
> +///
> +/// For more details see [`topsis_criteria`].
> +const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
> +
> +/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with
here as well
> +/// [`topsis_criteria`].
> +pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
> + cpu_weight: 1.0,
> + mem_weight: 5.0,
> +};
> +
> +/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
> +///
> +/// The criteria for the average usage of each resource get the respective weight in `weights`,
> +/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
> +fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {
nit: topsis_criteria_from_weights() or make this a method of Scheduler,
which is its only user
> + let NodeStatsWeights {
> + cpu_weight,
> + mem_weight,
> + } = *weights;
> +
> + topsis::Criteria::new([
> + topsis::Criterion::new("average CPU".to_owned(), -cpu_weight),
> + topsis::Criterion::new(
> + "highest CPU".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * cpu_weight,
> + ),
> + topsis::Criterion::new("average memory".to_owned(), -mem_weight),
> + topsis::Criterion::new(
> + "highest memory".to_owned(),
> + -TOPSIS_HIGHEST_USAGE_FACTOR * mem_weight,
> + ),
> + ])
> }
>
> #[derive(Clone, Debug)]
> pub struct Scheduler {
> nodes: Vec<NodeUsage>,
> + weights: NodeStatsWeights,
> }
>
> /// A possible migration.
> @@ -176,9 +223,28 @@ impl Scheduler {
> {
> Self {
> nodes: nodes.into_iter().map(|node| node.into()).collect(),
> + weights: NodeStatsWeights::EQUAL,
> }
> }
>
> + /// Sets the `weights` controlling how much balancing the CPU load and the memory usage of the
> + /// nodes is prioritized relative to each other.
> + ///
> + /// For [`Self::node_imbalance`], [`Self::node_imbalance_with_migration_candidate`], the
> + /// bruteforce method [`Self::score_best_balancing_migration_candidates`] and the imbalance
> + /// reported by [`Self::score_best_balancing_migration_candidates_topsis`], the `weights`
> + /// determine the [`NodeStats::weighted_load`] of each node.
> + ///
> + /// For the TOPSIS-based methods, [`Self::score_best_balancing_migration_candidates_topsis`] and
> + /// [`Self::score_nodes_to_start_resource`], the `weights` also weigh the average CPU and
> + /// average memory criteria, see [`topsis_criteria`]. Use [`LEGACY_TOPSIS_WEIGHTS`] to keep the
> + /// criteria weights introduced in commit 2b325aa5.
> + pub fn with_weights(mut self, weights: NodeStatsWeights) -> Self {
> + self.weights = weights;
> +
> + self
> + }
> +
> /// Maps the current node usages to a [`PveTopsisAlternative`].
> ///
> /// The [`PveTopsisAlternative`] is derived by calculating a modified version of the root mean
> @@ -221,7 +287,7 @@ impl Scheduler {
> ///
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance(&self) -> f64 {
> - calculate_node_imbalance(&self.nodes, |node| node.stats.load())
> + calculate_node_imbalance(&self.nodes, |node| node.stats.weighted_load(&self.weights))
> }
>
> /// Returns the load imbalance among the nodes as if a specific resource was moved.
> @@ -229,7 +295,7 @@ impl Scheduler {
> /// See [`calculate_node_imbalance`] for more information.
> pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> - node_stats_after_migration(candidate, node).load()
> + node_stats_after_migration(candidate, node).weighted_load(&self.weights)
> })
> }
>
> @@ -297,8 +363,9 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> + let criteria = topsis_criteria(&self.weights)?;
> let best_alternatives =
> - topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + topsis::rank_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(best_alternatives
> .into_iter()
> @@ -343,8 +410,8 @@ impl Scheduler {
> })
> .collect::<Vec<_>>();
>
> - let scores =
> - topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &PVE_HA_TOPSIS_CRITERIA)?;
> + let criteria = topsis_criteria(&self.weights)?;
> + let scores = topsis::score_alternatives(&topsis::Matrix::new(matrix)?, &criteria)?;
>
> Ok(scores
> .into_iter()
> diff --git a/proxmox-resource-scheduling/tests/scheduler.rs b/proxmox-resource-scheduling/tests/scheduler.rs
> index 61fdd840..603a9418 100644
> --- a/proxmox-resource-scheduling/tests/scheduler.rs
> +++ b/proxmox-resource-scheduling/tests/scheduler.rs
> @@ -2,7 +2,7 @@ use anyhow::Error;
> use proxmox_resource_scheduling::{
> node::NodeStats,
> resource::ResourceStats,
> - scheduler::{Migration, MigrationCandidate, NodeUsage, Scheduler},
> + scheduler::{LEGACY_TOPSIS_WEIGHTS, Migration, MigrationCandidate, NodeUsage, Scheduler},
> };
>
> fn new_empty_cluster_scheduler() -> Scheduler {
> @@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
is the scheduler here intentionally not augmented with
.with_weights(LEGACY_TOPSIS_WEIGHTS) ?
>
> assert_eq!(
> rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
> - vec!["node2", "node3", "node1"]
> + vec!["node1", "node3", "node2"]
> );
>
> Ok(())
> @@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> #[test]
> fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
> - let scheduler = new_heterogeneous_cluster_scheduler();
> + let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
>
> let heavy_memory_resource_stats = ResourceStats {
> cpu: 0.0,
> @@ -377,3 +377,61 @@ fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> Ok(())
> }
> +
> +#[test]
> +fn test_legacy_topsis_weights_keep_unweighted_rankings() -> Result<(), Error> {
> + let resource_stats =
> + [(1.0, 12 << 30), (12.0, 0), (0.0, 0), (12.0, 12 << 30)].map(|(maxcpu, maxmem)| {
> + ResourceStats {
> + cpu: 0.0,
> + maxcpu,
> + mem: 0,
> + maxmem,
> + }
> + });
> +
> + // rankings with the TOPSIS criteria weights from before the weights became adjustable
> + let expected_homogeneous_rankings = [
> + ["node2", "node3", "node1"],
> + ["node1", "node3", "node2"],
> + ["node1", "node3", "node2"],
> + ["node2", "node3", "node1"],
> + ];
> + let expected_heterogeneous_rankings = [
> + ["node2", "node1", "node3"],
> + ["node3", "node2", "node1"],
> + ["node1", "node3", "node2"],
> + ["node2", "node1", "node3"],
> + ];
> +
> + let scenarios: [(fn() -> Scheduler, _); 2] = [
> + (
> + new_homogeneous_cluster_scheduler,
> + expected_homogeneous_rankings,
> + ),
> + (
> + new_heterogeneous_cluster_scheduler,
> + expected_heterogeneous_rankings,
> + ),
> + ];
> +
> + for (new_scheduler, expected_rankings) in scenarios {
> + let legacy = new_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
> +
> + for (resource_stats, expected_ranking) in resource_stats.iter().zip(expected_rankings) {
> + assert_eq!(
> + rank_nodes_to_start_resource(&legacy, *resource_stats)?,
> + expected_ranking
> + );
> + }
> +
> + let (candidates, migration1, migration2) = new_simple_migration_candidates();
> +
> + assert_eq!(
> + rank_best_balancing_migration_candidates_topsis(&legacy, &candidates, 2)?,
> + vec![migration1, migration2]
> + );
> + }
> +
> + Ok(())
> +}
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
@ 2026-11-19 12:41 ` Daniel Kral
2026-11-19 12:52 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:41 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> this is made public, so callers like pve-rs bindings can compute load
> imbalance with migration candidates
maybe also note here that this is because of backwards compatibility
that is done in pve-rs, because imbalance has been calculated without
the hardcoded 1:5 there, but now that value would also be weighted with
the weights applied
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> no changes since v1, new
>
> ultimately, required to avoid mismatch in imbalance comparison in older
> HA manager versions
>
> proxmox-resource-scheduling/src/scheduler.rs | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
> index d6508c15..a9390357 100644
> --- a/proxmox-resource-scheduling/src/scheduler.rs
> +++ b/proxmox-resource-scheduling/src/scheduler.rs
> @@ -227,7 +227,7 @@ impl Scheduler {
> /// Returns the load imbalance among the nodes as if a specific resource was moved.
> ///
> /// See [`calculate_node_imbalance`] for more information.
> - fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> + pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
> calculate_node_imbalance(&self.nodes, |node| {
> node_stats_after_migration(candidate, node).load()
> })
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
@ 2026-11-19 12:45 ` Daniel Kral
2026-11-19 13:28 ` Dominik Rusovac
0 siblings, 1 reply; 33+ messages in thread
From: Daniel Kral @ 2026-11-19 12:45 UTC (permalink / raw)
To: Dominik Rusovac, pve-devel
On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
> allow passing options to the constructor of the static and dynamic
> scheduler, which specify the weights of the CPU load and memory usage
> applied to every load balancing computation. these options are a single
> optional parameter with only optional fields, so the signatures of the
> existing methods stay the same
>
> with specified weights, every rebalancing method uses them for the load
> imbalance and the TOPSIS criteria alike, so that for example 1:1 means
> 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS
> criteria weights
>
> without specified weights, every method keeps its previous behavior: the
> load imbalance and the bruteforce method use equal weights (1:1); the
> TOPSIS methods use weights of 1:5 for their criteria
>
> NOTE: weights are not validated. if both are 0, there is simply no load
> imbalance and the TOPSIS methods fail, so callers must not pass such
> weights
>
> NOTE: to decide whether a migration improves the balance enough, the HA
> manager compares the load imbalance reported for the migrations ranked
> by TOPSIS with the current load imbalance. without specified weights,
> the TOPSIS ranking would report an imbalance weighted 1:5, while the
> load imbalance is unweighted. so, without specified weights, recompute
> the reported imbalance of the ranked migrations with equal weights, as
> before. while this requires reiterating all migrations candidates, it is
> required to keep older versions of the HA manager working as before by
> preventing them from running into this undesirable load imbalance
> mismatch
>
> add test cases checking that:
> - without options or with unspecified options, the load imbalance and
> the node scores stay the same as before
> - equal weights keep the unweighted load imbalance, regardless of their
> magnitude
> - weights of 1:5 keep the previous node scores, but change the load
> imbalance, as it's also being weighted 1:5
> - weighting only the CPU load or only the memory usage results in a
> different load imbalance
> - the TOPSIS ranking reports the same load imbalance as the bruteforce
> method, both without specified options and with weights of 1:5, as the
> HA manager compares it with the current load imbalance
> - without specified options, the TOPSIS ranking is the same as with
> weights of 1:5
> - the dynamic scheduler accepts options as well
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
>
> Notes:
> changes since v1:
> * reword commit
> * pass the weights once as optional scheduler options to the constructor
> instead of adding weight parameters to every method, as suggested in
> the review of v1, so the signatures of the existing methods stay the
> same and older HA manager versions can still call them
> * options are a single optional hash with only optional fields, a
> missing weight defaults to 1
> * without options, every method keeps its previous behavior, including
> the TOPSIS ranking with weights of 1:5
> * without options, recompute the imbalance reported with the TOPSIS
> ranking with equal weights, so that it stays comparable to the node
> imbalance for older HA manager version
> * weights also apply to score_nodes_to_start_service()
> * drop the adjustment of the existing test, as calling the methods
> without weights keeps working
> * add test cases for the scheduler options
>
> requires the proxmox-resource-scheduling patches of this series
>
> .../src/bindings/resource_scheduling/mod.rs | 1 +
> .../bindings/resource_scheduling/options.rs | 59 ++++++++
> .../resource_scheduling/pve_dynamic.rs | 71 +++++++--
> .../resource_scheduling/pve_static.rs | 64 ++++++--
> pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++
> 5 files changed, 304 insertions(+), 32 deletions(-)
> create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/mod.rs b/pve-rs/src/bindings/resource_scheduling/mod.rs
> index 87b4a03..dc94f21 100644
> --- a/pve-rs/src/bindings/resource_scheduling/mod.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/mod.rs
> @@ -1,5 +1,6 @@
> //! Resource scheduling related bindings.
>
> +mod options;
> mod resource;
> mod usage;
>
> diff --git a/pve-rs/src/bindings/resource_scheduling/options.rs b/pve-rs/src/bindings/resource_scheduling/options.rs
> new file mode 100644
> index 0000000..6759cc3
> --- /dev/null
> +++ b/pve-rs/src/bindings/resource_scheduling/options.rs
> @@ -0,0 +1,59 @@
> +//! Options for the resource scheduling bindings.
> +
> +use serde::Deserialize;
> +
> +use proxmox_resource_scheduling::node::NodeStatsWeights;
> +use proxmox_resource_scheduling::scheduler::{MigrationCandidate, ScoredMigration};
> +use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
> +
> +/// Options for a scheduler instance.
> +#[derive(Clone, Copy, Debug, Default, Deserialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct SchedulerOptions {
> + /// The weight of the CPU load of nodes. Defaults to 1 if only the other weight is specified.
> + pub cpu_weight: Option<usize>,
> + /// The weight of the memory usage of nodes. Defaults to 1 if only the other weight is
> + /// specified.
> + pub mem_weight: Option<usize>,
> +}
> +
> +impl SchedulerOptions {
> + /// Returns the weights or [`None`] if no weight is set.
> + pub fn weights(&self) -> Option<NodeStatsWeights> {
> + if self.cpu_weight.is_none() && self.mem_weight.is_none() {
> + return None;
> + }
> +
> + let mut weights = NodeStatsWeights::default();
> + // unspecified weight defaults to 1
> + weights.cpu_weight = self.cpu_weight.unwrap_or(1) as f64;
> + weights.mem_weight = self.mem_weight.unwrap_or(1) as f64;
> +
> + Some(weights)
> + }
> +}
> +
> +/// Replaces the imbalance of each of the `scored_migrations` with the unweighted imbalance of the
> +/// respective migration in `candidates`, for the cluster usage `usage` aggregated by `F`.
> +pub fn with_unweighted_imbalance<F: UsageAggregator>(
> + usage: &Usage,
> + candidates: &[MigrationCandidate],
> + scored_migrations: Vec<ScoredMigration>,
> +) -> Vec<ScoredMigration> {
> + let scheduler = usage.to_scheduler::<F>();
> +
> + scored_migrations
> + .into_iter()
> + // every scored migration should belong to some candidate
> + .flat_map(|scored_migration| {
> + candidates
> + .iter()
> + .find(|candidate| candidate.migration == scored_migration.migration)
> + .map(|candidate| {
> + let imbalance = scheduler.node_imbalance_with_migration_candidate(candidate);
> +
> + ScoredMigration::new(scored_migration.migration, imbalance)
> + })
> + })
> + .collect()
> +}
> diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> index 8e066e0..62d6426 100644
> --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
> @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
> use serde::{Deserialize, Serialize};
>
> use perlmod::Value;
> - use proxmox_resource_scheduling::node::NodeStats;
> + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
> use proxmox_resource_scheduling::resource::ResourceStats;
> - use proxmox_resource_scheduling::scheduler::ScoredMigration;
> - use proxmox_resource_scheduling::usage::Usage;
> + use proxmox_resource_scheduling::scheduler::{
> + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
> + };
> + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
>
> + use crate::bindings::resource_scheduling::options::{
> + SchedulerOptions, with_unweighted_imbalance,
> + };
> use crate::bindings::resource_scheduling::resource::{
> CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
> };
> @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic {
> /// A scheduler instance contains the cluster usage.
> pub struct Scheduler {
> inner: Mutex<Usage>,
> + weights: Option<NodeStatsWeights>,
> + }
> +
> + impl Scheduler {
> + /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
> + /// this instance or `default_weights` if this instance has none.
> + fn to_weighted_scheduler<F: UsageAggregator>(
> + &self,
> + usage: &Usage,
> + default_weights: NodeStatsWeights,
> + ) -> CoreScheduler {
> + usage
> + .to_scheduler::<F>()
> + .with_weights(self.weights.unwrap_or(default_weights))
> + }
Hm, not a big one, but this could also just live in
proxmox_resource_scheduling itself, where Usage implements a
pub fn to_weighted_scheduler<F: UsageAggregator>(&self, weights: NodeStatsWeights) -> Scheduler {
let node_usages = F::aggregate(self);
Scheduler::from_nodes(node_usages).with_weights(weights)
}
and then the unwrap_or() could be made explicitly where it's called to
it's clear that it's a default fallback in there.
No hard feelings though.
> }
>
> #[derive(Clone, Copy, Debug, Serialize, Deserialize)]
> @@ -83,15 +103,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> type DynamicResource = PveResource<DynamicResourceStats>;
>
> - /// Class method: Create a new [`Scheduler`] instance.
> + /// Class method: Create a new [`Scheduler`] instance with the optional `options`.
> ///
> - /// See [`proxmox_resource_scheduling::usage::Usage::new`].
> + /// See [`proxmox_resource_scheduling::usage::Usage::new`] and [`SchedulerOptions`].
> #[export(raw_return)]
> - pub fn new(#[raw] class: Value) -> Result<Value, Error> {
> + pub fn new(#[raw] class: Value, options: Option<SchedulerOptions>) -> Result<Value, Error> {
> let inner = Usage::new();
> + let weights = options.unwrap_or_default().weights();
>
> Ok(perlmod::instantiate_magic!(
> - &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner) })
> + &class, MAGIC => Box::new(Scheduler { inner: Mutex::new(inner), weights })
> ))
> }
>
> @@ -169,7 +190,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
> pub fn calculate_node_imbalance(#[try_from_ref] this: &Scheduler) -> f64 {
> let usage = this.inner.lock().unwrap();
>
> - usage.to_scheduler::<IdentityAggregator>().node_imbalance()
> + this.to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
> + .node_imbalance()
> }
>
> /// Method: Scores the given migration `candidates` by the best node imbalance improvement with
> @@ -186,8 +208,8 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>
> - Ok(usage
> - .to_scheduler::<IdentityAggregator>()
> + Ok(this
> + .to_weighted_scheduler::<IdentityAggregator>(&usage, NodeStatsWeights::EQUAL)
> .score_best_balancing_migration_candidates(candidates, limit))
> }
>
> @@ -205,9 +227,24 @@ pub mod pve_rs_resource_scheduling_dynamic {
>
> let candidates = decompose_compact_migration_candidates(&usage, candidates)?;
>
> - usage
> - .to_scheduler::<IdentityAggregator>()
> - .score_best_balancing_migration_candidates_topsis(&candidates, limit)
> + let scored_migrations = this
> + .to_weighted_scheduler::<IdentityAggregator>(&usage, LEGACY_TOPSIS_WEIGHTS)
> + .score_best_balancing_migration_candidates_topsis(&candidates, limit)?;
> +
> + if this.weights.is_some() {
> + return Ok(scored_migrations);
> + }
> +
> + // callers not passing any weights would compare the 1:5 weighted imbalance reported for the
> + // migrations ranked with the LEGACY_TOPSIS_WEIGHTS based TOPSIS method to the unweighted
> + // imbalance
> + //
> + // to avoid this mismatch, report the unweighted imbalance instead
good comment!
> + Ok(with_unweighted_imbalance::<IdentityAggregator>(
> + &usage,
> + &candidates,
> + scored_migrations,
> + ))
> }
>
> /// Method: Scores nodes to start a resource with the usage statistics `resource_stats` on.
> @@ -220,8 +257,10 @@ pub mod pve_rs_resource_scheduling_dynamic {
> ) -> Result<Vec<(String, f64)>, Error> {
> let usage = this.inner.lock().unwrap();
>
> - usage
> - .to_scheduler::<StartingAsStartedResourceAggregator>()
> - .score_nodes_to_start_resource(resource_stats)
> + this.to_weighted_scheduler::<StartingAsStartedResourceAggregator>(
> + &usage,
> + LEGACY_TOPSIS_WEIGHTS,
> + )
> + .score_nodes_to_start_resource(resource_stats)
> }
> }
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs
2026-11-19 12:27 ` Daniel Kral
@ 2026-11-19 12:47 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 12:47 UTC (permalink / raw)
To: Daniel Kral, pve-devel
thx for the review!
On Thu Nov 19, 2026 at 1:27 PM CET, Daniel Kral wrote:
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
[snip]
>> to be more precise, refine parts of docs of functions responsible for
>> scoring migration candidates and selecting node to start service on,
>> respectively
>>
>> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
>> ---
>>
>> Notes:
>> new, split out of patch [0] in v1
>>
>> respective changes:
>> * slightly reword docs
>>
>> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>>
>> proxmox-resource-scheduling/src/scheduler.rs | 12 ++++++------
>> 1 file changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
>> index 9078fa72..60e60d3c 100644
>> --- a/proxmox-resource-scheduling/src/scheduler.rs
>> +++ b/proxmox-resource-scheduling/src/scheduler.rs
>> @@ -228,15 +228,15 @@ impl Scheduler {
>> })
>> }
>>
>> - /// Scores the given migration `candidates` by the best node imbalance improvement with
>> - /// exhaustive search.
>> + /// Uses bruteforce search to score how much each of the given migration `candidates` improves
>> + /// node imbalance and ranks them accordingly.
>
> nit: if possible start this with 'Scores'/'Ranks' here as well just to
> be consistent with the TOPSIS doc
>
ack
>> ///
>> /// The `candidates` are assumed to be consistent with the scheduler. No further validation is
>> /// done whether the given nodenames actually exist in the scheduler.
>> ///
>> /// The scoring is done as if each resource migration has already been done. This assumes that
>> /// the already migrated resource consumes the same amount of each stat as on the previous node
>> - /// according to its `stats`.
>> + /// according to its `stats`. The lower the score, the better.
>> ///
>> /// Returns up to `limit` of the best scored migrations.
>> pub fn score_best_balancing_migration_candidates<I>(
>> @@ -269,8 +269,7 @@ impl Scheduler {
>> best_migrations
>> }
>>
>> - /// Scores the given migration `candidates` by the best node imbalance improvement with the
>> - /// TOPSIS method.
>> + /// Scores the given migration `candidates` using the TOPSIS method.
>
> nit: maybe use 'Ranks' here as well and clarify that the 'score_' is a
> misnomer in the commit message (I'm sorry ;-))
>
ack
[snip]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic
2026-11-19 12:28 ` Daniel Kral
@ 2026-11-19 12:47 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 12:47 UTC (permalink / raw)
To: Daniel Kral, pve-devel
On Thu Nov 19, 2026 at 1:28 PM CET, Daniel Kral wrote:
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
>> DRY and hide repeatedly used closure inside reusable function
>>
>> Reviewed-by: Daniel Kral <d.kral@proxmox.com>
>> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
>
> The Git trailers should be ordered causally, so you Signed-off-by first
> and then the Reviews/Tests/etc. follow.
my bad, will fix this in v3
thx for the review!
>
> Patch LGTM, nice refactoring! Anyway:
>
> Reviewed-by: Daniel Kral <d.kral@proxmox.com>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load
2026-11-19 12:28 ` Daniel Kral
@ 2026-11-19 12:50 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 12:50 UTC (permalink / raw)
To: Daniel Kral, pve-devel
On Thu Nov 19, 2026 at 1:28 PM CET, Daniel Kral wrote:
> nit: drop the 'fix #7993' as this is only prepatory for adding weights
> later, which is the actual fix
ack, thx for the review!
>
> With or without the nits, consider as:
>
> Reviewed-by: Daniel Kral <d.kral@proxmox.com>
>
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
[snip]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public
2026-11-19 12:41 ` Daniel Kral
@ 2026-11-19 12:52 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 12:52 UTC (permalink / raw)
To: Daniel Kral, pve-devel
On Thu Nov 19, 2026 at 1:41 PM CET, Daniel Kral wrote:
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
>> this is made public, so callers like pve-rs bindings can compute load
>> imbalance with migration candidates
>
> maybe also note here that this is because of backwards compatibility
> that is done in pve-rs, because imbalance has been calculated without
> the hardcoded 1:5 there, but now that value would also be weighted with
> the weights applied
>
ack, thx for the review!
>>
>> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
>> ---
>>
>> Notes:
>> no changes since v1, new
>>
>> ultimately, required to avoid mismatch in imbalance comparison in older
>> HA manager versions
>>
>> proxmox-resource-scheduling/src/scheduler.rs | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/proxmox-resource-scheduling/src/scheduler.rs b/proxmox-resource-scheduling/src/scheduler.rs
>> index d6508c15..a9390357 100644
>> --- a/proxmox-resource-scheduling/src/scheduler.rs
>> +++ b/proxmox-resource-scheduling/src/scheduler.rs
>> @@ -227,7 +227,7 @@ impl Scheduler {
>> /// Returns the load imbalance among the nodes as if a specific resource was moved.
>> ///
>> /// See [`calculate_node_imbalance`] for more information.
>> - fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
>> + pub fn node_imbalance_with_migration_candidate(&self, candidate: &MigrationCandidate) -> f64 {
>> calculate_node_imbalance(&self.nodes, |node| {
>> node_stats_after_migration(candidate, node).load()
>> })
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options
2026-11-19 12:45 ` Daniel Kral
@ 2026-11-19 13:28 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 13:28 UTC (permalink / raw)
To: Daniel Kral, pve-devel
thx for your review!
On Thu Nov 19, 2026 at 1:45 PM CET, Daniel Kral wrote:
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
>> allow passing options to the constructor of the static and dynamic
>> scheduler, which specify the weights of the CPU load and memory usage
>> applied to every load balancing computation. these options are a single
>> optional parameter with only optional fields, so the signatures of the
>> existing methods stay the same
>>
>> with specified weights, every rebalancing method uses them for the load
>> imbalance and the TOPSIS criteria alike, so that for example 1:1 means
>> 1:1 for every method. weights of 1:5 reproduce the previous TOPSIS
>> criteria weights
>>
>> without specified weights, every method keeps its previous behavior: the
>> load imbalance and the bruteforce method use equal weights (1:1); the
>> TOPSIS methods use weights of 1:5 for their criteria
>>
>> NOTE: weights are not validated. if both are 0, there is simply no load
>> imbalance and the TOPSIS methods fail, so callers must not pass such
>> weights
>>
>> NOTE: to decide whether a migration improves the balance enough, the HA
>> manager compares the load imbalance reported for the migrations ranked
>> by TOPSIS with the current load imbalance. without specified weights,
>> the TOPSIS ranking would report an imbalance weighted 1:5, while the
>> load imbalance is unweighted. so, without specified weights, recompute
>> the reported imbalance of the ranked migrations with equal weights, as
>> before. while this requires reiterating all migrations candidates, it is
>> required to keep older versions of the HA manager working as before by
>> preventing them from running into this undesirable load imbalance
>> mismatch
>>
>> add test cases checking that:
>> - without options or with unspecified options, the load imbalance and
>> the node scores stay the same as before
>> - equal weights keep the unweighted load imbalance, regardless of their
>> magnitude
>> - weights of 1:5 keep the previous node scores, but change the load
>> imbalance, as it's also being weighted 1:5
>> - weighting only the CPU load or only the memory usage results in a
>> different load imbalance
>> - the TOPSIS ranking reports the same load imbalance as the bruteforce
>> method, both without specified options and with weights of 1:5, as the
>> HA manager compares it with the current load imbalance
>> - without specified options, the TOPSIS ranking is the same as with
>> weights of 1:5
>> - the dynamic scheduler accepts options as well
>>
>> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
>> ---
>>
>> Notes:
>> changes since v1:
>> * reword commit
>> * pass the weights once as optional scheduler options to the constructor
>> instead of adding weight parameters to every method, as suggested in
>> the review of v1, so the signatures of the existing methods stay the
>> same and older HA manager versions can still call them
>> * options are a single optional hash with only optional fields, a
>> missing weight defaults to 1
>> * without options, every method keeps its previous behavior, including
>> the TOPSIS ranking with weights of 1:5
>> * without options, recompute the imbalance reported with the TOPSIS
>> ranking with equal weights, so that it stays comparable to the node
>> imbalance for older HA manager version
>> * weights also apply to score_nodes_to_start_service()
>> * drop the adjustment of the existing test, as calling the methods
>> without weights keeps working
>> * add test cases for the scheduler options
>>
>> requires the proxmox-resource-scheduling patches of this series
>>
>> .../src/bindings/resource_scheduling/mod.rs | 1 +
>> .../bindings/resource_scheduling/options.rs | 59 ++++++++
>> .../resource_scheduling/pve_dynamic.rs | 71 +++++++--
>> .../resource_scheduling/pve_static.rs | 64 ++++++--
>> pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++
>> 5 files changed, 304 insertions(+), 32 deletions(-)
>> create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs
[snip]
>> diff --git a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
>> index 8e066e0..62d6426 100644
>> --- a/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
>> +++ b/pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs
>> @@ -12,11 +12,16 @@ pub mod pve_rs_resource_scheduling_dynamic {
>> use serde::{Deserialize, Serialize};
>>
>> use perlmod::Value;
>> - use proxmox_resource_scheduling::node::NodeStats;
>> + use proxmox_resource_scheduling::node::{NodeStats, NodeStatsWeights};
>> use proxmox_resource_scheduling::resource::ResourceStats;
>> - use proxmox_resource_scheduling::scheduler::ScoredMigration;
>> - use proxmox_resource_scheduling::usage::Usage;
>> + use proxmox_resource_scheduling::scheduler::{
>> + LEGACY_TOPSIS_WEIGHTS, Scheduler as CoreScheduler, ScoredMigration,
>> + };
>> + use proxmox_resource_scheduling::usage::{Usage, UsageAggregator};
>>
>> + use crate::bindings::resource_scheduling::options::{
>> + SchedulerOptions, with_unweighted_imbalance,
>> + };
>> use crate::bindings::resource_scheduling::resource::{
>> CompactMigrationCandidate, PveResource, decompose_compact_migration_candidates,
>> };
>> @@ -29,6 +34,21 @@ pub mod pve_rs_resource_scheduling_dynamic {
>> /// A scheduler instance contains the cluster usage.
>> pub struct Scheduler {
>> inner: Mutex<Usage>,
>> + weights: Option<NodeStatsWeights>,
>> + }
>> +
>> + impl Scheduler {
>> + /// Returns a scheduler for the current `usage`, aggregated by `F`, with the weights of
>> + /// this instance or `default_weights` if this instance has none.
>> + fn to_weighted_scheduler<F: UsageAggregator>(
>> + &self,
>> + usage: &Usage,
>> + default_weights: NodeStatsWeights,
>> + ) -> CoreScheduler {
>> + usage
>> + .to_scheduler::<F>()
>> + .with_weights(self.weights.unwrap_or(default_weights))
>> + }
>
> Hm, not a big one, but this could also just live in
> proxmox_resource_scheduling itself, where Usage implements a
>
> pub fn to_weighted_scheduler<F: UsageAggregator>(&self, weights: NodeStatsWeights) -> Scheduler {
> let node_usages = F::aggregate(self);
>
> Scheduler::from_nodes(node_usages).with_weights(weights)
> }
>
> and then the unwrap_or() could be made explicitly where it's called to
> it's clear that it's a default fallback in there.
>
technically, yes, but I'd rather keep it in proxmox-perl-rs
> No hard feelings though.
just to give my 2 cents on why I'd rather keep it there
the `weights` are only optional in there, because the bindings can be
constructed without scheduler options, so that older HA managers keep
working. in contrast, proxmox-resource-scheduling's scheduler always has
weights, so the `unwrap_or` call would stay in the bindings either way,
and on the proxmox-resource-scheduling side
usage.to_scheduler::<F>().with_weights(weights)
already does the same as the proposed method
also, the fallback is already explicit where the helper is called, as
each caller passes its default weights
[snip]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights
2026-11-19 12:40 ` Daniel Kral
@ 2026-11-19 13:52 ` Dominik Rusovac
0 siblings, 0 replies; 33+ messages in thread
From: Dominik Rusovac @ 2026-11-19 13:52 UTC (permalink / raw)
To: Daniel Kral, pve-devel
On Thu Nov 19, 2026 at 1:40 PM CET, Daniel Kral wrote:
> On Thu Nov 12, 2026 at 10:44 AM CET, Dominik Rusovac wrote:
>> CRS did not rebalance a cluster that was clearly imbalanced in CPU load
>> or memory usage alone, because the combined node load looked balanced
>> enough [0]
>>
>> add weights to the scheduler, so that callers can prioritize balancing
>> one resource over the other
>>
>> load imbalance computations and the bruteforce method use the weighted
>> node load
>>
>> for the TOPSIS methods, the weights replace the hardcoded criteria
>> weights from commit 2b325aa5 ("add pve_static module"). those favor
>> memory usage 5 times over CPU load. Scaling them with the weights
>> instead would make the weights relative to that built-in emphasis, so
>> 1:1 would effectively mean 1:5, which is undesirable. only the factor of
>> 2 between the criteria for the highest and the average usage of each
>> resource is kept, so custom weights of 1:5 reproduce the previous
>> criteria weights of 1:5 (average) and 2:10 (highest)
>>
>> the same weights are used for the TOPSIS criteria and the node load, so
>> the imbalance reported for the migrations ranked by TOPSIS is weighted
>> like the criteria that ranked them
>>
>> the weights default to 1:1. This changes the default ranking of the
>> TOPSIS methods (expected result of one affected test was adjusted), so
>> callers have to pass hardcoded legacy topsis weights to keep their
>> previous rankings. with these, the node load is weighted 1:5 as well and
>> the previous rankings are preserved, which is also being tested
>>
>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>>
>> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
>> ---
>>
>> Notes:
>> new with changes regarding patch [0] in v1:
>> * add weights as field to Scheduler struct
>> * weigh the TOPSIS criteria directly with the weights, keeping the factor
>> of 2 between the highest and the average usage criteria of each resource
>> - weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10
>> - add LEGACY_TOPSIS_WEIGHTS for callers keeping the previous rankings
>> * fall back to previous TOPSIS ranking of migration candidates
>> * no longer expose score and keep ScoredMigration as is
>> * adjust one scheduler test's expected result
>> * test that legacy topsis weights preserve previous rankings
>>
>> [0] https://lore.proxmox.com/all/20261021085932.1196302-4-d.rusovac@proxmox.com/
>>
>> proxmox-resource-scheduling/src/pve_static.rs | 4 +-
>> proxmox-resource-scheduling/src/scheduler.rs | 105 ++++++++++++++----
>> .../tests/scheduler.rs | 64 ++++++++++-
>> 3 files changed, 149 insertions(+), 24 deletions(-)
[snip]
>> - const N_CRITERIA;
>> - static PVE_HA_TOPSIS_CRITERIA;
>> +/// Factor to compute the weight of the criterion for the highest usage of a resource relative to
>> +/// the criterion for the average usage of that resource. This keeps the ratio of the criteria
>> +/// weights introduced in commit 2b325aa5.
>
> nit: include the commit title here, i.e.
>
> 2b325aa5 ("add pve_static module")
>
ack
>> +///
>> +/// For more details see [`topsis_criteria`].
>> +const TOPSIS_HIGHEST_USAGE_FACTOR: f64 = 2.0;
>> +
>> +/// Weights that reproduce the TOPSIS criteria weights introduced in commit 2b325aa5 with
>
> here as well
>
ack
>> +/// [`topsis_criteria`].
>> +pub const LEGACY_TOPSIS_WEIGHTS: NodeStatsWeights = NodeStatsWeights {
>> + cpu_weight: 1.0,
>> + mem_weight: 5.0,
>> +};
>> +
>> +/// Returns the TOPSIS criteria for a [`PveTopsisAlternative`], weighted by `weights`.
>> +///
>> +/// The criteria for the average usage of each resource get the respective weight in `weights`,
>> +/// the criteria for the highest usage get [`TOPSIS_HIGHEST_USAGE_FACTOR`] times that weight.
>> +fn topsis_criteria(weights: &NodeStatsWeights) -> Result<topsis::Criteria<N_CRITERIA>, Error> {
>
> nit: topsis_criteria_from_weights() or make this a method of Scheduler,
> which is its only user
>
I had `topsis_criteria_from_weights` at some point, can go back to it
[snip]
>> fn new_empty_cluster_scheduler() -> Scheduler {
>> @@ -317,7 +317,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>
> is the scheduler here intentionally not augmented with
>
> .with_weights(LEGACY_TOPSIS_WEIGHTS) ?
>
yes, the existing tests are meant to check the default (equal) weights,
while test_legacy_topsis_weights_keep_unweighted_rankings covers the
legacy weights
good catch tho, using the weights for the heterogeneous one is
inconsistent with that, will drop it there and adapt its expected
rankings in v3
>>
>> assert_eq!(
>> rank_nodes_to_start_resource(&scheduler, combined_resource_stats)?,
>> - vec!["node2", "node3", "node1"]
>> + vec!["node1", "node3", "node2"]
>> );
>>
>> Ok(())
>> @@ -325,7 +325,7 @@ fn test_score_homogeneous_nodes_to_start_resource() -> Result<(), Error> {
>>
>> #[test]
>> fn test_score_heterogeneous_nodes_to_start_resource() -> Result<(), Error> {
>> - let scheduler = new_heterogeneous_cluster_scheduler();
>> + let scheduler = new_heterogeneous_cluster_scheduler().with_weights(LEGACY_TOPSIS_WEIGHTS);
>>
will drop it here ^
[snip]
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2026-11-19 13:52 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-12 9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-12 9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:47 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28 ` Daniel Kral
2026-11-19 12:50 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41 ` Daniel Kral
2026-11-19 12:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40 ` Daniel Kral
2026-11-19 13:52 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45 ` Daniel Kral
2026-11-19 13:28 ` Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12 9:44 ` [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Dominik Rusovac
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox