From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id DA18A1FF0B6 for ; Thu, 12 Nov 2026 10:47:19 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 63DB221746; Thu, 12 Nov 2026 10:46:24 +0100 (CET) From: Dominik Rusovac To: pve-devel@lists.proxmox.com Subject: [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Date: Thu, 12 Nov 2026 10:44:18 +0100 Message-ID: <20261112094437.1951271-1-d.rusovac@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794476685019 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.032 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NBMJ4YBL6ROGKWJ7KCY5F7C5D63AD6AK X-Message-ID-Hash: NBMJ4YBL6ROGKWJ7KCY5F7C5D63AD6AK X-MailFrom: d.rusovac@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Let users express a preference for prioritizing CPU load balancing over memory usage balancing, or vice versa, by means of weights. Tested basic functionality in virtualized 3-node cluster. Added property tests, unit tests, and test cases documenting the effect of the weights on automatic rebalancing. Will follow this up with a separate series updating CRS docs accordingly, once this one is considered. # TL;DR This series enables weighting CPU load against memory usage when computing node load. The node load becomes the weighted arithmetic mean of both (see Details). Only the weight ratio matters, and equal weights reproduce the old plain mean. This addresses cases [0] where the load balancer didn't act because a blended load imbalance looked fine even though the cluster was rather imbalanced in CPU load or memory usage alone. The weights are configured with the new `ha-cpu-weight` and `ha-mem-weight` CRS options. They apply to the static and dynamic CRS modes, that is, to the load imbalance, both automatic rebalancing methods, and the selection of a node to start or recover a resource on. ## Bruteforce - a higher weight on a resource steers migrations toward balancing that resource; this is a tendency, not a guarantee - a migration that leaves the variance of both resources higher can still look better balanced, because CPU and memory deviations can cancel out (negative covariance); weights don't disable this - load imbalance values computed with different weights, and the thresholds they are compared against, aren't directly comparable ## TOPSIS - the weights replace the fixed criteria weights; weights of 1:5 reproduce the previous criteria weights - the weight of the highest usage criterion of a resource stays twice the weight of its average usage criterion, which keeps the ratio introduced in commit 2b325aa5 ("add pve_static module") ## Defaults and compatibility - without configured weights, the bruteforce method and the load imbalance keep using equal weights, and TOPSIS keeps its previous ranking (1:5) - with TOPSIS as automatic rebalancing method and no configured weights, the HA manager weighs the load imbalance 1:5 as well, so that it reflects what TOPSIS balances; for these setups, the imbalance is dominated by memory usage after upgrading - HA manager logs the weights it passes to the scheduler, so the implicit 1:5 shows up in the log once weights are configured or the method changes - older HA managers don't pass weights and get the same results as before from the new bindings (see Compatibility) # Details A weight is any non-negative integer. Weights are blended into the computation of node loads. The load of a node is currently the arithmetic mean of the node's CPU load `cpu(node)` and memory usage `mem(node)`: (F1) (cpu(node) + mem(node)) / 2 This series replaces F1 with the weighted arithmetic mean [1]: (F2) (cpu(node) * cpu_w + mem(node) * mem_w) / (cpu_w + mem_w) `cpu_w` controls how much `cpu(node)` contributes to F2, analogously, `mem_w` controls how much `mem(node)` contributes to F2; the higher the weight, the more that resource dominates the result of F2. NOTE: - a weight of 0 means no contribution to F2; - only the weights' ratio matters, not their absolute magnitude; and - F2 reduces exactly to the plain arithmetic mean F1 whenever `cpu_w` and `mem_w` are equal (not just when both equal 1). Let `X` be the population of loads according to F2 with weights `cpu_w` and `mem_w` of all nodes in a cluster consisting of `n` nodes. Since weights `cpu_w` and `mem_w` affect the result of F2, they also shape how sensitively the load imbalance computation: (F3) coefficient_of_variation(X) / sqrt(n - 1) reacts to CPU load changes relative to memory usage changes. Accordingly, this series enables users to control how much a given change in CPU load or memory usage moves the imbalance metric by means of weights. This is to address: "In our case CRS has not moved any VM around because the imbalance percentage is low (7%). Eventhough the cluster is very imbalance [sic] CPU wise, or memory wise alone." [0] Weights of course also factor into the load balancer's decision, the 'how' depends on which method is used. ## Bruteforce Method The higher the weight of a computational resource, the more the search will tend to prioritize balancing out that resource, e.g., a much greater 'cpu_w' than 'mem_w' will steer migrations toward evening out CPU load, even at the cost of leaving memory usage less balanced. This is a general tendency, not a guarantee. The following deep(er) dive into the load imbalance formula gives more insights. Recall F3. Note that (F4) coefficient_of_variation(X) := standard_deviation(X) / mean(X) (F5) standard_deviation(X) := sqrt(variance(X)) Hence, `variance(X)` is the part of F3 that captures how dispersed the node loads are. Define `C := [cpu(x) | x in cluster]` and `M := [mem(x) | x in cluster]`. In fact, one can derive that `variance(X)` can be rewritten as: (F6) (cpu_t + mem_t + cov_t) / (cpu_w + mem_w)^2 where cpu_t := cpu_w^2 * variance(C) mem_t := mem_w^2 * variance(M) cov_t := 2 * cpu_w * mem_w * covariance(M,C) F6 dissects how the weights shape the variance part of the load imbalance. `cpu_t` and `mem_t` relate to the respective resource's dispersion weighted by the square of its weight (note that doubling a weight roughly quadruples that term's pull on the total). Note that `variance(C)` as part of `cpu_t` and `variance(M)` as part of `mem_t` are each an average of squared deviations [3], so they can never be negative, and neither can the squared weights; hence `cpu_t` and `mem_t` can only ever add to `variance(X)`, never subtract from it. They are also where each weight acts on its own resource's dispersion, and this is the mechanism behind letting users express a preference for balancing one computational resource over the other. Because `cpu_t` scales with `cpu_w^2` and `mem_t` scales with `mem_w^2`, raising `cpu_w` relative to `mem_w` makes `variance(C)`'s own dispersion count for a disproportionately larger share of the numerator of F6, which steers the bruteforce method toward candidates that keep CPU load tight, even at the cost of leaving memory usage more spread out. This is the sense in which a higher `cpu_w` means "prioritize balancing CPU load": not that memory usage is ignored, but that `mem_t`'s pull on the numerator shrinks relative to `cpu_t`'s as the weight ratio skews toward CPU load. `cov_t` is the only one of the three terms that can be negative, which makes it the only lever capable of pulling `variance(X)` below what the two resources' own dispersions would otherwise force it to be. A rebalancing migration can exploit this by arranging `cpu(·)` and `mem(·)` to deviate from their means in opposite directions across nodes. A node that ends up above-average on `mem(·)` tends to end up below-average on `cpu(·)`, and vice versa. Averaged into F2 per node, those two errors partially cancel out, so `variance(X)` (and hence the reported load imbalance) can come out looking well-balanced even when neither `variance(C)` nor `variance(M)` individually is. One caveat that follows directly: weighting a resource more heavily increases its pull on `variance(X)`, but it doesn't disable `cov_t`. A candidate with a sufficiently favorable (negative) covariance can still beat one that's honestly better on the heavily-weighted resource's own dispersion. In short, weighting changes how much a resource's balance matters to the outcome, but it does not make bruteforce immune to the cancellation mechanism described above. Besides `variance(X)`, the weights also enter `mean(X)`, the denominator of F4. By definition, `mean(X)` is the plain (unweighted) arithmetic mean over the nodes of the weighted node loads, that is, the sum of the loads according to F2 divided by the number of nodes `n`. The weights act across the two resources within each node, not across nodes. In fact, `mean(X)` can be rewritten as the weighted arithmetic mean of the two resources' mean loads: (F7) (cpu_w * mean(C) + mem_w * mean(M)) / (cpu_w + mem_w) The result of F7 lies between `mean(C)` and `mean(M)` and moves toward the mean of the more heavily weighted resource. Scaling both weights by the same factor leaves it unchanged. Note that a single rebalancing migration only redistributes computational resources between nodes, it never changes the cluster-wide total CPU load or memory usage. Hence, for nodes of equal capacity (homogeneous cluster), `mean(C)`, `mean(M)` and thus `mean(X)` are the same constant for every migration candidate at fixed weights: the mean scales the load imbalance (F3) but does not influence which candidate has the lowest one, which is why the weights' steering effect on the ranking is dissected by F6. Two caveats follow: - Different weights: Both `variance(X)` (F6) and `mean(X)` (F7) change with the choice of weights, so the same state of the cluster yields different load imbalance values for different weights, in particular if `mean(C)` and `mean(M)` or `variance(C)` and `variance(M)` differ considerably. Values computed with different weights, and thresholds they are compared against, are therefore not directly comparable. - Nodes of different capacity (heterogeneous cluster): The same amount of a resource corresponds to a different fraction of the capacity of different nodes, so migrating it to different targets can change the sum of node loads. Then `mean(C)`, `mean(M)` and `mean(X)` can differ between migration candidates, and the mean influences which candidate has the lowest load imbalance in addition to `variance(X)`. F6 remains valid but in this case only explains the variance part of the ranking. ## TOPSIS Method Weights have always been part of the TOPSIS [2] method, weighing the criteria of the decision process. Previously, all of these weights were fixed and favored memory usage 5 times over CPU load. With this series, the configured weights replace them directly: `cpu_w` is the weight of the average CPU criterion and `mem_w` the weight of the average memory criterion. The weights of highest CPU and highest memory are not independently configurable; each is twice the weight of its average counterpart. This makes it possible for users to prioritize balancing a specific computational resource while also preserving the ratio between a resource's two criteria, as introduced in commit 2b325aa5 ("add pve_static module"). Configured weights of 1:5 reproduce the previous criteria weights 1, 2, 5, 10. Scaling the fixed weights with the configured ones instead would have made the configured weights relative to the built-in emphasis on memory, so that 1:1 would effectively mean 1:5. The same weights are used for the TOPSIS criteria and the node load, so the imbalance reported for the migrations ranked by TOPSIS is weighted like the criteria that ranked them, and it is compared to a load imbalance weighted the same way. Unlike for the bruteforce method, TOPSIS' per-resource criteria never combine CPU load and memory usage into a single quantity, so there is no known cancellation-style lever to be aware of here. # Compatibility As suggested in the v1 review, the weights are part of the scheduler struct in proxmox-resource-scheduling and are passed once to the constructor of the pve-rs bindings as optional scheduler options, instead of to every method. So, signatures of the existing methods stay the same. The crate defaults to equal weights, which changes the default TOPSIS ranking. Callers keep the previous ranking with hardcoded legacy topsis weights. The bindings keep the previous behavior of every method if no weights are passed: the load imbalance and the bruteforce method use equal weights, TOPSIS uses 1:5. The HA manager compares the imbalance reported for the migrations ranked by TOPSIS with the current load imbalance. To keep these comparable for older HA managers, which don't pass weights, the bindings recompute the imbalance of the ranked migrations with equal weights in that case, as before. Hence, the new bindings shouldn't break older HA manager versions. Newer HA managers need a versioned dependency on the new bindings, as older bindings reject the options, which makes the HA manager fall back to the 'basic' CRS mode. If both weights are set to 0, no resource contributes to the node load. The HA manager then warns once, skips automatic rebalancing, uses the weights for selecting nodes to start or recover resources on, and computes the load imbalance using 1:1, as before. # Summary of main changes since v1 - weights are part of the scheduler instead of being passed to each of its methods - the weights apply to selecting a node to start or recover a resource on as well, so the options were renamed to `ha-cpu-weight` and `ha-mem-weight` - weights are passed as optional scheduler options on construction, so older HA managers keep working - factor out the CRS option logging into a helper, first for the scheduler mode, keeping its log messages, then for the other options; values differing from the defaults are logged on startup - log the implicit 1:5 weights for TOPSIS - skip automatic rebalancing and warn once if both weights are 0 - in the ui the weights are enabled for the static and dynamic modes regardless of automatic rebalancing, and are advanced options together with the hold duration and the minimum imbalance improvement [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993 [1] https://en.wikipedia.org/wiki/Weighted_arithmetic_mean [2] https://en.wikipedia.org/wiki/TOPSIS [3] https://en.wikipedia.org/wiki/Variance proxmox: Dominik Rusovac (8): resource-scheduling: fix typos resource-scheduling: scheduler: refine score_* docs resource-scheduling: scheduler: abstract post-migration stats logic fix #7993: resource-scheduling: node: add weighted node load resource-scheduling: scheduler: make imbalance with candidate public fix #7993: resource-scheduling: scheduler: add weights workspace,resource-scheduling: add proptest as dev-dependency resource-scheduling: scheduler: add property tests for weights Cargo.toml | 1 + proxmox-resource-scheduling/Cargo.toml | 3 + proxmox-resource-scheduling/src/node.rs | 114 ++++ proxmox-resource-scheduling/src/pve_static.rs | 4 +- proxmox-resource-scheduling/src/scheduler.rs | 485 ++++++++++++++++-- proxmox-resource-scheduling/src/topsis.rs | 2 +- .../tests/scheduler.rs | 64 ++- 7 files changed, 616 insertions(+), 57 deletions(-) proxmox-perl-rs: Dominik Rusovac (1): fix #7993: pve-rs: resource-scheduling: add scheduler options .../src/bindings/resource_scheduling/mod.rs | 1 + .../bindings/resource_scheduling/options.rs | 59 ++++++++ .../resource_scheduling/pve_dynamic.rs | 71 +++++++-- .../resource_scheduling/pve_static.rs | 64 ++++++-- pve-rs/test/resource_scheduling.pl | 141 ++++++++++++++++++ 5 files changed, 304 insertions(+), 32 deletions(-) create mode 100644 pve-rs/src/bindings/resource_scheduling/options.rs pve-cluster: Dominik Rusovac (1): fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) pve-manager: Dominik Rusovac (2): ui: form: crs options: make fine-tuning options advanced fix #7993: ui: form: crs options: add weights www/manager6/form/CRSOptions.js | 47 +++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) pve-ha-manager: Dominik Rusovac (7): manager: remove obsolete argument manager: rename update_crs_scheduler_mod to update_crs_settings manager: factor out crs option update helper manager: log changes to crs settings sim: hardware: allow changing datacenter settings at runtime fix #7993: manager,usage: use weights for static and dynamic mode test: add automatic rebalancing test cases with weights src/PVE/HA/Manager.pm | 140 ++++++++++++++---- src/PVE/HA/Sim/Hardware.pm | 62 ++++++++ src/PVE/HA/Usage/Dynamic.pm | 6 +- src/PVE/HA/Usage/Static.pm | 6 +- .../log.expect | 2 + .../log.expect | 2 + .../log.expect | 15 +- .../log.expect | 3 + .../README | 4 + .../cmdlist | 4 + .../datacenter.cfg | 6 + .../dynamic_service_stats | 8 + .../hardware_status | 5 + .../log.expect | 61 ++++++++ .../manager_status | 1 + .../service_config | 8 + .../static_service_stats | 8 + .../README | 4 + .../cmdlist | 6 + .../datacenter.cfg | 6 + .../dynamic_service_stats | 8 + .../hardware_status | 5 + .../log.expect | 65 ++++++++ .../manager_status | 1 + .../service_config | 8 + .../static_service_stats | 8 + .../README | 5 + .../cmdlist | 4 + .../datacenter.cfg | 6 + .../dynamic_service_stats | 8 + .../hardware_status | 5 + .../log.expect | 61 ++++++++ .../manager_status | 1 + .../service_config | 8 + .../static_service_stats | 8 + .../README | 5 + .../cmdlist | 4 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 8 + .../hardware_status | 5 + .../log.expect | 62 ++++++++ .../manager_status | 1 + .../service_config | 8 + .../static_service_stats | 8 + .../README | 5 + .../cmdlist | 4 + .../datacenter.cfg | 6 + .../dynamic_service_stats | 8 + .../hardware_status | 5 + .../log.expect | 61 ++++++++ .../manager_status | 1 + .../service_config | 8 + .../static_service_stats | 8 + .../README | 6 + .../cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 41 +++++ .../manager_status | 1 + .../service_config | 6 + .../static_service_stats | 6 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 2 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 2 + .../log.expect | 1 + .../test-crs-dynamic-rebalance1/log.expect | 1 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 1 + .../log.expect | 4 + .../test-crs-static-rebalance1/log.expect | 1 + .../test-crs-static-rebalance2/log.expect | 1 + 79 files changed, 833 insertions(+), 43 deletions(-) create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats Summary over all repositories: 93 files changed, 1828 insertions(+), 132 deletions(-) -- Generated by murpp 0.11.0