From: "Daniel Kral" <d.kral@proxmox.com>
To: "Dominik Rusovac" <d.rusovac@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing
Date: Thu, 22 Oct 2026 11:58:10 +0200 [thread overview]
Message-ID: <DMBA12N76K0Z.31DGWX02RMAYD@proxmox.com> (raw)
In-Reply-To: <20261021085932.1196302-11-d.rusovac@proxmox.com>
On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote:
> Include ha-auto-rebalance-cpu-weight and ha-auto-rebalance-mem-weight in
> load imbalance calculation and decision making.
As noted, these options control the decision making for intial placement
(rebalance-on-start) as well, so should probably drop the 'auto-' part.
>
> The weights only play into decisions in the context of automatic
> rebalancing. The bruteforce method optimizes the weighted load
> imbalance. For the TOPSIS method ha-auto-rebalance-cpu-weight and
> ha-auto-rebalance-mem-weight are assigned to the respective average cpu
> and average memory criteria. The weights of the highest cpu and highest
> memory criteria of TOPSIS are not independently configurable. Nodes to
> start service are scored with the same fixed (TOPSIS) weights as before.
Oh, so the weights don't influence score_nodes_to_start_resource()...
Hm, but maybe we could do that as well? What do you think?
>
> The weights also shape how sensitively the load balancer reacts to CPU
> load changes relative to memory usage changes. This is to address: "In
> our case CRS has not moved any VM around because the imbalance
> percentage is low (7%). Eventhough the cluster is very imbalance [sic]
> CPU wise, or memory wise alone." [0]
>
> Skip load balancing with warning if both weights are 0.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7993
>
> Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
> ---
> src/PVE/HA/Manager.pm | 47 ++++++++++++++++---
> src/PVE/HA/Usage.pm | 21 +++++----
> src/PVE/HA/Usage/Basic.pm | 14 ++++--
> src/PVE/HA/Usage/Dynamic.pm | 27 +++++++----
> src/PVE/HA/Usage/Static.pm | 27 +++++++----
> .../log.expect | 13 +----
> 6 files changed, 102 insertions(+), 47 deletions(-)
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 7a5a685..c6818e2 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -119,6 +119,12 @@ sub update_crs_scheduler_mode {
> $self->{crs}->{auto_rebalance}->{margin} =
> ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
>
> + my $default_weight = 1;
> + $self->{crs}->{auto_rebalance}->{cpu_weight} = $crs_cfg->{'ha-auto-rebalance-cpu-weight'} // $default_weight;
> + # by default TOPSIS weights mem higher than cpu
> + $default_weight = 5 if ($self->{crs}->{auto_rebalance}->{method} eq 'topsis');
> + $self->{crs}->{auto_rebalance}->{mem_weight} = $crs_cfg->{'ha-auto-rebalance-mem-weight'} // $default_weight;
nit: might be slightly nicer to read with a
my ($cpu_weight, $mem_weight) = $crs_cfg->@{qw(ha-rebalance-cpu-weight ha-rebalance-mem-weight)};
if ($method eq 'topsis') {
$self->{crs}->{cpu_weight} = $cpu_weight // 1;
$self->{crs}->{mem_weight} = $mem_weight // 5;
} else {
$self->{crs}->{cpu_weight} = $cpu_weight // 1;
$self->{crs}->{mem_weight} = $mem_weight // 1;
}
but no hard feelings.
Put the fields outside of auto_rebalance as these are also used for
rebalance-on-start, but no hard feelings for that either.
Another idea, that would be a bit more refactoring beforehand, would be
to actually make 'method' a intialization parameter for
$online_node_usage on the pve-ha-manager side.
Then $cpu_weight and $mem_weight could also become initialization
parameters, and we wouldn't need to pass them to
calculate_node_imbalance() and the score_best_balancing_migrations()
methods anymore.
Then we would also need only one score_best_balancing_migrations(),
which under its hood would then either call the bruteforce or topsis
pve-rs version.
This works because $online_node_usage is reconstructed in every HA
Manager round anyway.
> +
> my $old_mode = $self->{crs}->{scheduler};
> my $new_mode = $crs_cfg->{ha} || 'basic';
>
> @@ -236,8 +242,20 @@ sub load_balance {
> return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
> return if $self->any_resource_motion_queued_or_running();
>
> - my ($threshold, $method, $hold_duration, $margin) =
> - $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
> + my (
> + $threshold, $method, $hold_duration, $margin, $cpu_weight, $mem_weight,
> + ) =
> + $auto_rebalance_opts->@{qw(threshold method hold_duration margin cpu_weight mem_weight)};
> +
> + if ($cpu_weight + $mem_weight eq 0) {
> + $haenv->log(
> + 'warning',
> + "auto-rebalance weights are both set to 0, skipping load balancing -"
nit: all resource weights set to 0, skipping load balancing -
> + . " set at least one to a positive value to avoid constant load"
> + . " imbalance of 0",
> + );
> + return;
> + }
Though maybe this should come after the threshold + duration check, so
it is only skips an unnecessary call to
score_best_balancing_migrations() and so it doesn't get logged in each
call to load_balance()?
No hard feelings though
>
> # with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
> # migration, churning the cluster forever - skip and warn instead of silently looping.
> @@ -250,7 +268,7 @@ sub load_balance {
> return;
> }
>
> - my $imbalance = $online_node_usage->calculate_node_imbalance();
> + my $imbalance = $online_node_usage->calculate_node_imbalance($cpu_weight, $mem_weight);
>
> # do not load balance unless imbalance threshold has been exceeded
> # consecutively for $hold_duration calls to load_balance();
> @@ -268,15 +286,24 @@ sub load_balance {
>
> my $result;
> if ($method eq 'bruteforce') {
> - $result = $online_node_usage->select_best_balancing_migration($candidates);
> + $result = $online_node_usage->select_best_balancing_migration(
> + $candidates,
> + $cpu_weight,
> + $mem_weight,
> + );
> } elsif ($method eq 'topsis') {
> - $result = $online_node_usage->select_best_balancing_migration_topsis($candidates);
> + $result = $online_node_usage->select_best_balancing_migration_topsis(
> + $candidates,
> + $cpu_weight,
> + $mem_weight,
> + );
> }
>
> # happens if $candidates is empty or $method isn't handled above
> return if !$result;
>
> - my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
> + my ($migration, $target_imbalance, $score) = $result->@{qw(migration imbalance score)};
> + $target_imbalance = $score if $method eq 'bruteforce';
>
> my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
> return if $relative_change < $margin;
> @@ -311,7 +338,13 @@ sub flush_master_status {
> $ms->{node_status} = $ns->{status};
> $ms->{service_status} = $ss;
> $ms->{timestamp} = $haenv->get_time();
> - $ms->{imbalance} = $self->{online_node_usage}->calculate_node_imbalance();
> +
> + my ($cpu_weight, $mem_weight) = (
> + $self->{crs}->{auto_rebalance}->{cpu_weight},
> + $self->{crs}->{auto_rebalance}->{mem_weight},
> + );
nit: can be
my ($cpu_weight, $mem_weight) = $self->{crs}->@{qw(cpu_weight mem_weight)};
> + $ms->{imbalance} =
> + $self->{online_node_usage}->calculate_node_imbalance($cpu_weight, $mem_weight);
>
> $haenv->write_manager_status($ms);
> }
> diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
> index 659ab30..fbd8800 100644
> --- a/src/PVE/HA/Usage.pm
> +++ b/src/PVE/HA/Usage.pm
> @@ -61,42 +61,47 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
>
> sub select_best_balancing_migration {
> - my ($self, $migration_candidates) = @_;
> + my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
>
> - my $migrations = $self->score_best_balancing_migrations($migration_candidates, 1);
> + my $migrations =
> + $self->score_best_balancing_migrations($migration_candidates, 1, $cpu_weight, $mem_weight);
>
> return $migrations->[0];
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> die "implement in subclass";
> }
>
> sub select_best_balancing_migration_topsis {
> - my ($self, $migration_candidates) = @_;
> + my ($self, $migration_candidates, $cpu_weight, $mem_weight) = @_;
>
> - my $migrations = $self->score_best_balancing_migrations_topsis($migration_candidates, 1);
> + my $migrations = $self->score_best_balancing_migrations_topsis(
> + $migration_candidates, 1, $cpu_weight, $mem_weight,
> + );
>
> return $migrations->[0];
> }
>
> # Returns a hash with $nodename => $score pairs. A lower $score is better.
> sub score_nodes_to_start_service {
> - my ($self, $sid) = @_;
> + my ($self, $sid, $cpu_weight, $mem_weight) = @_;
>
> die "implement in subclass";
> }
> diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
> index 4dce9e1..57eecd3 100644
> --- a/src/PVE/HA/Usage/Basic.pm
> +++ b/src/PVE/HA/Usage/Basic.pm
> @@ -67,19 +67,27 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> return 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> return [];
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self,
> + $migration_candidates,
> + $limit,
> + $cpu_weight,
> + $max_cpu_weight,
> + $mem_weight,
> + $max_mem_weight,
$max_{cpu,mem}_weight seem to be from a former design of this series :-)
> + ) = @_;
>
> return [];
> }
> diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
> index 76d0fea..bd03e89 100644
> --- a/src/PVE/HA/Usage/Dynamic.pm
> +++ b/src/PVE/HA/Usage/Dynamic.pm
> @@ -105,20 +105,22 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> - my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
> + my $node_imbalance =
> + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
> $self->{haenv}->log('warning', "unable to calculate dynamic node imbalance - $@") if $@;
>
> return $node_imbalance // 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -126,11 +128,14 @@ sub score_best_balancing_migrations {
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -140,9 +145,13 @@ sub score_best_balancing_migrations_topsis {
> sub score_nodes_to_start_service {
> my ($self, $sid) = @_;
>
> + my ($cpu_weight, $mem_weight) = (1, 5);
> +
> my $score_list = eval {
> my $service_usage = get_service_usage($self, $sid);
> - $self->{scheduler}->score_nodes_to_start_resource($service_usage);
> + $self->{scheduler}->score_nodes_to_start_resource(
> + $service_usage, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}
> ->log('err', "unable to score nodes according to dynamic usage for service '$sid' - $@")
> diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
> index e67d5f5..39308ff 100644
> --- a/src/PVE/HA/Usage/Static.pm
> +++ b/src/PVE/HA/Usage/Static.pm
> @@ -112,20 +112,22 @@ sub remove_service_usage {
> }
>
> sub calculate_node_imbalance {
> - my ($self) = @_;
> + my ($self, $cpu_weight, $mem_weight) = @_;
>
> - my $node_imbalance = eval { $self->{scheduler}->calculate_node_imbalance() };
> + my $node_imbalance =
> + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight, $mem_weight) };
> $self->{haenv}->log('warning', "unable to calculate static node imbalance - $@") if $@;
>
> return $node_imbalance // 0.0;
> }
>
> sub score_best_balancing_migrations {
> - my ($self, $migration_candidates, $limit) = @_;
> + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -133,11 +135,14 @@ sub score_best_balancing_migrations {
> }
>
> sub score_best_balancing_migrations_topsis {
> - my ($self, $migration_candidates, $limit) = @_;
> + my (
> + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + ) = @_;
>
> my $migrations = eval {
> - $self->{scheduler}
> - ->score_best_balancing_migration_candidates_topsis($migration_candidates, $limit);
> + $self->{scheduler}->score_best_balancing_migration_candidates_topsis(
> + $migration_candidates, $limit, $cpu_weight, $mem_weight,
> + );
> };
> $self->{haenv}->log('warning', "unable to score best balancing migration - $@") if $@;
>
> @@ -147,9 +152,13 @@ sub score_best_balancing_migrations_topsis {
> sub score_nodes_to_start_service {
> my ($self, $sid) = @_;
>
> + my ($cpu_weight, $mem_weight) = (1, 5);
> +
> my $score_list = eval {
> my $service_usage = get_service_usage($self, $sid);
> - $self->{scheduler}->score_nodes_to_start_service($service_usage);
> + $self->{scheduler}->score_nodes_to_start_service(
> + $service_usage, $cpu_weight, $mem_weight,
> + );
> };
> if (my $err = $@) {
> $self->{haenv}->log(
> diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> index 2f19c72..95163d4 100644
> --- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> +++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
> @@ -53,7 +53,7 @@ info 25 node3/lrm: service status vm:107 started
> info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912
> info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192
> info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011
> -info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%)
> +info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%)
> info 160 node1/crm: got crm command: migrate vm:105 node2
> info 160 node1/crm: migrate service 'vm:105' to node 'node2'
> info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2)
> @@ -68,7 +68,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8
> info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201
> info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211
> info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191
> -info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%)
> +info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%)
> info 260 node1/crm: got crm command: migrate vm:103 node3
> info 260 node1/crm: migrate service 'vm:103' to node 'node3'
> info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3)
> @@ -77,13 +77,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
> info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
> info 285 node3/lrm: starting service vm:103
> info 285 node3/lrm: service status vm:103 started
> -info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
> -info 340 node1/crm: got crm command: migrate vm:105 node1
> -info 340 node1/crm: migrate service 'vm:105' to node 'node1'
> -info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
> -info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
> -info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
> -info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
> -info 361 node1/lrm: starting service vm:105
> -info 361 node1/lrm: service status vm:105 started
> info 820 hardware: exit simulation - done
next prev parent reply other threads:[~2026-10-22 9:58 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-21 8:59 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} 00/15] fix #7993: implement weighted auto rebalancing Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 01/15] resource-scheduling: scheduler: fix typos Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 02/15] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-10-21 12:49 ` Daniel Kral
2026-10-21 8:59 ` [PATCH proxmox 03/15] fix #7993: resource-scheduling: lift to weighted load balancing Dominik Rusovac
2026-10-22 9:15 ` Daniel Kral
2026-10-22 11:34 ` Daniel Kral
2026-10-21 8:59 ` [PATCH proxmox 04/15] workspace,resource-scheduling: add proptest crate Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox 05/15] resource-scheduling: scheduler: proptest weights Dominik Rusovac
2026-10-21 8:59 ` [PATCH proxmox-perl-rs 06/15] fix #7993: pve-rs: resource-scheduling: add weights to pve bindings Dominik Rusovac
2026-10-22 9:26 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-manager 07/15] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-manager 08/15] ui: form: crs options: make options advanced Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 09/15] manager: remove obsolete argument Dominik Rusovac
2026-10-22 9:29 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing Dominik Rusovac
2026-10-22 9:58 ` Daniel Kral [this message]
2026-10-21 8:59 ` [PATCH pve-ha-manager 11/15] sim: hardware: enable datacenter config changes at runtime Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-ha-manager 12/15] manager: log changes to crs settings Dominik Rusovac
2026-10-22 12:05 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Dominik Rusovac
2026-10-22 12:20 ` Daniel Kral
2026-10-21 8:59 ` [PATCH pve-ha-manager 14/15] manager: rename update_crs_scheduler_mode Dominik Rusovac
2026-10-21 8:59 ` [PATCH pve-cluster 15/15] fix #7993: datacenter config: add ha-auto-rebalance-{cpu,mem}-weight Dominik Rusovac
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DMBA12N76K0Z.31DGWX02RMAYD@proxmox.com \
--to=d.kral@proxmox.com \
--cc=d.rusovac@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox