From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B13231FF0AF for ; Thu, 22 Oct 2026 11:58:20 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 6A4B0215F8; Thu, 22 Oct 2026 11:58:17 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 22 Oct 2026 11:58:10 +0200 Message-Id: From: "Daniel Kral" To: "Dominik Rusovac" , Subject: Re: [PATCH pve-ha-manager 10/15] fix #7993: manager,usage: implement weighted auto rebalancing X-Mailer: aerc 0.22.0-10-g6373ac9d2179-dirty References: <20261021085932.1196302-1-d.rusovac@proxmox.com> <20261021085932.1196302-11-d.rusovac@proxmox.com> In-Reply-To: <20261021085932.1196302-11-d.rusovac@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792663090452 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.736 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WD2UDBEFD5DLZ4ZFIM3LVRMCIHEDNF7J X-Message-ID-Hash: WD2UDBEFD5DLZ4ZFIM3LVRMCIHEDNF7J X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote: > Include ha-auto-rebalance-cpu-weight and ha-auto-rebalance-mem-weight in > load imbalance calculation and decision making. As noted, these options control the decision making for intial placement (rebalance-on-start) as well, so should probably drop the 'auto-' part. > > The weights only play into decisions in the context of automatic > rebalancing. The bruteforce method optimizes the weighted load > imbalance. For the TOPSIS method ha-auto-rebalance-cpu-weight and > ha-auto-rebalance-mem-weight are assigned to the respective average cpu > and average memory criteria. The weights of the highest cpu and highest > memory criteria of TOPSIS are not independently configurable. Nodes to > start service are scored with the same fixed (TOPSIS) weights as before. Oh, so the weights don't influence score_nodes_to_start_resource()... Hm, but maybe we could do that as well? What do you think? > > The weights also shape how sensitively the load balancer reacts to CPU > load changes relative to memory usage changes. This is to address: "In > our case CRS has not moved any VM around because the imbalance > percentage is low (7%). Eventhough the cluster is very imbalance [sic] > CPU wise, or memory wise alone." [0] > > Skip load balancing with warning if both weights are 0. > > [0] https://bugzilla.proxmox.com/show_bug.cgi?id=3D7993 > > Signed-off-by: Dominik Rusovac > --- > src/PVE/HA/Manager.pm | 47 ++++++++++++++++--- > src/PVE/HA/Usage.pm | 21 +++++---- > src/PVE/HA/Usage/Basic.pm | 14 ++++-- > src/PVE/HA/Usage/Dynamic.pm | 27 +++++++---- > src/PVE/HA/Usage/Static.pm | 27 +++++++---- > .../log.expect | 13 +---- > 6 files changed, 102 insertions(+), 47 deletions(-) > > diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm > index 7a5a685..c6818e2 100644 > --- a/src/PVE/HA/Manager.pm > +++ b/src/PVE/HA/Manager.pm > @@ -119,6 +119,12 @@ sub update_crs_scheduler_mode { > $self->{crs}->{auto_rebalance}->{margin} =3D > ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0; > =20 > + my $default_weight =3D 1; > + $self->{crs}->{auto_rebalance}->{cpu_weight} =3D $crs_cfg->{'ha-auto= -rebalance-cpu-weight'} // $default_weight; > + # by default TOPSIS weights mem higher than cpu > + $default_weight =3D 5 if ($self->{crs}->{auto_rebalance}->{method} e= q 'topsis'); > + $self->{crs}->{auto_rebalance}->{mem_weight} =3D $crs_cfg->{'ha-auto= -rebalance-mem-weight'} // $default_weight; nit: might be slightly nicer to read with a my ($cpu_weight, $mem_weight) =3D $crs_cfg->@{qw(ha-rebalance-cpu-weight ha= -rebalance-mem-weight)}; if ($method eq 'topsis') { $self->{crs}->{cpu_weight} =3D $cpu_weight // 1; $self->{crs}->{mem_weight} =3D $mem_weight // 5; } else { $self->{crs}->{cpu_weight} =3D $cpu_weight // 1; $self->{crs}->{mem_weight} =3D $mem_weight // 1; } but no hard feelings. Put the fields outside of auto_rebalance as these are also used for rebalance-on-start, but no hard feelings for that either. Another idea, that would be a bit more refactoring beforehand, would be to actually make 'method' a intialization parameter for $online_node_usage on the pve-ha-manager side. Then $cpu_weight and $mem_weight could also become initialization parameters, and we wouldn't need to pass them to calculate_node_imbalance() and the score_best_balancing_migrations() methods anymore. Then we would also need only one score_best_balancing_migrations(), which under its hood would then either call the bruteforce or topsis pve-rs version. This works because $online_node_usage is reconstructed in every HA Manager round anyway. > + > my $old_mode =3D $self->{crs}->{scheduler}; > my $new_mode =3D $crs_cfg->{ha} || 'basic'; > =20 > @@ -236,8 +242,20 @@ sub load_balance { > return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dyn= amic'; > return if $self->any_resource_motion_queued_or_running(); > =20 > - my ($threshold, $method, $hold_duration, $margin) =3D > - $auto_rebalance_opts->@{qw(threshold method hold_duration margin= )}; > + my ( > + $threshold, $method, $hold_duration, $margin, $cpu_weight, $mem_= weight, > + ) =3D > + $auto_rebalance_opts->@{qw(threshold method hold_duration margin= cpu_weight mem_weight)}; > + > + if ($cpu_weight + $mem_weight eq 0) { > + $haenv->log( > + 'warning', > + "auto-rebalance weights are both set to 0, skipping load bal= ancing -" nit: all resource weights set to 0, skipping load balancing - > + . " set at least one to a positive value to avoid consta= nt load" > + . " imbalance of 0", > + ); > + return; > + } Though maybe this should come after the threshold + duration check, so it is only skips an unnecessary call to score_best_balancing_migrations() and so it doesn't get logged in each call to load_balance()? No hard feelings though > =20 > # with threshold and margin both at 0% any non-zero imbalance trigge= rs a non-improving > # migration, churning the cluster forever - skip and warn instead of= silently looping. > @@ -250,7 +268,7 @@ sub load_balance { > return; > } > =20 > - my $imbalance =3D $online_node_usage->calculate_node_imbalance(); > + my $imbalance =3D $online_node_usage->calculate_node_imbalance($cpu_= weight, $mem_weight); > =20 > # do not load balance unless imbalance threshold has been exceeded > # consecutively for $hold_duration calls to load_balance(); > @@ -268,15 +286,24 @@ sub load_balance { > =20 > my $result; > if ($method eq 'bruteforce') { > - $result =3D $online_node_usage->select_best_balancing_migration(= $candidates); > + $result =3D $online_node_usage->select_best_balancing_migration( > + $candidates, > + $cpu_weight, > + $mem_weight, > + ); > } elsif ($method eq 'topsis') { > - $result =3D $online_node_usage->select_best_balancing_migration_= topsis($candidates); > + $result =3D $online_node_usage->select_best_balancing_migration_= topsis( > + $candidates, > + $cpu_weight, > + $mem_weight, > + ); > } > =20 > # happens if $candidates is empty or $method isn't handled above > return if !$result; > =20 > - my ($migration, $target_imbalance) =3D $result->@{qw(migration imbal= ance)}; > + my ($migration, $target_imbalance, $score) =3D $result->@{qw(migrati= on imbalance score)}; > + $target_imbalance =3D $score if $method eq 'bruteforce'; > =20 > my $relative_change =3D ($imbalance - $target_imbalance) / $imbalanc= e; > return if $relative_change < $margin; > @@ -311,7 +338,13 @@ sub flush_master_status { > $ms->{node_status} =3D $ns->{status}; > $ms->{service_status} =3D $ss; > $ms->{timestamp} =3D $haenv->get_time(); > - $ms->{imbalance} =3D $self->{online_node_usage}->calculate_node_imba= lance(); > + > + my ($cpu_weight, $mem_weight) =3D ( > + $self->{crs}->{auto_rebalance}->{cpu_weight}, > + $self->{crs}->{auto_rebalance}->{mem_weight}, > + ); nit: can be my ($cpu_weight, $mem_weight) =3D $self->{crs}->@{qw(cpu_weight mem_weight)= }; > + $ms->{imbalance} =3D > + $self->{online_node_usage}->calculate_node_imbalance($cpu_weight= , $mem_weight); > =20 > $haenv->write_manager_status($ms); > } > diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm > index 659ab30..fbd8800 100644 > --- a/src/PVE/HA/Usage.pm > +++ b/src/PVE/HA/Usage.pm > @@ -61,42 +61,47 @@ sub remove_service_usage { > } > =20 > sub calculate_node_imbalance { > - my ($self) =3D @_; > + my ($self, $cpu_weight, $mem_weight) =3D @_; > =20 > die "implement in subclass"; > } > =20 > sub score_best_balancing_migrations { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = =3D @_; > =20 > die "implement in subclass"; > } > =20 > sub select_best_balancing_migration { > - my ($self, $migration_candidates) =3D @_; > + my ($self, $migration_candidates, $cpu_weight, $mem_weight) =3D @_; > =20 > - my $migrations =3D $self->score_best_balancing_migrations($migration= _candidates, 1); > + my $migrations =3D > + $self->score_best_balancing_migrations($migration_candidates, 1,= $cpu_weight, $mem_weight); > =20 > return $migrations->[0]; > } > =20 > sub score_best_balancing_migrations_topsis { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ( > + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ) =3D @_; > =20 > die "implement in subclass"; > } > =20 > sub select_best_balancing_migration_topsis { > - my ($self, $migration_candidates) =3D @_; > + my ($self, $migration_candidates, $cpu_weight, $mem_weight) =3D @_; > =20 > - my $migrations =3D $self->score_best_balancing_migrations_topsis($mi= gration_candidates, 1); > + my $migrations =3D $self->score_best_balancing_migrations_topsis( > + $migration_candidates, 1, $cpu_weight, $mem_weight, > + ); > =20 > return $migrations->[0]; > } > =20 > # Returns a hash with $nodename =3D> $score pairs. A lower $score is bet= ter. > sub score_nodes_to_start_service { > - my ($self, $sid) =3D @_; > + my ($self, $sid, $cpu_weight, $mem_weight) =3D @_; > =20 > die "implement in subclass"; > } > diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm > index 4dce9e1..57eecd3 100644 > --- a/src/PVE/HA/Usage/Basic.pm > +++ b/src/PVE/HA/Usage/Basic.pm > @@ -67,19 +67,27 @@ sub remove_service_usage { > } > =20 > sub calculate_node_imbalance { > - my ($self) =3D @_; > + my ($self, $cpu_weight, $mem_weight) =3D @_; > =20 > return 0.0; > } > =20 > sub score_best_balancing_migrations { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = =3D @_; > =20 > return []; > } > =20 > sub score_best_balancing_migrations_topsis { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ( > + $self, > + $migration_candidates, > + $limit, > + $cpu_weight, > + $max_cpu_weight, > + $mem_weight, > + $max_mem_weight, $max_{cpu,mem}_weight seem to be from a former design of this series :-) > + ) =3D @_; > =20 > return []; > } > diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm > index 76d0fea..bd03e89 100644 > --- a/src/PVE/HA/Usage/Dynamic.pm > +++ b/src/PVE/HA/Usage/Dynamic.pm > @@ -105,20 +105,22 @@ sub remove_service_usage { > } > =20 > sub calculate_node_imbalance { > - my ($self) =3D @_; > + my ($self, $cpu_weight, $mem_weight) =3D @_; > =20 > - my $node_imbalance =3D eval { $self->{scheduler}->calculate_node_imb= alance() }; > + my $node_imbalance =3D > + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight,= $mem_weight) }; > $self->{haenv}->log('warning', "unable to calculate dynamic node imb= alance - $@") if $@; > =20 > return $node_imbalance // 0.0; > } > =20 > sub score_best_balancing_migrations { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = =3D @_; > =20 > my $migrations =3D eval { > - $self->{scheduler} > - ->score_best_balancing_migration_candidates($migration_candi= dates, $limit); > + $self->{scheduler}->score_best_balancing_migration_candidates( > + $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ); > }; > $self->{haenv}->log('warning', "unable to score best balancing migra= tion - $@") if $@; > =20 > @@ -126,11 +128,14 @@ sub score_best_balancing_migrations { > } > =20 > sub score_best_balancing_migrations_topsis { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ( > + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ) =3D @_; > =20 > my $migrations =3D eval { > - $self->{scheduler} > - ->score_best_balancing_migration_candidates_topsis($migratio= n_candidates, $limit); > + $self->{scheduler}->score_best_balancing_migration_candidates_to= psis( > + $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ); > }; > $self->{haenv}->log('warning', "unable to score best balancing migra= tion - $@") if $@; > =20 > @@ -140,9 +145,13 @@ sub score_best_balancing_migrations_topsis { > sub score_nodes_to_start_service { > my ($self, $sid) =3D @_; > =20 > + my ($cpu_weight, $mem_weight) =3D (1, 5); > + > my $score_list =3D eval { > my $service_usage =3D get_service_usage($self, $sid); > - $self->{scheduler}->score_nodes_to_start_resource($service_usage= ); > + $self->{scheduler}->score_nodes_to_start_resource( > + $service_usage, $cpu_weight, $mem_weight, > + ); > }; > $self->{haenv} > ->log('err', "unable to score nodes according to dynamic usage f= or service '$sid' - $@") > diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm > index e67d5f5..39308ff 100644 > --- a/src/PVE/HA/Usage/Static.pm > +++ b/src/PVE/HA/Usage/Static.pm > @@ -112,20 +112,22 @@ sub remove_service_usage { > } > =20 > sub calculate_node_imbalance { > - my ($self) =3D @_; > + my ($self, $cpu_weight, $mem_weight) =3D @_; > =20 > - my $node_imbalance =3D eval { $self->{scheduler}->calculate_node_imb= alance() }; > + my $node_imbalance =3D > + eval { $self->{scheduler}->calculate_node_imbalance($cpu_weight,= $mem_weight) }; > $self->{haenv}->log('warning', "unable to calculate static node imba= lance - $@") if $@; > =20 > return $node_imbalance // 0.0; > } > =20 > sub score_best_balancing_migrations { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ($self, $migration_candidates, $limit, $cpu_weight, $mem_weight) = =3D @_; > =20 > my $migrations =3D eval { > - $self->{scheduler} > - ->score_best_balancing_migration_candidates($migration_candi= dates, $limit); > + $self->{scheduler}->score_best_balancing_migration_candidates( > + $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ); > }; > $self->{haenv}->log('warning', "unable to score best balancing migra= tion - $@") if $@; > =20 > @@ -133,11 +135,14 @@ sub score_best_balancing_migrations { > } > =20 > sub score_best_balancing_migrations_topsis { > - my ($self, $migration_candidates, $limit) =3D @_; > + my ( > + $self, $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ) =3D @_; > =20 > my $migrations =3D eval { > - $self->{scheduler} > - ->score_best_balancing_migration_candidates_topsis($migratio= n_candidates, $limit); > + $self->{scheduler}->score_best_balancing_migration_candidates_to= psis( > + $migration_candidates, $limit, $cpu_weight, $mem_weight, > + ); > }; > $self->{haenv}->log('warning', "unable to score best balancing migra= tion - $@") if $@; > =20 > @@ -147,9 +152,13 @@ sub score_best_balancing_migrations_topsis { > sub score_nodes_to_start_service { > my ($self, $sid) =3D @_; > =20 > + my ($cpu_weight, $mem_weight) =3D (1, 5); > + > my $score_list =3D eval { > my $service_usage =3D get_service_usage($self, $sid); > - $self->{scheduler}->score_nodes_to_start_service($service_usage)= ; > + $self->{scheduler}->score_nodes_to_start_service( > + $service_usage, $cpu_weight, $mem_weight, > + ); > }; > if (my $err =3D $@) { > $self->{haenv}->log( > diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect = b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect > index 2f19c72..95163d4 100644 > --- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect > +++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect > @@ -53,7 +53,7 @@ info 25 node3/lrm: service status vm:107 started > info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7= .8 mem 7912 > info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5= .7 mem 8192 > info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6= .0 mem 8011 > -info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expe= cted change for imbalance from 60.2% to 29.8%) > +info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expe= cted change for imbalance from 48.2% to 24.0%) > info 160 node1/crm: got crm command: migrate vm:105 node2 > info 160 node1/crm: migrate service 'vm:105' to node 'node2' > info 160 node1/crm: service 'vm:105': state changed from 'started'= to 'migrate' (node =3D node3, target =3D node2) > @@ -68,7 +68,7 @@ info 220 cmdlist: execute service vm:104 set-dy= namic-stats cpu 6.7 mem 8 > info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1= .8 mem 1201 > info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2= .1 mem 1211 > info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0= .9 mem 1191 > -info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expe= cted change for imbalance from 57.3% to 30.1%) > +info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expe= cted change for imbalance from 58.4% to 27.4%) > info 260 node1/crm: got crm command: migrate vm:103 node3 > info 260 node1/crm: migrate service 'vm:103' to node 'node3' > info 260 node1/crm: service 'vm:103': state changed from 'started'= to 'migrate' (node =3D node2, target =3D node3) > @@ -77,13 +77,4 @@ info 263 node2/lrm: service vm:103 - end migrate= to node 'node3' > info 280 node1/crm: service 'vm:103': state changed from 'migrate'= to 'started' (node =3D node3) > info 285 node3/lrm: starting service vm:103 > info 285 node3/lrm: service status vm:103 started > -info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expe= cted change for imbalance from 30.1% to 17.0%) > -info 340 node1/crm: got crm command: migrate vm:105 node1 > -info 340 node1/crm: migrate service 'vm:105' to node 'node1' > -info 340 node1/crm: service 'vm:105': state changed from 'started'= to 'migrate' (node =3D node2, target =3D node1) > -info 343 node2/lrm: service vm:105 - start migrate to node 'node1' > -info 343 node2/lrm: service vm:105 - end migrate to node 'node1' > -info 360 node1/crm: service 'vm:105': state changed from 'migrate'= to 'started' (node =3D node1) > -info 361 node1/lrm: starting service vm:105 > -info 361 node1/lrm: service status vm:105 started > info 820 hardware: exit simulation - done