From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2547E1FF0B6 for ; Thu, 12 Nov 2026 10:47:37 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A2FF821788; Thu, 12 Nov 2026 10:46:25 +0100 (CET) From: Dominik Rusovac To: pve-devel@lists.proxmox.com Subject: [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Date: Thu, 12 Nov 2026 10:44:36 +0100 Message-ID: <20261112094437.1951271-19-d.rusovac@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261112094437.1951271-1-d.rusovac@proxmox.com> References: <20261112094437.1951271-1-d.rusovac@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794476686656 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.030 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RXYGCACAU6COEDKLQZWQDVMENFYTOLH4 X-Message-ID-Hash: RXYGCACAU6COEDKLQZWQDVMENFYTOLH4 X-MailFrom: d.rusovac@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: pass the ha-cpu-weight and ha-mem-weight CRS options to the static and dynamic scheduler, so that users can prioritize balancing the CPU load or the memory usage of nodes. weights apply to the load imbalance, both automatic rebalancing methods and the selection of a node to start or recover a resource on without configured weights, the TOPSIS method keeps favoring memory usage 5 times over CPU load. if TOPSIS is the automatic rebalancing method, pass weights of 1:5 in that case, so that the node imbalance is weighted the same way and reflects what the TOPSIS method balances. this changes the behavior for existing setups using the TOPSIS method: after upgrading, their load imbalance is dominated by memory usage. setups using the bruteforce method keep the unweighted load imbalance log changes to the weights as they are passed to the scheduler, so that the implicit memory weight of 5 shows up in the log when configuring or removing weights, or when switching the automatic rebalancing method if both weights are set to 0, no resource contributes to the node load, so there is nothing to balance. skip automatic rebalancing and log a warning once in that case. the weights are not passed to the scheduler then, so selecting a node to start or recover a resource on keeps working with the default weights instead of failing to score allow changing the weights in the simulator at runtime, and add a test case with both weights set to 0, which neither rebalances nor fails to start the resources. Update the expected log of the existing TOPSIS test case whose node imbalance changes with the weights of 1:5: the reported imbalance values differ, and the last rebalancing migration is not done anymore passing weights requires a version of libpve-rs-perl that supports scheduler options. older versions reject the options, so the scheduler fails to initialize and falls back to the 'basic' mode. this happens whenever weights are configured, and also without configured weights if TOPSIS is the rebalancing method, as weights of 1:5 are passed then Signed-off-by: Dominik Rusovac --- Notes: changes since v1: * reword commit * name options ha-cpu-weight and ha-mem-weight, as they also apply to placing resources on start and recovery * weights are passed once on scheduler construction instead of to each method, see the proxmox-perl-rs patch * weights are kept outside of auto_rebalance * skip rebalancing and warn once if both weights are 0 * log the weights as passed to the scheduler, including the implicit memory weight of 5 for TOPSIS * add zero-weights test case * adjust existing topsis test using weights requires a libpve-rs-perl version with scheduler options support src/PVE/HA/Manager.pm | 46 ++++++++++++++++++- src/PVE/HA/Sim/Hardware.pm | 2 + src/PVE/HA/Usage/Dynamic.pm | 6 ++- src/PVE/HA/Usage/Static.pm | 6 ++- .../log.expect | 13 +----- .../README | 6 +++ .../cmdlist | 3 ++ .../datacenter.cfg | 8 ++++ .../dynamic_service_stats | 6 +++ .../hardware_status | 5 ++ .../log.expect | 41 +++++++++++++++++ .../manager_status | 1 + .../service_config | 6 +++ .../static_service_stats | 6 +++ 14 files changed, 138 insertions(+), 17 deletions(-) create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 7de8781..3db5402 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -174,6 +174,39 @@ sub update_crs_settings { 0.1, ); + my ($cpu_weight, $mem_weight) = $crs_cfg->@{qw(ha-cpu-weight ha-mem-weight)}; + my $weights_configured = defined($cpu_weight) || defined($mem_weight); + # the legacy TOPSIS criteria favor memory usage 5 times over CPU load by default, weigh the node + # load the same way, so that the imbalance reflects what the TOPSIS method balances + my $use_legacy_topsis_weights = !$weights_configured && $auto_rebalance->{method} eq 'topsis'; + + $self->update_crs_option('ha-cpu-weight', \$crs->{cpu_weight}, $cpu_weight, 1); + $self->update_crs_option( + 'ha-mem-weight', + \$crs->{mem_weight}, + $mem_weight, + $use_legacy_topsis_weights ? 5 : 1, + ); + + my $scheduler_options; + my $all_weights_zero = 0; + if (!$crs->{cpu_weight} && !$crs->{mem_weight}) { + # no resource would contribute to the node load, so there is nothing to balance; keep + # the default weights for selecting nodes to start or recover resources on + $haenv->log( + 'warning', + "all CRS weights are set to 0, ignoring them and skipping automatic rebalancing", + ) if !$crs->{all_weights_zero}; + $all_weights_zero = 1; + } elsif ($weights_configured || $use_legacy_topsis_weights) { + $scheduler_options = { + 'cpu-weight' => $crs->{cpu_weight}, + 'mem-weight' => $crs->{mem_weight}, + }; + } + $crs->{scheduler_options} = $scheduler_options; + $crs->{all_weights_zero} = $all_weights_zero; + return; } @@ -276,6 +309,7 @@ sub load_balance { return if !$auto_rebalance_opts->{enable}; return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic'; + return if $crs->{all_weights_zero}; return if $self->any_resource_motion_queued_or_running(); my ($threshold, $method, $hold_duration, $margin) = @@ -489,7 +523,11 @@ sub recompute_online_node_usage { if ($have_static_scheduling) { $online_node_usage = eval { $service_stats = $haenv->get_static_service_stats(); - my $scheduler = PVE::HA::Usage::Static->new($haenv, $service_stats); + my $scheduler = PVE::HA::Usage::Static->new( + $haenv, + $service_stats, + $self->{crs}->{scheduler_options}, + ); $scheduler->add_node($_) for $online_nodes->@*; return $scheduler; }; @@ -504,7 +542,11 @@ sub recompute_online_node_usage { if ($have_dynamic_scheduling) { $online_node_usage = eval { $service_stats = $haenv->get_dynamic_service_stats(); - my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats); + my $scheduler = PVE::HA::Usage::Dynamic->new( + $haenv, + $service_stats, + $self->{crs}->{scheduler_options}, + ); $scheduler->add_node($_) for $online_nodes->@*; return $scheduler; }; diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm index f3d8301..957548a 100644 --- a/src/PVE/HA/Sim/Hardware.pm +++ b/src/PVE/HA/Sim/Hardware.pm @@ -712,6 +712,8 @@ my $datacenter_settings = { 'ha-auto-rebalance-method' => qr/^(?:bruteforce|topsis)$/, 'ha-auto-rebalance-hold-duration' => qr/^\d+$/, 'ha-auto-rebalance-margin' => qr/^\d+(?:\.\d+)?$/, + 'ha-cpu-weight' => qr/^\d+$/, + 'ha-mem-weight' => qr/^\d+$/, }, }; diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm index 76d0fea..0da6115 100644 --- a/src/PVE/HA/Usage/Dynamic.pm +++ b/src/PVE/HA/Usage/Dynamic.pm @@ -9,12 +9,14 @@ use PVE::RS::ResourceScheduling::Dynamic; use base qw(PVE::HA::Usage); sub new { - my ($class, $haenv, $service_stats) = @_; + my ($class, $haenv, $service_stats, $scheduler_options) = @_; my $node_stats = eval { $haenv->get_dynamic_node_stats() }; die "did not get dynamic node usage information - $@" if $@; - my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() }; + # only pass options if there are any, so that older versions of the bindings still work + my @args = defined($scheduler_options) ? ($scheduler_options) : (); + my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new(@args) }; die "unable to initialize dynamic scheduling - $@" if $@; return bless { diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm index e67d5f5..9b91ba6 100644 --- a/src/PVE/HA/Usage/Static.pm +++ b/src/PVE/HA/Usage/Static.pm @@ -9,12 +9,14 @@ use PVE::RS::ResourceScheduling::Static; use base qw(PVE::HA::Usage); sub new { - my ($class, $haenv, $service_stats) = @_; + my ($class, $haenv, $service_stats, $scheduler_options) = @_; my $node_stats = eval { $haenv->get_static_node_stats() }; die "did not get static node usage information - $@" if $@; - my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(); }; + # only pass options if there are any, so that older versions of the bindings still work + my @args = defined($scheduler_options) ? ($scheduler_options) : (); + my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(@args) }; die "unable to initialize static scheduling - $@" if $@; return bless { diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect index 8779848..0847a04 100644 --- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect +++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect @@ -55,7 +55,7 @@ info 25 node3/lrm: service status vm:107 started info 120 cmdlist: execute service vm:105 set-dynamic-stats cpu 7.8 mem 7912 info 120 cmdlist: execute service vm:106 set-dynamic-stats cpu 5.7 mem 8192 info 120 cmdlist: execute service vm:107 set-dynamic-stats cpu 6.0 mem 8011 -info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 60.2% to 29.8%) +info 160 node1/crm: auto rebalance - migrate vm:105 to node2 (expected change for imbalance from 48.2% to 24.0%) info 160 node1/crm: got crm command: migrate vm:105 node2 info 160 node1/crm: migrate service 'vm:105' to node 'node2' info 160 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node3, target = node2) @@ -70,7 +70,7 @@ info 220 cmdlist: execute service vm:104 set-dynamic-stats cpu 6.7 mem 8 info 220 cmdlist: execute service vm:105 set-dynamic-stats cpu 1.8 mem 1201 info 220 cmdlist: execute service vm:106 set-dynamic-stats cpu 2.1 mem 1211 info 220 cmdlist: execute service vm:107 set-dynamic-stats cpu 0.9 mem 1191 -info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 57.3% to 30.1%) +info 260 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 58.4% to 27.4%) info 260 node1/crm: got crm command: migrate vm:103 node3 info 260 node1/crm: migrate service 'vm:103' to node 'node3' info 260 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node2, target = node3) @@ -79,13 +79,4 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3' info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3) info 285 node3/lrm: starting service vm:103 info 285 node3/lrm: service status vm:103 started -info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%) -info 340 node1/crm: got crm command: migrate vm:105 node1 -info 340 node1/crm: migrate service 'vm:105' to node 'node1' -info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1) -info 343 node2/lrm: service vm:105 - start migrate to node 'node1' -info 343 node2/lrm: service vm:105 - end migrate to node 'node1' -info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1) -info 361 node1/lrm: starting service vm:105 -info 361 node1/lrm: service status vm:105 started info 820 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README new file mode 100644 index 0000000..8108448 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/README @@ -0,0 +1,6 @@ +Test that the auto rebalance system with dynamic usage information does not +rebalance if all CRS weights are set to 0, as no resource contributes to the +node load then. + +The same setup without weights causes two rebalancing migrations, see +test-crs-dynamic-auto-rebalance2. A warning is logged once instead. diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist new file mode 100644 index 0000000..13f90cd --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/cmdlist @@ -0,0 +1,3 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg new file mode 100644 index 0000000..c395bc8 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/datacenter.cfg @@ -0,0 +1,8 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1, + "ha-cpu-weight": 0, + "ha-mem-weight": 0 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats new file mode 100644 index 0000000..f01fd76 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/dynamic_service_stats @@ -0,0 +1,6 @@ +{ + "vm:101": { "cpu": 1.0, "mem": 4294967296 }, + "vm:102": { "cpu": 1.0, "mem": 4294967296 }, + "vm:103": { "cpu": 1.0, "mem": 4294967296 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status new file mode 100644 index 0000000..ce8cf0e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect new file mode 100644 index 0000000..792a3a2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/log.expect @@ -0,0 +1,41 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: using ha-cpu-weight '0' +info 20 node1/crm: using ha-mem-weight '0' +warn 20 node1/crm: all CRS weights are set to 0, ignoring them and skipping automatic rebalancing +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node1' +info 20 node1/crm: adding new service 'vm:104' on node 'node1' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 21 node1/lrm: starting service vm:103 +info 21 node1/lrm: service status vm:103 started +info 21 node1/lrm: starting service vm:104 +info 21 node1/lrm: service status vm:104 started +info 22 node2/crm: status change wait_for_quorum => slave +info 24 node3/crm: status change wait_for_quorum => slave +info 620 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config new file mode 100644 index 0000000..b5960cb --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/service_config @@ -0,0 +1,6 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node1", "state": "started" }, + "vm:104": { "node": "node1", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats new file mode 100644 index 0000000..6cf8c10 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-weights/static_service_stats @@ -0,0 +1,6 @@ +{ + "vm:101": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 } +} -- 2.47.3