From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id F1FF81FF0B3 for ; Wed, 14 Oct 2026 16:29:17 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id A7C6321472; Wed, 14 Oct 2026 16:29:15 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Wed, 14 Oct 2026 16:29:08 +0200 Message-Id: Subject: Re: [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure From: "Daniel Kral" To: "Michael Ryom" , X-Mailer: aerc 0.22.0-10-g6373ac9d2179-dirty References: <20260920162220.574802-1-Michael@RyomHerold.dk> <20260920162220.574802-8-Michael@RyomHerold.dk> In-Reply-To: <20260920162220.574802-8-Michael@RyomHerold.dk> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1791988148448 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.789 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GXSSGVOKA5C3HJQL4O4XMFD52IZSEDSX X-Message-ID-Hash: GXSSGVOKA5C3HJQL4O4XMFD52IZSEDSX X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote: > The load balancer acted on load asymmetry alone. But asymmetric node > loads are not, by themselves, a performance problem: on an almost idle > cluster, the imbalance metric (a coefficient of variation, std/mean) > becomes hypersensitive, and observed live, a barely-qualifying > migration (60.4% -> 54.0% expected imbalance) was issued at 2-7% node > load, with nothing to gain for any workload. The purpose of the load > balancer is better performance for the guests, not symmetric load > figures. > > Gate the balancer on the pressure stall information (PSI) that the > nodes already broadcast in their RRD stats: only balance while some > node reports a cpu or memory some-pressure (avg10) of at least 10%, > i.e. work on that node actually loses a meaningful share of its time > waiting on resources and could run better with more headroom. Unlike a > load threshold, this criterion is independent of the cluster's size and > load level, and it directly expresses the question "would anything > perform better elsewhere?" - akin to the contention-driven (rather than > load-driven) approach of other schedulers like VMware DRS. > > Deliberately only the node pressure is considered, although the guests' > own pressure stats are collected as well: a guest saturating its own > vCPUs or its own memory reports pressure too, but gets neither more > cores nor more memory by being migrated, while host-caused starvation > (contention with other guests or host processes) shows up in the node > pressure - and only that kind can be improved by a migration. > > The kernel's avg10 pressure values are already exponentially averaged > over ten seconds, so no additional smoothing is applied. IO pressure is > deliberately not considered for now, since with shared storage it > usually cannot be improved by a migration. If no pressure information > is available at all (e.g. nodes still broadcasting an older stats > schema), balancing proceeds as before. We cannot answer the question "Would anything perform better elsewhere?" with the PSI information as said above. A high PSI value can only say "This doesn't work well right now, do something." That is, in the context of the CRS Load Balancer a rising PSI value can only tell us to move something away, but not what should be moved away. The idea is that a relatively similar load across cluster nodes does reduce the likelihood of contention/stalling/resource pressure to happen on all nodes as long as the cluster isn't heavily overcommitted itself. We can predict the movement of resource usage of one guest from one node to another and how it affects the whole cluster relatively well. (Of course, improvements always welcome to make predictions better and the model more aligned with reality wrt. KSM, frequency scaling, performance/efficiency cores, topologies, cache hierarchies, etc. without too much computational effort) In comparison, the PSI information is local information that has no guarantee to be reflected on other nodes at all. We cannot say whether or in what quantity contention will change because of a migration. Especially what do we do in case all nodes tell us that they have high contention? That will likely mean that the load is pretty high on all nodes. Then we don't want to do trigger any rebalancing migrations, because we can't do any better than suggesting that the cluster size/hardware resources should be increased or less-important guests should be shutdown. Another point is that PSI values are delayed, so we would only react as there are already problems surfacing. And in case of memory pressure, that would be already very late as those are much more serious than CPU pressure scenarios. > > The minimum pressure is a constant for now; like the other balancer > constants, it could be exposed as a ha-auto-rebalance-* option later. This should definitely be configurable as it changes the behavior for existing setups that depend on this. This should at least be configurable with a default value of 0. > > Signed-off-by: Michael Ryom > --- > src/PVE/HA/Env/PVE2.pm | 25 +++++++++++++ > src/PVE/HA/Manager.pm | 16 +++++++++ > src/PVE/HA/Sim/Hardware.pm | 16 +++++++-- > src/PVE/HA/Usage.pm | 9 +++++ > src/PVE/HA/Usage/Dynamic.pm | 23 ++++++++++++ > .../test-crs-dynamic-auto-rebalance9/README | 15 ++++++++ > .../test-crs-dynamic-auto-rebalance9/cmdlist | 4 +++ > .../datacenter.cfg | 6 ++++ > .../dynamic_service_stats | 4 +++ > .../hardware_status | 4 +++ > .../log.expect | 36 +++++++++++++++++++ > .../manager_status | 1 + > .../service_config | 4 +++ > .../static_service_stats | 4 +++ > 14 files changed, 165 insertions(+), 2 deletions(-) > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.= cfg > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_ser= vice_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_st= atus > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_sta= tus > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_con= fig > create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_serv= ice_stats > > diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm > index 8c2b03d..7d7fceb 100644 > --- a/src/PVE/HA/Env/PVE2.pm > +++ b/src/PVE/HA/Env/PVE2.pm > @@ -47,6 +47,8 @@ use constant { > RRD_VM_INDEX_MAXMEM =3D> 7, > RRD_VM_INDEX_MEM =3D> 8, > RRD_VM_INDEX_MEMHOST =3D> 15, > + RRD_VM_INDEX_PRESSURE_CPU_SOME =3D> 16, > + RRD_VM_INDEX_PRESSURE_MEM_SOME =3D> 20, nit: redundant PSI information that isn't used > }; > =20 > # rrd entry indices for PVE nodes > @@ -56,8 +58,21 @@ use constant { > RRD_NODE_INDEX_CPU =3D> 5, > RRD_NODE_INDEX_MAXMEM =3D> 7, > RRD_NODE_INDEX_MEM =3D> 8, > + RRD_NODE_INDEX_PRESSURE_CPU_SOME =3D> 17, > + RRD_NODE_INDEX_PRESSURE_MEM_SOME =3D> 20, > }; > =20 > +# returns the numeric value of the given RRD column, or undef if the col= umn > +# is not present (e.g. nodes broadcasting an older schema) > +my sub rrd_column_or_undef { > + my ($rrdentry, $index) =3D @_; > + > + my $value =3D $rrdentry->[$index]; > + return undef if !defined($value) || $value eq '' || $value eq 'U'; > + > + return $value + 0.0; > +} > + > my $HOSTNAME_RE =3D qr/(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{,61}?[a-zA-Z0-9])?= )/; > =20 > sub new { > @@ -615,6 +630,11 @@ sub get_dynamic_service_stats { > cpu =3D> (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $m= axcpu, > maxmem =3D> int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0), > mem =3D> $mem, > + # PSI avg10 in percent; undef if the guest broadcasts an old= er > + # schema without pressure information > + pressurecpusome =3D> rrd_column_or_undef($rrdentry, RRD_VM_I= NDEX_PRESSURE_CPU_SOME), > + pressurememorysome =3D> > + rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_MEM= _SOME), nit: redundant PSI information that isn't used > }; > } > =20 > @@ -660,6 +680,11 @@ sub get_dynamic_node_stats { > cpu =3D> (($rrdentry->[RRD_NODE_INDEX_CPU] || 0.0) + 0.0) * = $maxcpu, > maxmem =3D> int($rrdentry->[RRD_NODE_INDEX_MAXMEM] || 0), > mem =3D> int($rrdentry->[RRD_NODE_INDEX_MEM] || 0), > + # PSI avg10 in percent; undef if the node broadcasts an olde= r > + # schema without pressure information > + pressurecpusome =3D> rrd_column_or_undef($rrdentry, RRD_NODE= _INDEX_PRESSURE_CPU_SOME), > + pressurememorysome =3D> > + rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_M= EM_SOME), > }; > } > =20 > diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm > index 6e1827f..c34da89 100644 > --- a/src/PVE/HA/Manager.pm > +++ b/src/PVE/HA/Manager.pm > @@ -155,6 +155,15 @@ my $auto_rebalance_failure_backoff_max =3D 3600; > # minimum time before a resource is considered for rebalancing again aft= er it > # was successfully moved by the load balancer > my $auto_rebalance_resource_cooldown =3D 600; > +# minimum pressure stall value (PSI avg10, in percent) among the nodes f= or > +# the load balancer to act at all: load asymmetry by itself is not a > +# performance problem, and the node pressure stall information directly > +# measures which share of its time work on the node spends waiting on > +# resources - i.e. whether anything could actually run better with more > +# headroom - independent of the cluster's size or load level; if no pres= sure > +# information is available at all (older stats schema), balancing procee= ds > +# as before I'd rather have the feature be gated through a switch or as described above a new option value with a default that wouldn't change existing setups than whether a node sends this information or not. > +my $auto_rebalance_min_pressure =3D 10.0; > # minimum absolute improvement of the node imbalance for a rebalance mot= ion; > # the relative margin alone provides almost no hysteresis when a single > # resource dominates the cluster load: with two nodes, a resource load v= and > @@ -368,6 +377,13 @@ sub load_balance { > return; > } > =20 > + # nothing to gain from balancing while no workload is short on resou= rces > + my $max_pressure =3D $online_node_usage->max_pressure(); > + if (defined($max_pressure) && $max_pressure < $auto_rebalance_min_pr= essure) { > + $self->{sustained_imbalance_round} =3D 0; > + return; > + } > + > my $imbalance =3D $online_node_usage->calculate_node_imbalance(); > =20 > # do not load balance unless imbalance threshold has been exceeded > diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm > index aa1a886..8766a6f 100644 > --- a/src/PVE/HA/Sim/Hardware.pm > +++ b/src/PVE/HA/Sim/Hardware.pm > @@ -791,7 +791,8 @@ sub get_cfs_state { > # crm enable-node-maintenance > # crm disable-node-maintenance > # pve-manager-version set # note: this is NOT the *= ha*-manager version > -# node set-dynamic-stats [cpu ] [mem ] # load outs= ide HA services > +# node set-dynamic-stats [cpu ] [mem ] [pressurecpu= some ] [pressurememorysome ] > +# (load and pressure outside of the HA-managed services) > # reboot > # shutdown > # restart-lrm > @@ -888,7 +889,12 @@ sub sim_hardware_cmd { > die "sim_hardware_cmd: missing target stat for '$action' com= mand" > if !@params; > =20 > - my $conversions =3D { cpu =3D> sub { 0.0 + $_[0] }, mem =3D>= sub { $_[0] * 1024**2 } }; > + my $conversions =3D { > + cpu =3D> sub { 0.0 + $_[0] }, > + mem =3D> sub { $_[0] * 1024**2 }, > + pressurecpusome =3D> sub { 0.0 + $_[0] }, > + pressurememorysome =3D> sub { 0.0 + $_[0] }, > + }; > =20 > for my ($target, $val) (@params) { > die "sim_hardware_cmd: missing value for '$action $targe= t' command" > @@ -1312,6 +1318,12 @@ sub get_dynamic_node_stats { > $stats->{$node}->{cpu} =3D $cstatus->{$node}->{cpu} // 0.0; > $stats->{$node}->{maxmem} =3D $stats->{$node}->{maxmem} // $defa= ult_node_maxmem; > $stats->{$node}->{mem} =3D $cstatus->{$node}->{mem} // 0; > + # PSI avg10 in percent; keep undef (pressure information not > + # available) distinct from an explicit zero (no pressure) > + for my $pressure (qw(pressurecpusome pressurememorysome)) { > + $stats->{$node}->{$pressure} =3D $cstatus->{$node}->{$pressu= re} > + if defined($cstatus->{$node}->{$pressure}); > + } > } > =20 > my $service_conf =3D $self->read_service_config(); > diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm > index 659ab30..e0890fb 100644 > --- a/src/PVE/HA/Usage.pm > +++ b/src/PVE/HA/Usage.pm > @@ -66,6 +66,15 @@ sub calculate_node_imbalance { > die "implement in subclass"; > } > =20 > +# Returns the highest pressure stall value (PSI avg10, in percent) among= the > +# nodes, or undef if the usage implementation has no notion of pressure > +# (e.g. basic or static) or no node reported pressure information. > +sub max_pressure { > + my ($self) =3D @_; > + > + return undef; > +} > + > sub score_best_balancing_migrations { > my ($self, $migration_candidates, $limit) =3D @_; > =20 > diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm > index fc88ddf..b2ed79a 100644 > --- a/src/PVE/HA/Usage/Dynamic.pm > +++ b/src/PVE/HA/Usage/Dynamic.pm > @@ -184,6 +184,29 @@ sub calculate_node_imbalance { > return $node_imbalance // 0.0; > } > =20 > +# NOTE only the *node* pressure is considered: a guest saturating its ow= n > +# vCPUs or its own memory reports pressure too, but gets neither more co= res > +# nor more memory by being migrated, while starvation caused by the host > +# (contention with other guests or host processes) shows up in the node'= s > +# pressure - and only that kind can be improved by a migration This information should go right next to some code why this exact decision was made, maybe a bit more concise > +sub max_pressure { > + my ($self) =3D @_; > + > + my $max_pressure; > + > + for my $node ($self->list_nodes()) { > + my $stats =3D $self->{'node-stats'}->{$node} or next; > + > + for my $pressure ($stats->@{qw(pressurecpusome pressurememorysom= e)}) { > + next if !defined($pressure); > + $max_pressure =3D $pressure > + if !defined($max_pressure) || $pressure > $max_pressure; > + } CPU and memory PSI values do have very different meanings. The SOME CPU PSI value does say that some processes are delayed on running on the CPU because of CPU scheduling contention to serve all of them fairly. The SOME memory PSI value does say that some processes are delayed because there is memory reclaiming going on. That is, there is more memory demand than can be currently provided. That is much more severe, because memory is a truly finite resource (that can end in OOM kills), while CPU pressure will just slow down workloads. So if we would go for this, we should measure them by different standards. > + } > + > + return $max_pressure; > +} > + > sub score_best_balancing_migrations { > my ($self, $migration_candidates, $limit) =3D @_; > =20