Re: [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure

["Daniel Kral" <d.kral@proxmox.com>]

On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote:
> The load balancer acted on load asymmetry alone. But asymmetric node
> loads are not, by themselves, a performance problem: on an almost idle
> cluster, the imbalance metric (a coefficient of variation, std/mean)
> becomes hypersensitive, and observed live, a barely-qualifying
> migration (60.4% -> 54.0% expected imbalance) was issued at 2-7% node
> load, with nothing to gain for any workload. The purpose of the load
> balancer is better performance for the guests, not symmetric load
> figures.
>
> Gate the balancer on the pressure stall information (PSI) that the
> nodes already broadcast in their RRD stats: only balance while some
> node reports a cpu or memory some-pressure (avg10) of at least 10%,
> i.e. work on that node actually loses a meaningful share of its time
> waiting on resources and could run better with more headroom. Unlike a
> load threshold, this criterion is independent of the cluster's size and
> load level, and it directly expresses the question "would anything
> perform better elsewhere?" - akin to the contention-driven (rather than
> load-driven) approach of other schedulers like VMware DRS.
>
> Deliberately only the node pressure is considered, although the guests'
> own pressure stats are collected as well: a guest saturating its own
> vCPUs or its own memory reports pressure too, but gets neither more
> cores nor more memory by being migrated, while host-caused starvation
> (contention with other guests or host processes) shows up in the node
> pressure - and only that kind can be improved by a migration.
>
> The kernel's avg10 pressure values are already exponentially averaged
> over ten seconds, so no additional smoothing is applied. IO pressure is
> deliberately not considered for now, since with shared storage it
> usually cannot be improved by a migration. If no pressure information
> is available at all (e.g. nodes still broadcasting an older stats
> schema), balancing proceeds as before.

We cannot answer the question "Would anything perform better elsewhere?"
with the PSI information as said above. A high PSI value can only say
"This doesn't work well right now, do something."

That is, in the context of the CRS Load Balancer a rising PSI value can
only tell us to move something away, but not what should be moved away.

The idea is that a relatively similar load across cluster nodes does
reduce the likelihood of contention/stalling/resource pressure to happen
on all nodes as long as the cluster isn't heavily overcommitted itself.

We can predict the movement of resource usage of one guest from one node
to another and how it affects the whole cluster relatively well.
(Of course, improvements always welcome to make predictions better and
the model more aligned with reality wrt. KSM, frequency scaling,
performance/efficiency cores, topologies, cache hierarchies, etc.
without too much computational effort)

In comparison, the PSI information is local information that has no
guarantee to be reflected on other nodes at all. We cannot say whether
or in what quantity contention will change because of a migration.


Especially what do we do in case all nodes tell us that they have high
contention? That will likely mean that the load is pretty high on all
nodes. Then we don't want to do trigger any rebalancing migrations,
because we can't do any better than suggesting that the cluster
size/hardware resources should be increased or less-important guests
should be shutdown.


Another point is that PSI values are delayed, so we would only react as
there are already problems surfacing. And in case of memory pressure,
that would be already very late as those are much more serious than CPU
pressure scenarios.

>
> The minimum pressure is a constant for now; like the other balancer
> constants, it could be exposed as a ha-auto-rebalance-* option later.

This should definitely be configurable as it changes the behavior for
existing setups that depend on this. This should at least be
configurable with a default value of 0.

>
> Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
> ---
>  src/PVE/HA/Env/PVE2.pm                        | 25 +++++++++++++
>  src/PVE/HA/Manager.pm                         | 16 +++++++++
>  src/PVE/HA/Sim/Hardware.pm                    | 16 +++++++--
>  src/PVE/HA/Usage.pm                           |  9 +++++
>  src/PVE/HA/Usage/Dynamic.pm                   | 23 ++++++++++++
>  .../test-crs-dynamic-auto-rebalance9/README   | 15 ++++++++
>  .../test-crs-dynamic-auto-rebalance9/cmdlist  |  4 +++
>  .../datacenter.cfg                            |  6 ++++
>  .../dynamic_service_stats                     |  4 +++
>  .../hardware_status                           |  4 +++
>  .../log.expect                                | 36 +++++++++++++++++++
>  .../manager_status                            |  1 +
>  .../service_config                            |  4 +++
>  .../static_service_stats                      |  4 +++
>  14 files changed, 165 insertions(+), 2 deletions(-)
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_status
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_status
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_config
>  create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
>
> diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
> index 8c2b03d..7d7fceb 100644
> --- a/src/PVE/HA/Env/PVE2.pm
> +++ b/src/PVE/HA/Env/PVE2.pm
> @@ -47,6 +47,8 @@ use constant {
>      RRD_VM_INDEX_MAXMEM => 7,
>      RRD_VM_INDEX_MEM => 8,
>      RRD_VM_INDEX_MEMHOST => 15,
> +    RRD_VM_INDEX_PRESSURE_CPU_SOME => 16,
> +    RRD_VM_INDEX_PRESSURE_MEM_SOME => 20,

nit: redundant PSI information that isn't used

>  };
>  
>  # rrd entry indices for PVE nodes
> @@ -56,8 +58,21 @@ use constant {
>      RRD_NODE_INDEX_CPU => 5,
>      RRD_NODE_INDEX_MAXMEM => 7,
>      RRD_NODE_INDEX_MEM => 8,
> +    RRD_NODE_INDEX_PRESSURE_CPU_SOME => 17,
> +    RRD_NODE_INDEX_PRESSURE_MEM_SOME => 20,
>  };
>  
> +# returns the numeric value of the given RRD column, or undef if the column
> +# is not present (e.g. nodes broadcasting an older schema)
> +my sub rrd_column_or_undef {
> +    my ($rrdentry, $index) = @_;
> +
> +    my $value = $rrdentry->[$index];
> +    return undef if !defined($value) || $value eq '' || $value eq 'U';
> +
> +    return $value + 0.0;
> +}
> +
>  my $HOSTNAME_RE = qr/(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{,61}?[a-zA-Z0-9])?)/;
>  
>  sub new {
> @@ -615,6 +630,11 @@ sub get_dynamic_service_stats {
>              cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
>              maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0),
>              mem => $mem,
> +            # PSI avg10 in percent; undef if the guest broadcasts an older
> +            # schema without pressure information
> +            pressurecpusome => rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_CPU_SOME),
> +            pressurememorysome =>
> +                rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_MEM_SOME),

nit: redundant PSI information that isn't used

>          };
>      }
>  
> @@ -660,6 +680,11 @@ sub get_dynamic_node_stats {
>              cpu => (($rrdentry->[RRD_NODE_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
>              maxmem => int($rrdentry->[RRD_NODE_INDEX_MAXMEM] || 0),
>              mem => int($rrdentry->[RRD_NODE_INDEX_MEM] || 0),
> +            # PSI avg10 in percent; undef if the node broadcasts an older
> +            # schema without pressure information
> +            pressurecpusome => rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_CPU_SOME),
> +            pressurememorysome =>
> +                rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_MEM_SOME),
>          };
>      }
>  
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 6e1827f..c34da89 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -155,6 +155,15 @@ my $auto_rebalance_failure_backoff_max = 3600;
>  # minimum time before a resource is considered for rebalancing again after it
>  # was successfully moved by the load balancer
>  my $auto_rebalance_resource_cooldown = 600;
> +# minimum pressure stall value (PSI avg10, in percent) among the nodes for
> +# the load balancer to act at all: load asymmetry by itself is not a
> +# performance problem, and the node pressure stall information directly
> +# measures which share of its time work on the node spends waiting on
> +# resources - i.e. whether anything could actually run better with more
> +# headroom - independent of the cluster's size or load level; if no pressure
> +# information is available at all (older stats schema), balancing proceeds
> +# as before

I'd rather have the feature be gated through a switch or as described
above a new option value with a default that wouldn't change existing
setups than whether a node sends this information or not.

> +my $auto_rebalance_min_pressure = 10.0;
>  # minimum absolute improvement of the node imbalance for a rebalance motion;
>  # the relative margin alone provides almost no hysteresis when a single
>  # resource dominates the cluster load: with two nodes, a resource load v and
> @@ -368,6 +377,13 @@ sub load_balance {
>          return;
>      }
>  
> +    # nothing to gain from balancing while no workload is short on resources
> +    my $max_pressure = $online_node_usage->max_pressure();
> +    if (defined($max_pressure) && $max_pressure < $auto_rebalance_min_pressure) {
> +        $self->{sustained_imbalance_round} = 0;
> +        return;
> +    }
> +
>      my $imbalance = $online_node_usage->calculate_node_imbalance();
>  
>      # do not load balance unless imbalance threshold has been exceeded
> diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
> index aa1a886..8766a6f 100644
> --- a/src/PVE/HA/Sim/Hardware.pm
> +++ b/src/PVE/HA/Sim/Hardware.pm
> @@ -791,7 +791,8 @@ sub get_cfs_state {
>  #   crm enable-node-maintenance <node>
>  #   crm disable-node-maintenance <node>
>  #   pve-manager-version <node> set <version>   # note: this is NOT the *ha*-manager version
> -#   node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>]  # load outside HA services
> +#   node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>] [pressurecpusome <pct>] [pressurememorysome <pct>]
> +#     (load and pressure outside of the HA-managed services)
>  #   reboot <node>
>  #   shutdown <node>
>  #   restart-lrm <node>
> @@ -888,7 +889,12 @@ sub sim_hardware_cmd {
>              die "sim_hardware_cmd: missing target stat for '$action' command"
>                  if !@params;
>  
> -            my $conversions = { cpu => sub { 0.0 + $_[0] }, mem => sub { $_[0] * 1024**2 } };
> +            my $conversions = {
> +                cpu => sub { 0.0 + $_[0] },
> +                mem => sub { $_[0] * 1024**2 },
> +                pressurecpusome => sub { 0.0 + $_[0] },
> +                pressurememorysome => sub { 0.0 + $_[0] },
> +            };
>  
>              for my ($target, $val) (@params) {
>                  die "sim_hardware_cmd: missing value for '$action $target' command"
> @@ -1312,6 +1318,12 @@ sub get_dynamic_node_stats {
>          $stats->{$node}->{cpu} = $cstatus->{$node}->{cpu} // 0.0;
>          $stats->{$node}->{maxmem} = $stats->{$node}->{maxmem} // $default_node_maxmem;
>          $stats->{$node}->{mem} = $cstatus->{$node}->{mem} // 0;
> +        # PSI avg10 in percent; keep undef (pressure information not
> +        # available) distinct from an explicit zero (no pressure)
> +        for my $pressure (qw(pressurecpusome pressurememorysome)) {
> +            $stats->{$node}->{$pressure} = $cstatus->{$node}->{$pressure}
> +                if defined($cstatus->{$node}->{$pressure});
> +        }
>      }
>  
>      my $service_conf = $self->read_service_config();
> diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
> index 659ab30..e0890fb 100644
> --- a/src/PVE/HA/Usage.pm
> +++ b/src/PVE/HA/Usage.pm
> @@ -66,6 +66,15 @@ sub calculate_node_imbalance {
>      die "implement in subclass";
>  }
>  
> +# Returns the highest pressure stall value (PSI avg10, in percent) among the
> +# nodes, or undef if the usage implementation has no notion of pressure
> +# (e.g. basic or static) or no node reported pressure information.
> +sub max_pressure {
> +    my ($self) = @_;
> +
> +    return undef;
> +}
> +
>  sub score_best_balancing_migrations {
>      my ($self, $migration_candidates, $limit) = @_;
>  
> diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
> index fc88ddf..b2ed79a 100644
> --- a/src/PVE/HA/Usage/Dynamic.pm
> +++ b/src/PVE/HA/Usage/Dynamic.pm
> @@ -184,6 +184,29 @@ sub calculate_node_imbalance {
>      return $node_imbalance // 0.0;
>  }
>  
> +# NOTE only the *node* pressure is considered: a guest saturating its own
> +# vCPUs or its own memory reports pressure too, but gets neither more cores
> +# nor more memory by being migrated, while starvation caused by the host
> +# (contention with other guests or host processes) shows up in the node's
> +# pressure - and only that kind can be improved by a migration

This information should go right next to some code why this exact
decision was made, maybe a bit more concise

> +sub max_pressure {
> +    my ($self) = @_;
> +
> +    my $max_pressure;
> +
> +    for my $node ($self->list_nodes()) {
> +        my $stats = $self->{'node-stats'}->{$node} or next;
> +
> +        for my $pressure ($stats->@{qw(pressurecpusome pressurememorysome)}) {
> +            next if !defined($pressure);
> +            $max_pressure = $pressure
> +                if !defined($max_pressure) || $pressure > $max_pressure;
> +        }

CPU and memory PSI values do have very different meanings.

The SOME CPU PSI value does say that some processes are delayed on
running on the CPU because of CPU scheduling contention to serve all of
them fairly.

The SOME memory PSI value does say that some processes are delayed
because there is memory reclaiming going on. That is, there is more
memory demand than can be currently provided. That is much more severe,
because memory is a truly finite resource (that can end in OOM kills),
while CPU pressure will just slow down workloads.

So if we would go for this, we should measure them by different
standards.

> +    }
> +
> +    return $max_pressure;
> +}
> +
>  sub score_best_balancing_migrations {
>      my ($self, $migration_candidates, $limit) = @_;
>