Re: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement

[Michael Ryom <Michael@RyomHerold.dk>]

Thanks for the review!
rather than using a constant, I'd expose this as an ha-auto-rebalance
option right away. I'd suggest using 0.0 as the default value, meaning
this kind of protection is disabled by default
Done in v2. The value is now the datacenter option
'ha-auto-rebalance-min-improvement' with a default of 0, so the guard is
disabled unless it is configured and the behavior of an existing setup does
not change. The regression test sets the option explicitly, since 5
percentage points is what stopped the oscillation on the reported cluster.
This turns the series into a cross-repository one: a new patch 1 adds the
schema in pve-cluster and a new patch 9 the matching UI field in
pve-manager. pve-ha-manager falls back to the default when the key is
absent, so it still works with an older libpve-cluster-perl.
[factor out the absolute change]
Also done; the difference is computed once as $absolute_change and the
relative one is derived from it.
One note on the follow-up work, so the option naming does not have to be
changed again later: I had reused this value in a follow-up patch as the
share of a node's capacity that a motion has to free in a pressured
dimension. With the new default of 0 that would have silently disabled that
requirement as well, so the two are now separate options - the follow-up
introduces 'ha-auto-rebalance-min-relief' (default 5%) for the pressure
relief share, while 'ha-auto-rebalance-min-improvement' only ever applies
to the imbalance metric itself.
v2 of the series follows; the memhost patch carries your Reviewed-by, as
the only change there is the suggested '//' instead of '||'.
________________________________
From: Dominik Rusovac <d.rusovac@proxmox.com>
Sent: 22 September 2026 08:18
To: Michael Ryom <Michael@RyomHerold.dk>; pve-devel@lists.proxmox.com <pve-devel@lists.proxmox.com>
Cc: d.kral@proxmox.com <d.kral@proxmox.com>
Subject: Re: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement

thanks for all the helpful details in the related report [0]!

see comments inline

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=8059

On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote:
> The automatic load balancer only requires a relative improvement of the
> node imbalance (default 10%). When a single resource dominates the
> cluster load, the relative margin provides almost no hysteresis: with
> two nodes, a resource load v and a node base load difference d, the
> relative improvement of moving the resource is 2d / (v + d), so it
> exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
> that is a base load difference of about half a percentage point, well
> within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
> each fluctuation then qualifies a motion (and, after it was carried
> out, its reverse).
>
> Additionally require the imbalance improvement to exceed a minimum
> absolute value of 5 percentage points. This creates a dead band for
> motions that shuffle load without meaningfully improving the cluster
> balance. No expected motion of an existing regression test is affected;
> the smallest occurring improvement there is 9.6 percentage points.
>

thanks for unveiling this!

the math deriving d > v * m / (2 - m) in [0] checks out

TL;DR: in the given scenario the load-heavy resource keeps getting
migrated back and forth between the two nodes, because the load
balancer's decisions are triggered by background noise in d, rather than
by a genuine change in the cluster's load imbalance that the relative
margin m is meant to filter for

to me, checking that the imbalance improvement exceeds an "absolute"
margin appears to be a good way of guarding against this issue

> Like the backoff and cooldown constants, this could be exposed as a
> ha-auto-rebalance-* option later.

rather than using a constant, I'd expose this as an ha-auto-rebalance
option right away. I'd suggest using 0.0 as the default value, meaning
this kind of protection is disabled by default

[snip]

>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 30b9ae2..2c4c273 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -151,6 +151,13 @@ my $auto_rebalance_failure_backoff_max = 3600;
>  # minimum time before a resource is considered for rebalancing again after it
>  # was successfully moved by the load balancer
>  my $auto_rebalance_resource_cooldown = 600;
> +# minimum absolute improvement of the node imbalance for a rebalance motion;
> +# the relative margin alone provides almost no hysteresis when a single
> +# resource dominates the cluster load: with two nodes, a resource load v and
> +# a node base load difference d, the relative improvement of a motion is
> +# 2d / (v + d), so for large v tiny fluctuations of d around zero are enough
> +# to make the motion (and afterwards its reverse) qualify
> +my $auto_rebalance_min_imbalance_improvement = 0.05;
>
>  my $is_on_rebalance_cooldown = sub {
>      my ($self, $sid, $now) = @_;
> @@ -387,6 +394,7 @@ sub load_balance {
>
>      my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
>      return if $relative_change < $margin;
> +    return if ($imbalance - $target_imbalance) < $auto_rebalance_min_imbalance_improvement;

nit: factor out the absolute change and reuse it

       my $absolute_change = $imbalance - $target_imbalance;
       my $relative_change = $absolute_change / $imbalance;
       return if $relative_change < $margin;
       return if $absolute_change < $auto_rebalance_min_imbalance_improvement;

[snip]

> diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
> new file mode 100644
> index 0000000..488a018
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance7/README
> @@ -0,0 +1,12 @@
> +Test that the auto rebalance system does not act on a rebalance motion whose
> +absolute improvement of the node imbalance is below the minimum, even though
> +the relative improvement exceeds the configured margin.
> +
> +The cluster has two nodes with a HA resource excluded from auto rebalancing
> +on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and
> +one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays
> +above the default trigger threshold of 30%, and migrating vm:101 to node2
> +would improve the imbalance to 31.1%, i.e. by 12.5% relative, which exceeds
> +the default margin of 10%. However, the absolute improvement of 4.4
> +percentage points is below the required minimum of 5 percentage points, so
> +the motion must not be issued.

the numbers check out

thx for adding a test right away!

[snip]