Re: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement

["Dominik Rusovac" <d.rusovac@proxmox.com>]

thanks for all the helpful details in the related report [0]!

see comments inline 

[0]: https://bugzilla.proxmox.com/show_bug.cgi?id=8059

On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote:
> The automatic load balancer only requires a relative improvement of the
> node imbalance (default 10%). When a single resource dominates the
> cluster load, the relative margin provides almost no hysteresis: with
> two nodes, a resource load v and a node base load difference d, the
> relative improvement of moving the resource is 2d / (v + d), so it
> exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
> that is a base load difference of about half a percentage point, well
> within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
> each fluctuation then qualifies a motion (and, after it was carried
> out, its reverse).
>
> Additionally require the imbalance improvement to exceed a minimum
> absolute value of 5 percentage points. This creates a dead band for
> motions that shuffle load without meaningfully improving the cluster
> balance. No expected motion of an existing regression test is affected;
> the smallest occurring improvement there is 9.6 percentage points.
>

thanks for unveiling this!

the math deriving d > v * m / (2 - m) in [0] checks out

TL;DR: in the given scenario the load-heavy resource keeps getting
migrated back and forth between the two nodes, because the load
balancer's decisions are triggered by background noise in d, rather than
by a genuine change in the cluster's load imbalance that the relative
margin m is meant to filter for

to me, checking that the imbalance improvement exceeds an "absolute"
margin appears to be a good way of guarding against this issue

> Like the backoff and cooldown constants, this could be exposed as a
> ha-auto-rebalance-* option later.

rather than using a constant, I'd expose this as an ha-auto-rebalance
option right away. I'd suggest using 0.0 as the default value, meaning
this kind of protection is disabled by default

[snip] 

>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index 30b9ae2..2c4c273 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -151,6 +151,13 @@ my $auto_rebalance_failure_backoff_max = 3600;
>  # minimum time before a resource is considered for rebalancing again after it
>  # was successfully moved by the load balancer
>  my $auto_rebalance_resource_cooldown = 600;
> +# minimum absolute improvement of the node imbalance for a rebalance motion;
> +# the relative margin alone provides almost no hysteresis when a single
> +# resource dominates the cluster load: with two nodes, a resource load v and
> +# a node base load difference d, the relative improvement of a motion is
> +# 2d / (v + d), so for large v tiny fluctuations of d around zero are enough
> +# to make the motion (and afterwards its reverse) qualify
> +my $auto_rebalance_min_imbalance_improvement = 0.05;
>  
>  my $is_on_rebalance_cooldown = sub {
>      my ($self, $sid, $now) = @_;
> @@ -387,6 +394,7 @@ sub load_balance {
>  
>      my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
>      return if $relative_change < $margin;
> +    return if ($imbalance - $target_imbalance) < $auto_rebalance_min_imbalance_improvement;

nit: factor out the absolute change and reuse it

       my $absolute_change = $imbalance - $target_imbalance;
       my $relative_change = $absolute_change / $imbalance;
       return if $relative_change < $margin;
       return if $absolute_change < $auto_rebalance_min_imbalance_improvement;

[snip]

> diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
> new file mode 100644
> index 0000000..488a018
> --- /dev/null
> +++ b/src/test/test-crs-dynamic-auto-rebalance7/README
> @@ -0,0 +1,12 @@
> +Test that the auto rebalance system does not act on a rebalance motion whose
> +absolute improvement of the node imbalance is below the minimum, even though
> +the relative improvement exceeds the configured margin.
> +
> +The cluster has two nodes with a HA resource excluded from auto rebalancing
> +on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and
> +one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays
> +above the default trigger threshold of 30%, and migrating vm:101 to node2
> +would improve the imbalance to 31.1%, i.e. by 12.5% relative, which exceeds
> +the default margin of 10%. However, the absolute improvement of 4.4
> +percentage points is below the required minimum of 5 percentage points, so
> +the motion must not be issued.

the numbers check out

thx for adding a test right away!

[snip]