[PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin

[Michael Ryom <Michael@RyomHerold.dk>]

The automatic load balancer only requires a relative improvement of the
node imbalance (default 10%). When a single resource dominates the
cluster load, the relative margin provides almost no hysteresis: with
two nodes, a resource load v and a node base load difference d, the
relative improvement of moving the resource is 2d / (v + d), so it
exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
that is a base load difference of about half a percentage point, well
within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
each fluctuation then qualifies a motion (and, after it was carried
out, its reverse) (see bug #8059).

Add the ha-auto-rebalance-margin-mode option, which allows to interpret
the existing margin as the minimum absolute improvement in percentage
points instead. This creates a dead band for motions that shuffle load
without meaningfully improving the cluster balance. The relative margin
was introduced when the imbalance was not yet bounded for different
cluster sizes; now that it is in [0, 1] for all cluster sizes, an
absolute margin is meaningful on its own.

The margin mode defaults to 'relative', i.e. the behavior of existing
setups does not change.

The expected log of test-crs-dynamic-auto-rebalance8 changes, as the
improvement of 4.4 percentage points of migrating vm:101 is below the
configured absolute margin of 5 percentage points.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                              | 14 ++++++++++----
 .../test-crs-dynamic-auto-rebalance8/log.expect    |  9 ---------
 2 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index c480b73..b804602 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -141,6 +141,8 @@ sub update_crs_scheduler_mode {
         // 3;
     $self->{crs}->{auto_rebalance}->{margin} =
         ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+    $self->{crs}->{auto_rebalance}->{margin_mode} = $crs_cfg->{'ha-auto-rebalance-margin-mode'}
+        // 'relative';
     $self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0;
 
     my $old_mode = $self->{crs}->{scheduler};
@@ -370,8 +372,8 @@ sub load_balance {
 
     return if $self->any_resource_motion_queued_or_running();
 
-    my ($threshold, $method, $hold_duration, $margin) =
-        $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
+    my ($threshold, $method, $hold_duration, $margin, $margin_mode) =
+        $auto_rebalance_opts->@{qw(threshold method hold_duration margin margin_mode)};
 
     # with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
     # migration, churning the cluster forever - skip and warn instead of silently looping.
@@ -421,8 +423,12 @@ sub load_balance {
 
     my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
 
-    my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
-    return if $relative_change < $margin;
+    # the relative margin alone provides almost no hysteresis if a single resource
+    # dominates the cluster load, so tiny fluctuations of the node base loads can
+    # make a motion and its reverse qualify; an absolute margin guards against it
+    my $absolute_change = $imbalance - $target_imbalance;
+    my $change = $margin_mode eq 'absolute' ? $absolute_change : $absolute_change / $imbalance;
+    return if $change < $margin;
 
     my ($sid, $source, $target) = $migration->@{qw(sid source-node target-node)};
 
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
index df11212..6c84e42 100644
--- a/src/test/test-crs-dynamic-auto-rebalance8/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
@@ -27,13 +27,4 @@ info     23    node2/lrm: got lock 'ha_agent_node2_lock'
 info     23    node2/lrm: status change wait_for_agent_lock => active
 info     23    node2/lrm: starting service vm:200
 info     23    node2/lrm: service status vm:200 started
-info     80    node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 35.6% to 31.1%)
-info     80    node1/crm: got crm command: migrate vm:101 node2
-info     80    node1/crm: migrate service 'vm:101' to node 'node2'
-info     80    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
-info     81    node1/lrm: service vm:101 - start migrate to node 'node2'
-info     81    node1/lrm: service vm:101 - end migrate to node 'node2'
-info    100    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node2)
-info    103    node2/lrm: starting service vm:101
-info    103    node2/lrm: service status vm:101 started
 info    620     hardware: exit simulation - done
-- 
2.47.3