[PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources

[Michael Ryom <Michael@RyomHerold.dk>]

When a single resource dominates the cluster load, the relative
improvement of moving it between two nodes is 2d / (v + d), where v is
the load of the resource and d is the difference of the node base loads.
Small fluctuations of d around zero are enough to make a motion and,
right afterwards, its reverse qualify, so the resource is moved back and
forth (see bug #8059).

Add the ha-auto-rebalance-cooldown option, which keeps resources that
were successfully moved by the load balancer from being considered for
another rebalance motion for the given time. The cooldown applies to all
resources of a moved resource bundle, as they were all moved, and a
bundle is skipped if any of its resources is still on cooldown.

The cooldown defaults to 0, i.e. it is disabled and the behavior of
existing setups does not change.

Like the other auto rebalance state, the cooldown is not persisted
across a CRM failover.

The expected log of test-crs-dynamic-auto-rebalance7 changes, as vm:101
is not migrated back right after it was migrated, but only after the
configured cooldown of 600 seconds has expired.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                         | 27 +++++++++++++++++++
 .../log.expect                                | 15 +++++------
 2 files changed, 33 insertions(+), 9 deletions(-)

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 18ad12f..c480b73 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -91,6 +91,9 @@ sub new {
             # rebalance motions which failed $count times in a row, the last
             # time in rebalancing attempt $attempt
             failures => {},
+            # $sid => $time until which a resource successfully moved by
+            # load_balance() is not considered for rebalancing
+            cooldown => {},
         },
         group_migration_round => 3, # wait a little bit
     }, $class;
@@ -138,6 +141,7 @@ sub update_crs_scheduler_mode {
         // 3;
     $self->{crs}->{auto_rebalance}->{margin} =
         ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+    $self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0;
 
     my $old_mode = $self->{crs}->{scheduler};
     my $new_mode = $crs_cfg->{ha} || 'basic';
@@ -259,6 +263,10 @@ sub record_rebalance_motion_result {
     if ($success) {
         delete $state->{failures}->{$key};
         $state->{attempt_ongoing} = 0;
+
+        my $cooldown = $self->{crs}->{auto_rebalance}->{cooldown};
+        $state->{cooldown}->{$sid} = $self->{haenv}->get_time() + $cooldown if $cooldown > 0;
+
         return;
     }
 
@@ -277,6 +285,20 @@ sub record_rebalance_motion_result {
     $self->{haenv}->log('warning', $msg);
 }
 
+# Returns whether $sid was moved by load_balance() too recently to be considered
+# for rebalancing again.
+my sub is_on_rebalance_cooldown {
+    my ($self, $sid, $now) = @_;
+
+    my $cooldown = $self->{auto_rebalance_state}->{cooldown};
+    my $until = $cooldown->{$sid} // return 0;
+
+    return 1 if $now < $until;
+
+    delete $cooldown->{$sid};
+    return 0;
+}
+
 # Returns a hash of hashes, where each item contains the resource bundle's
 # leader, the list of HA resources in the resource bundle, and the list of
 # possible nodes to migrate to.
@@ -291,8 +313,13 @@ sub get_resource_migration_candidates {
     my $resource_bundles =
         get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity);
 
+    my $now = $self->{haenv}->get_time();
+
     my @compact_migration_candidates = ();
     for my $leader_sid (sort keys %$resource_bundles) {
+        my $bundle = $resource_bundles->{$leader_sid};
+        next if grep { is_on_rebalance_cooldown($self, $_, $now) } @$bundle;
+
         my $current_leader_node = $ss->{$leader_sid}->{node};
         my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
 
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
index c87f487..a82f4bc 100644
--- a/src/test/test-crs-dynamic-auto-rebalance7/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -38,13 +38,10 @@ info    103    node2/lrm: starting service vm:101
 info    103    node2/lrm: service status vm:101 started
 info    120      cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
 info    120      cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
-info    160    node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
-info    160    node1/crm: got crm command: migrate vm:101 node1
-info    160    node1/crm: migrate service 'vm:101' to node 'node1'
-info    160    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node2, target = node1)
-info    163    node2/lrm: service vm:101 - start migrate to node 'node1'
-info    163    node2/lrm: service vm:101 - end migrate to node 'node1'
-info    180    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node1)
-info    181    node1/lrm: starting service vm:101
-info    181    node1/lrm: service status vm:101 started
+info    700    node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
+info    700    node1/crm: got crm command: migrate vm:101 node1
+info    700    node1/crm: migrate service 'vm:101' to node 'node1'
+info    700    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node2, target = node1)
+info    703    node2/lrm: service vm:101 - start migrate to node 'node1'
+info    703    node2/lrm: service vm:101 - end migrate to node 'node1'
 info    720     hardware: exit simulation - done
-- 
2.47.3