[PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown

[Michael Ryom <Michael@RyomHerold.dk>]

The automatic load balancer only refrains from issuing new motions while
a migrate/relocate is queued or running. It never records the outcome of
a motion it issued. If the motion fails (e.g. because the guest has a
local CD-ROM image attached), the resource returns to 'started' on its
source node, and the very same motion is selected again as soon as the
hold duration has passed. This results in an endless loop of failing
migrations, in practice about every 40 seconds with the default hold
duration.

Track the last motion issued by load_balance() and, once it is no
longer queued or running, check whether the resource ended up on the
target node:

- on failure, back off that (resource, target) pair with an exponential
  delay (60s, 120s, ... capped at 1h) and log a warning;
- on success, put the resource on a cooldown (10 minutes), so that a
  resource is not moved back and forth in quick succession, which can
  happen when a single resource dominates the cluster load and the node
  base loads fluctuate around each other.

Backed-off targets and resources on cooldown are pruned in
get_resource_migration_candidates() before scoring, so the balancer can
still pick other, possibly worse but feasible, migrations.

Like sustained_imbalance_round, this state is not persisted across a
CRM failover. The delays are constants for now; they could be exposed
as ha-auto-rebalance-* options later.

The expected log of test-crs-dynamic-auto-rebalance-topsis3 changes,
because the resource vm:105, which was moved to node2 when its load was
high, is on cooldown when its load has gone down again and is not moved
back immediately anymore; the balancer instead improves the imbalance
by moving vm:106.

Add two regression tests: one for the backoff of failed motions (using
a fa: resource, which fails its first two relocations) and one for the
cooldown suppressing an immediate qualifying reverse migration after a
successful rebalance motion.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                         | 118 ++++++++++++++++++
 .../log.expect                                |  18 +--
 .../test-crs-dynamic-auto-rebalance5/README   |  13 ++
 .../test-crs-dynamic-auto-rebalance5/cmdlist  |   3 +
 .../datacenter.cfg                            |   6 +
 .../dynamic_service_stats                     |   6 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  77 ++++++++++++
 .../manager_status                            |   1 +
 .../service_config                            |   6 +
 .../static_service_stats                      |   6 +
 .../test-crs-dynamic-auto-rebalance6/README   |  17 +++
 .../test-crs-dynamic-auto-rebalance6/cmdlist  |   7 ++
 .../datacenter.cfg                            |   6 +
 .../dynamic_service_stats                     |   5 +
 .../hardware_status                           |   4 +
 .../log.expect                                |  41 ++++++
 .../manager_status                            |   1 +
 .../service_config                            |   5 +
 .../static_service_stats                      |   5 +
 20 files changed, 341 insertions(+), 9 deletions(-)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..30b9ae2 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -72,6 +72,15 @@ sub new {
         # this is not persisted for a CRM failover as in the mean time
         # the usage statistics might have change quite a bit already
         sustained_imbalance_round => 0,
+        # outcome tracking for resource motions issued by load_balance()
+        #
+        # like sustained_imbalance_round, this is not persisted for a CRM
+        # failover
+        auto_rebalance_state => {
+            pending => undef, # last issued motion, until it has finished
+            failures => {}, # "$sid:$target" => { count => $count, until => $time }
+            cooldown => {}, # $sid => $time
+        },
         group_migration_round => 3, # wait a little bit
     }, $class;
 
@@ -135,6 +144,92 @@ sub update_crs_scheduler_mode {
     return;
 }
 
+# delay before a failed rebalance motion to the same target node is retried,
+# doubled for each consecutive failure up to the maximum
+my $auto_rebalance_failure_backoff_base = 60;
+my $auto_rebalance_failure_backoff_max = 3600;
+# minimum time before a resource is considered for rebalancing again after it
+# was successfully moved by the load balancer
+my $auto_rebalance_resource_cooldown = 600;
+
+my $is_on_rebalance_cooldown = sub {
+    my ($self, $sid, $now) = @_;
+
+    my $cooldown = $self->{auto_rebalance_state}->{cooldown};
+    my $until = $cooldown->{$sid} // return 0;
+
+    if ($now >= $until) {
+        delete $cooldown->{$sid};
+        return 0;
+    }
+
+    return 1;
+};
+
+my $is_rebalance_motion_backed_off = sub {
+    my ($self, $sid, $target, $now) = @_;
+
+    my $failure = $self->{auto_rebalance_state}->{failures}->{"$sid:$target"} // return 0;
+
+    # keep the entry after expiry, so that the failure count keeps growing if
+    # the next attempt fails as well; it is only reset on success
+    return $now < $failure->{until} ? 1 : 0;
+};
+
+# Checks whether the last resource motion issued by load_balance() has
+# finished and records its outcome: a per-resource cooldown on success, or an
+# exponential backoff for the ($sid, $target) pair on failure.
+my $check_last_rebalance_motion = sub {
+    my ($self) = @_;
+
+    my ($haenv, $ss) = $self->@{qw(haenv ss)};
+    my $state = $self->{auto_rebalance_state};
+
+    my $pending = $state->{pending} // return;
+    my ($sid, $target, $task) = $pending->@{qw(sid target task)};
+
+    my $sd = $ss->{$sid};
+    if (!defined($sd)) {
+        # resource was removed in the meantime
+        $state->{pending} = undef;
+        return;
+    }
+
+    my ($cmd, $sstate) = $sd->@{qw(cmd state)};
+
+    # still queued or in progress
+    return if $sstate eq 'migrate' || $sstate eq 'relocate';
+    return if defined($cmd) && ($cmd->[0] eq 'migrate' || $cmd->[0] eq 'relocate');
+
+    $state->{pending} = undef;
+
+    my $now = $haenv->get_time();
+    my $key = "$sid:$target";
+
+    if (defined($sd->{node}) && $sd->{node} eq $target) {
+        delete $state->{failures}->{$key};
+        $state->{cooldown}->{$sid} = $now + $auto_rebalance_resource_cooldown;
+        return;
+    }
+
+    my $failure = $state->{failures}->{$key} //= { count => 0 };
+    $failure->{count}++;
+
+    my $delay = $auto_rebalance_failure_backoff_base * 2**($failure->{count} - 1);
+    $delay = $auto_rebalance_failure_backoff_max
+        if $delay > $auto_rebalance_failure_backoff_max;
+
+    $failure->{until} = $now + $delay;
+
+    $haenv->log(
+        'warning',
+        "auto rebalance - $task $sid to $target did not succeed"
+            . " (attempt $failure->{count}), not retrying it for ${delay}s",
+    );
+
+    return;
+};
+
 # Returns a hash of lists, which contain the running, movable, non-moving HA resource
 # bundles, which are on the same node, implied by the strict positive resource
 # affinity rules.
@@ -200,9 +295,12 @@ sub get_resource_migration_candidates {
 
     my $resource_bundles =
         get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity);
+    my $now = $self->{haenv}->get_time();
 
     my @compact_migration_candidates = ();
     for my $leader_sid (sort keys %$resource_bundles) {
+        next if $is_on_rebalance_cooldown->($self, $leader_sid, $now);
+
         my $current_leader_node = $ss->{$leader_sid}->{node};
         my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
 
@@ -213,6 +311,11 @@ sub get_resource_migration_candidates {
 
         delete $target_nodes->{$current_leader_node};
 
+        for my $node (keys %$target_nodes) {
+            delete $target_nodes->{$node}
+                if $is_rebalance_motion_backed_off->($self, $leader_sid, $node, $now);
+        }
+
         next if !%$target_nodes;
 
         push @compact_migration_candidates,
@@ -234,6 +337,10 @@ sub load_balance {
 
     return if !$auto_rebalance_opts->{enable};
     return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
+
+    # record the outcome of the last motion issued here, if it has finished
+    $check_last_rebalance_motion->($self);
+
     return if $self->any_resource_motion_queued_or_running();
 
     my ($threshold, $method, $hold_duration, $margin) =
@@ -295,6 +402,17 @@ sub load_balance {
     $haenv->log('info', "auto rebalance - $task $sid to $target ($imbalance_change_str)");
 
     $self->queue_resource_motion($cmd, $task, $sid, $target);
+
+    # queue_resource_motion() may refuse to queue the motion (e.g. blocking
+    # resources on the target node), so only track it if it was queued
+    if (defined($self->{ss}->{$sid}->{cmd})) {
+        $self->{auto_rebalance_state}->{pending} = {
+            sid => $sid,
+            source => $source,
+            target => $target,
+            task => $task,
+        };
+    }
 }
 
 sub cleanup {
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 2f19c72..dfbc904 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -77,13 +77,13 @@ info    263    node2/lrm: service vm:103 - end migrate to node 'node3'
 info    280    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node3)
 info    285    node3/lrm: starting service vm:103
 info    285    node3/lrm: service status vm:103 started
-info    340    node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
-info    340    node1/crm: got crm command: migrate vm:105 node1
-info    340    node1/crm: migrate service 'vm:105' to node 'node1'
-info    340    node1/crm: service 'vm:105': state changed from 'started' to 'migrate'  (node = node2, target = node1)
-info    343    node2/lrm: service vm:105 - start migrate to node 'node1'
-info    343    node2/lrm: service vm:105 - end migrate to node 'node1'
-info    360    node1/crm: service 'vm:105': state changed from 'migrate' to 'started'  (node = node1)
-info    361    node1/lrm: starting service vm:105
-info    361    node1/lrm: service status vm:105 started
+info    340    node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 30.1% to 20.5%)
+info    340    node1/crm: got crm command: migrate vm:106 node1
+info    340    node1/crm: migrate service 'vm:106' to node 'node1'
+info    340    node1/crm: service 'vm:106': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+info    345    node3/lrm: service vm:106 - start migrate to node 'node1'
+info    345    node3/lrm: service vm:106 - end migrate to node 'node1'
+info    360    node1/crm: service 'vm:106': state changed from 'migrate' to 'started'  (node = node1)
+info    361    node1/lrm: starting service vm:106
+info    361    node1/lrm: service status vm:106 started
 info    820     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README
new file mode 100644
index 0000000..c24c44d
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/README
@@ -0,0 +1,13 @@
+Test that the auto rebalance system backs off failed rebalance motions
+instead of retrying them indefinitely.
+
+The HA resource fa:102 fails its first two migrate/relocate attempts. The
+auto rebalancer first selects fa:102 to be relocated to node2, which fails.
+The (fa:102, node2) pair is backed off for 60 seconds, so on the next
+balancing round the rebalancer relocates fa:102 to node3 instead, which
+fails as well and backs off the (fa:102, node3) pair. On the third attempt,
+the backoff for node2 has expired and the relocation to node2 succeeds.
+
+After the successful relocation, fa:102 is put on the per-resource cooldown,
+so the following balancing round moves vm:102 to node3 instead to reach the
+minimum cluster node imbalance.
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
new file mode 100644
index 0000000..13f90cd
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
@@ -0,0 +1,3 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
new file mode 100644
index 0000000..4eba71c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
@@ -0,0 +1,6 @@
+{
+    "fa:102": { "cpu": 1.0, "mem": 4294967296 },
+    "vm:102": { "cpu": 1.0, "mem": 4294967296 },
+    "vm:103": { "cpu": 1.0, "mem": 4294967296 },
+    "vm:104": { "cpu": 1.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
new file mode 100644
index 0000000..ce8cf0e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
new file mode 100644
index 0000000..1267366
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
@@ -0,0 +1,77 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'fa:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node1'
+info     20    node1/crm: adding new service 'vm:104' on node 'node1'
+info     20    node1/crm: service 'fa:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service fa:102
+info     21    node1/lrm: service status fa:102 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     21    node1/lrm: starting service vm:103
+info     21    node1/lrm: service status vm:103 started
+info     21    node1/lrm: starting service vm:104
+info     21    node1/lrm: service status vm:104 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info     80    node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info     80    node1/crm: got crm command: relocate fa:102 node2
+info     80    node1/crm: relocate service 'fa:102' to node 'node2'
+info     80    node1/crm: service 'fa:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+err      81    node1/lrm: service fa:102 not moved (migration error)
+info     83    node2/lrm: got lock 'ha_agent_node2_lock'
+info     83    node2/lrm: status change wait_for_agent_lock => active
+err     100    node1/crm: service 'fa:102' - migration failed (exit code 1)
+info    100    node1/crm: service 'fa:102': state changed from 'relocate' to 'started'  (node = node1)
+warn    120    node1/crm: auto rebalance - relocate fa:102 to node2 did not succeed (attempt 1), not retrying it for 60s
+info    160    node1/crm: auto rebalance - relocate fa:102 to node3 (expected change for imbalance from 100.0% to 66.1%)
+info    160    node1/crm: got crm command: relocate fa:102 node3
+info    160    node1/crm: relocate service 'fa:102' to node 'node3'
+info    160    node1/crm: service 'fa:102': state changed from 'started' to 'relocate'  (node = node1, target = node3)
+err     161    node1/lrm: service fa:102 not moved (migration error)
+info    165    node3/lrm: got lock 'ha_agent_node3_lock'
+info    165    node3/lrm: status change wait_for_agent_lock => active
+err     180    node1/crm: service 'fa:102' - migration failed (exit code 1)
+info    180    node1/crm: service 'fa:102': state changed from 'relocate' to 'started'  (node = node1)
+warn    200    node1/crm: auto rebalance - relocate fa:102 to node3 did not succeed (attempt 1), not retrying it for 60s
+info    240    node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info    240    node1/crm: got crm command: relocate fa:102 node2
+info    240    node1/crm: relocate service 'fa:102' to node 'node2'
+info    240    node1/crm: service 'fa:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    241    node1/lrm: service fa:102 - start relocate to node 'node2'
+info    241    node1/lrm: stopping service fa:102 (relocate)
+info    241    node1/lrm: service status fa:102 stopped
+info    241    node1/lrm: service fa:102 - end relocate to node 'node2'
+info    260    node1/crm: service 'fa:102': state changed from 'relocate' to 'started'  (node = node2)
+info    263    node2/lrm: starting service fa:102
+info    263    node2/lrm: service status fa:102 started
+info    320    node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 66.1% to 25.0%)
+info    320    node1/crm: got crm command: migrate vm:102 node3
+info    320    node1/crm: migrate service 'vm:102' to node 'node3'
+info    320    node1/crm: service 'vm:102': state changed from 'started' to 'migrate'  (node = node1, target = node3)
+info    321    node1/lrm: service vm:102 - start migrate to node 'node3'
+info    321    node1/lrm: service vm:102 - end migrate to node 'node3'
+info    340    node1/crm: service 'vm:102': state changed from 'migrate' to 'started'  (node = node3)
+info    345    node3/lrm: starting service vm:102
+info    345    node3/lrm: service status vm:102 started
+info    620     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config
new file mode 100644
index 0000000..6c4ad4b
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config
@@ -0,0 +1,6 @@
+{
+    "fa:102": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node1", "state": "started" },
+    "vm:104": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
new file mode 100644
index 0000000..41dc446
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
@@ -0,0 +1,6 @@
+{
+    "fa:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+    "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+    "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 },
+    "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README
new file mode 100644
index 0000000..1931f77
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/README
@@ -0,0 +1,17 @@
+Test that a HA resource which was successfully moved by the auto rebalancer
+is put on a cooldown, during which it is not moved again, even if a
+qualifying rebalance motion exists.
+
+The cluster has two nodes, where each node has a HA resource excluded from
+the auto rebalancing (vm:100 on node1 and vm:200 on node2), which act as the
+node base loads, and one movable HA resource vm:101 on node1. Initially,
+node1 is much more loaded than node2, so vm:101 is migrated to node2.
+
+Afterwards, the base loads are swapped, i.e. node2 becomes much more loaded
+than node1. Without the cooldown, the auto rebalancer would immediately
+migrate vm:101 back to node1, as the expected relative imbalance improvement
+qualifies. With the cooldown, vm:101 must not be moved again for 10 minutes,
+so there are no further migrations until the end of the test.
+
+This mimics a single dominant HA resource oscillating between two nodes,
+whose base loads fluctuate around each other.
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
new file mode 100644
index 0000000..1de33e4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
@@ -0,0 +1,7 @@
+[
+    [ "power node1 on", "power node2 on" ],
+    [
+        "service vm:100 set-dynamic-stats cpu 0.96 mem 0",
+        "service vm:200 set-dynamic-stats cpu 14.4 mem 0"
+    ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
new file mode 100644
index 0000000..3d2d743
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "cpu": 16.8, "mem": 0 },
+    "vm:101": { "cpu": 4.8, "mem": 0 },
+    "vm:200": { "cpu": 0.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
@@ -0,0 +1,4 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
new file mode 100644
index 0000000..37f5cce
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
@@ -0,0 +1,41 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:100' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:200' on node 'node2'
+info     20    node1/crm: service 'vm:100': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:200': state changed from 'request_start' to 'started'  (node = node2)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:200
+info     23    node2/lrm: service status vm:200 started
+info     80    node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 91.5% to 48.9%)
+info     80    node1/crm: got crm command: migrate vm:101 node2
+info     80    node1/crm: migrate service 'vm:101' to node 'node2'
+info     80    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info     81    node1/lrm: service vm:101 - start migrate to node 'node2'
+info     81    node1/lrm: service vm:101 - end migrate to node 'node2'
+info    100    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node2)
+info    103    node2/lrm: starting service vm:101
+info    103    node2/lrm: service status vm:101 started
+info    120      cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
+info    120      cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
new file mode 100644
index 0000000..f7d1a27
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "maxcpu": 20.0, "maxmem": 8589934592 },
+    "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 },
+    "vm:200": { "maxcpu": 16.0, "maxmem": 8589934592 }
+}
-- 
2.47.3