[PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion

[Michael Ryom <Michael@RyomHerold.dk>]

When a single resource dominates the cluster load and the node base
loads fluctuate around each other, the automatic load balancer moves
the resource back and forth between the nodes (see bug #8059).

Add a test case, where the base loads of two nodes are swapped right
after the only movable resource vm:101 was migrated, which shows that
vm:101 is migrated back immediately.

The test case already configures the ha-auto-rebalance-cooldown option,
which is not used yet, and its expected log records the current
behavior. It is adapted by the following patch.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 .../test-crs-dynamic-auto-rebalance7/README   | 17 +++++++
 .../test-crs-dynamic-auto-rebalance7/cmdlist  |  7 +++
 .../datacenter.cfg                            |  7 +++
 .../dynamic_service_stats                     |  5 ++
 .../hardware_status                           |  4 ++
 .../log.expect                                | 50 +++++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  5 ++
 .../static_service_stats                      |  5 ++
 9 files changed, 101 insertions(+)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/static_service_stats

diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
new file mode 100644
index 0000000..db7bb00
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/README
@@ -0,0 +1,17 @@
+Test that a HA resource which was successfully moved by the auto rebalancer
+is not moved again during the configured cooldown, even if a qualifying
+rebalance motion exists.
+
+The cluster has two nodes, where each node has a HA resource excluded from
+the auto rebalancing (vm:100 on node1 and vm:200 on node2), which act as the
+node base loads, and one movable HA resource vm:101 on node1. Initially,
+node1 is much more loaded than node2, so vm:101 is migrated to node2.
+
+Afterwards, the base loads are swapped, i.e. node2 becomes much more loaded
+than node1. Without a cooldown, the auto rebalancer immediately migrates
+vm:101 back to node1, as the expected relative imbalance improvement
+qualifies. With ha-auto-rebalance-cooldown set to 600 seconds, vm:101 must
+not be moved again until 10 minutes after its first migration has finished.
+
+This mimics a single dominant HA resource oscillating between two nodes,
+whose base loads fluctuate around each other.
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/cmdlist b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
new file mode 100644
index 0000000..1de33e4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
@@ -0,0 +1,7 @@
+[
+    [ "power node1 on", "power node2 on" ],
+    [
+        "service vm:100 set-dynamic-stats cpu 0.96 mem 0",
+        "service vm:200 set-dynamic-stats cpu 14.4 mem 0"
+    ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
new file mode 100644
index 0000000..f34bb81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
@@ -0,0 +1,7 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1,
+        "ha-auto-rebalance-cooldown": 600
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
new file mode 100644
index 0000000..3d2d743
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "cpu": 16.8, "mem": 0 },
+    "vm:101": { "cpu": 4.8, "mem": 0 },
+    "vm:200": { "cpu": 0.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/hardware_status b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
@@ -0,0 +1,4 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
new file mode 100644
index 0000000..c87f487
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -0,0 +1,50 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:100' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:200' on node 'node2'
+info     20    node1/crm: service 'vm:100': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:200': state changed from 'request_start' to 'started'  (node = node2)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:200
+info     23    node2/lrm: service status vm:200 started
+info     80    node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 91.5% to 48.9%)
+info     80    node1/crm: got crm command: migrate vm:101 node2
+info     80    node1/crm: migrate service 'vm:101' to node 'node2'
+info     80    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info     81    node1/lrm: service vm:101 - start migrate to node 'node2'
+info     81    node1/lrm: service vm:101 - end migrate to node 'node2'
+info    100    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node2)
+info    103    node2/lrm: starting service vm:101
+info    103    node2/lrm: service status vm:101 started
+info    120      cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
+info    120      cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
+info    160    node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
+info    160    node1/crm: got crm command: migrate vm:101 node1
+info    160    node1/crm: migrate service 'vm:101' to node 'node1'
+info    160    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node2, target = node1)
+info    163    node2/lrm: service vm:101 - start migrate to node 'node1'
+info    163    node2/lrm: service vm:101 - end migrate to node 'node1'
+info    180    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node1)
+info    181    node1/lrm: starting service vm:101
+info    181    node1/lrm: service status vm:101 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/manager_status b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/service_config b/src/test/test-crs-dynamic-auto-rebalance7/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/service_config
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
new file mode 100644
index 0000000..f7d1a27
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "maxcpu": 20.0, "maxmem": 8589934592 },
+    "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 },
+    "vm:200": { "maxcpu": 16.0, "maxmem": 8589934592 }
+}
-- 
2.47.3