[PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads

[Michael Ryom <Michael@RyomHerold.dk>]

Add a test case, where the base loads of two nodes, which are not caused
by any HA resource, are flipped by about half a percentage point of node
load in alternating directions, while a single dominant HA resource runs
on one of them.

The expected log records the current behavior, where vm:100 is migrated
back and forth after each flip (see bug #8059). It is adapted by the
following patch.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 .../test-crs-dynamic-auto-rebalance9/README   | 16 +++++
 .../test-crs-dynamic-auto-rebalance9/cmdlist  | 19 ++++++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  3 +
 .../hardware_status                           |  4 ++
 .../log.expect                                | 66 +++++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  3 +
 .../static_service_stats                      |  3 +
 9 files changed, 121 insertions(+)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_service_stats

diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README
new file mode 100644
index 0000000..93b27c4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/README
@@ -0,0 +1,16 @@
+Test that a single dominant HA resource does not oscillate between two nodes
+whose base loads (i.e. load not caused by any HA-managed service, here set via
+the node base load of the simulated hardware) fluctuate around each other.
+
+The cluster has two nodes with an equal base load and one dominant movable
+HA resource vm:100 on node1. The imbalance is permanently above the trigger
+threshold, but no motion can improve it, so the load balancer stays armed
+without acting. Then the node base loads are repeatedly flipped by about half
+a percentage point of node load in alternating directions.
+
+Each raw sample after a flip qualifies a migration of vm:100 to the less
+loaded node (and after the next flip, back again), as the expected relative
+imbalance improvement of about 10.4% exceeds the default margin of 10%.
+Acting on the raw samples moves the dominant resource back and forth with
+every flip, even though the fluctuation is well within the noise of a real
+node's base load.
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/cmdlist b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
new file mode 100644
index 0000000..eeee26c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
@@ -0,0 +1,19 @@
+[
+    [ "power node1 on", "power node2 on" ],
+    [
+        "node node1 set-dynamic-stats cpu 5.064 mem 0",
+        "node node2 set-dynamic-stats cpu 4.536 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 4.536 mem 0",
+        "node node2 set-dynamic-stats cpu 5.064 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 5.064 mem 0",
+        "node node2 set-dynamic-stats cpu 4.536 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 4.536 mem 0",
+        "node node2 set-dynamic-stats cpu 5.064 mem 0"
+    ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
new file mode 100644
index 0000000..ef14918
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "cpu": 9.6, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/hardware_status b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
new file mode 100644
index 0000000..bb0cf81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
@@ -0,0 +1,4 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
new file mode 100644
index 0000000..efd4d2c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
@@ -0,0 +1,66 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:100' on node 'node1'
+info     20    node1/crm: service 'vm:100': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info    120      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info    140    node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
+info    140    node1/crm: got crm command: migrate vm:100 node2
+info    140    node1/crm: migrate service 'vm:100' to node 'node2'
+info    140    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    141    node1/lrm: service vm:100 - start migrate to node 'node2'
+info    141    node1/lrm: service vm:100 - end migrate to node 'node2'
+info    143    node2/lrm: got lock 'ha_agent_node2_lock'
+info    143    node2/lrm: status change wait_for_agent_lock => active
+info    160    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node2)
+info    163    node2/lrm: starting service vm:100
+info    163    node2/lrm: service status vm:100 started
+info    220      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info    220      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info    220    node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
+info    220    node1/crm: got crm command: migrate vm:100 node1
+info    220    node1/crm: migrate service 'vm:100' to node 'node1'
+info    220    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node2, target = node1)
+info    223    node2/lrm: service vm:100 - start migrate to node 'node1'
+info    223    node2/lrm: service vm:100 - end migrate to node 'node1'
+info    240    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node1)
+info    241    node1/lrm: starting service vm:100
+info    241    node1/lrm: service status vm:100 started
+info    320      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info    320      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info    360    node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
+info    360    node1/crm: got crm command: migrate vm:100 node2
+info    360    node1/crm: migrate service 'vm:100' to node 'node2'
+info    360    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    361    node1/lrm: service vm:100 - start migrate to node 'node2'
+info    361    node1/lrm: service vm:100 - end migrate to node 'node2'
+info    380    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node2)
+info    383    node2/lrm: starting service vm:100
+info    383    node2/lrm: service status vm:100 started
+info    420      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info    420      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info    440    node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
+info    440    node1/crm: got crm command: migrate vm:100 node1
+info    440    node1/crm: migrate service 'vm:100' to node 'node1'
+info    440    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node2, target = node1)
+info    443    node2/lrm: service vm:100 - start migrate to node 'node1'
+info    443    node2/lrm: service vm:100 - end migrate to node 'node1'
+info    460    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node1)
+info    461    node1/lrm: starting service vm:100
+info    461    node1/lrm: service status vm:100 started
+info   1020     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/manager_status b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/service_config b/src/test/test-crs-dynamic-auto-rebalance9/service_config
new file mode 100644
index 0000000..60688cf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/service_config
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
new file mode 100644
index 0000000..7eee778
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 }
+}
-- 
2.47.3