[PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement

[Michael Ryom <Michael@RyomHerold.dk>]

The automatic load balancer only requires a relative improvement of the
node imbalance (default 10%). When a single resource dominates the
cluster load, the relative margin provides almost no hysteresis: with
two nodes, a resource load v and a node base load difference d, the
relative improvement of moving the resource is 2d / (v + d), so it
exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
that is a base load difference of about half a percentage point, well
within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
each fluctuation then qualifies a motion (and, after it was carried
out, its reverse).

Additionally require the imbalance improvement to exceed a minimum
absolute value of 5 percentage points. This creates a dead band for
motions that shuffle load without meaningfully improving the cluster
balance. No expected motion of an existing regression test is affected;
the smallest occurring improvement there is 9.6 percentage points.

Like the backoff and cooldown constants, this could be exposed as a
ha-auto-rebalance-* option later.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                         |  8 +++++
 .../test-crs-dynamic-auto-rebalance7/README   | 12 ++++++++
 .../test-crs-dynamic-auto-rebalance7/cmdlist  |  3 ++
 .../datacenter.cfg                            |  6 ++++
 .../dynamic_service_stats                     |  5 ++++
 .../hardware_status                           |  4 +++
 .../log.expect                                | 30 +++++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  5 ++++
 .../static_service_stats                      |  5 ++++
 10 files changed, 79 insertions(+)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/static_service_stats

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 30b9ae2..2c4c273 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -151,6 +151,13 @@ my $auto_rebalance_failure_backoff_max = 3600;
 # minimum time before a resource is considered for rebalancing again after it
 # was successfully moved by the load balancer
 my $auto_rebalance_resource_cooldown = 600;
+# minimum absolute improvement of the node imbalance for a rebalance motion;
+# the relative margin alone provides almost no hysteresis when a single
+# resource dominates the cluster load: with two nodes, a resource load v and
+# a node base load difference d, the relative improvement of a motion is
+# 2d / (v + d), so for large v tiny fluctuations of d around zero are enough
+# to make the motion (and afterwards its reverse) qualify
+my $auto_rebalance_min_imbalance_improvement = 0.05;
 
 my $is_on_rebalance_cooldown = sub {
     my ($self, $sid, $now) = @_;
@@ -387,6 +394,7 @@ sub load_balance {
 
     my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
     return if $relative_change < $margin;
+    return if ($imbalance - $target_imbalance) < $auto_rebalance_min_imbalance_improvement;
 
     my ($sid, $source, $target) = $migration->@{qw(sid source-node target-node)};
 
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
new file mode 100644
index 0000000..488a018
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/README
@@ -0,0 +1,12 @@
+Test that the auto rebalance system does not act on a rebalance motion whose
+absolute improvement of the node imbalance is below the minimum, even though
+the relative improvement exceeds the configured margin.
+
+The cluster has two nodes with a HA resource excluded from auto rebalancing
+on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and
+one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays
+above the default trigger threshold of 30%, and migrating vm:101 to node2
+would improve the imbalance to 31.1%, i.e. by 12.5% relative, which exceeds
+the default margin of 10%. However, the absolute improvement of 4.4
+percentage points is below the required minimum of 5 percentage points, so
+the motion must not be issued.
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/cmdlist b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
new file mode 100644
index 0000000..76f0313
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
@@ -0,0 +1,3 @@
+[
+    [ "power node1 on", "power node2 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
new file mode 100644
index 0000000..231af30
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "cpu": 7.44, "mem": 0 },
+    "vm:101": { "cpu": 7.2, "mem": 0 },
+    "vm:200": { "cpu": 6.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/hardware_status b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
@@ -0,0 +1,4 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
new file mode 100644
index 0000000..6c84e42
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -0,0 +1,30 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:100' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:200' on node 'node2'
+info     20    node1/crm: service 'vm:100': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:200': state changed from 'request_start' to 'started'  (node = node2)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:200
+info     23    node2/lrm: service status vm:200 started
+info    620     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/manager_status b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/service_config b/src/test/test-crs-dynamic-auto-rebalance7/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/service_config
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
new file mode 100644
index 0000000..300fb7a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:100": { "maxcpu": 8.0, "maxmem": 8589934592 },
+    "vm:101": { "maxcpu": 8.0, "maxmem": 8589934592 },
+    "vm:200": { "maxcpu": 8.0, "maxmem": 8589934592 }
+}
-- 
2.47.3