[PATCH ha-manager 6/7] usage: dynamic: smooth the unaccounted node load

[Michael Ryom <Michael@RyomHerold.dk>]

The dynamic scheduler acts on point-in-time samples of the node and
guest usage stats. The part of the node load that is not caused by any
HA-managed service (non-HA guests, the ZFS ARC, Ceph daemons, the IO of
a rebalance migration itself, sampling skew between node and guest
stats) is noisy, and acting on a single sample makes the automatic load
balancer prone to oscillation: a single dominant HA resource on a small
cluster qualifies for a migration (and afterwards, for its reverse) as
soon as the node base loads fluctuate around each other by a fraction
of the resource's own load.

Decompose each node's load into the usage of the HA-managed services
actually running on it plus a residual, and smooth only the residual
with an exponentially moving average that the manager keeps across
scheduling rounds (not persisted for a CRM failover, like the other
balancer state).

The service usages themselves are deliberately not smoothed: they are
fully modeled by the scheduler, so known structural changes (a service
was migrated, started or stopped, or its measured load changed) take
effect immediately, transient service load spikes remain handled by the
hold duration, and the expected log of no existing regression test
changes. The residual, on the other hand, cannot be attributed to
anything the balancer could act on, and its expected reaction to a
rebalance motion is zero, so a lagging average is safe to act on.

Add a regression test where the node base loads of a two-node cluster
fluctuate around each other by half a percentage point under a dominant
HA resource. Without the smoothing, every fluctuation qualifies a
migration of the dominant resource to the other node, moving it back
and forth indefinitely (with the per-resource cooldown, once its
cooldown has expired); with the smoothing, the effective base load
difference stays well below the qualification point and no motion is
issued.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                         | 10 ++-
 src/PVE/HA/Usage/Dynamic.pm                   | 73 ++++++++++++++++++-
 .../test-crs-dynamic-auto-rebalance8/README   | 19 +++++
 .../test-crs-dynamic-auto-rebalance8/cmdlist  | 19 +++++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  3 +
 .../hardware_status                           |  4 +
 .../log.expect                                | 28 +++++++
 .../manager_status                            |  1 +
 .../service_config                            |  3 +
 .../static_service_stats                      |  3 +
 11 files changed, 167 insertions(+), 2 deletions(-)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/static_service_stats

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 2c4c273..6e1827f 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -81,6 +81,10 @@ sub new {
             failures => {}, # "$sid:$target" => { count => $count, until => $time }
             cooldown => {}, # $sid => $time
         },
+        # exponentially moving average of the unaccounted node load, kept
+        # across rounds by PVE::HA::Usage::Dynamic; also not persisted for a
+        # CRM failover
+        dynamic_stats_smoothing_state => {},
         group_migration_round => 3, # wait a little bit
     }, $class;
 
@@ -588,7 +592,11 @@ sub recompute_online_node_usage {
             if ($have_dynamic_scheduling) {
                 $online_node_usage = eval {
                     $service_stats = $haenv->get_dynamic_service_stats();
-                    my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats);
+                    my $scheduler = PVE::HA::Usage::Dynamic->new(
+                        $haenv,
+                        $service_stats,
+                        $self->{dynamic_stats_smoothing_state},
+                    );
                     $scheduler->add_node($_) for $online_nodes->@*;
                     return $scheduler;
                 };
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..fc88ddf 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -8,12 +8,83 @@ use PVE::RS::ResourceScheduling::Dynamic;
 
 use base qw(PVE::HA::Usage);
 
+# weight of the newest sample in the exponentially moving average of the
+# unaccounted node load; with one sample per CRM scheduling round (~10s),
+# 0.25 averages over roughly the last minute
+my $usage_smoothing_alpha = 0.25;
+
+# Smooths the *unaccounted* part of each node's load with an exponentially
+# moving average kept in $state across invocations.
+#
+# The node load is decomposed into the usage of the HA-managed services
+# running on it plus a residual, and only the residual is smoothed. The
+# residual captures the load that cannot be attributed to any HA-managed
+# service (non-HA guests, ZFS ARC, Ceph daemons, the IO of a rebalance
+# migration itself, sampling skew between the node and guest stats). Those
+# point-in-time samples are noisy, and acting on a single sample makes the
+# load balancer prone to oscillation, while their expected reaction to a
+# rebalance motion is zero, so a lagging average is safe to act on.
+#
+# The service usages themselves are deliberately not smoothed: they are
+# fully modeled by the scheduler, i.e. known structural changes (a service
+# was migrated, started or stopped, or its load changed) take effect
+# immediately, and transient service load spikes are already handled by the
+# hold duration of the load balancer.
+my $smooth_unaccounted_node_load = sub {
+    my ($state, $node_stats, $service_stats) = @_;
+
+    # usage per node of the HA-managed services actually running on it
+    my $accounted = {};
+    for my $sid (keys %$service_stats) {
+        my ($node, $running, $usage) = $service_stats->{$sid}->@{qw(node running usage)};
+        next if !$running || !defined($node) || !$usage;
+        next if !defined($usage->{cpu}) || !defined($usage->{mem});
+
+        $accounted->{$node}->{cpu} += $usage->{cpu};
+        $accounted->{$node}->{mem} += $usage->{mem};
+    }
+
+    for my $node (sort keys %$node_stats) {
+        my $stats = $node_stats->{$node};
+        next if !defined($stats->{cpu}) || !defined($stats->{mem});
+
+        # the residual may become negative if the samples disagree, e.g.
+        # right after a service stopped; only the reconstructed node load is
+        # clamped, so that the residual average is not biased upwards
+        my $residual_sample = {
+            cpu => $stats->{cpu} - ($accounted->{$node}->{cpu} // 0.0),
+            mem => $stats->{mem} - ($accounted->{$node}->{mem} // 0),
+        };
+
+        my $residual = $state->{$node};
+        if (!defined($residual)) {
+            $residual = $state->{$node} = $residual_sample;
+        } else {
+            for my $field (qw(cpu mem)) {
+                $residual->{$field} = $usage_smoothing_alpha * $residual_sample->{$field} +
+                    (1 - $usage_smoothing_alpha) * $residual->{$field};
+            }
+        }
+
+        my $cpu = $residual->{cpu} + ($accounted->{$node}->{cpu} // 0.0);
+        my $mem = $residual->{mem} + ($accounted->{$node}->{mem} // 0);
+        $stats->{cpu} = $cpu > 0.0 ? $cpu : 0.0;
+        $stats->{mem} = $mem > 0 ? int($mem) : 0;
+    }
+
+    # drop state of meanwhile removed nodes
+    delete $state->{$_} for grep { !$node_stats->{$_} } keys %$state;
+};
+
 sub new {
-    my ($class, $haenv, $service_stats) = @_;
+    my ($class, $haenv, $service_stats, $smoothing_state) = @_;
 
     my $node_stats = eval { $haenv->get_dynamic_node_stats() };
     die "did not get dynamic node usage information - $@" if $@;
 
+    $smooth_unaccounted_node_load->($smoothing_state, $node_stats, $service_stats)
+        if defined($smoothing_state);
+
     my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() };
     die "unable to initialize dynamic scheduling - $@" if $@;
 
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/README b/src/test/test-crs-dynamic-auto-rebalance8/README
new file mode 100644
index 0000000..0264b58
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/README
@@ -0,0 +1,19 @@
+Test that smoothing the unaccounted node load prevents a single dominant HA
+resource from oscillating between two nodes whose base loads (i.e. load not
+caused by any HA-managed service, here set via the node base load of the
+simulated hardware) fluctuate around each other.
+
+The cluster has two nodes with an equal base load and one dominant movable
+HA resource vm:100 on node1. The imbalance is permanently above the trigger
+threshold, but no motion can improve it, so the load balancer stays armed
+without acting. Then the node base loads are repeatedly flipped by about half
+a percentage point of node load in alternating directions.
+
+Each raw sample after a flip would qualify a migration of vm:100 to the less
+loaded node (and after the next flip, back again): the expected relative
+imbalance improvement of about 10.4% exceeds the 10% margin and the absolute
+improvement of about 5.5 percentage points exceeds the required minimum of 5.
+Acting on the raw samples would move the dominant resource back and forth
+indefinitely. With the exponentially moving average over the unaccounted
+node load, the effective base load difference stays well below the point
+where a motion qualifies, so no motion may be issued at any point.
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/cmdlist b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
new file mode 100644
index 0000000..eeee26c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
@@ -0,0 +1,19 @@
+[
+    [ "power node1 on", "power node2 on" ],
+    [
+        "node node1 set-dynamic-stats cpu 5.064 mem 0",
+        "node node2 set-dynamic-stats cpu 4.536 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 4.536 mem 0",
+        "node node2 set-dynamic-stats cpu 5.064 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 5.064 mem 0",
+        "node node2 set-dynamic-stats cpu 4.536 mem 0"
+    ],
+    [
+        "node node1 set-dynamic-stats cpu 4.536 mem 0",
+        "node node2 set-dynamic-stats cpu 5.064 mem 0"
+    ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
new file mode 100644
index 0000000..ef14918
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "cpu": 9.6, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/hardware_status b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
new file mode 100644
index 0000000..bb0cf81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
@@ -0,0 +1,4 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
new file mode 100644
index 0000000..73e381a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
@@ -0,0 +1,28 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:100' on node 'node1'
+info     20    node1/crm: service 'vm:100': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:100
+info     21    node1/lrm: service status vm:100 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info    120      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info    220      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info    220      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info    320      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info    320      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info    420      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info    420      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info   1020     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/manager_status b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/service_config b/src/test/test-crs-dynamic-auto-rebalance8/service_config
new file mode 100644
index 0000000..60688cf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/service_config
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
new file mode 100644
index 0000000..7eee778
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
@@ -0,0 +1,3 @@
+{
+    "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 }
+}
-- 
2.47.3