[PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load

[Michael Ryom <Michael@RyomHerold.dk>]

The dynamic scheduler acts on point-in-time samples of the node and
guest usage stats. The part of the node load that is not caused by any
HA-managed service (non-HA guests, the ZFS ARC, Ceph daemons, the IO of
a rebalance migration itself, sampling skew between node and guest
stats) is noisy, and acting on a single sample makes the automatic load
balancer prone to oscillation: a single dominant HA resource on a small
cluster qualifies for a migration (and afterwards, for its reverse) as
soon as the node base loads fluctuate around each other by a fraction
of the resource's own load.

Decompose each node's load into the usage of the HA-managed services
actually running on it plus a residual, and smooth only the residual
with an exponentially moving average that the manager keeps across
scheduling rounds (not persisted for a CRM failover, like the other
balancer state).

The service usages themselves are deliberately not smoothed: they are
fully modeled by the scheduler, so known structural changes (a service
was migrated, started or stopped, or its measured load changed) take
effect immediately, transient service load spikes remain handled by the
hold duration, and the expected log of no existing regression test
changes. The residual, on the other hand, cannot be attributed to
anything the balancer could act on, and its expected reaction to a
rebalance motion is zero, so a lagging average is safe to act on.

The expected log of test-crs-dynamic-auto-rebalance9 changes, as the
effective base load difference now stays well below the qualification
point and vm:100 is not migrated back and forth anymore.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/HA/Manager.pm                         | 10 ++-
 src/PVE/HA/Usage/Dynamic.pm                   | 73 ++++++++++++++++++-
 .../test-crs-dynamic-auto-rebalance9/README   |  4 +
 .../log.expect                                | 38 ----------
 4 files changed, 85 insertions(+), 40 deletions(-)

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index b804602..1721fae 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -95,6 +95,10 @@ sub new {
             # load_balance() is not considered for rebalancing
             cooldown => {},
         },
+        # exponentially moving average of the unaccounted node load, kept
+        # across rounds by PVE::HA::Usage::Dynamic; also not persisted for a
+        # CRM failover
+        dynamic_stats_smoothing_state => {},
         group_migration_round => 3, # wait a little bit
     }, $class;
 
@@ -623,7 +627,11 @@ sub recompute_online_node_usage {
             if ($have_dynamic_scheduling) {
                 $online_node_usage = eval {
                     $service_stats = $haenv->get_dynamic_service_stats();
-                    my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats);
+                    my $scheduler = PVE::HA::Usage::Dynamic->new(
+                        $haenv,
+                        $service_stats,
+                        $self->{dynamic_stats_smoothing_state},
+                    );
                     $scheduler->add_node($_) for $online_nodes->@*;
                     return $scheduler;
                 };
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..fc88ddf 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -8,12 +8,83 @@ use PVE::RS::ResourceScheduling::Dynamic;
 
 use base qw(PVE::HA::Usage);
 
+# weight of the newest sample in the exponentially moving average of the
+# unaccounted node load; with one sample per CRM scheduling round (~10s),
+# 0.25 averages over roughly the last minute
+my $usage_smoothing_alpha = 0.25;
+
+# Smooths the *unaccounted* part of each node's load with an exponentially
+# moving average kept in $state across invocations.
+#
+# The node load is decomposed into the usage of the HA-managed services
+# running on it plus a residual, and only the residual is smoothed. The
+# residual captures the load that cannot be attributed to any HA-managed
+# service (non-HA guests, ZFS ARC, Ceph daemons, the IO of a rebalance
+# migration itself, sampling skew between the node and guest stats). Those
+# point-in-time samples are noisy, and acting on a single sample makes the
+# load balancer prone to oscillation, while their expected reaction to a
+# rebalance motion is zero, so a lagging average is safe to act on.
+#
+# The service usages themselves are deliberately not smoothed: they are
+# fully modeled by the scheduler, i.e. known structural changes (a service
+# was migrated, started or stopped, or its load changed) take effect
+# immediately, and transient service load spikes are already handled by the
+# hold duration of the load balancer.
+my $smooth_unaccounted_node_load = sub {
+    my ($state, $node_stats, $service_stats) = @_;
+
+    # usage per node of the HA-managed services actually running on it
+    my $accounted = {};
+    for my $sid (keys %$service_stats) {
+        my ($node, $running, $usage) = $service_stats->{$sid}->@{qw(node running usage)};
+        next if !$running || !defined($node) || !$usage;
+        next if !defined($usage->{cpu}) || !defined($usage->{mem});
+
+        $accounted->{$node}->{cpu} += $usage->{cpu};
+        $accounted->{$node}->{mem} += $usage->{mem};
+    }
+
+    for my $node (sort keys %$node_stats) {
+        my $stats = $node_stats->{$node};
+        next if !defined($stats->{cpu}) || !defined($stats->{mem});
+
+        # the residual may become negative if the samples disagree, e.g.
+        # right after a service stopped; only the reconstructed node load is
+        # clamped, so that the residual average is not biased upwards
+        my $residual_sample = {
+            cpu => $stats->{cpu} - ($accounted->{$node}->{cpu} // 0.0),
+            mem => $stats->{mem} - ($accounted->{$node}->{mem} // 0),
+        };
+
+        my $residual = $state->{$node};
+        if (!defined($residual)) {
+            $residual = $state->{$node} = $residual_sample;
+        } else {
+            for my $field (qw(cpu mem)) {
+                $residual->{$field} = $usage_smoothing_alpha * $residual_sample->{$field} +
+                    (1 - $usage_smoothing_alpha) * $residual->{$field};
+            }
+        }
+
+        my $cpu = $residual->{cpu} + ($accounted->{$node}->{cpu} // 0.0);
+        my $mem = $residual->{mem} + ($accounted->{$node}->{mem} // 0);
+        $stats->{cpu} = $cpu > 0.0 ? $cpu : 0.0;
+        $stats->{mem} = $mem > 0 ? int($mem) : 0;
+    }
+
+    # drop state of meanwhile removed nodes
+    delete $state->{$_} for grep { !$node_stats->{$_} } keys %$state;
+};
+
 sub new {
-    my ($class, $haenv, $service_stats) = @_;
+    my ($class, $haenv, $service_stats, $smoothing_state) = @_;
 
     my $node_stats = eval { $haenv->get_dynamic_node_stats() };
     die "did not get dynamic node usage information - $@" if $@;
 
+    $smooth_unaccounted_node_load->($smoothing_state, $node_stats, $service_stats)
+        if defined($smoothing_state);
+
     my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() };
     die "unable to initialize dynamic scheduling - $@" if $@;
 
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README
index 93b27c4..56bb608 100644
--- a/src/test/test-crs-dynamic-auto-rebalance9/README
+++ b/src/test/test-crs-dynamic-auto-rebalance9/README
@@ -14,3 +14,7 @@ imbalance improvement of about 10.4% exceeds the default margin of 10%.
 Acting on the raw samples moves the dominant resource back and forth with
 every flip, even though the fluctuation is well within the noise of a real
 node's base load.
+
+With the exponentially moving average over the unaccounted node load, the
+effective base load difference stays well below the point where a motion
+qualifies, so no motion may be issued at any point.
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
index efd4d2c..73e381a 100644
--- a/src/test/test-crs-dynamic-auto-rebalance9/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
@@ -19,48 +19,10 @@ info     21    node1/lrm: service status vm:100 started
 info     22    node2/crm: status change wait_for_quorum => slave
 info    120      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
 info    120      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
-info    140    node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
-info    140    node1/crm: got crm command: migrate vm:100 node2
-info    140    node1/crm: migrate service 'vm:100' to node 'node2'
-info    140    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node1, target = node2)
-info    141    node1/lrm: service vm:100 - start migrate to node 'node2'
-info    141    node1/lrm: service vm:100 - end migrate to node 'node2'
-info    143    node2/lrm: got lock 'ha_agent_node2_lock'
-info    143    node2/lrm: status change wait_for_agent_lock => active
-info    160    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node2)
-info    163    node2/lrm: starting service vm:100
-info    163    node2/lrm: service status vm:100 started
 info    220      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
 info    220      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
-info    220    node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
-info    220    node1/crm: got crm command: migrate vm:100 node1
-info    220    node1/crm: migrate service 'vm:100' to node 'node1'
-info    220    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node2, target = node1)
-info    223    node2/lrm: service vm:100 - start migrate to node 'node1'
-info    223    node2/lrm: service vm:100 - end migrate to node 'node1'
-info    240    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node1)
-info    241    node1/lrm: starting service vm:100
-info    241    node1/lrm: service status vm:100 started
 info    320      cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
 info    320      cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
-info    360    node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
-info    360    node1/crm: got crm command: migrate vm:100 node2
-info    360    node1/crm: migrate service 'vm:100' to node 'node2'
-info    360    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node1, target = node2)
-info    361    node1/lrm: service vm:100 - start migrate to node 'node2'
-info    361    node1/lrm: service vm:100 - end migrate to node 'node2'
-info    380    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node2)
-info    383    node2/lrm: starting service vm:100
-info    383    node2/lrm: service status vm:100 started
 info    420      cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
 info    420      cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
-info    440    node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
-info    440    node1/crm: got crm command: migrate vm:100 node1
-info    440    node1/crm: migrate service 'vm:100' to node 'node1'
-info    440    node1/crm: service 'vm:100': state changed from 'started' to 'migrate'  (node = node2, target = node1)
-info    443    node2/lrm: service vm:100 - start migrate to node 'node1'
-info    443    node2/lrm: service vm:100 - end migrate to node 'node1'
-info    460    node1/crm: service 'vm:100': state changed from 'migrate' to 'started'  (node = node1)
-info    461    node1/lrm: starting service vm:100
-info    461    node1/lrm: service status vm:100 started
 info   1020     hardware: exit simulation - done
-- 
2.47.3