* [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn
@ 2026-09-20 16:22 Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Michael Ryom
` (6 more replies)
0 siblings, 7 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel
The automatic load balancer of the dynamic scheduler shows several
problematic behaviors on a small cluster, found while deliberately
stress-testing CRS with ha=dynamic and ha-auto-rebalance on a two-node
PVE 9.2 cluster (pve-ha-manager 5.2.5):
1. A rebalance migration that fails (e.g. local CD-ROM attached) is
re-issued every ~40 seconds, indefinitely - 25 identical failing
attempts in 20 minutes were observed.
2. A single dominant resource oscillates between two nodes: the
relative margin provides almost no hysteresis (a base load
difference of ~0.5 percentage points qualifies a motion and later
its reverse), the point-in-time usage samples are noisy, and the
predicted post-migration state is systematically off because the
balloon-adjusted guest memory value is used instead of the host-side
footprint. Five moves of the same VM in 27 minutes were observed on
an otherwise idle cluster.
3. On an almost idle cluster, the imbalance metric (a coefficient of
variation) is hypersensitive and kept issuing barely-qualifying
migrations with nothing to gain for any workload.
Patches 1-3 address the acute symptoms: track the outcome of motions
issued by the balancer (exponential per-target backoff on failure, a
per-resource cooldown on success), use the host-side memory footprint
('memhost') for the dynamic service stats, and require a minimum
absolute imbalance improvement in addition to the relative margin.
Note that patch 2 overlaps with Dominik Rusovac's pending fix for
bug #7974 ("env: pve2: set dynamic service mem value to host memory
usage", 2026-08-28), which reports the same root cause independently;
happy to drop patch 2 in favor of that one — the rest of the series
applies either way. The oscillation does not fully disappear with
correct memory accounting alone, though: the noise-driven variant
(sub-percent base load fluctuations under a dominant resource) remains
and is what patches 3, 6 and 7 address.
Patches 4-5 extend the simulator (actual running state in the cluster
service stats, like the RRD-based stats on a real cluster; an optional
node base load) so that the following behavior is regression-testable.
Patch 6 decomposes each node's load into the usage of the HA-managed
services running on it plus a residual, and smooths only the residual
with an exponentially moving average kept across scheduling rounds.
Known structural changes take effect immediately; only load that cannot
be attributed to any HA-managed service - which drives the oscillation
but cannot be acted upon - is averaged. No existing expected test log
changes.
Patch 7 changes when the balancer acts at all: only while some node
reports a cpu or memory some-pressure (PSI avg10, already broadcast in
the 9.0 RRD schemas) of at least 10%. Load asymmetry by itself is not a
performance problem; the pressure stall information directly measures
whether anything could run better with more headroom, independent of
cluster size and load level - akin to the contention-driven approach of
other schedulers. Deliberately only node pressure is considered, since
a guest saturating its own vCPUs or memory cannot be helped by a
migration. Fail-open when no pressure telemetry is available.
The series was validated on the live test cluster: failed motions back
off with the expected 60/120s delays, transient spikes and sub-dead-band
asymmetries no longer trigger, real node pressure leads to a single
strongly qualifying evacuation about two minutes after the pressure
appears, and an idle night (imbalance metric standing at ~49% the whole
time) passes with zero balancer actions. The full src/test suite (158
tests, including 5 new regression tests) passes.
A related qemu-server patch (sent separately) handles the case where a
rebalance-issued live migration cannot converge because the guest
dirties memory faster than it can be transferred.
Known remaining gaps, kept out of this series and described in the bug
report (bug #8059): candidate migratability is not checked before
scoring (local CD-ROM etc. - the backoff turns this from an endless
loop into a bounded nuisance), migration cost is not part of the
scoring (the balancer prefers the largest imbalance improvement over
the cheapest adequate motion - see also bug #7650), and a fully
guest-centric scoring model
(per-guest contention score, DRS-style) would be the natural next step
on top of patch 7 - the required per-guest PSI telemetry is already
collected.
Changes are constants for now (backoff/cooldown durations, dead band,
minimum pressure); they could be exposed as ha-auto-rebalance-* options
if preferred.
src/PVE/HA/Env/PVE2.pm | ...
src/PVE/HA/Manager.pm | ...
src/PVE/HA/Sim/Hardware.pm | ...
src/PVE/HA/Usage.pm | ...
src/PVE/HA/Usage/Dynamic.pm | ...
src/test/... | ...
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests Michael Ryom
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer only refrains from issuing new motions while
a migrate/relocate is queued or running. It never records the outcome of
a motion it issued. If the motion fails (e.g. because the guest has a
local CD-ROM image attached), the resource returns to 'started' on its
source node, and the very same motion is selected again as soon as the
hold duration has passed. This results in an endless loop of failing
migrations, in practice about every 40 seconds with the default hold
duration.
Track the last motion issued by load_balance() and, once it is no
longer queued or running, check whether the resource ended up on the
target node:
- on failure, back off that (resource, target) pair with an exponential
delay (60s, 120s, ... capped at 1h) and log a warning;
- on success, put the resource on a cooldown (10 minutes), so that a
resource is not moved back and forth in quick succession, which can
happen when a single resource dominates the cluster load and the node
base loads fluctuate around each other.
Backed-off targets and resources on cooldown are pruned in
get_resource_migration_candidates() before scoring, so the balancer can
still pick other, possibly worse but feasible, migrations.
Like sustained_imbalance_round, this state is not persisted across a
CRM failover. The delays are constants for now; they could be exposed
as ha-auto-rebalance-* options later.
The expected log of test-crs-dynamic-auto-rebalance-topsis3 changes,
because the resource vm:105, which was moved to node2 when its load was
high, is on cooldown when its load has gone down again and is not moved
back immediately anymore; the balancer instead improves the imbalance
by moving vm:106.
Add two regression tests: one for the backoff of failed motions (using
a fa: resource, which fails its first two relocations) and one for the
cooldown suppressing an immediate qualifying reverse migration after a
successful rebalance motion.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 118 ++++++++++++++++++
.../log.expect | 18 +--
.../test-crs-dynamic-auto-rebalance5/README | 13 ++
.../test-crs-dynamic-auto-rebalance5/cmdlist | 3 +
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 6 +
.../hardware_status | 5 +
.../log.expect | 77 ++++++++++++
.../manager_status | 1 +
.../service_config | 6 +
.../static_service_stats | 6 +
.../test-crs-dynamic-auto-rebalance6/README | 17 +++
.../test-crs-dynamic-auto-rebalance6/cmdlist | 7 ++
.../datacenter.cfg | 6 +
.../dynamic_service_stats | 5 +
.../hardware_status | 4 +
.../log.expect | 41 ++++++
.../manager_status | 1 +
.../service_config | 5 +
.../static_service_stats | 5 +
20 files changed, 341 insertions(+), 9 deletions(-)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..30b9ae2 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -72,6 +72,15 @@ sub new {
# this is not persisted for a CRM failover as in the mean time
# the usage statistics might have change quite a bit already
sustained_imbalance_round => 0,
+ # outcome tracking for resource motions issued by load_balance()
+ #
+ # like sustained_imbalance_round, this is not persisted for a CRM
+ # failover
+ auto_rebalance_state => {
+ pending => undef, # last issued motion, until it has finished
+ failures => {}, # "$sid:$target" => { count => $count, until => $time }
+ cooldown => {}, # $sid => $time
+ },
group_migration_round => 3, # wait a little bit
}, $class;
@@ -135,6 +144,92 @@ sub update_crs_scheduler_mode {
return;
}
+# delay before a failed rebalance motion to the same target node is retried,
+# doubled for each consecutive failure up to the maximum
+my $auto_rebalance_failure_backoff_base = 60;
+my $auto_rebalance_failure_backoff_max = 3600;
+# minimum time before a resource is considered for rebalancing again after it
+# was successfully moved by the load balancer
+my $auto_rebalance_resource_cooldown = 600;
+
+my $is_on_rebalance_cooldown = sub {
+ my ($self, $sid, $now) = @_;
+
+ my $cooldown = $self->{auto_rebalance_state}->{cooldown};
+ my $until = $cooldown->{$sid} // return 0;
+
+ if ($now >= $until) {
+ delete $cooldown->{$sid};
+ return 0;
+ }
+
+ return 1;
+};
+
+my $is_rebalance_motion_backed_off = sub {
+ my ($self, $sid, $target, $now) = @_;
+
+ my $failure = $self->{auto_rebalance_state}->{failures}->{"$sid:$target"} // return 0;
+
+ # keep the entry after expiry, so that the failure count keeps growing if
+ # the next attempt fails as well; it is only reset on success
+ return $now < $failure->{until} ? 1 : 0;
+};
+
+# Checks whether the last resource motion issued by load_balance() has
+# finished and records its outcome: a per-resource cooldown on success, or an
+# exponential backoff for the ($sid, $target) pair on failure.
+my $check_last_rebalance_motion = sub {
+ my ($self) = @_;
+
+ my ($haenv, $ss) = $self->@{qw(haenv ss)};
+ my $state = $self->{auto_rebalance_state};
+
+ my $pending = $state->{pending} // return;
+ my ($sid, $target, $task) = $pending->@{qw(sid target task)};
+
+ my $sd = $ss->{$sid};
+ if (!defined($sd)) {
+ # resource was removed in the meantime
+ $state->{pending} = undef;
+ return;
+ }
+
+ my ($cmd, $sstate) = $sd->@{qw(cmd state)};
+
+ # still queued or in progress
+ return if $sstate eq 'migrate' || $sstate eq 'relocate';
+ return if defined($cmd) && ($cmd->[0] eq 'migrate' || $cmd->[0] eq 'relocate');
+
+ $state->{pending} = undef;
+
+ my $now = $haenv->get_time();
+ my $key = "$sid:$target";
+
+ if (defined($sd->{node}) && $sd->{node} eq $target) {
+ delete $state->{failures}->{$key};
+ $state->{cooldown}->{$sid} = $now + $auto_rebalance_resource_cooldown;
+ return;
+ }
+
+ my $failure = $state->{failures}->{$key} //= { count => 0 };
+ $failure->{count}++;
+
+ my $delay = $auto_rebalance_failure_backoff_base * 2**($failure->{count} - 1);
+ $delay = $auto_rebalance_failure_backoff_max
+ if $delay > $auto_rebalance_failure_backoff_max;
+
+ $failure->{until} = $now + $delay;
+
+ $haenv->log(
+ 'warning',
+ "auto rebalance - $task $sid to $target did not succeed"
+ . " (attempt $failure->{count}), not retrying it for ${delay}s",
+ );
+
+ return;
+};
+
# Returns a hash of lists, which contain the running, movable, non-moving HA resource
# bundles, which are on the same node, implied by the strict positive resource
# affinity rules.
@@ -200,9 +295,12 @@ sub get_resource_migration_candidates {
my $resource_bundles =
get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity);
+ my $now = $self->{haenv}->get_time();
my @compact_migration_candidates = ();
for my $leader_sid (sort keys %$resource_bundles) {
+ next if $is_on_rebalance_cooldown->($self, $leader_sid, $now);
+
my $current_leader_node = $ss->{$leader_sid}->{node};
my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
@@ -213,6 +311,11 @@ sub get_resource_migration_candidates {
delete $target_nodes->{$current_leader_node};
+ for my $node (keys %$target_nodes) {
+ delete $target_nodes->{$node}
+ if $is_rebalance_motion_backed_off->($self, $leader_sid, $node, $now);
+ }
+
next if !%$target_nodes;
push @compact_migration_candidates,
@@ -234,6 +337,10 @@ sub load_balance {
return if !$auto_rebalance_opts->{enable};
return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
+
+ # record the outcome of the last motion issued here, if it has finished
+ $check_last_rebalance_motion->($self);
+
return if $self->any_resource_motion_queued_or_running();
my ($threshold, $method, $hold_duration, $margin) =
@@ -295,6 +402,17 @@ sub load_balance {
$haenv->log('info', "auto rebalance - $task $sid to $target ($imbalance_change_str)");
$self->queue_resource_motion($cmd, $task, $sid, $target);
+
+ # queue_resource_motion() may refuse to queue the motion (e.g. blocking
+ # resources on the target node), so only track it if it was queued
+ if (defined($self->{ss}->{$sid}->{cmd})) {
+ $self->{auto_rebalance_state}->{pending} = {
+ sid => $sid,
+ source => $source,
+ target => $target,
+ task => $task,
+ };
+ }
}
sub cleanup {
diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
index 2f19c72..dfbc904 100644
--- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect
@@ -77,13 +77,13 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3'
info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
info 285 node3/lrm: starting service vm:103
info 285 node3/lrm: service status vm:103 started
-info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%)
-info 340 node1/crm: got crm command: migrate vm:105 node1
-info 340 node1/crm: migrate service 'vm:105' to node 'node1'
-info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 343 node2/lrm: service vm:105 - start migrate to node 'node1'
-info 343 node2/lrm: service vm:105 - end migrate to node 'node1'
-info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1)
-info 361 node1/lrm: starting service vm:105
-info 361 node1/lrm: service status vm:105 started
+info 340 node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 30.1% to 20.5%)
+info 340 node1/crm: got crm command: migrate vm:106 node1
+info 340 node1/crm: migrate service 'vm:106' to node 'node1'
+info 340 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 345 node3/lrm: service vm:106 - start migrate to node 'node1'
+info 345 node3/lrm: service vm:106 - end migrate to node 'node1'
+info 360 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node1)
+info 361 node1/lrm: starting service vm:106
+info 361 node1/lrm: service status vm:106 started
info 820 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README
new file mode 100644
index 0000000..c24c44d
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/README
@@ -0,0 +1,13 @@
+Test that the auto rebalance system backs off failed rebalance motions
+instead of retrying them indefinitely.
+
+The HA resource fa:102 fails its first two migrate/relocate attempts. The
+auto rebalancer first selects fa:102 to be relocated to node2, which fails.
+The (fa:102, node2) pair is backed off for 60 seconds, so on the next
+balancing round the rebalancer relocates fa:102 to node3 instead, which
+fails as well and backs off the (fa:102, node3) pair. On the third attempt,
+the backoff for node2 has expired and the relocation to node2 succeeds.
+
+After the successful relocation, fa:102 is put on the per-resource cooldown,
+so the following balancing round moves vm:102 to node3 instead to reach the
+minimum cluster node imbalance.
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
new file mode 100644
index 0000000..13f90cd
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
new file mode 100644
index 0000000..4eba71c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:102": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:102": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:103": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
new file mode 100644
index 0000000..ce8cf0e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
new file mode 100644
index 0000000..1267366
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
@@ -0,0 +1,77 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'fa:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 20 node1/crm: adding new service 'vm:104' on node 'node1'
+info 20 node1/crm: service 'fa:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service fa:102
+info 21 node1/lrm: service status fa:102 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 21 node1/lrm: starting service vm:104
+info 21 node1/lrm: service status vm:104 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 80 node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 80 node1/crm: got crm command: relocate fa:102 node2
+info 80 node1/crm: relocate service 'fa:102' to node 'node2'
+info 80 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 81 node1/lrm: service fa:102 not moved (migration error)
+info 83 node2/lrm: got lock 'ha_agent_node2_lock'
+info 83 node2/lrm: status change wait_for_agent_lock => active
+err 100 node1/crm: service 'fa:102' - migration failed (exit code 1)
+info 100 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node1)
+warn 120 node1/crm: auto rebalance - relocate fa:102 to node2 did not succeed (attempt 1), not retrying it for 60s
+info 160 node1/crm: auto rebalance - relocate fa:102 to node3 (expected change for imbalance from 100.0% to 66.1%)
+info 160 node1/crm: got crm command: relocate fa:102 node3
+info 160 node1/crm: relocate service 'fa:102' to node 'node3'
+info 160 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node3)
+err 161 node1/lrm: service fa:102 not moved (migration error)
+info 165 node3/lrm: got lock 'ha_agent_node3_lock'
+info 165 node3/lrm: status change wait_for_agent_lock => active
+err 180 node1/crm: service 'fa:102' - migration failed (exit code 1)
+info 180 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node1)
+warn 200 node1/crm: auto rebalance - relocate fa:102 to node3 did not succeed (attempt 1), not retrying it for 60s
+info 240 node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 240 node1/crm: got crm command: relocate fa:102 node2
+info 240 node1/crm: relocate service 'fa:102' to node 'node2'
+info 240 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 241 node1/lrm: service fa:102 - start relocate to node 'node2'
+info 241 node1/lrm: stopping service fa:102 (relocate)
+info 241 node1/lrm: service status fa:102 stopped
+info 241 node1/lrm: service fa:102 - end relocate to node 'node2'
+info 260 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node2)
+info 263 node2/lrm: starting service fa:102
+info 263 node2/lrm: service status fa:102 started
+info 320 node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 66.1% to 25.0%)
+info 320 node1/crm: got crm command: migrate vm:102 node3
+info 320 node1/crm: migrate service 'vm:102' to node 'node3'
+info 320 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 321 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 321 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 340 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 345 node3/lrm: starting service vm:102
+info 345 node3/lrm: service status vm:102 started
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config
new file mode 100644
index 0000000..6c4ad4b
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config
@@ -0,0 +1,6 @@
+{
+ "fa:102": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node1", "state": "started" },
+ "vm:104": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
new file mode 100644
index 0000000..41dc446
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README
new file mode 100644
index 0000000..1931f77
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/README
@@ -0,0 +1,17 @@
+Test that a HA resource which was successfully moved by the auto rebalancer
+is put on a cooldown, during which it is not moved again, even if a
+qualifying rebalance motion exists.
+
+The cluster has two nodes, where each node has a HA resource excluded from
+the auto rebalancing (vm:100 on node1 and vm:200 on node2), which act as the
+node base loads, and one movable HA resource vm:101 on node1. Initially,
+node1 is much more loaded than node2, so vm:101 is migrated to node2.
+
+Afterwards, the base loads are swapped, i.e. node2 becomes much more loaded
+than node1. Without the cooldown, the auto rebalancer would immediately
+migrate vm:101 back to node1, as the expected relative imbalance improvement
+qualifies. With the cooldown, vm:101 must not be moved again for 10 minutes,
+so there are no further migrations until the end of the test.
+
+This mimics a single dominant HA resource oscillating between two nodes,
+whose base loads fluctuate around each other.
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
new file mode 100644
index 0000000..1de33e4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
@@ -0,0 +1,7 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [
+ "service vm:100 set-dynamic-stats cpu 0.96 mem 0",
+ "service vm:200 set-dynamic-stats cpu 14.4 mem 0"
+ ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
new file mode 100644
index 0000000..3d2d743
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "cpu": 16.8, "mem": 0 },
+ "vm:101": { "cpu": 4.8, "mem": 0 },
+ "vm:200": { "cpu": 0.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
new file mode 100644
index 0000000..37f5cce
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
@@ -0,0 +1,41 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:200' on node 'node2'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:200': state changed from 'request_start' to 'started' (node = node2)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:200
+info 23 node2/lrm: service status vm:200 started
+info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 91.5% to 48.9%)
+info 80 node1/crm: got crm command: migrate vm:101 node2
+info 80 node1/crm: migrate service 'vm:101' to node 'node2'
+info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 81 node1/lrm: service vm:101 - start migrate to node 'node2'
+info 81 node1/lrm: service vm:101 - end migrate to node 'node2'
+info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 103 node2/lrm: starting service vm:101
+info 103 node2/lrm: service status vm:101 started
+info 120 cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
+info 120 cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
new file mode 100644
index 0000000..f7d1a27
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "maxcpu": 20.0, "maxmem": 8589934592 },
+ "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 },
+ "vm:200": { "maxcpu": 16.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-21 9:18 ` Dominik Rusovac
2026-09-20 16:22 ` [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement Michael Ryom
` (4 subsequent siblings)
6 siblings, 1 reply; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The dynamic service stats used the 'mem' RRD column of the guests, which
is the guest-reported memory usage (total_mem - free_mem) whenever the
balloon driver is active, and can be considerably lower than the actual
host-side footprint of the guest process (QEMU overhead, page cache
still mapped, etc.). The node stats, on the other hand, always contain
the full host view.
The scheduler simulates a migration by subtracting exactly the
service's stats from the source node and adding them to the target
node. With the guest-reported value, the predicted post-migration state
is systematically off by the difference between guest-reported and
host-side usage (over 1 GiB for a mid-size VM). This bias is of the
same order of magnitude as the trigger condition of the automatic load
balancer on a two-node cluster with a single dominant resource, and
contributes to a resource being moved back and forth.
Use the 'memhost' column (host-side cgroup usage, without KSM) instead
and fall back to 'mem' if it is unset: containers do not broadcast
'memhost', because their 'mem' already is the host-side cgroup value.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Env/PVE2.pm | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 782d19d..8c2b03d 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -46,6 +46,7 @@ use constant {
RRD_VM_INDEX_CPU => 6,
RRD_VM_INDEX_MAXMEM => 7,
RRD_VM_INDEX_MEM => 8,
+ RRD_VM_INDEX_MEMHOST => 15,
};
# rrd entry indices for PVE nodes
@@ -598,11 +599,22 @@ sub get_dynamic_service_stats {
# NOTE the guests' broadcasted vmstatus() caps maxcpu at the node's maxcpu
my $maxcpu = ($rrdentry->[RRD_VM_INDEX_MAXCPU] || 0.0) + 0.0;
+ # prefer the host-side memory footprint over 'mem', which is the
+ # guest-reported usage (total_mem - free_mem) when the balloon driver
+ # is active and can be considerably lower than what running the guest
+ # actually consumes on the node; the node stats contain the full host
+ # view, so mixing in the guest view makes the scheduler's prediction
+ # of the post-migration state systematically off
+ #
+ # 'memhost' is only broadcasted for VMs, for CTs 'mem' already is the
+ # host-side (cgroup) usage
+ my $mem = int($rrdentry->[RRD_VM_INDEX_MEMHOST] || $rrdentry->[RRD_VM_INDEX_MEM] || 0);
+
$stats->{$sid}->{usage} = {
maxcpu => $maxcpu,
cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0),
- mem => int($rrdentry->[RRD_VM_INDEX_MEM] || 0),
+ mem => $mem,
};
}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 4/7] sim: hardware: report actual running state in cluster service stats Michael Ryom
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer only requires a relative improvement of the
node imbalance (default 10%). When a single resource dominates the
cluster load, the relative margin provides almost no hysteresis: with
two nodes, a resource load v and a node base load difference d, the
relative improvement of moving the resource is 2d / (v + d), so it
exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
that is a base load difference of about half a percentage point, well
within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
each fluctuation then qualifies a motion (and, after it was carried
out, its reverse).
Additionally require the imbalance improvement to exceed a minimum
absolute value of 5 percentage points. This creates a dead band for
motions that shuffle load without meaningfully improving the cluster
balance. No expected motion of an existing regression test is affected;
the smallest occurring improvement there is 9.6 percentage points.
Like the backoff and cooldown constants, this could be exposed as a
ha-auto-rebalance-* option later.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 8 +++++
.../test-crs-dynamic-auto-rebalance7/README | 12 ++++++++
.../test-crs-dynamic-auto-rebalance7/cmdlist | 3 ++
.../datacenter.cfg | 6 ++++
.../dynamic_service_stats | 5 ++++
.../hardware_status | 4 +++
.../log.expect | 30 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 5 ++++
.../static_service_stats | 5 ++++
10 files changed, 79 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 30b9ae2..2c4c273 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -151,6 +151,13 @@ my $auto_rebalance_failure_backoff_max = 3600;
# minimum time before a resource is considered for rebalancing again after it
# was successfully moved by the load balancer
my $auto_rebalance_resource_cooldown = 600;
+# minimum absolute improvement of the node imbalance for a rebalance motion;
+# the relative margin alone provides almost no hysteresis when a single
+# resource dominates the cluster load: with two nodes, a resource load v and
+# a node base load difference d, the relative improvement of a motion is
+# 2d / (v + d), so for large v tiny fluctuations of d around zero are enough
+# to make the motion (and afterwards its reverse) qualify
+my $auto_rebalance_min_imbalance_improvement = 0.05;
my $is_on_rebalance_cooldown = sub {
my ($self, $sid, $now) = @_;
@@ -387,6 +394,7 @@ sub load_balance {
my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
return if $relative_change < $margin;
+ return if ($imbalance - $target_imbalance) < $auto_rebalance_min_imbalance_improvement;
my ($sid, $source, $target) = $migration->@{qw(sid source-node target-node)};
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
new file mode 100644
index 0000000..488a018
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/README
@@ -0,0 +1,12 @@
+Test that the auto rebalance system does not act on a rebalance motion whose
+absolute improvement of the node imbalance is below the minimum, even though
+the relative improvement exceeds the configured margin.
+
+The cluster has two nodes with a HA resource excluded from auto rebalancing
+on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and
+one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays
+above the default trigger threshold of 30%, and migrating vm:101 to node2
+would improve the imbalance to 31.1%, i.e. by 12.5% relative, which exceeds
+the default margin of 10%. However, the absolute improvement of 4.4
+percentage points is below the required minimum of 5 percentage points, so
+the motion must not be issued.
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/cmdlist b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
new file mode 100644
index 0000000..76f0313
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
new file mode 100644
index 0000000..231af30
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "cpu": 7.44, "mem": 0 },
+ "vm:101": { "cpu": 7.2, "mem": 0 },
+ "vm:200": { "cpu": 6.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/hardware_status b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
new file mode 100644
index 0000000..6c84e42
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -0,0 +1,30 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:200' on node 'node2'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:200': state changed from 'request_start' to 'started' (node = node2)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:200
+info 23 node2/lrm: service status vm:200 started
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/manager_status b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/service_config b/src/test/test-crs-dynamic-auto-rebalance7/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
new file mode 100644
index 0000000..300fb7a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "maxcpu": 8.0, "maxmem": 8589934592 },
+ "vm:101": { "maxcpu": 8.0, "maxmem": 8589934592 },
+ "vm:200": { "maxcpu": 8.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 4/7] sim: hardware: report actual running state in cluster service stats
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
` (2 preceding siblings ...)
2026-09-20 16:22 ` [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 5/7] sim: hardware: allow setting a base load for nodes Michael Ryom
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The simulated cluster service stats reported a service as running
whenever its request state was 'started', even before the LRM actually
started it. On the real cluster, the 'running' flag of the service
stats is derived from the actual RRD status of the guest, and the node
usage stats only contain load of actually running guests.
Report the running state as marked by the LRM in the node service
status instead, so that the service stats and the node usage stats of
the simulated hardware are consistent with each other, like they are on
the real cluster.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Sim/Hardware.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index e1a9ddd..538bc4e 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -1198,6 +1198,9 @@ sub watchdog_update {
my sub get_cluster_service_stats {
my ($self) = @_;
+ my $cstatus = $self->read_hardware_status_nolock();
+ my $node_service_status = { map { $_ => $self->read_service_status($_) } keys %$cstatus };
+
my $stats = {};
for my $sid (keys $self->{service_config}->%*) {
my $cfg = $self->{service_config}->{$sid};
@@ -1205,7 +1208,11 @@ my sub get_cluster_service_stats {
$stats->{$sid} = {
node => $cfg->{node},
state => $cfg->{state},
- running => $cfg->{state} eq 'started',
+ # like the RRD-based stats on the real cluster, report whether
+ # the service is actually running (as marked by the LRM), not
+ # whether its request state is 'started'; the node usage stats
+ # only account for actually running services as well
+ running => $node_service_status->{ $cfg->{node} }->{$sid} ? 1 : 0,
usage => {},
};
}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 5/7] sim: hardware: allow setting a base load for nodes
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
` (3 preceding siblings ...)
2026-09-20 16:22 ` [PATCH ha-manager 4/7] sim: hardware: report actual running state in cluster service stats Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 6/7] usage: dynamic: smooth the unaccounted node load Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure Michael Ryom
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The dynamic node usage stats of the simulated hardware only consisted
of the usage of the running HA-managed services. On a real cluster, the
node usage additionally contains everything else running on the node:
non-HA guests, storage daemons like Ceph or the ZFS ARC, and other host
processes, which fluctuate independently of the HA-managed services.
Read an optional base load ('cpu' and 'mem') for each node from the
hardware status and add a hardware command
node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>]
to change it during a test, so that regression tests can simulate load
that cannot be attributed to any HA-managed service.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Sim/Hardware.pm | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index 538bc4e..aa1a886 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -791,6 +791,7 @@ sub get_cfs_state {
# crm enable-node-maintenance <node>
# crm disable-node-maintenance <node>
# pve-manager-version <node> set <version> # note: this is NOT the *ha*-manager version
+# node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>] # load outside HA services
# reboot <node>
# shutdown <node>
# restart-lrm <node>
@@ -881,6 +882,26 @@ sub sim_hardware_cmd {
$self->write_hardware_status_nolock($cstatus);
+ } elsif ($cmd eq 'node') {
+ die "sim_hardware_cmd: unknown node action '$action'"
+ if $action ne 'set-dynamic-stats';
+ die "sim_hardware_cmd: missing target stat for '$action' command"
+ if !@params;
+
+ my $conversions = { cpu => sub { 0.0 + $_[0] }, mem => sub { $_[0] * 1024**2 } };
+
+ for my ($target, $val) (@params) {
+ die "sim_hardware_cmd: missing value for '$action $target' command"
+ if !defined($val);
+
+ my $convert = $conversions->{$target}
+ or die "sim_hardware_cmd: unknown target stat '$target' for '$action' command";
+
+ $cstatus->{$node}->{$target} = $convert->($val);
+ }
+
+ $self->write_hardware_status_nolock($cstatus);
+
} elsif ($cmd eq 'cfs') {
die "sim_hardware_cmd: unknown cfs action '$action' for node '$node'"
if $action !~ m/^(rw|update)$/;
@@ -1281,18 +1302,21 @@ sub get_static_node_stats {
sub get_dynamic_node_stats {
my ($self) = @_;
+ my $cstatus = $self->read_hardware_status_nolock();
+
my $stats = $self->get_static_node_stats();
for my $node (keys %$stats) {
$stats->{$node}->{maxcpu} = $stats->{$node}->{maxcpu} // $default_node_maxcpu;
- $stats->{$node}->{cpu} = $stats->{$node}->{cpu} // 0.0;
+ # base load of the node outside of the HA-managed services, e.g.
+ # non-HA guests or other host processes
+ $stats->{$node}->{cpu} = $cstatus->{$node}->{cpu} // 0.0;
$stats->{$node}->{maxmem} = $stats->{$node}->{maxmem} // $default_node_maxmem;
- $stats->{$node}->{mem} = $stats->{$node}->{mem} // 0;
+ $stats->{$node}->{mem} = $cstatus->{$node}->{mem} // 0;
}
my $service_conf = $self->read_service_config();
my $dynamic_service_stats = $self->get_dynamic_service_stats();
- my $cstatus = $self->read_hardware_status_nolock();
my $node_service_status = { map { $_ => $self->read_service_status($_) } keys %$cstatus };
for my $sid (keys %$service_conf) {
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 6/7] usage: dynamic: smooth the unaccounted node load
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
` (4 preceding siblings ...)
2026-09-20 16:22 ` [PATCH ha-manager 5/7] sim: hardware: allow setting a base load for nodes Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure Michael Ryom
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The dynamic scheduler acts on point-in-time samples of the node and
guest usage stats. The part of the node load that is not caused by any
HA-managed service (non-HA guests, the ZFS ARC, Ceph daemons, the IO of
a rebalance migration itself, sampling skew between node and guest
stats) is noisy, and acting on a single sample makes the automatic load
balancer prone to oscillation: a single dominant HA resource on a small
cluster qualifies for a migration (and afterwards, for its reverse) as
soon as the node base loads fluctuate around each other by a fraction
of the resource's own load.
Decompose each node's load into the usage of the HA-managed services
actually running on it plus a residual, and smooth only the residual
with an exponentially moving average that the manager keeps across
scheduling rounds (not persisted for a CRM failover, like the other
balancer state).
The service usages themselves are deliberately not smoothed: they are
fully modeled by the scheduler, so known structural changes (a service
was migrated, started or stopped, or its measured load changed) take
effect immediately, transient service load spikes remain handled by the
hold duration, and the expected log of no existing regression test
changes. The residual, on the other hand, cannot be attributed to
anything the balancer could act on, and its expected reaction to a
rebalance motion is zero, so a lagging average is safe to act on.
Add a regression test where the node base loads of a two-node cluster
fluctuate around each other by half a percentage point under a dominant
HA resource. Without the smoothing, every fluctuation qualifies a
migration of the dominant resource to the other node, moving it back
and forth indefinitely (with the per-resource cooldown, once its
cooldown has expired); with the smoothing, the effective base load
difference stays well below the qualification point and no motion is
issued.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 10 ++-
src/PVE/HA/Usage/Dynamic.pm | 73 ++++++++++++++++++-
.../test-crs-dynamic-auto-rebalance8/README | 19 +++++
.../test-crs-dynamic-auto-rebalance8/cmdlist | 19 +++++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 3 +
.../hardware_status | 4 +
.../log.expect | 28 +++++++
.../manager_status | 1 +
.../service_config | 3 +
.../static_service_stats | 3 +
11 files changed, 167 insertions(+), 2 deletions(-)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 2c4c273..6e1827f 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -81,6 +81,10 @@ sub new {
failures => {}, # "$sid:$target" => { count => $count, until => $time }
cooldown => {}, # $sid => $time
},
+ # exponentially moving average of the unaccounted node load, kept
+ # across rounds by PVE::HA::Usage::Dynamic; also not persisted for a
+ # CRM failover
+ dynamic_stats_smoothing_state => {},
group_migration_round => 3, # wait a little bit
}, $class;
@@ -588,7 +592,11 @@ sub recompute_online_node_usage {
if ($have_dynamic_scheduling) {
$online_node_usage = eval {
$service_stats = $haenv->get_dynamic_service_stats();
- my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats);
+ my $scheduler = PVE::HA::Usage::Dynamic->new(
+ $haenv,
+ $service_stats,
+ $self->{dynamic_stats_smoothing_state},
+ );
$scheduler->add_node($_) for $online_nodes->@*;
return $scheduler;
};
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..fc88ddf 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -8,12 +8,83 @@ use PVE::RS::ResourceScheduling::Dynamic;
use base qw(PVE::HA::Usage);
+# weight of the newest sample in the exponentially moving average of the
+# unaccounted node load; with one sample per CRM scheduling round (~10s),
+# 0.25 averages over roughly the last minute
+my $usage_smoothing_alpha = 0.25;
+
+# Smooths the *unaccounted* part of each node's load with an exponentially
+# moving average kept in $state across invocations.
+#
+# The node load is decomposed into the usage of the HA-managed services
+# running on it plus a residual, and only the residual is smoothed. The
+# residual captures the load that cannot be attributed to any HA-managed
+# service (non-HA guests, ZFS ARC, Ceph daemons, the IO of a rebalance
+# migration itself, sampling skew between the node and guest stats). Those
+# point-in-time samples are noisy, and acting on a single sample makes the
+# load balancer prone to oscillation, while their expected reaction to a
+# rebalance motion is zero, so a lagging average is safe to act on.
+#
+# The service usages themselves are deliberately not smoothed: they are
+# fully modeled by the scheduler, i.e. known structural changes (a service
+# was migrated, started or stopped, or its load changed) take effect
+# immediately, and transient service load spikes are already handled by the
+# hold duration of the load balancer.
+my $smooth_unaccounted_node_load = sub {
+ my ($state, $node_stats, $service_stats) = @_;
+
+ # usage per node of the HA-managed services actually running on it
+ my $accounted = {};
+ for my $sid (keys %$service_stats) {
+ my ($node, $running, $usage) = $service_stats->{$sid}->@{qw(node running usage)};
+ next if !$running || !defined($node) || !$usage;
+ next if !defined($usage->{cpu}) || !defined($usage->{mem});
+
+ $accounted->{$node}->{cpu} += $usage->{cpu};
+ $accounted->{$node}->{mem} += $usage->{mem};
+ }
+
+ for my $node (sort keys %$node_stats) {
+ my $stats = $node_stats->{$node};
+ next if !defined($stats->{cpu}) || !defined($stats->{mem});
+
+ # the residual may become negative if the samples disagree, e.g.
+ # right after a service stopped; only the reconstructed node load is
+ # clamped, so that the residual average is not biased upwards
+ my $residual_sample = {
+ cpu => $stats->{cpu} - ($accounted->{$node}->{cpu} // 0.0),
+ mem => $stats->{mem} - ($accounted->{$node}->{mem} // 0),
+ };
+
+ my $residual = $state->{$node};
+ if (!defined($residual)) {
+ $residual = $state->{$node} = $residual_sample;
+ } else {
+ for my $field (qw(cpu mem)) {
+ $residual->{$field} = $usage_smoothing_alpha * $residual_sample->{$field} +
+ (1 - $usage_smoothing_alpha) * $residual->{$field};
+ }
+ }
+
+ my $cpu = $residual->{cpu} + ($accounted->{$node}->{cpu} // 0.0);
+ my $mem = $residual->{mem} + ($accounted->{$node}->{mem} // 0);
+ $stats->{cpu} = $cpu > 0.0 ? $cpu : 0.0;
+ $stats->{mem} = $mem > 0 ? int($mem) : 0;
+ }
+
+ # drop state of meanwhile removed nodes
+ delete $state->{$_} for grep { !$node_stats->{$_} } keys %$state;
+};
+
sub new {
- my ($class, $haenv, $service_stats) = @_;
+ my ($class, $haenv, $service_stats, $smoothing_state) = @_;
my $node_stats = eval { $haenv->get_dynamic_node_stats() };
die "did not get dynamic node usage information - $@" if $@;
+ $smooth_unaccounted_node_load->($smoothing_state, $node_stats, $service_stats)
+ if defined($smoothing_state);
+
my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() };
die "unable to initialize dynamic scheduling - $@" if $@;
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/README b/src/test/test-crs-dynamic-auto-rebalance8/README
new file mode 100644
index 0000000..0264b58
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/README
@@ -0,0 +1,19 @@
+Test that smoothing the unaccounted node load prevents a single dominant HA
+resource from oscillating between two nodes whose base loads (i.e. load not
+caused by any HA-managed service, here set via the node base load of the
+simulated hardware) fluctuate around each other.
+
+The cluster has two nodes with an equal base load and one dominant movable
+HA resource vm:100 on node1. The imbalance is permanently above the trigger
+threshold, but no motion can improve it, so the load balancer stays armed
+without acting. Then the node base loads are repeatedly flipped by about half
+a percentage point of node load in alternating directions.
+
+Each raw sample after a flip would qualify a migration of vm:100 to the less
+loaded node (and after the next flip, back again): the expected relative
+imbalance improvement of about 10.4% exceeds the 10% margin and the absolute
+improvement of about 5.5 percentage points exceeds the required minimum of 5.
+Acting on the raw samples would move the dominant resource back and forth
+indefinitely. With the exponentially moving average over the unaccounted
+node load, the effective base load difference stays well below the point
+where a motion qualifies, so no motion may be issued at any point.
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/cmdlist b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
new file mode 100644
index 0000000..eeee26c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
@@ -0,0 +1,19 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [
+ "node node1 set-dynamic-stats cpu 5.064 mem 0",
+ "node node2 set-dynamic-stats cpu 4.536 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 4.536 mem 0",
+ "node node2 set-dynamic-stats cpu 5.064 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 5.064 mem 0",
+ "node node2 set-dynamic-stats cpu 4.536 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 4.536 mem 0",
+ "node node2 set-dynamic-stats cpu 5.064 mem 0"
+ ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
new file mode 100644
index 0000000..ef14918
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "cpu": 9.6, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/hardware_status b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
new file mode 100644
index 0000000..bb0cf81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
new file mode 100644
index 0000000..73e381a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
@@ -0,0 +1,28 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info 120 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info 220 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info 220 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info 320 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info 320 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info 420 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info 420 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info 1020 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/manager_status b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/service_config b/src/test/test-crs-dynamic-auto-rebalance8/service_config
new file mode 100644
index 0000000..60688cf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
new file mode 100644
index 0000000..7eee778
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
` (5 preceding siblings ...)
2026-09-20 16:22 ` [PATCH ha-manager 6/7] usage: dynamic: smooth the unaccounted node load Michael Ryom
@ 2026-09-20 16:22 ` Michael Ryom
6 siblings, 0 replies; 9+ messages in thread
From: Michael Ryom @ 2026-09-20 16:22 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The load balancer acted on load asymmetry alone. But asymmetric node
loads are not, by themselves, a performance problem: on an almost idle
cluster, the imbalance metric (a coefficient of variation, std/mean)
becomes hypersensitive, and observed live, a barely-qualifying
migration (60.4% -> 54.0% expected imbalance) was issued at 2-7% node
load, with nothing to gain for any workload. The purpose of the load
balancer is better performance for the guests, not symmetric load
figures.
Gate the balancer on the pressure stall information (PSI) that the
nodes already broadcast in their RRD stats: only balance while some
node reports a cpu or memory some-pressure (avg10) of at least 10%,
i.e. work on that node actually loses a meaningful share of its time
waiting on resources and could run better with more headroom. Unlike a
load threshold, this criterion is independent of the cluster's size and
load level, and it directly expresses the question "would anything
perform better elsewhere?" - akin to the contention-driven (rather than
load-driven) approach of other schedulers like VMware DRS.
Deliberately only the node pressure is considered, although the guests'
own pressure stats are collected as well: a guest saturating its own
vCPUs or its own memory reports pressure too, but gets neither more
cores nor more memory by being migrated, while host-caused starvation
(contention with other guests or host processes) shows up in the node
pressure - and only that kind can be improved by a migration.
The kernel's avg10 pressure values are already exponentially averaged
over ten seconds, so no additional smoothing is applied. IO pressure is
deliberately not considered for now, since with shared storage it
usually cannot be improved by a migration. If no pressure information
is available at all (e.g. nodes still broadcasting an older stats
schema), balancing proceeds as before.
The minimum pressure is a constant for now; like the other balancer
constants, it could be exposed as a ha-auto-rebalance-* option later.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Env/PVE2.pm | 25 +++++++++++++
src/PVE/HA/Manager.pm | 16 +++++++++
src/PVE/HA/Sim/Hardware.pm | 16 +++++++--
src/PVE/HA/Usage.pm | 9 +++++
src/PVE/HA/Usage/Dynamic.pm | 23 ++++++++++++
.../test-crs-dynamic-auto-rebalance9/README | 15 ++++++++
.../test-crs-dynamic-auto-rebalance9/cmdlist | 4 +++
.../datacenter.cfg | 6 ++++
.../dynamic_service_stats | 4 +++
.../hardware_status | 4 +++
.../log.expect | 36 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 4 +++
.../static_service_stats | 4 +++
14 files changed, 165 insertions(+), 2 deletions(-)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 8c2b03d..7d7fceb 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -47,6 +47,8 @@ use constant {
RRD_VM_INDEX_MAXMEM => 7,
RRD_VM_INDEX_MEM => 8,
RRD_VM_INDEX_MEMHOST => 15,
+ RRD_VM_INDEX_PRESSURE_CPU_SOME => 16,
+ RRD_VM_INDEX_PRESSURE_MEM_SOME => 20,
};
# rrd entry indices for PVE nodes
@@ -56,8 +58,21 @@ use constant {
RRD_NODE_INDEX_CPU => 5,
RRD_NODE_INDEX_MAXMEM => 7,
RRD_NODE_INDEX_MEM => 8,
+ RRD_NODE_INDEX_PRESSURE_CPU_SOME => 17,
+ RRD_NODE_INDEX_PRESSURE_MEM_SOME => 20,
};
+# returns the numeric value of the given RRD column, or undef if the column
+# is not present (e.g. nodes broadcasting an older schema)
+my sub rrd_column_or_undef {
+ my ($rrdentry, $index) = @_;
+
+ my $value = $rrdentry->[$index];
+ return undef if !defined($value) || $value eq '' || $value eq 'U';
+
+ return $value + 0.0;
+}
+
my $HOSTNAME_RE = qr/(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{,61}?[a-zA-Z0-9])?)/;
sub new {
@@ -615,6 +630,11 @@ sub get_dynamic_service_stats {
cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0),
mem => $mem,
+ # PSI avg10 in percent; undef if the guest broadcasts an older
+ # schema without pressure information
+ pressurecpusome => rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_CPU_SOME),
+ pressurememorysome =>
+ rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_MEM_SOME),
};
}
@@ -660,6 +680,11 @@ sub get_dynamic_node_stats {
cpu => (($rrdentry->[RRD_NODE_INDEX_CPU] || 0.0) + 0.0) * $maxcpu,
maxmem => int($rrdentry->[RRD_NODE_INDEX_MAXMEM] || 0),
mem => int($rrdentry->[RRD_NODE_INDEX_MEM] || 0),
+ # PSI avg10 in percent; undef if the node broadcasts an older
+ # schema without pressure information
+ pressurecpusome => rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_CPU_SOME),
+ pressurememorysome =>
+ rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_MEM_SOME),
};
}
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 6e1827f..c34da89 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -155,6 +155,15 @@ my $auto_rebalance_failure_backoff_max = 3600;
# minimum time before a resource is considered for rebalancing again after it
# was successfully moved by the load balancer
my $auto_rebalance_resource_cooldown = 600;
+# minimum pressure stall value (PSI avg10, in percent) among the nodes for
+# the load balancer to act at all: load asymmetry by itself is not a
+# performance problem, and the node pressure stall information directly
+# measures which share of its time work on the node spends waiting on
+# resources - i.e. whether anything could actually run better with more
+# headroom - independent of the cluster's size or load level; if no pressure
+# information is available at all (older stats schema), balancing proceeds
+# as before
+my $auto_rebalance_min_pressure = 10.0;
# minimum absolute improvement of the node imbalance for a rebalance motion;
# the relative margin alone provides almost no hysteresis when a single
# resource dominates the cluster load: with two nodes, a resource load v and
@@ -368,6 +377,13 @@ sub load_balance {
return;
}
+ # nothing to gain from balancing while no workload is short on resources
+ my $max_pressure = $online_node_usage->max_pressure();
+ if (defined($max_pressure) && $max_pressure < $auto_rebalance_min_pressure) {
+ $self->{sustained_imbalance_round} = 0;
+ return;
+ }
+
my $imbalance = $online_node_usage->calculate_node_imbalance();
# do not load balance unless imbalance threshold has been exceeded
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index aa1a886..8766a6f 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -791,7 +791,8 @@ sub get_cfs_state {
# crm enable-node-maintenance <node>
# crm disable-node-maintenance <node>
# pve-manager-version <node> set <version> # note: this is NOT the *ha*-manager version
-# node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>] # load outside HA services
+# node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>] [pressurecpusome <pct>] [pressurememorysome <pct>]
+# (load and pressure outside of the HA-managed services)
# reboot <node>
# shutdown <node>
# restart-lrm <node>
@@ -888,7 +889,12 @@ sub sim_hardware_cmd {
die "sim_hardware_cmd: missing target stat for '$action' command"
if !@params;
- my $conversions = { cpu => sub { 0.0 + $_[0] }, mem => sub { $_[0] * 1024**2 } };
+ my $conversions = {
+ cpu => sub { 0.0 + $_[0] },
+ mem => sub { $_[0] * 1024**2 },
+ pressurecpusome => sub { 0.0 + $_[0] },
+ pressurememorysome => sub { 0.0 + $_[0] },
+ };
for my ($target, $val) (@params) {
die "sim_hardware_cmd: missing value for '$action $target' command"
@@ -1312,6 +1318,12 @@ sub get_dynamic_node_stats {
$stats->{$node}->{cpu} = $cstatus->{$node}->{cpu} // 0.0;
$stats->{$node}->{maxmem} = $stats->{$node}->{maxmem} // $default_node_maxmem;
$stats->{$node}->{mem} = $cstatus->{$node}->{mem} // 0;
+ # PSI avg10 in percent; keep undef (pressure information not
+ # available) distinct from an explicit zero (no pressure)
+ for my $pressure (qw(pressurecpusome pressurememorysome)) {
+ $stats->{$node}->{$pressure} = $cstatus->{$node}->{$pressure}
+ if defined($cstatus->{$node}->{$pressure});
+ }
}
my $service_conf = $self->read_service_config();
diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
index 659ab30..e0890fb 100644
--- a/src/PVE/HA/Usage.pm
+++ b/src/PVE/HA/Usage.pm
@@ -66,6 +66,15 @@ sub calculate_node_imbalance {
die "implement in subclass";
}
+# Returns the highest pressure stall value (PSI avg10, in percent) among the
+# nodes, or undef if the usage implementation has no notion of pressure
+# (e.g. basic or static) or no node reported pressure information.
+sub max_pressure {
+ my ($self) = @_;
+
+ return undef;
+}
+
sub score_best_balancing_migrations {
my ($self, $migration_candidates, $limit) = @_;
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index fc88ddf..b2ed79a 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -184,6 +184,29 @@ sub calculate_node_imbalance {
return $node_imbalance // 0.0;
}
+# NOTE only the *node* pressure is considered: a guest saturating its own
+# vCPUs or its own memory reports pressure too, but gets neither more cores
+# nor more memory by being migrated, while starvation caused by the host
+# (contention with other guests or host processes) shows up in the node's
+# pressure - and only that kind can be improved by a migration
+sub max_pressure {
+ my ($self) = @_;
+
+ my $max_pressure;
+
+ for my $node ($self->list_nodes()) {
+ my $stats = $self->{'node-stats'}->{$node} or next;
+
+ for my $pressure ($stats->@{qw(pressurecpusome pressurememorysome)}) {
+ next if !defined($pressure);
+ $max_pressure = $pressure
+ if !defined($max_pressure) || $pressure > $max_pressure;
+ }
+ }
+
+ return $max_pressure;
+}
+
sub score_best_balancing_migrations {
my ($self, $migration_candidates, $limit) = @_;
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README
new file mode 100644
index 0000000..d19bd97
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/README
@@ -0,0 +1,15 @@
+Test that the auto rebalance system does not act while no node or HA-managed
+service reports meaningful resource pressure, even though the relative
+imbalance is high and a strongly qualifying rebalance motion exists.
+
+The cluster has two nodes with all load on node1: a HA resource excluded
+from auto rebalancing (vm:101) and a movable HA resource vm:100. The
+imbalance is 100% and migrating vm:100 to the empty node2 would improve it
+far beyond the margin and the minimum absolute improvement. But both nodes
+report zero pressure stall values, i.e. no workload spends any time waiting
+on resources, so there is no performance problem a migration could solve and
+no motion may be issued - regardless of node1's 30% load.
+
+Then node1 reports a CPU pressure of 25%, i.e. some workload now loses a
+quarter of its time waiting for CPU, and the pending migration of vm:100 to
+node2 must be carried out.
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/cmdlist b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
new file mode 100644
index 0000000..3cb6c2a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [ "node node1 set-dynamic-stats pressurecpusome 25" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
new file mode 100644
index 0000000..7705333
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
@@ -0,0 +1,4 @@
+{
+ "vm:100": { "cpu": 9.6, "mem": 0 },
+ "vm:101": { "cpu": 4.8, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/hardware_status b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
new file mode 100644
index 0000000..eeef22a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "pressurecpusome": 0.0, "pressurememorysome": 0.0 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "pressurecpusome": 0.0, "pressurememorysome": 0.0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
new file mode 100644
index 0000000..aed4165
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
@@ -0,0 +1,36 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute node node1 set-dynamic-stats pressurecpusome 25
+info 160 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 100.0% to 33.3%)
+info 160 node1/crm: got crm command: migrate vm:100 node2
+info 160 node1/crm: migrate service 'vm:100' to node 'node2'
+info 160 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 161 node1/lrm: service vm:100 - start migrate to node 'node2'
+info 161 node1/lrm: service vm:100 - end migrate to node 'node2'
+info 163 node2/lrm: got lock 'ha_agent_node2_lock'
+info 163 node2/lrm: status change wait_for_agent_lock => active
+info 180 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2)
+info 183 node2/lrm: starting service vm:100
+info 183 node2/lrm: service status vm:100 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/manager_status b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/service_config b/src/test/test-crs-dynamic-auto-rebalance9/service_config
new file mode 100644
index 0000000..3f12542
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:100": { "node": "node1", "state": "started" },
+ "vm:101": { "node": "node1", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
new file mode 100644
index 0000000..0dfad75
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
@@ -0,0 +1,4 @@
+{
+ "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 },
+ "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests
2026-09-20 16:22 ` [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests Michael Ryom
@ 2026-09-21 9:18 ` Dominik Rusovac
0 siblings, 0 replies; 9+ messages in thread
From: Dominik Rusovac @ 2026-09-21 9:18 UTC (permalink / raw)
To: Michael Ryom, pve-devel
thank you for sending this patch!
one nit inline, otherwise lgtm
NOTE: as mentioned in the cover letter, this overlaps with
https://lore.proxmox.com/all/20260828123239.1110595-2-d.rusovac@proxmox.com/
Reviewed-by: Dominik Rusovac <d.rusovac@proxmox.com>
On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote:
> The dynamic service stats used the 'mem' RRD column of the guests, which
> is the guest-reported memory usage (total_mem - free_mem) whenever the
> balloon driver is active, and can be considerably lower than the actual
> host-side footprint of the guest process (QEMU overhead, page cache
> still mapped, etc.). The node stats, on the other hand, always contain
> the full host view.
>
> The scheduler simulates a migration by subtracting exactly the
> service's stats from the source node and adding them to the target
> node. With the guest-reported value, the predicted post-migration state
> is systematically off by the difference between guest-reported and
> host-side usage (over 1 GiB for a mid-size VM). This bias is of the
> same order of magnitude as the trigger condition of the automatic load
> balancer on a two-node cluster with a single dominant resource, and
> contributes to a resource being moved back and forth.
>
> Use the 'memhost' column (host-side cgroup usage, without KSM) instead
> and fall back to 'mem' if it is unset: containers do not broadcast
> 'memhost', because their 'mem' already is the host-side cgroup value.
>
> Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
> ---
> src/PVE/HA/Env/PVE2.pm | 14 +++++++++++++-
> 1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
> index 782d19d..8c2b03d 100644
> --- a/src/PVE/HA/Env/PVE2.pm
> +++ b/src/PVE/HA/Env/PVE2.pm
> @@ -46,6 +46,7 @@ use constant {
> RRD_VM_INDEX_CPU => 6,
> RRD_VM_INDEX_MAXMEM => 7,
> RRD_VM_INDEX_MEM => 8,
> + RRD_VM_INDEX_MEMHOST => 15,
> };
>
> # rrd entry indices for PVE nodes
> @@ -598,11 +599,22 @@ sub get_dynamic_service_stats {
> # NOTE the guests' broadcasted vmstatus() caps maxcpu at the node's maxcpu
> my $maxcpu = ($rrdentry->[RRD_VM_INDEX_MAXCPU] || 0.0) + 0.0;
>
> + # prefer the host-side memory footprint over 'mem', which is the
> + # guest-reported usage (total_mem - free_mem) when the balloon driver
> + # is active and can be considerably lower than what running the guest
> + # actually consumes on the node; the node stats contain the full host
> + # view, so mixing in the guest view makes the scheduler's prediction
> + # of the post-migration state systematically off
> + #
> + # 'memhost' is only broadcasted for VMs, for CTs 'mem' already is the
> + # host-side (cgroup) usage
> + my $mem = int($rrdentry->[RRD_VM_INDEX_MEMHOST] || $rrdentry->[RRD_VM_INDEX_MEM] || 0);
nit: while this kind of declaration shouldn't cause problems, since it
can be assumed that
$rrdentry->[RRD_VM_INDEX_MEMHOST] >= $rrdentry->[RRD_VM_INDEX_MEM],
technically, we aim to fall back to 'mem' only if 'memhost' is undef and
not if 'memhost' is 0.
[snip]
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-09-21 9:18 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-09-20 16:22 [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 2/7] env: dynamic service stats: use host-side memory footprint of guests Michael Ryom
2026-09-21 9:18 ` Dominik Rusovac
2026-09-20 16:22 ` [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 4/7] sim: hardware: report actual running state in cluster service stats Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 5/7] sim: hardware: allow setting a base load for nodes Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 6/7] usage: dynamic: smooth the unaccounted node load Michael Ryom
2026-09-20 16:22 ` [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure Michael Ryom
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.