* [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation
@ 2026-10-18 12:45 Michael Ryom
2026-10-18 12:45 ` [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown Michael Ryom
` (13 more replies)
0 siblings, 14 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer shows two problematic behaviors on a small
cluster, found while deliberately stress-testing CRS with ha=dynamic and
ha-auto-rebalance on a two-node PVE 9.2 cluster, see bug #8059:
1. A rebalance migration that fails (e.g. local CD-ROM attached) is
re-issued every ~40 seconds, indefinitely - 25 identical failing
attempts in 20 minutes were observed.
2. A single dominant resource oscillates between two nodes: the
relative margin provides almost no hysteresis (a base load
difference of ~0.5 percentage points qualifies a motion and later
its reverse), and the point-in-time usage samples of the load that
is not caused by HA resources are noisy. Five moves of the same VM
in 27 minutes were observed on an otherwise idle cluster.
Changes since v1
----------------
Thanks to Dominik and Daniel for the reviews!
v1: https://lore.proxmox.com/pve-devel/20260920162220.574802-1-Michael@RyomHerold.dk/
* Failed rebalance motions (patches 3-4) are now handled per
rebalancing attempt instead of with a time-based backoff, as
suggested by Daniel: a failed motion is never retried in the same
attempt, so the next best motion is tried instead; it is considered
again in the next attempt, and if it keeps failing, it is skipped in
exponentially more of the following attempts. With the time-based
backoff of v1, two failing motions of the same resource to different
nodes could take turns before the next best motion was ever tried -
the new test-crs-dynamic-auto-rebalance5 reproduces exactly the
fa:109 example from the review.
* Failures are recorded in next_state_migrate_relocate() when the LRM
reports them, for every resource moved by the balancer, i.e. also for
the dependent resources of a resource bundle. A change of the
bundle's leader no longer resets the failure state.
* The cooldown after a successful motion is a separate patch (5-6) and
an option (ha-auto-rebalance-cooldown), which defaults to 0, i.e. it
is disabled.
* The minimum absolute improvement is no longer a separate threshold,
but a mode of the existing margin (ha-auto-rebalance-margin-mode,
relative or absolute), defaulting to relative (patches 7-8).
* The pressure-gating patch (7/7 of v1) is dropped. I agree with
Daniel's points: PSI can tell that a node has a problem, but not what
should be moved where, it reacts late, cpu and memory pressure need
different standards, and it should not depend on whether a node
broadcasts the information. I'll follow up separately with a
proposal that addresses the "what and where" with a per-guest model
instead of a node-local gate.
* Each behavior change is preceded by a patch which adds its test case
with the current behavior, so the change is visible in the expected
log of the changing patch (patches 4, 6, 8 and 12).
* Nits: is_resource_motion_queued_or_running() factored out (patch 2),
`my sub` instead of `my $name = sub`, no combined `//=` with an
increment.
* The memhost patch of v1 is dropped, as the same fix has landed as
commit 28c31e4 ("fix #7974: env: pve2: set dynamic service mem value
to host memory usage"). The series is rebased on top of it.
The series
----------
Patch 1 adds the schema of the two new options to pve-cluster.
pve-ha-manager falls back to the defaults when the options are absent,
so it also works with an older libpve-cluster-perl.
Patch 2 is a preparatory refactoring.
Patches 3-4 fix the retry loop of failed rebalance motions (bug #8059).
Patches 5-6 add an optional cooldown for resources moved by the
balancer, which suppresses an immediate reverse motion.
Patches 7-8 add the absolute margin mode, which creates a dead band for
motions that only shuffle load without meaningfully improving the
cluster balance.
Patches 9-10 extend the simulator (actual running state in the cluster
service stats, like the RRD-based stats on a real cluster; an optional
node base load), so that the following patches are regression-testable.
Patches 11-12 decompose each node's load into the usage of the
HA-managed services running on it plus a residual, and smooth only the
residual with an exponentially moving average kept across scheduling
rounds. Known structural changes (a service was migrated, started or
stopped) take effect immediately; only load that cannot be attributed
to any HA-managed service - which drives the oscillation but cannot be
acted upon - is averaged. This is unchanged from v1 and is the only
behavior change of the series that is not opt-in; no expected log of an
existing test case changes.
Patch 13 adds the matching UI fields to pve-manager, and patch 14
documents the new options and the handling of failed migrations,
together with an overview table of all load balancer options.
Testing
-------
The full src/test suite (158 test cases, including 5 new ones) passes,
both against the pve-rs bindings and in a build chroot, and each
test-adding patch was run on its own to record the behavior before the
change. The residual smoothing (patches 11-12) ran on the reporting
cluster for several weeks as part of v1. The new attempt-based failure
handling and the margin mode are tested in the simulator only so far.
Known remaining gaps, kept out of this series and described in bug
#8059: candidate migratability is not checked before scoring (Daniel
mentioned he is working on inferring such constraints from the guest
configuration - I have a draft for the same that I'm happy to drop or
contribute to his series), and migration cost is not part of the
scoring (see also bug #7650).
pve-ha-manager:
src/PVE/HA/Manager.pm | 201 ++++++-
src/PVE/HA/Sim/Hardware.pm | 39 +-
src/PVE/HA/Usage/Dynamic.pm | 73 ++-
src/test/... | 503 ++++++++ (45 files, 5 new tests)
pve-cluster:
src/PVE/DataCenterConfig.pm | 28 +-
pve-manager:
www/manager6/form/CRSOptions.js | 35 +-
pve-docs:
ha-manager.adoc | 62 +
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 02/14] manager: factor out checking whether a resource motion is queued or running Michael Ryom
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The relative improvement margin of the automatic resource balancing
provides almost no hysteresis when a single resource dominates the
cluster load: with two nodes, a resource load v and a node base load
difference d, the relative improvement of moving the resource is
2d / (v + d), so background fluctuations of the node base loads are
enough to qualify a migration and, afterwards, its reverse (see bug
#8059).
Add a margin mode, which allows to interpret the existing margin as a
minimum absolute improvement in percentage points instead, which creates
a dead band for such migrations. It defaults to 'relative', i.e. the
behavior of existing setups does not change.
Also add a cooldown, which keeps resources moved by a resource
balancing migration from being considered for another one for the given
time. It defaults to 0, i.e. it is disabled.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/DataCenterConfig.pm | 28 ++++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
diff --git a/src/PVE/DataCenterConfig.pm b/src/PVE/DataCenterConfig.pm
index 004122e..e637e63 100644
--- a/src/PVE/DataCenterConfig.pm
+++ b/src/PVE/DataCenterConfig.pm
@@ -75,8 +75,32 @@ EODESC
maximum => 100,
default => 10,
requires => 'ha-auto-rebalance',
- description => "The minimum relative improvement in cluster node imbalance,"
- . " in percent, to commit to a resource balancing migration.",
+ description => "The minimum improvement in cluster node imbalance to commit to a"
+ . " resource balancing migration. Depending on the margin mode, either in"
+ . " percent of the current imbalance (relative) or in percentage points"
+ . " (absolute).",
+ },
+ 'ha-auto-rebalance-margin-mode' => {
+ type => 'string',
+ enum => ['relative', 'absolute'],
+ optional => 1,
+ default => 'relative',
+ requires => 'ha-auto-rebalance',
+ description => "Whether the margin is the minimum relative or absolute improvement"
+ . " in cluster node imbalance. An absolute margin prevents migrations which"
+ . " only shuffle load without meaningfully improving the cluster balance, e.g."
+ . " a single dominating resource being moved back and forth because of"
+ . " fluctuations of the node base loads.",
+ },
+ 'ha-auto-rebalance-cooldown' => {
+ type => 'integer',
+ optional => 1,
+ minimum => 0,
+ default => 0,
+ requires => 'ha-auto-rebalance',
+ description => "The time, in seconds, after a successful resource balancing"
+ . " migration in which the moved resources are not considered for another"
+ . " resource balancing migration. Set to 0 to disable the cooldown.",
},
};
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 02/14] manager: factor out checking whether a resource motion is queued or running
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
2026-10-18 12:45 ` [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions Michael Ryom
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The check is needed for single resources in the following patches.
No functional change intended.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76..e05f0a4 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -651,16 +651,22 @@ sub queue_resource_motion {
}
}
-sub any_resource_motion_queued_or_running {
- my ($self) = @_;
+sub is_resource_motion_queued_or_running {
+ my ($self, $sid) = @_;
+
+ my ($cmd, $state) = $self->{ss}->{$sid}->@{qw(cmd state)};
+
+ return 1 if $state eq 'migrate' || $state eq 'relocate';
+ return 1 if defined($cmd) && ($cmd->[0] eq 'migrate' || $cmd->[0] eq 'relocate');
- my ($ss) = $self->@{qw(ss)};
+ return 0;
+}
- for my $sid (keys %$ss) {
- my ($cmd, $state) = $ss->{$sid}->@{qw(cmd state)};
+sub any_resource_motion_queued_or_running {
+ my ($self) = @_;
- return 1 if $state eq 'migrate' || $state eq 'relocate';
- return 1 if defined($cmd) && ($cmd->[0] eq 'migrate' || $cmd->[0] eq 'relocate');
+ for my $sid (keys $self->{ss}->%*) {
+ return 1 if $self->is_resource_motion_queued_or_running($sid);
}
return 0;
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
2026-10-18 12:45 ` [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 02/14] manager: factor out checking whether a resource motion is queued or running Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 04/14] fix #8059: manager: auto rebalance: do not retry failed rebalance motions right away Michael Ryom
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer does not record the outcome of the motions
it issues. If a motion fails, the resource is put back to 'started' on
its source node and the very same motion is selected again as soon as
the imbalance has exceeded the threshold for the hold duration again
(see bug #8059).
Add two test cases which show this behavior:
- test-crs-dynamic-auto-rebalance5: a failing relocation of fa:109 is
retried every 80 seconds, even though migrating any of the other HA
resources would improve the imbalance just as well.
- test-crs-dynamic-auto-rebalance6: the failing relocation of fa:109 is
the only possible rebalance motion and is retried every 80 seconds as
well.
The expected logs record the current behavior and are adapted by the
following patch.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
.../test-crs-dynamic-auto-rebalance5/README | 9 ++
.../test-crs-dynamic-auto-rebalance5/cmdlist | 3 +
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 6 ++
.../hardware_status | 5 +
.../log.expect | 88 +++++++++++++++
.../manager_status | 1 +
.../service_config | 6 ++
.../static_service_stats | 6 ++
.../test-crs-dynamic-auto-rebalance6/README | 10 ++
.../test-crs-dynamic-auto-rebalance6/cmdlist | 17 +++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 4 +
.../hardware_status | 4 +
.../log.expect | 100 ++++++++++++++++++
.../manager_status | 1 +
.../service_config | 4 +
.../static_service_stats | 4 +
18 files changed, 280 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README
new file mode 100644
index 0000000..018e604
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/README
@@ -0,0 +1,9 @@
+Test that the auto rebalance system does not retry a failing rebalance motion
+over and over again.
+
+The HA resource fa:109 fails its first nine migrate/relocate attempts. The
+auto rebalancer selects fa:109 to be relocated, which fails, and fa:109 is put
+back to 'started' on node1. As long as the imbalance stays above the threshold,
+the auto rebalancer should neither select the very same failed motion again
+nor keep alternating between failing motions of fa:109 to the other nodes, but
+try the next best rebalance motion, i.e. migrate one of the other HA resources.
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
new file mode 100644
index 0000000..13f90cd
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
new file mode 100644
index 0000000..921df6a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:102": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:103": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
new file mode 100644
index 0000000..ce8cf0e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
new file mode 100644
index 0000000..a5dad5a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
@@ -0,0 +1,88 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'fa:109' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 20 node1/crm: adding new service 'vm:104' on node 'node1'
+info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service fa:109
+info 21 node1/lrm: service status fa:109 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 21 node1/lrm: starting service vm:104
+info 21 node1/lrm: service status vm:104 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 80 node1/crm: got crm command: relocate fa:109 node2
+info 80 node1/crm: relocate service 'fa:109' to node 'node2'
+info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 81 node1/lrm: service fa:109 not moved (migration error)
+info 83 node2/lrm: got lock 'ha_agent_node2_lock'
+info 83 node2/lrm: status change wait_for_agent_lock => active
+err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 160 node1/crm: got crm command: relocate fa:109 node2
+info 160 node1/crm: relocate service 'fa:109' to node 'node2'
+info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 161 node1/lrm: service fa:109 not moved (migration error)
+err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 240 node1/crm: got crm command: relocate fa:109 node2
+info 240 node1/crm: relocate service 'fa:109' to node 'node2'
+info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 241 node1/lrm: service fa:109 not moved (migration error)
+err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 320 node1/crm: got crm command: relocate fa:109 node2
+info 320 node1/crm: relocate service 'fa:109' to node 'node2'
+info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 321 node1/lrm: service fa:109 not moved (migration error)
+err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 400 node1/crm: got crm command: relocate fa:109 node2
+info 400 node1/crm: relocate service 'fa:109' to node 'node2'
+info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 401 node1/lrm: service fa:109 not moved (migration error)
+err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 480 node1/crm: got crm command: relocate fa:109 node2
+info 480 node1/crm: relocate service 'fa:109' to node 'node2'
+info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 481 node1/lrm: service fa:109 not moved (migration error)
+err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 560 node1/crm: got crm command: relocate fa:109 node2
+info 560 node1/crm: relocate service 'fa:109' to node 'node2'
+info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 561 node1/lrm: service fa:109 not moved (migration error)
+err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config
new file mode 100644
index 0000000..3685b14
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node1", "state": "started" },
+ "vm:104": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
new file mode 100644
index 0000000..6dab3d1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README
new file mode 100644
index 0000000..76e9411
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/README
@@ -0,0 +1,10 @@
+Test that the auto rebalance system retries a rebalance motion, which failed
+before, less and less often if it keeps failing and there is no other
+rebalance motion to fall back to.
+
+The HA resource vm:100 is excluded from auto rebalancing and only acts as a
+node base load on node1. The only movable HA resource fa:109 fails its first
+nine migrate/relocate attempts, so the auto rebalancer can only ever select
+the failing relocation of fa:109 to node2. The time between two attempts should
+grow with each failure instead of retrying it every time the imbalance has
+exceeded the threshold for the hold duration.
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
new file mode 100644
index 0000000..c8c644e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
@@ -0,0 +1,17 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ []
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
new file mode 100644
index 0000000..77ef5c2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:100": { "cpu": 4.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
new file mode 100644
index 0000000..9758af5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
@@ -0,0 +1,100 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'fa:109' on node 'node1'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service fa:109
+info 21 node1/lrm: service status fa:109 started
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 80 node1/crm: got crm command: relocate fa:109 node2
+info 80 node1/crm: relocate service 'fa:109' to node 'node2'
+info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 81 node1/lrm: service fa:109 not moved (migration error)
+info 83 node2/lrm: got lock 'ha_agent_node2_lock'
+info 83 node2/lrm: status change wait_for_agent_lock => active
+err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 160 node1/crm: got crm command: relocate fa:109 node2
+info 160 node1/crm: relocate service 'fa:109' to node 'node2'
+info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 161 node1/lrm: service fa:109 not moved (migration error)
+err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 240 node1/crm: got crm command: relocate fa:109 node2
+info 240 node1/crm: relocate service 'fa:109' to node 'node2'
+info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 241 node1/lrm: service fa:109 not moved (migration error)
+err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 320 node1/crm: got crm command: relocate fa:109 node2
+info 320 node1/crm: relocate service 'fa:109' to node 'node2'
+info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 321 node1/lrm: service fa:109 not moved (migration error)
+err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 400 node1/crm: got crm command: relocate fa:109 node2
+info 400 node1/crm: relocate service 'fa:109' to node 'node2'
+info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 401 node1/lrm: service fa:109 not moved (migration error)
+err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 480 node1/crm: got crm command: relocate fa:109 node2
+info 480 node1/crm: relocate service 'fa:109' to node 'node2'
+info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 481 node1/lrm: service fa:109 not moved (migration error)
+err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 560 node1/crm: got crm command: relocate fa:109 node2
+info 560 node1/crm: relocate service 'fa:109' to node 'node2'
+info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 561 node1/lrm: service fa:109 not moved (migration error)
+err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 640 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 640 node1/crm: got crm command: relocate fa:109 node2
+info 640 node1/crm: relocate service 'fa:109' to node 'node2'
+info 640 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 641 node1/lrm: service fa:109 not moved (migration error)
+err 660 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 660 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 720 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 720 node1/crm: got crm command: relocate fa:109 node2
+info 720 node1/crm: relocate service 'fa:109' to node 'node2'
+info 720 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 721 node1/lrm: service fa:109 not moved (migration error)
+err 740 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 740 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 800 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 800 node1/crm: got crm command: relocate fa:109 node2
+info 800 node1/crm: relocate service 'fa:109' to node 'node2'
+info 800 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 801 node1/lrm: service fa:109 - start relocate to node 'node2'
+info 801 node1/lrm: stopping service fa:109 (relocate)
+info 801 node1/lrm: service status fa:109 stopped
+info 801 node1/lrm: service fa:109 - end relocate to node 'node2'
+info 820 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node2)
+info 823 node2/lrm: starting service fa:109
+info 823 node2/lrm: service status fa:109 started
+info 1520 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config
new file mode 100644
index 0000000..ccffeab
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "node": "node1", "state": "started" },
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
new file mode 100644
index 0000000..7d2c248
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:100": { "maxcpu": 4.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 04/14] fix #8059: manager: auto rebalance: do not retry failed rebalance motions right away
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (2 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion Michael Ryom
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer only refrains from issuing new motions while
a migrate/relocate is queued or running. It never records the outcome of
a motion it issued. If the motion fails (e.g. because the guest has a
local CD-ROM image attached), the resource returns to 'started' on its
source node, and the very same motion is selected again as soon as the
imbalance has exceeded the threshold for the hold duration again. In
practice, this resulted in an endless loop of failing migrations about
every 40 seconds with the default hold duration.
Record the result of each resource motion issued by load_balance() when
the LRM reports it in next_state_migrate_relocate(). This also covers
the dependent resources of a resource bundle, which are moved together
with the bundle's leader, so a failure is recorded for the resource that
actually failed and is not lost if the bundle's leader changes.
Failed motions are handled per rebalancing attempt. An attempt starts
when the imbalance has exceeded the threshold for the hold duration and
is concluded when a rebalance motion succeeds, when there is no
rebalance motion left to try, or when the imbalance drops below the
threshold again:
- a failed motion is not tried again in the same rebalancing attempt,
so the next best motion is tried instead;
- in the following attempt, it is considered again, since the cause of
the failure might have been temporary;
- if it fails again, it is skipped in exponentially more of the
following attempts (2, 4, 8, ..., at most 32), as the cause is likely
not temporary.
Counting attempts instead of time avoids that two failing motions of the
same resource to different nodes take turns, because the time-based
backoff of one of them has just expired when the other one fails, before
the next best motion is ever tried.
Like sustained_imbalance_round, this state is not persisted across a
CRM failover.
The expected logs of the test cases added in the previous patch change
accordingly:
- test-crs-dynamic-auto-rebalance5: after the relocations of fa:109 to
node2 and node3 failed, vm:102 is migrated instead; when the imbalance
is still too high afterwards, fa:109 is tried once more in the next
attempt, fails again, and vm:103 is migrated instead.
- test-crs-dynamic-auto-rebalance6: the only possible motion is retried
less and less often (at 80, 220, 480, 860 and 1480 seconds) instead of
every 80 seconds.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 130 ++++++++++++++++++
.../log.expect | 67 +++++----
.../log.expect | 90 ++++--------
3 files changed, 191 insertions(+), 96 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index e05f0a4..18ad12f 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -72,6 +72,26 @@ sub new {
# this is not persisted for a CRM failover as in the mean time
# the usage statistics might have change quite a bit already
sustained_imbalance_round => 0,
+ # outcome tracking of the resource motions issued by load_balance()
+ #
+ # like sustained_imbalance_round, this is not persisted for a CRM
+ # failover
+ auto_rebalance_state => {
+ # number of rebalancing attempts so far; an attempt starts when the
+ # imbalance has exceeded the threshold for the hold duration and is
+ # concluded when a rebalance motion succeeds, when there is no
+ # rebalance motion left to try or when the imbalance drops below
+ # the threshold again
+ attempt => 0,
+ attempt_ongoing => 0,
+ # $sid => { target => $node, task => $task } for issued motions,
+ # which have not finished yet
+ issued => {},
+ # "$sid:$target" => { count => $count, attempt => $attempt } for
+ # rebalance motions which failed $count times in a row, the last
+ # time in rebalancing attempt $attempt
+ failures => {},
+ },
group_migration_round => 3, # wait a little bit
}, $class;
@@ -187,6 +207,76 @@ sub get_active_stationary_movable_resource_bundles {
return $resource_bundles;
}
+# Upper bound for the number of rebalancing attempts, in which a rebalance motion
+# that failed repeatedly is skipped.
+my $auto_rebalance_max_skipped_attempts = 32;
+
+# Returns the number of rebalancing attempts following the last failure, in
+# which a rebalance motion that failed $count times in a row is skipped.
+my sub skipped_rebalance_attempts {
+ my ($count) = @_;
+
+ return 0 if $count <= 1;
+
+ my $skipped_attempts = 2**($count - 1);
+
+ return $skipped_attempts < $auto_rebalance_max_skipped_attempts
+ ? $skipped_attempts
+ : $auto_rebalance_max_skipped_attempts;
+}
+
+# Returns whether the motion of $sid to $target is not to be considered in the
+# current rebalancing attempt, because it failed before.
+#
+# A failed motion is never retried in the same rebalancing attempt, so that the
+# next best motion is tried instead. If it fails again in a later attempt, it is
+# skipped for exponentially more of the following attempts, as it might not be
+# a temporary issue that caused the failure.
+my sub is_rebalance_motion_backed_off {
+ my ($self, $sid, $target) = @_;
+
+ my $state = $self->{auto_rebalance_state};
+ my $failure = $state->{failures}->{"$sid:$target"} or return 0;
+
+ my $skipped_attempts = skipped_rebalance_attempts($failure->{count});
+
+ return $state->{attempt} <= $failure->{attempt} + $skipped_attempts;
+}
+
+# Records the result of a resource motion, if it was issued by load_balance().
+sub record_rebalance_motion_result {
+ my ($self, $sid, $target, $success) = @_;
+
+ my $state = $self->{auto_rebalance_state};
+
+ my $issued = $state->{issued}->{$sid};
+ return if !$issued || $issued->{target} ne $target;
+
+ delete $state->{issued}->{$sid};
+
+ my $key = "$sid:$target";
+
+ if ($success) {
+ delete $state->{failures}->{$key};
+ $state->{attempt_ongoing} = 0;
+ return;
+ }
+
+ my $failure = $state->{failures}->{$key} //= { count => 0 };
+ $failure->{count}++;
+ $failure->{attempt} = $state->{attempt};
+
+ my $msg = "auto rebalance - $issued->{task} $sid to $target failed";
+ if ($failure->{count} == 1) {
+ $msg .= ", not trying it again in this rebalancing attempt";
+ } else {
+ my $skipped_attempts = skipped_rebalance_attempts($failure->{count});
+ $msg .= " $failure->{count} times in a row, not trying it again in this"
+ . " and the next $skipped_attempts rebalancing attempts";
+ }
+ $self->{haenv}->log('warning', $msg);
+}
+
# Returns a hash of hashes, where each item contains the resource bundle's
# leader, the list of HA resources in the resource bundle, and the list of
# possible nodes to migrate to.
@@ -213,6 +303,14 @@ sub get_resource_migration_candidates {
delete $target_nodes->{$current_leader_node};
+ for my $node (keys %$target_nodes) {
+ for my $sid ($resource_bundles->{$leader_sid}->@*) {
+ next if !is_rebalance_motion_backed_off($self, $sid, $node);
+ delete $target_nodes->{$node};
+ last;
+ }
+ }
+
next if !%$target_nodes;
push @compact_migration_candidates,
@@ -234,6 +332,15 @@ sub load_balance {
return if !$auto_rebalance_opts->{enable};
return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic';
+
+ my $state = $self->{auto_rebalance_state};
+
+ # forget issued motions which were dropped before they were carried out
+ for my $sid (keys $state->{issued}->%*) {
+ delete $state->{issued}->{$sid}
+ if !defined($self->{ss}->{$sid}) || !$self->is_resource_motion_queued_or_running($sid);
+ }
+
return if $self->any_resource_motion_queued_or_running();
my ($threshold, $method, $hold_duration, $margin) =
@@ -257,6 +364,7 @@ sub load_balance {
# the <= relation prevents load balancing from triggering for $imbalance = 0.0
if ($imbalance <= $threshold) {
$self->{sustained_imbalance_round} = 0;
+ $state->{attempt_ongoing} = 0;
return;
} else {
$self->{sustained_imbalance_round}++;
@@ -264,6 +372,14 @@ sub load_balance {
$self->{sustained_imbalance_round} = 0;
}
+ if (!$state->{attempt_ongoing}) {
+ $state->{attempt}++;
+ $state->{attempt_ongoing} = 1;
+ }
+
+ # concluded unless a rebalance motion is issued below
+ $state->{attempt_ongoing} = 0;
+
my $candidates = $self->get_resource_migration_candidates();
my $result;
@@ -295,6 +411,18 @@ sub load_balance {
$haenv->log('info', "auto rebalance - $task $sid to $target ($imbalance_change_str)");
$self->queue_resource_motion($cmd, $task, $sid, $target);
+
+ # queue_resource_motion() refuses to queue the motion if there are blocking
+ # resources on the target node
+ return if !$self->is_resource_motion_queued_or_running($sid);
+
+ $state->{attempt_ongoing} = 1;
+
+ # the whole resource bundle is moved together with its leader $sid
+ my ($candidate) = grep { $_->{leader} eq $sid } @$candidates;
+ for my $csid ($candidate ? $candidate->{resources}->@* : ($sid)) {
+ $state->{issued}->{$csid} = { target => $target, task => $task };
+ }
}
sub cleanup {
@@ -1269,6 +1397,7 @@ sub next_state_migrate_relocate {
my $exit_code = $lrm_res->{exit_code};
my $req_state = $cd->{state} eq 'started' ? 'started' : 'request_stop';
if ($exit_code == SUCCESS) {
+ $self->record_rebalance_motion_result($sid, $sd->{target}, 1);
&$change_service_state($self, $sid, $req_state, node => $sd->{target});
return;
} elsif ($exit_code == EWRONG_NODE) {
@@ -1285,6 +1414,7 @@ sub next_state_migrate_relocate {
$change_service_state->($self, $sid, $req_state, node => $sd->{node});
} else {
$haenv->log('err', "service '$sid' - migration failed (exit code $exit_code)");
+ $self->record_rebalance_motion_result($sid, $sd->{target}, 0);
&$change_service_state($self, $sid, $req_state, node => $sd->{node});
return;
}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
index a5dad5a..bc80ea1 100644
--- a/src/test/test-crs-dynamic-auto-rebalance5/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
@@ -42,47 +42,42 @@ err 81 node1/lrm: service fa:109 not moved (migration error)
info 83 node2/lrm: got lock 'ha_agent_node2_lock'
info 83 node2/lrm: status change wait_for_agent_lock => active
err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 100 node1/crm: auto rebalance - relocate fa:109 to node2 failed, not trying it again in this rebalancing attempt
info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 160 node1/crm: got crm command: relocate fa:109 node2
-info 160 node1/crm: relocate service 'fa:109' to node 'node2'
-info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 160 node1/crm: auto rebalance - relocate fa:109 to node3 (expected change for imbalance from 100.0% to 66.1%)
+info 160 node1/crm: got crm command: relocate fa:109 node3
+info 160 node1/crm: relocate service 'fa:109' to node 'node3'
+info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node3)
err 161 node1/lrm: service fa:109 not moved (migration error)
+info 165 node3/lrm: got lock 'ha_agent_node3_lock'
+info 165 node3/lrm: status change wait_for_agent_lock => active
err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 180 node1/crm: auto rebalance - relocate fa:109 to node3 failed, not trying it again in this rebalancing attempt
info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 240 node1/crm: got crm command: relocate fa:109 node2
-info 240 node1/crm: relocate service 'fa:109' to node 'node2'
-info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 241 node1/lrm: service fa:109 not moved (migration error)
-err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 320 node1/crm: got crm command: relocate fa:109 node2
-info 320 node1/crm: relocate service 'fa:109' to node 'node2'
-info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 240 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 240 node1/crm: got crm command: migrate vm:102 node2
+info 240 node1/crm: migrate service 'vm:102' to node 'node2'
+info 240 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 241 node1/lrm: service vm:102 - start migrate to node 'node2'
+info 241 node1/lrm: service vm:102 - end migrate to node 'node2'
+info 260 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2)
+info 263 node2/lrm: starting service vm:102
+info 263 node2/lrm: service status vm:102 started
+info 320 node1/crm: auto rebalance - relocate fa:109 to node3 (expected change for imbalance from 66.1% to 25.0%)
+info 320 node1/crm: got crm command: relocate fa:109 node3
+info 320 node1/crm: relocate service 'fa:109' to node 'node3'
+info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node3)
err 321 node1/lrm: service fa:109 not moved (migration error)
err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 340 node1/crm: auto rebalance - relocate fa:109 to node3 failed 2 times in a row, not trying it again in this and the next 2 rebalancing attempts
info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 400 node1/crm: got crm command: relocate fa:109 node2
-info 400 node1/crm: relocate service 'fa:109' to node 'node2'
-info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 401 node1/lrm: service fa:109 not moved (migration error)
-err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 480 node1/crm: got crm command: relocate fa:109 node2
-info 480 node1/crm: relocate service 'fa:109' to node 'node2'
-info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 481 node1/lrm: service fa:109 not moved (migration error)
-err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
-info 560 node1/crm: got crm command: relocate fa:109 node2
-info 560 node1/crm: relocate service 'fa:109' to node 'node2'
-info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 561 node1/lrm: service fa:109 not moved (migration error)
-err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 400 node1/crm: auto rebalance - migrate vm:103 to node3 (expected change for imbalance from 66.1% to 25.0%)
+info 400 node1/crm: got crm command: migrate vm:103 node3
+info 400 node1/crm: migrate service 'vm:103' to node 'node3'
+info 400 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 401 node1/lrm: service vm:103 - start migrate to node 'node3'
+info 401 node1/lrm: service vm:103 - end migrate to node 'node3'
+info 420 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3)
+info 425 node3/lrm: starting service vm:103
+info 425 node3/lrm: service status vm:103 started
info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
index 9758af5..7401c45 100644
--- a/src/test/test-crs-dynamic-auto-rebalance6/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
@@ -29,72 +29,42 @@ err 81 node1/lrm: service fa:109 not moved (migration error)
info 83 node2/lrm: got lock 'ha_agent_node2_lock'
info 83 node2/lrm: status change wait_for_agent_lock => active
err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 100 node1/crm: auto rebalance - relocate fa:109 to node2 failed, not trying it again in this rebalancing attempt
info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 160 node1/crm: got crm command: relocate fa:109 node2
-info 160 node1/crm: relocate service 'fa:109' to node 'node2'
-info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 161 node1/lrm: service fa:109 not moved (migration error)
-err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 240 node1/crm: got crm command: relocate fa:109 node2
-info 240 node1/crm: relocate service 'fa:109' to node 'node2'
-info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 241 node1/lrm: service fa:109 not moved (migration error)
-err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 320 node1/crm: got crm command: relocate fa:109 node2
-info 320 node1/crm: relocate service 'fa:109' to node 'node2'
-info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 321 node1/lrm: service fa:109 not moved (migration error)
-err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 400 node1/crm: got crm command: relocate fa:109 node2
-info 400 node1/crm: relocate service 'fa:109' to node 'node2'
-info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 401 node1/lrm: service fa:109 not moved (migration error)
-err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 220 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 220 node1/crm: got crm command: relocate fa:109 node2
+info 220 node1/crm: relocate service 'fa:109' to node 'node2'
+info 220 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 221 node1/lrm: service fa:109 not moved (migration error)
+err 240 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 240 node1/crm: auto rebalance - relocate fa:109 to node2 failed 2 times in a row, not trying it again in this and the next 2 rebalancing attempts
+info 240 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
info 480 node1/crm: got crm command: relocate fa:109 node2
info 480 node1/crm: relocate service 'fa:109' to node 'node2'
info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
err 481 node1/lrm: service fa:109 not moved (migration error)
err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 500 node1/crm: auto rebalance - relocate fa:109 to node2 failed 3 times in a row, not trying it again in this and the next 4 rebalancing attempts
info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 560 node1/crm: got crm command: relocate fa:109 node2
-info 560 node1/crm: relocate service 'fa:109' to node 'node2'
-info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 561 node1/lrm: service fa:109 not moved (migration error)
-err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 640 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 640 node1/crm: got crm command: relocate fa:109 node2
-info 640 node1/crm: relocate service 'fa:109' to node 'node2'
-info 640 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 641 node1/lrm: service fa:109 not moved (migration error)
-err 660 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 660 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 720 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 720 node1/crm: got crm command: relocate fa:109 node2
-info 720 node1/crm: relocate service 'fa:109' to node 'node2'
-info 720 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-err 721 node1/lrm: service fa:109 not moved (migration error)
-err 740 node1/crm: service 'fa:109' - migration failed (exit code 1)
-info 740 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
-info 800 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
-info 800 node1/crm: got crm command: relocate fa:109 node2
-info 800 node1/crm: relocate service 'fa:109' to node 'node2'
-info 800 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
-info 801 node1/lrm: service fa:109 - start relocate to node 'node2'
-info 801 node1/lrm: stopping service fa:109 (relocate)
-info 801 node1/lrm: service status fa:109 stopped
-info 801 node1/lrm: service fa:109 - end relocate to node 'node2'
-info 820 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node2)
-info 823 node2/lrm: starting service fa:109
-info 823 node2/lrm: service status fa:109 started
+info 860 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 860 node1/crm: got crm command: relocate fa:109 node2
+info 860 node1/crm: relocate service 'fa:109' to node 'node2'
+info 860 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 861 node1/lrm: service fa:109 not moved (migration error)
+err 880 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 880 node1/crm: auto rebalance - relocate fa:109 to node2 failed 4 times in a row, not trying it again in this and the next 8 rebalancing attempts
+info 880 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 1303 node2/lrm: node had no service configured for 60 rounds, going idle.
+info 1303 node2/lrm: status change active => wait_for_agent_lock
+info 1480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 1480 node1/crm: got crm command: relocate fa:109 node2
+info 1480 node1/crm: relocate service 'fa:109' to node 'node2'
+info 1480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 1481 node1/lrm: service fa:109 not moved (migration error)
+info 1483 node2/lrm: got lock 'ha_agent_node2_lock'
+info 1483 node2/lrm: status change wait_for_agent_lock => active
+err 1500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+warn 1500 node1/crm: auto rebalance - relocate fa:109 to node2 failed 5 times in a row, not trying it again in this and the next 16 rebalancing attempts
+info 1500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
info 1520 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (3 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 04/14] fix #8059: manager: auto rebalance: do not retry failed rebalance motions right away Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources Michael Ryom
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
When a single resource dominates the cluster load and the node base
loads fluctuate around each other, the automatic load balancer moves
the resource back and forth between the nodes (see bug #8059).
Add a test case, where the base loads of two nodes are swapped right
after the only movable resource vm:101 was migrated, which shows that
vm:101 is migrated back immediately.
The test case already configures the ha-auto-rebalance-cooldown option,
which is not used yet, and its expected log records the current
behavior. It is adapted by the following patch.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
.../test-crs-dynamic-auto-rebalance7/README | 17 +++++++
.../test-crs-dynamic-auto-rebalance7/cmdlist | 7 +++
.../datacenter.cfg | 7 +++
.../dynamic_service_stats | 5 ++
.../hardware_status | 4 ++
.../log.expect | 50 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 5 ++
.../static_service_stats | 5 ++
9 files changed, 101 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README
new file mode 100644
index 0000000..db7bb00
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/README
@@ -0,0 +1,17 @@
+Test that a HA resource which was successfully moved by the auto rebalancer
+is not moved again during the configured cooldown, even if a qualifying
+rebalance motion exists.
+
+The cluster has two nodes, where each node has a HA resource excluded from
+the auto rebalancing (vm:100 on node1 and vm:200 on node2), which act as the
+node base loads, and one movable HA resource vm:101 on node1. Initially,
+node1 is much more loaded than node2, so vm:101 is migrated to node2.
+
+Afterwards, the base loads are swapped, i.e. node2 becomes much more loaded
+than node1. Without a cooldown, the auto rebalancer immediately migrates
+vm:101 back to node1, as the expected relative imbalance improvement
+qualifies. With ha-auto-rebalance-cooldown set to 600 seconds, vm:101 must
+not be moved again until 10 minutes after its first migration has finished.
+
+This mimics a single dominant HA resource oscillating between two nodes,
+whose base loads fluctuate around each other.
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/cmdlist b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
new file mode 100644
index 0000000..1de33e4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/cmdlist
@@ -0,0 +1,7 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [
+ "service vm:100 set-dynamic-stats cpu 0.96 mem 0",
+ "service vm:200 set-dynamic-stats cpu 14.4 mem 0"
+ ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
new file mode 100644
index 0000000..f34bb81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/datacenter.cfg
@@ -0,0 +1,7 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1,
+ "ha-auto-rebalance-cooldown": 600
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
new file mode 100644
index 0000000..3d2d743
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "cpu": 16.8, "mem": 0 },
+ "vm:101": { "cpu": 4.8, "mem": 0 },
+ "vm:200": { "cpu": 0.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/hardware_status b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
new file mode 100644
index 0000000..c87f487
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -0,0 +1,50 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:200' on node 'node2'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:200': state changed from 'request_start' to 'started' (node = node2)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:200
+info 23 node2/lrm: service status vm:200 started
+info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 91.5% to 48.9%)
+info 80 node1/crm: got crm command: migrate vm:101 node2
+info 80 node1/crm: migrate service 'vm:101' to node 'node2'
+info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 81 node1/lrm: service vm:101 - start migrate to node 'node2'
+info 81 node1/lrm: service vm:101 - end migrate to node 'node2'
+info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 103 node2/lrm: starting service vm:101
+info 103 node2/lrm: service status vm:101 started
+info 120 cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
+info 120 cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
+info 160 node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
+info 160 node1/crm: got crm command: migrate vm:101 node1
+info 160 node1/crm: migrate service 'vm:101' to node 'node1'
+info 160 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node1)
+info 163 node2/lrm: service vm:101 - start migrate to node 'node1'
+info 163 node2/lrm: service vm:101 - end migrate to node 'node1'
+info 180 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 181 node1/lrm: starting service vm:101
+info 181 node1/lrm: service status vm:101 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/manager_status b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/service_config b/src/test/test-crs-dynamic-auto-rebalance7/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
new file mode 100644
index 0000000..f7d1a27
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance7/static_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "maxcpu": 20.0, "maxmem": 8589934592 },
+ "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 },
+ "vm:200": { "maxcpu": 16.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (4 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 07/14] test: add auto rebalance test case for a marginal absolute improvement Michael Ryom
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
When a single resource dominates the cluster load, the relative
improvement of moving it between two nodes is 2d / (v + d), where v is
the load of the resource and d is the difference of the node base loads.
Small fluctuations of d around zero are enough to make a motion and,
right afterwards, its reverse qualify, so the resource is moved back and
forth (see bug #8059).
Add the ha-auto-rebalance-cooldown option, which keeps resources that
were successfully moved by the load balancer from being considered for
another rebalance motion for the given time. The cooldown applies to all
resources of a moved resource bundle, as they were all moved, and a
bundle is skipped if any of its resources is still on cooldown.
The cooldown defaults to 0, i.e. it is disabled and the behavior of
existing setups does not change.
Like the other auto rebalance state, the cooldown is not persisted
across a CRM failover.
The expected log of test-crs-dynamic-auto-rebalance7 changes, as vm:101
is not migrated back right after it was migrated, but only after the
configured cooldown of 600 seconds has expired.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 27 +++++++++++++++++++
.../log.expect | 15 +++++------
2 files changed, 33 insertions(+), 9 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 18ad12f..c480b73 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -91,6 +91,9 @@ sub new {
# rebalance motions which failed $count times in a row, the last
# time in rebalancing attempt $attempt
failures => {},
+ # $sid => $time until which a resource successfully moved by
+ # load_balance() is not considered for rebalancing
+ cooldown => {},
},
group_migration_round => 3, # wait a little bit
}, $class;
@@ -138,6 +141,7 @@ sub update_crs_scheduler_mode {
// 3;
$self->{crs}->{auto_rebalance}->{margin} =
($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ $self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0;
my $old_mode = $self->{crs}->{scheduler};
my $new_mode = $crs_cfg->{ha} || 'basic';
@@ -259,6 +263,10 @@ sub record_rebalance_motion_result {
if ($success) {
delete $state->{failures}->{$key};
$state->{attempt_ongoing} = 0;
+
+ my $cooldown = $self->{crs}->{auto_rebalance}->{cooldown};
+ $state->{cooldown}->{$sid} = $self->{haenv}->get_time() + $cooldown if $cooldown > 0;
+
return;
}
@@ -277,6 +285,20 @@ sub record_rebalance_motion_result {
$self->{haenv}->log('warning', $msg);
}
+# Returns whether $sid was moved by load_balance() too recently to be considered
+# for rebalancing again.
+my sub is_on_rebalance_cooldown {
+ my ($self, $sid, $now) = @_;
+
+ my $cooldown = $self->{auto_rebalance_state}->{cooldown};
+ my $until = $cooldown->{$sid} // return 0;
+
+ return 1 if $now < $until;
+
+ delete $cooldown->{$sid};
+ return 0;
+}
+
# Returns a hash of hashes, where each item contains the resource bundle's
# leader, the list of HA resources in the resource bundle, and the list of
# possible nodes to migrate to.
@@ -291,8 +313,13 @@ sub get_resource_migration_candidates {
my $resource_bundles =
get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity);
+ my $now = $self->{haenv}->get_time();
+
my @compact_migration_candidates = ();
for my $leader_sid (sort keys %$resource_bundles) {
+ my $bundle = $resource_bundles->{$leader_sid};
+ next if grep { is_on_rebalance_cooldown($self, $_, $now) } @$bundle;
+
my $current_leader_node = $ss->{$leader_sid}->{node};
my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
index c87f487..a82f4bc 100644
--- a/src/test/test-crs-dynamic-auto-rebalance7/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect
@@ -38,13 +38,10 @@ info 103 node2/lrm: starting service vm:101
info 103 node2/lrm: service status vm:101 started
info 120 cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0
info 120 cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0
-info 160 node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
-info 160 node1/crm: got crm command: migrate vm:101 node1
-info 160 node1/crm: migrate service 'vm:101' to node 'node1'
-info 160 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 163 node2/lrm: service vm:101 - start migrate to node 'node1'
-info 163 node2/lrm: service vm:101 - end migrate to node 'node1'
-info 180 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
-info 181 node1/lrm: starting service vm:101
-info 181 node1/lrm: service status vm:101 started
+info 700 node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%)
+info 700 node1/crm: got crm command: migrate vm:101 node1
+info 700 node1/crm: migrate service 'vm:101' to node 'node1'
+info 700 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node1)
+info 703 node2/lrm: service vm:101 - start migrate to node 'node1'
+info 703 node2/lrm: service vm:101 - end migrate to node 'node1'
info 720 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 07/14] test: add auto rebalance test case for a marginal absolute improvement
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (5 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin Michael Ryom
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
When a single resource dominates the cluster load, the relative margin
provides almost no hysteresis, so a rebalance motion with an absolute
improvement of only a few percentage points qualifies.
Add a test case, where migrating vm:101 would improve the imbalance from
35.6% to 31.1%, i.e. by 12.5% relative, but only by 4.4 percentage
points absolute.
The test case already configures an absolute margin of 5 percentage
points with the ha-auto-rebalance-margin-mode option, which is not used
yet, so the margin of 5 is still interpreted as relative and its
expected log records that vm:101 is migrated. It is adapted by the
following patch.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
.../test-crs-dynamic-auto-rebalance8/README | 12 ++++++
.../test-crs-dynamic-auto-rebalance8/cmdlist | 3 ++
.../datacenter.cfg | 8 ++++
.../dynamic_service_stats | 5 +++
.../hardware_status | 4 ++
.../log.expect | 39 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 5 +++
.../static_service_stats | 5 +++
9 files changed, 82 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/README b/src/test/test-crs-dynamic-auto-rebalance8/README
new file mode 100644
index 0000000..80e9ac3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/README
@@ -0,0 +1,12 @@
+Test that the auto rebalance system with an absolute margin does not act on a
+rebalance motion whose absolute improvement of the node imbalance is below
+the margin, even though its relative improvement is well above it.
+
+The cluster has two nodes with a HA resource excluded from auto rebalancing
+on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and
+one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays
+above the default trigger threshold of 30%, and migrating vm:101 to node2
+would improve the imbalance to 31.1%, i.e. by 12.5% relative. With a relative
+margin of 5% the motion would qualify, but with the margin mode set to
+'absolute', the improvement of 4.4 percentage points is below the margin of 5
+percentage points, so the motion must not be issued.
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/cmdlist b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
new file mode 100644
index 0000000..76f0313
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
new file mode 100644
index 0000000..63c1d6b
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/datacenter.cfg
@@ -0,0 +1,8 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1,
+ "ha-auto-rebalance-margin": 5,
+ "ha-auto-rebalance-margin-mode": "absolute"
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
new file mode 100644
index 0000000..231af30
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/dynamic_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "cpu": 7.44, "mem": 0 },
+ "vm:101": { "cpu": 7.2, "mem": 0 },
+ "vm:200": { "cpu": 6.96, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/hardware_status b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
new file mode 100644
index 0000000..df11212
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
@@ -0,0 +1,39 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:200' on node 'node2'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:200': state changed from 'request_start' to 'started' (node = node2)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:200
+info 23 node2/lrm: service status vm:200 started
+info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 35.6% to 31.1%)
+info 80 node1/crm: got crm command: migrate vm:101 node2
+info 80 node1/crm: migrate service 'vm:101' to node 'node2'
+info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 81 node1/lrm: service vm:101 - start migrate to node 'node2'
+info 81 node1/lrm: service vm:101 - end migrate to node 'node2'
+info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 103 node2/lrm: starting service vm:101
+info 103 node2/lrm: service status vm:101 started
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/manager_status b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/service_config b/src/test/test-crs-dynamic-auto-rebalance8/service_config
new file mode 100644
index 0000000..83bf42e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 },
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
new file mode 100644
index 0000000..300fb7a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance8/static_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:100": { "maxcpu": 8.0, "maxmem": 8589934592 },
+ "vm:101": { "maxcpu": 8.0, "maxmem": 8589934592 },
+ "vm:200": { "maxcpu": 8.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (6 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 07/14] test: add auto rebalance test case for a marginal absolute improvement Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 09/14] sim: hardware: report actual running state in cluster service stats Michael Ryom
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The automatic load balancer only requires a relative improvement of the
node imbalance (default 10%). When a single resource dominates the
cluster load, the relative margin provides almost no hysteresis: with
two nodes, a resource load v and a node base load difference d, the
relative improvement of moving the resource is 2d / (v + d), so it
exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM
that is a base load difference of about half a percentage point, well
within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and
each fluctuation then qualifies a motion (and, after it was carried
out, its reverse) (see bug #8059).
Add the ha-auto-rebalance-margin-mode option, which allows to interpret
the existing margin as the minimum absolute improvement in percentage
points instead. This creates a dead band for motions that shuffle load
without meaningfully improving the cluster balance. The relative margin
was introduced when the imbalance was not yet bounded for different
cluster sizes; now that it is in [0, 1] for all cluster sizes, an
absolute margin is meaningful on its own.
The margin mode defaults to 'relative', i.e. the behavior of existing
setups does not change.
The expected log of test-crs-dynamic-auto-rebalance8 changes, as the
improvement of 4.4 percentage points of migrating vm:101 is below the
configured absolute margin of 5 percentage points.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 14 ++++++++++----
.../test-crs-dynamic-auto-rebalance8/log.expect | 9 ---------
2 files changed, 10 insertions(+), 13 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index c480b73..b804602 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -141,6 +141,8 @@ sub update_crs_scheduler_mode {
// 3;
$self->{crs}->{auto_rebalance}->{margin} =
($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0;
+ $self->{crs}->{auto_rebalance}->{margin_mode} = $crs_cfg->{'ha-auto-rebalance-margin-mode'}
+ // 'relative';
$self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0;
my $old_mode = $self->{crs}->{scheduler};
@@ -370,8 +372,8 @@ sub load_balance {
return if $self->any_resource_motion_queued_or_running();
- my ($threshold, $method, $hold_duration, $margin) =
- $auto_rebalance_opts->@{qw(threshold method hold_duration margin)};
+ my ($threshold, $method, $hold_duration, $margin, $margin_mode) =
+ $auto_rebalance_opts->@{qw(threshold method hold_duration margin margin_mode)};
# with threshold and margin both at 0% any non-zero imbalance triggers a non-improving
# migration, churning the cluster forever - skip and warn instead of silently looping.
@@ -421,8 +423,12 @@ sub load_balance {
my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)};
- my $relative_change = ($imbalance - $target_imbalance) / $imbalance;
- return if $relative_change < $margin;
+ # the relative margin alone provides almost no hysteresis if a single resource
+ # dominates the cluster load, so tiny fluctuations of the node base loads can
+ # make a motion and its reverse qualify; an absolute margin guards against it
+ my $absolute_change = $imbalance - $target_imbalance;
+ my $change = $margin_mode eq 'absolute' ? $absolute_change : $absolute_change / $imbalance;
+ return if $change < $margin;
my ($sid, $source, $target) = $migration->@{qw(sid source-node target-node)};
diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
index df11212..6c84e42 100644
--- a/src/test/test-crs-dynamic-auto-rebalance8/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect
@@ -27,13 +27,4 @@ info 23 node2/lrm: got lock 'ha_agent_node2_lock'
info 23 node2/lrm: status change wait_for_agent_lock => active
info 23 node2/lrm: starting service vm:200
info 23 node2/lrm: service status vm:200 started
-info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 35.6% to 31.1%)
-info 80 node1/crm: got crm command: migrate vm:101 node2
-info 80 node1/crm: migrate service 'vm:101' to node 'node2'
-info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
-info 81 node1/lrm: service vm:101 - start migrate to node 'node2'
-info 81 node1/lrm: service vm:101 - end migrate to node 'node2'
-info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
-info 103 node2/lrm: starting service vm:101
-info 103 node2/lrm: service status vm:101 started
info 620 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 09/14] sim: hardware: report actual running state in cluster service stats
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (7 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 10/14] sim: hardware: allow setting a base load for nodes Michael Ryom
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The simulated cluster service stats reported a service as running
whenever its request state was 'started', even before the LRM actually
started it. On the real cluster, the 'running' flag of the service
stats is derived from the actual RRD status of the guest, and the node
usage stats only contain load of actually running guests.
Report the running state as marked by the LRM in the node service
status instead, so that the service stats and the node usage stats of
the simulated hardware are consistent with each other, like they are on
the real cluster.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Sim/Hardware.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index e1a9ddd..538bc4e 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -1198,6 +1198,9 @@ sub watchdog_update {
my sub get_cluster_service_stats {
my ($self) = @_;
+ my $cstatus = $self->read_hardware_status_nolock();
+ my $node_service_status = { map { $_ => $self->read_service_status($_) } keys %$cstatus };
+
my $stats = {};
for my $sid (keys $self->{service_config}->%*) {
my $cfg = $self->{service_config}->{$sid};
@@ -1205,7 +1208,11 @@ my sub get_cluster_service_stats {
$stats->{$sid} = {
node => $cfg->{node},
state => $cfg->{state},
- running => $cfg->{state} eq 'started',
+ # like the RRD-based stats on the real cluster, report whether
+ # the service is actually running (as marked by the LRM), not
+ # whether its request state is 'started'; the node usage stats
+ # only account for actually running services as well
+ running => $node_service_status->{ $cfg->{node} }->{$sid} ? 1 : 0,
usage => {},
};
}
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 10/14] sim: hardware: allow setting a base load for nodes
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (8 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 09/14] sim: hardware: report actual running state in cluster service stats Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads Michael Ryom
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The dynamic node usage stats of the simulated hardware only consisted
of the usage of the running HA-managed services. On a real cluster, the
node usage additionally contains everything else running on the node:
non-HA guests, storage daemons like Ceph or the ZFS ARC, and other host
processes, which fluctuate independently of the HA-managed services.
Read an optional base load ('cpu' and 'mem') for each node from the
hardware status and add a hardware command
node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>]
to change it during a test, so that regression tests can simulate load
that cannot be attributed to any HA-managed service.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Sim/Hardware.pm | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index 538bc4e..aa1a886 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -791,6 +791,7 @@ sub get_cfs_state {
# crm enable-node-maintenance <node>
# crm disable-node-maintenance <node>
# pve-manager-version <node> set <version> # note: this is NOT the *ha*-manager version
+# node <node> set-dynamic-stats [cpu <cores>] [mem <MiB>] # load outside HA services
# reboot <node>
# shutdown <node>
# restart-lrm <node>
@@ -881,6 +882,26 @@ sub sim_hardware_cmd {
$self->write_hardware_status_nolock($cstatus);
+ } elsif ($cmd eq 'node') {
+ die "sim_hardware_cmd: unknown node action '$action'"
+ if $action ne 'set-dynamic-stats';
+ die "sim_hardware_cmd: missing target stat for '$action' command"
+ if !@params;
+
+ my $conversions = { cpu => sub { 0.0 + $_[0] }, mem => sub { $_[0] * 1024**2 } };
+
+ for my ($target, $val) (@params) {
+ die "sim_hardware_cmd: missing value for '$action $target' command"
+ if !defined($val);
+
+ my $convert = $conversions->{$target}
+ or die "sim_hardware_cmd: unknown target stat '$target' for '$action' command";
+
+ $cstatus->{$node}->{$target} = $convert->($val);
+ }
+
+ $self->write_hardware_status_nolock($cstatus);
+
} elsif ($cmd eq 'cfs') {
die "sim_hardware_cmd: unknown cfs action '$action' for node '$node'"
if $action !~ m/^(rw|update)$/;
@@ -1281,18 +1302,21 @@ sub get_static_node_stats {
sub get_dynamic_node_stats {
my ($self) = @_;
+ my $cstatus = $self->read_hardware_status_nolock();
+
my $stats = $self->get_static_node_stats();
for my $node (keys %$stats) {
$stats->{$node}->{maxcpu} = $stats->{$node}->{maxcpu} // $default_node_maxcpu;
- $stats->{$node}->{cpu} = $stats->{$node}->{cpu} // 0.0;
+ # base load of the node outside of the HA-managed services, e.g.
+ # non-HA guests or other host processes
+ $stats->{$node}->{cpu} = $cstatus->{$node}->{cpu} // 0.0;
$stats->{$node}->{maxmem} = $stats->{$node}->{maxmem} // $default_node_maxmem;
- $stats->{$node}->{mem} = $stats->{$node}->{mem} // 0;
+ $stats->{$node}->{mem} = $cstatus->{$node}->{mem} // 0;
}
my $service_conf = $self->read_service_config();
my $dynamic_service_stats = $self->get_dynamic_service_stats();
- my $cstatus = $self->read_hardware_status_nolock();
my $node_service_status = { map { $_ => $self->read_service_status($_) } keys %$cstatus };
for my $sid (keys %$service_conf) {
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (9 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 10/14] sim: hardware: allow setting a base load for nodes Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load Michael Ryom
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
Add a test case, where the base loads of two nodes, which are not caused
by any HA resource, are flipped by about half a percentage point of node
load in alternating directions, while a single dominant HA resource runs
on one of them.
The expected log records the current behavior, where vm:100 is migrated
back and forth after each flip (see bug #8059). It is adapted by the
following patch.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
.../test-crs-dynamic-auto-rebalance9/README | 16 +++++
.../test-crs-dynamic-auto-rebalance9/cmdlist | 19 ++++++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 3 +
.../hardware_status | 4 ++
.../log.expect | 66 +++++++++++++++++++
.../manager_status | 1 +
.../service_config | 3 +
.../static_service_stats | 3 +
9 files changed, 121 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README
new file mode 100644
index 0000000..93b27c4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/README
@@ -0,0 +1,16 @@
+Test that a single dominant HA resource does not oscillate between two nodes
+whose base loads (i.e. load not caused by any HA-managed service, here set via
+the node base load of the simulated hardware) fluctuate around each other.
+
+The cluster has two nodes with an equal base load and one dominant movable
+HA resource vm:100 on node1. The imbalance is permanently above the trigger
+threshold, but no motion can improve it, so the load balancer stays armed
+without acting. Then the node base loads are repeatedly flipped by about half
+a percentage point of node load in alternating directions.
+
+Each raw sample after a flip qualifies a migration of vm:100 to the less
+loaded node (and after the next flip, back again), as the expected relative
+imbalance improvement of about 10.4% exceeds the default margin of 10%.
+Acting on the raw samples moves the dominant resource back and forth with
+every flip, even though the fluctuation is well within the noise of a real
+node's base load.
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/cmdlist b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
new file mode 100644
index 0000000..eeee26c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist
@@ -0,0 +1,19 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [
+ "node node1 set-dynamic-stats cpu 5.064 mem 0",
+ "node node2 set-dynamic-stats cpu 4.536 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 4.536 mem 0",
+ "node node2 set-dynamic-stats cpu 5.064 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 5.064 mem 0",
+ "node node2 set-dynamic-stats cpu 4.536 mem 0"
+ ],
+ [
+ "node node1 set-dynamic-stats cpu 4.536 mem 0",
+ "node node2 set-dynamic-stats cpu 5.064 mem 0"
+ ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
new file mode 100644
index 0000000..ef14918
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "cpu": 9.6, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/hardware_status b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
new file mode 100644
index 0000000..bb0cf81
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "cpu": 4.8, "mem": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
new file mode 100644
index 0000000..efd4d2c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
@@ -0,0 +1,66 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info 120 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info 140 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
+info 140 node1/crm: got crm command: migrate vm:100 node2
+info 140 node1/crm: migrate service 'vm:100' to node 'node2'
+info 140 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 141 node1/lrm: service vm:100 - start migrate to node 'node2'
+info 141 node1/lrm: service vm:100 - end migrate to node 'node2'
+info 143 node2/lrm: got lock 'ha_agent_node2_lock'
+info 143 node2/lrm: status change wait_for_agent_lock => active
+info 160 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2)
+info 163 node2/lrm: starting service vm:100
+info 163 node2/lrm: service status vm:100 started
+info 220 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info 220 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info 220 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
+info 220 node1/crm: got crm command: migrate vm:100 node1
+info 220 node1/crm: migrate service 'vm:100' to node 'node1'
+info 220 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1)
+info 223 node2/lrm: service vm:100 - start migrate to node 'node1'
+info 223 node2/lrm: service vm:100 - end migrate to node 'node1'
+info 240 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1)
+info 241 node1/lrm: starting service vm:100
+info 241 node1/lrm: service status vm:100 started
+info 320 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
+info 320 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
+info 360 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
+info 360 node1/crm: got crm command: migrate vm:100 node2
+info 360 node1/crm: migrate service 'vm:100' to node 'node2'
+info 360 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 361 node1/lrm: service vm:100 - start migrate to node 'node2'
+info 361 node1/lrm: service vm:100 - end migrate to node 'node2'
+info 380 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2)
+info 383 node2/lrm: starting service vm:100
+info 383 node2/lrm: service status vm:100 started
+info 420 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
+info 420 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
+info 440 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
+info 440 node1/crm: got crm command: migrate vm:100 node1
+info 440 node1/crm: migrate service 'vm:100' to node 'node1'
+info 440 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1)
+info 443 node2/lrm: service vm:100 - start migrate to node 'node1'
+info 443 node2/lrm: service vm:100 - end migrate to node 'node1'
+info 460 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1)
+info 461 node1/lrm: starting service vm:100
+info 461 node1/lrm: service status vm:100 started
+info 1020 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/manager_status b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/service_config b/src/test/test-crs-dynamic-auto-rebalance9/service_config
new file mode 100644
index 0000000..60688cf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
new file mode 100644
index 0000000..7eee778
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats
@@ -0,0 +1,3 @@
+{
+ "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (10 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH manager v2 13/14] ui: crs options: expose the auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations Michael Ryom
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The dynamic scheduler acts on point-in-time samples of the node and
guest usage stats. The part of the node load that is not caused by any
HA-managed service (non-HA guests, the ZFS ARC, Ceph daemons, the IO of
a rebalance migration itself, sampling skew between node and guest
stats) is noisy, and acting on a single sample makes the automatic load
balancer prone to oscillation: a single dominant HA resource on a small
cluster qualifies for a migration (and afterwards, for its reverse) as
soon as the node base loads fluctuate around each other by a fraction
of the resource's own load.
Decompose each node's load into the usage of the HA-managed services
actually running on it plus a residual, and smooth only the residual
with an exponentially moving average that the manager keeps across
scheduling rounds (not persisted for a CRM failover, like the other
balancer state).
The service usages themselves are deliberately not smoothed: they are
fully modeled by the scheduler, so known structural changes (a service
was migrated, started or stopped, or its measured load changed) take
effect immediately, transient service load spikes remain handled by the
hold duration, and the expected log of no existing regression test
changes. The residual, on the other hand, cannot be attributed to
anything the balancer could act on, and its expected reaction to a
rebalance motion is zero, so a lagging average is safe to act on.
The expected log of test-crs-dynamic-auto-rebalance9 changes, as the
effective base load difference now stays well below the qualification
point and vm:100 is not migrated back and forth anymore.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
src/PVE/HA/Manager.pm | 10 ++-
src/PVE/HA/Usage/Dynamic.pm | 73 ++++++++++++++++++-
.../test-crs-dynamic-auto-rebalance9/README | 4 +
.../log.expect | 38 ----------
4 files changed, 85 insertions(+), 40 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index b804602..1721fae 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -95,6 +95,10 @@ sub new {
# load_balance() is not considered for rebalancing
cooldown => {},
},
+ # exponentially moving average of the unaccounted node load, kept
+ # across rounds by PVE::HA::Usage::Dynamic; also not persisted for a
+ # CRM failover
+ dynamic_stats_smoothing_state => {},
group_migration_round => 3, # wait a little bit
}, $class;
@@ -623,7 +627,11 @@ sub recompute_online_node_usage {
if ($have_dynamic_scheduling) {
$online_node_usage = eval {
$service_stats = $haenv->get_dynamic_service_stats();
- my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats);
+ my $scheduler = PVE::HA::Usage::Dynamic->new(
+ $haenv,
+ $service_stats,
+ $self->{dynamic_stats_smoothing_state},
+ );
$scheduler->add_node($_) for $online_nodes->@*;
return $scheduler;
};
diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm
index 76d0fea..fc88ddf 100644
--- a/src/PVE/HA/Usage/Dynamic.pm
+++ b/src/PVE/HA/Usage/Dynamic.pm
@@ -8,12 +8,83 @@ use PVE::RS::ResourceScheduling::Dynamic;
use base qw(PVE::HA::Usage);
+# weight of the newest sample in the exponentially moving average of the
+# unaccounted node load; with one sample per CRM scheduling round (~10s),
+# 0.25 averages over roughly the last minute
+my $usage_smoothing_alpha = 0.25;
+
+# Smooths the *unaccounted* part of each node's load with an exponentially
+# moving average kept in $state across invocations.
+#
+# The node load is decomposed into the usage of the HA-managed services
+# running on it plus a residual, and only the residual is smoothed. The
+# residual captures the load that cannot be attributed to any HA-managed
+# service (non-HA guests, ZFS ARC, Ceph daemons, the IO of a rebalance
+# migration itself, sampling skew between the node and guest stats). Those
+# point-in-time samples are noisy, and acting on a single sample makes the
+# load balancer prone to oscillation, while their expected reaction to a
+# rebalance motion is zero, so a lagging average is safe to act on.
+#
+# The service usages themselves are deliberately not smoothed: they are
+# fully modeled by the scheduler, i.e. known structural changes (a service
+# was migrated, started or stopped, or its load changed) take effect
+# immediately, and transient service load spikes are already handled by the
+# hold duration of the load balancer.
+my $smooth_unaccounted_node_load = sub {
+ my ($state, $node_stats, $service_stats) = @_;
+
+ # usage per node of the HA-managed services actually running on it
+ my $accounted = {};
+ for my $sid (keys %$service_stats) {
+ my ($node, $running, $usage) = $service_stats->{$sid}->@{qw(node running usage)};
+ next if !$running || !defined($node) || !$usage;
+ next if !defined($usage->{cpu}) || !defined($usage->{mem});
+
+ $accounted->{$node}->{cpu} += $usage->{cpu};
+ $accounted->{$node}->{mem} += $usage->{mem};
+ }
+
+ for my $node (sort keys %$node_stats) {
+ my $stats = $node_stats->{$node};
+ next if !defined($stats->{cpu}) || !defined($stats->{mem});
+
+ # the residual may become negative if the samples disagree, e.g.
+ # right after a service stopped; only the reconstructed node load is
+ # clamped, so that the residual average is not biased upwards
+ my $residual_sample = {
+ cpu => $stats->{cpu} - ($accounted->{$node}->{cpu} // 0.0),
+ mem => $stats->{mem} - ($accounted->{$node}->{mem} // 0),
+ };
+
+ my $residual = $state->{$node};
+ if (!defined($residual)) {
+ $residual = $state->{$node} = $residual_sample;
+ } else {
+ for my $field (qw(cpu mem)) {
+ $residual->{$field} = $usage_smoothing_alpha * $residual_sample->{$field} +
+ (1 - $usage_smoothing_alpha) * $residual->{$field};
+ }
+ }
+
+ my $cpu = $residual->{cpu} + ($accounted->{$node}->{cpu} // 0.0);
+ my $mem = $residual->{mem} + ($accounted->{$node}->{mem} // 0);
+ $stats->{cpu} = $cpu > 0.0 ? $cpu : 0.0;
+ $stats->{mem} = $mem > 0 ? int($mem) : 0;
+ }
+
+ # drop state of meanwhile removed nodes
+ delete $state->{$_} for grep { !$node_stats->{$_} } keys %$state;
+};
+
sub new {
- my ($class, $haenv, $service_stats) = @_;
+ my ($class, $haenv, $service_stats, $smoothing_state) = @_;
my $node_stats = eval { $haenv->get_dynamic_node_stats() };
die "did not get dynamic node usage information - $@" if $@;
+ $smooth_unaccounted_node_load->($smoothing_state, $node_stats, $service_stats)
+ if defined($smoothing_state);
+
my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() };
die "unable to initialize dynamic scheduling - $@" if $@;
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README
index 93b27c4..56bb608 100644
--- a/src/test/test-crs-dynamic-auto-rebalance9/README
+++ b/src/test/test-crs-dynamic-auto-rebalance9/README
@@ -14,3 +14,7 @@ imbalance improvement of about 10.4% exceeds the default margin of 10%.
Acting on the raw samples moves the dominant resource back and forth with
every flip, even though the fluctuation is well within the noise of a real
node's base load.
+
+With the exponentially moving average over the unaccounted node load, the
+effective base load difference stays well below the point where a motion
+qualifies, so no motion may be issued at any point.
diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
index efd4d2c..73e381a 100644
--- a/src/test/test-crs-dynamic-auto-rebalance9/log.expect
+++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect
@@ -19,48 +19,10 @@ info 21 node1/lrm: service status vm:100 started
info 22 node2/crm: status change wait_for_quorum => slave
info 120 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
info 120 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
-info 140 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
-info 140 node1/crm: got crm command: migrate vm:100 node2
-info 140 node1/crm: migrate service 'vm:100' to node 'node2'
-info 140 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2)
-info 141 node1/lrm: service vm:100 - start migrate to node 'node2'
-info 141 node1/lrm: service vm:100 - end migrate to node 'node2'
-info 143 node2/lrm: got lock 'ha_agent_node2_lock'
-info 143 node2/lrm: status change wait_for_agent_lock => active
-info 160 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2)
-info 163 node2/lrm: starting service vm:100
-info 163 node2/lrm: service status vm:100 started
info 220 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
info 220 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
-info 220 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
-info 220 node1/crm: got crm command: migrate vm:100 node1
-info 220 node1/crm: migrate service 'vm:100' to node 'node1'
-info 220 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 223 node2/lrm: service vm:100 - start migrate to node 'node1'
-info 223 node2/lrm: service vm:100 - end migrate to node 'node1'
-info 240 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1)
-info 241 node1/lrm: starting service vm:100
-info 241 node1/lrm: service status vm:100 started
info 320 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0
info 320 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0
-info 360 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%)
-info 360 node1/crm: got crm command: migrate vm:100 node2
-info 360 node1/crm: migrate service 'vm:100' to node 'node2'
-info 360 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2)
-info 361 node1/lrm: service vm:100 - start migrate to node 'node2'
-info 361 node1/lrm: service vm:100 - end migrate to node 'node2'
-info 380 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2)
-info 383 node2/lrm: starting service vm:100
-info 383 node2/lrm: service status vm:100 started
info 420 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0
info 420 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0
-info 440 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%)
-info 440 node1/crm: got crm command: migrate vm:100 node1
-info 440 node1/crm: migrate service 'vm:100' to node 'node1'
-info 440 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1)
-info 443 node2/lrm: service vm:100 - start migrate to node 'node1'
-info 443 node2/lrm: service vm:100 - end migrate to node 'node1'
-info 460 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1)
-info 461 node1/lrm: starting service vm:100
-info 461 node1/lrm: service status vm:100 started
info 1020 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH manager v2 13/14] ui: crs options: expose the auto-rebalance margin mode and cooldown
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (11 preceding siblings ...)
2026-10-18 12:45 ` [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
2026-10-18 12:45 ` [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations Michael Ryom
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
Add form fields for the ha-auto-rebalance-margin-mode and
ha-auto-rebalance-cooldown datacenter options. They follow the same
pattern as the existing auto-rebalance tuning fields, including being
disabled while automatic rebalancing is off.
As the unit of the margin now depends on the margin mode, drop the
percent sign from its label and show the unit in the mode selection
instead.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
www/manager6/form/CRSOptions.js | 35 ++++++++++++++++++++++++++++++++-
1 file changed, 34 insertions(+), 1 deletion(-)
diff --git a/www/manager6/form/CRSOptions.js b/www/manager6/form/CRSOptions.js
index 84f6d506..bdf64177 100644
--- a/www/manager6/form/CRSOptions.js
+++ b/www/manager6/form/CRSOptions.js
@@ -117,7 +117,7 @@ Ext.define('PVE.form.CRSOptions', {
{
xtype: 'numberfield',
name: 'ha-auto-rebalance-margin',
- fieldLabel: gettext('Minimum Imbalance Improvement (%)'),
+ fieldLabel: gettext('Minimum Imbalance Improvement'),
emptyText: Proxmox.Utils.defaultText + ' (10)',
minValue: 0,
maxValue: 100,
@@ -126,6 +126,39 @@ Ext.define('PVE.form.CRSOptions', {
disabled: '{autoRebalancingDisabled}',
},
},
+ {
+ xtype: 'proxmoxKVComboBox',
+ name: 'ha-auto-rebalance-margin-mode',
+ fieldLabel: gettext('Improvement Mode'),
+ deleteEmpty: false,
+ value: '__default__',
+ comboItems: [
+ [
+ '__default__',
+ Proxmox.Utils.defaultText + ' (' + gettext('Relative') + ')',
+ ],
+ ['relative', gettext('Relative') + ' (%)'],
+ [
+ 'absolute',
+ gettext('Absolute') + ' (' + gettext('percentage points') + ')',
+ ],
+ ],
+ defaultValue: '__default__',
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ },
+ },
+ {
+ xtype: 'numberfield',
+ name: 'ha-auto-rebalance-cooldown',
+ fieldLabel: gettext('Cooldown (s)'),
+ emptyText: Proxmox.Utils.defaultText + ' (0)',
+ minValue: 0,
+ step: 60,
+ bind: {
+ disabled: '{autoRebalancingDisabled}',
+ },
+ },
],
},
],
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
` (12 preceding siblings ...)
2026-10-18 12:45 ` [PATCH manager v2 13/14] ui: crs options: expose the auto-rebalance margin mode and cooldown Michael Ryom
@ 2026-10-18 12:45 ` Michael Ryom
13 siblings, 0 replies; 15+ messages in thread
From: Michael Ryom @ 2026-10-18 12:45 UTC (permalink / raw)
To: pve-devel; +Cc: Michael Ryom
The relative imbalance improvement provides almost no hysteresis when a
single HA resource dominates the cluster load (see bug #8059), which the
new ha-auto-rebalance-margin-mode and ha-auto-rebalance-cooldown options
guard against. Also describe how the load balancer handles failed
rebalancing migrations now.
While at it, add an overview table of the load balancer options, since
the surrounding text refers to them by their user interface labels only.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
ha-manager.adoc | 62 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/ha-manager.adoc b/ha-manager.adoc
index 95ab799..bbf1ce1 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -1614,6 +1614,68 @@ the expected imbalance does not change at all. The value can never be
negative, so the balancer will never commit to a migration with an expected
imbalance worse than the current one.
+By default, the _imbalance improvement_ is relative to the current imbalance.
+That provides very little hysteresis if a single HA resource dominates the
+cluster load: moving that resource in either direction can look like a large
+relative improvement, even though the cluster ends up about as imbalanced as
+it was before, so small fluctuations of the node loads can make the load
+balancer move the resource back and forth. With the _improvement mode_ set to
+`absolute`, the _imbalance improvement_ is the number of percentage points by
+which the expected imbalance has to drop instead, which ignores such marginal
+migrations. Additionally, the _cooldown_ option keeps HA resources, which were
+moved by the load balancer, from being moved again for the given number of
+seconds. It defaults to `0`, which disables the cooldown.
+
+If a rebalancing migration fails, for example because the guest uses a local
+resource that is not available on the target node, the load balancer does not
+try the same migration again until the imbalance has exceeded the threshold
+for the hold duration anew and no other rebalancing migration succeeded in
+the meantime. Instead, it tries the next best rebalancing migration. If the
+same migration keeps failing, it is skipped for exponentially more of these
+rebalancing attempts.
+
+[[ha_manager_crs_load_balancer_options]]
+Load Balancer Options
+^^^^^^^^^^^^^^^^^^^^^
+
+The automatic load balancer is configured with the following options of the
+`crs` setting in `/etc/pve/datacenter.cfg`, which can also be set in the web
+interface under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`:
+
+[width="100%",cols="3m,1d,2d,5d",options="header"]
+|==========================================================================
+|Option |Default |Values |Description
+
+|ha-auto-rebalance |0 |0, 1
+|Enable the automatic load balancer. The options below have no effect unless
+it is enabled.
+
+|ha-auto-rebalance-threshold |30 |0-100%
+|Cluster imbalance above which the load balancer becomes active. With `0` it
+is always active.
+
+|ha-auto-rebalance-hold-duration |3 |>= 0 rounds
+|Number of consecutive HA Manager rounds, each around 10 seconds, for which
+the threshold has to be exceeded before a migration is selected.
+
+|ha-auto-rebalance-method |bruteforce |bruteforce, topsis
+|Method used to score the possible rebalancing migrations.
+
+|ha-auto-rebalance-margin |10 |0-100
+|_Imbalance improvement_: the reduction of the cluster imbalance a migration
+has to achieve to be committed, in percent of the current imbalance or in
+percentage points, depending on the improvement mode.
+
+|ha-auto-rebalance-margin-mode |relative |relative, absolute
+|_Improvement mode_: whether the imbalance improvement is relative to the
+current imbalance or absolute.
+
+|ha-auto-rebalance-cooldown |0 |>= 0 seconds
+|_Cooldown_: the time after a successful rebalancing migration in which the
+moved HA resources are not moved again by the load balancer. With `0` the
+cooldown is disabled.
+|==========================================================================
+
NOTE: It is planned that load balancing will also include non-HA resources in
the future.
--
2.47.3
^ permalink raw reply related [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-10-18 12:48 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
2026-10-18 12:45 ` [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 02/14] manager: factor out checking whether a resource motion is queued or running Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 04/14] fix #8059: manager: auto rebalance: do not retry failed rebalance motions right away Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 07/14] test: add auto rebalance test case for a marginal absolute improvement Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 09/14] sim: hardware: report actual running state in cluster service stats Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 10/14] sim: hardware: allow setting a base load for nodes Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load Michael Ryom
2026-10-18 12:45 ` [PATCH manager v2 13/14] ui: crs options: expose the auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations Michael Ryom
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox