[PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn

[Michael Ryom <Michael@RyomHerold.dk>]

The automatic load balancer of the dynamic scheduler shows several
problematic behaviors on a small cluster, found while deliberately
stress-testing CRS with ha=dynamic and ha-auto-rebalance on a two-node
PVE 9.2 cluster (pve-ha-manager 5.2.5):

1. A rebalance migration that fails (e.g. local CD-ROM attached) is
   re-issued every ~40 seconds, indefinitely - 25 identical failing
   attempts in 20 minutes were observed.
2. A single dominant resource oscillates between two nodes: the
   relative margin provides almost no hysteresis (a base load
   difference of ~0.5 percentage points qualifies a motion and later
   its reverse), the point-in-time usage samples are noisy, and the
   predicted post-migration state is systematically off because the
   balloon-adjusted guest memory value is used instead of the host-side
   footprint. Five moves of the same VM in 27 minutes were observed on
   an otherwise idle cluster.
3. On an almost idle cluster, the imbalance metric (a coefficient of
   variation) is hypersensitive and kept issuing barely-qualifying
   migrations with nothing to gain for any workload.

Patches 1-3 address the acute symptoms: track the outcome of motions
issued by the balancer (exponential per-target backoff on failure, a
per-resource cooldown on success), use the host-side memory footprint
('memhost') for the dynamic service stats, and require a minimum
absolute imbalance improvement in addition to the relative margin.

Note that patch 2 overlaps with Dominik Rusovac's pending fix for
bug #7974 ("env: pve2: set dynamic service mem value to host memory
usage", 2026-08-28), which reports the same root cause independently;
happy to drop patch 2 in favor of that one — the rest of the series
applies either way. The oscillation does not fully disappear with
correct memory accounting alone, though: the noise-driven variant
(sub-percent base load fluctuations under a dominant resource) remains
and is what patches 3, 6 and 7 address.

Patches 4-5 extend the simulator (actual running state in the cluster
service stats, like the RRD-based stats on a real cluster; an optional
node base load) so that the following behavior is regression-testable.

Patch 6 decomposes each node's load into the usage of the HA-managed
services running on it plus a residual, and smooths only the residual
with an exponentially moving average kept across scheduling rounds.
Known structural changes take effect immediately; only load that cannot
be attributed to any HA-managed service - which drives the oscillation
but cannot be acted upon - is averaged. No existing expected test log
changes.

Patch 7 changes when the balancer acts at all: only while some node
reports a cpu or memory some-pressure (PSI avg10, already broadcast in
the 9.0 RRD schemas) of at least 10%. Load asymmetry by itself is not a
performance problem; the pressure stall information directly measures
whether anything could run better with more headroom, independent of
cluster size and load level - akin to the contention-driven approach of
other schedulers. Deliberately only node pressure is considered, since
a guest saturating its own vCPUs or memory cannot be helped by a
migration. Fail-open when no pressure telemetry is available.

The series was validated on the live test cluster: failed motions back
off with the expected 60/120s delays, transient spikes and sub-dead-band
asymmetries no longer trigger, real node pressure leads to a single
strongly qualifying evacuation about two minutes after the pressure
appears, and an idle night (imbalance metric standing at ~49% the whole
time) passes with zero balancer actions. The full src/test suite (158
tests, including 5 new regression tests) passes.

A related qemu-server patch (sent separately) handles the case where a
rebalance-issued live migration cannot converge because the guest
dirties memory faster than it can be transferred.

Known remaining gaps, kept out of this series and described in the bug
report (bug #8059): candidate migratability is not checked before
scoring (local CD-ROM etc. - the backoff turns this from an endless
loop into a bounded nuisance), migration cost is not part of the
scoring (the balancer prefers the largest imbalance improvement over
the cheapest adequate motion - see also bug #7650), and a fully
guest-centric scoring model
(per-guest contention score, DRS-style) would be the natural next step
on top of patch 7 - the required per-guest PSI telemetry is already
collected.

Changes are constants for now (backoff/cooldown durations, dead band,
minimum pressure); they could be exposed as ha-auto-rebalance-* options
if preferred.

src/PVE/HA/Env/PVE2.pm          |  ...
src/PVE/HA/Manager.pm           |  ...
src/PVE/HA/Sim/Hardware.pm      |  ...
src/PVE/HA/Usage.pm             |  ...
src/PVE/HA/Usage/Dynamic.pm     |  ...
src/test/...                    |  ...