* [PATCH pve-ha-manager 1/2] fix #7974: run make tidy @ 2026-08-28 12:32 Dominik Rusovac 2026-08-28 12:32 ` [PATCH pve-ha-manager 2/2] fix #7974: env: pve2: set dynamic service mem value to host memory usage Dominik Rusovac 0 siblings, 1 reply; 3+ messages in thread From: Dominik Rusovac @ 2026-08-28 12:32 UTC (permalink / raw) To: pve-devel Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com> --- src/PVE/HA/Manager.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 9a4f70c..5840a76 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -146,7 +146,7 @@ sub get_active_stationary_movable_resource_bundles { my ($ss, $sc, $resource_affinity) = @_; my $resource_bundles = {}; -OUTER: for my $sid (sort keys %$ss) { + OUTER: for my $sid (sort keys %$ss) { # do not consider non-started resource as 'active' leading resource next if $ss->{$sid}->{state} ne 'started'; # do not consider resource if it is not movable -- 2.47.3 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH pve-ha-manager 2/2] fix #7974: env: pve2: set dynamic service mem value to host memory usage 2026-08-28 12:32 [PATCH pve-ha-manager 1/2] fix #7974: run make tidy Dominik Rusovac @ 2026-08-28 12:32 ` Dominik Rusovac 2026-09-20 15:24 ` Michael Ryom 0 siblings, 1 reply; 3+ messages in thread From: Dominik Rusovac @ 2026-08-28 12:32 UTC (permalink / raw) To: pve-devel Make memory accounting in dynamic CRS consistent. Atm the imbalance value that triggers rebalancing is derived from the guests' host memory usage, which corresponds to the respective host-side cgroup footprints of the guests. However, the scheduler's decision procedure predicts the changes in imbalance caused by migration candidates based on memory usage (inside the guests). This discrepancy can lead to oscillating migrations, as described in detail in [0]. For VMs, host memory usage is the relevant stat, as it better describes the actual cost of running on a node than memory usage. Host memory usage is not available for containers, so fall back to memory usage for containers. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=7974 Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com> --- src/PVE/HA/Env/PVE2.pm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm index 782d19d..5f195da 100644 --- a/src/PVE/HA/Env/PVE2.pm +++ b/src/PVE/HA/Env/PVE2.pm @@ -46,6 +46,7 @@ use constant { RRD_VM_INDEX_CPU => 6, RRD_VM_INDEX_MAXMEM => 7, RRD_VM_INDEX_MEM => 8, + RRD_VM_INDEX_MEMHOST => 15, }; # rrd entry indices for PVE nodes @@ -598,11 +599,14 @@ sub get_dynamic_service_stats { # NOTE the guests' broadcasted vmstatus() caps maxcpu at the node's maxcpu my $maxcpu = ($rrdentry->[RRD_VM_INDEX_MAXCPU] || 0.0) + 0.0; + # designated fallback for containers + my $mem = $rrdentry->[RRD_VM_INDEX_MEMHOST] // $rrdentry->[RRD_VM_INDEX_MEM]; + $stats->{$sid}->{usage} = { maxcpu => $maxcpu, cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu, maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0), - mem => int($rrdentry->[RRD_VM_INDEX_MEM] || 0), + mem => int($mem || 0), }; } -- 2.47.3 ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH pve-ha-manager 2/2] fix #7974: env: pve2: set dynamic service mem value to host memory usage 2026-08-28 12:32 ` [PATCH pve-ha-manager 2/2] fix #7974: env: pve2: set dynamic service mem value to host memory usage Dominik Rusovac @ 2026-09-20 15:24 ` Michael Ryom 0 siblings, 0 replies; 3+ messages in thread From: Michael Ryom @ 2026-09-20 15:24 UTC (permalink / raw) To: pve-devel; +Cc: Michael Ryom On 28/08/2026 14:32, Dominik Rusovac wrote: > Make memory accounting in dynamic CRS consistent. > > Atm the imbalance value that triggers rebalancing is derived from the > guests' host memory usage, which corresponds to the respective host-side > cgroup footprints of the guests. However, the scheduler's decision > procedure predicts the changes in imbalance caused by migration > candidates based on memory usage (inside the guests). This discrepancy > can lead to oscillating migrations, as described in detail in [0]. > > For VMs, host memory usage is the relevant stat, as it better describes > the actual cost of running on a node than memory usage. Host memory > usage is not available for containers, so fall back to memory usage for > containers. Ran into the same issue independently while deliberately stress-testing CRS with ha=dynamic on a two-node PVE 9.2.20 cluster (pve-ha-manager 5.2.5): a single dominant VM with an active balloon driver was accounted with mem = 6.94 GiB while its host-side footprint was ~8.15 GiB (QEMU RSS 7,950,016 KiB). The ~1.2 GiB difference is the same order of magnitude as the balancer's trigger condition on a small cluster, and contributed to the VM being migrated back and forth - five moves in 27 minutes on an otherwise idle cluster, matching the oscillation described in the bug report. Tested an equivalent change (memhost with fallback to mem) on the same cluster: the predicted post-migration imbalance now matches the measured state, where it was systematically off by ~1-2 percentage points before. Tested-by: Michael Ryom <Michael@RyomHerold.dk> One note: with the memory accounting fixed, the oscillation shrinks but does not fully disappear here. With a dominant resource on two nodes, sub-percent fluctuations of the node base loads (ZFS ARC, Ceph daemons, the IO of a rebalance migration itself) are still enough to qualify a motion - and, once carried out, its reverse - since the relative margin provides almost no hysteresis in that regime. I am about to send a series addressing that (plus the endless retry of failing rebalance migrations); one patch in it overlaps with this one, which I will note there so whichever lands first wins. ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-09-20 15:26 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-08-28 12:32 [PATCH pve-ha-manager 1/2] fix #7974: run make tidy Dominik Rusovac 2026-08-28 12:32 ` [PATCH pve-ha-manager 2/2] fix #7974: env: pve2: set dynamic service mem value to host memory usage Dominik Rusovac 2026-09-20 15:24 ` Michael Ryom
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.