From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 5D3A01FF0AA for ; Tue, 22 Sep 2026 12:10:11 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 29EC2215BC; Tue, 22 Sep 2026 12:09:58 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cbA9ELpBaIHkwt8o22F/0MGcKnW2hvKway2yTOlEo0JyIZA29chJLA3SN7YudHVrgpMAyDrURTMVmqvhIsFRy1kBv/TwTrIM1bX+YNV5vfyLzmU9vMKa0WofZ79wsnSeUYXd+d6ViGIw+hrI6tDNiJZ6J8u6H4VBVX2iCt0vQSAkujIYCvV9UiruJfTTP8PX99WMEHYnHgEO9vXe5DCgXmtvjeKuszUREuZdKtO9WxKBTMMZKoOXah5/zsRY3i/zP9JeomwJfGz9AP1fWSxVShEsXCp0BenleWy9zsoeR80CzclaPDU+prTB+eEO/PKmCtl2IzcuEomcAD+okyHrcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=l5Q+8UMpyoGdwRdyjMtPOSSuTeHZ0qaVTRsufpk6Kdc=; b=xi5/n6pc8KhVJCWWwUuxvPZmrqeY2lPR0zCN+/+N+TQTywgvnTuFWOdNqP+WrWDAgO8X2uKtTzu6YEuBGRFH8Kgr+tHFld6GxxtWAs8X4bvhSjKSGQ4ryiLQzp6tmneaVyhw+6mkmlV7hQgHVCgdsER5ilWpw8ptp2sPt0lxNGO/2skg32IhLEmVWW0KKqE6/IEYkSGBLIrJKnER+7i8aAaS4Pa1Zvzgg5zoPPmMhQPxIdO/cxe9CVnens4Y+esISvYQuuzxPOQ0wH4ZGDWKztzcECq4D4YzzWhM2UBhN5TqMll1Q5zWXwOmsfk/V/bfFa/n+Xmiyun1OQuT1rBRKw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l5Q+8UMpyoGdwRdyjMtPOSSuTeHZ0qaVTRsufpk6Kdc=; b=rOd2EcHa3I3Y93ORkHLzTJfT8FVK0le8mEuhwnFdNcwNRb7Mz9lAcbwDHLMO//eqFvtsexLVM1LZxRGACkpOtk8aqWNOGC6Av4LyGaCUhRZivaOUxi675yoS8e+buVUQ9FyA8Yla+Xe3ZYKVGa5v4pboUsyh9d+HhBEfMH3VW93gtkhP6GyjGpmXP2HYNKAwwyyp4dO2HGusdTOccyujsr23K2TRGbAUjp2q4OWpvdSwAw2EZkGmF5fbRLGrryEKpdQBD2ms1ttKhd6BbW+dIuff+QPx8ck3BQGXVYI1cJ4ZttQMWg/eOEHckLansWnRpM4BHtMVu7smin5R291rbg== From: Michael Ryom To: Dominik Rusovac , "pve-devel@lists.proxmox.com" Subject: Re: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement Thread-Topic: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement Thread-Index: AQHdSRxMvY8UFgl5BUCVTosQg88YuLbaIkmAgABAUcg= Date: Tue, 22 Sep 2026 10:09:40 +0000 Message-ID: References: <20260920162220.574802-1-Michael@RyomHerold.dk> <20260920162220.574802-4-Michael@RyomHerold.dk> In-Reply-To: Accept-Language: en-GB, da-DK, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: AS8PR10MB7231:EE_|PAVPR10MB7418:EE_ x-ms-office365-filtering-correlation-id: d6eabbad-150a-40a2-0750-08df18919deb x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|1800799024|376014|23010399003|366016|56012099006|6133799003|38070700021|3023799007|8096899003|10067099003|13003099007|4143699003|22082099003|18002099003; x-microsoft-antispam-message-info: J8Q/TbxnZHa10PiwlvRXIdTjgt0Si48Swr0/LF1F9OuEDswtNWaNgrjCKFKGBWPLXPeFwGQg0fKRVesmSNb4ubeFaX23qR1wfayTXwY8am4pv5L2o//b/HTlN749ipuZPGCvWEM6e5BAjF6wzzuNk1+c2sf4A57cA4B2fITBBAkZ015liHj2AoJhhZRwDQtqpMDOpZD+w2b/bF//QEhgl4D0EfOtlwbSmeIf9k4BOYSg7LH2XM+jUQHilJzDEj/PPuEThGJgpGjMpHZ9KS38Sf4j2pcVr2MISeKZVxsJ+Whkk90YWOgZkGhbz4cgTX0pQ9t1d+aYoTYnjnzM6JerRgbXFp0dbnpPgprkPv9QPKFob/vPMa1oGPurtipkHItiSHzRVJbK4kFSV58T+C0i958ZMGAKemFlM316czxHDXO00J2N58+02iLdOOeL72V9vwIwOl3ZJyNBsewLkM7XjtuX3mUNJDUmL2OylP7YCAiEZBV/Id75ijYy7EihmqcsKkKqGbbhJqlI5zI+jlWp0UDEblzHGsolX11iVxUwkR2FfbrPyEr7/fUKdDxvZDO3httL4n6hQmwbo900B+H9KNq7OD8YPt71z28GH0pnsJZVb9D4klpjIhMtLFc0lN+o+On9bXZTKPkzL7BceosVXGlvboerR87mwGiRMPiAxFs= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(23010399003)(366016)(56012099006)(6133799003)(38070700021)(3023799007)(8096899003)(10067099003)(13003099007)(4143699003)(22082099003)(18002099003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?/WsGfIQYeUA6nZY2MzsELeretDJflYbGDsScdem1UPPmmr98IcBV5ZHj7XQN?= =?us-ascii?Q?6roV03fK2aQ9Sw/U3tcJGb32MoHyByCO2gkekNtn4rUMW+lKkg2AkjoTeLtk?= =?us-ascii?Q?mYtbf13VkT9Mh4UHkaQD8D4bVL+2HGiVnoGk577IQpPyGRorY5TMednhGtsF?= =?us-ascii?Q?9RANHMvFG7HKB+bw/wLS2m5LXwVI/jNyWvUIqzHnY8GS11sGcpZwmmOr9MJI?= =?us-ascii?Q?tV+j4+HAr/uU9u+L/4t9N0DT2+6O0YxPOBJv0yKFBHP/lTxpR2TBIiyGTIbI?= =?us-ascii?Q?lAjzlHHpEtvv9Z0eG+XhaNqFKPVxFOtg0rFYdnYfPePOGi19Y/Eb54YEByes?= =?us-ascii?Q?Lm1H3Ik1sjfyrYvBQS1IwCU5eWMlln3ogtkIFam45qF9gc1nOMbyjRuT7cT9?= =?us-ascii?Q?iY6FJtj+6rLmjjzpKLT68Xa3gMeLqyYdnJ0BEqxCov92wgMnGidH+QMAMBcU?= =?us-ascii?Q?rEqMdoi8vhPcAwZXlQV9Yr1RPJzZTFncHIkW6IXckNtqcMpNl806VkY6eUiQ?= =?us-ascii?Q?VzlFnVjkZ+iTOBxDSrvEvi5zv/X4eLyhRup07PJOvxr+3WCeU4IdU9xHo2wV?= =?us-ascii?Q?UVMjlqHtqsyR6NgjkJisTtyLXc118+vKEFyb52UrSOuCbV6RPBeqojNceQgz?= =?us-ascii?Q?UnNImKqzaxIouitOz3p6akJBk48dO+4/X8RsI8ND3ACRrKydGIfcr4279TWw?= =?us-ascii?Q?xBX2y4ZuiPXelJjJarLYNk1uhh8IzNvdf25TwzLlg2AE+n4OG/GUikNJY6OQ?= =?us-ascii?Q?2geP+wML2tVOSiF0M1HTyEmp0IX+K68gi6Aokx2qX7jJnMkocQ5VuhzWO6DE?= =?us-ascii?Q?czLAqRnvlVj05mC7Tt53P5VDxbIB41sdImC3hdn3Q0sFAJjPFbXzU+McLvVq?= =?us-ascii?Q?azRKqeqQ/YjYCtncNPoTLoICS6f3cQ2mWJobe8qFop5R1zv06J1ecnMkPmao?= =?us-ascii?Q?RFznmIbm+DiWo2nG29wQ9eDq2DcKrP90VWoMCEOoXPmnY7OvbaO1zAEXftFH?= =?us-ascii?Q?8J8gcd2fwM2hekQqTlrsC++5uIB6kUr6vY2+1zwaeznYI5pJFrSBtiB6CHLW?= =?us-ascii?Q?ETaOkOCMpNyVZRbYvtFxJ8jIkQKKpPzEAn73J/0LQaK4049HYQH1oU2I34lh?= =?us-ascii?Q?q9Vvl8D4eZ3Ip0cW6p4BEN+facpxdRfxNu/0dDeY2q1qJ+JukTQCT0tBMl4p?= =?us-ascii?Q?GUVcbw5veur1Lgfw+DTre8cd9zKX1Ts++3j2zz09RbFWgyQvSD5VzXMpiLDY?= =?us-ascii?Q?Rig5uMvuSu6Rruvn0iyxdYa8uJ7HAJJ/PvPdmsnt3Et1K3IgjU1wT9c+tJey?= =?us-ascii?Q?f8EVwdncT9oYRzcxznccDKco37wAq0fqu+UXIqtkUy4DPAlIhAI+pOGyWiST?= =?us-ascii?Q?7EVoQGv7XWNL4bvW9wd3sW3YSBW1MGaDXYeFN4i5nFjA4Gl1B6TsrS1QU9LF?= =?us-ascii?Q?3u9RKiiUqNLdCj9hJ1niP0OGwp+z/RtU7HxG3e6Tv2yv1iGun/jieXVYWMfp?= =?us-ascii?Q?eMdDiv5EaqVkI9V/J9xYENxYCSRDmiBU6STlBirBgvjf+v7tlDoq7UQDf4V7?= =?us-ascii?Q?m7CRmcVIWaeuHKvyOcyPz2GaInxaJjFWDxIbm2pmYRpon49K8fH1QfNBxPfI?= =?us-ascii?Q?2InxOz672CibhvffwCL/W8730qnfj1P6AJmiiwjujm8+nCmmA/kBW6fW82Yx?= =?us-ascii?Q?aoKMa0lw3XzwiF/FzhI4ri3bPWJzAeQM4/h3+R8QalFUKlab?= MIME-Version: 1.0 X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: d6eabbad-150a-40a2-0750-08df18919deb X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Sep 2026 10:09:40.8308 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rG8FpToe3V1/BK+npNrT5xZdaoyuvGeRxTVmTDDbJ7PnnrFtN8YpyvlZ4P1wMXhihxRSD5dBoVkHw4TkZbFMmQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR10MB7418 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.001 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: T5AFCLF6UWJMYDW5A2Z265G7C5USG3TZ X-Message-ID-Hash: T5AFCLF6UWJMYDW5A2Z265G7C5USG3TZ X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 3.3.10 X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Thanks for the review! rather than using a constant, I'd expose this as an ha-auto-rebalance option right away. I'd suggest using 0.0 as the default value, meaning this kind of protection is disabled by default Done in v2. The value is now the datacenter option 'ha-auto-rebalance-min-improvement' with a default of 0, so the guard is disabled unless it is configured and the behavior of an existing setup does not change. The regression test sets the option explicitly, since 5 percentage points is what stopped the oscillation on the reported cluster. This turns the series into a cross-repository one: a new patch 1 adds the schema in pve-cluster and a new patch 9 the matching UI field in pve-manager. pve-ha-manager falls back to the default when the key is absent, so it still works with an older libpve-cluster-perl. [factor out the absolute change] Also done; the difference is computed once as $absolute_change and the relative one is derived from it. One note on the follow-up work, so the option naming does not have to be changed again later: I had reused this value in a follow-up patch as the share of a node's capacity that a motion has to free in a pressured dimension. With the new default of 0 that would have silently disabled that requirement as well, so the two are now separate options - the follow-up introduces 'ha-auto-rebalance-min-relief' (default 5%) for the pressure relief share, while 'ha-auto-rebalance-min-improvement' only ever applies to the imbalance metric itself. v2 of the series follows; the memhost patch carries your Reviewed-by, as the only change there is the suggested '//' instead of '||'. ________________________________ From: Dominik Rusovac Sent: 22 September 2026 08:18 To: Michael Ryom ; pve-devel@lists.proxmox.com Cc: d.kral@proxmox.com Subject: Re: [PATCH ha-manager 3/7] manager: auto rebalance: require a minimum absolute imbalance improvement thanks for all the helpful details in the related report [0]! see comments inline [0]: https://bugzilla.proxmox.com/show_bug.cgi?id=8059 On Sun Sep 20, 2026 at 6:22 PM CEST, Michael Ryom wrote: > The automatic load balancer only requires a relative improvement of the > node imbalance (default 10%). When a single resource dominates the > cluster load, the relative margin provides almost no hysteresis: with > two nodes, a resource load v and a node base load difference d, the > relative improvement of moving the resource is 2d / (v + d), so it > exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM > that is a base load difference of about half a percentage point, well > within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and > each fluctuation then qualifies a motion (and, after it was carried > out, its reverse). > > Additionally require the imbalance improvement to exceed a minimum > absolute value of 5 percentage points. This creates a dead band for > motions that shuffle load without meaningfully improving the cluster > balance. No expected motion of an existing regression test is affected; > the smallest occurring improvement there is 9.6 percentage points. > thanks for unveiling this! the math deriving d > v * m / (2 - m) in [0] checks out TL;DR: in the given scenario the load-heavy resource keeps getting migrated back and forth between the two nodes, because the load balancer's decisions are triggered by background noise in d, rather than by a genuine change in the cluster's load imbalance that the relative margin m is meant to filter for to me, checking that the imbalance improvement exceeds an "absolute" margin appears to be a good way of guarding against this issue > Like the backoff and cooldown constants, this could be exposed as a > ha-auto-rebalance-* option later. rather than using a constant, I'd expose this as an ha-auto-rebalance option right away. I'd suggest using 0.0 as the default value, meaning this kind of protection is disabled by default [snip] > > diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm > index 30b9ae2..2c4c273 100644 > --- a/src/PVE/HA/Manager.pm > +++ b/src/PVE/HA/Manager.pm > @@ -151,6 +151,13 @@ my $auto_rebalance_failure_backoff_max = 3600; > # minimum time before a resource is considered for rebalancing again after it > # was successfully moved by the load balancer > my $auto_rebalance_resource_cooldown = 600; > +# minimum absolute improvement of the node imbalance for a rebalance motion; > +# the relative margin alone provides almost no hysteresis when a single > +# resource dominates the cluster load: with two nodes, a resource load v and > +# a node base load difference d, the relative improvement of a motion is > +# 2d / (v + d), so for large v tiny fluctuations of d around zero are enough > +# to make the motion (and afterwards its reverse) qualify > +my $auto_rebalance_min_imbalance_improvement = 0.05; > > my $is_on_rebalance_cooldown = sub { > my ($self, $sid, $now) = @_; > @@ -387,6 +394,7 @@ sub load_balance { > > my $relative_change = ($imbalance - $target_imbalance) / $imbalance; > return if $relative_change < $margin; > + return if ($imbalance - $target_imbalance) < $auto_rebalance_min_imbalance_improvement; nit: factor out the absolute change and reuse it my $absolute_change = $imbalance - $target_imbalance; my $relative_change = $absolute_change / $imbalance; return if $relative_change < $margin; return if $absolute_change < $auto_rebalance_min_imbalance_improvement; [snip] > diff --git a/src/test/test-crs-dynamic-auto-rebalance7/README b/src/test/test-crs-dynamic-auto-rebalance7/README > new file mode 100644 > index 0000000..488a018 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance7/README > @@ -0,0 +1,12 @@ > +Test that the auto rebalance system does not act on a rebalance motion whose > +absolute improvement of the node imbalance is below the minimum, even though > +the relative improvement exceeds the configured margin. > + > +The cluster has two nodes with a HA resource excluded from auto rebalancing > +on each (vm:100 on node1 and vm:200 on node2) acting as node base loads, and > +one movable HA resource vm:101 on node1. The imbalance is 35.6% and stays > +above the default trigger threshold of 30%, and migrating vm:101 to node2 > +would improve the imbalance to 31.1%, i.e. by 12.5% relative, which exceeds > +the default margin of 10%. However, the absolute improvement of 4.4 > +percentage points is below the required minimum of 5 percentage points, so > +the motion must not be issued. the numbers check out thx for adding a test right away! [snip]