From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9D69E1FF0AD for ; Sun, 18 Oct 2026 14:48:47 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E1B4421664; Sun, 18 Oct 2026 14:47:44 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=InC44mq4ZVdXJCrgnNPT5FAAy4y6oxm7KcKDMzGZSUyC9FzlSh7D9hXBRi2GaxzuVDLftxYle5k4vQoldaVo7XC4FBgkB93ZLlT2Mp4vzVfMjup1x8vWalgXPslNtay7/f1YYACvYoR+EPTcyaJcf4gkGwg/vJ5FgN7S0Syp0WfiGWij/vwCW52ag3xUHwTY4U3BaCEdmmMiwf1EnAJfaae58HcZBJ154/mmIGzgPfj2esFh+T+gZk+9MQ29ZIZDFQd3QfDw2jShE1sTeEFzB7zE9UuJ3H7iq8dXMNlUS0gy/vnDtIDiVS6E7UJYkfheO7J3crMEM3d6SQ09+Qyzmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F/llT+FvK1TARKo4AqPhfB+qipVljmv3Vwb3z7PBnWg=; b=qxE4WPiXg0MDCtMnBDJX/K5m4Wtz2qf3YDk0+g0+hEkOKNvFrh7WZg1JmgyYC97jkrQj4SYjQUblgE1Ax3tYE3IYa63Ymdy72lIkQSSbnnQ791LZChqdDtrn6S6awR5v3BnkdooLzVq/LxGyqX0vPKtp0knY6oJtfF8R2K5DOQNaocb0uHTXIWTUlFjKIvY2E3rRmgLozDJ9WtSBLi5RURYlAFMBWLI6+lBLOFmKsOwM+2/+m07WrmA3izvfDL8laY5cOL63T8S7QYugPelGP8zg1US2DY4L9wn0ig1Eb+U4ehHqGnCb5225SgyBOrNiAeS1Jql+WCHNywoRYuO/6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F/llT+FvK1TARKo4AqPhfB+qipVljmv3Vwb3z7PBnWg=; b=i71Ni3Gp+eNd81cFhzrY6wVG5o/u/x3O6QkhWL6+zH5YfbuvcM6pEylpQRFN6zYxoLFBFKIGDfDdgnzHmUMf5OOdm9MxcK3Hlv1X9X87CmRhJs6Jo8wkOVkL9rh5w5r0gwKNeRruYwyrVHdn4oGQaAArJNAG128BwHD1/ctw9fWU8xTTFfQirvZ7g82Oke+jAUPHNTIQZ4TMrNjny5UmJLCs8oHI2UUWybcq9yfzwKhVSTu866zxXZTmYs9z88WoxI23U3w7Y4JL8dee5O3YAXvRVkbAGkVzYfN+a/rRI4zqe4neu2v8DNJeshcVzTVDpZUECX3fD8m8Xlkydf6qmg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations Date: Sun, 18 Oct 2026 14:45:58 +0200 Message-ID: <20261018124559.24305-15-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk> References: <20261018124559.24305-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: d457de32-6596-414a-f9e2-08df2d15ce1d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|22082099003|10067099003|18002099003|261009223027099003|261009221227099003|56012099006; X-Microsoft-Antispam-Message-Info: 3+e/NqVuFWOuGLlKcfcFrFRkjQ4vgGrjsZDtTkrqHa7E0w2wIvfnfYUNi6q2kiK/CO4xco6+QmgCHMnG0TY9OMYMDw9ONQTq8ExNkgqOeoKym57KU2i6jUriRqSijkaRgbi/xVxAdVHvaKP9wsWCfEi5AuYpjp+lxWxzfd4TGIrAJKFoGavYqzmQMQ0+BsHX1oQurvEWh7ZsfAFnfadS0NXY6pKd14zjqulZd6F2P+7lwcFTdpsfpXKv95w9D0PM/lKL7ooAxGMOcZ8d6QZQZ5Rt3sLbk8owSPlYqFGpeYVC25YuqSIhECeSkKIoGdHsh+GbgitKOY60chsCz/asOmIW0gz7EfHnFHM+Q/bmTgQoyk1D1pUO/6nMcddxHoxjtnafbKkWZ4268mqTDAwAf+p4shy/98tamtmCAP4gwUhrUZz3PF6YeX8KIUEuFGCcuYJGOEN6ttiaSB/+ARp1tiZ12ORgL5u+Bdre8hFCPkLwthactQrqowbYIx1Nc3Z1hejjUBwBKibNM6VHew0kTxKYUGcUncGtrEggQkkFQSjUB4MvQ13yPmXyf615zQ4KFnrmJdwv9/h4r2Uvf13JIFmUAy8OrDMq1o4DOV+h6/f/+kOOdzNgUsxLWKfKk3FUXHmNwwA6uDBPfYzUSTSJe6MdPBmrHr2/ubVWLFma2GI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(22082099003)(10067099003)(18002099003)(261009223027099003)(261009221227099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?g19OoufnzOBrePfjFpeuUjdXkC5MvXm23Gl3WY8cZeGal7bTFuDRIEo5pQ7R?= =?us-ascii?Q?FCsR4bcQYNHh4NZE1fjYbBExNGoGtSzfIZCmSpylA+KR6bpvSEy4ebNthviC?= =?us-ascii?Q?dvjVBhoXxtpHxomJZF6Ai8eLG23jG1iwq6w5++W6jTbfllGEaWUkXLATau/p?= =?us-ascii?Q?0GE9h3r+YXl53/1rf8qsm74qdDCvyIex3sd4latNsm/Cyh3bJie1cVMEANsx?= =?us-ascii?Q?WG5lO/jJpAeKMjDw4TRQ7MPsknvnGC0BRqbb5X4X1g8BjLN4cFXeBRPmRYMg?= =?us-ascii?Q?Q4w9tujJ/4aH7YoVxcliImSn/HcO9chnehCHd+Y1ZXm3IBsN4pRBz12wig5o?= =?us-ascii?Q?A2ExweRX0/vOZqGW5if4RD8ubZc8R5sTh8uSDL4OBIWxYk4OTtoy0BYve537?= =?us-ascii?Q?/yAHLLCWWPjzjyUYcGa4htQmuBAZ/HhutOSqLwgS8hCuQ2x/lc7WyiIX/syG?= =?us-ascii?Q?RdiXwztrHDd3HkENgQ6N1uC9hTaMJs3a7r/1Vixy/bdgEc+OIBltPr9GVsD+?= =?us-ascii?Q?2m528BbW7knbp61/PrQtfoKe5qwTWxpe58PXk3aIBbjanwTZG9gwT/3AlZ+d?= =?us-ascii?Q?yQr3uSuv9zp6jeNhEvZtIyUSMpszi01RjUGsjYZXwVeAw/kvBbBamOKHlkHg?= =?us-ascii?Q?D3ILLP5mXXCm8awWZxMVFdkeD4DH2WZ/vquIT08OqBrcqcz4RZZ+7PGPYANa?= =?us-ascii?Q?M1oNpSYtcATqeRUChVRkZueKnt2YnhEnXhzVlVbn/88/QXmZnYFClNExkNr8?= =?us-ascii?Q?ZdWwjcc5R4Y3t2avSfoTbTGFFF+7t3Kvr3Ik82lnjfVWG7/gvJNZaKK/XlgD?= =?us-ascii?Q?u/jWw5Ibi/1DSwsUs6yKnMhFZ6/HEKSHktGoAtCo3Dch+bn3cA9JX7P8ALgm?= =?us-ascii?Q?5mN6tQpJ8jsxJB75aSeOhuk3W57CpeVzjaYTj2aweWP8y/aWvreDmgrPSQfb?= =?us-ascii?Q?wHeIUxoPlzmyzEYU5VwzC1eF0XGq2693Eu8wbFgF9HJyOSWPdtenWT4x8vYs?= =?us-ascii?Q?XRawHunKvY/r8jktPs2pbsB8lQuJhrZ/KdQSe/vRYFy++yUN8BW7mJJMW73/?= =?us-ascii?Q?mMRXLUYTiPsXRBAvAZzdIMZtx6BfX2FTPFl6ubbPVuMSUpeb+48swUMMjROC?= =?us-ascii?Q?FNJSGsYtAMqZKvtb0m07z+oPoYPVeeTDYLsOINtBIijBgXBDG+Dj9oEAZkjL?= =?us-ascii?Q?zA5MSt0VuKkzTS9y5/ki2zd5RxObLqdzAx11rabiWmlN4ToaSgGTAr+hoh2I?= =?us-ascii?Q?SQkpVHqXE+bZW1OVBB60zGJn/qQGes/BSbCrXSkX68wLyM5gnxo1BHqA3t6i?= =?us-ascii?Q?UWVmkxwE/JJOd+la2qMpXzbLRoiu2VHh5JJpbpb/QmTDemZqfMd/Y36rWar6?= =?us-ascii?Q?1US5/YOI4CZDvuEBLwoR/h+siAOc+Vn+dHo5QyQodiYwWmglA0fwQs2BtUQZ?= =?us-ascii?Q?JMK+owbQJXEXwAlZWuCS4Y0bEMD89dJKJzdhS4SKw8mxVWCcqj4+DtTnWazj?= =?us-ascii?Q?W0Zza1uuWD/Afp1Oxh9b+XjAdb/q2yhLdUHcFVqAK41yzxl4ndB1bTe3Sjgo?= =?us-ascii?Q?xSP50Es1G9StZBCWMkvGl+HT4vnbpBDWUvppsdcZPemMzOlwy7dKW2CpdPPm?= =?us-ascii?Q?8Pa9kOUJEvveWi0t4+jtdoPH9sCsSSIBnoyRz77A/GT+3Br/soR+mTC4QnYD?= =?us-ascii?Q?LLUrjXQaM6pkKy0MgnwIwhW985/iZYUVs9IClMoemQ4qkdHZcewv6pAD2uQV?= =?us-ascii?Q?kndaJvTa1g=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: d457de32-6596-414a-f9e2-08df2d15ce1d X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:18.6441 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: P4R+7wF5AnmWmr5riuLyPYrWuufBWkph6ajay6BbV4ZFRUESUWQOgvvI4t9whj1AtjJvr85gm9qdnuabpvegcw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.403 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: B464DBXN26DORX3UIX2WWMB6HCKQRW57 X-Message-ID-Hash: B464DBXN26DORX3UIX2WWMB6HCKQRW57 X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The relative imbalance improvement provides almost no hysteresis when a single HA resource dominates the cluster load (see bug #8059), which the new ha-auto-rebalance-margin-mode and ha-auto-rebalance-cooldown options guard against. Also describe how the load balancer handles failed rebalancing migrations now. While at it, add an overview table of the load balancer options, since the surrounding text refers to them by their user interface labels only. Signed-off-by: Michael Ryom --- ha-manager.adoc | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/ha-manager.adoc b/ha-manager.adoc index 95ab799..bbf1ce1 100644 --- a/ha-manager.adoc +++ b/ha-manager.adoc @@ -1614,6 +1614,68 @@ the expected imbalance does not change at all. The value can never be negative, so the balancer will never commit to a migration with an expected imbalance worse than the current one. +By default, the _imbalance improvement_ is relative to the current imbalance. +That provides very little hysteresis if a single HA resource dominates the +cluster load: moving that resource in either direction can look like a large +relative improvement, even though the cluster ends up about as imbalanced as +it was before, so small fluctuations of the node loads can make the load +balancer move the resource back and forth. With the _improvement mode_ set to +`absolute`, the _imbalance improvement_ is the number of percentage points by +which the expected imbalance has to drop instead, which ignores such marginal +migrations. Additionally, the _cooldown_ option keeps HA resources, which were +moved by the load balancer, from being moved again for the given number of +seconds. It defaults to `0`, which disables the cooldown. + +If a rebalancing migration fails, for example because the guest uses a local +resource that is not available on the target node, the load balancer does not +try the same migration again until the imbalance has exceeded the threshold +for the hold duration anew and no other rebalancing migration succeeded in +the meantime. Instead, it tries the next best rebalancing migration. If the +same migration keeps failing, it is skipped for exponentially more of these +rebalancing attempts. + +[[ha_manager_crs_load_balancer_options]] +Load Balancer Options +^^^^^^^^^^^^^^^^^^^^^ + +The automatic load balancer is configured with the following options of the +`crs` setting in `/etc/pve/datacenter.cfg`, which can also be set in the web +interface under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`: + +[width="100%",cols="3m,1d,2d,5d",options="header"] +|========================================================================== +|Option |Default |Values |Description + +|ha-auto-rebalance |0 |0, 1 +|Enable the automatic load balancer. The options below have no effect unless +it is enabled. + +|ha-auto-rebalance-threshold |30 |0-100% +|Cluster imbalance above which the load balancer becomes active. With `0` it +is always active. + +|ha-auto-rebalance-hold-duration |3 |>= 0 rounds +|Number of consecutive HA Manager rounds, each around 10 seconds, for which +the threshold has to be exceeded before a migration is selected. + +|ha-auto-rebalance-method |bruteforce |bruteforce, topsis +|Method used to score the possible rebalancing migrations. + +|ha-auto-rebalance-margin |10 |0-100 +|_Imbalance improvement_: the reduction of the cluster imbalance a migration +has to achieve to be committed, in percent of the current imbalance or in +percentage points, depending on the improvement mode. + +|ha-auto-rebalance-margin-mode |relative |relative, absolute +|_Improvement mode_: whether the imbalance improvement is relative to the +current imbalance or absolute. + +|ha-auto-rebalance-cooldown |0 |>= 0 seconds +|_Cooldown_: the time after a successful rebalancing migration in which the +moved HA resources are not moved again by the load balancer. With `0` the +cooldown is disabled. +|========================================================================== + NOTE: It is planned that load balancing will also include non-HA resources in the future. -- 2.47.3