From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id D74D31FF0AD for ; Sun, 18 Oct 2026 14:46:33 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 631F321491; Sun, 18 Oct 2026 14:46:30 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PM80pbiqX3iT+LkjZdfzAZewbqq+mHc9iBmfCDNPT8o3SIzc2TD3Wisgc7ETxbrYXhDmbLc89W4BygQsJeNyQROrtzJPrzuApK07y10G/poGrOP4Hr6IxQCtJAvESBxLcOGBOSQNeIltuT7b+ZF6Pa+QLs0cZ0/DCWhCs4DH4P3LZwIyg5ySZB7dLsYCMg1ga8O990vX6yO1hrsd0JmfP30C7egRt3wx1t0CKxaB9eloWc1zL05pd2Xsfr0TiaRI8VBGf1oODXQ2cnt92iY5hixf3BvTn8sSjkd1EYFxdCANyH26tKxj4vD0MW9V/DLYwUy+mSy7rDi1EDGZ21CoVQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zOvATbeT/1m4//CNj8ZOfzKdE5zy8GrT4b9/DxkI3Ek=; b=kHSue7nwDRGjLZ9nvfTJ2kbQsYgy6O3wi9XLPbwjUtwZdJzfcHIYFGguKwHVta2sGvw4EzUAbwgn1xp6kDPS3757reLvAYOT0b+NYIrd8aIe7OHatSV4p8LrPAR66FaKYO/X1Tsi5uOJjCbxLviSe1all5G20u46RrHezrEHgJY9Ig95kIlYBU6/JPFB9JOnbAMidlrsHuSWFy29q5qKj+maADGpNXMdhEsj0/hHuR+rHAAlnpfBiIAN3R9XTmJP/me9TtcDhkZffiaysRvBLy6u19FpAapq1FgGGyNhiUTdAzTibMLJUwnAOwjlxY2lRFHqAKes6uHt+o0gu0zzsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zOvATbeT/1m4//CNj8ZOfzKdE5zy8GrT4b9/DxkI3Ek=; b=ttfdioiW8vL9zFZYomulI+5/rvc1fB0DkRDkMboSP6/MFpjbu0IvopGfcxWhYjONziL//R6af30/TVFP0Wl8Go95NeTV5Sd3E/Lu5eSr19rcw8PivfJnuKdUhmiwSuB3aFq8E8XbRyFmPV0u74eI/uOjxAcZBFu9BH7J7m6WwV/5P6QJQxkJgmdw1kCntQMh34UwfZ1dvan6eNVgrs1lDe3I96O/q5YMDcjnMccvwhEOc0dXaJDVuPT3ZSqT1zhwRoLDu4LTd6zyotnVsbnA1TS9bAhrxgvU87/row2ynUGntsmZ/nEfVbA5FIOQO1EhxSIbMn80ygtRK9+GslTA6w== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Date: Sun, 18 Oct 2026 14:45:44 +0200 Message-ID: <20261018124559.24305-1-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: ca0f2bf8-aedc-468f-e576-08df2d15cb59 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|6133799003|10067099003|18002099003|3023799007|261009223027099003|261009221227099003|56012099006; X-Microsoft-Antispam-Message-Info: tZfDQTmG/U3zq73ALVuknzVSCcIvV1h1a0upNzYLPB21ESvVZ/6VXcFWevnLYj6x/Uw0X7rTe9IsQmp47QCeceQ80aJePn9tvXL8pCEjCDgIFaBQG8dTOdyqHtGR0HBU9IXZ1gVQXse3W6lj+e/QXRpSWeM1zBKddqlB+ctMTikuCK7si9uaHVo6psHVlV8mTjCOomCpa+t7jljCW+eLIMjmvv3EV5ElXR1xzuTjuNATA6VGew2cBhBo/INfee3B4yrmdgwabXRGQDWVJzEUMzBRmrrP5uTvODMIccAO4tVJk1kwza07UTpzGsRmvVhW/kvd/ZFQYm8PaGPXMTj3iJSXdM89czDGt/CNBTU7ADrneEME9cWHtgTV0H85Yw7kVB/PKQpfBS+Gb9uFLqI/GhnPMMXxZnklmq87FGmO00D4hzyFrlQPAaXDLg01dP3blhdv+z9KRX0ZVANPo4IY6Q97shazlXGbSWZ9gh82YYZloT3O2XIzxVr9ffReor/YDW6XJqCvmRCVHwb6AnCc5X5whs2OuiEm33xolqHe9ODYo+6P6gAeC25+xtX5zZXZ2sPgfbaCvdedcePt/BuBzLfTVKMd8MOxSi10lzUKsuM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(6133799003)(10067099003)(18002099003)(3023799007)(261009223027099003)(261009221227099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?DvIXaPIvHhqt/Tyrny13xXMKdFdvNwrloBrK5YkzRNDKPNhcc/J4sNBOydhr?= =?us-ascii?Q?zP+SFvAJvsPffJkwxwZgJaUThXPOBITA0W0l3OvkCiU53dKFNIfOSmjD59wO?= =?us-ascii?Q?+n633pWjKOXP+L9Bm+KhkEkiQc/4AW1PKRxwSCkA51Z46iVYgem+VLVlRDLA?= =?us-ascii?Q?iRbOh38ck/LaZELzLA+QDMGkXt1GgyTkUWca97VwiunQBkjSptzrjUL3jERr?= =?us-ascii?Q?T/bK48GKWpVi0Qjfnq8pc6Z1e6Mt1mI7eDAcQ1s5QKc95NxvXKoq8GIattEA?= =?us-ascii?Q?c02qKo3Oo5WzYVWNIZuEyRJn5ukp/hdH6Ml8dGoRZgM+ZPyYeRaco18iKNvb?= =?us-ascii?Q?wX2FLG6bnLUTRzfl0oSaa7nORln8uYm87q8YnW3IMM4kRZe7P76vu9Ch6hBI?= =?us-ascii?Q?0wEvKZHYXljmHImrAi2TJRWCLdd3Tn2iyOjL7P3AyrpG7vKpqL8RpKC0scV+?= =?us-ascii?Q?UZF4IStPas29O6aFWDifAsTc96acOq8coFa+d8pzxTcPDIj+xOa1EbOaNmmf?= =?us-ascii?Q?Kx/9OqV/IlBCWF8m85F2AMgGoIShNX0L+PgM54eO/zYm6mmeT/5Am0pu2pb4?= =?us-ascii?Q?Ys8MdxmTapu9ZHYGO6c/U19xIQvjIUSVHUIU6CJgptqlX5YQi/eZNkj9jXr7?= =?us-ascii?Q?655E2IM5KSt8HdOMunTgT5uomtEvFgTsGy9GPZ8h8GnkBz90n+MCD27jY+WV?= =?us-ascii?Q?dEG5vFPnSF/lq6lNsVUMdypaH+7mjOcLebUBWQytXsGg8nrTnDI4jS3ussKl?= =?us-ascii?Q?+3dmsbFiohVkyYC7eMaixd0biS1Xek7h6Mku7ame+1rE3hG7KuVx6fxmPIge?= =?us-ascii?Q?WfE1VU/AvADKqgDy00LUzXBFD8ia1klwCkaatduQ1+aNHYpcKYN7ZdZsNE6P?= =?us-ascii?Q?IRqqqtrc0GxV1Ffgoo4UAds+g6PP6rBz3s4c0udmFJdh+wkMU+hZxLYyB6z4?= =?us-ascii?Q?+l+tjEHcINV+jBJvlUnzS5/m0b/kEYPZX363rvgyVkH/IpP60nAAy+JZ7xzF?= =?us-ascii?Q?DDAZ+rWFyGkX2D0KHyIxSLDNMC9y11fc1Pxb4mUtfC06Un3yTOE5TRvVOSB8?= =?us-ascii?Q?DBrtwJM3u8dieqJMqQDDnVSHu0QsRwEQcl5VXN7kx0wzgOgS8wyyRG7TuKYy?= =?us-ascii?Q?LKFbRwK3VpxCV227mJdYkfrr907ao6sqlgjoDJWwHvk7rBqmp9WXUWto5+77?= =?us-ascii?Q?nJeoB1iLcHriR5z0Ppbhs13dgmI22AwuxyhsbHLGYHf2nWtkyhvJC2nwBj6j?= =?us-ascii?Q?Da6fE1drs4fjqizFzL7cCtXBTbovM8PGr37hkRt0ulxWFMNsbLinzySg4hj9?= =?us-ascii?Q?iJ3h0FJqvBNIvT7xNaOOW9qq/eHHTSLou+hziFmGH8pTEsnVWOLO72YixoDL?= =?us-ascii?Q?w5IovFLQGHHBuTzp+B6Qe3jyUNqLUpoTREqC0SGDU/y92wzNdRJw4wq2GnL1?= =?us-ascii?Q?wCKpZfAfkLLzoVTk2cEEi0p/5W0ELzhCibqF8E9We9ELURnK+hTPve3brGHa?= =?us-ascii?Q?zxrIE2TXxr+jNFXlWTSbULo5rPgjR6WluKWY3X6c3Hm5vRE2LFcQ6XEjAbq3?= =?us-ascii?Q?layPhUTi/RJOe1EA2eU3XW4pzf0Q1s697VzDVEdZGEvzZ9PKb2N0vW322Zmj?= =?us-ascii?Q?lPs784FR9TirBkoaMc/N43ullgZr03zC8gUN+AEf238K3AgPYvVBh6q0+i5e?= =?us-ascii?Q?Y+8uSCS4MzQi70vZdHbVfO0d/2Gq6Xu60Pib68be3CAmAhLrNANQhUsteaMJ?= =?us-ascii?Q?xlB2K++HYw=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: ca0f2bf8-aedc-468f-e576-08df2d15cb59 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:14.0331 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /WuqVdKp3Wy+mbUoiTILwoaCM+xercEs4pg0Eqv/mMHoyiOZQ9226Eud2w2T4PXXZUtaxD4Ex86/yfeRSsqPzw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.005 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NF6ZDNFBLKTXIVQZO5V3KVTP2GPBW2YQ X-Message-ID-Hash: NF6ZDNFBLKTXIVQZO5V3KVTP2GPBW2YQ X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The automatic load balancer shows two problematic behaviors on a small cluster, found while deliberately stress-testing CRS with ha=dynamic and ha-auto-rebalance on a two-node PVE 9.2 cluster, see bug #8059: 1. A rebalance migration that fails (e.g. local CD-ROM attached) is re-issued every ~40 seconds, indefinitely - 25 identical failing attempts in 20 minutes were observed. 2. A single dominant resource oscillates between two nodes: the relative margin provides almost no hysteresis (a base load difference of ~0.5 percentage points qualifies a motion and later its reverse), and the point-in-time usage samples of the load that is not caused by HA resources are noisy. Five moves of the same VM in 27 minutes were observed on an otherwise idle cluster. Changes since v1 ---------------- Thanks to Dominik and Daniel for the reviews! v1: https://lore.proxmox.com/pve-devel/20260920162220.574802-1-Michael@RyomHerold.dk/ * Failed rebalance motions (patches 3-4) are now handled per rebalancing attempt instead of with a time-based backoff, as suggested by Daniel: a failed motion is never retried in the same attempt, so the next best motion is tried instead; it is considered again in the next attempt, and if it keeps failing, it is skipped in exponentially more of the following attempts. With the time-based backoff of v1, two failing motions of the same resource to different nodes could take turns before the next best motion was ever tried - the new test-crs-dynamic-auto-rebalance5 reproduces exactly the fa:109 example from the review. * Failures are recorded in next_state_migrate_relocate() when the LRM reports them, for every resource moved by the balancer, i.e. also for the dependent resources of a resource bundle. A change of the bundle's leader no longer resets the failure state. * The cooldown after a successful motion is a separate patch (5-6) and an option (ha-auto-rebalance-cooldown), which defaults to 0, i.e. it is disabled. * The minimum absolute improvement is no longer a separate threshold, but a mode of the existing margin (ha-auto-rebalance-margin-mode, relative or absolute), defaulting to relative (patches 7-8). * The pressure-gating patch (7/7 of v1) is dropped. I agree with Daniel's points: PSI can tell that a node has a problem, but not what should be moved where, it reacts late, cpu and memory pressure need different standards, and it should not depend on whether a node broadcasts the information. I'll follow up separately with a proposal that addresses the "what and where" with a per-guest model instead of a node-local gate. * Each behavior change is preceded by a patch which adds its test case with the current behavior, so the change is visible in the expected log of the changing patch (patches 4, 6, 8 and 12). * Nits: is_resource_motion_queued_or_running() factored out (patch 2), `my sub` instead of `my $name = sub`, no combined `//=` with an increment. * The memhost patch of v1 is dropped, as the same fix has landed as commit 28c31e4 ("fix #7974: env: pve2: set dynamic service mem value to host memory usage"). The series is rebased on top of it. The series ---------- Patch 1 adds the schema of the two new options to pve-cluster. pve-ha-manager falls back to the defaults when the options are absent, so it also works with an older libpve-cluster-perl. Patch 2 is a preparatory refactoring. Patches 3-4 fix the retry loop of failed rebalance motions (bug #8059). Patches 5-6 add an optional cooldown for resources moved by the balancer, which suppresses an immediate reverse motion. Patches 7-8 add the absolute margin mode, which creates a dead band for motions that only shuffle load without meaningfully improving the cluster balance. Patches 9-10 extend the simulator (actual running state in the cluster service stats, like the RRD-based stats on a real cluster; an optional node base load), so that the following patches are regression-testable. Patches 11-12 decompose each node's load into the usage of the HA-managed services running on it plus a residual, and smooth only the residual with an exponentially moving average kept across scheduling rounds. Known structural changes (a service was migrated, started or stopped) take effect immediately; only load that cannot be attributed to any HA-managed service - which drives the oscillation but cannot be acted upon - is averaged. This is unchanged from v1 and is the only behavior change of the series that is not opt-in; no expected log of an existing test case changes. Patch 13 adds the matching UI fields to pve-manager, and patch 14 documents the new options and the handling of failed migrations, together with an overview table of all load balancer options. Testing ------- The full src/test suite (158 test cases, including 5 new ones) passes, both against the pve-rs bindings and in a build chroot, and each test-adding patch was run on its own to record the behavior before the change. The residual smoothing (patches 11-12) ran on the reporting cluster for several weeks as part of v1. The new attempt-based failure handling and the margin mode are tested in the simulator only so far. Known remaining gaps, kept out of this series and described in bug #8059: candidate migratability is not checked before scoring (Daniel mentioned he is working on inferring such constraints from the guest configuration - I have a draft for the same that I'm happy to drop or contribute to his series), and migration cost is not part of the scoring (see also bug #7650). pve-ha-manager: src/PVE/HA/Manager.pm | 201 ++++++- src/PVE/HA/Sim/Hardware.pm | 39 +- src/PVE/HA/Usage/Dynamic.pm | 73 ++- src/test/... | 503 ++++++++ (45 files, 5 new tests) pve-cluster: src/PVE/DataCenterConfig.pm | 28 +- pve-manager: www/manager6/form/CRSOptions.js | 35 +- pve-docs: ha-manager.adoc | 62 +