From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 084621FF0AD for ; Sun, 18 Oct 2026 14:48:58 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1E4E42160B; Sun, 18 Oct 2026 14:48:55 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vGK9NKTSvYeVX4vL6MQQebg2uO2kcVqvgfVBfdw1+4Z4+NRbfEVRDBaL+mD+JAIDKyvmx+4s6i50QqawQUJeFZrFLe6UMrqPt83Hbu7mHxMIjj0OckS+nYsNHYlNXJPwgl0WUNcZ0PKGe6KKy8kzgckJAR+jTenl6LPRPUSnYZz6UnP/sDpplVwp1WQDAhK1oyC3oPT5yvQDU540XoYk60mgii6tDxrmPT2PEdaC72QYSt2eT1fCbYW+/s3Uz+J6DJLRbOj/F1lohub932NQYWIXVdVoCBJYc2tWkyc7L0qm+Mmf2RZnaGFyohPObovj+rIGB58uzf1uwkYy726QdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=upKX5cor4zFjBZ9XxJlwmf/51Uccn0m6XSPVl1wT8Kg=; b=vigOyCgQNIwNAtmhusMzeBdeSM+gIwqK9YSIaUX4Tp+IJ+2h0glDfFpbvS1/+x5KYqwA5An1EJHhjTmpbAQzUNwIBQX3tee3bfFx1StCSEp1dceP7qlsN7Di3V0KjOb9VkYjf55wCsNANU+FnWjs2ZCOuLOZwy003Th1gSzlGBdWOG3YoyIGQ6x23YdhsGnbx/XCCRo8dl3SgR824ytSuQY109wLkJSdIrc8tASeLTQx2OzYTlCz2Suw3jLtuBrgIo0YYaa6bJbRi9gwSOGfaEYCZZP4hkEPsLfb2W3PDH9MNj6+9rxCWpspxHWIaUEEDq2YqW+II2wbGc+jtS8SOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=upKX5cor4zFjBZ9XxJlwmf/51Uccn0m6XSPVl1wT8Kg=; b=eUpeC68P/0q8df/Z2lF4/X98LElWwst6jgwBnIuHx2x8Xt2Px72M08jTLxJQ2Q9DZrRtCvCxSVwOd5gVAoRLPDSpYULyUaookZ3EDRlq6ZslExKUtFidkvYVEkgDN6Ar6yyb1VEEHYLev9f7fpfQppBeKlnNTbi+Qyr64AFWYnNTS6QPT4BPNMBcLsbP7VUn6He9Tj2k9NoqV880x44C7fFLkj0si5WlqOpIg9Ycn+n7ZeVmhYDnW7DCr4I6KzVF2tUe4PNHkNKP5tYJRQ5GkQbx4Luyz3exSPapRPU/O10/ta7hwJBQ72DZRbEBSJfskO37x8sy2stEjtF2NkJA+w== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com, Daniel Kral Subject: Re: [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Date: Sun, 18 Oct 2026 14:48:09 +0200 Message-ID: <20261018124809.24466-1-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: References: <20260920162220.574802-1-Michael@RyomHerold.dk> <20260920162220.574802-2-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: AS9PR06CA0165.eurprd06.prod.outlook.com (2603:10a6:20b:45c::12) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: d414dcf3-25f9-4b39-2384-08df2d1615d7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|22082099003|10067099003|18002099003|3023799007|5023799004|261009223027099003|4143699003|56012099006; X-Microsoft-Antispam-Message-Info: dJkosFIuD7ItFMQwWtAqfUqJ2Jq28rf0svj48vqa1sMbGu73O+6xNmlWo4uKnBtSZl7KK0ELvScDKjg3sgbdK9oKbKK3b5GZ4hYmYW1kS3YiabIPIz/MWcdLWXiIdLmSC9T1lNvrAIdG5Qof/2Ia29pLj0fDC9lmng0081EvsCwYDWaye7GKrTsRfMnetVsbuw52ZDReelNtfng+mKf2BLbwf2hkZsd1AEkIozQj02ak1jKeHI+2QqiEJmBkZNX07U03EbGyJqmVHZYXNYj0dHkFWgdHlVPd6XC+Ert33k/HHfpa4D8Yejid18ClzESjcxY2cES7kaLRIipm2Oi/UCgYzz3VPjjcTJ52QvDLrPKxk6Jzl1tjie5ssBK3+cmSLMvHgxTyvyXSnvq1HC5Nxu1YoOG7ZNqw5CGOAwJ/xelfXQ8FwXaFKgpyNB1fnN2kdeLD9PkbZWArMiHDFIfEpP5PBdTu0OTAop02i3OtLr1FvVZfWi4/5sm0IHAl0PLwtNqf66sNCi9SY/i+iGFG4uP4Y5JxZ0K0oVKkbmAn5Q8REZ8RMc5KjzMcOoEIofoyOFGQT2Tu0vLiRWwdZ+8P8MQ/35gDmUwChY2NpnSmIs6MxVhJQphKZmu1tYlv3LALISQ71PZ2DWgsJbqg36c8dMo+Zk2INfwb9W0BfrQlWtU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(22082099003)(10067099003)(18002099003)(3023799007)(5023799004)(261009223027099003)(4143699003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?axlTS3k5wrXV47cADHDBsVv1pvHuOgs5qL9sDA/DjhzugKfJBkr5G8zIQXAL?= =?us-ascii?Q?4NwKs76+cLwonFUcrXhbvCa0fLl5IGWbWjsK9uyH0PElOxRyRDBS/1FtliLw?= =?us-ascii?Q?SspXXTvEoPryhzE67WeypRMBT733i/3FmqDdpOUmr65s9yIpLr/YezCVRyPR?= =?us-ascii?Q?w/4kxLzvZVtABuHkA+cCfM2gI89A/Xaso/oZDaNhEwKzFSbihF2/oFvhEF6N?= =?us-ascii?Q?d/slOdZpLGsristoZfF7tprZ3OV7xvB4ILJjFIN/jFHmRCPO2mskVfJ0NcfB?= =?us-ascii?Q?pWMaQfTIioBzkdKHfBXqQtgiWqY2WqQECh2mhDi/BDz7Cyi1UuAqwW11A220?= =?us-ascii?Q?aEyzd78gzkxjrPlHEPkQG8MfUJv+7rhi5fcSXpsVyvxERu4GS76+7fvJ+92n?= =?us-ascii?Q?TbpFhD5+6sq3beKI0WK2d2ti+AtdlYb6kV6SRuSe3J+1HvwIQSN6Hx9AQ2ba?= =?us-ascii?Q?/8G+qn2TZlVchSA+NyTlmYzsEoWwiwFMNk9BPUoSG+BwROsBGynttxGgi6zN?= =?us-ascii?Q?H9N3cecsJ3RKuMeTrDYlNfuK2Uro3Hxr1MWuHfg+iS1fN4dIZJF4PRregFBi?= =?us-ascii?Q?hVjIpj5pasJqtwK6G7asSAvGMRYXE9sB53UdmByeigBvSmdyYM6+yRTQi/eB?= =?us-ascii?Q?GMW/bUY6huFJamK5COPDk28mRp5Jc7sopZMH27WJ/EGmtkl5iCz95cdJtfC5?= =?us-ascii?Q?E7y+wNvOFn4QqlWIufMZX4qZCZEbMkhdPNrgxRvRNlZmPnJFN1KdBaDyacRr?= =?us-ascii?Q?Ge8ZJtHAuMoJRggRtLNHOpJo5I1VJBwgvhX9G3VplYqtxb1D0G+r1BMR00k+?= =?us-ascii?Q?GQDAF2hCj2MnD02Cwb3o/lGKV8cQ9U4tRbdimD0F6/zG5oQOeqrsYlzB6HF5?= =?us-ascii?Q?7BtQekvoiUSDbQlxVH0zIqumpFM0Z9w0BsR7J+s/8NXoJEcHG4L2e5l4MxcF?= =?us-ascii?Q?f7tCIjSpDT4K+VtIjA5UkojkihsTj1T1A95NysVn85yz4HK/UhyTNkr6koQc?= =?us-ascii?Q?8WK9ttMQTZiksLSSRMVMCRhgWNLKGIB8gu2i1Vy6YJhwRpxA7uxEYraGu/TL?= =?us-ascii?Q?hixrvQ2tRf9UKh0LJijeSMM6qhXL+gnTRmc40q4tcd0zwTAknj7Mk9bRCxPE?= =?us-ascii?Q?b1n4QACI4IfcsCLfpTdAg1bUTBms/6grwgvUoYTsUJYWqYc6srIBHA9wUbfd?= =?us-ascii?Q?bDr7cTn4mvOOEzxJCdT2qBiz7CO0V/ZyLCV1zihcsOpy1hJxMaYMRMiDuBS4?= =?us-ascii?Q?ArH6c0wmcyCUt4qfn76yLaV8bARfcpoaOfUDj4pg+E6VEkkcj39gb6TsrNNw?= =?us-ascii?Q?hABgtzzcJJwGHDYtUFKHNXU41SJZHhTputjUUGv2Ptvf23M+vmj2gkshUDDe?= =?us-ascii?Q?3nb7DK8W22eWvK5x5ysy411Er9TZGhM6eb4souDCBFEAPQ+cUwftRhOSFLiN?= =?us-ascii?Q?OGfFdvKODDC2gR/3YQKc8lyNvcIy+EBlodY6VXeELbZV9qnoIPtR+eStZppD?= =?us-ascii?Q?00GNnOlp7FltzXPF8nDJ/D5VI8Ne7w7UGdOVNcuwRu6/2SdeOGrHwcJF1jX4?= =?us-ascii?Q?Ntt35u43n+cpLSs6RB6odtq6tqnxOCsSFrqfeHSFuDmJSrPqVHJNUo74lhoZ?= =?us-ascii?Q?9L9fic6HFnLNhr/DFy8pvXvMN4qVApyF33BWxfARDVUH6kq19e/AO49ytQui?= =?us-ascii?Q?su1pZyGrnTB3Uw+UPTxhEcml8D91LWZyLYP/D6+y6ef5iH70Ii7iTmplo7xR?= =?us-ascii?Q?Ku2FNXBZ8A=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: d414dcf3-25f9-4b39-2384-08df2d1615d7 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:48:18.9946 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2dvSZQW/CC0e09pU9rkdtrqFLmvfKlyI2Cb8YpQrmCkgQfpc66fWQ2mnivjzC3Web5kyoFJIMz2u/PB4fh8f8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.013 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KYMLKV44ACM3DFNXQLKCB6YI53PXKVBK X-Message-ID-Hash: KYMLKV44ACM3DFNXQLKCB6YI53PXKVBK X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Daniel, thanks a lot for the thorough review. > Most of these are inferrable by the guest config alone, so I'm > working on a patch series which automatically encodes these > constraints obtained from the guest's config files Good to hear. I had drafted a precheck that mirrors the hard failures of QemuMigrate/LXC::Migrate (local CD-ROM/files/devices, storage availability per node) as a get_migration_blockers() plugin method, evaluated once per balancing round. I won't send it, as yours will cover it - but if any part of it is useful to you, I'm happy to share it or test your series on the cluster where the issue showed up. > This makes me believe that the exponential backoff isn't solving this > problem well [...] I think it would be better to just remove the > migration candidates that failed in previous attempts in the same > series of trying out a rebalancing migration alltogether and only > retry them in another series of rebalancing migrations. You're right - I could reproduce the fa:109 cycle exactly as you described it (node2, node3, node2, ... six failed attempts before vm:102 was tried). v2 implements your suggestion: a failed motion is not tried again in the same rebalancing attempt, it is considered again in the next attempt, and if it fails again, it is skipped for 2, 4, 8, ... (at most 32) of the following attempts. An attempt is concluded when a rebalance motion succeeds, when there is nothing left to try or when the imbalance drops below the threshold. The fa:109 variant is now the test case added in patch 3/14, and a second test case covers the case where the failing motion is the only one possible. > Otherwise, I'd tend to record any failed HA resource motion when it > was reported as unsuccessful by the LRM in > next_state_migrate_relocate(). Done, restricted to motions issued by load_balance(). As every resource moved by the balancer is recorded, this also covers the dependent resources of a bundle, which addresses your point about a leader change resetting the backoff: a bundle candidate is skipped for a target node if any of its resources failed to move there. > The separate cooldown for previous rebalancing migrations should go > in its own patch Done (patches 5-6), and it is now an option (ha-auto-rebalance- cooldown) which defaults to 0, so existing setups behave as before. > IMO a patch series that changes some behavior of an existing system > should add test cases in a patch before that change Done for all behavior changes in v2. > nit: I'd prefer `my sub name {}` [...] > This should be factored out from the similar > any_resource_motion_queued_or_running() [...] > nit: use two lines for this All done. Best regards, Michael