From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 40B051FF0AD for ; Sun, 20 Sep 2026 18:55:08 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 920D421480; Sun, 20 Sep 2026 18:55:07 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BmnJdzyFKF8KkVuWKu5pEdLacr6sdy8xQFi26hZsDi/SpcFs3/xKXDxp7Og8Lygix5ipxqYfmRdnP8cgBzsO7kLlJb5amAvhi7cVnX3WgCUJyCCtxLlb9TVi+mnPYRkdmVaA5uhSYxZfVNkbsp93/2dFkPQfBbzXP63Xj+aRzG1DO62ZkYavk+ekkffA2BWuBCkvaHhtzTaNDdvSX3Up/BpMCcKJpdxyHJnu3jmhl449sJsPPI7G2PqztXxsYXXmneco//LQr/MxakB8xMRV3Lt7Ms1TqGH0qauDMEF2svdBKaQn1IQU9amJlM17zryuxG7sAG31HZ76V17NB6CLTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=80atesIMt3qsFDeKY9Z620FL6Funp2AlUqkPO8BrJGw=; b=gx8QTv88jeHWytOLCJi/cmDzsZm8y1iJVEDMv4Ot5y7F40mUaMX/K0g9pLKEyXKtntF8TtSVfmCdoYXvNouBiG1DcHbA8+a0+Apx9E32+c8ZeWMcV1KYDSGmtziSuEzsBGinSH80+zeu0Z4VgAl8aIAlhVdkirunMqXkQkn+lhp5ZxiC4Vn05DNHkNLShVi5vvVmd09+4zCVhzSo15/g5ogG93QZmuVjyji6QjCh52X5NzvftPpJdAsl7SQL7W08pCntRzHDIYqyBoy55bbdcfPJTkY8Ay61HupQZTsdMV8y90rRl7/DXb1N+Pjr5U0Z+Y11i4cHPhujtjgCAitB7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=80atesIMt3qsFDeKY9Z620FL6Funp2AlUqkPO8BrJGw=; b=B9o99U0nB6/eQ3OmqjtWlPj9eUgafFkKZxhXdcpoo9/wG8tlHVs1nbvF91UJXiXp1cKd7SFqDJD7Au61ax76yVmf6sGfxFBxP6cQOibVRFFlDhPQak+sQtr/X79XGyJuMsBLApV7NWbGc8fgvahtR/grVEWSFGqG957ovzbtDLHCPa/V9yecZpWLnR8NtK5Moi5rOCEOGwEMd6mPplzSgjhSvCGpmh1ENISrXQx9voId2sWPVdVBQNJoOwNm+ROEnaj68gOaqEN/DZfE6SL9BUkqZLxBG8pAZ2dp/oAwv7oxeuafEkYRYXGEqPQYySCf+pvqC+izzVaYx2K85YhaRg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager 0/7] auto rebalance: fix failure retry loop, oscillation and idle-cluster churn Date: Sun, 20 Sep 2026 18:22:07 +0200 Message-ID: <20260920162220.574802-1-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: TL2P290CA0008.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:2::11) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|AM0PR10MB883391:EE_ X-MS-Office365-Filtering-Correlation-Id: 9975d8dd-4ebd-4ded-ab24-08df17335d57 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|366016|10067099003|3023799007|6133799003|5023799004|18002099003|56012099006; X-Microsoft-Antispam-Message-Info: 6uQ211BBXskqv4MxBBBBtWP2wTdWJ5X1kwRtZY7pjAhn+mrsHf0lbzYMX4IFFgfd+6nZvQB+n2BMEpReNyVmOM37DddQqIaGRIZWlCAsnPZEYHlz4Et5t3p4z/7N7Jf4Q3OZN45maRlSWCp5Th4HS97Y1oUvNXpVg8PU7aKJyDjQJMNa37lPFpw/Va3Ptg220tkQ84M0+zFJe81O8fv+E8p7+PtQIAZDzdggjcUwk/Km0+A0M6ZcxuN7iK4zLbdydrxoc2ZrLYh6Gf65eiqlQ6wiKPQEXig02nQ8VTANQRjf9GRxJk45JG34BGCVWgNipg9gkK49ZNqA/e/GO/B0EbntTrkZCO+j5qFAUXFOvkAIS1j2fJKZUpfsaqjsj74aO64t04Bf4I9tJkSd3rRreHeeWTzTwbPEBrkwQX6LdU6GgAnmZSudOBtWPDhUakEmo2RVDDFR1plA0/8qDlTRytkIKtQXxUYehG2thSm6t7D8R3Omi5rxDnfvQyXGAaAYHw5/AntjR+2JPeMtXr9Gvk7116xWX+WqSAmMhoaHARaQOoCoXcGMX/dDM1iAyK4vKyeIAziqIpm+Mx3qckkYzWZWlmRPfBryCq7Giz6oFbuA0GwdCo/otuEz8MF5UrbvGDRMTSb9UJwFjcgRILy/aBc2lSUBk7mmiEUzFeI9/lU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(366016)(10067099003)(3023799007)(6133799003)(5023799004)(18002099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?clFiSFN2V2ZsZEViSWluMVRsaGh1eWdxVEhpY2hCdG0rdjlTVlN5MmxlVVZx?= =?utf-8?B?MllhdVYrRzZwTEFvWjdoVi9pOTNTN1AxL0ZvbnQ0ei9mWWFSU1hha3ZCa2gz?= =?utf-8?B?dlBvTFFic01XNUZnWjZISC9GUHNHSTBTd2xZUWVwYkdIaHNubG9KdVpyR0FI?= =?utf-8?B?RlZKSExqZVdpN3FxUTFjR3NxWUZZNmxDZWZSU0R3eXpKNm9HcDQ3OXlrbW85?= =?utf-8?B?YXdSa2Fjc242N3ZHM0U3L3JLMDA2WGlUK1I4d2RWaEc1TjNta3RNT2dYS1Zz?= =?utf-8?B?eW15OFgyNS90VmFtL0x1aythU0ttcFFvNWx0R2xZTys2bTlKQXFhbENISnhC?= =?utf-8?B?cjFpejJ6UU9SSEFUZjN2ZDhNdjV5cnBRRXJxa0lNMG1aVnZpQUxpTXFkVW5E?= =?utf-8?B?SmRWczVGeDMwRjVFTXVuU2JyTHdtTFRqeHJxL1R3RnZYd3hTN0xXK2lCdGN1?= =?utf-8?B?YXg0VEhvNk5uT2VnZDFhV01jZkMwY3FFb1N0dzJBOGJhaXF2NDBiSzcrVXk3?= =?utf-8?B?Nnhua2UzTUpZKzRGYXB4d0FHTSs0WWZxbEhTVG55OTVpcnQ1eFk5U3duek5k?= =?utf-8?B?VkIydGZ5ZTR0eTdjY3hhZEdxMFlkVUFFZ0VUbWRkNVFwT3VqUUFMb0htRWxZ?= =?utf-8?B?d0NFUFc3dWNUZXpXRS85bU02aDZnSDVlb3RXSDZ0N0RvRGdKbmw0ZlN6ZTFZ?= =?utf-8?B?NFI2NWRxK0x1YjVYc1F5TE8vOHdUa2VlNFpjSWorMkQ5R2pHSjU2Ymh5aXpT?= =?utf-8?B?MEZxb3N0TmtUY0FqK2NmaUpOL0tXeXZNUFZldlh6dW5teDRyRjQ5aHkxUVhM?= =?utf-8?B?aVBiMjArNWhsYStuaDdpZ1UxVDlSblBhZ1g0bEhDWUVROE9jYStxZHYwTEhl?= =?utf-8?B?U1lpTHdBUWZrWThqMXQwcFpnaDJwalZHcEg5M3Zkci9WcS9hRzlKcjNaakpu?= =?utf-8?B?MGZtdThOalJjVGNWazN2czZNWGJzNktBaGJobDlUTEVwdVZ4WlhHU1FKNjdj?= =?utf-8?B?T1R1VzVkam5kaVVIUjlyeEd3VEl0eWxoZlF5bjVqb1ZJYVRLejRHTzVLRGNR?= =?utf-8?B?ZFAxZVBUM3RHMUV4aDAwbTFNTVBtODVLR201VkNWUSt6cXdWYUVoWWV3K2cz?= =?utf-8?B?RjBDNkVsTEVCNHNOYVpxK0lEenJRb2VJTE83WUduRmUvb2hTSjdYZU5rNXdR?= =?utf-8?B?WERWTjdVNEV2eGhObGFSM2lIUVhkSDJBOE56ZHprOXhXRTViVjZTZjBmN0FQ?= =?utf-8?B?VXppM1YzWXZCODhJY3EwOSt3WUV6YW9RRW85RytSVkZUOVdSZzZaaGNMd1BG?= =?utf-8?B?SG1Keml5dHVYQWMxWHJzWDIvWHpZRXN0L0NLYUJnZXRjQ2ZodHVOTWk2cFNC?= =?utf-8?B?c2xtbGlBMitDSGN3em5zZTFZTnJLTGJwWUpvYzlGM0ZKOHZGTWRrL01hcW01?= =?utf-8?B?bndVS1VaTmpvRTRRdnRGakZMUlJhSy96eWFhTjl3REV3S2ZhR1dteHBMbnE4?= =?utf-8?B?M2tUZWZJZlpzbURlUGlvVUlzdG9UeW1lZlNoR1hDRi9DampQU2tvRTd0bEtr?= =?utf-8?B?ZlZTUXk3TndJWWZHeVVpMzFsVHNVcmViOHpCMUVmTTFHc0VlZ0R0cm41YXM4?= =?utf-8?B?d1A0bjhPVlBoQzVSMHJlR2U3U0dDVkRtOWpucXZoU3BKRy9HdXV4eFYzU1FO?= =?utf-8?B?czZqb3FSRW01ZU12emZvK3A5YWZyQTFZWkFaS3hqNlBKdnNwTWMxV1B0T2ds?= =?utf-8?B?T0VUdFBpREEvbWRtd2pybzh0QllUSlF1OVpuanJ0NVFTWVZwODNQM0dvNXRD?= =?utf-8?B?Z3Q5dndCRzFiNlB0aCtQd1Frbnd4aDJaT0I5aFVhWmxhckVwbVhPQjJPL2RI?= =?utf-8?B?Uk5Ya0xMYmRGdGhqS1pIbUhSVlNMK2dzY0JlMExYdytGV0h5VkdOK2JsTnRx?= =?utf-8?B?YmFxeEYwbXdsL2I5N0pqTVdrTE1FSXF1MU4yZmpvR2l6dEl3R1RHVUVPVHVr?= =?utf-8?B?Vi9qbFdzSFUvL3BoWHhQY3RMajBOMEdGNWpkWDhEK1lJTnBLZWd0MWZBWGJL?= =?utf-8?B?WXZ5eVFEY2thL21nOEgxQ0xCTFNKZ1owR0FnVjc3Wms2QklveWhlTGxSeVJC?= =?utf-8?B?YjkvMFdVS3U3TFpsclBKR2MyY25RejFyaWpVNzFyYlpjK3V6VldXT3RpQXhL?= =?utf-8?B?OERWM2ovcHhac0s3Tk12bFcvSmxCeUNzM21nVDVHbDU3L09FZit0VjJUSTNJ?= =?utf-8?B?NmloY3JoL1grU3RqR25wdmVzaitLTEZta1RoRmxabzhLK0V3STg0SkpMNXhk?= =?utf-8?B?V2NnYXFrNTBSL2UyVjQxcWNuVkYybzJRVy9LdENxeTF1SGRXSlE2QT09?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: 9975d8dd-4ebd-4ded-ab24-08df17335d57 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2026 16:22:28.9316 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4G1GEUoWd7KFRsoQyeo0S3jg6MI7LGkEHLLWQfSucHcFNckPuPDpD/EbYoisEA7ltNX38yTBGr4dULcZmHNcoA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB883391 X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WKDF5TLMFH5XWC7PPI7WI76MCOBVCWGM X-Message-ID-Hash: WKDF5TLMFH5XWC7PPI7WI76MCOBVCWGM X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The automatic load balancer of the dynamic scheduler shows several problematic behaviors on a small cluster, found while deliberately stress-testing CRS with ha=dynamic and ha-auto-rebalance on a two-node PVE 9.2 cluster (pve-ha-manager 5.2.5): 1. A rebalance migration that fails (e.g. local CD-ROM attached) is re-issued every ~40 seconds, indefinitely - 25 identical failing attempts in 20 minutes were observed. 2. A single dominant resource oscillates between two nodes: the relative margin provides almost no hysteresis (a base load difference of ~0.5 percentage points qualifies a motion and later its reverse), the point-in-time usage samples are noisy, and the predicted post-migration state is systematically off because the balloon-adjusted guest memory value is used instead of the host-side footprint. Five moves of the same VM in 27 minutes were observed on an otherwise idle cluster. 3. On an almost idle cluster, the imbalance metric (a coefficient of variation) is hypersensitive and kept issuing barely-qualifying migrations with nothing to gain for any workload. Patches 1-3 address the acute symptoms: track the outcome of motions issued by the balancer (exponential per-target backoff on failure, a per-resource cooldown on success), use the host-side memory footprint ('memhost') for the dynamic service stats, and require a minimum absolute imbalance improvement in addition to the relative margin. Note that patch 2 overlaps with Dominik Rusovac's pending fix for bug #7974 ("env: pve2: set dynamic service mem value to host memory usage", 2026-08-28), which reports the same root cause independently; happy to drop patch 2 in favor of that one — the rest of the series applies either way. The oscillation does not fully disappear with correct memory accounting alone, though: the noise-driven variant (sub-percent base load fluctuations under a dominant resource) remains and is what patches 3, 6 and 7 address. Patches 4-5 extend the simulator (actual running state in the cluster service stats, like the RRD-based stats on a real cluster; an optional node base load) so that the following behavior is regression-testable. Patch 6 decomposes each node's load into the usage of the HA-managed services running on it plus a residual, and smooths only the residual with an exponentially moving average kept across scheduling rounds. Known structural changes take effect immediately; only load that cannot be attributed to any HA-managed service - which drives the oscillation but cannot be acted upon - is averaged. No existing expected test log changes. Patch 7 changes when the balancer acts at all: only while some node reports a cpu or memory some-pressure (PSI avg10, already broadcast in the 9.0 RRD schemas) of at least 10%. Load asymmetry by itself is not a performance problem; the pressure stall information directly measures whether anything could run better with more headroom, independent of cluster size and load level - akin to the contention-driven approach of other schedulers. Deliberately only node pressure is considered, since a guest saturating its own vCPUs or memory cannot be helped by a migration. Fail-open when no pressure telemetry is available. The series was validated on the live test cluster: failed motions back off with the expected 60/120s delays, transient spikes and sub-dead-band asymmetries no longer trigger, real node pressure leads to a single strongly qualifying evacuation about two minutes after the pressure appears, and an idle night (imbalance metric standing at ~49% the whole time) passes with zero balancer actions. The full src/test suite (158 tests, including 5 new regression tests) passes. A related qemu-server patch (sent separately) handles the case where a rebalance-issued live migration cannot converge because the guest dirties memory faster than it can be transferred. Known remaining gaps, kept out of this series and described in the bug report (bug #8059): candidate migratability is not checked before scoring (local CD-ROM etc. - the backoff turns this from an endless loop into a bounded nuisance), migration cost is not part of the scoring (the balancer prefers the largest imbalance improvement over the cheapest adequate motion - see also bug #7650), and a fully guest-centric scoring model (per-guest contention score, DRS-style) would be the natural next step on top of patch 7 - the required per-guest PSI telemetry is already collected. Changes are constants for now (backoff/cooldown durations, dead band, minimum pressure); they could be exposed as ha-auto-rebalance-* options if preferred. src/PVE/HA/Env/PVE2.pm | ... src/PVE/HA/Manager.pm | ... src/PVE/HA/Sim/Hardware.pm | ... src/PVE/HA/Usage.pm | ... src/PVE/HA/Usage/Dynamic.pm | ... src/test/... | ...