From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 46D3A1FF0AD for ; Sun, 18 Oct 2026 14:47:26 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D42D52168F; Sun, 18 Oct 2026 14:46:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BnfKRftxfvRI95hAPxpqFxDSxiaot7z8pZQsZSir9zOTUB48VU/3q2AD0TXWGYbi90gKj3r/3wJELE37Nc1GcLwYdBs8JS9haqiRufqgEv3ILHksdHbK62ITactBBIJVyKvigECRWh9S7KeMoCtiIMU6I/sDuQPZqEbloT+w9ovAfM3UsZ+l6hofYcZ81gdwKzogZiYjPlo56PN8E17ef8rn4WrsSwDut+kfKJQLtr+FkNNp+Qc1aEGzottj0fjcS5TfQwaCmIhTmzXctgttZFyjvBQH34qMxDgsdTlhdy1zeWEHIcouUrWJiwzVtZrbeIbPd3BPaTQJuFveaQTaUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hTCM2zlXB2jiF+ovX1GWK29mrOjyPnkvwNT9WjLCi9w=; b=RVu4WgElgffw2ifR7CpfyHOIHYd47qfLdnCcGjKGH4dDnPTSPAKoR8FuZQ1yyilci2sqD4IKzls18vkwQDhrntE/tevYAnmRk9jp0ha44+jmR3C7aSEzPsn/BxcmSuJ9qE7t4BP6sfYvjIN8zp//VTg7l3P2kyn/Fq9zkykX9OWqutQIBx3/urd3xViYTk3hpsDqTkvLd/CkPKK41at19+16vVyny7Mmsgh4rhokm5pi++LVp05Qumx34VW9HMFGBiSBbLXfPcdpcjOkrM4h6RyNjv5DyyzTtEd76KHvdbMspNvxwNI+kL8rFbfFcmLajTvv0yNjTNMFAzR+UZAuZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hTCM2zlXB2jiF+ovX1GWK29mrOjyPnkvwNT9WjLCi9w=; b=XDZAVP7fNcV25zgrqvy6X3zmqDIRK/OGrSXZ9m6kgrgEO0erUmjy1xqEoZtGCk5oZmY+cIArpMp6BnHUd8kfY7AlxKl/qfLft6ZzzzkyFFOpsuhPToruP3L25IrjCF60m86L6Ac55pzH3VHgtYObK+O96Pi1IlYre2FlNAp/84voQpZs4wsk/vmInIbbGAQmG1lXgvJ7AI3pZrj3FsyyNeZ8hApStn61cIIMCQ0oVuZRgW1AP2mD0SQ4RZf9Hyl48Syd9noSl2XIIXeQguqDdgAfRo0CTIzQHijs9J0t7xHkfJ3VX85+56y+90u+rCo3LGfz1kkZ606ly8bh4UuIwQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin Date: Sun, 18 Oct 2026 14:45:52 +0200 Message-ID: <20261018124559.24305-9-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk> References: <20261018124559.24305-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: 18582c1d-e445-490d-a983-08df2d15ccef X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|6133799003|22082099003|10067099003|18002099003|261009223027099003|261009221227099003|56012099006; X-Microsoft-Antispam-Message-Info: EeSRrbZvvS0N7Z5FWc3BpAAveURxiSQyHzN3IbImxxZK1X9y/jC80MyQxRB/q3AbCZ6Nj+JJ4gmlrpp92v9en+fBwTOfOkCaB23kiOwCIuzLH9yDYqnaPPIj5Xpuf6i1bvKoPq3XkeUx6s8VC2onpWRV53594+00JgES970MAk0PhSMoyRJZ4U7TC+b9u8cQ5YXxplzhCyKDkVLxiNzVYzzy1MWpbKCHOnmrvaUqO4b/3GqtIeJw/tMjZ+Vvdnkg5LSalH5MnA37cilmdNuBlMQ6BLjHwM1tokaXNDOY+qMdguHA3PbfStVlwkkkdj5vJYpijS3tr3TPvkCzgf2bPv7rcFMUlrNQpQldIliWKao+Cu8lBGqLp8s+FO8nFp49B7aFQnM4Nu7bYPhoFuCVG/jB+SKE7GVXLQKU4E3OsI7a/4D4o3VuNgbTLD0A3h6g//1g1o2mPlR2VX5iwRH+mDiO45xdsph4Cwyg0lMhfMSDe4OGPx5wydo6UQidQKzfHe3+8ChC5PVcIn2+2+6rxwOmdwOTtWwijg0mvyl8T/WLTgK0xo4Q+dCq8qSGFpwyPimP7e4S9Q+d6LDe/GCyZD5na12v9zIz1cIzCoyGYOALy/D0DQUNRpK4ELS9XTzj37mk8KhopcRgyA5MB4vHoaEpRPEdCvvpMEgSmlah2ls= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(6133799003)(22082099003)(10067099003)(18002099003)(261009223027099003)(261009221227099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?JH3LoAhtZn/T6KDX9OqJOS2vwwwN59niIQOZERVQ90zgnR8Kkvj707PPXrnd?= =?us-ascii?Q?vWvdqi5+D5gfXyBD5Rhn9EUVpvCH537Wp8Ngzd/466uFp0Np4+YLvIT01Pf6?= =?us-ascii?Q?D+Ykf/RDS8ykkWdJmqVJEitw6k7Jpdo4xEsrZnQznfw9yJ8BArB77VNpNFSj?= =?us-ascii?Q?QXZNFdIBM9vyY3yTT2y/HQTCUIbkFirYHlQmvA+vis0pKrg2fdzay4tn+AI2?= =?us-ascii?Q?ockTp1SZB4MMvpylaHMO2+5BM9Ew3oLuJeF8nu090Qg1g8zgoWo8utFr6Lf3?= =?us-ascii?Q?8jcBGegYnMWHnmytUHqVldTORUOHjg1nbyZjofw8YeY6y1tKrsGMyXyLBaYA?= =?us-ascii?Q?XqDyvj4EcmSMOKYHIBU1TnMUhOOgd8ts+daFBnrH7V942txXL5cxi13yKHwc?= =?us-ascii?Q?NZltlMMsnn1SWm3XGK7/H7m+nc0ABIICa3Bz8G8s0j7eY73d/qAJGJ15X5FO?= =?us-ascii?Q?M6ULNOgmZvfA985paPOsfG7S2K4zRFjeisdeifY41OcCydMeT8ScNCV9aXTy?= =?us-ascii?Q?+unaV+5YGa+AS+xPCuXBpioo3CVK2u06ekrhbJpo4+cM302CDlY7bvNM25da?= =?us-ascii?Q?4kBlAQCQAk7Gwk+WwdfIXh9TwDu+PEje4honBc3+/5D0Y2seAUYHDboSDfMl?= =?us-ascii?Q?szgjd20/Yh9/pQhzTpceuRNKSUO257KjAy2K77oJLTnwImoX83yoMDhBKZcd?= =?us-ascii?Q?ZIrKAJ8OdQeYBdh1hNa7mufnfuV8EqaXKkGHZIgD2bV8mzFODw2+V6mf9L4h?= =?us-ascii?Q?PfqXugfxp0ec22JOl5HY0RygdwMQWQChlsveHCXeybPPv1BD4LSejFk45b7c?= =?us-ascii?Q?XE7sPPNrtndC9LG31UhT3Bbfqx8tfOztkdEGvjk/ZrFuhQwvnfHrsgT7ixox?= =?us-ascii?Q?aCalTXhE/B0TUJnyycOLwaNhbUphohWeZVfBFBd+1ERJLvzokjwy4at3XBTC?= =?us-ascii?Q?kkwMgCom5jP2FW/8eBWeuNqLyBBy4m5GTj8KMGOQVRncfcvVptgZzhYYLpMl?= =?us-ascii?Q?06fC3olFDnuVQxPzAPIRhW0TLfwvUxTXgwOZisedjfS4IchWMfj/Yu0CAiQq?= =?us-ascii?Q?wnHut7GBtveAqIUxCSgvS+87p8t4UOBHJ1l2yi7neE+UJLT/0OqZLDzXY2bz?= =?us-ascii?Q?WExIxRsY17TrO7Kkz0Hh+8uyK9xLDiZKQIyeiAqFljQCODadoJhVEQJ+oNLH?= =?us-ascii?Q?JYwNWnwUJ9CCdcmTXLZrnkGaTGyDfimb29ql1H2+00hbLQKZH9a8jh3Bi/ZU?= =?us-ascii?Q?QVDXmS8iDRIkzWqxRDIjbdsm8ncNqxAPTb46grMce7bKfjQT3BG2vw1FLyvJ?= =?us-ascii?Q?WqH998RjiOy/YAE8wg6pNo7abPY9pB3Qkrl8dqAZw6CTqB7n85x9TN/H/n0A?= =?us-ascii?Q?Yw4j+qivYHcc7xyzwGXdCmpmDqO5ODkL3bCV7PkYt2b7AtYD7XQIMVztkk+R?= =?us-ascii?Q?/alfhu9JK1E3ae1XFvzuII09rI9Q4UtBgFa0mxm9G8vcP2UZIQceO+G81+6F?= =?us-ascii?Q?z+s93kLxm1v7HO7MGOmRTF6rK6Pj00Vi5IxVCwns45+97w6fQosXP8RK/2z2?= =?us-ascii?Q?Yu6NnGh1rjucdWqHufOhfHAbdGQKiV9PW7Jx9hxAvM1HFYCNU1tAKoZLmI44?= =?us-ascii?Q?rq5miwPxtwPdaKMXxXNcWefxvvi7UREBPzbxU1LmuGs992wMKQMGC75O6GLH?= =?us-ascii?Q?h28CnYGZCQKJow/bntimBCOhCO9JLWRzFH7lQ3FG00QduJ2OzGBOxy/Yc0qT?= =?us-ascii?Q?i2/AyJncLQ=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: 18582c1d-e445-490d-a983-08df2d15ccef X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:16.6581 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: chnXYtSFlQEIXeaUlGwJBbkn2odsPk8AtpCmqFwiOzUN13fc1t/4rcv15NnGmMda0ur5k8ZFHhfuZgH0Jxuo+w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.003 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FBY67Z2C6UZ36MA6LSXZKVIQRIJHEQ32 X-Message-ID-Hash: FBY67Z2C6UZ36MA6LSXZKVIQRIJHEQ32 X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The automatic load balancer only requires a relative improvement of the node imbalance (default 10%). When a single resource dominates the cluster load, the relative margin provides almost no hysteresis: with two nodes, a resource load v and a node base load difference d, the relative improvement of moving the resource is 2d / (v + d), so it exceeds a margin m as soon as d > v * m / (2 - m). For a mid-size VM that is a base load difference of about half a percentage point, well within the normal fluctuation of e.g. the ZFS ARC or Ceph daemons, and each fluctuation then qualifies a motion (and, after it was carried out, its reverse) (see bug #8059). Add the ha-auto-rebalance-margin-mode option, which allows to interpret the existing margin as the minimum absolute improvement in percentage points instead. This creates a dead band for motions that shuffle load without meaningfully improving the cluster balance. The relative margin was introduced when the imbalance was not yet bounded for different cluster sizes; now that it is in [0, 1] for all cluster sizes, an absolute margin is meaningful on its own. The margin mode defaults to 'relative', i.e. the behavior of existing setups does not change. The expected log of test-crs-dynamic-auto-rebalance8 changes, as the improvement of 4.4 percentage points of migrating vm:101 is below the configured absolute margin of 5 percentage points. Signed-off-by: Michael Ryom --- src/PVE/HA/Manager.pm | 14 ++++++++++---- .../test-crs-dynamic-auto-rebalance8/log.expect | 9 --------- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index c480b73..b804602 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -141,6 +141,8 @@ sub update_crs_scheduler_mode { // 3; $self->{crs}->{auto_rebalance}->{margin} = ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0; + $self->{crs}->{auto_rebalance}->{margin_mode} = $crs_cfg->{'ha-auto-rebalance-margin-mode'} + // 'relative'; $self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0; my $old_mode = $self->{crs}->{scheduler}; @@ -370,8 +372,8 @@ sub load_balance { return if $self->any_resource_motion_queued_or_running(); - my ($threshold, $method, $hold_duration, $margin) = - $auto_rebalance_opts->@{qw(threshold method hold_duration margin)}; + my ($threshold, $method, $hold_duration, $margin, $margin_mode) = + $auto_rebalance_opts->@{qw(threshold method hold_duration margin margin_mode)}; # with threshold and margin both at 0% any non-zero imbalance triggers a non-improving # migration, churning the cluster forever - skip and warn instead of silently looping. @@ -421,8 +423,12 @@ sub load_balance { my ($migration, $target_imbalance) = $result->@{qw(migration imbalance)}; - my $relative_change = ($imbalance - $target_imbalance) / $imbalance; - return if $relative_change < $margin; + # the relative margin alone provides almost no hysteresis if a single resource + # dominates the cluster load, so tiny fluctuations of the node base loads can + # make a motion and its reverse qualify; an absolute margin guards against it + my $absolute_change = $imbalance - $target_imbalance; + my $change = $margin_mode eq 'absolute' ? $absolute_change : $absolute_change / $imbalance; + return if $change < $margin; my ($sid, $source, $target) = $migration->@{qw(sid source-node target-node)}; diff --git a/src/test/test-crs-dynamic-auto-rebalance8/log.expect b/src/test/test-crs-dynamic-auto-rebalance8/log.expect index df11212..6c84e42 100644 --- a/src/test/test-crs-dynamic-auto-rebalance8/log.expect +++ b/src/test/test-crs-dynamic-auto-rebalance8/log.expect @@ -27,13 +27,4 @@ info 23 node2/lrm: got lock 'ha_agent_node2_lock' info 23 node2/lrm: status change wait_for_agent_lock => active info 23 node2/lrm: starting service vm:200 info 23 node2/lrm: service status vm:200 started -info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 35.6% to 31.1%) -info 80 node1/crm: got crm command: migrate vm:101 node2 -info 80 node1/crm: migrate service 'vm:101' to node 'node2' -info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2) -info 81 node1/lrm: service vm:101 - start migrate to node 'node2' -info 81 node1/lrm: service vm:101 - end migrate to node 'node2' -info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2) -info 103 node2/lrm: starting service vm:101 -info 103 node2/lrm: service status vm:101 started info 620 hardware: exit simulation - done -- 2.47.3