From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 10BBF1FF0AD for ; Sun, 18 Oct 2026 14:47:13 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5B3C82164D; Sun, 18 Oct 2026 14:46:32 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=U/h8nt2tdE68ONRml4n++Xc8ozqAK1/H1l6GLUNyPWDChh4XYYHlL6kN9yJkNdxpSKGjI5zCMomdhJfkhtWtF46MtCvAwf9gwUmEQtdItpFMO0RFJIjre4YwioMNC51hkvAM7bryKpRf5vE1Woo4ZYeHbH1Jurx0thWW24EEj6eIew+aHcFDemXR57RO88LdJje0TF/3GzSUXKOfDuunAyy5gesHgQz0FS8477UQ2C0ZHb4qaRx0W2KmXFb/epKR7Zjp1WuF+xDdCPeftJrO0IslztFfd70U1pTGGh4ZW7201xFw2X3K2CYxxOROJy49UtLqMkcOiT4nF4495Kxr4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vFCey4xo+SRgJ6nrvrsBJG9uAAwuDCWmnAJXNaT0aK4=; b=JuNXvieL+JHIKoTUsiGE6nD6gItp+5MN+8eAMV3uLjV24dfvmOj9k1VSbCBgDrTW5/RfqO4Bv6H38U6Ww9VQgMm+9L2kIHWosFqNh1/KNn8VMHusps6M2ijQepwetvhHEpk8jyRytmW3k1ORO+r0ZuhQ9bfJ7mg80fSk5YwUzn/tVwDVeklphGAEhmJuxMQvsyUYt5Fvde3plEhrrNWoiGIQ+f9uU3mmGkyshvacQYfeix/HlluOslavG1zH2P4DNhVciQq+06eRbweOWZjCZFe5Dqj7KIDd20MOO93ZwsH2inkZlh1yCAwErCGxfoXy3c9mFuF/mhpGpSGScA4RNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vFCey4xo+SRgJ6nrvrsBJG9uAAwuDCWmnAJXNaT0aK4=; b=3KKNR6tx0FKMqyLavWWMiJluprVt/YqU4F214JVdfCj2mQsDNY1nUZ6RYbRcNnJyXv9YZ2jdAwcueXjGq8morprrynvrFmeTWaeOSuHFocCXi6hW1HiNAOchi4owFWduN6KOZ+a7vC1HvNjBHanl3Qd5+IrXJxPH3iBhGKwU20t+eitOD7okbApUzkq699sCVCJtmx25XqRVWe5DWPDF8YDRsPxdEWY07Lzhsnq+h8mM/67boYIuEC0ltKrfky2nC7x4WSSFBiei5PrnS+kocOiNBg0VYTZE/Fvu1mupXTQe5uc0Up2yeSO4JnQCZFftf6YbfHyYpLVfyJsAI8FOqQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources Date: Sun, 18 Oct 2026 14:45:50 +0200 Message-ID: <20261018124559.24305-7-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk> References: <20261018124559.24305-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: 09880af8-ed99-4f0f-0cf2-08df2d15cc87 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|6133799003|22082099003|10067099003|18002099003|261009223027099003|261009221227099003|56012099006; X-Microsoft-Antispam-Message-Info: csfxYjkjwYQDe7tJCEKzfRsVLYIbb+KhvBH8UmmiQnJIztHhzQLojDTiPmlkNPi8ZZIXNhvL5fKXrpWLL6MF9orvCxFkhoH1tFyYzAP2jS/mwJys7v9XYeOt/Dxst8uHE9GMP+qyy/LnqWftoi6qt7of2lWL6Smc2YL+9KPh54gJE2D9kH3R0OGkK7ji/WbUFpPCKGsg/V7i8OGU9HXkAwbDHVPRghnMhSIkpf5bHz20dAq6bTjJX+idN6EH4NnZ4iL9F9t+BRXnDL9W1gZq02yfGGJEWg20TVeMAGJ1N0RLfGuYkkntxzZO49fLDPBIdr/V9QHMoIciTLkvpSOLcbB2niPLfdkL5ElkRlcD+LZVew6C9tJr4FsLuQ+RdmmCHKtLc8R5FuzQvifERCivf2vpVKFG6HYNXoqxuYD5xKpvzm7UF2xSsIYncXYr0h3WBlG1P+eM9u/C1/l2N5fiXc6ykhlPmucSPw83QnQDCNDOMGxZtNSQJ5XyZoRcP0gyZSYM2O+xhwGrRzTOGp+A6RHqvQpq6WtJrZ4N3++0HH7GZhXfawX7Al0724HHStgDVW5SoCxVG9hygfQ44AKou2K1ObSyZYC8p1jJuSPU2geEEmFamFlG8ab24dCbqaeL2jt0MQ6XV8X/LPWHGkyhFgH79R7hHHWJO6bu8IVYt5U= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(6133799003)(22082099003)(10067099003)(18002099003)(261009223027099003)(261009221227099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LuhzuIgyqs2S7p84qOvUR5X1P+Q0ummmfgEm2EodVEVEiHzAx35ux4oGiHMt?= =?us-ascii?Q?D/T4vB5V/X81/yKeyrsFP4GR8uCAzFx79gC/JKgIiSFfoTSspUzkDttzlXbD?= =?us-ascii?Q?CiQucyU8A1i3PD1FriiJQySFCHSXOE7/zhReeBVIn4kUcS/EyZNx3X/7A8Ey?= =?us-ascii?Q?X6DqgJHSE36iH78NoXHKkfFYRylA6QhqZY1sAo7sEjRTDSJUdGhYbx/BEd6r?= =?us-ascii?Q?UjtPjAo4GyW2pjFLhM8N+mUdCpRs+EOluLzhEEfj3KObxDcPWN8JjYxgXPue?= =?us-ascii?Q?ZG2MQ8rNRB8VgIaJpnH4uSSsRgEcEuyHcTbngTgCv1vzzlpGnTRl6UtvVvvl?= =?us-ascii?Q?FPvZU9g0m4QEGaOw8WeTPC2zQnqrsAUKUDs5O3gNgJFJ1x1hsCezTX/W73SD?= =?us-ascii?Q?OtgLt3JvftNewM/Wxqfw3Uh6GihRLdgyljVU0MlXpqcdg8M2guOEWg5a12yj?= =?us-ascii?Q?vYfbn6VYSx5RG26Z4HuQ87cWaoZSJMu3PrSGQ2iP0kHESHE+AxnOotWuEB/U?= =?us-ascii?Q?ylbiI2leU2gMNNOohzA0nf3rBab8jqWxtnKrOJHPIgtv1ndcT3e6tA25xhBI?= =?us-ascii?Q?ICYaTM4SVIIMtYysJPmkhFPBsgBsUUaLIH2M+GKL3k/SUhA0UQ3WIJIvc9vJ?= =?us-ascii?Q?A7qZYCJIMxaPx1KJbqCBNvrfvqkCbZiwhuUSy1E1aC9RuIKiIjQOZlc5NbI6?= =?us-ascii?Q?6RU/IVT53YQM/GNt3mStdl5M9JmP+Xy5NqWNS53UQc5DxMz1EQIRV/bLv0ot?= =?us-ascii?Q?6y8C9jNS13SoYmAHegmjlkjQCc1IEr9Z6x2I/IvbcXlWWi8kAWZmn5FmCR8v?= =?us-ascii?Q?Nf3yKrPK7yj0ZYqw4BDECzJFV4YGaCaEQJCp9Fs6l6HW3dLu1h57xTMcCscJ?= =?us-ascii?Q?L1igV+O9WB18gide4Neo+KpmZRSXEuUj0vv2C2coaXTUk2gTU6ezkFbuP98N?= =?us-ascii?Q?ptTcr4mHt0z4Pfw3v/LTi+PRMWD+eZPYEX/fchswlRt4x9dyCQA7TToWLF3k?= =?us-ascii?Q?n+ducDWiHjyAx14YKWYOXHkA9J6EVTaErrcMFYylM2j3jn8k0RNKdNWUEuSt?= =?us-ascii?Q?GNCiyZY6zl1k2lIxFG1HHTkET7HOUk2GwcOQGfZXP4F/9+ZpT26t8UreFGpX?= =?us-ascii?Q?+XugioH2Bo3DJ9qA6DghIgNFq+NPw6Kw3rLSvqiDKMyTW5IFXVwc5S4JeUzt?= =?us-ascii?Q?DkyATME9cg8glhTW7uphPporGAU0Cbz5ItAss+cFQh8cnaqvndRF3yuUJi7m?= =?us-ascii?Q?Ct4eLFhUiWweFFYmFqHzqwVFvvB3nAD5ctPCw2fg+sNMwD5ii8Ek44Gp/ao/?= =?us-ascii?Q?NRDajtdOF9JVpaO1i8pk1Izh31JdVV2py68e9qcLmdGxTptTUSmdLO+xCgHI?= =?us-ascii?Q?8ilgpdIlBOFDVztk7WXdzauDpv1E+ieDWNPx/K5N6f51lkwsw6BNaw/tZrQr?= =?us-ascii?Q?H4T8xvA+/zmJI3OtYaUSURdYBlPj0KQEcNUr51SLb++I22g2L62fUSzVLthw?= =?us-ascii?Q?1lo46Fs2OfzrEMtcUdAhFAsPwfnmgvZr1frNaWwD2fcj0eKukXuahp/epx7o?= =?us-ascii?Q?8NgPpq3HdBl9WsnRzaYRbHKDyY1ZSJ+6P9QHNGRDrvxa5TvOsmDpOAW053R9?= =?us-ascii?Q?JwyeY5u+UkXkho2uCPH5naOeYSHEGEBz4mTkzUVM5ny2VRPWqA0nwQrZ7/n7?= =?us-ascii?Q?PoFpnKYMvuXQEp7y7NdTxSVvz1zTT8uJaZZ4GoNagwRpsSUdgB86XdxgvH0w?= =?us-ascii?Q?EhBVijiJeA=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: 09880af8-ed99-4f0f-0cf2-08df2d15cc87 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:15.9851 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9oFXhbq8/6BtaaNlumJNgp7kngZhZxqANUk+1PGlNjsQA012B9m3/jHLL0RR3Ec5Doqn0efj36roBxeMsbBjFA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.004 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: A3KKHTZ5XZ7TVKJZELHXCMSL2BKBKUJD X-Message-ID-Hash: A3KKHTZ5XZ7TVKJZELHXCMSL2BKBKUJD X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: When a single resource dominates the cluster load, the relative improvement of moving it between two nodes is 2d / (v + d), where v is the load of the resource and d is the difference of the node base loads. Small fluctuations of d around zero are enough to make a motion and, right afterwards, its reverse qualify, so the resource is moved back and forth (see bug #8059). Add the ha-auto-rebalance-cooldown option, which keeps resources that were successfully moved by the load balancer from being considered for another rebalance motion for the given time. The cooldown applies to all resources of a moved resource bundle, as they were all moved, and a bundle is skipped if any of its resources is still on cooldown. The cooldown defaults to 0, i.e. it is disabled and the behavior of existing setups does not change. Like the other auto rebalance state, the cooldown is not persisted across a CRM failover. The expected log of test-crs-dynamic-auto-rebalance7 changes, as vm:101 is not migrated back right after it was migrated, but only after the configured cooldown of 600 seconds has expired. Signed-off-by: Michael Ryom --- src/PVE/HA/Manager.pm | 27 +++++++++++++++++++ .../log.expect | 15 +++++------ 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 18ad12f..c480b73 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -91,6 +91,9 @@ sub new { # rebalance motions which failed $count times in a row, the last # time in rebalancing attempt $attempt failures => {}, + # $sid => $time until which a resource successfully moved by + # load_balance() is not considered for rebalancing + cooldown => {}, }, group_migration_round => 3, # wait a little bit }, $class; @@ -138,6 +141,7 @@ sub update_crs_scheduler_mode { // 3; $self->{crs}->{auto_rebalance}->{margin} = ($crs_cfg->{'ha-auto-rebalance-margin'} // 10) / 100.0; + $self->{crs}->{auto_rebalance}->{cooldown} = $crs_cfg->{'ha-auto-rebalance-cooldown'} // 0; my $old_mode = $self->{crs}->{scheduler}; my $new_mode = $crs_cfg->{ha} || 'basic'; @@ -259,6 +263,10 @@ sub record_rebalance_motion_result { if ($success) { delete $state->{failures}->{$key}; $state->{attempt_ongoing} = 0; + + my $cooldown = $self->{crs}->{auto_rebalance}->{cooldown}; + $state->{cooldown}->{$sid} = $self->{haenv}->get_time() + $cooldown if $cooldown > 0; + return; } @@ -277,6 +285,20 @@ sub record_rebalance_motion_result { $self->{haenv}->log('warning', $msg); } +# Returns whether $sid was moved by load_balance() too recently to be considered +# for rebalancing again. +my sub is_on_rebalance_cooldown { + my ($self, $sid, $now) = @_; + + my $cooldown = $self->{auto_rebalance_state}->{cooldown}; + my $until = $cooldown->{$sid} // return 0; + + return 1 if $now < $until; + + delete $cooldown->{$sid}; + return 0; +} + # Returns a hash of hashes, where each item contains the resource bundle's # leader, the list of HA resources in the resource bundle, and the list of # possible nodes to migrate to. @@ -291,8 +313,13 @@ sub get_resource_migration_candidates { my $resource_bundles = get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity); + my $now = $self->{haenv}->get_time(); + my @compact_migration_candidates = (); for my $leader_sid (sort keys %$resource_bundles) { + my $bundle = $resource_bundles->{$leader_sid}; + next if grep { is_on_rebalance_cooldown($self, $_, $now) } @$bundle; + my $current_leader_node = $ss->{$leader_sid}->{node}; my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() }; diff --git a/src/test/test-crs-dynamic-auto-rebalance7/log.expect b/src/test/test-crs-dynamic-auto-rebalance7/log.expect index c87f487..a82f4bc 100644 --- a/src/test/test-crs-dynamic-auto-rebalance7/log.expect +++ b/src/test/test-crs-dynamic-auto-rebalance7/log.expect @@ -38,13 +38,10 @@ info 103 node2/lrm: starting service vm:101 info 103 node2/lrm: service status vm:101 started info 120 cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0 info 120 cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0 -info 160 node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%) -info 160 node1/crm: got crm command: migrate vm:101 node1 -info 160 node1/crm: migrate service 'vm:101' to node 'node1' -info 160 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node1) -info 163 node2/lrm: service vm:101 - start migrate to node 'node1' -info 163 node2/lrm: service vm:101 - end migrate to node 'node1' -info 180 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1) -info 181 node1/lrm: starting service vm:101 -info 181 node1/lrm: service status vm:101 started +info 700 node1/crm: auto rebalance - migrate vm:101 to node1 (expected change for imbalance from 90.5% to 42.9%) +info 700 node1/crm: got crm command: migrate vm:101 node1 +info 700 node1/crm: migrate service 'vm:101' to node 'node1' +info 700 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node1) +info 703 node2/lrm: service vm:101 - start migrate to node 'node1' +info 703 node2/lrm: service vm:101 - end migrate to node 'node1' info 720 hardware: exit simulation - done -- 2.47.3