From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 1CC461FF0AD for ; Sun, 20 Sep 2026 18:57:23 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id DE3E821509; Sun, 20 Sep 2026 18:57:22 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xTZaYRpYHyBtUsHRDGn8kLPNrM3yulgzCeNeQ3SrRdRh9vJb6Ac0GaavhK+ZAqZCI07jCv9ZOLrghun+yjV8KmBqCZGEnUQapKOsybFk8PYDGE0mzZwCJpmEaRt40NmjVxzkYg2kxhXrfkFHliPx/eYLsdFDLoiJZVlCOGok8AQ//ctgDFj15bsNZNevN3IWBEkSbEqJ2Nige/8/1e2mOYc7hBFjfDGrCJOqN6rgva0W8PoM0xztMjfYpo3tVycmNFsxCT84i5qQ7h+K9tJ0DKI8gYw0jBosS4H7jDmH6ilalWQJ17xoI79B5XWiuALkTnplGGrXzu5SgSyCiKrjnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fo3UREmfFvnnqKw1Mimy5x5TVhnnVtCdQR4hjUzro0Y=; b=g+iwEfimkIkQWkx3YkqiCW03XhNWRmkRJsaDshOYRrFPhMRy7fuBXmVKsFtwoxjp+bsCGKRy4tl6df0brmUUFsaw4fThpRIkn6UgY/gU8Mok+EIeKbNs4VNI78hij1z5p6NlnPbQdovb4MAYpyblhwmNZc3MK7XhF7MVm1iOv7IUDJFvUDhMRltCWDCUkzZinJOgk8jTxhJL157BdFyZUSkz1MuB9M3B5KQ755XE22cUrR9YagI0fccUIILpTsTGx3cv01qhx+sgA7+o6EizyC73/Ni/FzkSXdSK5J2Hk+zkhOL/+XTdrnlcZx8/Oy48sU7G3tz6Bdp8jAIsAocvjA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fo3UREmfFvnnqKw1Mimy5x5TVhnnVtCdQR4hjUzro0Y=; b=gCYfVf9U7M7GO0Ga6dJcucuzE/7mXqirDUPCDCd4mgxdS/erDDl57MwzR1LUJQHD1Y81MFf+tnIRKETVNy0UZf5mAHTQYJPjvabIJF8A8O4a0RN1Eg8DoR9D8f8geEKD+a9PW3Gu4c1su9B1iC3/kW/JsdcMG9wCyCzSN0gKgleshilbDp68YW+IktAymn/68rxyiMxecRnMf4x3WWyaoCZuFJtbPEp2xAOX13NpUErOrZdgfZZQxuEK/OobLvgIHUa7AauESsXe+NMhM0k4i5Wlpod+/r9uXVDmaRtbmvTv8gbQw74H1vp2/19ngf2lkzszBbBZhVBjY+BSqdBIsQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager 1/7] fix #8059: manager: auto rebalance: back off failed motions and add per-resource cooldown Date: Sun, 20 Sep 2026 18:22:08 +0200 Message-ID: <20260920162220.574802-2-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260920162220.574802-1-Michael@RyomHerold.dk> References: <20260920162220.574802-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: TL2P290CA0008.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:2::11) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|AM0PR10MB883391:EE_ X-MS-Office365-Filtering-Correlation-Id: e92673d9-b39f-4c41-354f-08df1733669f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|366016|10067099003|6133799003|22082099003|18002099003|56012099006; X-Microsoft-Antispam-Message-Info: ui6bFxUuQue+HJCgFglZBUg2Op1nzrDxaulFNa7gltCRmd7Wy4dUVtnWPlO/G+n4LYeX281uq8ip0nHIo0yaNKk53oo4MfPKTa2ulaCe1KhxpRSxletp6d62Y9n9KsLlUDW0CHqdwkSGoJIoI8+tE6j1tk6ktY2rAEESoVwSf4ofNphilUKgQGebffRitbjJJucCuxejKIs8uWeMYeBx4QxCAZ5N9Vgi1ukEcOfigsHP2j81mVd300PcAMBkGPab53kx0elFxhMMzSHajpif9Bmt7GefDUrrkLAUpbNQcUyAdITr+QgScEFZDFSxMM9WmcNs5oiItDEtaBjJALIg6B+NGUgYDyZ7WoiJP8GmWPzuRROwFizPneZxZDbgrSFT2n8uNexbo1Y0vRzMYod7WxKWA7QhSagUnyYAegcBR4WYMjuCi1Xf/DGeHorGAg1fbdT58WvTGGd89XDvDCg8UhcS1ZrCGHsL79qLksZLtOlqZEw9RUPpxguzgjzLSMt0gxbAtu47WB6Dwn09ZnwecQT+cMI0WhvIt3qYmAaS3bYoFyb61ZTYj8w/OatYoEZycHk66gRDSbN8vgQS8GmN4TsVmxqTPdQjzcHQHFOcohfCzhFGZmuLjsycynqj5nEGjh6k8irtmfFCk39iRy1/mbpHyW1uiGwbJDGDpziv4g8= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(366016)(10067099003)(6133799003)(22082099003)(18002099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+W/hgY7YMxoyZv7CfYetsODObYtg3X1M547E09/rwVgw0m06Q9fp4FFMaWyE?= =?us-ascii?Q?UGmlnWoKiyjP1eH0vdEVw2hSlfQ+4Wt4s33a+WOjD2aCdcjQxNNY+ZD1tI1O?= =?us-ascii?Q?v0+8xDW4K+7HPCwwl/f+Fh8PznwnljRuJgWOHO5zs5bykpvtdU2yrEj+cGCf?= =?us-ascii?Q?Ozr1yGzA3mkDt7UEGpYAtV+slpuU4GMzf643or8T8xLrDMHsrjW8m29sT2tB?= =?us-ascii?Q?oNhCMiYDYoX4JdNGP4uVjyZkk89GfbTJizqO3EQJRgic2zgEHwPikfrUyoAj?= =?us-ascii?Q?hdbeEeZukpGQHrVNn7g46PnQ1Nu86WwhyewUwda4cR6f2P/5rDm3ROyiSe21?= =?us-ascii?Q?VO/cJQy1hUH9as4cZtekJuAJ9C9UBWEdZdzpOs7q5Lw0eqVckm7vl43oqJHV?= =?us-ascii?Q?WGZBosWgsr/6270pUDujayBZmCAuDlEU1q813WMlJCTkcQ+jrUvKIxZVKaBy?= =?us-ascii?Q?5TytqfzRW2BMQMcgYtk5cWUmPRc69YwbKR2rWyE+zLI/ZbVQmoYSIIA+iNX+?= =?us-ascii?Q?SKOZf1hEydRu4XFLcguK9bIBW3TUkfcPqOBjZC40l6lV4xjit0yqYyWCzce1?= =?us-ascii?Q?2ARuDn8P7KYl3Ky+YAUzY5tdtVjuqDOQbAVVglmVrnOsbCuQjeI/eLXxIA9V?= =?us-ascii?Q?7zkIXGQ7QE/xOb4qtAep2rK5JO+dOCZOjWMrubL668LE+GsmladB4pnMibqV?= =?us-ascii?Q?Gic9s3rWrwpfQvpmDrPqNm7QVtf3iVA+sBtzaHM66xAVDiohURCZVb1pHnfG?= =?us-ascii?Q?MIfVpY9UKes/HoCgBx22Fcxxb83IksuvVcFpZdA5Exg6wyYoLJJNUhI/3m8m?= =?us-ascii?Q?YzvI/++GlXOh/O2Am11sNpGECYuauutaOEP4d5Tp3g4Oma9h6Ie6+Yp/17xd?= =?us-ascii?Q?p5JZZdjp8aUTu/gcoC5nehygcl5uBewvbkFRqyyWmrNqPO1JRTyLlwiK22Ev?= =?us-ascii?Q?kj9qzLLksd9cEpTSRYRgSV4mEX3dOKLyGfR5w4wKoZYhfT9V3z2eNtUjV+4c?= =?us-ascii?Q?qbuOHBDVvHFU/FczzppaC2eJwm/ArNqIZsZviVooaeBtXKn/YXLEzHcbbKNm?= =?us-ascii?Q?xtycNe/ysvYMAWKScSMEu+rsCUL6MPPm7THAZ10d+Aq+3T8kUdXx+PsArvQM?= =?us-ascii?Q?DVToo5ta/oroNYCpPda/QgZTo866teNjyK9b42i+KbtdVqIjv99yZaxFhoRl?= =?us-ascii?Q?Qw4QpDM3gMsctsWsxYOdDDuXk6QcPzxtDlBuwsThzogIUaVvNulROGIUQeY5?= =?us-ascii?Q?RmUcoogUmQZBwWnljSp6Kjm2arHewmhi+gI3be0BV7v5R8dLUwKNa4WcMzxK?= =?us-ascii?Q?dTQHlLnDvBNat9KIzR09vKiOoeqIFSCow1/qi1TqqFMcW6SjVbHpadQeEPge?= =?us-ascii?Q?XnIVnhhwqDoqI8kWG6PVwYGIQkKF0mg+HrERa2dgYjY5PqMr7xpJzlOt4Of8?= =?us-ascii?Q?VaeoE18vq48Iu+JMqofw7yxPu0Pi1gJhtbyYun8h8NnfUQ4auoc5ZItZoUcf?= =?us-ascii?Q?na2Swmccc3AUhL+an96BArTwO3XEAZYlWGDPuBu7cJQDG/YqE8sHfNFVMrdZ?= =?us-ascii?Q?wnRVa7CAXf6lz0qo99CTITeXmj3HXwoBz0xV+dCVgni0nS3UW/wiaLY6PJYp?= =?us-ascii?Q?jWTOO6nHGSmoLH/jBcziM6zm51SnROYofcz8ABJXVfOpKXdX9oQ4B/gGvzS7?= =?us-ascii?Q?StUUphCWhI/rl8KOVMAKNMgZLhJX14q9bZGAejY5mL3zE5tWRwLuUUBy4gaX?= =?us-ascii?Q?GhqECoxdEw=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: e92673d9-b39f-4c41-354f-08df1733669f X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2026 16:22:44.6108 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GgPK/axzshsjZRv7Qxlm50P/L2ACSFzq06M94aUftECdOSnzKDxzLbvuV8om7BgQYccKdR7qDcpoxm64W/6bPw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB883391 X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FRP2EFIKPSEIWDT3ZRR46Z5CMVN7IRTC X-Message-ID-Hash: FRP2EFIKPSEIWDT3ZRR46Z5CMVN7IRTC X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The automatic load balancer only refrains from issuing new motions while a migrate/relocate is queued or running. It never records the outcome of a motion it issued. If the motion fails (e.g. because the guest has a local CD-ROM image attached), the resource returns to 'started' on its source node, and the very same motion is selected again as soon as the hold duration has passed. This results in an endless loop of failing migrations, in practice about every 40 seconds with the default hold duration. Track the last motion issued by load_balance() and, once it is no longer queued or running, check whether the resource ended up on the target node: - on failure, back off that (resource, target) pair with an exponential delay (60s, 120s, ... capped at 1h) and log a warning; - on success, put the resource on a cooldown (10 minutes), so that a resource is not moved back and forth in quick succession, which can happen when a single resource dominates the cluster load and the node base loads fluctuate around each other. Backed-off targets and resources on cooldown are pruned in get_resource_migration_candidates() before scoring, so the balancer can still pick other, possibly worse but feasible, migrations. Like sustained_imbalance_round, this state is not persisted across a CRM failover. The delays are constants for now; they could be exposed as ha-auto-rebalance-* options later. The expected log of test-crs-dynamic-auto-rebalance-topsis3 changes, because the resource vm:105, which was moved to node2 when its load was high, is on cooldown when its load has gone down again and is not moved back immediately anymore; the balancer instead improves the imbalance by moving vm:106. Add two regression tests: one for the backoff of failed motions (using a fa: resource, which fails its first two relocations) and one for the cooldown suppressing an immediate qualifying reverse migration after a successful rebalance motion. Signed-off-by: Michael Ryom --- src/PVE/HA/Manager.pm | 118 ++++++++++++++++++ .../log.expect | 18 +-- .../test-crs-dynamic-auto-rebalance5/README | 13 ++ .../test-crs-dynamic-auto-rebalance5/cmdlist | 3 + .../datacenter.cfg | 6 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 77 ++++++++++++ .../manager_status | 1 + .../service_config | 6 + .../static_service_stats | 6 + .../test-crs-dynamic-auto-rebalance6/README | 17 +++ .../test-crs-dynamic-auto-rebalance6/cmdlist | 7 ++ .../datacenter.cfg | 6 + .../dynamic_service_stats | 5 + .../hardware_status | 4 + .../log.expect | 41 ++++++ .../manager_status | 1 + .../service_config | 5 + .../static_service_stats | 5 + 20 files changed, 341 insertions(+), 9 deletions(-) create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 5840a76..30b9ae2 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -72,6 +72,15 @@ sub new { # this is not persisted for a CRM failover as in the mean time # the usage statistics might have change quite a bit already sustained_imbalance_round => 0, + # outcome tracking for resource motions issued by load_balance() + # + # like sustained_imbalance_round, this is not persisted for a CRM + # failover + auto_rebalance_state => { + pending => undef, # last issued motion, until it has finished + failures => {}, # "$sid:$target" => { count => $count, until => $time } + cooldown => {}, # $sid => $time + }, group_migration_round => 3, # wait a little bit }, $class; @@ -135,6 +144,92 @@ sub update_crs_scheduler_mode { return; } +# delay before a failed rebalance motion to the same target node is retried, +# doubled for each consecutive failure up to the maximum +my $auto_rebalance_failure_backoff_base = 60; +my $auto_rebalance_failure_backoff_max = 3600; +# minimum time before a resource is considered for rebalancing again after it +# was successfully moved by the load balancer +my $auto_rebalance_resource_cooldown = 600; + +my $is_on_rebalance_cooldown = sub { + my ($self, $sid, $now) = @_; + + my $cooldown = $self->{auto_rebalance_state}->{cooldown}; + my $until = $cooldown->{$sid} // return 0; + + if ($now >= $until) { + delete $cooldown->{$sid}; + return 0; + } + + return 1; +}; + +my $is_rebalance_motion_backed_off = sub { + my ($self, $sid, $target, $now) = @_; + + my $failure = $self->{auto_rebalance_state}->{failures}->{"$sid:$target"} // return 0; + + # keep the entry after expiry, so that the failure count keeps growing if + # the next attempt fails as well; it is only reset on success + return $now < $failure->{until} ? 1 : 0; +}; + +# Checks whether the last resource motion issued by load_balance() has +# finished and records its outcome: a per-resource cooldown on success, or an +# exponential backoff for the ($sid, $target) pair on failure. +my $check_last_rebalance_motion = sub { + my ($self) = @_; + + my ($haenv, $ss) = $self->@{qw(haenv ss)}; + my $state = $self->{auto_rebalance_state}; + + my $pending = $state->{pending} // return; + my ($sid, $target, $task) = $pending->@{qw(sid target task)}; + + my $sd = $ss->{$sid}; + if (!defined($sd)) { + # resource was removed in the meantime + $state->{pending} = undef; + return; + } + + my ($cmd, $sstate) = $sd->@{qw(cmd state)}; + + # still queued or in progress + return if $sstate eq 'migrate' || $sstate eq 'relocate'; + return if defined($cmd) && ($cmd->[0] eq 'migrate' || $cmd->[0] eq 'relocate'); + + $state->{pending} = undef; + + my $now = $haenv->get_time(); + my $key = "$sid:$target"; + + if (defined($sd->{node}) && $sd->{node} eq $target) { + delete $state->{failures}->{$key}; + $state->{cooldown}->{$sid} = $now + $auto_rebalance_resource_cooldown; + return; + } + + my $failure = $state->{failures}->{$key} //= { count => 0 }; + $failure->{count}++; + + my $delay = $auto_rebalance_failure_backoff_base * 2**($failure->{count} - 1); + $delay = $auto_rebalance_failure_backoff_max + if $delay > $auto_rebalance_failure_backoff_max; + + $failure->{until} = $now + $delay; + + $haenv->log( + 'warning', + "auto rebalance - $task $sid to $target did not succeed" + . " (attempt $failure->{count}), not retrying it for ${delay}s", + ); + + return; +}; + # Returns a hash of lists, which contain the running, movable, non-moving HA resource # bundles, which are on the same node, implied by the strict positive resource # affinity rules. @@ -200,9 +295,12 @@ sub get_resource_migration_candidates { my $resource_bundles = get_active_stationary_movable_resource_bundles($ss, $sc, $resource_affinity); + my $now = $self->{haenv}->get_time(); my @compact_migration_candidates = (); for my $leader_sid (sort keys %$resource_bundles) { + next if $is_on_rebalance_cooldown->($self, $leader_sid, $now); + my $current_leader_node = $ss->{$leader_sid}->{node}; my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() }; @@ -213,6 +311,11 @@ sub get_resource_migration_candidates { delete $target_nodes->{$current_leader_node}; + for my $node (keys %$target_nodes) { + delete $target_nodes->{$node} + if $is_rebalance_motion_backed_off->($self, $leader_sid, $node, $now); + } + next if !%$target_nodes; push @compact_migration_candidates, @@ -234,6 +337,10 @@ sub load_balance { return if !$auto_rebalance_opts->{enable}; return if $crs->{scheduler} ne 'static' && $crs->{scheduler} ne 'dynamic'; + + # record the outcome of the last motion issued here, if it has finished + $check_last_rebalance_motion->($self); + return if $self->any_resource_motion_queued_or_running(); my ($threshold, $method, $hold_duration, $margin) = @@ -295,6 +402,17 @@ sub load_balance { $haenv->log('info', "auto rebalance - $task $sid to $target ($imbalance_change_str)"); $self->queue_resource_motion($cmd, $task, $sid, $target); + + # queue_resource_motion() may refuse to queue the motion (e.g. blocking + # resources on the target node), so only track it if it was queued + if (defined($self->{ss}->{$sid}->{cmd})) { + $self->{auto_rebalance_state}->{pending} = { + sid => $sid, + source => $source, + target => $target, + task => $task, + }; + } } sub cleanup { diff --git a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect index 2f19c72..dfbc904 100644 --- a/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect +++ b/src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect @@ -77,13 +77,13 @@ info 263 node2/lrm: service vm:103 - end migrate to node 'node3' info 280 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node3) info 285 node3/lrm: starting service vm:103 info 285 node3/lrm: service status vm:103 started -info 340 node1/crm: auto rebalance - migrate vm:105 to node1 (expected change for imbalance from 30.1% to 17.0%) -info 340 node1/crm: got crm command: migrate vm:105 node1 -info 340 node1/crm: migrate service 'vm:105' to node 'node1' -info 340 node1/crm: service 'vm:105': state changed from 'started' to 'migrate' (node = node2, target = node1) -info 343 node2/lrm: service vm:105 - start migrate to node 'node1' -info 343 node2/lrm: service vm:105 - end migrate to node 'node1' -info 360 node1/crm: service 'vm:105': state changed from 'migrate' to 'started' (node = node1) -info 361 node1/lrm: starting service vm:105 -info 361 node1/lrm: service status vm:105 started +info 340 node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 30.1% to 20.5%) +info 340 node1/crm: got crm command: migrate vm:106 node1 +info 340 node1/crm: migrate service 'vm:106' to node 'node1' +info 340 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node1) +info 345 node3/lrm: service vm:106 - start migrate to node 'node1' +info 345 node3/lrm: service vm:106 - end migrate to node 'node1' +info 360 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node1) +info 361 node1/lrm: starting service vm:106 +info 361 node1/lrm: service status vm:106 started info 820 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README new file mode 100644 index 0000000..c24c44d --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/README @@ -0,0 +1,13 @@ +Test that the auto rebalance system backs off failed rebalance motions +instead of retrying them indefinitely. + +The HA resource fa:102 fails its first two migrate/relocate attempts. The +auto rebalancer first selects fa:102 to be relocated to node2, which fails. +The (fa:102, node2) pair is backed off for 60 seconds, so on the next +balancing round the rebalancer relocates fa:102 to node3 instead, which +fails as well and backs off the (fa:102, node3) pair. On the third attempt, +the backoff for node2 has expired and the relocation to node2 succeeds. + +After the successful relocation, fa:102 is put on the per-resource cooldown, +so the following balancing round moves vm:102 to node3 instead to reach the +minimum cluster node imbalance. diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist new file mode 100644 index 0000000..13f90cd --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist @@ -0,0 +1,3 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats new file mode 100644 index 0000000..4eba71c --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats @@ -0,0 +1,6 @@ +{ + "fa:102": { "cpu": 1.0, "mem": 4294967296 }, + "vm:102": { "cpu": 1.0, "mem": 4294967296 }, + "vm:103": { "cpu": 1.0, "mem": 4294967296 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status new file mode 100644 index 0000000..ce8cf0e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect new file mode 100644 index 0000000..1267366 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect @@ -0,0 +1,77 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'fa:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node1' +info 20 node1/crm: adding new service 'vm:104' on node 'node1' +info 20 node1/crm: service 'fa:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service fa:102 +info 21 node1/lrm: service status fa:102 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 21 node1/lrm: starting service vm:103 +info 21 node1/lrm: service status vm:103 started +info 21 node1/lrm: starting service vm:104 +info 21 node1/lrm: service status vm:104 started +info 22 node2/crm: status change wait_for_quorum => slave +info 24 node3/crm: status change wait_for_quorum => slave +info 80 node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 80 node1/crm: got crm command: relocate fa:102 node2 +info 80 node1/crm: relocate service 'fa:102' to node 'node2' +info 80 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node2) +err 81 node1/lrm: service fa:102 not moved (migration error) +info 83 node2/lrm: got lock 'ha_agent_node2_lock' +info 83 node2/lrm: status change wait_for_agent_lock => active +err 100 node1/crm: service 'fa:102' - migration failed (exit code 1) +info 100 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node1) +warn 120 node1/crm: auto rebalance - relocate fa:102 to node2 did not succeed (attempt 1), not retrying it for 60s +info 160 node1/crm: auto rebalance - relocate fa:102 to node3 (expected change for imbalance from 100.0% to 66.1%) +info 160 node1/crm: got crm command: relocate fa:102 node3 +info 160 node1/crm: relocate service 'fa:102' to node 'node3' +info 160 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node3) +err 161 node1/lrm: service fa:102 not moved (migration error) +info 165 node3/lrm: got lock 'ha_agent_node3_lock' +info 165 node3/lrm: status change wait_for_agent_lock => active +err 180 node1/crm: service 'fa:102' - migration failed (exit code 1) +info 180 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node1) +warn 200 node1/crm: auto rebalance - relocate fa:102 to node3 did not succeed (attempt 1), not retrying it for 60s +info 240 node1/crm: auto rebalance - relocate fa:102 to node2 (expected change for imbalance from 100.0% to 66.1%) +info 240 node1/crm: got crm command: relocate fa:102 node2 +info 240 node1/crm: relocate service 'fa:102' to node 'node2' +info 240 node1/crm: service 'fa:102': state changed from 'started' to 'relocate' (node = node1, target = node2) +info 241 node1/lrm: service fa:102 - start relocate to node 'node2' +info 241 node1/lrm: stopping service fa:102 (relocate) +info 241 node1/lrm: service status fa:102 stopped +info 241 node1/lrm: service fa:102 - end relocate to node 'node2' +info 260 node1/crm: service 'fa:102': state changed from 'relocate' to 'started' (node = node2) +info 263 node2/lrm: starting service fa:102 +info 263 node2/lrm: service status fa:102 started +info 320 node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 66.1% to 25.0%) +info 320 node1/crm: got crm command: migrate vm:102 node3 +info 320 node1/crm: migrate service 'vm:102' to node 'node3' +info 320 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3) +info 321 node1/lrm: service vm:102 - start migrate to node 'node3' +info 321 node1/lrm: service vm:102 - end migrate to node 'node3' +info 340 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3) +info 345 node3/lrm: starting service vm:102 +info 345 node3/lrm: service status vm:102 started +info 620 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config new file mode 100644 index 0000000..6c4ad4b --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config @@ -0,0 +1,6 @@ +{ + "fa:102": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node1", "state": "started" }, + "vm:104": { "node": "node1", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats new file mode 100644 index 0000000..41dc446 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats @@ -0,0 +1,6 @@ +{ + "fa:102": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 }, + "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README new file mode 100644 index 0000000..1931f77 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/README @@ -0,0 +1,17 @@ +Test that a HA resource which was successfully moved by the auto rebalancer +is put on a cooldown, during which it is not moved again, even if a +qualifying rebalance motion exists. + +The cluster has two nodes, where each node has a HA resource excluded from +the auto rebalancing (vm:100 on node1 and vm:200 on node2), which act as the +node base loads, and one movable HA resource vm:101 on node1. Initially, +node1 is much more loaded than node2, so vm:101 is migrated to node2. + +Afterwards, the base loads are swapped, i.e. node2 becomes much more loaded +than node1. Without the cooldown, the auto rebalancer would immediately +migrate vm:101 back to node1, as the expected relative imbalance improvement +qualifies. With the cooldown, vm:101 must not be moved again for 10 minutes, +so there are no further migrations until the end of the test. + +This mimics a single dominant HA resource oscillating between two nodes, +whose base loads fluctuate around each other. diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist new file mode 100644 index 0000000..1de33e4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist @@ -0,0 +1,7 @@ +[ + [ "power node1 on", "power node2 on" ], + [ + "service vm:100 set-dynamic-stats cpu 0.96 mem 0", + "service vm:200 set-dynamic-stats cpu 14.4 mem 0" + ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats new file mode 100644 index 0000000..3d2d743 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats @@ -0,0 +1,5 @@ +{ + "vm:100": { "cpu": 16.8, "mem": 0 }, + "vm:101": { "cpu": 4.8, "mem": 0 }, + "vm:200": { "cpu": 0.96, "mem": 0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status new file mode 100644 index 0000000..864adb3 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status @@ -0,0 +1,4 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect new file mode 100644 index 0000000..37f5cce --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect @@ -0,0 +1,41 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:100' on node 'node1' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:200' on node 'node2' +info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:200': state changed from 'request_start' to 'started' (node = node2) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:100 +info 21 node1/lrm: service status vm:100 started +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:200 +info 23 node2/lrm: service status vm:200 started +info 80 node1/crm: auto rebalance - migrate vm:101 to node2 (expected change for imbalance from 91.5% to 48.9%) +info 80 node1/crm: got crm command: migrate vm:101 node2 +info 80 node1/crm: migrate service 'vm:101' to node 'node2' +info 80 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2) +info 81 node1/lrm: service vm:101 - start migrate to node 'node2' +info 81 node1/lrm: service vm:101 - end migrate to node 'node2' +info 100 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2) +info 103 node2/lrm: starting service vm:101 +info 103 node2/lrm: service status vm:101 started +info 120 cmdlist: execute service vm:100 set-dynamic-stats cpu 0.96 mem 0 +info 120 cmdlist: execute service vm:200 set-dynamic-stats cpu 14.4 mem 0 +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config new file mode 100644 index 0000000..83bf42e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config @@ -0,0 +1,5 @@ +{ + "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 }, + "vm:101": { "node": "node1", "state": "started" }, + "vm:200": { "node": "node2", "state": "started", "auto-rebalance": 0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats new file mode 100644 index 0000000..f7d1a27 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats @@ -0,0 +1,5 @@ +{ + "vm:100": { "maxcpu": 20.0, "maxmem": 8589934592 }, + "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 }, + "vm:200": { "maxcpu": 16.0, "maxmem": 8589934592 } +} -- 2.47.3