From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B44361FF0AF for ; Fri, 06 Nov 2026 10:57:13 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 045C5215F7; Fri, 06 Nov 2026 10:57:10 +0100 (CET) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node Date: Fri, 6 Nov 2026 10:56:44 +0100 Message-ID: <20261106095650.72865-2-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261106095650.72865-1-d.kral@proxmox.com> References: <20261106095650.72865-1-d.kral@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793959024360 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.700 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FIB6XN2H3WX72U3CKCO5Y2WFPUX4CC6Q X-Message-ID-Hash: FIB6XN2H3WX72U3CKCO5Y2WFPUX4CC6Q X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: There are scenarios, where an HA resource cannot be moved away from its current node when it changes to maintenance mode. This stems from select_service_node() not being able to find any replacement node. A common reason can be that the available node set for these HA resources is constrained so much by online nodes and/or affinity rules that there aren't any other options for select_service_node() than the current node. While some of these affinity-related scenarios need to be resolved in a better way, admins should always be warned about such a situation, so that they know that a node isn't fully in maintenance mode yet. Signed-off-by: Daniel Kral --- changes since v1: - reword comment - put comment at a better place src/PVE/HA/Manager.pm | 18 +++++++++++++----- .../test-stale-maintenance-node/log.expect | 3 +++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 5840a76e..753da1a8 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -1589,20 +1589,28 @@ sub next_state_started { ); } + # Having the maintenance node set to its current node here means that the + # HA resource was never started on a different node since it was set, + # because select_service_node() could not find any replacement node. + # Common reasons are affinity rules or the whole cluster was shut down at + # the same time while the 'migrate' shutdown policy was configured. if ($sd->{maintenance_node} && $sd->{node} eq $sd->{maintenance_node}) { my $node_state = $ns->get_node_state($sd->{node}); if ($node_state eq 'online') { - # Having the maintenance node set here means that the service was never - # started on a different node since it was set. This can happen in the edge - # case that the whole cluster is shut down at the same time while the - # 'migrate' policy was configured. Node is not in maintenance mode anymore - # and service is started on this node, so it's fine to clear the setting. + # Node is not in maintenance mode anymore and service is started on this + # node, so it's fine to clear the setting. $haenv->log( 'info', "service '$sid': clearing stale maintenance node " . "'$sd->{maintenance_node}' setting (is current node)", ); delete $sd->{maintenance_node}; + } else { + $haenv->log( + 'warning', + "service '$sid': cannot find a replacement node while" + . " its current node is in maintenance", + ); } } diff --git a/src/test/test-stale-maintenance-node/log.expect b/src/test/test-stale-maintenance-node/log.expect index 092db8be..fce96fd4 100644 --- a/src/test/test-stale-maintenance-node/log.expect +++ b/src/test/test-stale-maintenance-node/log.expect @@ -33,6 +33,7 @@ info 120 node3/lrm: shutdown LRM, doing maintenance, removing this node fr info 120 node1/crm: node 'node1': state changed from 'online' => 'maintenance' info 120 node1/crm: node 'node2': state changed from 'online' => 'maintenance' info 120 node1/crm: node 'node3': state changed from 'online' => 'maintenance' +warn 120 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance info 121 node1/lrm: status change active => maintenance info 124 node2/lrm: exit (loop end) info 124 shutdown: execute crm node2 stop @@ -40,6 +41,7 @@ info 123 node2/crm: server received shutdown request info 126 node3/lrm: exit (loop end) info 126 shutdown: execute crm node3 stop info 125 node3/crm: server received shutdown request +warn 140 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance info 143 node2/crm: exit (loop end) info 143 shutdown: execute power node2 off info 144 node3/crm: exit (loop end) @@ -64,6 +66,7 @@ info 220 cmdlist: execute power node3 on info 220 node3/crm: status change startup => wait_for_quorum info 220 node3/lrm: status change startup => wait_for_agent_lock info 220 node1/crm: status change wait_for_quorum => master +warn 220 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance info 221 node1/lrm: status change wait_for_agent_lock => active info 221 node1/lrm: starting service vm:103 info 221 node1/lrm: service status vm:103 started -- 2.47.3