[PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node

[Daniel Kral <d.kral@proxmox.com>]

There are scenarios, where an HA resource cannot be moved away from its
current node when it changes to maintenance mode. This stems from
select_service_node() not being able to find any replacement node.

A common reason can be that the available node set for these HA
resources is constrained so much by online nodes and/or affinity rules
that there aren't any other options for select_service_node() than the
current node.

While some of these affinity-related scenarios need to be resolved in a
better way, admins should always be warned about such a situation, so
that they know that a node isn't fully in maintenance mode yet.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- reword comment
- put comment at a better place

 src/PVE/HA/Manager.pm                          | 18 +++++++++++++-----
 .../test-stale-maintenance-node/log.expect     |  3 +++
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76e..753da1a8 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -1589,20 +1589,28 @@ sub next_state_started {
                     );
                 }
 
+                # Having the maintenance node set to its current node here means that the
+                # HA resource was never started on a different node since it was set,
+                # because select_service_node() could not find any replacement node.
+                # Common reasons are affinity rules or the whole cluster was shut down at
+                # the same time while the 'migrate' shutdown policy was configured.
                 if ($sd->{maintenance_node} && $sd->{node} eq $sd->{maintenance_node}) {
                     my $node_state = $ns->get_node_state($sd->{node});
                     if ($node_state eq 'online') {
-                        # Having the maintenance node set here means that the service was never
-                        # started on a different node since it was set. This can happen in the edge
-                        # case that the whole cluster is shut down at the same time while the
-                        # 'migrate' policy was configured. Node is not in maintenance mode anymore
-                        # and service is started on this node, so it's fine to clear the setting.
+                        # Node is not in maintenance mode anymore and service is started on this
+                        # node, so it's fine to clear the setting.
                         $haenv->log(
                             'info',
                             "service '$sid': clearing stale maintenance node "
                                 . "'$sd->{maintenance_node}' setting (is current node)",
                         );
                         delete $sd->{maintenance_node};
+                    } else {
+                        $haenv->log(
+                            'warning',
+                            "service '$sid': cannot find a replacement node while"
+                                . " its current node is in maintenance",
+                        );
                     }
                 }
 
diff --git a/src/test/test-stale-maintenance-node/log.expect b/src/test/test-stale-maintenance-node/log.expect
index 092db8be..fce96fd4 100644
--- a/src/test/test-stale-maintenance-node/log.expect
+++ b/src/test/test-stale-maintenance-node/log.expect
@@ -33,6 +33,7 @@ info    120    node3/lrm: shutdown LRM, doing maintenance, removing this node fr
 info    120    node1/crm: node 'node1': state changed from 'online' => 'maintenance'
 info    120    node1/crm: node 'node2': state changed from 'online' => 'maintenance'
 info    120    node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn    120    node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
 info    121    node1/lrm: status change active => maintenance
 info    124    node2/lrm: exit (loop end)
 info    124     shutdown: execute crm node2 stop
@@ -40,6 +41,7 @@ info    123    node2/crm: server received shutdown request
 info    126    node3/lrm: exit (loop end)
 info    126     shutdown: execute crm node3 stop
 info    125    node3/crm: server received shutdown request
+warn    140    node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
 info    143    node2/crm: exit (loop end)
 info    143     shutdown: execute power node2 off
 info    144    node3/crm: exit (loop end)
@@ -64,6 +66,7 @@ info    220      cmdlist: execute power node3 on
 info    220    node3/crm: status change startup => wait_for_quorum
 info    220    node3/lrm: status change startup => wait_for_agent_lock
 info    220    node1/crm: status change wait_for_quorum => master
+warn    220    node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
 info    221    node1/lrm: status change wait_for_agent_lock => active
 info    221    node1/lrm: starting service vm:103
 info    221    node1/lrm: service status vm:103 started
-- 
2.47.3