Re: [PATCH ha-manager 7/7] manager: try multiple priority classes when applying negative resource affinity

["Daniel Kral" <d.kral@proxmox.com>]

Sorry for the late reply, I had a draft reply for this lying around for
some time, but didn't get to test my statements below until now.

On Wed Apr 29, 2026 at 6:58 PM CEST, Michael Köppl wrote:
> On Wed Apr 22, 2026 at 12:00 PM CEST, Daniel Kral wrote:
>
> [snip]
>
>> @@ -354,7 +363,15 @@ sub select_service_node {
>>          }
>>      }
>>  
>> -    apply_negative_resource_affinity($separate, $pri_nodes);
>> +    # do not consider nodes where HA resources from a possible negative resource
>> +    # affinity rule are running on.
>> +    # as such a negative resource affinity could end up emptying the current
>> +    # priority class, try the succeeding priority classes which result in a
>> +    # non-empty node set or else end up with an empty set.
>> +    do {
>> +        apply_negative_resource_affinity($separate, $pri_nodes);
>> +    } while (keys %$pri_nodes < 1 && ($pri_nodes = shift @$priority_classes));
>> +    $pri_nodes = {} if !defined($pri_nodes);
>
> Just above this, $tried_nodes is removed from $pri_nodes. If the
> $tried_nodes are not also removed from other priority classes, couldn't
> this run into a migration loop where it tries again and again to migrate
> to a failed node from a lower priority class? I'm not entirely sure
> about this, it just seemed a bit odd to me at first glance. The only
> scenario I can think of is one where you have vm:100, vm:101 with a
> non-strict node affinity rule for vm:100 (so you'd have 2 priority
> classes. don't think it's possible otherwise because you cannot combine
> negative resource affinity rule and node affinity rule with priorities.)
> and a negative resource affinity rule that keeps vm:100 and vm:101 apart
> and then migrating vm:100 to the other available node (one not in the
> node affinity rule, so with prio -1 and therefore lower prio class)
> fails.

Good thought!

I'm not sure if I can follow the description correctly, but manually
migrating one of these HA resources to the node outside of the node
affinity rule will either (1) move it back to a node within the node
affinity rule with failback enabled or (2) do nothing if failback is
disabled for the HA resource.

A node is only marked as 'failed to start' if an HA resource is in
'started' state, but starting the HA resource is reported as
unsuccessful by the LRM. Then, the HA Manager calls
select_service_node() with 'try-next', which removes the failed nodes
from the available node set on from highest priority class...

I tried this out in a few different ways, but couldn't come up how it
would end up in a migration loop because of this... Though I found some
odd behavior here as described in the following two scenarios, which is
pre-existing.

Nonetheless, I think it is a good idea to just remove the failed nodes
from $online_nodes at the start to make sure that this doesn't cause any
headaches in the future.

I'll see whether I find any fault with doing this, otherwise I'll
include it in a v2.


== Scenario 1

node-affinity: na-scenario-1
        nodes node1:1,node2:2,node3:3
        resources vm:101

Let's say vm:101 is currently on node3, will always fail on node2, and
has failback enabled, max_relocate = 1, and max_restart = 1.

1. As node3 goes into maintenance mode, vm:101 is automatically moved to
   node2 as a recovery action, because it is the second priority class.

2. As vm:101 always fails on node2, the LRM will mark vm:101 as failed
   starting as the tries run out.

3. The failed starts will trigger the HA Manager to make vm:101 move to
   node1, which is the last remaining node that is available, i.e., not
   in maintenance and vm:101 hasn't failed on it before.
   (as dictated by
    select_service_node(sid = 'vm:101', node_preference = 'try-next'))

4. As vm:101 is acknowledged in state 'started' on node1, the HA Manager
   calls select_service_node(sid = 'vm:101', node_preference = 'none')
   and will not remove the failed nodes at all. The HA Manager will
   decide that node2 is the highest-priority available node (as
   node_preference = 'none', which does not remove the failed nodes from
   the allowed node set) and move vm:101 to node2 again.

5. After vm:101 fails again on node2, the max_relocate condition will be
   finally met (scalar($sd->{failed_nodes}->@*) > $cd->{max_relocate})
   and therefore vm:101 goes into error state after all.


So this doesn't end up in any migration loops.

It is still a surprising behavior as the HA Manager doesn't make node1 a
valid option, because the non-strict node affinity rule and
failback-enabled vm:101 is enforced more strictly than the failed node
set {node2}.

The reason is that in (4) the HA Manager doesn't even have an LRM
response for the vm:101 from the LRM as it just was put in 'started'
state by the HA Manager because of the successful migration to the
target node.

So we should also look into fixing this case as there would have been a
possible running target. Though the solution might require holding the
failed_nodes for longer to not go back to node2 immediately as it is in
'started' state again (and has failback enabled!). But for how long?


== Scenario 2

As I understand your description, your scenario would be something like
the following, where the non-strict node affinity rule implicitly
creates two priority classes for both guests ([{node1,node2}, {node3}]).

node-affinity: na-scenario-2
        nodes node1,node2
        resources vm:101,vm:102

resource-affinity: ra-scenario-2
        resources vm:101,vm:102
        affinity negative

Let's say:
- a 3-node cluster,
- vm:101 is on node1,
- vm:102 is on node2, and
- vm:101 always fails on node3.

1. As node1 goes into maintenance mode, vm:101 is automatically migrated
   to node3 as a recovery action.

2. As vm:101 always fails on node3, the LRM will mark vm:101 as failed
   starting as the tries run out.

3. The failed node set {node3} will trigger the HA Manager to find a new
   node to try restarting vm:101, but
   select_service_node(sid = 'vm:101', node_preference = 'try-next')
   will return node3 as that's the only option for vm:101 now.

Though this would be a very constrained situation... And we cannot do
anything other than maybe setting vm:101 in error state or retrying it
until it hopefully works.

If there would be another possibility in the same node prority class,
e.g. node4 (so [{node1,node2}, {node3,node4}]), then it would select
node4.

If node4 is in the lowest-priority node class (i.e.,
[{node1,node2}, {node3}, {node4}]), then the same as in scenario 1 would
happen as we cannot do anything here either if fallback is enabled.
Though that cannot be, as explicitly multi-prority class node affinity
rules aren't allowed for HA resources which are already in resource
affinity rules.