From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 87D251FF0BC for ; Thu, 29 Oct 2026 10:32:17 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 891DE21226; Thu, 29 Oct 2026 10:32:14 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Thu, 29 Oct 2026 10:32:08 +0100 Message-Id: Subject: Re: [PATCH ha-manager 7/7] manager: try multiple priority classes when applying negative resource affinity From: "Daniel Kral" To: =?utf-8?q?Michael_K=C3=B6ppl?= , Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: aerc 0.21.0-145-gf21bb67f8cab-dirty References: <20260422100035.232716-1-d.kral@proxmox.com> <20260422100035.232716-8-d.kral@proxmox.com> In-Reply-To: X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793266328947 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.706 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: FZW2EXCF4C4WAOZMUFRNC5OZPREBP6I5 X-Message-ID-Hash: FZW2EXCF4C4WAOZMUFRNC5OZPREBP6I5 X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Sorry for the late reply, I had a draft reply for this lying around for some time, but didn't get to test my statements below until now. On Wed Apr 29, 2026 at 6:58 PM CEST, Michael K=C3=B6ppl wrote: > On Wed Apr 22, 2026 at 12:00 PM CEST, Daniel Kral wrote: > > [snip] > >> @@ -354,7 +363,15 @@ sub select_service_node { >> } >> } >> =20 >> - apply_negative_resource_affinity($separate, $pri_nodes); >> + # do not consider nodes where HA resources from a possible negative= resource >> + # affinity rule are running on. >> + # as such a negative resource affinity could end up emptying the cu= rrent >> + # priority class, try the succeeding priority classes which result = in a >> + # non-empty node set or else end up with an empty set. >> + do { >> + apply_negative_resource_affinity($separate, $pri_nodes); >> + } while (keys %$pri_nodes < 1 && ($pri_nodes =3D shift @$priority_c= lasses)); >> + $pri_nodes =3D {} if !defined($pri_nodes); > > Just above this, $tried_nodes is removed from $pri_nodes. If the > $tried_nodes are not also removed from other priority classes, couldn't > this run into a migration loop where it tries again and again to migrate > to a failed node from a lower priority class? I'm not entirely sure > about this, it just seemed a bit odd to me at first glance. The only > scenario I can think of is one where you have vm:100, vm:101 with a > non-strict node affinity rule for vm:100 (so you'd have 2 priority > classes. don't think it's possible otherwise because you cannot combine > negative resource affinity rule and node affinity rule with priorities.) > and a negative resource affinity rule that keeps vm:100 and vm:101 apart > and then migrating vm:100 to the other available node (one not in the > node affinity rule, so with prio -1 and therefore lower prio class) > fails. Good thought! I'm not sure if I can follow the description correctly, but manually migrating one of these HA resources to the node outside of the node affinity rule will either (1) move it back to a node within the node affinity rule with failback enabled or (2) do nothing if failback is disabled for the HA resource. A node is only marked as 'failed to start' if an HA resource is in 'started' state, but starting the HA resource is reported as unsuccessful by the LRM. Then, the HA Manager calls select_service_node() with 'try-next', which removes the failed nodes from the available node set on from highest priority class... I tried this out in a few different ways, but couldn't come up how it would end up in a migration loop because of this... Though I found some odd behavior here as described in the following two scenarios, which is pre-existing. Nonetheless, I think it is a good idea to just remove the failed nodes from $online_nodes at the start to make sure that this doesn't cause any headaches in the future. I'll see whether I find any fault with doing this, otherwise I'll include it in a v2. =3D=3D Scenario 1 node-affinity: na-scenario-1 nodes node1:1,node2:2,node3:3 resources vm:101 Let's say vm:101 is currently on node3, will always fail on node2, and has failback enabled, max_relocate =3D 1, and max_restart =3D 1. 1. As node3 goes into maintenance mode, vm:101 is automatically moved to node2 as a recovery action, because it is the second priority class. 2. As vm:101 always fails on node2, the LRM will mark vm:101 as failed starting as the tries run out. 3. The failed starts will trigger the HA Manager to make vm:101 move to node1, which is the last remaining node that is available, i.e., not in maintenance and vm:101 hasn't failed on it before. (as dictated by select_service_node(sid =3D 'vm:101', node_preference =3D 'try-next')) 4. As vm:101 is acknowledged in state 'started' on node1, the HA Manager calls select_service_node(sid =3D 'vm:101', node_preference =3D 'none') and will not remove the failed nodes at all. The HA Manager will decide that node2 is the highest-priority available node (as node_preference =3D 'none', which does not remove the failed nodes from the allowed node set) and move vm:101 to node2 again. 5. After vm:101 fails again on node2, the max_relocate condition will be finally met (scalar($sd->{failed_nodes}->@*) > $cd->{max_relocate}) and therefore vm:101 goes into error state after all. So this doesn't end up in any migration loops. It is still a surprising behavior as the HA Manager doesn't make node1 a valid option, because the non-strict node affinity rule and failback-enabled vm:101 is enforced more strictly than the failed node set {node2}. The reason is that in (4) the HA Manager doesn't even have an LRM response for the vm:101 from the LRM as it just was put in 'started' state by the HA Manager because of the successful migration to the target node. So we should also look into fixing this case as there would have been a possible running target. Though the solution might require holding the failed_nodes for longer to not go back to node2 immediately as it is in 'started' state again (and has failback enabled!). But for how long? =3D=3D Scenario 2 As I understand your description, your scenario would be something like the following, where the non-strict node affinity rule implicitly creates two priority classes for both guests ([{node1,node2}, {node3}]). node-affinity: na-scenario-2 nodes node1,node2 resources vm:101,vm:102 resource-affinity: ra-scenario-2 resources vm:101,vm:102 affinity negative Let's say: - a 3-node cluster, - vm:101 is on node1, - vm:102 is on node2, and - vm:101 always fails on node3. 1. As node1 goes into maintenance mode, vm:101 is automatically migrated to node3 as a recovery action. 2. As vm:101 always fails on node3, the LRM will mark vm:101 as failed starting as the tries run out. 3. The failed node set {node3} will trigger the HA Manager to find a new node to try restarting vm:101, but select_service_node(sid =3D 'vm:101', node_preference =3D 'try-next') will return node3 as that's the only option for vm:101 now. Though this would be a very constrained situation... And we cannot do anything other than maybe setting vm:101 in error state or retrying it until it hopefully works. If there would be another possibility in the same node prority class, e.g. node4 (so [{node1,node2}, {node3,node4}]), then it would select node4. If node4 is in the lowest-priority node class (i.e., [{node1,node2}, {node3}, {node4}]), then the same as in scenario 1 would happen as we cannot do anything here either if fallback is enabled. Though that cannot be, as explicitly multi-prority class node affinity rules aren't allowed for HA resources which are already in resource affinity rules.