[pve-devel] [PATCH docs 1/5] ha: affinity rules: simplify overly verbose rule conflicts and errors section

[pve-devel] [PATCH docs 1/5] ha: affinity rules: simplify overly verbose rule conflicts and errors section

[Daniel Kral]

Suggested-by: Hannes Dürr <h.duerr@proxmox.com>
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
I only included some of the suggestions for the rule conflicts & errors
part from Hannes for now as these are more important to be correct and
understandable for the release.

 ha-manager.adoc | 29 ++++++++++-------------------
 1 file changed, 10 insertions(+), 19 deletions(-)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index 77d7a65..d6ac75d 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -849,34 +849,25 @@ Rule Conflicts and Errors
 HA rules can impose rather complex constraints on the HA resources. To ensure
 that a new or modified HA rule does not introduce uncertainty into the HA
 stack's CRS scheduler, HA rules are tested for feasibility before these are
-applied. If a rule does fail any of these tests, the rule is disabled until the
-conflicts and errors is resolved.
+applied. For any rules that fail these tests, these rules are disabled until
+the conflicts and errors have been resolved.
 
 Currently, HA rules are checked for the following feasibility tests:
 
-* A HA resource can only be referenced by a single HA node affinity rule in
-  total. If two or more HA node affinity rules specify the same HA resource,
-  these HA node affinity rules will be disabled.
+* An HA resource can only be part of a single HA node affinity rule.
 
-* A HA resource affinity rule must specify at least two HA resources to be
-  feasible. If a HA resource affinity rule specifies only one HA resource, the
-  HA resource affinity rule will be disabled.
+* An HA resource affinity rule must have at least two HA resources.
 
-* A HA resource affinity rule must specify no more HA resources than there are
-  nodes in the cluster. If a HA resource affinity rule specifies more HA
-  resources than there are in the cluster, the HA resource affinity rule will
-  be disabled.
+* A negative HA resource affinity rule cannot specify more HA resources than
+  there are nodes in the cluster. Otherwise, the HA resources do not have
+  enough nodes to be separated.
 
 * A positive HA resource affinity rule cannot specify the same two or more HA
   resources as a negative HA resources affinity rule. That is, two or more HA
-  resources cannot be kept together and separate at the same time. If any pair
-  of positive and negative HA resource affinity rules do specify the same two
-  or more HA resources, both HA resource affinity rules will be disabled.
+  resources cannot be kept together and separate at the same time.
 
-* A HA resource can only be referenced by either a HA node affinity rule or HA
-  resource affinity rules, but not both at the same time. If there is at least
-  one HA node affinity rule and at least one HA resource affinity rule, which
-  specify the same HA resource, these rules will be disabled.
+* A HA resource can only be part of either a HA node affinity rule or HA
+  resource affinity rules, but not both at the same time.
 
 [[ha_manager_fencing]]
 Fencing
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docs 2/5] ha: rules: update about mixed usage of node and resource affinity rules

[Daniel Kral]

As HA resources can be part of node and resource affinity rules at the
same time in simpler cases, update the relevant sections about specific
interactions and restrictions.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 ha-manager.adoc | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index d6ac75d..5d75287 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -837,6 +837,23 @@ Two or more HA resources cannot be kept on the same node and separated on
 different nodes at the same time. For more information on these cases, see the
 section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.
 
+Interactions between Node and Positive Resource Affinity Rules
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+If there are HA resources in a node affinity rule, which are also part of a
+positive resource affinity rules, then all the other HA resources in the
+positive resource affinity rule inherit the node affinity rule as well.
+
+For example, if the HA resources `vm:100`, `vm:101`, and `vm:102` are in a
+positive resource affinity rule, and `vm:102` is in a node affinity rule, which
+restricts `vm:102` to be only on `node3`, then `vm:100` and `vm:101` are
+restricted to be only on `node3` as well.
+
+Note that if there are two or more HA resources of a positive resource affinity
+rules, which are in different node affinity rules, then those will be disabled
+as it is currently not supported. For more information on these cases, see the
+section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.
+
 Resource Affinity Rule Properties
 +++++++++++++++++++++++++++++++++
 
@@ -866,8 +883,16 @@ Currently, HA rules are checked for the following feasibility tests:
   resources as a negative HA resources affinity rule. That is, two or more HA
   resources cannot be kept together and separate at the same time.
 
-* A HA resource can only be part of either a HA node affinity rule or HA
-  resource affinity rules, but not both at the same time.
+* An HA resource can only be part of a HA node affinity rule and a HA resource
+  affinity rule at the same time, if the HA node affinity rule has a single
+  priority class.
+
+* The HA resources of a positive HA resource affinity rule can only be part of
+  a single HA node affinity rule at most.
+
+* The HA resources of a negative HA resource affinity rule cannot be restricted
+  to less nodes than HA resources by their node affinity rules. Otherwise, the
+  HA resources do not have enough nodes to be separated.
 
 [[ha_manager_fencing]]
 Fencing
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docs 3/5] ha: rules: document crs behavior for split positive resource affinity

[Daniel Kral]

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 ha-manager.adoc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index 5d75287..e18a14d 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -787,6 +787,11 @@ HA resources `vm:101` and `vm:102` are each in a positive resource affinity
 rule, then it is the same as if `vm:100`, `vm:101` and `vm:102` would have been
 in a single positive resource affinity rule.
 
+NOTE: If the HA resources of a positive resource affinity rule are currently
+running on different nodes, the CRS will move the HA resources to the node,
+where most of them are running already. If there is a tie in the HA resource
+count, the node with the alphabetically first name is chosen.
+
 However, suppose there are computationally expensive, and/or distributed
 programs running on the HA resources `vm:200` and `ct:300`, for example,
 sharded database instances. In that case, running them on the same node could
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docs 4/5] ha: mark ha groups as deprecated and note migration to node affinity rules

[Daniel Kral]

As the migration is handled automatically by the HA Manager now instead
of manual user migration, the notice should be placed in the HA Groups
section instead.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
Alternatively, we could already remove the HA groups section now/after
the next patch, which replaces any references to groups in the remaining
text in ha-manager.adoc (AFAIK there are still some minor references in
pve-docs, e.g. ha-groups-opts.adoc, references to
/etc/pve/ha/groups.cfg, etc.).

 ha-manager.adoc | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index e18a14d..ffab83c 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -594,6 +594,9 @@ The above config was generated using the `ha-manager` command-line tool:
 Groups
 ~~~~~~
 
+NOTE: HA Groups are deprecated and migrated to HA Node Affinity rules since
+Proxmox VE 9.0.
+
 [thumbnail="screenshot/gui-ha-manager-groups-view.png"]
 
 The HA group configuration file `/etc/pve/ha/groups.cfg` is used to
@@ -700,9 +703,6 @@ on the same node.
 Node Affinity Rules
 ^^^^^^^^^^^^^^^^^^^
 
-NOTE: HA Node Affinity rules are equivalent to HA Groups and will replace them
-in an upcoming major release.
-
 By default, a HA resource is able to run on any cluster node, but a common
 requirement is that a HA resource should run on a specific node. That can be
 implemented by defining a HA node affinity rule to make the HA resource
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH docs 5/5] ha: replace in-text references to ha groups with ha rules

[Daniel Kral]

As HA groups are replaced by HA node affinity rules and user can
implement new CRS behavior with HA resource affinity rules now, update
texts that reference HA groups with references to HA rules instead.

While at it, also replace references to "HA services" with "HA
resources" for short sections that are touched in the process as new
references should use the latter term only.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 ha-manager.adoc | 49 +++++++++++++++++++++++++------------------------
 1 file changed, 25 insertions(+), 24 deletions(-)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index ffab83c..f63fd05 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -314,9 +314,8 @@ recovery state.
 recovery::
 
 Wait for recovery of the service. The HA manager tries to find a new node where
-the service can run on. This search depends not only on the list of online and
-quorate nodes, but also if the service is a group member and how such a group
-is limited.
+the service can run on. This search depends on the list of online and quorate
+nodes as well as the affinity rules the service is part of, if any.
 As soon as a new available node is found, the service will be moved there and
 initially placed into stopped state. If it's configured to run the new node
 will do so.
@@ -977,20 +976,24 @@ Recover Fenced Services
 ~~~~~~~~~~~~~~~~~~~~~~~
 
 After a node failed and its fencing was successful, the CRM tries to
-move services from the failed node to nodes which are still online.
+move HA resources from the failed node to nodes which are still online.
 
-The selection of nodes, on which those services gets recovered, is
-influenced by the resource `group` settings, the list of currently active
-nodes, and their respective active service count.
+The selection of the recovery nodes is influenced by the list of
+currently active nodes, their respective loads depending on the used
+scheduler, and the affinity rules the resource is part of, if any.
 
-The CRM first builds a set out of the intersection between user selected
-nodes (from `group` setting) and available nodes. It then choose the
-subset of nodes with the highest priority, and finally select the node
-with the lowest active service count. This minimizes the possibility
+First, the CRM builds a set of nodes available to the HA resource. If the
+resource is part of a node affinity rule, the set is reduced to the
+highest priority nodes in the node affinity rule. If the resource is part
+of a resource affinity rule, the set is further reduced to fufill their
+constraints, which is either keeping the HA resource on the same node as
+some other HA resources or keeping the HA resource on a different node
+than some other HA resources. Finally, the CRM selects the node with the
+lowest load according to the used scheduler to minimize the possibility
 of an overloaded node.
 
-CAUTION: On node failure, the CRM distributes services to the
-remaining nodes. This increases the service count on those nodes, and
+CAUTION: On node failure, the CRM distributes resources to the
+remaining nodes. This increases the resource count on those nodes, and
 can lead to high load, especially on small clusters. Please design
 your cluster so that it can handle such worst case scenarios.
 
@@ -1102,7 +1105,7 @@ You can use the manual maintenance mode to mark the node as unavailable for HA
 operation, prompting all services managed by HA to migrate to other nodes.
 
 The target nodes for these migrations are selected from the other currently
-available nodes, and determined by the HA group configuration and the configured
+available nodes, and determined by the HA rules configuration and the configured
 cluster resource scheduler (CRS) mode.
 During each migration, the original node will be recorded in the HA managers'
 state, so that the service can be moved back again automatically once the
@@ -1173,14 +1176,12 @@ This triggers a migration of all HA Services currently located on this node.
 The LRM will try to delay the shutdown process, until all running services get
 moved away. But, this expects that the running services *can* be migrated to
 another node. In other words, the service must not be locally bound, for example
-by using hardware passthrough. As non-group member nodes are considered as
-runnable target if no group member is available, this policy can still be used
-when making use of HA groups with only some nodes selected. But, marking a group
-as 'restricted' tells the HA manager that the service cannot run outside of the
-chosen set of nodes. If all of those nodes are unavailable, the shutdown will
-hang until you manually intervene. Once the shut down node comes back online
-again, the previously displaced services will be moved back, if they were not
-already manually migrated in-between.
+by using hardware passthrough. For example, strict node affinity rules tell the
+HA Manager that the service cannot run outside of the chosen set of nodes. If all
+of those nodes are unavailable, the shutdown will hang until you manually
+intervene. Once the shut down node comes back online again, the previously
+displaced services will be moved back, if they were not already manually migrated
+in-between.
 
 NOTE: The watchdog is still active during the migration process on shutdown.
 If the node loses quorum it will be fenced and the services will be recovered.
@@ -1266,8 +1267,8 @@ The change will be in effect starting with the next manager round (after a few
 seconds).
 
 For each service that needs to be recovered or migrated, the scheduler
-iteratively chooses the best node among the nodes with the highest priority in
-the service's group.
+iteratively chooses the best node among the nodes that are available to
+the service according to their HA rules, if any.
 
 NOTE: There are plans to add modes for (static and dynamic) load-balancing in
 the future.
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 2/5] ha: rules: update about mixed usage of node and resource affinity rules

[Hannes Duerr]

On Mon Aug 4, 2025 at 4:11 PM CEST, Daniel Kral wrote:
> As HA resources can be part of node and resource affinity rules at the
> same time in simpler cases, update the relevant sections about specific
> interactions and restrictions.
>
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
> ---
>  ha-manager.adoc | 29 +++++++++++++++++++++++++++--
>  1 file changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/ha-manager.adoc b/ha-manager.adoc
> index d6ac75d..5d75287 100644
> --- a/ha-manager.adoc
> +++ b/ha-manager.adoc
> @@ -837,6 +837,23 @@ Two or more HA resources cannot be kept on the same node and separated on
>  different nodes at the same time. For more information on these cases, see the
>  section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.
>  
> +Interactions between Node and Positive Resource Affinity Rules
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> +
> +If there are HA resources in a node affinity rule, which are also part of a
> +positive resource affinity rules, then all the other HA resources in the
s/rules/rule/
> +positive resource affinity rule inherit the node affinity rule as well.
> +
> +For example, if the HA resources `vm:100`, `vm:101`, and `vm:102` are in a
> +positive resource affinity rule, and `vm:102` is in a node affinity rule, which
> +restricts `vm:102` to be only on `node3`, then `vm:100` and `vm:101` are
> +restricted to be only on `node3` as well.
> +
> +Note that if there are two or more HA resources of a positive resource affinity
> +rules, which are in different node affinity rules, then those will be disabled
> +as it is currently not supported. For more information on these cases, see the
Please note that if there are two or more HA resources in a positive
resource affinity rule and in different node affinity rules, these
will be disabled, as this is not currently supported.
> +section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.

>  Resource Affinity Rule Properties
>  +++++++++++++++++++++++++++++++++
>  
> @@ -866,8 +883,16 @@ Currently, HA rules are checked for the following feasibility tests:
>    resources as a negative HA resources affinity rule. That is, two or more HA
>    resources cannot be kept together and separate at the same time.
>  
> -* A HA resource can only be part of either a HA node affinity rule or HA
> -  resource affinity rules, but not both at the same time.
> +* An HA resource can only be part of a HA node affinity rule and a HA resource
> +  affinity rule at the same time, if the HA node affinity rule has a single
> +  priority class.
> +
> +* The HA resources of a positive HA resource affinity rule can only be part of
> +  a single HA node affinity rule at most.
> +
> +* The HA resources of a negative HA resource affinity rule cannot be restricted
> +  to less nodes than HA resources by their node affinity rules. Otherwise, the
> +  HA resources do not have enough nodes to be separated.
>  
>  [[ha_manager_fencing]]
>  Fencing



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 5/5] ha: replace in-text references to ha groups with ha rules

[Hannes Duerr]

On Mon Aug 4, 2025 at 4:11 PM CEST, Daniel Kral wrote:
> As HA groups are replaced by HA node affinity rules and user can
> implement new CRS behavior with HA resource affinity rules now, update
> texts that reference HA groups with references to HA rules instead.
>
> While at it, also replace references to "HA services" with "HA
> resources" for short sections that are touched in the process as new
> references should use the latter term only.
>
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
> ---
>  ha-manager.adoc | 49 +++++++++++++++++++++++++------------------------
>  1 file changed, 25 insertions(+), 24 deletions(-)
>
> diff --git a/ha-manager.adoc b/ha-manager.adoc
> index ffab83c..f63fd05 100644
> --- a/ha-manager.adoc
> +++ b/ha-manager.adoc
> @@ -314,9 +314,8 @@ recovery state.
>  recovery::
>  
>  Wait for recovery of the service. The HA manager tries to find a new node where
forgot to change the `service` here?
> -the service can run on. This search depends not only on the list of online and
> -quorate nodes, but also if the service is a group member and how such a group
> -is limited.
> +the service can run on. This search depends on the list of online and quorate
s/service/resource/
> +nodes as well as the affinity rules the service is part of, if any.
s/service/resource/
>  As soon as a new available node is found, the service will be moved there and
forgot to change the `service` here?
>  initially placed into stopped state. If it's configured to run the new node
>  will do so.
> @@ -977,20 +976,24 @@ Recover Fenced Services
>  ~~~~~~~~~~~~~~~~~~~~~~~
>  
>  After a node failed and its fencing was successful, the CRM tries to
> -move services from the failed node to nodes which are still online.
> +move HA resources from the failed node to nodes which are still online.
>  
> -The selection of nodes, on which those services gets recovered, is
> -influenced by the resource `group` settings, the list of currently active
> -nodes, and their respective active service count.
> +The selection of the recovery nodes is influenced by the list of
> +currently active nodes, their respective loads depending on the used
> +scheduler, and the affinity rules the resource is part of, if any.
>  
> -The CRM first builds a set out of the intersection between user selected
> -nodes (from `group` setting) and available nodes. It then choose the
> -subset of nodes with the highest priority, and finally select the node
> -with the lowest active service count. This minimizes the possibility
> +First, the CRM builds a set of nodes available to the HA resource. If the
> +resource is part of a node affinity rule, the set is reduced to the
> +highest priority nodes in the node affinity rule. If the resource is part
> +of a resource affinity rule, the set is further reduced to fufill their
> +constraints, which is either keeping the HA resource on the same node as
> +some other HA resources or keeping the HA resource on a different node
> +than some other HA resources. Finally, the CRM selects the node with the
> +lowest load according to the used scheduler to minimize the possibility
>  of an overloaded node.
>  
> -CAUTION: On node failure, the CRM distributes services to the
> -remaining nodes. This increases the service count on those nodes, and
> +CAUTION: On node failure, the CRM distributes resources to the
> +remaining nodes. This increases the resource count on those nodes, and
>  can lead to high load, especially on small clusters. Please design
>  your cluster so that it can handle such worst case scenarios.
>  
> @@ -1102,7 +1105,7 @@ You can use the manual maintenance mode to mark the node as unavailable for HA
>  operation, prompting all services managed by HA to migrate to other nodes.
forgot to change the `service` here?
>  
>  The target nodes for these migrations are selected from the other currently
> -available nodes, and determined by the HA group configuration and the configured
> +available nodes, and determined by the HA rules configuration and the configured
>  cluster resource scheduler (CRS) mode.
>  During each migration, the original node will be recorded in the HA managers'
>  state, so that the service can be moved back again automatically once the
forgot to change the `service` here?
> @@ -1173,14 +1176,12 @@ This triggers a migration of all HA Services currently located on this node.
forgot to change the `service` here?
>  The LRM will try to delay the shutdown process, until all running services get
forgot to change the `service` here?
>  moved away. But, this expects that the running services *can* be migrated to
forgot to change the `service` here?
>  another node. In other words, the service must not be locally bound, for example
forgot to change the `service` here?
> -by using hardware passthrough. As non-group member nodes are considered as
> -runnable target if no group member is available, this policy can still be used
> -when making use of HA groups with only some nodes selected. But, marking a group
> -as 'restricted' tells the HA manager that the service cannot run outside of the
> -chosen set of nodes. If all of those nodes are unavailable, the shutdown will
> -hang until you manually intervene. Once the shut down node comes back online
> -again, the previously displaced services will be moved back, if they were not
> -already manually migrated in-between.
> +by using hardware passthrough. For example, strict node affinity rules tell the
s/For example, s/S/
> +HA Manager that the service cannot run outside of the chosen set of nodes. If all
> +of those nodes are unavailable, the shutdown will hang until you manually
s/those/these/
> +intervene. Once the shut down node comes back online again, the previously
> +displaced services will be moved back, if they were not already manually migrated
> +in-between.
>  
>  NOTE: The watchdog is still active during the migration process on shutdown.
>  If the node loses quorum it will be fenced and the services will be recovered.
> @@ -1266,8 +1267,8 @@ The change will be in effect starting with the next manager round (after a few
>  seconds).
>  
>  For each service that needs to be recovered or migrated, the scheduler
> -iteratively chooses the best node among the nodes with the highest priority in
> -the service's group.
> +iteratively chooses the best node among the nodes that are available to
> +the service according to their HA rules, if any.
Doesn't the scheduler take the ha node affinity priority into
consideration here?
And:
s/service/resource/
>  
>  NOTE: There are plans to add modes for (static and dynamic) load-balancing in
>  the future.



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 2/5] ha: rules: update about mixed usage of node and resource affinity rules

[Daniel Kral]

Hey Hannes,

thanks for the quick review!

On Mon Aug 4, 2025 at 5:09 PM CEST, Hannes Duerr wrote:
> On Mon Aug 4, 2025 at 4:11 PM CEST, Daniel Kral wrote:
>> +If there are HA resources in a node affinity rule, which are also part of a
>> +positive resource affinity rules, then all the other HA resources in the
> s/rules/rule/

ACK!

>> +positive resource affinity rule inherit the node affinity rule as well.
>> +
>> +For example, if the HA resources `vm:100`, `vm:101`, and `vm:102` are in a
>> +positive resource affinity rule, and `vm:102` is in a node affinity rule, which
>> +restricts `vm:102` to be only on `node3`, then `vm:100` and `vm:101` are
>> +restricted to be only on `node3` as well.
>> +
>> +Note that if there are two or more HA resources of a positive resource affinity
>> +rules, which are in different node affinity rules, then those will be disabled
>> +as it is currently not supported. For more information on these cases, see the
> Please note that if there are two or more HA resources in a positive
> resource affinity rule and in different node affinity rules, these
> will be disabled, as this is not currently supported.

Thanks, will change that with slightly more emphasis that the rules will
be disabled and not the HA resources.

>> +section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 5/5] ha: replace in-text references to ha groups with ha rules

[Daniel Kral]

Thanks for the quick review!

On Mon Aug 4, 2025 at 5:24 PM CEST, Hannes Duerr wrote:
> On Mon Aug 4, 2025 at 4:11 PM CEST, Daniel Kral wrote:
>> As HA groups are replaced by HA node affinity rules and user can
>> implement new CRS behavior with HA resource affinity rules now, update
>> texts that reference HA groups with references to HA rules instead.
>>
>> While at it, also replace references to "HA services" with "HA
>> resources" for short sections that are touched in the process as new
>> references should use the latter term only.
>>
>> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
>> ---
>>  ha-manager.adoc | 49 +++++++++++++++++++++++++------------------------
>>  1 file changed, 25 insertions(+), 24 deletions(-)
>>
>> diff --git a/ha-manager.adoc b/ha-manager.adoc
>> index ffab83c..f63fd05 100644
>> --- a/ha-manager.adoc
>> +++ b/ha-manager.adoc
>> @@ -314,9 +314,8 @@ recovery state.
>>  recovery::
>>  
>>  Wait for recovery of the service. The HA manager tries to find a new node where
> forgot to change the `service` here?

For all the following: I didn't change it in other sections because it
would have needed more changes than relevant to this patch. I only
changed it in the next section "Recover Fenced Services", because it was
its own section and changing it was trivial, even though I'll

>> -the service can run on. This search depends not only on the list of online and
>> -quorate nodes, but also if the service is a group member and how such a group
>> -is limited.
>> +the service can run on. This search depends on the list of online and quorate
> s/service/resource/
>> +nodes as well as the affinity rules the service is part of, if any.
> s/service/resource/
>>  As soon as a new available node is found, the service will be moved there and
> forgot to change the `service` here?
>>  initially placed into stopped state. If it's configured to run the new node
>>  will do so.
>> @@ -977,20 +976,24 @@ Recover Fenced Services
>>  ~~~~~~~~~~~~~~~~~~~~~~~
>>  
>>  After a node failed and its fencing was successful, the CRM tries to
>> -move services from the failed node to nodes which are still online.
>> +move HA resources from the failed node to nodes which are still online.
>>  
>> -The selection of nodes, on which those services gets recovered, is
>> -influenced by the resource `group` settings, the list of currently active
>> -nodes, and their respective active service count.
>> +The selection of the recovery nodes is influenced by the list of
>> +currently active nodes, their respective loads depending on the used
>> +scheduler, and the affinity rules the resource is part of, if any.
>>  
>> -The CRM first builds a set out of the intersection between user selected
>> -nodes (from `group` setting) and available nodes. It then choose the
>> -subset of nodes with the highest priority, and finally select the node
>> -with the lowest active service count. This minimizes the possibility
>> +First, the CRM builds a set of nodes available to the HA resource. If the
>> +resource is part of a node affinity rule, the set is reduced to the
>> +highest priority nodes in the node affinity rule. If the resource is part
>> +of a resource affinity rule, the set is further reduced to fufill their
>> +constraints, which is either keeping the HA resource on the same node as
>> +some other HA resources or keeping the HA resource on a different node
>> +than some other HA resources. Finally, the CRM selects the node with the
>> +lowest load according to the used scheduler to minimize the possibility
>>  of an overloaded node.
>>  
>> -CAUTION: On node failure, the CRM distributes services to the
>> -remaining nodes. This increases the service count on those nodes, and
>> +CAUTION: On node failure, the CRM distributes resources to the
>> +remaining nodes. This increases the resource count on those nodes, and
>>  can lead to high load, especially on small clusters. Please design
>>  your cluster so that it can handle such worst case scenarios.
>>  
>> @@ -1102,7 +1105,7 @@ You can use the manual maintenance mode to mark the node as unavailable for HA
>>  operation, prompting all services managed by HA to migrate to other nodes.
> forgot to change the `service` here?
>>  
>>  The target nodes for these migrations are selected from the other currently
>> -available nodes, and determined by the HA group configuration and the configured
>> +available nodes, and determined by the HA rules configuration and the configured
>>  cluster resource scheduler (CRS) mode.
>>  During each migration, the original node will be recorded in the HA managers'
>>  state, so that the service can be moved back again automatically once the
> forgot to change the `service` here?
>> @@ -1173,14 +1176,12 @@ This triggers a migration of all HA Services currently located on this node.
> forgot to change the `service` here?
>>  The LRM will try to delay the shutdown process, until all running services get
> forgot to change the `service` here?
>>  moved away. But, this expects that the running services *can* be migrated to
> forgot to change the `service` here?
>>  another node. In other words, the service must not be locally bound, for example
> forgot to change the `service` here?
>> -by using hardware passthrough. As non-group member nodes are considered as
>> -runnable target if no group member is available, this policy can still be used
>> -when making use of HA groups with only some nodes selected. But, marking a group
>> -as 'restricted' tells the HA manager that the service cannot run outside of the
>> -chosen set of nodes. If all of those nodes are unavailable, the shutdown will
>> -hang until you manually intervene. Once the shut down node comes back online
>> -again, the previously displaced services will be moved back, if they were not
>> -already manually migrated in-between.
>> +by using hardware passthrough. For example, strict node affinity rules tell the
> s/For example, s/S/

Hm, I'd rather let that "for example" stay as it's an example how rules
could restrict the HA Manager to be unable to find a node to migrate to,
not the definition or only way to do that.

>> +HA Manager that the service cannot run outside of the chosen set of nodes. If all
>> +of those nodes are unavailable, the shutdown will hang until you manually
> s/those/these/

ACK

>> +intervene. Once the shut down node comes back online again, the previously
>> +displaced services will be moved back, if they were not already manually migrated
>> +in-between.
>>  
>>  NOTE: The watchdog is still active during the migration process on shutdown.
>>  If the node loses quorum it will be fenced and the services will be recovered.
>> @@ -1266,8 +1267,8 @@ The change will be in effect starting with the next manager round (after a few
>>  seconds).
>>  
>>  For each service that needs to be recovered or migrated, the scheduler
>> -iteratively chooses the best node among the nodes with the highest priority in
>> -the service's group.
>> +iteratively chooses the best node among the nodes that are available to
>> +the service according to their HA rules, if any.
> Doesn't the scheduler take the ha node affinity priority into
> consideration here?

Yes it does, but I shortened it to "according to HA rules" as the
selection criteria was explained multiple times above already and I
thought it might be a maintenance burden if it needs to be checked and
changed at every change of the scheduler.

I think the explanation in the HA rules section and the CRS scheduling
points is enough to warrant less duplication here, but I'm happy to
change it if there are any objections.

> And:
> s/service/resource/
>>  
>>  NOTE: There are plans to add modes for (static and dynamic) load-balancing in
>>  the future.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 5/5] ha: replace in-text references to ha groups with ha rules

[Michael Köppl]

1 typo noted inline

On 8/4/25 16:12, Daniel Kral wrote:
> As HA groups are replaced by HA node affinity rules and user can
> implement new CRS behavior with HA resource affinity rules now, update
> texts that reference HA groups with references to HA rules instead.
> 
> While at it, also replace references to "HA services" with "HA
> resources" for short sections that are touched in the process as new
> references should use the latter term only.
> 
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
> ---
>  ha-manager.adoc | 49 +++++++++++++++++++++++++------------------------
>  1 file changed, 25 insertions(+), 24 deletions(-)
> 
> diff --git a/ha-manager.adoc b/ha-manager.adoc
> index ffab83c..f63fd05 100644
> --- a/ha-manager.adoc
> +++ b/ha-manager.adoc
> @@ -314,9 +314,8 @@ recovery state.
>  recovery::
>  
>  Wait for recovery of the service. The HA manager tries to find a new node where
> -the service can run on. This search depends not only on the list of online and
> -quorate nodes, but also if the service is a group member and how such a group
> -is limited.
> +the service can run on. This search depends on the list of online and quorate
> +nodes as well as the affinity rules the service is part of, if any.
>  As soon as a new available node is found, the service will be moved there and
>  initially placed into stopped state. If it's configured to run the new node
>  will do so.
> @@ -977,20 +976,24 @@ Recover Fenced Services
>  ~~~~~~~~~~~~~~~~~~~~~~~
>  
>  After a node failed and its fencing was successful, the CRM tries to
> -move services from the failed node to nodes which are still online.
> +move HA resources from the failed node to nodes which are still online.
>  
> -The selection of nodes, on which those services gets recovered, is
> -influenced by the resource `group` settings, the list of currently active
> -nodes, and their respective active service count.
> +The selection of the recovery nodes is influenced by the list of
> +currently active nodes, their respective loads depending on the used
> +scheduler, and the affinity rules the resource is part of, if any.
>  
> -The CRM first builds a set out of the intersection between user selected
> -nodes (from `group` setting) and available nodes. It then choose the
> -subset of nodes with the highest priority, and finally select the node
> -with the lowest active service count. This minimizes the possibility
> +First, the CRM builds a set of nodes available to the HA resource. If the
> +resource is part of a node affinity rule, the set is reduced to the
> +highest priority nodes in the node affinity rule. If the resource is part
> +of a resource affinity rule, the set is further reduced to fufill their

typo: s/fufill/fulfill

> +constraints, which is either keeping the HA resource on the same node as
> +some other HA resources or keeping the HA resource on a different node
> +than some other HA resources. Finally, the CRM selects the node with the
> +lowest load according to the used scheduler to minimize the possibility
>  of an overloaded node.
>  
> -CAUTION: On node failure, the CRM distributes services to the
> -remaining nodes. This increases the service count on those nodes, and
> +CAUTION: On node failure, the CRM distributes resources to the
> +remaining nodes. This increases the resource count on those nodes, and
>  can lead to high load, especially on small clusters. Please design
>  your cluster so that it can handle such worst case scenarios.
>  
> @@ -1102,7 +1105,7 @@ You can use the manual maintenance mode to mark the node as unavailable for HA
>  operation, prompting all services managed by HA to migrate to other nodes.
>  
>  The target nodes for these migrations are selected from the other currently
> -available nodes, and determined by the HA group configuration and the configured
> +available nodes, and determined by the HA rules configuration and the configured
>  cluster resource scheduler (CRS) mode.
>  During each migration, the original node will be recorded in the HA managers'
>  state, so that the service can be moved back again automatically once the
> @@ -1173,14 +1176,12 @@ This triggers a migration of all HA Services currently located on this node.
>  The LRM will try to delay the shutdown process, until all running services get
>  moved away. But, this expects that the running services *can* be migrated to
>  another node. In other words, the service must not be locally bound, for example
> -by using hardware passthrough. As non-group member nodes are considered as
> -runnable target if no group member is available, this policy can still be used
> -when making use of HA groups with only some nodes selected. But, marking a group
> -as 'restricted' tells the HA manager that the service cannot run outside of the
> -chosen set of nodes. If all of those nodes are unavailable, the shutdown will
> -hang until you manually intervene. Once the shut down node comes back online
> -again, the previously displaced services will be moved back, if they were not
> -already manually migrated in-between.
> +by using hardware passthrough. For example, strict node affinity rules tell the
> +HA Manager that the service cannot run outside of the chosen set of nodes. If all
> +of those nodes are unavailable, the shutdown will hang until you manually
> +intervene. Once the shut down node comes back online again, the previously
> +displaced services will be moved back, if they were not already manually migrated
> +in-between.
>  
>  NOTE: The watchdog is still active during the migration process on shutdown.
>  If the node loses quorum it will be fenced and the services will be recovered.
> @@ -1266,8 +1267,8 @@ The change will be in effect starting with the next manager round (after a few
>  seconds).
>  
>  For each service that needs to be recovered or migrated, the scheduler
> -iteratively chooses the best node among the nodes with the highest priority in
> -the service's group.
> +iteratively chooses the best node among the nodes that are available to
> +the service according to their HA rules, if any.
>  
>  NOTE: There are plans to add modes for (static and dynamic) load-balancing in
>  the future.



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 3/5] ha: rules: document crs behavior for split positive resource affinity

[Michael Köppl]

On 8/4/25 16:12, Daniel Kral wrote:
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
> ---
>  ha-manager.adoc | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/ha-manager.adoc b/ha-manager.adoc
> index 5d75287..e18a14d 100644
> --- a/ha-manager.adoc
> +++ b/ha-manager.adoc
> @@ -787,6 +787,11 @@ HA resources `vm:101` and `vm:102` are each in a positive resource affinity
>  rule, then it is the same as if `vm:100`, `vm:101` and `vm:102` would have been
>  in a single positive resource affinity rule.
>  
> +NOTE: If the HA resources of a positive resource affinity rule are currently
> +running on different nodes, the CRS will move the HA resources to the node,
> +where most of them are running already. If there is a tie in the HA resource
> +count, the node with the alphabetically first name is chosen.

maybe something like:

"the node whose name appears first alphabetically is selected"

just a suggestion, though

> +
>  However, suppose there are computationally expensive, and/or distributed
>  programs running on the HA resources `vm:200` and `ct:300`, for example,
>  sharded database instances. In that case, running them on the same node could



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 1/5] ha: affinity rules: simplify overly verbose rule conflicts and errors section

[Michael Köppl]

Apart from a typo I noted on 5/5 and the comments already made by
Hannes, I did not notice any errors and the changes improve the
comprehensibility of the docs IMO, so consider these changes:

Reviewed-by: Michael Köppl <m.koeppl@proxmox.com>

On 8/4/25 16:12, Daniel Kral wrote:
> Suggested-by: Hannes Dürr <h.duerr@proxmox.com>
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
> ---
> I only included some of the suggestions for the rule conflicts & errors
> part from Hannes for now as these are more important to be correct and
> understandable for the release.
> 
>  ha-manager.adoc | 29 ++++++++++-------------------
>  1 file changed, 10 insertions(+), 19 deletions(-)
> 
> diff --git a/ha-manager.adoc b/ha-manager.adoc
> index 77d7a65..d6ac75d 100644
> --- a/ha-manager.adoc
> +++ b/ha-manager.adoc
> @@ -849,34 +849,25 @@ Rule Conflicts and Errors
>  HA rules can impose rather complex constraints on the HA resources. To ensure
>  that a new or modified HA rule does not introduce uncertainty into the HA
>  stack's CRS scheduler, HA rules are tested for feasibility before these are
> -applied. If a rule does fail any of these tests, the rule is disabled until the
> -conflicts and errors is resolved.
> +applied. For any rules that fail these tests, these rules are disabled until
> +the conflicts and errors have been resolved.
>  
>  Currently, HA rules are checked for the following feasibility tests:
>  
> -* A HA resource can only be referenced by a single HA node affinity rule in
> -  total. If two or more HA node affinity rules specify the same HA resource,
> -  these HA node affinity rules will be disabled.
> +* An HA resource can only be part of a single HA node affinity rule.
>  
> -* A HA resource affinity rule must specify at least two HA resources to be
> -  feasible. If a HA resource affinity rule specifies only one HA resource, the
> -  HA resource affinity rule will be disabled.
> +* An HA resource affinity rule must have at least two HA resources.
>  
> -* A HA resource affinity rule must specify no more HA resources than there are
> -  nodes in the cluster. If a HA resource affinity rule specifies more HA
> -  resources than there are in the cluster, the HA resource affinity rule will
> -  be disabled.
> +* A negative HA resource affinity rule cannot specify more HA resources than
> +  there are nodes in the cluster. Otherwise, the HA resources do not have
> +  enough nodes to be separated.
>  
>  * A positive HA resource affinity rule cannot specify the same two or more HA
>    resources as a negative HA resources affinity rule. That is, two or more HA
> -  resources cannot be kept together and separate at the same time. If any pair
> -  of positive and negative HA resource affinity rules do specify the same two
> -  or more HA resources, both HA resource affinity rules will be disabled.
> +  resources cannot be kept together and separate at the same time.
>  
> -* A HA resource can only be referenced by either a HA node affinity rule or HA
> -  resource affinity rules, but not both at the same time. If there is at least
> -  one HA node affinity rule and at least one HA resource affinity rule, which
> -  specify the same HA resource, these rules will be disabled.
> +* A HA resource can only be part of either a HA node affinity rule or HA
> +  resource affinity rules, but not both at the same time.
>  
>  [[ha_manager_fencing]]
>  Fencing



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH docs 3/5] ha: rules: document crs behavior for split positive resource affinity

[Daniel Kral]

Hey Michael!

Thanks for taking a look!

On Tue Aug 5, 2025 at 9:42 AM CEST, Michael Köppl wrote:
> On 8/4/25 16:12, Daniel Kral wrote:
>> Signed-off-by: Daniel Kral <d.kral@proxmox.com>
>> ---
>>  ha-manager.adoc | 5 +++++
>>  1 file changed, 5 insertions(+)
>> 
>> diff --git a/ha-manager.adoc b/ha-manager.adoc
>> index 5d75287..e18a14d 100644
>> --- a/ha-manager.adoc
>> +++ b/ha-manager.adoc
>> @@ -787,6 +787,11 @@ HA resources `vm:101` and `vm:102` are each in a positive resource affinity
>>  rule, then it is the same as if `vm:100`, `vm:101` and `vm:102` would have been
>>  in a single positive resource affinity rule.
>>  
>> +NOTE: If the HA resources of a positive resource affinity rule are currently
>> +running on different nodes, the CRS will move the HA resources to the node,
>> +where most of them are running already. If there is a tie in the HA resource
>> +count, the node with the alphabetically first name is chosen.
>
> maybe something like:
>
> "the node whose name appears first alphabetically is selected"

ACK, I'll use "first in alphabetical order" though as it seems ambiguous
to me whether first comes before or after alphabetically

>
> just a suggestion, though
>
>> +
>>  However, suppose there are computationally expensive, and/or distributed
>>  programs running on the HA resources `vm:200` and `ct:300`, for example,
>>  sharded database instances. In that case, running them on the same node could



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] applied: [PATCH docs 1/5] ha: affinity rules: simplify overly verbose rule conflicts and errors section

[Daniel Kral]

On Mon Aug 4, 2025 at 4:11 PM CEST, Daniel Kral wrote:
> Suggested-by: Hannes Dürr <h.duerr@proxmox.com>
> Signed-off-by: Daniel Kral <d.kral@proxmox.com>

This has already been applied [0]:

c61403f ha: affinity rules: simplify overly verbose rule conflicts and errors section

https://git.proxmox.com/?p=pve-docs.git;a=commit;h=c61403fb668f586612ac6287df2613395350332d


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel