From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id BE30B1FF17C for ; Wed, 9 Jul 2025 08:18:55 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6CC543424B; Wed, 9 Jul 2025 08:19:38 +0200 (CEST) Message-ID: <97360323-0fe0-4f72-addd-89d4fe1cdee6@proxmox.com> Date: Wed, 9 Jul 2025 08:19:02 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Proxmox VE development discussion , Shannon Sterz , Daniel Kral References: <20250704182102.467624-1-d.kral@proxmox.com> <20250704182102.467624-15-d.kral@proxmox.com> Content-Language: en-US From: Friedrich Weber In-Reply-To: X-SPAM-LEVEL: Spam detection results: 0 AWL -0.391 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [gen-pct-network-opts.pl, gen-ha-rules-opts.pl, 5-opts.pl, proxmox.com, gen-ha-resources-opts.pl, gen-ha-rules-node-affinity-opts.pl, gen-ha-rules-resource-affinity-opts.pl] Subject: Re: [pve-devel] [PATCH docs v3 1/1] ha: add documentation about ha resource affinity rules X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" On 08/07/2025 18:08, Shannon Sterz wrote: > On Fri Jul 4, 2025 at 8:20 PM CEST, Daniel Kral wrote: >> Add documentation about HA Resource Affinity rules, what effects those >> have on the CRS scheduler, and what users can expect when those are >> changed. >> >> There are also a few points on the rule conflicts/errors list which >> describe some conflicts that can arise from a mixed usage of HA Node >> Affinity rules and HA Resource Affinity rules. >> >> Signed-off-by: Daniel Kral >> --- >> Makefile | 1 + >> gen-ha-rules-resource-affinity-opts.pl | 20 ++++ >> ha-manager.adoc | 133 +++++++++++++++++++++++++ >> ha-rules-resource-affinity-opts.adoc | 8 ++ >> 4 files changed, 162 insertions(+) >> create mode 100755 gen-ha-rules-resource-affinity-opts.pl >> create mode 100644 ha-rules-resource-affinity-opts.adoc >> >> diff --git a/Makefile b/Makefile >> index c5e506e..4d9e2f0 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -51,6 +51,7 @@ GEN_SCRIPTS= \ >> gen-ha-resources-opts.pl \ >> gen-ha-rules-node-affinity-opts.pl \ >> gen-ha-rules-opts.pl \ >> + gen-ha-rules-resource-affinity-opts.pl \ >> gen-datacenter.cfg.5-opts.pl \ >> gen-pct.conf.5-opts.pl \ >> gen-pct-network-opts.pl \ >> diff --git a/gen-ha-rules-resource-affinity-opts.pl b/gen-ha-rules-resource-affinity-opts.pl >> new file mode 100755 >> index 0000000..5abed50 >> --- /dev/null >> +++ b/gen-ha-rules-resource-affinity-opts.pl >> @@ -0,0 +1,20 @@ >> +#!/usr/bin/perl >> + >> +use lib '.'; >> +use strict; >> +use warnings; >> +use PVE::RESTHandler; >> + >> +use Data::Dumper; >> + >> +use PVE::HA::Rules; >> +use PVE::HA::Rules::ResourceAffinity; >> + >> +my $private = PVE::HA::Rules::private(); >> +my $resource_affinity_rule_props = PVE::HA::Rules::ResourceAffinity::properties(); >> +my $properties = { >> + resources => $private->{propertyList}->{resources}, >> + $resource_affinity_rule_props->%*, >> +}; >> + >> +print PVE::RESTHandler::dump_properties($properties); >> diff --git a/ha-manager.adoc b/ha-manager.adoc >> index ec26c22..8d06885 100644 >> --- a/ha-manager.adoc >> +++ b/ha-manager.adoc >> @@ -692,6 +692,10 @@ include::ha-rules-opts.adoc[] >> | HA Rule Type | Description >> | `node-affinity` | Places affinity from one or more HA resources to one or >> more nodes. >> +| `resource-affinity` | Places affinity between two or more HA resources. The >> +affinity `separate` specifies that HA resources are to be kept on separate >> +nodes, while the affinity `together` specifies that HA resources are to be kept >> +on the same node. > > here it's calleged "together" (or "separate")... > >> |=========================================================== >> >> [[ha_manager_node_affinity_rules]] >> @@ -758,6 +762,88 @@ Node Affinity Rule Properties >> >> include::ha-rules-node-affinity-opts.adoc[] >> >> +[[ha_manager_resource_affinity_rules]] >> +Resource Affinity Rules >> +^^^^^^^^^^^^^^^^^^^^^^^ >> + >> +Another common requirement is that two or more HA resources should run on >> +either the same node, or should be distributed on separate nodes. These are >> +also commonly called "Affinity/Anti-Affinity constraints". >> + >> +For example, suppose there is a lot of communication traffic between the HA >> +resources `vm:100` and `vm:200`, e.g., a web server communicating with a > > nit: just a small heads up, we recommend avoid "e.g." as it often gets > confused with "i.e." [1]. you could use `for example` instead to make > this a bit clearer (same below) > > [1]: https://pve.proxmox.com/wiki/Technical_Writing_Style_Guide#Abbreviations > >> +database server. If those HA resources are on separate nodes, this could >> +potentially result in a higher latency and unnecessary network load. Resource >> +affinity rules with the affinity `positive` implement the constraint to keep >> +the HA resources on the same node: >> + >> +---- >> +# ha-manager rules add resource-affinity keep-together \ >> + --affinity positive --resources vm:100,vm:200 >> +---- > > ... here it is specified as "positive"? did i miss something or is that > incorrect? Good catch, but I think it should be "positive"/"negative", so it's "together" and "separate" that are outdated. A lot of the naming was changed between v2 and v3, including "together"->"positive", and "separate"->"negative" [1] so they're probably leftovers from before the rename. [1] https://lore.proxmox.com/pve-devel/7fb94369-d8b6-47c6-b36c-428db5bb85de@proxmox.com/ > >> + >> +NOTE: If there are two or more positive resource affinity rules, which have >> +common HA resources, then these are treated as a single positive resource >> +affinity rule. For example, if the HA resources `vm:100` and `vm:101` and the >> +HA resources `vm:101` and `vm:102` are each in a positive resource affinity >> +rule, then it is the same as if `vm:100`, `vm:101` and `vm:102` would have been >> +in a single positive resource affinity rule. >> + >> +However, suppose there are computationally expensive, and/or distributed >> +programs running on the HA resources `vm:200` and `ct:300`, e.g., sharded >> +database instances. In that case, running them on the same node could >> +potentially result in pressure on the hardware resources of the node and will >> +slow down the operations of these HA resources. Resource affinity rules with >> +the affinity `negative` implement the constraint to spread the HA resources on >> +separate nodes: >> + >> +---- >> +# ha-manager rules add resource-affinity keep-separate \ >> + --affinity negative --resources vm:200,ct:300 >> +---- > > ... same here with "separate" or "negative" > >> + >> +Other than node affinity rules, resource affinity rules are strict by default, >> +i.e., if the constraints imposed by the resource affinity rules cannot be met >> +for a HA resource, the HA Manager will put the HA resource in recovery state in >> +case of a failover or in error state elsewhere. >> + >> +The above commands created the following rules in the rules configuration file: >> + >> +.Resource Affinity Rules Configuration Example (`/etc/pve/ha/rules.cfg`) >> +---- >> +resource-affinity: keep-together >> + resources vm:100,vm:200 >> + affinity positive >> + >> +resource-affinity: keep-separate >> + resources vm:200,ct:300 >> + affinity negative >> +---- >> + >> +Interactions between Positive and Negative Resource Affinity Rules >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> + >> +If there are HA resources in a positive resource affinity rule, which are also >> +part of a negative resource affinity rule, then all the other HA resources in >> +the positive resource affinity rule are in negative affinity with the HA >> +resources of these negative resource affinity rules as well. >> + >> +For example, if the HA resources `vm:100`, `vm:101`, and `vm:102` are in a >> +positive resource affinity rule, and `vm:100` is in a negative resource affinity >> +rule with the HA resource `ct:200`, then `vm:101` and `vm:102` are each in >> +negative resource affinity with `ct:200` as well. >> + >> +Note that if there are two or more HA resources in both a positive and negative >> +resource affinity rule, then those will be disabled as they cause a conflict: >> +Two or more HA resources cannot be kept on the same node and separated on >> +different nodes at the same time. For more information on these cases, see the >> +section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below. >> + >> +Resource Affinity Rule Properties >> ++++++++++++++++++++++++++++++++++ >> + >> +include::ha-rules-resource-affinity-opts.adoc[] >> + >> [[ha_manager_rule_conflicts]] >> Rule Conflicts and Errors >> ~~~~~~~~~~~~~~~~~~~~~~~~~ >> @@ -774,6 +860,43 @@ Currently, HA rules are checked for the following feasibility tests: >> total. If two or more HA node affinity rules specify the same HA resource, >> these HA node affinity rules will be disabled. >> >> +* A HA resource affinity rule must specify at least two HA resources to be >> + feasible. If a HA resource affinity rule does specify only one HA resource, > > nit: get rid of the "does" it makes this already very long and hard to > parse sentence ven hader to read. > >> + the HA resource affinity rule will be disabled. >> + >> +* A HA resource affinity rule must specify no more HA resources than there are >> + nodes in the cluster. If a HA resource affinity rule does specify more HA > > same here > >> + resources than there are in the cluster, the HA resource affinity rule will be >> + disabled. >> + >> +* A positive HA resource affinity rule cannot specify the same two or more HA >> + resources as a negative HA resources affinity rule. That is, two or more HA >> + resources cannot be kept together and separate at the same time. If any pair >> + of positive and negative HA resource affinity rules do specify the same two or >> + more HA resources, both HA resource affinity rules will be disabled. >> + >> +* A HA resource, which is already constrained by a HA node affinity rule, can >> + only be referenced by a HA resource affinity rule, if the HA node affinity >> + rule does only use a single priority group. That is, the specified nodes in > > and here > >> + the HA node affinity rule have the same priority. If one of the HA resources >> + in a HA resource affinity rule is constrainted by a HA node affinity rule with > > typo: constrainted -> constrained > >> + multiple priority groups, the HA resource affinity rule will be disabled. >> + >> +* The HA resources of a positive HA resource affinity rule, which are >> + constrained by HA node affinity rules, must have at least one common node, >> + where the HA resources are allowed to run on. Otherwise, the HA resources >> + could only run on separate nodes. In other words, if two or more HA resources >> + of a positive HA resource affinity rule are constrained to different nodes, >> + the positive HA resource affinity rule will be disabled. >> + >> +* The HA resources of a negative HA resource affinity rule, which are >> + constrained by HA node affinity rules, must have at least enough nodes to >> + separate these constrained HA resources on. Otherwise, the HA resources do not > > nit: the "on" here is not necessary. > >> + have enough nodes to be separated on. In other words, if two or more HA > > same here. > >> + resources of a negative HA resource affinity rule are constrained to less >> + nodes than needed to separate them on, the negative HA resource affinity rule > > and here > >> + will be disabled. >> + >> [[ha_manager_fencing]] >> Fencing >> ------- >> @@ -1205,6 +1328,16 @@ The CRS is currently used at the following scheduling points: >> algorithm to ensure that these HA resources are assigned according to their >> node and priority constraints. >> >> +** Positive resource affinity rules: If a positive resource affinity rule is >> + created or HA resources are added to an existing positive resource affinity >> + rule, the HA stack will use the CRS algorithm to ensure that these HA >> + resources are moved to a common node. >> + >> +** Negative resource affinity rules: If a negative resource affinity rule is >> + created or HA resources are added to an existing negative resource affinity >> + rule, the HA stack will use the CRS algorithm to ensure that these HA >> + resources are moved to separate nodes. >> + >> - HA service stopped -> start transition (opt-in). Requesting that a stopped >> service should be started is an good opportunity to check for the best suited >> node as per the CRS algorithm, as moving stopped services is cheaper to do >> diff --git a/ha-rules-resource-affinity-opts.adoc b/ha-rules-resource-affinity-opts.adoc >> new file mode 100644 >> index 0000000..596ec3c >> --- /dev/null >> +++ b/ha-rules-resource-affinity-opts.adoc >> @@ -0,0 +1,8 @@ >> +`affinity`: `` :: >> + >> +Describes whether the HA resources are supposed to be kept on the same node ('positive'), or are supposed to be kept on separate nodes ('negative'). >> + >> +`resources`: `:{,:}*` :: >> + >> +List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101). >> + > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel