From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 6E10C1FF183 for ; Wed, 30 Jul 2025 20:01:37 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 80533173E3; Wed, 30 Jul 2025 20:00:46 +0200 (CEST) From: Daniel Kral To: pve-devel@lists.proxmox.com Date: Wed, 30 Jul 2025 19:59:27 +0200 Message-ID: <20250730175957.386674-1-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.2 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1753898389058 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.014 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH docs/ha-manager/manager v5 00/29] HA Rules X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Here's a (hopefully) final update on the core HA rules series. @Michael and @Hannes Duerr have been a great help for testing this series, thank you very much! For maintainers: ha-manager patch #19 should be updated to the correct pve-manager version that is dependent on the pve-ha-manager package which can interpret the HA rules config. Changelog since v4 ------------------ - rebased on newest available master - postponed persistent rules migration only if all ha groups migration checks passed - also check the lrm status of all nodes as additional ha groups migration checks - do not allow modifying ha rules api calls unless the ha groups have been fully migrated - do not allow reading/modifying ha group api calls after the ha groups have been fully migrated (after thought: we could be more loose on the reading api endpoints?) - do not allow 'group' parameter for ha resources api calls after the ha groups have been fully migrated - the migrated ha groups' comment is not overwritten anymore - fixed the ha-manager CLI for 'ha-manager rules config' to display an overview of the ha rules on the CLI too ha-manager: Daniel Kral (23): tree-wide: make arguments for select_service_node explicit manager: improve signature of select_service_node introduce rules base plugin rules: introduce node affinity rule plugin config, env, hw: add rules read and parse methods config: delete services from rules if services are deleted from config manager: read and update rules config test: ha tester: add test cases for future node affinity rules resources: introduce failback property in ha resource config manager: migrate ha groups to node affinity rules in-memory manager: apply node affinity rules when selecting service nodes test: add test cases for rules config api: introduce ha rules api endpoints cli: expose ha rules api endpoints to ha-manager cli sim: do not create default groups config test: ha tester: migrate groups to service and rules config test: ha tester: replace any reference to groups with node affinity rules env: add property delete for update_service_config manager: persistently migrate ha groups to ha rules api: groups: disallow calls to ha groups endpoints if fully migrated api: resources: exclude group property in reading endpoints if migrated api: resources: disallow group prop in modifying endpoints if migrated api: rules: disallow modifying api calls if ha groups not migrated .gitignore | 1 + debian/pve-ha-manager.install | 3 + src/PVE/API2/HA/Groups.pm | 25 +- src/PVE/API2/HA/Makefile | 2 +- src/PVE/API2/HA/Resources.pm | 24 +- src/PVE/API2/HA/Rules.pm | 394 +++++++++++++++ src/PVE/API2/HA/Status.pm | 11 +- src/PVE/CLI/ha_manager.pm | 36 ++ src/PVE/HA/Config.pm | 67 ++- src/PVE/HA/Env.pm | 34 +- src/PVE/HA/Env/PVE2.pm | 45 +- src/PVE/HA/Groups.pm | 49 ++ src/PVE/HA/Makefile | 3 +- src/PVE/HA/Manager.pm | 321 ++++++++---- src/PVE/HA/Resources.pm | 9 + src/PVE/HA/Resources/PVECT.pm | 1 + src/PVE/HA/Resources/PVEVM.pm | 1 + src/PVE/HA/Rules.pm | 455 ++++++++++++++++++ src/PVE/HA/Rules/Makefile | 6 + src/PVE/HA/Rules/NodeAffinity.pm | 296 ++++++++++++ src/PVE/HA/Sim/Env.pm | 48 +- src/PVE/HA/Sim/Hardware.pm | 70 ++- src/PVE/HA/Tools.pm | 46 ++ src/test/Makefile | 4 +- .../defaults-for-node-affinity-rules.cfg | 22 + ...efaults-for-node-affinity-rules.cfg.expect | 60 +++ ...e-resource-refs-in-node-affinity-rules.cfg | 31 ++ ...rce-refs-in-node-affinity-rules.cfg.expect | 63 +++ src/test/test-basic5/groups | 2 - src/test/test-basic5/rules_config | 3 + src/test/test-basic5/service_config | 2 +- src/test/test-crs-static2/README | 3 +- src/test/test-crs-static2/groups | 2 - src/test/test-crs-static2/rules_config | 3 + src/test/test-crs-static2/service_config | 2 +- src/test/test-group-migrate1/README | 4 + src/test/test-group-migrate1/cmdlist | 3 + src/test/test-group-migrate1/groups | 7 + src/test/test-group-migrate1/hardware_status | 5 + src/test/test-group-migrate1/log.expect | 101 ++++ src/test/test-group-migrate1/manager_status | 1 + src/test/test-group-migrate1/service_config | 5 + src/test/test-group-migrate2/README | 3 + src/test/test-group-migrate2/cmdlist | 3 + src/test/test-group-migrate2/groups | 7 + src/test/test-group-migrate2/hardware_status | 5 + src/test/test-group-migrate2/log.expect | 50 ++ src/test/test-group-migrate2/manager_status | 1 + src/test/test-group-migrate2/service_config | 5 + src/test/test-node-affinity-nonstrict1/README | 11 + .../test-node-affinity-nonstrict1/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict1/log.expect | 40 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict2/README | 12 + .../test-node-affinity-nonstrict2/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict2/log.expect | 35 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict3/README | 10 + .../test-node-affinity-nonstrict3/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict3/log.expect | 56 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 5 + src/test/test-node-affinity-nonstrict4/README | 14 + .../test-node-affinity-nonstrict4/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict4/log.expect | 54 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 5 + src/test/test-node-affinity-nonstrict5/README | 16 + .../test-node-affinity-nonstrict5/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-nonstrict5/log.expect | 66 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict6/README | 14 + .../test-node-affinity-nonstrict6/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-nonstrict6/log.expect | 52 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-strict1/README | 11 + src/test/test-node-affinity-strict1/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict1/log.expect | 40 ++ .../test-node-affinity-strict1/manager_status | 1 + .../test-node-affinity-strict1/rules_config | 4 + .../test-node-affinity-strict1/service_config | 3 + src/test/test-node-affinity-strict2/README | 11 + src/test/test-node-affinity-strict2/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict2/log.expect | 40 ++ .../test-node-affinity-strict2/manager_status | 1 + .../test-node-affinity-strict2/rules_config | 4 + .../test-node-affinity-strict2/service_config | 3 + src/test/test-node-affinity-strict3/README | 10 + src/test/test-node-affinity-strict3/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict3/log.expect | 74 +++ .../test-node-affinity-strict3/manager_status | 1 + .../test-node-affinity-strict3/rules_config | 4 + .../test-node-affinity-strict3/service_config | 5 + src/test/test-node-affinity-strict4/README | 14 + src/test/test-node-affinity-strict4/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict4/log.expect | 54 +++ .../test-node-affinity-strict4/manager_status | 1 + .../test-node-affinity-strict4/rules_config | 4 + .../test-node-affinity-strict4/service_config | 5 + src/test/test-node-affinity-strict5/README | 16 + src/test/test-node-affinity-strict5/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-strict5/log.expect | 66 +++ .../test-node-affinity-strict5/manager_status | 1 + .../test-node-affinity-strict5/rules_config | 4 + .../test-node-affinity-strict5/service_config | 3 + src/test/test-node-affinity-strict6/README | 15 + src/test/test-node-affinity-strict6/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-strict6/log.expect | 52 ++ .../test-node-affinity-strict6/manager_status | 1 + .../test-node-affinity-strict6/rules_config | 4 + .../test-node-affinity-strict6/service_config | 3 + src/test/test-recovery1/groups | 4 - src/test/test-recovery1/rules_config | 4 + src/test/test-recovery1/service_config | 2 +- src/test/test-recovery2/README | 4 +- src/test/test-recovery2/groups | 4 - src/test/test-recovery2/rules_config | 4 + src/test/test-recovery2/service_config | 2 +- src/test/test-recovery3/groups | 4 - src/test/test-recovery3/rules_config | 4 + src/test/test-recovery3/service_config | 2 +- src/test/test-recovery4/groups | 4 - src/test/test-recovery4/rules_config | 4 + src/test/test-recovery4/service_config | 2 +- .../test-relocate-policy-default-group/README | 6 +- src/test/test-resource-failure2/groups | 2 - src/test/test-resource-failure2/rules_config | 3 + .../test-resource-failure2/service_config | 2 +- .../test-resource-failure3/service_config | 2 +- src/test/test-resource-failure6/README | 6 +- src/test/test-shutdown2/groups | 2 - src/test/test-shutdown2/rules_config | 3 + src/test/test-shutdown2/service_config | 4 +- src/test/test-shutdown3/groups | 2 - src/test/test-shutdown3/rules_config | 3 + src/test/test-shutdown3/service_config | 4 +- src/test/test_failover1.pl | 27 +- src/test/test_rules_config.pl | 100 ++++ 160 files changed, 3374 insertions(+), 173 deletions(-) create mode 100644 src/PVE/API2/HA/Rules.pm create mode 100644 src/PVE/HA/Rules.pm create mode 100644 src/PVE/HA/Rules/Makefile create mode 100644 src/PVE/HA/Rules/NodeAffinity.pm create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg.expect create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg.expect delete mode 100644 src/test/test-basic5/groups create mode 100644 src/test/test-basic5/rules_config delete mode 100644 src/test/test-crs-static2/groups create mode 100644 src/test/test-crs-static2/rules_config create mode 100644 src/test/test-group-migrate1/README create mode 100644 src/test/test-group-migrate1/cmdlist create mode 100644 src/test/test-group-migrate1/groups create mode 100644 src/test/test-group-migrate1/hardware_status create mode 100644 src/test/test-group-migrate1/log.expect create mode 100644 src/test/test-group-migrate1/manager_status create mode 100644 src/test/test-group-migrate1/service_config create mode 100644 src/test/test-group-migrate2/README create mode 100644 src/test/test-group-migrate2/cmdlist create mode 100644 src/test/test-group-migrate2/groups create mode 100644 src/test/test-group-migrate2/hardware_status create mode 100644 src/test/test-group-migrate2/log.expect create mode 100644 src/test/test-group-migrate2/manager_status create mode 100644 src/test/test-group-migrate2/service_config create mode 100644 src/test/test-node-affinity-nonstrict1/README create mode 100644 src/test/test-node-affinity-nonstrict1/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict1/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict1/log.expect create mode 100644 src/test/test-node-affinity-nonstrict1/manager_status create mode 100644 src/test/test-node-affinity-nonstrict1/rules_config create mode 100644 src/test/test-node-affinity-nonstrict1/service_config create mode 100644 src/test/test-node-affinity-nonstrict2/README create mode 100644 src/test/test-node-affinity-nonstrict2/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict2/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict2/log.expect create mode 100644 src/test/test-node-affinity-nonstrict2/manager_status create mode 100644 src/test/test-node-affinity-nonstrict2/rules_config create mode 100644 src/test/test-node-affinity-nonstrict2/service_config create mode 100644 src/test/test-node-affinity-nonstrict3/README create mode 100644 src/test/test-node-affinity-nonstrict3/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict3/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict3/log.expect create mode 100644 src/test/test-node-affinity-nonstrict3/manager_status create mode 100644 src/test/test-node-affinity-nonstrict3/rules_config create mode 100644 src/test/test-node-affinity-nonstrict3/service_config create mode 100644 src/test/test-node-affinity-nonstrict4/README create mode 100644 src/test/test-node-affinity-nonstrict4/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict4/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict4/log.expect create mode 100644 src/test/test-node-affinity-nonstrict4/manager_status create mode 100644 src/test/test-node-affinity-nonstrict4/rules_config create mode 100644 src/test/test-node-affinity-nonstrict4/service_config create mode 100644 src/test/test-node-affinity-nonstrict5/README create mode 100644 src/test/test-node-affinity-nonstrict5/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict5/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict5/log.expect create mode 100644 src/test/test-node-affinity-nonstrict5/manager_status create mode 100644 src/test/test-node-affinity-nonstrict5/rules_config create mode 100644 src/test/test-node-affinity-nonstrict5/service_config create mode 100644 src/test/test-node-affinity-nonstrict6/README create mode 100644 src/test/test-node-affinity-nonstrict6/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict6/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict6/log.expect create mode 100644 src/test/test-node-affinity-nonstrict6/manager_status create mode 100644 src/test/test-node-affinity-nonstrict6/rules_config create mode 100644 src/test/test-node-affinity-nonstrict6/service_config create mode 100644 src/test/test-node-affinity-strict1/README create mode 100644 src/test/test-node-affinity-strict1/cmdlist create mode 100644 src/test/test-node-affinity-strict1/hardware_status create mode 100644 src/test/test-node-affinity-strict1/log.expect create mode 100644 src/test/test-node-affinity-strict1/manager_status create mode 100644 src/test/test-node-affinity-strict1/rules_config create mode 100644 src/test/test-node-affinity-strict1/service_config create mode 100644 src/test/test-node-affinity-strict2/README create mode 100644 src/test/test-node-affinity-strict2/cmdlist create mode 100644 src/test/test-node-affinity-strict2/hardware_status create mode 100644 src/test/test-node-affinity-strict2/log.expect create mode 100644 src/test/test-node-affinity-strict2/manager_status create mode 100644 src/test/test-node-affinity-strict2/rules_config create mode 100644 src/test/test-node-affinity-strict2/service_config create mode 100644 src/test/test-node-affinity-strict3/README create mode 100644 src/test/test-node-affinity-strict3/cmdlist create mode 100644 src/test/test-node-affinity-strict3/hardware_status create mode 100644 src/test/test-node-affinity-strict3/log.expect create mode 100644 src/test/test-node-affinity-strict3/manager_status create mode 100644 src/test/test-node-affinity-strict3/rules_config create mode 100644 src/test/test-node-affinity-strict3/service_config create mode 100644 src/test/test-node-affinity-strict4/README create mode 100644 src/test/test-node-affinity-strict4/cmdlist create mode 100644 src/test/test-node-affinity-strict4/hardware_status create mode 100644 src/test/test-node-affinity-strict4/log.expect create mode 100644 src/test/test-node-affinity-strict4/manager_status create mode 100644 src/test/test-node-affinity-strict4/rules_config create mode 100644 src/test/test-node-affinity-strict4/service_config create mode 100644 src/test/test-node-affinity-strict5/README create mode 100644 src/test/test-node-affinity-strict5/cmdlist create mode 100644 src/test/test-node-affinity-strict5/hardware_status create mode 100644 src/test/test-node-affinity-strict5/log.expect create mode 100644 src/test/test-node-affinity-strict5/manager_status create mode 100644 src/test/test-node-affinity-strict5/rules_config create mode 100644 src/test/test-node-affinity-strict5/service_config create mode 100644 src/test/test-node-affinity-strict6/README create mode 100644 src/test/test-node-affinity-strict6/cmdlist create mode 100644 src/test/test-node-affinity-strict6/hardware_status create mode 100644 src/test/test-node-affinity-strict6/log.expect create mode 100644 src/test/test-node-affinity-strict6/manager_status create mode 100644 src/test/test-node-affinity-strict6/rules_config create mode 100644 src/test/test-node-affinity-strict6/service_config delete mode 100644 src/test/test-recovery1/groups create mode 100644 src/test/test-recovery1/rules_config delete mode 100644 src/test/test-recovery2/groups create mode 100644 src/test/test-recovery2/rules_config delete mode 100644 src/test/test-recovery3/groups create mode 100644 src/test/test-recovery3/rules_config delete mode 100644 src/test/test-recovery4/groups create mode 100644 src/test/test-recovery4/rules_config delete mode 100644 src/test/test-resource-failure2/groups create mode 100644 src/test/test-resource-failure2/rules_config delete mode 100644 src/test/test-shutdown2/groups create mode 100644 src/test/test-shutdown2/rules_config delete mode 100644 src/test/test-shutdown3/groups create mode 100644 src/test/test-shutdown3/rules_config create mode 100755 src/test/test_rules_config.pl docs: Daniel Kral (2): ha: add documentation about ha rules and ha node affinity rules ha: crs: add effects of ha node affinity rule on the crs scheduler Makefile | 2 + gen-ha-rules-node-affinity-opts.pl | 20 +++++ gen-ha-rules-opts.pl | 17 +++++ ha-manager.adoc | 113 +++++++++++++++++++++++++++++ ha-rules-node-affinity-opts.adoc | 18 +++++ ha-rules-opts.adoc | 12 +++ pmxcfs.adoc | 1 + 7 files changed, 183 insertions(+) create mode 100755 gen-ha-rules-node-affinity-opts.pl create mode 100755 gen-ha-rules-opts.pl create mode 100644 ha-rules-node-affinity-opts.adoc create mode 100644 ha-rules-opts.adoc manager: Daniel Kral (4): api: ha: add ha rules api endpoints ui: ha: remove ha groups from ha resource components ui: ha: show failback flag in resources status view ui: ha: replace ha groups with ha node affinity rules PVE/API2/HAConfig.pm | 8 +- www/manager6/Makefile | 8 +- www/manager6/StateProvider.js | 2 +- www/manager6/dc/Config.js | 8 +- www/manager6/ha/GroupSelector.js | 71 ------- www/manager6/ha/Groups.js | 117 ----------- www/manager6/ha/ResourceEdit.js | 16 +- www/manager6/ha/Resources.js | 17 +- www/manager6/ha/RuleEdit.js | 146 +++++++++++++ www/manager6/ha/RuleErrorsModal.js | 50 +++++ www/manager6/ha/Rules.js | 196 ++++++++++++++++++ www/manager6/ha/StatusView.js | 5 +- .../NodeAffinityRuleEdit.js} | 105 ++-------- www/manager6/ha/rules/NodeAffinityRules.js | 36 ++++ 14 files changed, 481 insertions(+), 304 deletions(-) delete mode 100644 www/manager6/ha/GroupSelector.js delete mode 100644 www/manager6/ha/Groups.js create mode 100644 www/manager6/ha/RuleEdit.js create mode 100644 www/manager6/ha/RuleErrorsModal.js create mode 100644 www/manager6/ha/Rules.js rename www/manager6/ha/{GroupEdit.js => rules/NodeAffinityRuleEdit.js} (67%) create mode 100644 www/manager6/ha/rules/NodeAffinityRules.js Summary over all repositories: 181 files changed, 4038 insertions(+), 477 deletions(-) -- Generated by git-murpp 0.8.0 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel