From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 9FD651FF16B for ; Tue, 29 Jul 2025 20:02:12 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BE5E51A5A9; Tue, 29 Jul 2025 20:02:03 +0200 (CEST) From: Daniel Kral To: pve-devel@lists.proxmox.com Date: Tue, 29 Jul 2025 20:00:40 +0200 Message-ID: <20250729180107.428855-1-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.2 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1753812063178 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.112 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [PATCH docs/ha-manager/manager v4 00/25] HA Rules X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" Here's a quick update on the core HA rules series. This cleans up the series so that all tests are running again and includes the missing ui patch that I didn't see missing last time. The persistent migration path has been tested for at least four full upgrade runs now, always with one node being behind and checking that the group config is only removed as soon as all nodes are on the right version. I'll wait for tomorrow if something comes up and will do some testing myself, so I'm anticipating to follow up on this tomorrow. I'll also want to get a more mature version of the HA resource affinity series ready for tomorrow on the mailing list. For maintainers: ha-manager patch #19 should be updated to the correct pve-manager version that is dependent on the pve-ha-manager package which can interpret the HA rules config. Changelog since v3 ------------------ - rebased on newest available master - included missing ui patch for web interface - correction in failback property description (does not influence the ha node affinity rules) - migrated the groups configs in the test cases to node affinity rules in rules configs (except two test cases for the persistent migration) - improved persistent ha group migration process - try a persistent upgrade only every 10 HA manager rounds - various other minor touches TODO ---- - More testing on edge cases for the HA Manager migration path - Some more testing of the ha-manager CLI and adding a deprecation warning on the HA Groups API and disallowing requests as soon as the groups config is fully migrated ha-manager: Daniel Kral (19): tree-wide: make arguments for select_service_node explicit manager: improve signature of select_service_node introduce rules base plugin rules: introduce node affinity rule plugin config, env, hw: add rules read and parse methods config: delete services from rules if services are deleted from config manager: read and update rules config test: ha tester: add test cases for future node affinity rules resources: introduce failback property in ha resource config manager: migrate ha groups to node affinity rules in-memory manager: apply node affinity rules when selecting service nodes test: add test cases for rules config api: introduce ha rules api endpoints cli: expose ha rules api endpoints to ha-manager cli sim: do not create default groups for test cases test: ha tester: migrate groups to service and rules config test: ha tester: replace any reference to groups with node affinity rules env: add property delete for update_service_config manager: persistently migrate ha groups to ha rules .gitignore | 1 + debian/pve-ha-manager.install | 3 + src/PVE/API2/HA/Makefile | 2 +- src/PVE/API2/HA/Resources.pm | 9 + src/PVE/API2/HA/Rules.pm | 391 +++++++++++++++ src/PVE/API2/HA/Status.pm | 11 +- src/PVE/CLI/ha_manager.pm | 32 ++ src/PVE/HA/Config.pm | 58 ++- src/PVE/HA/Env.pm | 34 +- src/PVE/HA/Env/PVE2.pm | 45 +- src/PVE/HA/Groups.pm | 47 ++ src/PVE/HA/Makefile | 3 +- src/PVE/HA/Manager.pm | 279 +++++++---- src/PVE/HA/Resources.pm | 9 + src/PVE/HA/Resources/PVECT.pm | 1 + src/PVE/HA/Resources/PVEVM.pm | 1 + src/PVE/HA/Rules.pm | 455 ++++++++++++++++++ src/PVE/HA/Rules/Makefile | 6 + src/PVE/HA/Rules/NodeAffinity.pm | 296 ++++++++++++ src/PVE/HA/Sim/Env.pm | 48 +- src/PVE/HA/Sim/Hardware.pm | 70 ++- src/PVE/HA/Sim/RTHardware.pm | 18 + src/PVE/HA/Tools.pm | 46 ++ src/test/Makefile | 4 +- .../defaults-for-node-affinity-rules.cfg | 22 + ...efaults-for-node-affinity-rules.cfg.expect | 60 +++ ...e-resource-refs-in-node-affinity-rules.cfg | 31 ++ ...rce-refs-in-node-affinity-rules.cfg.expect | 63 +++ src/test/test-basic5/groups | 2 - src/test/test-basic5/rules_config | 3 + src/test/test-basic5/service_config | 2 +- src/test/test-crs-static2/README | 3 +- src/test/test-crs-static2/groups | 2 - src/test/test-crs-static2/rules_config | 3 + src/test/test-crs-static2/service_config | 2 +- src/test/test-group-migrate1/README | 4 + src/test/test-group-migrate1/cmdlist | 4 + src/test/test-group-migrate1/groups | 7 + src/test/test-group-migrate1/hardware_status | 5 + src/test/test-group-migrate1/log.expect | 87 ++++ src/test/test-group-migrate1/manager_status | 1 + src/test/test-group-migrate1/service_config | 5 + src/test/test-group-migrate2/README | 3 + src/test/test-group-migrate2/cmdlist | 3 + src/test/test-group-migrate2/groups | 7 + src/test/test-group-migrate2/hardware_status | 5 + src/test/test-group-migrate2/log.expect | 47 ++ src/test/test-group-migrate2/manager_status | 1 + src/test/test-group-migrate2/service_config | 5 + src/test/test-node-affinity-nonstrict1/README | 11 + .../test-node-affinity-nonstrict1/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict1/log.expect | 40 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict2/README | 12 + .../test-node-affinity-nonstrict2/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict2/log.expect | 35 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict3/README | 10 + .../test-node-affinity-nonstrict3/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict3/log.expect | 56 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 5 + src/test/test-node-affinity-nonstrict4/README | 14 + .../test-node-affinity-nonstrict4/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-nonstrict4/log.expect | 54 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 5 + src/test/test-node-affinity-nonstrict5/README | 16 + .../test-node-affinity-nonstrict5/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-nonstrict5/log.expect | 66 +++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-nonstrict6/README | 14 + .../test-node-affinity-nonstrict6/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-nonstrict6/log.expect | 52 ++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + src/test/test-node-affinity-strict1/README | 11 + src/test/test-node-affinity-strict1/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict1/log.expect | 40 ++ .../test-node-affinity-strict1/manager_status | 1 + .../test-node-affinity-strict1/rules_config | 4 + .../test-node-affinity-strict1/service_config | 3 + src/test/test-node-affinity-strict2/README | 11 + src/test/test-node-affinity-strict2/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict2/log.expect | 40 ++ .../test-node-affinity-strict2/manager_status | 1 + .../test-node-affinity-strict2/rules_config | 4 + .../test-node-affinity-strict2/service_config | 3 + src/test/test-node-affinity-strict3/README | 10 + src/test/test-node-affinity-strict3/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict3/log.expect | 74 +++ .../test-node-affinity-strict3/manager_status | 1 + .../test-node-affinity-strict3/rules_config | 4 + .../test-node-affinity-strict3/service_config | 5 + src/test/test-node-affinity-strict4/README | 14 + src/test/test-node-affinity-strict4/cmdlist | 4 + .../hardware_status | 5 + .../test-node-affinity-strict4/log.expect | 54 +++ .../test-node-affinity-strict4/manager_status | 1 + .../test-node-affinity-strict4/rules_config | 4 + .../test-node-affinity-strict4/service_config | 5 + src/test/test-node-affinity-strict5/README | 16 + src/test/test-node-affinity-strict5/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-strict5/log.expect | 66 +++ .../test-node-affinity-strict5/manager_status | 1 + .../test-node-affinity-strict5/rules_config | 4 + .../test-node-affinity-strict5/service_config | 3 + src/test/test-node-affinity-strict6/README | 15 + src/test/test-node-affinity-strict6/cmdlist | 5 + .../hardware_status | 5 + .../test-node-affinity-strict6/log.expect | 52 ++ .../test-node-affinity-strict6/manager_status | 1 + .../test-node-affinity-strict6/rules_config | 4 + .../test-node-affinity-strict6/service_config | 3 + src/test/test-recovery1/groups | 4 - src/test/test-recovery1/rules_config | 4 + src/test/test-recovery1/service_config | 2 +- src/test/test-recovery2/README | 4 +- src/test/test-recovery2/groups | 4 - src/test/test-recovery2/rules_config | 4 + src/test/test-recovery2/service_config | 2 +- src/test/test-recovery3/groups | 4 - src/test/test-recovery3/rules_config | 4 + src/test/test-recovery3/service_config | 2 +- src/test/test-recovery4/groups | 4 - src/test/test-recovery4/rules_config | 4 + src/test/test-recovery4/service_config | 2 +- .../test-relocate-policy-default-group/README | 6 +- src/test/test-resource-failure2/groups | 2 - src/test/test-resource-failure2/rules_config | 3 + .../test-resource-failure2/service_config | 2 +- .../test-resource-failure3/service_config | 2 +- src/test/test-resource-failure6/README | 6 +- src/test/test-shutdown2/groups | 2 - src/test/test-shutdown2/rules_config | 3 + src/test/test-shutdown2/service_config | 4 +- src/test/test-shutdown3/groups | 2 - src/test/test-shutdown3/rules_config | 3 + src/test/test-shutdown3/service_config | 4 +- src/test/test_failover1.pl | 27 +- src/test/test_rules_config.pl | 100 ++++ 160 files changed, 3284 insertions(+), 165 deletions(-) create mode 100644 src/PVE/API2/HA/Rules.pm create mode 100644 src/PVE/HA/Rules.pm create mode 100644 src/PVE/HA/Rules/Makefile create mode 100644 src/PVE/HA/Rules/NodeAffinity.pm create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg.expect create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg.expect delete mode 100644 src/test/test-basic5/groups create mode 100644 src/test/test-basic5/rules_config delete mode 100644 src/test/test-crs-static2/groups create mode 100644 src/test/test-crs-static2/rules_config create mode 100644 src/test/test-group-migrate1/README create mode 100644 src/test/test-group-migrate1/cmdlist create mode 100644 src/test/test-group-migrate1/groups create mode 100644 src/test/test-group-migrate1/hardware_status create mode 100644 src/test/test-group-migrate1/log.expect create mode 100644 src/test/test-group-migrate1/manager_status create mode 100644 src/test/test-group-migrate1/service_config create mode 100644 src/test/test-group-migrate2/README create mode 100644 src/test/test-group-migrate2/cmdlist create mode 100644 src/test/test-group-migrate2/groups create mode 100644 src/test/test-group-migrate2/hardware_status create mode 100644 src/test/test-group-migrate2/log.expect create mode 100644 src/test/test-group-migrate2/manager_status create mode 100644 src/test/test-group-migrate2/service_config create mode 100644 src/test/test-node-affinity-nonstrict1/README create mode 100644 src/test/test-node-affinity-nonstrict1/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict1/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict1/log.expect create mode 100644 src/test/test-node-affinity-nonstrict1/manager_status create mode 100644 src/test/test-node-affinity-nonstrict1/rules_config create mode 100644 src/test/test-node-affinity-nonstrict1/service_config create mode 100644 src/test/test-node-affinity-nonstrict2/README create mode 100644 src/test/test-node-affinity-nonstrict2/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict2/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict2/log.expect create mode 100644 src/test/test-node-affinity-nonstrict2/manager_status create mode 100644 src/test/test-node-affinity-nonstrict2/rules_config create mode 100644 src/test/test-node-affinity-nonstrict2/service_config create mode 100644 src/test/test-node-affinity-nonstrict3/README create mode 100644 src/test/test-node-affinity-nonstrict3/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict3/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict3/log.expect create mode 100644 src/test/test-node-affinity-nonstrict3/manager_status create mode 100644 src/test/test-node-affinity-nonstrict3/rules_config create mode 100644 src/test/test-node-affinity-nonstrict3/service_config create mode 100644 src/test/test-node-affinity-nonstrict4/README create mode 100644 src/test/test-node-affinity-nonstrict4/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict4/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict4/log.expect create mode 100644 src/test/test-node-affinity-nonstrict4/manager_status create mode 100644 src/test/test-node-affinity-nonstrict4/rules_config create mode 100644 src/test/test-node-affinity-nonstrict4/service_config create mode 100644 src/test/test-node-affinity-nonstrict5/README create mode 100644 src/test/test-node-affinity-nonstrict5/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict5/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict5/log.expect create mode 100644 src/test/test-node-affinity-nonstrict5/manager_status create mode 100644 src/test/test-node-affinity-nonstrict5/rules_config create mode 100644 src/test/test-node-affinity-nonstrict5/service_config create mode 100644 src/test/test-node-affinity-nonstrict6/README create mode 100644 src/test/test-node-affinity-nonstrict6/cmdlist create mode 100644 src/test/test-node-affinity-nonstrict6/hardware_status create mode 100644 src/test/test-node-affinity-nonstrict6/log.expect create mode 100644 src/test/test-node-affinity-nonstrict6/manager_status create mode 100644 src/test/test-node-affinity-nonstrict6/rules_config create mode 100644 src/test/test-node-affinity-nonstrict6/service_config create mode 100644 src/test/test-node-affinity-strict1/README create mode 100644 src/test/test-node-affinity-strict1/cmdlist create mode 100644 src/test/test-node-affinity-strict1/hardware_status create mode 100644 src/test/test-node-affinity-strict1/log.expect create mode 100644 src/test/test-node-affinity-strict1/manager_status create mode 100644 src/test/test-node-affinity-strict1/rules_config create mode 100644 src/test/test-node-affinity-strict1/service_config create mode 100644 src/test/test-node-affinity-strict2/README create mode 100644 src/test/test-node-affinity-strict2/cmdlist create mode 100644 src/test/test-node-affinity-strict2/hardware_status create mode 100644 src/test/test-node-affinity-strict2/log.expect create mode 100644 src/test/test-node-affinity-strict2/manager_status create mode 100644 src/test/test-node-affinity-strict2/rules_config create mode 100644 src/test/test-node-affinity-strict2/service_config create mode 100644 src/test/test-node-affinity-strict3/README create mode 100644 src/test/test-node-affinity-strict3/cmdlist create mode 100644 src/test/test-node-affinity-strict3/hardware_status create mode 100644 src/test/test-node-affinity-strict3/log.expect create mode 100644 src/test/test-node-affinity-strict3/manager_status create mode 100644 src/test/test-node-affinity-strict3/rules_config create mode 100644 src/test/test-node-affinity-strict3/service_config create mode 100644 src/test/test-node-affinity-strict4/README create mode 100644 src/test/test-node-affinity-strict4/cmdlist create mode 100644 src/test/test-node-affinity-strict4/hardware_status create mode 100644 src/test/test-node-affinity-strict4/log.expect create mode 100644 src/test/test-node-affinity-strict4/manager_status create mode 100644 src/test/test-node-affinity-strict4/rules_config create mode 100644 src/test/test-node-affinity-strict4/service_config create mode 100644 src/test/test-node-affinity-strict5/README create mode 100644 src/test/test-node-affinity-strict5/cmdlist create mode 100644 src/test/test-node-affinity-strict5/hardware_status create mode 100644 src/test/test-node-affinity-strict5/log.expect create mode 100644 src/test/test-node-affinity-strict5/manager_status create mode 100644 src/test/test-node-affinity-strict5/rules_config create mode 100644 src/test/test-node-affinity-strict5/service_config create mode 100644 src/test/test-node-affinity-strict6/README create mode 100644 src/test/test-node-affinity-strict6/cmdlist create mode 100644 src/test/test-node-affinity-strict6/hardware_status create mode 100644 src/test/test-node-affinity-strict6/log.expect create mode 100644 src/test/test-node-affinity-strict6/manager_status create mode 100644 src/test/test-node-affinity-strict6/rules_config create mode 100644 src/test/test-node-affinity-strict6/service_config delete mode 100644 src/test/test-recovery1/groups create mode 100644 src/test/test-recovery1/rules_config delete mode 100644 src/test/test-recovery2/groups create mode 100644 src/test/test-recovery2/rules_config delete mode 100644 src/test/test-recovery3/groups create mode 100644 src/test/test-recovery3/rules_config delete mode 100644 src/test/test-recovery4/groups create mode 100644 src/test/test-recovery4/rules_config delete mode 100644 src/test/test-resource-failure2/groups create mode 100644 src/test/test-resource-failure2/rules_config delete mode 100644 src/test/test-shutdown2/groups create mode 100644 src/test/test-shutdown2/rules_config delete mode 100644 src/test/test-shutdown3/groups create mode 100644 src/test/test-shutdown3/rules_config create mode 100755 src/test/test_rules_config.pl docs: Daniel Kral (2): ha: add documentation about ha rules and ha node affinity rules ha: crs: add effects of ha node affinity rule on the crs scheduler Makefile | 2 + gen-ha-rules-node-affinity-opts.pl | 20 +++++ gen-ha-rules-opts.pl | 17 +++++ ha-manager.adoc | 113 +++++++++++++++++++++++++++++ ha-rules-node-affinity-opts.adoc | 18 +++++ ha-rules-opts.adoc | 12 +++ pmxcfs.adoc | 1 + 7 files changed, 183 insertions(+) create mode 100755 gen-ha-rules-node-affinity-opts.pl create mode 100755 gen-ha-rules-opts.pl create mode 100644 ha-rules-node-affinity-opts.adoc create mode 100644 ha-rules-opts.adoc manager: Daniel Kral (4): api: ha: add ha rules api endpoints ui: ha: remove ha groups from ha resource components ui: ha: show failback flag in resources status view ui: ha: replace ha groups with ha node affinity rules PVE/API2/HAConfig.pm | 8 +- www/manager6/Makefile | 8 +- www/manager6/StateProvider.js | 2 +- www/manager6/dc/Config.js | 8 +- www/manager6/ha/GroupSelector.js | 71 ------- www/manager6/ha/Groups.js | 117 ----------- www/manager6/ha/ResourceEdit.js | 16 +- www/manager6/ha/Resources.js | 17 +- www/manager6/ha/RuleEdit.js | 146 +++++++++++++ www/manager6/ha/RuleErrorsModal.js | 50 +++++ www/manager6/ha/Rules.js | 196 ++++++++++++++++++ www/manager6/ha/StatusView.js | 5 +- .../NodeAffinityRuleEdit.js} | 105 ++-------- www/manager6/ha/rules/NodeAffinityRules.js | 36 ++++ 14 files changed, 481 insertions(+), 304 deletions(-) delete mode 100644 www/manager6/ha/GroupSelector.js delete mode 100644 www/manager6/ha/Groups.js create mode 100644 www/manager6/ha/RuleEdit.js create mode 100644 www/manager6/ha/RuleErrorsModal.js create mode 100644 www/manager6/ha/Rules.js rename www/manager6/ha/{GroupEdit.js => rules/NodeAffinityRuleEdit.js} (67%) create mode 100644 www/manager6/ha/rules/NodeAffinityRules.js base-commit: 58f080d41079a0c2494a61cdaf382e7e7adf39db prerequisite-patch-id: f7b4b46d65fb44865ff09e2cc59cdf7fd659b48c prerequisite-patch-id: d9576d6a257f301a2396e2874c734b0d892a5761 prerequisite-patch-id: d580d2af8dba82526b35ccce0b0e943667666040 prerequisite-patch-id: 2ad790fd968ed7da823222103e17605498a1e5c8 prerequisite-patch-id: 30dd0ce7ccbc9c5b8a3153515ab320c9bcbff3b7 prerequisite-patch-id: 0420d015785effb38262f336c2f30566fa7f0ef5 prerequisite-patch-id: 5a975adc780d83762ebb2be29fc03f1cd3e6eb69 prerequisite-patch-id: 9a86ed00a5f9c565259de761bc1a141e49820874 prerequisite-patch-id: 84bba3312f1c47432c6d86075e422509653b8263 Summary over all repositories: 181 files changed, 3948 insertions(+), 469 deletions(-) -- Generated by git-murpp 0.8.0 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel