From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 652901FF0AF for ; Fri, 06 Nov 2026 10:57:31 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 49D582161C; Fri, 06 Nov 2026 10:57:13 +0100 (CET) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Date: Fri, 6 Nov 2026 10:56:43 +0100 Message-ID: <20261106095650.72865-1-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793959024303 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.689 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ES6ZMCFQUFEOD6CESKJFZCO4J66O33GF X-Message-ID-Hash: ES6ZMCFQUFEOD6CESKJFZCO4J66O33GF X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Changes since v1: - introduce patch to remove failed nodes earlier (thanks @Michael!) - drop patch for get_node_affinity() returning all priority classes as it isn't needed anymore - simplify handling of earlier negative resource affinity apply and moving back to maintenance node for failback-disabled HA resources - reorder patches according to these changes - improve readability of the comments v1: https://lore.proxmox.com/pve-devel/20260422100035.232716-1-d.kral@proxmox.com/ As reported by a Proxmox forum post [0], there are some cases where either HA resources are not moved away from maintenance nodes or are not moved back to the maintenance nodes after these are put out of maintenance again. Even though we cannot resolve all situations (for example, the affinity rules constrain the HA resource so that it cannot be moved anywhere but the maintenance node), this patch series improves the handling with: - log warnings if HA resources cannot be moved to a replacement node - improve negative resource affinity rule handling to allow using the non-highest-priority node priority classes as possible nodes - allow HA resources with fallback disabled move back to their previous maintenance node - make HA resource bundles move back to their previous maintenance node [0] https://forum.proxmox.com/threads/182890/ Daniel Kral (7): manager: warn if HA resources cannot be moved away from maintenance node test: add test cases for node affinity rules with maintenance mode test: add test cases for resource affinity rules with maintenance mode manager: apply negative resource affinity before get_node_affinity manager: make HA resource bundles move back to maintenance node manager: remove failed nodes earlier in select_service_node manager: move HA resources without failback back to maintenance node src/PVE/HA/Manager.pm | 56 ++++++++++----- .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 48 +++++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 48 +++++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + .../README | 9 +++ .../cmdlist | 6 ++ .../hardware_status | 5 ++ .../log.expect | 39 +++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 1 + .../README | 10 +++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 40 +++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 3 + .../README | 3 + .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 36 ++++++++++ .../manager_status | 1 + .../rules_config | 4 ++ .../service_config | 3 + .../README | 3 + .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 36 ++++++++++ .../manager_status | 1 + .../rules_config | 4 ++ .../service_config | 3 + .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 48 +++++++++++++ .../manager_status | 1 + .../rules_config | 4 ++ .../service_config | 3 + .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 48 +++++++++++++ .../manager_status | 1 + .../rules_config | 4 ++ .../service_config | 3 + .../README | 5 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 54 +++++++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 4 ++ .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 48 +++++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 5 ++ .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 67 ++++++++++++++++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 4 ++ .../README | 10 +++ .../cmdlist | 4 ++ .../hardware_status | 5 ++ .../log.expect | 69 +++++++++++++++++++ .../manager_status | 34 +++++++++ .../rules_config | 3 + .../service_config | 5 ++ .../README | 9 +++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 54 +++++++++++++++ .../manager_status | 1 + .../rules_config | 7 ++ .../service_config | 4 ++ .../README | 7 +- .../log.expect | 16 +++-- .../test-stale-maintenance-node/log.expect | 3 + 95 files changed, 1028 insertions(+), 29 deletions(-) create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/README create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/log.expect create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/manager_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/rules_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/service_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/README create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/log.expect create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/manager_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/rules_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/service_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/README create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/log.expect create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/manager_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/rules_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/service_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/README create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/log.expect create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/manager_status create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/rules_config create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/service_config create mode 100644 src/test/test-node-affinity-maintenance-strict1/README create mode 100644 src/test/test-node-affinity-maintenance-strict1/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-strict1/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-strict1/log.expect create mode 100644 src/test/test-node-affinity-maintenance-strict1/manager_status create mode 100644 src/test/test-node-affinity-maintenance-strict1/rules_config create mode 100644 src/test/test-node-affinity-maintenance-strict1/service_config create mode 100644 src/test/test-node-affinity-maintenance-strict2/README create mode 100644 src/test/test-node-affinity-maintenance-strict2/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-strict2/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-strict2/log.expect create mode 100644 src/test/test-node-affinity-maintenance-strict2/manager_status create mode 100644 src/test/test-node-affinity-maintenance-strict2/rules_config create mode 100644 src/test/test-node-affinity-maintenance-strict2/service_config create mode 100644 src/test/test-node-affinity-maintenance-strict3/README create mode 100644 src/test/test-node-affinity-maintenance-strict3/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-strict3/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-strict3/log.expect create mode 100644 src/test/test-node-affinity-maintenance-strict3/manager_status create mode 100644 src/test/test-node-affinity-maintenance-strict3/rules_config create mode 100644 src/test/test-node-affinity-maintenance-strict3/service_config create mode 100644 src/test/test-node-affinity-maintenance-strict4/README create mode 100644 src/test/test-node-affinity-maintenance-strict4/cmdlist create mode 100644 src/test/test-node-affinity-maintenance-strict4/hardware_status create mode 100644 src/test/test-node-affinity-maintenance-strict4/log.expect create mode 100644 src/test/test-node-affinity-maintenance-strict4/manager_status create mode 100644 src/test/test-node-affinity-maintenance-strict4/rules_config create mode 100644 src/test/test-node-affinity-maintenance-strict4/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/service_config create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config -- 2.47.3