From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 5A2BE1FF13B for ; Wed, 22 Apr 2026 12:01:38 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B95C21390F; Wed, 22 Apr 2026 12:01:31 +0200 (CEST) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager 3/7] test: add test cases for resource affinity rules with maintenance mode Date: Wed, 22 Apr 2026 12:00:21 +0200 Message-ID: <20260422100035.232716-4-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260422100035.232716-1-d.kral@proxmox.com> References: <20260422100035.232716-1-d.kral@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1776851951750 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.079 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 63HEZHL5VELS43U3HJ7R7VBO5GVAPFPD X-Message-ID-Hash: 63HEZHL5VELS43U3HJ7R7VBO5GVAPFPD X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: These test cases document how the HA Manager currently behaves for positive and negative resource affinity rules as well as resource affinity rules with node affinity rules mixed, whose relevant nodes are put in maintenance mode and available afterwards again. Signed-off-by: Daniel Kral --- .../README | 5 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 54 +++++++++++++++++++ .../manager_status | 1 + .../rules_config | 3 ++ .../service_config | 4 ++ .../README | 4 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 47 ++++++++++++++++ .../manager_status | 1 + .../rules_config | 3 ++ .../service_config | 5 ++ .../README | 3 ++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 51 ++++++++++++++++++ .../manager_status | 1 + .../rules_config | 3 ++ .../service_config | 4 ++ .../README | 9 ++++ .../cmdlist | 4 ++ .../hardware_status | 5 ++ .../log.expect | 46 ++++++++++++++++ .../manager_status | 34 ++++++++++++ .../rules_config | 3 ++ .../service_config | 5 ++ .../README | 8 +++ .../cmdlist | 5 ++ .../hardware_status | 5 ++ .../log.expect | 41 ++++++++++++++ .../manager_status | 1 + .../rules_config | 7 +++ .../service_config | 4 ++ 35 files changed, 396 insertions(+) create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/service_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/README create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/log.expect create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/manager_status create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/rules_config create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/service_config create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/README b/src/test/test-resource-affinity-maintenance-strict-negative1/README new file mode 100644 index 00000000..5365ebce --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/README @@ -0,0 +1,5 @@ +Tests whether a strict negative resource affinity rule among two HA resources +makes the HA resource, whose current node is in maintenance mode, move to a +replacement node (different node than the other HA resources' node) and moves +the HA resource back to its previous maintenance node as soon as it's available +again. diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist b/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist new file mode 100644 index 00000000..7e577b68 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist @@ -0,0 +1,5 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on"], + [ "crm node3 enable-node-maintenance" ], + [ "crm node3 disable-node-maintenance" ] +] diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status b/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status new file mode 100644 index 00000000..451beb13 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off" }, + "node2": { "power": "off", "network": "off" }, + "node3": { "power": "off", "network": "off" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect b/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect new file mode 100644 index 00000000..1fc25206 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect @@ -0,0 +1,54 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node2' +info 20 node1/crm: adding new service 'vm:102' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3) +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:101 +info 23 node2/lrm: service status vm:101 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:102 +info 25 node3/lrm: service status vm:102 started +info 120 cmdlist: execute crm node3 enable-node-maintenance +info 125 node3/lrm: status change active => maintenance +info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance' +info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running) +info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1) +info 141 node1/lrm: got lock 'ha_agent_node1_lock' +info 141 node1/lrm: status change wait_for_agent_lock => active +info 145 node3/lrm: service vm:102 - start migrate to node 'node1' +info 145 node3/lrm: service vm:102 - end migrate to node 'node1' +info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1) +info 161 node1/lrm: starting service vm:102 +info 161 node1/lrm: service status vm:102 started +info 220 cmdlist: execute crm node3 disable-node-maintenance +info 225 node3/lrm: got lock 'ha_agent_node3_lock' +info 225 node3/lrm: status change maintenance => active +info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online' +info 240 node1/crm: moving service 'vm:102' back to 'node3', node came back from maintenance. +info 240 node1/crm: migrate service 'vm:102' to node 'node3' (running) +info 240 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3) +info 241 node1/lrm: service vm:102 - start migrate to node 'node3' +info 241 node1/lrm: service vm:102 - end migrate to node 'node3' +info 260 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3) +info 265 node3/lrm: starting service vm:102 +info 265 node3/lrm: service status vm:102 started +info 820 hardware: exit simulation - done diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status b/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config b/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config new file mode 100644 index 00000000..20747760 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config @@ -0,0 +1,3 @@ +resource-affinity: lonely-must-vms-be + resources vm:101,vm:102 + affinity negative diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/service_config b/src/test/test-resource-affinity-maintenance-strict-negative1/service_config new file mode 100644 index 00000000..e42e5c79 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative1/service_config @@ -0,0 +1,4 @@ +{ + "vm:101": { "node": "node2", "state": "started" }, + "vm:102": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/README b/src/test/test-resource-affinity-maintenance-strict-negative2/README new file mode 100644 index 00000000..a2102c2f --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/README @@ -0,0 +1,4 @@ +Tests whether a strict negative resource affinity rule among three HA resources +makes the HA resource, whose current node is in maintenance mode, stay on its +current node, even though it is in maintenance mode, because it cannot find any +replacement node. diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist b/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist new file mode 100644 index 00000000..7e577b68 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist @@ -0,0 +1,5 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on"], + [ "crm node3 enable-node-maintenance" ], + [ "crm node3 disable-node-maintenance" ] +] diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status b/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status new file mode 100644 index 00000000..451beb13 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off" }, + "node2": { "power": "off", "network": "off" }, + "node3": { "power": "off", "network": "off" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect b/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect new file mode 100644 index 00000000..505702f7 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect @@ -0,0 +1,47 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node2' +info 20 node1/crm: adding new service 'vm:103' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:102 +info 23 node2/lrm: service status vm:102 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:103 +info 25 node3/lrm: service status vm:103 started +info 120 cmdlist: execute crm node3 enable-node-maintenance +info 125 node3/lrm: status change active => maintenance +info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance' +warn 140 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance +warn 160 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance +warn 180 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance +warn 200 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance +info 220 cmdlist: execute crm node3 disable-node-maintenance +warn 220 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance +info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online' +info 240 node1/crm: service 'vm:103': clearing stale maintenance node 'node3' setting (is current node) +info 820 hardware: exit simulation - done diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status b/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config b/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config new file mode 100644 index 00000000..44e6a02e --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config @@ -0,0 +1,3 @@ +resource-affinity: lonely-must-vms-be + resources vm:101,vm:102,vm:103 + affinity negative diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/service_config b/src/test/test-resource-affinity-maintenance-strict-negative2/service_config new file mode 100644 index 00000000..4b26f6b4 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-negative2/service_config @@ -0,0 +1,5 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node2", "state": "started" }, + "vm:103": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/README b/src/test/test-resource-affinity-maintenance-strict-positive1/README new file mode 100644 index 00000000..4b62e578 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/README @@ -0,0 +1,3 @@ +Tests whether a strict positive resource affinity rule among two HA resources +makes both HA resources move to the same replacement node in case their +current, common node is put in maintenance mode. diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist b/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist new file mode 100644 index 00000000..97fbc1ef --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist @@ -0,0 +1,5 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "crm node3 enable-node-maintenance" ], + [ "crm node3 disable-node-maintenance" ] +] diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status b/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status new file mode 100644 index 00000000..451beb13 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off" }, + "node2": { "power": "off", "network": "off" }, + "node3": { "power": "off", "network": "off" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect new file mode 100644 index 00000000..5f91b877 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect @@ -0,0 +1,51 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node3' +info 20 node1/crm: adding new service 'vm:102' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3) +info 22 node2/crm: status change wait_for_quorum => slave +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:101 +info 25 node3/lrm: service status vm:101 started +info 25 node3/lrm: starting service vm:102 +info 25 node3/lrm: service status vm:102 started +info 120 cmdlist: execute crm node3 enable-node-maintenance +info 125 node3/lrm: status change active => maintenance +info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance' +info 140 node1/crm: migrate service 'vm:101' to node 'node1' (running) +info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1) +info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running) +info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1) +info 141 node1/lrm: got lock 'ha_agent_node1_lock' +info 141 node1/lrm: status change wait_for_agent_lock => active +info 145 node3/lrm: service vm:101 - start migrate to node 'node1' +info 145 node3/lrm: service vm:101 - end migrate to node 'node1' +info 145 node3/lrm: service vm:102 - start migrate to node 'node1' +info 145 node3/lrm: service vm:102 - end migrate to node 'node1' +info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1) +info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1) +info 161 node1/lrm: starting service vm:101 +info 161 node1/lrm: service status vm:101 started +info 161 node1/lrm: starting service vm:102 +info 161 node1/lrm: service status vm:102 started +info 220 cmdlist: execute crm node3 disable-node-maintenance +info 225 node3/lrm: got lock 'ha_agent_node3_lock' +info 225 node3/lrm: status change maintenance => active +info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online' +info 820 hardware: exit simulation - done diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status b/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config b/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config new file mode 100644 index 00000000..9789d7cc --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config @@ -0,0 +1,3 @@ +resource-affinity: vms-must-stick-together + resources vm:101,vm:102 + affinity positive diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/service_config b/src/test/test-resource-affinity-maintenance-strict-positive1/service_config new file mode 100644 index 00000000..50ef1caa --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive1/service_config @@ -0,0 +1,4 @@ +{ + "vm:101": { "node": "node3", "state": "started" }, + "vm:102": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/README b/src/test/test-resource-affinity-maintenance-strict-positive2/README new file mode 100644 index 00000000..32f0942b --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/README @@ -0,0 +1,9 @@ +Tests whether a strict positive resource affinity rule among three HA +resources, where two of them are already on a common node but the other HA +resource is still on another node, makes the former two HA resources move to +the node of the other HA resource as their current common node is put in +maintenance mode. + +The "skip-round crm 1" command ensures that the HA Manager will not move the +dislocated, third HA resource to the common node, but make the LRM acknowledge +its maintenance mode request. diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist b/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist new file mode 100644 index 00000000..2185ee6e --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on", "skip-round crm 1" ], + [ "crm node1 disable-node-maintenance" ] +] diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status b/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status new file mode 100644 index 00000000..451beb13 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off" }, + "node2": { "power": "off", "network": "off" }, + "node3": { "power": "off", "network": "off" } +} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect new file mode 100644 index 00000000..ef63c8ca --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect @@ -0,0 +1,46 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute skip-round crm 1 +info 20 run-loop: skipping CRM round +info 20 node1/lrm: got lock 'ha_agent_node1_lock' +info 20 node1/lrm: status change wait_for_agent_lock => active +info 20 node1/lrm: starting service vm:101 +info 20 node1/lrm: service status vm:101 started +info 20 node1/lrm: starting service vm:102 +info 20 node1/lrm: service status vm:102 started +info 22 node3/lrm: got lock 'ha_agent_node3_lock' +info 22 node3/lrm: status change wait_for_agent_lock => active +info 22 node3/lrm: starting service vm:103 +info 22 node3/lrm: service status vm:103 started +info 40 node1/crm: got lock 'ha_manager_lock' +info 40 node1/crm: status change wait_for_quorum => master +info 40 node1/crm: migrate service 'vm:101' to node 'node3' (running) +info 40 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3) +info 40 node1/crm: migrate service 'vm:102' to node 'node3' (running) +info 40 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3) +info 41 node1/lrm: status change active => maintenance +info 41 node1/lrm: service vm:101 - start migrate to node 'node3' +info 41 node1/lrm: service vm:101 - end migrate to node 'node3' +info 41 node1/lrm: service vm:102 - start migrate to node 'node3' +info 41 node1/lrm: service vm:102 - end migrate to node 'node3' +info 42 node2/crm: status change wait_for_quorum => slave +info 44 node3/crm: status change wait_for_quorum => slave +info 60 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3) +info 60 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3) +info 65 node3/lrm: starting service vm:101 +info 65 node3/lrm: service status vm:101 started +info 65 node3/lrm: starting service vm:102 +info 65 node3/lrm: service status vm:102 started +info 120 cmdlist: execute crm node1 disable-node-maintenance +info 121 node1/lrm: got lock 'ha_agent_node1_lock' +info 121 node1/lrm: status change maintenance => active +info 140 node1/crm: node 'node1': state changed from 'maintenance' => 'online' +info 720 hardware: exit simulation - done diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status b/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status new file mode 100644 index 00000000..135a1d6f --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status @@ -0,0 +1,34 @@ +{ + "master_node": "node1", + "node_request": { + "node1": { + "maintenance": 1 + } + }, + "node_status": { + "node1": "maintenance", + "node2": "online", + "node3": "online" + }, + "service_status": { + "vm:101": { + "running": 1, + "node": "node1", + "state": "started", + "uid": "Xi3T+eaBD4iaN01s65D5/g" + }, + "vm:102": { + "running": 1, + "node": "node1", + "state": "started", + "uid": "F2xctkwVsaF2KY9gJYsz6g" + }, + "vm:103": { + "running": 1, + "node": "node3", + "state": "started", + "uid": "c5yeDFKYkhMe3Nv+XzmN0A" + } + }, + "timestamp": 40 +} diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config b/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config new file mode 100644 index 00000000..12da6e67 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config @@ -0,0 +1,3 @@ +resource-affinity: vms-must-stick-together + resources vm:101,vm:102,vm:103 + affinity positive diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/service_config b/src/test/test-resource-affinity-maintenance-strict-positive2/service_config new file mode 100644 index 00000000..32e61c84 --- /dev/null +++ b/src/test/test-resource-affinity-maintenance-strict-positive2/service_config @@ -0,0 +1,5 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README new file mode 100644 index 00000000..c6a11cec --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README @@ -0,0 +1,8 @@ +2 HA resources on a 3-node cluster, which are: + +- on node2 and node3 respectively, +- in a non-strict node affinity rule to node2 and node3 (equal priority), and +- in a strict negative resource affinity rule with each other. + +Tests whether the HA resource on node3 will stay there, even though node3 is +put in maintenance mode, because it cannot find any replacement node. diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist new file mode 100644 index 00000000..97fbc1ef --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist @@ -0,0 +1,5 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "crm node3 enable-node-maintenance" ], + [ "crm node3 disable-node-maintenance" ] +] diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status new file mode 100644 index 00000000..451beb13 --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off" }, + "node2": { "power": "off", "network": "off" }, + "node3": { "power": "off", "network": "off" } +} diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect new file mode 100644 index 00000000..8899f782 --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect @@ -0,0 +1,41 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node2' +info 20 node1/crm: adding new service 'vm:102' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3) +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:101 +info 23 node2/lrm: service status vm:101 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:102 +info 25 node3/lrm: service status vm:102 started +info 120 cmdlist: execute crm node3 enable-node-maintenance +info 125 node3/lrm: status change active => maintenance +info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance' +warn 140 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance +warn 160 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance +warn 180 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance +warn 200 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance +info 220 cmdlist: execute crm node3 disable-node-maintenance +warn 220 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance +info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online' +info 240 node1/crm: service 'vm:102': clearing stale maintenance node 'node3' setting (is current node) +info 820 hardware: exit simulation - done diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status new file mode 100644 index 00000000..0967ef42 --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config new file mode 100644 index 00000000..e5bf3e47 --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config @@ -0,0 +1,7 @@ +node-affinity: vm101-vm102-should-be-on-node2-node3 + resources vm:101,vm:102 + nodes node2,node3 + +resource-affinity: lonely-must-vms-be + resources vm:101,vm:102 + affinity negative diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config new file mode 100644 index 00000000..e42e5c79 --- /dev/null +++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config @@ -0,0 +1,4 @@ +{ + "vm:101": { "node": "node2", "state": "started" }, + "vm:102": { "node": "node3", "state": "started" } +} -- 2.47.3