* [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes
@ 2026-11-06 9:56 Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node Daniel Kral
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
Changes since v1:
- introduce patch to remove failed nodes earlier (thanks @Michael!)
- drop patch for get_node_affinity() returning all priority classes as
it isn't needed anymore
- simplify handling of earlier negative resource affinity apply and
moving back to maintenance node for failback-disabled HA resources
- reorder patches according to these changes
- improve readability of the comments
v1: https://lore.proxmox.com/pve-devel/20260422100035.232716-1-d.kral@proxmox.com/
As reported by a Proxmox forum post [0], there are some cases where
either HA resources are not moved away from maintenance nodes or are not
moved back to the maintenance nodes after these are put out of
maintenance again.
Even though we cannot resolve all situations (for example, the affinity
rules constrain the HA resource so that it cannot be moved anywhere but
the maintenance node), this patch series improves the handling with:
- log warnings if HA resources cannot be moved to a replacement node
- improve negative resource affinity rule handling to allow using the
non-highest-priority node priority classes as possible nodes
- allow HA resources with fallback disabled move back to their previous
maintenance node
- make HA resource bundles move back to their previous maintenance node
[0] https://forum.proxmox.com/threads/182890/
Daniel Kral (7):
manager: warn if HA resources cannot be moved away from maintenance
node
test: add test cases for node affinity rules with maintenance mode
test: add test cases for resource affinity rules with maintenance mode
manager: apply negative resource affinity before get_node_affinity
manager: make HA resource bundles move back to maintenance node
manager: remove failed nodes earlier in select_service_node
manager: move HA resources without failback back to maintenance node
src/PVE/HA/Manager.pm | 56 ++++++++++-----
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
.../README | 9 +++
.../cmdlist | 6 ++
.../hardware_status | 5 ++
.../log.expect | 39 +++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 1 +
.../README | 10 +++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 40 +++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
.../README | 3 +
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 36 ++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 +
.../README | 3 +
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 36 ++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 +
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 +
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 +
.../README | 5 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 54 +++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 4 ++
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 5 ++
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 67 ++++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 4 ++
.../README | 10 +++
.../cmdlist | 4 ++
.../hardware_status | 5 ++
.../log.expect | 69 +++++++++++++++++++
.../manager_status | 34 +++++++++
.../rules_config | 3 +
.../service_config | 5 ++
.../README | 9 +++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 54 +++++++++++++++
.../manager_status | 1 +
.../rules_config | 7 ++
.../service_config | 4 ++
.../README | 7 +-
.../log.expect | 16 +++--
.../test-stale-maintenance-node/log.expect | 3 +
95 files changed, 1028 insertions(+), 29 deletions(-)
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict1/README
create mode 100644 src/test/test-node-affinity-maintenance-strict1/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict1/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict1/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict1/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict1/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict1/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict2/README
create mode 100644 src/test/test-node-affinity-maintenance-strict2/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict2/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict2/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict2/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict2/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict2/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict3/README
create mode 100644 src/test/test-node-affinity-maintenance-strict3/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict3/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict3/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict3/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict3/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict3/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict4/README
create mode 100644 src/test/test-node-affinity-maintenance-strict4/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict4/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict4/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict4/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict4/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict4/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/service_config
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config
--
2.47.3
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 2/7] test: add test cases for node affinity rules with maintenance mode Daniel Kral
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
There are scenarios, where an HA resource cannot be moved away from its
current node when it changes to maintenance mode. This stems from
select_service_node() not being able to find any replacement node.
A common reason can be that the available node set for these HA
resources is constrained so much by online nodes and/or affinity rules
that there aren't any other options for select_service_node() than the
current node.
While some of these affinity-related scenarios need to be resolved in a
better way, admins should always be warned about such a situation, so
that they know that a node isn't fully in maintenance mode yet.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- reword comment
- put comment at a better place
src/PVE/HA/Manager.pm | 18 +++++++++++++-----
.../test-stale-maintenance-node/log.expect | 3 +++
2 files changed, 16 insertions(+), 5 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5840a76e..753da1a8 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -1589,20 +1589,28 @@ sub next_state_started {
);
}
+ # Having the maintenance node set to its current node here means that the
+ # HA resource was never started on a different node since it was set,
+ # because select_service_node() could not find any replacement node.
+ # Common reasons are affinity rules or the whole cluster was shut down at
+ # the same time while the 'migrate' shutdown policy was configured.
if ($sd->{maintenance_node} && $sd->{node} eq $sd->{maintenance_node}) {
my $node_state = $ns->get_node_state($sd->{node});
if ($node_state eq 'online') {
- # Having the maintenance node set here means that the service was never
- # started on a different node since it was set. This can happen in the edge
- # case that the whole cluster is shut down at the same time while the
- # 'migrate' policy was configured. Node is not in maintenance mode anymore
- # and service is started on this node, so it's fine to clear the setting.
+ # Node is not in maintenance mode anymore and service is started on this
+ # node, so it's fine to clear the setting.
$haenv->log(
'info',
"service '$sid': clearing stale maintenance node "
. "'$sd->{maintenance_node}' setting (is current node)",
);
delete $sd->{maintenance_node};
+ } else {
+ $haenv->log(
+ 'warning',
+ "service '$sid': cannot find a replacement node while"
+ . " its current node is in maintenance",
+ );
}
}
diff --git a/src/test/test-stale-maintenance-node/log.expect b/src/test/test-stale-maintenance-node/log.expect
index 092db8be..fce96fd4 100644
--- a/src/test/test-stale-maintenance-node/log.expect
+++ b/src/test/test-stale-maintenance-node/log.expect
@@ -33,6 +33,7 @@ info 120 node3/lrm: shutdown LRM, doing maintenance, removing this node fr
info 120 node1/crm: node 'node1': state changed from 'online' => 'maintenance'
info 120 node1/crm: node 'node2': state changed from 'online' => 'maintenance'
info 120 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn 120 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
info 121 node1/lrm: status change active => maintenance
info 124 node2/lrm: exit (loop end)
info 124 shutdown: execute crm node2 stop
@@ -40,6 +41,7 @@ info 123 node2/crm: server received shutdown request
info 126 node3/lrm: exit (loop end)
info 126 shutdown: execute crm node3 stop
info 125 node3/crm: server received shutdown request
+warn 140 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
info 143 node2/crm: exit (loop end)
info 143 shutdown: execute power node2 off
info 144 node3/crm: exit (loop end)
@@ -64,6 +66,7 @@ info 220 cmdlist: execute power node3 on
info 220 node3/crm: status change startup => wait_for_quorum
info 220 node3/lrm: status change startup => wait_for_agent_lock
info 220 node1/crm: status change wait_for_quorum => master
+warn 220 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
info 221 node1/lrm: status change wait_for_agent_lock => active
info 221 node1/lrm: starting service vm:103
info 221 node1/lrm: service status vm:103 started
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 2/7] test: add test cases for node affinity rules with maintenance mode
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 3/7] test: add test cases for resource " Daniel Kral
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
These test cases document how the HA Manager currently behaves for node
affinity with HA resources with failback enabled and disabled, whose
current nodes are put in maintenance mode and available afterwards
again.
The strict node affinity rule tests evaluate single and multi node
priority classes cases. For the non-strict node affinity rule test
cases, the single node member cases are sufficient, since non-strict
node affinity rules are implicitly multi-priority node affinity rules
with the non-member nodes having priority class -1.
As noted in the relevant test cases, it's up for debate whether HA
resources, whose maintenance node is a node of a lower node priority
class should move back as the maintenance mode is lifted. These test
cases do only document the current behavior of letting them stay on the
replacement node, which is in the highest node priority class.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- add test cases which document the behavior of an HA resource, which is
not on one of the nodes in its highest node priority class
- reword related test cases accordingly for consistency
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 3 ++
.../README | 3 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 40 ++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 3 ++
.../README | 9 ++++
.../cmdlist | 6 +++
.../hardware_status | 5 ++
.../log.expect | 39 +++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 1 +
.../README | 10 ++++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 40 ++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 3 ++
.../README | 3 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 36 ++++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 ++
.../README | 3 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 36 ++++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 ++
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 ++
.../README | 3 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 40 ++++++++++++++++
.../manager_status | 1 +
.../rules_config | 4 ++
.../service_config | 3 ++
56 files changed, 505 insertions(+)
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict1/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict2/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict3/service_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/README
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-nonstrict4/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict1/README
create mode 100644 src/test/test-node-affinity-maintenance-strict1/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict1/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict1/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict1/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict1/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict1/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict2/README
create mode 100644 src/test/test-node-affinity-maintenance-strict2/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict2/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict2/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict2/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict2/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict2/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict3/README
create mode 100644 src/test/test-node-affinity-maintenance-strict3/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict3/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict3/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict3/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict3/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict3/service_config
create mode 100644 src/test/test-node-affinity-maintenance-strict4/README
create mode 100644 src/test/test-node-affinity-maintenance-strict4/cmdlist
create mode 100644 src/test/test-node-affinity-maintenance-strict4/hardware_status
create mode 100644 src/test/test-node-affinity-maintenance-strict4/log.expect
create mode 100644 src/test/test-node-affinity-maintenance-strict4/manager_status
create mode 100644 src/test/test-node-affinity-maintenance-strict4/rules_config
create mode 100644 src/test/test-node-affinity-maintenance-strict4/service_config
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/README b/src/test/test-node-affinity-maintenance-nonstrict1/README
new file mode 100644
index 00000000..66111400
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/README
@@ -0,0 +1,4 @@
+Test whether an HA resource with failback enabled on the single member node of
+its non-strict node affinity rule will move to a replacement node if its
+current node is in maintenance mode and moves back to the previous maintenance
+node as soon as the node is available again.
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/cmdlist b/src/test/test-node-affinity-maintenance-nonstrict1/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/hardware_status b/src/test/test-node-affinity-maintenance-nonstrict1/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/log.expect b/src/test/test-node-affinity-maintenance-nonstrict1/log.expect
new file mode 100644
index 00000000..339ce3ab
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/log.expect
@@ -0,0 +1,48 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:101 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:101
+info 161 node1/lrm: service status vm:101 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:101' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 241 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:101
+info 265 node3/lrm: service status vm:101 started
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/manager_status b/src/test/test-node-affinity-maintenance-nonstrict1/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/rules_config b/src/test/test-node-affinity-maintenance-nonstrict1/rules_config
new file mode 100644
index 00000000..f758b512
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/rules_config
@@ -0,0 +1,3 @@
+node-affinity: vm101-should-be-on-node3
+ nodes node3
+ resources vm:101
diff --git a/src/test/test-node-affinity-maintenance-nonstrict1/service_config b/src/test/test-node-affinity-maintenance-nonstrict1/service_config
new file mode 100644
index 00000000..7f0b1bf9
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict1/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/README b/src/test/test-node-affinity-maintenance-nonstrict2/README
new file mode 100644
index 00000000..aa65a55a
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/README
@@ -0,0 +1,3 @@
+Test whether an HA resource with failback disabled on the single member node of
+its non-strict node affinity rule with a single node member will move to a
+replacement node if its current node is in maintenance mode.
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/cmdlist b/src/test/test-node-affinity-maintenance-nonstrict2/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/hardware_status b/src/test/test-node-affinity-maintenance-nonstrict2/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/log.expect b/src/test/test-node-affinity-maintenance-nonstrict2/log.expect
new file mode 100644
index 00000000..05a77a24
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/log.expect
@@ -0,0 +1,40 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:101 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:101
+info 161 node1/lrm: service status vm:101 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/manager_status b/src/test/test-node-affinity-maintenance-nonstrict2/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/rules_config b/src/test/test-node-affinity-maintenance-nonstrict2/rules_config
new file mode 100644
index 00000000..f758b512
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/rules_config
@@ -0,0 +1,3 @@
+node-affinity: vm101-should-be-on-node3
+ nodes node3
+ resources vm:101
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/service_config b/src/test/test-node-affinity-maintenance-nonstrict2/service_config
new file mode 100644
index 00000000..c7266eec
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started", "failback": 0 }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/README b/src/test/test-node-affinity-maintenance-nonstrict3/README
new file mode 100644
index 00000000..73217fba
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/README
@@ -0,0 +1,9 @@
+Test whether an HA resource with failback enabled on a non-member node of its
+non-strict node affinity rule will move to a replacement node if its current
+node is in maintenance mode. As the member node in the non-strict node affinity
+rule is available, the HA Manager will choose the member node as its
+replacement node.
+
+Moving the HA resource back to its previous maintenance node as the node is
+available again would be wasteful, because the HA resource would fail back to
+the member node immediately after.
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/cmdlist b/src/test/test-node-affinity-maintenance-nonstrict3/cmdlist
new file mode 100644
index 00000000..9d78d8cd
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "crm node2 enable-node-maintenance" ],
+ [ "service vm:101 add node2 started 1" ],
+ [ "crm node2 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/hardware_status b/src/test/test-node-affinity-maintenance-nonstrict3/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/log.expect b/src/test/test-node-affinity-maintenance-nonstrict3/log.expect
new file mode 100644
index 00000000..6f3ff1a2
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/log.expect
@@ -0,0 +1,39 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 120 cmdlist: execute crm node2 enable-node-maintenance
+info 120 node1/crm: got lock 'ha_manager_lock'
+info 120 node1/crm: status change wait_for_quorum => master
+info 120 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 120 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 120 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 140 node1/crm: node 'node2': state changed from 'online' => 'maintenance'
+info 220 cmdlist: execute service vm:101 add node2 started 1
+info 220 node1/crm: adding new service 'vm:101' on node 'node2'
+info 220 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node2)
+info 220 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 220 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node3)
+info 222 node2/crm: status change wait_for_quorum => slave
+info 223 node2/lrm: got lock 'ha_agent_node2_lock'
+info 223 node2/lrm: status change wait_for_agent_lock => active
+info 223 node2/lrm: service vm:101 - start migrate to node 'node3'
+info 223 node2/lrm: service vm:101 - end migrate to node 'node3'
+info 224 node3/crm: status change wait_for_quorum => slave
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change wait_for_agent_lock => active
+info 240 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 243 node2/lrm: status change active => maintenance
+info 245 node3/lrm: starting service vm:101
+info 245 node3/lrm: service status vm:101 started
+info 320 cmdlist: execute crm node2 disable-node-maintenance
+info 323 node2/lrm: got lock 'ha_agent_node2_lock'
+info 323 node2/lrm: status change maintenance => active
+info 340 node1/crm: node 'node2': state changed from 'maintenance' => 'online'
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/manager_status b/src/test/test-node-affinity-maintenance-nonstrict3/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/rules_config b/src/test/test-node-affinity-maintenance-nonstrict3/rules_config
new file mode 100644
index 00000000..f758b512
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/rules_config
@@ -0,0 +1,3 @@
+node-affinity: vm101-should-be-on-node3
+ nodes node3
+ resources vm:101
diff --git a/src/test/test-node-affinity-maintenance-nonstrict3/service_config b/src/test/test-node-affinity-maintenance-nonstrict3/service_config
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict3/service_config
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/README b/src/test/test-node-affinity-maintenance-nonstrict4/README
new file mode 100644
index 00000000..3239e369
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/README
@@ -0,0 +1,10 @@
+Test whether an HA resource with failback disabled on a non-member node of its
+non-strict node affinity rule will move to a replacement node if its current
+node is in maintenance mode. As the member node in the non-strict node affinity
+rule is available, the HA Manager will choose the member node as its
+replacement node.
+
+It's up for debate whether the HA resource should move back to its previous
+maintenance node as the node is available again, but currently the HA resource
+will be kept on the member node, because the member node is in the higher node
+priority class than the other non-member nodes.
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/cmdlist b/src/test/test-node-affinity-maintenance-nonstrict4/cmdlist
new file mode 100644
index 00000000..5aa42a6c
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "crm node1 enable-node-maintenance" ],
+ [ "crm node1 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/hardware_status b/src/test/test-node-affinity-maintenance-nonstrict4/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/log.expect b/src/test/test-node-affinity-maintenance-nonstrict4/log.expect
new file mode 100644
index 00000000..840f31be
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/log.expect
@@ -0,0 +1,40 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute crm node1 enable-node-maintenance
+info 121 node1/lrm: status change active => maintenance
+info 140 node1/crm: node 'node1': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 141 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 141 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 145 node3/lrm: got lock 'ha_agent_node3_lock'
+info 145 node3/lrm: status change wait_for_agent_lock => active
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 165 node3/lrm: starting service vm:101
+info 165 node3/lrm: service status vm:101 started
+info 220 cmdlist: execute crm node1 disable-node-maintenance
+info 221 node1/lrm: got lock 'ha_agent_node1_lock'
+info 221 node1/lrm: status change maintenance => active
+info 240 node1/crm: node 'node1': state changed from 'maintenance' => 'online'
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/manager_status b/src/test/test-node-affinity-maintenance-nonstrict4/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/rules_config b/src/test/test-node-affinity-maintenance-nonstrict4/rules_config
new file mode 100644
index 00000000..f758b512
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/rules_config
@@ -0,0 +1,3 @@
+node-affinity: vm101-should-be-on-node3
+ nodes node3
+ resources vm:101
diff --git a/src/test/test-node-affinity-maintenance-nonstrict4/service_config b/src/test/test-node-affinity-maintenance-nonstrict4/service_config
new file mode 100644
index 00000000..38f6090e
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-nonstrict4/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node1", "state": "started", "failback": 0 }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict1/README b/src/test/test-node-affinity-maintenance-strict1/README
new file mode 100644
index 00000000..a31be5db
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/README
@@ -0,0 +1,3 @@
+Test whether an HA resource with failback enabled in a strict node affinity
+rule with a single node member will stay on the current node even though it is
+in maintenance mode, because it cannot find any replacement node.
diff --git a/src/test/test-node-affinity-maintenance-strict1/cmdlist b/src/test/test-node-affinity-maintenance-strict1/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-strict1/hardware_status b/src/test/test-node-affinity-maintenance-strict1/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict1/log.expect b/src/test/test-node-affinity-maintenance-strict1/log.expect
new file mode 100644
index 00000000..e8ec162d
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/log.expect
@@ -0,0 +1,36 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn 140 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 160 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 180 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 200 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+warn 220 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: service 'vm:101': clearing stale maintenance node 'node3' setting (is current node)
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-strict1/manager_status b/src/test/test-node-affinity-maintenance-strict1/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-strict1/rules_config b/src/test/test-node-affinity-maintenance-strict1/rules_config
new file mode 100644
index 00000000..25aa655f
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/rules_config
@@ -0,0 +1,4 @@
+node-affinity: vm101-must-be-on-node3
+ nodes node3
+ resources vm:101
+ strict 1
diff --git a/src/test/test-node-affinity-maintenance-strict1/service_config b/src/test/test-node-affinity-maintenance-strict1/service_config
new file mode 100644
index 00000000..7f0b1bf9
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict1/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict2/README b/src/test/test-node-affinity-maintenance-strict2/README
new file mode 100644
index 00000000..8a7f768d
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/README
@@ -0,0 +1,3 @@
+Test whether an HA resource with failback disabled in a strict node affinity
+rule with a single node member will stay on the current node even though it is
+in maintenance mode, because it cannot find any replacement node.
diff --git a/src/test/test-node-affinity-maintenance-strict2/cmdlist b/src/test/test-node-affinity-maintenance-strict2/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-strict2/hardware_status b/src/test/test-node-affinity-maintenance-strict2/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict2/log.expect b/src/test/test-node-affinity-maintenance-strict2/log.expect
new file mode 100644
index 00000000..e8ec162d
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/log.expect
@@ -0,0 +1,36 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn 140 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 160 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 180 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+warn 200 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+warn 220 node1/crm: service 'vm:101': cannot find a replacement node while its current node is in maintenance
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: service 'vm:101': clearing stale maintenance node 'node3' setting (is current node)
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-strict2/manager_status b/src/test/test-node-affinity-maintenance-strict2/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-strict2/rules_config b/src/test/test-node-affinity-maintenance-strict2/rules_config
new file mode 100644
index 00000000..25aa655f
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/rules_config
@@ -0,0 +1,4 @@
+node-affinity: vm101-must-be-on-node3
+ nodes node3
+ resources vm:101
+ strict 1
diff --git a/src/test/test-node-affinity-maintenance-strict2/service_config b/src/test/test-node-affinity-maintenance-strict2/service_config
new file mode 100644
index 00000000..c7266eec
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict2/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started", "failback": 0 }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict3/README b/src/test/test-node-affinity-maintenance-strict3/README
new file mode 100644
index 00000000..b5f5dfbb
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/README
@@ -0,0 +1,4 @@
+Test whether an HA resource with failback enabled in a strict node affinity
+rule with two differently prioritized node members will move to the
+lower-priority node if its current node is in maintenance mode and moves back
+to the previous maintenance node as soon as it's available again.
diff --git a/src/test/test-node-affinity-maintenance-strict3/cmdlist b/src/test/test-node-affinity-maintenance-strict3/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-strict3/hardware_status b/src/test/test-node-affinity-maintenance-strict3/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict3/log.expect b/src/test/test-node-affinity-maintenance-strict3/log.expect
new file mode 100644
index 00000000..0bdf4fa0
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/log.expect
@@ -0,0 +1,48 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node2' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 143 node2/lrm: got lock 'ha_agent_node2_lock'
+info 143 node2/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:101 - start migrate to node 'node2'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node2'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 163 node2/lrm: starting service vm:101
+info 163 node2/lrm: service status vm:101 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:101' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node3)
+info 243 node2/lrm: service vm:101 - start migrate to node 'node3'
+info 243 node2/lrm: service vm:101 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:101
+info 265 node3/lrm: service status vm:101 started
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-strict3/manager_status b/src/test/test-node-affinity-maintenance-strict3/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-strict3/rules_config b/src/test/test-node-affinity-maintenance-strict3/rules_config
new file mode 100644
index 00000000..12539b76
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/rules_config
@@ -0,0 +1,4 @@
+node-affinity: vm101-must-be-on-node3-or-node2
+ nodes node2:1,node3:2
+ resources vm:101
+ strict 1
diff --git a/src/test/test-node-affinity-maintenance-strict3/service_config b/src/test/test-node-affinity-maintenance-strict3/service_config
new file mode 100644
index 00000000..7f0b1bf9
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict3/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict4/README b/src/test/test-node-affinity-maintenance-strict4/README
new file mode 100644
index 00000000..43c68463
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/README
@@ -0,0 +1,3 @@
+Test whether an HA resource with failback disabled in a strict node affinity
+rule with two differently prioritized node members will move to the
+lower-priority node if its current node is in maintenance mode.
diff --git a/src/test/test-node-affinity-maintenance-strict4/cmdlist b/src/test/test-node-affinity-maintenance-strict4/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-node-affinity-maintenance-strict4/hardware_status b/src/test/test-node-affinity-maintenance-strict4/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-node-affinity-maintenance-strict4/log.expect b/src/test/test-node-affinity-maintenance-strict4/log.expect
new file mode 100644
index 00000000..6f19258c
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/log.expect
@@ -0,0 +1,40 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node2' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node2)
+info 143 node2/lrm: got lock 'ha_agent_node2_lock'
+info 143 node2/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:101 - start migrate to node 'node2'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node2'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 163 node2/lrm: starting service vm:101
+info 163 node2/lrm: service status vm:101 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-strict4/manager_status b/src/test/test-node-affinity-maintenance-strict4/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-node-affinity-maintenance-strict4/rules_config b/src/test/test-node-affinity-maintenance-strict4/rules_config
new file mode 100644
index 00000000..12539b76
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/rules_config
@@ -0,0 +1,4 @@
+node-affinity: vm101-must-be-on-node3-or-node2
+ nodes node2:1,node3:2
+ resources vm:101
+ strict 1
diff --git a/src/test/test-node-affinity-maintenance-strict4/service_config b/src/test/test-node-affinity-maintenance-strict4/service_config
new file mode 100644
index 00000000..c7266eec
--- /dev/null
+++ b/src/test/test-node-affinity-maintenance-strict4/service_config
@@ -0,0 +1,3 @@
+{
+ "vm:101": { "node": "node3", "state": "started", "failback": 0 }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 3/7] test: add test cases for resource affinity rules with maintenance mode
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 2/7] test: add test cases for node affinity rules with maintenance mode Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 4/7] manager: apply negative resource affinity before get_node_affinity Daniel Kral
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
These test cases document how the HA Manager currently behaves for
positive and negative resource affinity rules as well as resource
affinity rules with node affinity rules mixed, whose relevant nodes are
put in maintenance mode and available afterwards again.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
-
.../README | 5 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 54 +++++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 4 ++
.../README | 4 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 48 +++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 5 ++
.../README | 3 ++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 51 ++++++++++++++++++
.../manager_status | 1 +
.../rules_config | 3 ++
.../service_config | 4 ++
.../README | 9 ++++
.../cmdlist | 4 ++
.../hardware_status | 5 ++
.../log.expect | 46 ++++++++++++++++
.../manager_status | 34 ++++++++++++
.../rules_config | 3 ++
.../service_config | 5 ++
.../README | 8 +++
.../cmdlist | 5 ++
.../hardware_status | 5 ++
.../log.expect | 42 +++++++++++++++
.../manager_status | 1 +
.../rules_config | 7 +++
.../service_config | 4 ++
35 files changed, 398 insertions(+)
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative1/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-negative2/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive1/service_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/README
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/manager_status
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/rules_config
create mode 100644 src/test/test-resource-affinity-maintenance-strict-positive2/service_config
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config
create mode 100644 src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/README b/src/test/test-resource-affinity-maintenance-strict-negative1/README
new file mode 100644
index 00000000..5365ebce
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/README
@@ -0,0 +1,5 @@
+Tests whether a strict negative resource affinity rule among two HA resources
+makes the HA resource, whose current node is in maintenance mode, move to a
+replacement node (different node than the other HA resources' node) and moves
+the HA resource back to its previous maintenance node as soon as it's available
+again.
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist b/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status b/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect b/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect
new file mode 100644
index 00000000..1fc25206
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/log.expect
@@ -0,0 +1,54 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node2'
+info 20 node1/crm: adding new service 'vm:102' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:101
+info 23 node2/lrm: service status vm:101 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:102
+info 25 node3/lrm: service status vm:102 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running)
+info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:102 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:102
+info 161 node1/lrm: service status vm:102 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:102' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:102' to node 'node3' (running)
+info 240 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 241 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:102
+info 265 node3/lrm: service status vm:102 started
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status b/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config b/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config
new file mode 100644
index 00000000..20747760
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/rules_config
@@ -0,0 +1,3 @@
+resource-affinity: lonely-must-vms-be
+ resources vm:101,vm:102
+ affinity negative
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative1/service_config b/src/test/test-resource-affinity-maintenance-strict-negative1/service_config
new file mode 100644
index 00000000..e42e5c79
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative1/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:101": { "node": "node2", "state": "started" },
+ "vm:102": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/README b/src/test/test-resource-affinity-maintenance-strict-negative2/README
new file mode 100644
index 00000000..a2102c2f
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/README
@@ -0,0 +1,4 @@
+Tests whether a strict negative resource affinity rule among three HA resources
+makes the HA resource, whose current node is in maintenance mode, stay on its
+current node, even though it is in maintenance mode, because it cannot find any
+replacement node.
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist b/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist
new file mode 100644
index 00000000..7e577b68
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status b/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect b/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect
new file mode 100644
index 00000000..36949bbe
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/log.expect
@@ -0,0 +1,48 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node2'
+info 20 node1/crm: adding new service 'vm:103' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:102
+info 23 node2/lrm: service status vm:102 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:103
+info 25 node3/lrm: service status vm:103 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn 140 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
+warn 160 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
+warn 180 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
+warn 200 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+warn 220 node1/crm: service 'vm:103': cannot find a replacement node while its current node is in maintenance
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: service 'vm:103': clearing stale maintenance node 'node3' setting (is current node)
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status b/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config b/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config
new file mode 100644
index 00000000..44e6a02e
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/rules_config
@@ -0,0 +1,3 @@
+resource-affinity: lonely-must-vms-be
+ resources vm:101,vm:102,vm:103
+ affinity negative
diff --git a/src/test/test-resource-affinity-maintenance-strict-negative2/service_config b/src/test/test-resource-affinity-maintenance-strict-negative2/service_config
new file mode 100644
index 00000000..4b26f6b4
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-negative2/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node2", "state": "started" },
+ "vm:103": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/README b/src/test/test-resource-affinity-maintenance-strict-positive1/README
new file mode 100644
index 00000000..4b62e578
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/README
@@ -0,0 +1,3 @@
+Tests whether a strict positive resource affinity rule among two HA resources
+makes both HA resources move to the same replacement node in case their
+current, common node is put in maintenance mode.
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist b/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist
new file mode 100644
index 00000000..97fbc1ef
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status b/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
new file mode 100644
index 00000000..5f91b877
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
@@ -0,0 +1,51 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node3'
+info 20 node1/crm: adding new service 'vm:102' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node3)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:101
+info 25 node3/lrm: service status vm:101 started
+info 25 node3/lrm: starting service vm:102
+info 25 node3/lrm: service status vm:102 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 140 node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running)
+info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:101 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:101
+info 161 node1/lrm: service status vm:101 started
+info 161 node1/lrm: starting service vm:102
+info 161 node1/lrm: service status vm:102 started
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status b/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config b/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config
new file mode 100644
index 00000000..9789d7cc
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/rules_config
@@ -0,0 +1,3 @@
+resource-affinity: vms-must-stick-together
+ resources vm:101,vm:102
+ affinity positive
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/service_config b/src/test/test-resource-affinity-maintenance-strict-positive1/service_config
new file mode 100644
index 00000000..50ef1caa
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:101": { "node": "node3", "state": "started" },
+ "vm:102": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/README b/src/test/test-resource-affinity-maintenance-strict-positive2/README
new file mode 100644
index 00000000..32f0942b
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/README
@@ -0,0 +1,9 @@
+Tests whether a strict positive resource affinity rule among three HA
+resources, where two of them are already on a common node but the other HA
+resource is still on another node, makes the former two HA resources move to
+the node of the other HA resource as their current common node is put in
+maintenance mode.
+
+The "skip-round crm 1" command ensures that the HA Manager will not move the
+dislocated, third HA resource to the common node, but make the LRM acknowledge
+its maintenance mode request.
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist b/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist
new file mode 100644
index 00000000..2185ee6e
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on", "skip-round crm 1" ],
+ [ "crm node1 disable-node-maintenance" ]
+]
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status b/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
new file mode 100644
index 00000000..ef63c8ca
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
@@ -0,0 +1,46 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute skip-round crm 1
+info 20 run-loop: skipping CRM round
+info 20 node1/lrm: got lock 'ha_agent_node1_lock'
+info 20 node1/lrm: status change wait_for_agent_lock => active
+info 20 node1/lrm: starting service vm:101
+info 20 node1/lrm: service status vm:101 started
+info 20 node1/lrm: starting service vm:102
+info 20 node1/lrm: service status vm:102 started
+info 22 node3/lrm: got lock 'ha_agent_node3_lock'
+info 22 node3/lrm: status change wait_for_agent_lock => active
+info 22 node3/lrm: starting service vm:103
+info 22 node3/lrm: service status vm:103 started
+info 40 node1/crm: got lock 'ha_manager_lock'
+info 40 node1/crm: status change wait_for_quorum => master
+info 40 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 40 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 40 node1/crm: migrate service 'vm:102' to node 'node3' (running)
+info 40 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 41 node1/lrm: status change active => maintenance
+info 41 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 41 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 41 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 41 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 42 node2/crm: status change wait_for_quorum => slave
+info 44 node3/crm: status change wait_for_quorum => slave
+info 60 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 60 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 65 node3/lrm: starting service vm:101
+info 65 node3/lrm: service status vm:101 started
+info 65 node3/lrm: starting service vm:102
+info 65 node3/lrm: service status vm:102 started
+info 120 cmdlist: execute crm node1 disable-node-maintenance
+info 121 node1/lrm: got lock 'ha_agent_node1_lock'
+info 121 node1/lrm: status change maintenance => active
+info 140 node1/crm: node 'node1': state changed from 'maintenance' => 'online'
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status b/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status
new file mode 100644
index 00000000..135a1d6f
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/manager_status
@@ -0,0 +1,34 @@
+{
+ "master_node": "node1",
+ "node_request": {
+ "node1": {
+ "maintenance": 1
+ }
+ },
+ "node_status": {
+ "node1": "maintenance",
+ "node2": "online",
+ "node3": "online"
+ },
+ "service_status": {
+ "vm:101": {
+ "running": 1,
+ "node": "node1",
+ "state": "started",
+ "uid": "Xi3T+eaBD4iaN01s65D5/g"
+ },
+ "vm:102": {
+ "running": 1,
+ "node": "node1",
+ "state": "started",
+ "uid": "F2xctkwVsaF2KY9gJYsz6g"
+ },
+ "vm:103": {
+ "running": 1,
+ "node": "node3",
+ "state": "started",
+ "uid": "c5yeDFKYkhMe3Nv+XzmN0A"
+ }
+ },
+ "timestamp": 40
+}
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config b/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config
new file mode 100644
index 00000000..12da6e67
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/rules_config
@@ -0,0 +1,3 @@
+resource-affinity: vms-must-stick-together
+ resources vm:101,vm:102,vm:103
+ affinity positive
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/service_config b/src/test/test-resource-affinity-maintenance-strict-positive2/service_config
new file mode 100644
index 00000000..32e61c84
--- /dev/null
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
new file mode 100644
index 00000000..c6a11cec
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
@@ -0,0 +1,8 @@
+2 HA resources on a 3-node cluster, which are:
+
+- on node2 and node3 respectively,
+- in a non-strict node affinity rule to node2 and node3 (equal priority), and
+- in a strict negative resource affinity rule with each other.
+
+Tests whether the HA resource on node3 will stay there, even though node3 is
+put in maintenance mode, because it cannot find any replacement node.
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist
new file mode 100644
index 00000000..97fbc1ef
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "crm node3 enable-node-maintenance" ],
+ [ "crm node3 disable-node-maintenance" ]
+]
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status
new file mode 100644
index 00000000..451beb13
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
new file mode 100644
index 00000000..5c7aa99c
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
@@ -0,0 +1,42 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'vm:101' on node 'node2'
+info 20 node1/crm: adding new service 'vm:102' on node 'node3'
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node2)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node3)
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 23 node2/lrm: starting service vm:101
+info 23 node2/lrm: service status vm:101 started
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service vm:102
+info 25 node3/lrm: service status vm:102 started
+info 120 cmdlist: execute crm node3 enable-node-maintenance
+info 125 node3/lrm: status change active => maintenance
+info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+warn 140 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+warn 160 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+warn 180 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+warn 200 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+info 220 cmdlist: execute crm node3 disable-node-maintenance
+warn 220 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+info 225 node3/lrm: status change maintenance => active
+info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: service 'vm:102': clearing stale maintenance node 'node3' setting (is current node)
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status
new file mode 100644
index 00000000..0967ef42
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config
new file mode 100644
index 00000000..e5bf3e47
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/rules_config
@@ -0,0 +1,7 @@
+node-affinity: vm101-vm102-should-be-on-node2-node3
+ resources vm:101,vm:102
+ nodes node2,node3
+
+resource-affinity: lonely-must-vms-be
+ resources vm:101,vm:102
+ affinity negative
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config
new file mode 100644
index 00000000..e42e5c79
--- /dev/null
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:101": { "node": "node2", "state": "started" },
+ "vm:102": { "node": "node3", "state": "started" }
+}
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 4/7] manager: apply negative resource affinity before get_node_affinity
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
` (2 preceding siblings ...)
2026-11-06 9:56 ` [PATCH ha-manager v2 3/7] test: add test cases for resource " Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 5/7] manager: make HA resource bundles move back to maintenance node Daniel Kral
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
When selecting possible HA resource node placements, get_node_affinity()
in select_service_node() and get_resource_migration_candidates() is
responsible to determine the highest node priority class with at least
one available node and the allowed set from an input node set, which
must only contain available nodes to the specific HA resource.
A necessary condition for a node to be considered as available to an HA
resource is whether the node is in 'online' state (i.e., not 'unknown',
'maintenance', 'fence', nor 'gone').
However, negative resource affinity rules introduced a new condition,
where nodes become unavailable to HA resources: another HA resource,
with which the HA resource must not share the node with, is already
running on the node.
Currently, the negative resource affinity rules are applied on the
highest node priority class selected by get_node_affinity(). This can
result in an empty set if all the remaining nodes in the highest node
priority class cannot be selected because of negative resource affinity
constraints.
Therefore, apply the negative resource affinity rules on the available
node set already, so get_node_affinity() will only select node priority
classes, which meet both conditions mentioned above.
It should be noted that the HA stack currently does not allow an HA
resource to be referenced in both resource affinity rules and explicit
multi-priority node affinity rules. However, non-strict node affinity
rules implicitly create multi-priority node affinity rules, because
there the member nodes are in the highest node priority class, while the
non-member nodes are in the least node priority class.
As seen in "test-resource-affinity-with-node-affinity-strict-negative3",
this makes the HA Manager move HA resources away from their preferred
node, if an HA resource, that it must not share the node with, starts
there. This is valid with the semantics of non-strict node affinity
rules. Nonetheless, handling negative affinity rules in these scenarios
should be improved in the future, e.g., by moving the starting HA
resource to another available node in request_start already.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- move this patch up first
- instead of looping through the node priority classes and effectively
letting apply_negative_resource_affinity() decide the node priority
class itself, remove them in $online_nodes already beforehand; this
results in the same behavior and is easier to read and think about
- adapt comment accordingly
src/PVE/HA/Manager.pm | 20 ++++++++++++----
.../README | 5 ++--
.../log.expect | 24 ++++++++++++++-----
.../README | 7 +++---
.../log.expect | 16 +++++++------
5 files changed, 49 insertions(+), 23 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 753da1a8..c38b1ddd 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -206,10 +206,15 @@ sub get_resource_migration_candidates {
my $current_leader_node = $ss->{$leader_sid}->{node};
my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
- my (undef, $target_nodes) = get_node_affinity($node_affinity, $leader_sid, $online_nodes);
my ($together, $separate) =
get_resource_affinity($resource_affinity, $leader_sid, $ss, $online_nodes);
- apply_negative_resource_affinity($separate, $target_nodes);
+
+ # Remove nodes with active HA resources that must not run on the same node as $leader_sid
+ # here already, so get_node_affinity() acknowledges these nodes as unavailable to
+ # $leader_sid when determining the highest available node priority class.
+ apply_negative_resource_affinity($separate, $online_nodes);
+
+ my (undef, $target_nodes) = get_node_affinity($node_affinity, $leader_sid, $online_nodes);
delete $target_nodes->{$current_leader_node};
@@ -352,12 +357,18 @@ sub select_service_node {
$compiled_rules->@{qw(node-affinity resource-affinity)};
my $online_nodes = { map { $_ => 1 } $online_node_usage->list_nodes() };
+
+ my ($together, $separate) = get_resource_affinity($resource_affinity, $sid, $ss, $online_nodes);
+
+ # Remove nodes with active HA resources that must not run on the same node as $sid
+ # here already, so get_node_affinity() acknowledges these nodes as unavailable to
+ # $sid when determining the highest available node priority class.
+ apply_negative_resource_affinity($separate, $online_nodes);
+
my ($allowed_nodes, $pri_nodes) = get_node_affinity($node_affinity, $sid, $online_nodes);
return undef if !%$pri_nodes;
- my ($together, $separate) = get_resource_affinity($resource_affinity, $sid, $ss, $online_nodes);
-
# stay on current node if possible (avoids random migrations)
if (
$node_preference eq 'none'
@@ -377,7 +388,6 @@ sub select_service_node {
}
}
- apply_negative_resource_affinity($separate, $pri_nodes);
apply_positive_resource_affinity($together, $pri_nodes);
return $maintenance_fallback
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
index c6a11cec..e1fc0d04 100644
--- a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/README
@@ -4,5 +4,6 @@
- in a non-strict node affinity rule to node2 and node3 (equal priority), and
- in a strict negative resource affinity rule with each other.
-Tests whether the HA resource on node3 will stay there, even though node3 is
-put in maintenance mode, because it cannot find any replacement node.
+Tests whether the HA resource on node3 will correctly move to a replacement
+node, which is different from the node of the other HA resource (node2), and
+moves back to its previous maintenance node as soon as it's available again.
diff --git a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
index 5c7aa99c..1fc25206 100644
--- a/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
+++ b/src/test/test-resource-affinity-with-node-affinity-maintenance-strict-negative1/log.expect
@@ -30,13 +30,25 @@ info 25 node3/lrm: service status vm:102 started
info 120 cmdlist: execute crm node3 enable-node-maintenance
info 125 node3/lrm: status change active => maintenance
info 140 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
-warn 140 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
-warn 160 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
-warn 180 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
-warn 200 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running)
+info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: service vm:102 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:102
+info 161 node1/lrm: service status vm:102 started
info 220 cmdlist: execute crm node3 disable-node-maintenance
-warn 220 node1/crm: service 'vm:102': cannot find a replacement node while its current node is in maintenance
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
info 225 node3/lrm: status change maintenance => active
info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
-info 240 node1/crm: service 'vm:102': clearing stale maintenance node 'node3' setting (is current node)
+info 240 node1/crm: moving service 'vm:102' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:102' to node 'node3' (running)
+info 240 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 241 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:102
+info 265 node3/lrm: service status vm:102 started
info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-with-node-affinity-strict-negative3/README b/src/test/test-resource-affinity-with-node-affinity-strict-negative3/README
index 062fc665..581b0e9a 100644
--- a/src/test/test-resource-affinity-with-node-affinity-strict-negative3/README
+++ b/src/test/test-resource-affinity-with-node-affinity-strict-negative3/README
@@ -1,7 +1,8 @@
Test whether a strict negative resource affinity rule among three resources,
-where two resources are restricted each to nodes they are not yet on, can be
-exchanged to the nodes described by their node affinity rules, if one of the
-resources is stopped.
+where all resources are restricted each to nodes they are not yet on, can be
+exchanged to the nodes described by their node affinity rules or fallback to
+another valid configuration within the semantics of non-strict node affinity
+rules, if one of the resources is stopped.
The test scenario is:
- vm:101, vm:102, and vm:103 should be on node2, node3 or node1 respectively
diff --git a/src/test/test-resource-affinity-with-node-affinity-strict-negative3/log.expect b/src/test/test-resource-affinity-with-node-affinity-strict-negative3/log.expect
index 1ed34c36..66974583 100644
--- a/src/test/test-resource-affinity-with-node-affinity-strict-negative3/log.expect
+++ b/src/test/test-resource-affinity-with-node-affinity-strict-negative3/log.expect
@@ -57,11 +57,13 @@ info 285 node3/lrm: service status vm:102 started
info 320 cmdlist: execute service vm:101 started
info 320 node1/crm: service 'vm:101': state changed from 'stopped' to 'request_start' (node = node1)
info 320 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
-info 320 node1/crm: migrate service 'vm:101' to node 'node2' (running)
-info 320 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
-info 321 node1/lrm: service vm:101 - start migrate to node 'node2'
-info 321 node1/lrm: service vm:101 - end migrate to node 'node2'
-info 340 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
-info 343 node2/lrm: starting service vm:101
-info 343 node2/lrm: service status vm:101 started
+info 320 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 320 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 321 node1/lrm: starting service vm:101
+info 321 node1/lrm: service status vm:101 started
+info 321 node1/lrm: service vm:103 - start migrate to node 'node2'
+info 321 node1/lrm: service vm:103 - end migrate to node 'node2'
+info 340 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node2)
+info 343 node2/lrm: starting service vm:103
+info 343 node2/lrm: service status vm:103 started
info 920 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 5/7] manager: make HA resource bundles move back to maintenance node
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
` (3 preceding siblings ...)
2026-11-06 9:56 ` [PATCH ha-manager v2 4/7] manager: apply negative resource affinity before get_node_affinity Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 6/7] manager: remove failed nodes earlier in select_service_node Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 7/7] manager: move HA resources without failback back to maintenance node Daniel Kral
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
HA resources in HA resource bundles always prefer their current, common
node as soon as at least one of their HA resources is actively assigned
to a node already. This is done by apply_positive_resource_affinity(),
which will reduce the node set to only this current, common node.
However, except the HA resource bundles' maintenance node is their
current, common node (e.g. no replacement node could be found), these
will never fallback to the maintenance node if this reduction is made
before the fallback logic.
Therefore, move the maintenance fallback before
apply_positive_resource_affinity() to make HA resources with the
maintenance node set fallback to their maintenance node, which is likely
the maintenance node of the other HA resources in the HA resource bundle
as well.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- move patch up
src/PVE/HA/Manager.pm | 4 ++--
.../README | 3 ++-
.../log.expect | 16 +++++++++++++
.../README | 3 ++-
.../log.expect | 23 +++++++++++++++++++
5 files changed, 45 insertions(+), 4 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index c38b1ddd..3c768a86 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -388,11 +388,11 @@ sub select_service_node {
}
}
- apply_positive_resource_affinity($together, $pri_nodes);
-
return $maintenance_fallback
if defined($maintenance_fallback) && $pri_nodes->{$maintenance_fallback};
+ apply_positive_resource_affinity($together, $pri_nodes);
+
return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
my $scores = $online_node_usage->score_nodes_to_start_service($sid, $current_node);
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/README b/src/test/test-resource-affinity-maintenance-strict-positive1/README
index 4b62e578..ab293cc5 100644
--- a/src/test/test-resource-affinity-maintenance-strict-positive1/README
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/README
@@ -1,3 +1,4 @@
Tests whether a strict positive resource affinity rule among two HA resources
makes both HA resources move to the same replacement node in case their
-current, common node is put in maintenance mode.
+current, common node is put in maintenance mode and moves them back as the
+previous maintenance node is available again.
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
index 5f91b877..91637279 100644
--- a/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
+++ b/src/test/test-resource-affinity-maintenance-strict-positive1/log.expect
@@ -48,4 +48,20 @@ info 220 cmdlist: execute crm node3 disable-node-maintenance
info 225 node3/lrm: got lock 'ha_agent_node3_lock'
info 225 node3/lrm: status change maintenance => active
info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:101' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 240 node1/crm: moving service 'vm:102' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:102' to node 'node3' (running)
+info 240 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 241 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 241 node1/lrm: service vm:102 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:102 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 260 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:101
+info 265 node3/lrm: service status vm:101 started
+info 265 node3/lrm: starting service vm:102
+info 265 node3/lrm: service status vm:102 started
info 820 hardware: exit simulation - done
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/README b/src/test/test-resource-affinity-maintenance-strict-positive2/README
index 32f0942b..dcc4c81d 100644
--- a/src/test/test-resource-affinity-maintenance-strict-positive2/README
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/README
@@ -2,7 +2,8 @@ Tests whether a strict positive resource affinity rule among three HA
resources, where two of them are already on a common node but the other HA
resource is still on another node, makes the former two HA resources move to
the node of the other HA resource as their current common node is put in
-maintenance mode.
+maintenance mode and moves them back as soon as the previous maintenance node
+is available again.
The "skip-round crm 1" command ensures that the HA Manager will not move the
dislocated, third HA resource to the common node, but make the LRM acknowledge
diff --git a/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
index ef63c8ca..9da6d968 100644
--- a/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
+++ b/src/test/test-resource-affinity-maintenance-strict-positive2/log.expect
@@ -43,4 +43,27 @@ info 120 cmdlist: execute crm node1 disable-node-maintenance
info 121 node1/lrm: got lock 'ha_agent_node1_lock'
info 121 node1/lrm: status change maintenance => active
info 140 node1/crm: node 'node1': state changed from 'maintenance' => 'online'
+info 140 node1/crm: moving service 'vm:101' back to 'node1', node came back from maintenance.
+info 140 node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 140 node1/crm: moving service 'vm:102' back to 'node1', node came back from maintenance.
+info 140 node1/crm: migrate service 'vm:102' to node 'node1' (running)
+info 140 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 140 node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info 140 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 145 node3/lrm: service vm:101 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:101 - end migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:102 - end migrate to node 'node1'
+info 145 node3/lrm: service vm:103 - start migrate to node 'node1'
+info 145 node3/lrm: service vm:103 - end migrate to node 'node1'
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 160 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node1)
+info 160 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 161 node1/lrm: starting service vm:101
+info 161 node1/lrm: service status vm:101 started
+info 161 node1/lrm: starting service vm:102
+info 161 node1/lrm: service status vm:102 started
+info 161 node1/lrm: starting service vm:103
+info 161 node1/lrm: service status vm:103 started
info 720 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 6/7] manager: remove failed nodes earlier in select_service_node
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
` (4 preceding siblings ...)
2026-11-06 9:56 ` [PATCH ha-manager v2 5/7] manager: make HA resource bundles move back to maintenance node Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 7/7] manager: move HA resources without failback back to maintenance node Daniel Kral
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
This is done in preparation of an upcoming patch, which moves the
maintenance node fallback behavior logic up as well, so move up the
failed nodes earlier here as well to maintain it.
This does not change the behavior of select_service_node(), because the
now subsequent random migrations check is only done if the
$node_preference = 'none', while the failed nodes are only removed in
$node_preference = 'try-next'.
Suggested-by: Michael Köppl <m.koeppl@proxmox.com>
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- new!
src/PVE/HA/Manager.pm | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 3c768a86..b2b74282 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -367,6 +367,13 @@ sub select_service_node {
my ($allowed_nodes, $pri_nodes) = get_node_affinity($node_affinity, $sid, $online_nodes);
+ # try to avoid nodes where the HA resource failed already if we want to relocate
+ if ($node_preference eq 'try-next') {
+ for my $node (@$tried_nodes) {
+ delete $pri_nodes->{$node};
+ }
+ }
+
return undef if !%$pri_nodes;
# stay on current node if possible (avoids random migrations)
@@ -381,13 +388,6 @@ sub select_service_node {
return $current_node;
}
- # try to avoid nodes where the service failed already if we want to relocate
- if ($node_preference eq 'try-next') {
- foreach my $node (@$tried_nodes) {
- delete $pri_nodes->{$node};
- }
- }
-
return $maintenance_fallback
if defined($maintenance_fallback) && $pri_nodes->{$maintenance_fallback};
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [PATCH ha-manager v2 7/7] manager: move HA resources without failback back to maintenance node
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
` (5 preceding siblings ...)
2026-11-06 9:56 ` [PATCH ha-manager v2 6/7] manager: remove failed nodes earlier in select_service_node Daniel Kral
@ 2026-11-06 9:56 ` Daniel Kral
6 siblings, 0 replies; 8+ messages in thread
From: Daniel Kral @ 2026-11-06 9:56 UTC (permalink / raw)
To: pve-devel
If the current node of an HA resource with failback disabled is put in
maintenance mode, the HA resource will correctly move to a replacement
node. Though as the previous node is put out of maintenance mode again,
the HA resource will stay on the new node, because of the disabled
failback flag.
As HA resources should move back to their previous maintenance node
independently of the failback flag, make the HA resource move to the
previous maintenance node as soon as it is available again.
This does not move HA resources without failback back to a maintenance
node that is not part of the highest node priority class. This behavior
is still up for debate and documented with relevant test cases.
Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
changes since v1:
- different patch now, but has the same behavior
- the maintenance fallback is moved up here now too as its easier to
think and reason about than the previous patch
- this also prevents the weird case where the "avoids random migrations"
is skipped if there is a maintenance_fallback set and isn't the
$current_node, making failbacks happen even though failback is cleared
for $sid
src/PVE/HA/Manager.pm | 6 +++---
src/test/test-node-affinity-maintenance-nonstrict2/README | 3 ++-
.../test-node-affinity-maintenance-nonstrict2/log.expect | 8 ++++++++
src/test/test-node-affinity-maintenance-strict4/README | 3 ++-
.../test-node-affinity-maintenance-strict4/log.expect | 8 ++++++++
5 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index b2b74282..2ce7e0b1 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -376,6 +376,9 @@ sub select_service_node {
return undef if !%$pri_nodes;
+ return $maintenance_fallback
+ if defined($maintenance_fallback) && $pri_nodes->{$maintenance_fallback};
+
# stay on current node if possible (avoids random migrations)
if (
$node_preference eq 'none'
@@ -388,9 +391,6 @@ sub select_service_node {
return $current_node;
}
- return $maintenance_fallback
- if defined($maintenance_fallback) && $pri_nodes->{$maintenance_fallback};
-
apply_positive_resource_affinity($together, $pri_nodes);
return $current_node if $node_preference eq 'none' && $pri_nodes->{$current_node};
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/README b/src/test/test-node-affinity-maintenance-nonstrict2/README
index aa65a55a..1958154f 100644
--- a/src/test/test-node-affinity-maintenance-nonstrict2/README
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/README
@@ -1,3 +1,4 @@
Test whether an HA resource with failback disabled on the single member node of
its non-strict node affinity rule with a single node member will move to a
-replacement node if its current node is in maintenance mode.
+replacement node if its current node is in maintenance mode and moves back to
+the previous maintenance node as soon as it's available again.
diff --git a/src/test/test-node-affinity-maintenance-nonstrict2/log.expect b/src/test/test-node-affinity-maintenance-nonstrict2/log.expect
index 05a77a24..339ce3ab 100644
--- a/src/test/test-node-affinity-maintenance-nonstrict2/log.expect
+++ b/src/test/test-node-affinity-maintenance-nonstrict2/log.expect
@@ -37,4 +37,12 @@ info 220 cmdlist: execute crm node3 disable-node-maintenance
info 225 node3/lrm: got lock 'ha_agent_node3_lock'
info 225 node3/lrm: status change maintenance => active
info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:101' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 241 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 241 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:101
+info 265 node3/lrm: service status vm:101 started
info 820 hardware: exit simulation - done
diff --git a/src/test/test-node-affinity-maintenance-strict4/README b/src/test/test-node-affinity-maintenance-strict4/README
index 43c68463..e6ad5c7e 100644
--- a/src/test/test-node-affinity-maintenance-strict4/README
+++ b/src/test/test-node-affinity-maintenance-strict4/README
@@ -1,3 +1,4 @@
Test whether an HA resource with failback disabled in a strict node affinity
rule with two differently prioritized node members will move to the
-lower-priority node if its current node is in maintenance mode.
+lower-priority node if its current node is in maintenance mode and moves back
+to the previous maintenance node as soon as it's available again.
diff --git a/src/test/test-node-affinity-maintenance-strict4/log.expect b/src/test/test-node-affinity-maintenance-strict4/log.expect
index 6f19258c..0bdf4fa0 100644
--- a/src/test/test-node-affinity-maintenance-strict4/log.expect
+++ b/src/test/test-node-affinity-maintenance-strict4/log.expect
@@ -37,4 +37,12 @@ info 220 cmdlist: execute crm node3 disable-node-maintenance
info 225 node3/lrm: got lock 'ha_agent_node3_lock'
info 225 node3/lrm: status change maintenance => active
info 240 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 240 node1/crm: moving service 'vm:101' back to 'node3', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node3)
+info 243 node2/lrm: service vm:101 - start migrate to node 'node3'
+info 243 node2/lrm: service vm:101 - end migrate to node 'node3'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 265 node3/lrm: starting service vm:101
+info 265 node3/lrm: service status vm:101 started
info 820 hardware: exit simulation - done
--
2.47.3
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-06 9:58 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-06 9:56 [PATCH-SERIES ha-manager v2 0/7] improve handling of maintenance nodes Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 1/7] manager: warn if HA resources cannot be moved away from maintenance node Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 2/7] test: add test cases for node affinity rules with maintenance mode Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 3/7] test: add test cases for resource " Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 4/7] manager: apply negative resource affinity before get_node_affinity Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 5/7] manager: make HA resource bundles move back to maintenance node Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 6/7] manager: remove failed nodes earlier in select_service_node Daniel Kral
2026-11-06 9:56 ` [PATCH ha-manager v2 7/7] manager: move HA resources without failback back to maintenance node Daniel Kral
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox