From: Michael Ryom <Michael@RyomHerold.dk>
To: pve-devel@lists.proxmox.com
Cc: Michael Ryom <Michael@RyomHerold.dk>
Subject: [PATCH ha-manager v2 03/14] test: add auto rebalance test cases for failing rebalance motions
Date: Sun, 18 Oct 2026 14:45:47 +0200 [thread overview]
Message-ID: <20261018124559.24305-4-Michael@RyomHerold.dk> (raw)
In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk>
The automatic load balancer does not record the outcome of the motions
it issues. If a motion fails, the resource is put back to 'started' on
its source node and the very same motion is selected again as soon as
the imbalance has exceeded the threshold for the hold duration again
(see bug #8059).
Add two test cases which show this behavior:
- test-crs-dynamic-auto-rebalance5: a failing relocation of fa:109 is
retried every 80 seconds, even though migrating any of the other HA
resources would improve the imbalance just as well.
- test-crs-dynamic-auto-rebalance6: the failing relocation of fa:109 is
the only possible rebalance motion and is retried every 80 seconds as
well.
The expected logs record the current behavior and are adapted by the
following patch.
Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
.../test-crs-dynamic-auto-rebalance5/README | 9 ++
.../test-crs-dynamic-auto-rebalance5/cmdlist | 3 +
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 6 ++
.../hardware_status | 5 +
.../log.expect | 88 +++++++++++++++
.../manager_status | 1 +
.../service_config | 6 ++
.../static_service_stats | 6 ++
.../test-crs-dynamic-auto-rebalance6/README | 10 ++
.../test-crs-dynamic-auto-rebalance6/cmdlist | 17 +++
.../datacenter.cfg | 6 ++
.../dynamic_service_stats | 4 +
.../hardware_status | 4 +
.../log.expect | 100 ++++++++++++++++++
.../manager_status | 1 +
.../service_config | 4 +
.../static_service_stats | 4 +
18 files changed, 280 insertions(+)
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/README
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/cmdlist
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/hardware_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/log.expect
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/manager_status
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/service_config
create mode 100644 src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/README b/src/test/test-crs-dynamic-auto-rebalance5/README
new file mode 100644
index 0000000..018e604
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/README
@@ -0,0 +1,9 @@
+Test that the auto rebalance system does not retry a failing rebalance motion
+over and over again.
+
+The HA resource fa:109 fails its first nine migrate/relocate attempts. The
+auto rebalancer selects fa:109 to be relocated, which fails, and fa:109 is put
+back to 'started' on node1. As long as the imbalance stays above the threshold,
+the auto rebalancer should neither select the very same failed motion again
+nor keep alternating between failing motions of fa:109 to the other nodes, but
+try the next best rebalance motion, i.e. migrate one of the other HA resources.
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/cmdlist b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
new file mode 100644
index 0000000..13f90cd
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/cmdlist
@@ -0,0 +1,3 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
new file mode 100644
index 0000000..921df6a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/dynamic_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:102": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:103": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:104": { "cpu": 1.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/hardware_status b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
new file mode 100644
index 0000000..ce8cf0e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node3": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/log.expect b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
new file mode 100644
index 0000000..a5dad5a
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/log.expect
@@ -0,0 +1,88 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'fa:109' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 20 node1/crm: adding new service 'vm:104' on node 'node1'
+info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service fa:109
+info 21 node1/lrm: service status fa:109 started
+info 21 node1/lrm: starting service vm:102
+info 21 node1/lrm: service status vm:102 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 21 node1/lrm: starting service vm:104
+info 21 node1/lrm: service status vm:104 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 80 node1/crm: got crm command: relocate fa:109 node2
+info 80 node1/crm: relocate service 'fa:109' to node 'node2'
+info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 81 node1/lrm: service fa:109 not moved (migration error)
+info 83 node2/lrm: got lock 'ha_agent_node2_lock'
+info 83 node2/lrm: status change wait_for_agent_lock => active
+err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 160 node1/crm: got crm command: relocate fa:109 node2
+info 160 node1/crm: relocate service 'fa:109' to node 'node2'
+info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 161 node1/lrm: service fa:109 not moved (migration error)
+err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 240 node1/crm: got crm command: relocate fa:109 node2
+info 240 node1/crm: relocate service 'fa:109' to node 'node2'
+info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 241 node1/lrm: service fa:109 not moved (migration error)
+err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 320 node1/crm: got crm command: relocate fa:109 node2
+info 320 node1/crm: relocate service 'fa:109' to node 'node2'
+info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 321 node1/lrm: service fa:109 not moved (migration error)
+err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 400 node1/crm: got crm command: relocate fa:109 node2
+info 400 node1/crm: relocate service 'fa:109' to node 'node2'
+info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 401 node1/lrm: service fa:109 not moved (migration error)
+err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 480 node1/crm: got crm command: relocate fa:109 node2
+info 480 node1/crm: relocate service 'fa:109' to node 'node2'
+info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 481 node1/lrm: service fa:109 not moved (migration error)
+err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 66.1%)
+info 560 node1/crm: got crm command: relocate fa:109 node2
+info 560 node1/crm: relocate service 'fa:109' to node 'node2'
+info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 561 node1/lrm: service fa:109 not moved (migration error)
+err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 620 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/manager_status b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/service_config b/src/test/test-crs-dynamic-auto-rebalance5/service_config
new file mode 100644
index 0000000..3685b14
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/service_config
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node1", "state": "started" },
+ "vm:104": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
new file mode 100644
index 0000000..6dab3d1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance5/static_service_stats
@@ -0,0 +1,6 @@
+{
+ "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:103": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:104": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/README b/src/test/test-crs-dynamic-auto-rebalance6/README
new file mode 100644
index 0000000..76e9411
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/README
@@ -0,0 +1,10 @@
+Test that the auto rebalance system retries a rebalance motion, which failed
+before, less and less often if it keeps failing and there is no other
+rebalance motion to fall back to.
+
+The HA resource vm:100 is excluded from auto rebalancing and only acts as a
+node base load on node1. The only movable HA resource fa:109 fails its first
+nine migrate/relocate attempts, so the auto rebalancer can only ever select
+the failing relocation of fa:109 to node2. The time between two attempts should
+grow with each failure instead of retrying it every time the imbalance has
+exceeded the threshold for the hold duration.
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/cmdlist b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
new file mode 100644
index 0000000..c8c644e
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/cmdlist
@@ -0,0 +1,17 @@
+[
+ [ "power node1 on", "power node2 on" ],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ [],
+ []
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "dynamic",
+ "ha-auto-rebalance": 1
+ }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
new file mode 100644
index 0000000..77ef5c2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/dynamic_service_stats
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "cpu": 1.0, "mem": 4294967296 },
+ "vm:100": { "cpu": 4.0, "mem": 4294967296 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/hardware_status b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
new file mode 100644
index 0000000..864adb3
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/hardware_status
@@ -0,0 +1,4 @@
+{
+ "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/log.expect b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
new file mode 100644
index 0000000..9758af5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/log.expect
@@ -0,0 +1,100 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'dynamic'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'fa:109' on node 'node1'
+info 20 node1/crm: adding new service 'vm:100' on node 'node1'
+info 20 node1/crm: service 'fa:109': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service fa:109
+info 21 node1/lrm: service status fa:109 started
+info 21 node1/lrm: starting service vm:100
+info 21 node1/lrm: service status vm:100 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 80 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 80 node1/crm: got crm command: relocate fa:109 node2
+info 80 node1/crm: relocate service 'fa:109' to node 'node2'
+info 80 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 81 node1/lrm: service fa:109 not moved (migration error)
+info 83 node2/lrm: got lock 'ha_agent_node2_lock'
+info 83 node2/lrm: status change wait_for_agent_lock => active
+err 100 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 100 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 160 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 160 node1/crm: got crm command: relocate fa:109 node2
+info 160 node1/crm: relocate service 'fa:109' to node 'node2'
+info 160 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 161 node1/lrm: service fa:109 not moved (migration error)
+err 180 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 180 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 240 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 240 node1/crm: got crm command: relocate fa:109 node2
+info 240 node1/crm: relocate service 'fa:109' to node 'node2'
+info 240 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 241 node1/lrm: service fa:109 not moved (migration error)
+err 260 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 260 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 320 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 320 node1/crm: got crm command: relocate fa:109 node2
+info 320 node1/crm: relocate service 'fa:109' to node 'node2'
+info 320 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 321 node1/lrm: service fa:109 not moved (migration error)
+err 340 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 340 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 400 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 400 node1/crm: got crm command: relocate fa:109 node2
+info 400 node1/crm: relocate service 'fa:109' to node 'node2'
+info 400 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 401 node1/lrm: service fa:109 not moved (migration error)
+err 420 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 420 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 480 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 480 node1/crm: got crm command: relocate fa:109 node2
+info 480 node1/crm: relocate service 'fa:109' to node 'node2'
+info 480 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 481 node1/lrm: service fa:109 not moved (migration error)
+err 500 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 500 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 560 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 560 node1/crm: got crm command: relocate fa:109 node2
+info 560 node1/crm: relocate service 'fa:109' to node 'node2'
+info 560 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 561 node1/lrm: service fa:109 not moved (migration error)
+err 580 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 580 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 640 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 640 node1/crm: got crm command: relocate fa:109 node2
+info 640 node1/crm: relocate service 'fa:109' to node 'node2'
+info 640 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 641 node1/lrm: service fa:109 not moved (migration error)
+err 660 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 660 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 720 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 720 node1/crm: got crm command: relocate fa:109 node2
+info 720 node1/crm: relocate service 'fa:109' to node 'node2'
+info 720 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 721 node1/lrm: service fa:109 not moved (migration error)
+err 740 node1/crm: service 'fa:109' - migration failed (exit code 1)
+info 740 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node1)
+info 800 node1/crm: auto rebalance - relocate fa:109 to node2 (expected change for imbalance from 100.0% to 27.3%)
+info 800 node1/crm: got crm command: relocate fa:109 node2
+info 800 node1/crm: relocate service 'fa:109' to node 'node2'
+info 800 node1/crm: service 'fa:109': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 801 node1/lrm: service fa:109 - start relocate to node 'node2'
+info 801 node1/lrm: stopping service fa:109 (relocate)
+info 801 node1/lrm: service status fa:109 stopped
+info 801 node1/lrm: service fa:109 - end relocate to node 'node2'
+info 820 node1/crm: service 'fa:109': state changed from 'relocate' to 'started' (node = node2)
+info 823 node2/lrm: starting service fa:109
+info 823 node2/lrm: service status fa:109 started
+info 1520 hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/manager_status b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/service_config b/src/test/test-crs-dynamic-auto-rebalance6/service_config
new file mode 100644
index 0000000..ccffeab
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/service_config
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "node": "node1", "state": "started" },
+ "vm:100": { "node": "node1", "state": "started", "auto-rebalance": 0 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
new file mode 100644
index 0000000..7d2c248
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance6/static_service_stats
@@ -0,0 +1,4 @@
+{
+ "fa:109": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:100": { "maxcpu": 4.0, "maxmem": 8589934592 }
+}
--
2.47.3
next prev parent reply other threads:[~2026-10-18 12:46 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-18 12:45 [PATCH cluster/ha-manager/manager/docs v2 00/14] auto rebalance: fix failure retry loop and oscillation Michael Ryom
2026-10-18 12:45 ` [PATCH cluster v2 01/14] datacenter config: add HA auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 02/14] manager: factor out checking whether a resource motion is queued or running Michael Ryom
2026-10-18 12:45 ` Michael Ryom [this message]
2026-10-18 12:45 ` [PATCH ha-manager v2 04/14] fix #8059: manager: auto rebalance: do not retry failed rebalance motions right away Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 05/14] test: add auto rebalance test case for an immediate reverse motion Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 06/14] manager: auto rebalance: add optional cooldown for moved resources Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 07/14] test: add auto rebalance test case for a marginal absolute improvement Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 08/14] manager: auto rebalance: allow an absolute margin Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 09/14] sim: hardware: report actual running state in cluster service stats Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 10/14] sim: hardware: allow setting a base load for nodes Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 11/14] test: add auto rebalance test case for fluctuating node base loads Michael Ryom
2026-10-18 12:45 ` [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load Michael Ryom
2026-10-18 12:45 ` [PATCH manager v2 13/14] ui: crs options: expose the auto-rebalance margin mode and cooldown Michael Ryom
2026-10-18 12:45 ` [PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations Michael Ryom
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261018124559.24305-4-Michael@RyomHerold.dk \
--to=michael@ryomherold.dk \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox