public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Dominik Rusovac <d.rusovac@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights
Date: Thu, 12 Nov 2026 10:44:37 +0100	[thread overview]
Message-ID: <20261112094437.1951271-20-d.rusovac@proxmox.com> (raw)
In-Reply-To: <20261112094437.1951271-1-d.rusovac@proxmox.com>

add test cases which document that the crs weights steer the automatic
rebalancing towards balancing the CPU load or the memory usage of the
nodes; increasing a weight lets the load imbalance exceed the threshold
for a cluster that is balanced with equal weights; and a weight of 0
balances the other resource only

Signed-off-by: Dominik Rusovac <d.rusovac@proxmox.com>
---

Notes:
    changes since v1:
    * reword commit
    * use the new option names
    * drop the redundant commands setting the weights to 1
    * add test cases with one of the weights set to 0

 .../README                                    |  4 ++
 .../cmdlist                                   |  4 ++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  8 +++
 .../hardware_status                           |  5 ++
 .../log.expect                                | 61 +++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  8 +++
 .../static_service_stats                      |  8 +++
 .../README                                    |  4 ++
 .../cmdlist                                   |  6 ++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  8 +++
 .../hardware_status                           |  5 ++
 .../log.expect                                | 65 +++++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  8 +++
 .../static_service_stats                      |  8 +++
 .../README                                    |  5 ++
 .../cmdlist                                   |  4 ++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  8 +++
 .../hardware_status                           |  5 ++
 .../log.expect                                | 61 +++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  8 +++
 .../static_service_stats                      |  8 +++
 .../README                                    |  5 ++
 .../cmdlist                                   |  4 ++
 .../datacenter.cfg                            |  7 ++
 .../dynamic_service_stats                     |  8 +++
 .../hardware_status                           |  5 ++
 .../log.expect                                | 62 ++++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  8 +++
 .../static_service_stats                      |  8 +++
 .../README                                    |  5 ++
 .../cmdlist                                   |  4 ++
 .../datacenter.cfg                            |  6 ++
 .../dynamic_service_stats                     |  8 +++
 .../hardware_status                           |  5 ++
 .../log.expect                                | 61 +++++++++++++++++
 .../manager_status                            |  1 +
 .../service_config                            |  8 +++
 .../static_service_stats                      |  8 +++
 45 files changed, 536 insertions(+)
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
 create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats

diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
new file mode 100644
index 0000000..05cb5df
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README
@@ -0,0 +1,4 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing ha-cpu-weight
+increases the node imbalance and triggers a rebalancing migration from
+the CPU-heavy node3 to node2, which has a low CPU load and memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
new file mode 100644
index 0000000..869d8e8
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "datacenter crs set ha-cpu-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+    "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+    "vm:103": { "cpu": 2.0, "mem": 8589934592  },
+    "vm:104": { "cpu": 1.0, "mem": 4294967296  },
+    "vm:105": { "cpu": 9.0, "mem": 5368709120  },
+    "vm:106": { "cpu": 6.0, "mem": 3221225472  }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
new file mode 100644
index 0000000..cf770d5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect
@@ -0,0 +1,61 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: using ha-auto-rebalance '1'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node2'
+info     20    node1/crm: adding new service 'vm:104' on node 'node2'
+info     20    node1/crm: adding new service 'vm:105' on node 'node3'
+info     20    node1/crm: adding new service 'vm:106' on node 'node3'
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:105': state changed from 'request_start' to 'started'  (node = node3)
+info     20    node1/crm: service 'vm:106': state changed from 'request_start' to 'started'  (node = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     25    node3/lrm: starting service vm:106
+info     25    node3/lrm: service status vm:106 started
+info    120      cmdlist: execute datacenter crs set ha-cpu-weight 2
+info    120    node1/crm: switching ha-cpu-weight from '1' to '2'
+info    160    node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%)
+info    160    node1/crm: got crm command: migrate vm:106 node2
+info    160    node1/crm: migrate service 'vm:106' to node 'node2'
+info    160    node1/crm: service 'vm:106': state changed from 'started' to 'migrate'  (node = node3, target = node2)
+info    165    node3/lrm: service vm:106 - start migrate to node 'node2'
+info    165    node3/lrm: service vm:106 - end migrate to node 'node2'
+info    180    node1/crm: service 'vm:106': state changed from 'migrate' to 'started'  (node = node2)
+info    183    node2/lrm: starting service vm:106
+info    183    node2/lrm: service status vm:106 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node2", "state": "started" },
+    "vm:104": { "node": "node2", "state": "started" },
+    "vm:105": { "node": "node3", "state": "started" },
+    "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
new file mode 100644
index 0000000..e2568fb
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README
@@ -0,0 +1,4 @@
+With equal weights, the node imbalance is 18.7%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that incrementing
+ha-mem-weight eventually triggers a rebalancing migration from the
+memory-heavy node3.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
new file mode 100644
index 0000000..375ab2f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist
@@ -0,0 +1,6 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "datacenter crs set ha-mem-weight 2" ],
+    [ "datacenter crs set ha-mem-weight 3" ],
+    [ "datacenter crs set ha-mem-weight 4" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
new file mode 100644
index 0000000..14740f1
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "cpu": 10.0, "mem": 3221225472  },
+    "vm:102": { "cpu": 5.0,  "mem": 1073741824  },
+    "vm:103": { "cpu": 5.0,  "mem": 3221225472  },
+    "vm:104": { "cpu": 3.0,  "mem": 1073741824  },
+    "vm:105": { "cpu": 0.5,  "mem": 32212254720 },
+    "vm:106": { "cpu": 0.5,  "mem": 6442450944  }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
new file mode 100644
index 0000000..6dd1901
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect
@@ -0,0 +1,65 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: using ha-auto-rebalance '1'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node2'
+info     20    node1/crm: adding new service 'vm:104' on node 'node2'
+info     20    node1/crm: adding new service 'vm:105' on node 'node3'
+info     20    node1/crm: adding new service 'vm:106' on node 'node3'
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:105': state changed from 'request_start' to 'started'  (node = node3)
+info     20    node1/crm: service 'vm:106': state changed from 'request_start' to 'started'  (node = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     25    node3/lrm: starting service vm:106
+info     25    node3/lrm: service status vm:106 started
+info    120      cmdlist: execute datacenter crs set ha-mem-weight 2
+info    120    node1/crm: switching ha-mem-weight from '1' to '2'
+info    220      cmdlist: execute datacenter crs set ha-mem-weight 3
+info    220    node1/crm: switching ha-mem-weight from '2' to '3'
+info    320      cmdlist: execute datacenter crs set ha-mem-weight 4
+info    320    node1/crm: switching ha-mem-weight from '3' to '4'
+info    360    node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%)
+info    360    node1/crm: got crm command: migrate vm:106 node2
+info    360    node1/crm: migrate service 'vm:106' to node 'node2'
+info    360    node1/crm: service 'vm:106': state changed from 'started' to 'migrate'  (node = node3, target = node2)
+info    365    node3/lrm: service vm:106 - start migrate to node 'node2'
+info    365    node3/lrm: service vm:106 - end migrate to node 'node2'
+info    380    node1/crm: service 'vm:106': state changed from 'migrate' to 'started'  (node = node2)
+info    383    node2/lrm: starting service vm:106
+info    383    node2/lrm: service status vm:106 started
+info    920     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node2", "state": "started" },
+    "vm:104": { "node": "node2", "state": "started" },
+    "vm:105": { "node": "node3", "state": "started" },
+    "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
new file mode 100644
index 0000000..e9ea92c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that increasing ha-mem-weight
+increases the node imbalance and triggers a rebalancing migration from
+the memory-heavy node1 to node2, which has a low CPU load and memory
+usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
new file mode 100644
index 0000000..ed207f5
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "datacenter crs set ha-mem-weight 2" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+    "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+    "vm:103": { "cpu": 2.0, "mem": 8589934592  },
+    "vm:104": { "cpu": 1.0, "mem": 4294967296  },
+    "vm:105": { "cpu": 9.0, "mem": 5368709120  },
+    "vm:106": { "cpu": 6.0, "mem": 3221225472  }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
new file mode 100644
index 0000000..91f6695
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect
@@ -0,0 +1,61 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: using ha-auto-rebalance '1'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node2'
+info     20    node1/crm: adding new service 'vm:104' on node 'node2'
+info     20    node1/crm: adding new service 'vm:105' on node 'node3'
+info     20    node1/crm: adding new service 'vm:106' on node 'node3'
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:105': state changed from 'request_start' to 'started'  (node = node3)
+info     20    node1/crm: service 'vm:106': state changed from 'request_start' to 'started'  (node = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     25    node3/lrm: starting service vm:106
+info     25    node3/lrm: service status vm:106 started
+info    120      cmdlist: execute datacenter crs set ha-mem-weight 2
+info    120    node1/crm: switching ha-mem-weight from '1' to '2'
+info    160    node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%)
+info    160    node1/crm: got crm command: migrate vm:102 node2
+info    160    node1/crm: migrate service 'vm:102' to node 'node2'
+info    160    node1/crm: service 'vm:102': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    161    node1/lrm: service vm:102 - start migrate to node 'node2'
+info    161    node1/lrm: service vm:102 - end migrate to node 'node2'
+info    180    node1/crm: service 'vm:102': state changed from 'migrate' to 'started'  (node = node2)
+info    183    node2/lrm: starting service vm:102
+info    183    node2/lrm: service status vm:102 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node2", "state": "started" },
+    "vm:104": { "node": "node2", "state": "started" },
+    "vm:105": { "node": "node3", "state": "started" },
+    "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
new file mode 100644
index 0000000..82e46f8
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 35%. Test that setting ha-cpu-weight to
+0 balances the memory usage only, which triggers a rebalancing
+migration from the memory-heavy node1 to node3, which has the highest
+CPU load but a low memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
new file mode 100644
index 0000000..863befe
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "datacenter crs set ha-cpu-weight 0" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
new file mode 100644
index 0000000..7fcaf00
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg
@@ -0,0 +1,7 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1,
+        "ha-auto-rebalance-threshold": 35
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+    "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+    "vm:103": { "cpu": 2.0, "mem": 8589934592  },
+    "vm:104": { "cpu": 1.0, "mem": 4294967296  },
+    "vm:105": { "cpu": 9.0, "mem": 5368709120  },
+    "vm:106": { "cpu": 6.0, "mem": 3221225472  }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
new file mode 100644
index 0000000..b0cdaaf
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect
@@ -0,0 +1,62 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: using ha-auto-rebalance '1'
+info     20    node1/crm: using ha-auto-rebalance-threshold '0.35'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node2'
+info     20    node1/crm: adding new service 'vm:104' on node 'node2'
+info     20    node1/crm: adding new service 'vm:105' on node 'node3'
+info     20    node1/crm: adding new service 'vm:106' on node 'node3'
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:105': state changed from 'request_start' to 'started'  (node = node3)
+info     20    node1/crm: service 'vm:106': state changed from 'request_start' to 'started'  (node = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     25    node3/lrm: starting service vm:106
+info     25    node3/lrm: service status vm:106 started
+info    120      cmdlist: execute datacenter crs set ha-cpu-weight 0
+info    120    node1/crm: switching ha-cpu-weight from '1' to '0'
+info    160    node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 62.6% to 30.4%)
+info    160    node1/crm: got crm command: migrate vm:102 node3
+info    160    node1/crm: migrate service 'vm:102' to node 'node3'
+info    160    node1/crm: service 'vm:102': state changed from 'started' to 'migrate'  (node = node1, target = node3)
+info    161    node1/lrm: service vm:102 - start migrate to node 'node3'
+info    161    node1/lrm: service vm:102 - end migrate to node 'node3'
+info    180    node1/crm: service 'vm:102': state changed from 'migrate' to 'started'  (node = node3)
+info    185    node3/lrm: starting service vm:102
+info    185    node3/lrm: service status vm:102 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node2", "state": "started" },
+    "vm:104": { "node": "node2", "state": "started" },
+    "vm:105": { "node": "node3", "state": "started" },
+    "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
new file mode 100644
index 0000000..8ede76c
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README
@@ -0,0 +1,5 @@
+With equal weights, the node imbalance is 27.5%, which is below the
+ha-auto-rebalance-threshold of 30%. Test that setting ha-mem-weight to
+0 balances the CPU load only, which triggers a rebalancing migration
+from the CPU-heavy node3 to node1, which has the lowest CPU load but
+the highest memory usage.
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
new file mode 100644
index 0000000..12b01a7
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "datacenter crs set ha-mem-weight 0" ]
+]
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
new file mode 100644
index 0000000..01c8114
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "dynamic",
+        "ha-auto-rebalance": 1
+    }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
new file mode 100644
index 0000000..0d0b3b2
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "cpu": 1.5, "mem": 42949672960 },
+    "vm:102": { "cpu": 0.5, "mem": 21474836480 },
+    "vm:103": { "cpu": 2.0, "mem": 8589934592  },
+    "vm:104": { "cpu": 1.0, "mem": 4294967296  },
+    "vm:105": { "cpu": 9.0, "mem": 5368709120  },
+    "vm:106": { "cpu": 6.0, "mem": 3221225472  }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
new file mode 100644
index 0000000..4395b83
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
new file mode 100644
index 0000000..1a7e746
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect
@@ -0,0 +1,61 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'dynamic'
+info     20    node1/crm: using ha-auto-rebalance '1'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node2'
+info     20    node1/crm: adding new service 'vm:104' on node 'node2'
+info     20    node1/crm: adding new service 'vm:105' on node 'node3'
+info     20    node1/crm: adding new service 'vm:106' on node 'node3'
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:104': state changed from 'request_start' to 'started'  (node = node2)
+info     20    node1/crm: service 'vm:105': state changed from 'request_start' to 'started'  (node = node3)
+info     20    node1/crm: service 'vm:106': state changed from 'request_start' to 'started'  (node = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:102
+info     21    node1/lrm: service status vm:102 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     23    node2/lrm: starting service vm:103
+info     23    node2/lrm: service status vm:103 started
+info     23    node2/lrm: starting service vm:104
+info     23    node2/lrm: service status vm:104 started
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     25    node3/lrm: starting service vm:105
+info     25    node3/lrm: service status vm:105 started
+info     25    node3/lrm: starting service vm:106
+info     25    node3/lrm: service status vm:106 started
+info    120      cmdlist: execute datacenter crs set ha-mem-weight 0
+info    120    node1/crm: switching ha-mem-weight from '1' to '0'
+info    160    node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 62.6% to 27.8%)
+info    160    node1/crm: got crm command: migrate vm:106 node1
+info    160    node1/crm: migrate service 'vm:106' to node 'node1'
+info    160    node1/crm: service 'vm:106': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+info    165    node3/lrm: service vm:106 - start migrate to node 'node1'
+info    165    node3/lrm: service vm:106 - end migrate to node 'node1'
+info    180    node1/crm: service 'vm:106': state changed from 'migrate' to 'started'  (node = node1)
+info    181    node1/lrm: starting service vm:106
+info    181    node1/lrm: service status vm:106 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
new file mode 100644
index 0000000..60cadca
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node2", "state": "started" },
+    "vm:104": { "node": "node2", "state": "started" },
+    "vm:105": { "node": "node3", "state": "started" },
+    "vm:106": { "node": "node3", "state": "started" }
+}
diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
new file mode 100644
index 0000000..9fcd31f
--- /dev/null
+++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats
@@ -0,0 +1,8 @@
+{
+    "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 },
+    "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 }
+}
-- 
2.47.3





      parent reply	other threads:[~2026-11-12  9:47 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12  9:44 [RFC PATCH-SERIES cluster/ha-manager/manager/proxmox{,-perl-rs} v2 00/19] fix #7993: implement weighted crs load balancing Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 01/19] resource-scheduling: fix typos Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-12  9:44 ` [PATCH proxmox v2 02/19] resource-scheduling: scheduler: refine score_* docs Dominik Rusovac
2026-11-19 12:27   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 03/19] resource-scheduling: scheduler: abstract post-migration stats logic Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:47     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 04/19] fix #7993: resource-scheduling: node: add weighted node load Dominik Rusovac
2026-11-19 12:28   ` Daniel Kral
2026-11-19 12:50     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 05/19] resource-scheduling: scheduler: make imbalance with candidate public Dominik Rusovac
2026-11-19 12:41   ` Daniel Kral
2026-11-19 12:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 06/19] fix #7993: resource-scheduling: scheduler: add weights Dominik Rusovac
2026-11-19 12:40   ` Daniel Kral
2026-11-19 13:52     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 07/19] workspace,resource-scheduling: add proptest as dev-dependency Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox v2 08/19] resource-scheduling: scheduler: add property tests for weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH proxmox-perl-rs v2 09/19] fix #7993: pve-rs: resource-scheduling: add scheduler options Dominik Rusovac
2026-11-19 12:45   ` Daniel Kral
2026-11-19 13:28     ` Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-cluster v2 10/19] fix #7993: datacenter config: ad ha-{cpu,mem}-weight crs options Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 11/19] ui: form: crs options: make fine-tuning options advanced Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-manager v2 12/19] fix #7993: ui: form: crs options: add weights Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 13/19] manager: remove obsolete argument Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 14/19] manager: rename update_crs_scheduler_mod to update_crs_settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 15/19] manager: factor out crs option update helper Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 16/19] manager: log changes to crs settings Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 17/19] sim: hardware: allow changing datacenter settings at runtime Dominik Rusovac
2026-11-12  9:44 ` [PATCH pve-ha-manager v2 18/19] fix #7993: manager,usage: use weights for static and dynamic mode Dominik Rusovac
2026-11-12  9:44 ` Dominik Rusovac [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261112094437.1951271-20-d.rusovac@proxmox.com \
    --to=d.rusovac@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal