From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9E1651FF0B6 for ; Thu, 12 Nov 2026 10:47:52 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BE968217D3; Thu, 12 Nov 2026 10:46:32 +0100 (CET) From: Dominik Rusovac To: pve-devel@lists.proxmox.com Subject: [PATCH pve-ha-manager v2 19/19] test: add automatic rebalancing test cases with weights Date: Thu, 12 Nov 2026 10:44:37 +0100 Message-ID: <20261112094437.1951271-20-d.rusovac@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261112094437.1951271-1-d.rusovac@proxmox.com> References: <20261112094437.1951271-1-d.rusovac@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794476686784 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.029 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: RRWPHOFAPAMDAKANEOO7VFGG6DIT3AFF X-Message-ID-Hash: RRWPHOFAPAMDAKANEOO7VFGG6DIT3AFF X-MailFrom: d.rusovac@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: add test cases which document that the crs weights steer the automatic rebalancing towards balancing the CPU load or the memory usage of the nodes; increasing a weight lets the load imbalance exceed the threshold for a cluster that is balanced with equal weights; and a weight of 0 balances the other resource only Signed-off-by: Dominik Rusovac --- Notes: changes since v1: * reword commit * use the new option names * drop the redundant commands setting the weights to 1 * add test cases with one of the weights set to 0 .../README | 4 ++ .../cmdlist | 4 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 61 +++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 4 ++ .../cmdlist | 6 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 65 +++++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 5 ++ .../cmdlist | 4 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 61 +++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 5 ++ .../cmdlist | 4 ++ .../datacenter.cfg | 7 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 62 ++++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 5 ++ .../cmdlist | 4 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 61 +++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ 45 files changed, 536 insertions(+) create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README new file mode 100644 index 0000000..05cb5df --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README @@ -0,0 +1,4 @@ +With equal weights, the node imbalance is 27.5%, which is below the +ha-auto-rebalance-threshold of 30%. Test that increasing ha-cpu-weight +increases the node imbalance and triggers a rebalancing migration from +the CPU-heavy node3 to node2, which has a low CPU load and memory usage. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist new file mode 100644 index 0000000..869d8e8 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-cpu-weight 2" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect new file mode 100644 index 0000000..cf770d5 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect @@ -0,0 +1,61 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-cpu-weight 2 +info 120 node1/crm: switching ha-cpu-weight from '1' to '2' +info 160 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%) +info 160 node1/crm: got crm command: migrate vm:106 node2 +info 160 node1/crm: migrate service 'vm:106' to node 'node2' +info 160 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2) +info 165 node3/lrm: service vm:106 - start migrate to node 'node2' +info 165 node3/lrm: service vm:106 - end migrate to node 'node2' +info 180 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2) +info 183 node2/lrm: starting service vm:106 +info 183 node2/lrm: service status vm:106 started +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README new file mode 100644 index 0000000..e2568fb --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README @@ -0,0 +1,4 @@ +With equal weights, the node imbalance is 18.7%, which is below the +ha-auto-rebalance-threshold of 30%. Test that incrementing +ha-mem-weight eventually triggers a rebalancing migration from the +memory-heavy node3. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist new file mode 100644 index 0000000..375ab2f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist @@ -0,0 +1,6 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-mem-weight 2" ], + [ "datacenter crs set ha-mem-weight 3" ], + [ "datacenter crs set ha-mem-weight 4" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats new file mode 100644 index 0000000..14740f1 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 10.0, "mem": 3221225472 }, + "vm:102": { "cpu": 5.0, "mem": 1073741824 }, + "vm:103": { "cpu": 5.0, "mem": 3221225472 }, + "vm:104": { "cpu": 3.0, "mem": 1073741824 }, + "vm:105": { "cpu": 0.5, "mem": 32212254720 }, + "vm:106": { "cpu": 0.5, "mem": 6442450944 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect new file mode 100644 index 0000000..6dd1901 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect @@ -0,0 +1,65 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-mem-weight 2 +info 120 node1/crm: switching ha-mem-weight from '1' to '2' +info 220 cmdlist: execute datacenter crs set ha-mem-weight 3 +info 220 node1/crm: switching ha-mem-weight from '2' to '3' +info 320 cmdlist: execute datacenter crs set ha-mem-weight 4 +info 320 node1/crm: switching ha-mem-weight from '3' to '4' +info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%) +info 360 node1/crm: got crm command: migrate vm:106 node2 +info 360 node1/crm: migrate service 'vm:106' to node 'node2' +info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2) +info 365 node3/lrm: service vm:106 - start migrate to node 'node2' +info 365 node3/lrm: service vm:106 - end migrate to node 'node2' +info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2) +info 383 node2/lrm: starting service vm:106 +info 383 node2/lrm: service status vm:106 started +info 920 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README new file mode 100644 index 0000000..e9ea92c --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README @@ -0,0 +1,5 @@ +With equal weights, the node imbalance is 27.5%, which is below the +ha-auto-rebalance-threshold of 30%. Test that increasing ha-mem-weight +increases the node imbalance and triggers a rebalancing migration from +the memory-heavy node1 to node2, which has a low CPU load and memory +usage. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist new file mode 100644 index 0000000..ed207f5 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-mem-weight 2" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect new file mode 100644 index 0000000..91f6695 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect @@ -0,0 +1,61 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-mem-weight 2 +info 120 node1/crm: switching ha-mem-weight from '1' to '2' +info 160 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%) +info 160 node1/crm: got crm command: migrate vm:102 node2 +info 160 node1/crm: migrate service 'vm:102' to node 'node2' +info 160 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2) +info 161 node1/lrm: service vm:102 - start migrate to node 'node2' +info 161 node1/lrm: service vm:102 - end migrate to node 'node2' +info 180 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2) +info 183 node2/lrm: starting service vm:102 +info 183 node2/lrm: service status vm:102 started +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README new file mode 100644 index 0000000..82e46f8 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/README @@ -0,0 +1,5 @@ +With equal weights, the node imbalance is 27.5%, which is below the +ha-auto-rebalance-threshold of 35%. Test that setting ha-cpu-weight to +0 balances the memory usage only, which triggers a rebalancing +migration from the memory-heavy node1 to node3, which has the highest +CPU load but a low memory usage. diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist new file mode 100644 index 0000000..863befe --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-cpu-weight 0" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg new file mode 100644 index 0000000..7fcaf00 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/datacenter.cfg @@ -0,0 +1,7 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1, + "ha-auto-rebalance-threshold": 35 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect new file mode 100644 index 0000000..b0cdaaf --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/log.expect @@ -0,0 +1,62 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: using ha-auto-rebalance-threshold '0.35' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-cpu-weight 0 +info 120 node1/crm: switching ha-cpu-weight from '1' to '0' +info 160 node1/crm: auto rebalance - migrate vm:102 to node3 (expected change for imbalance from 62.6% to 30.4%) +info 160 node1/crm: got crm command: migrate vm:102 node3 +info 160 node1/crm: migrate service 'vm:102' to node 'node3' +info 160 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node3) +info 161 node1/lrm: service vm:102 - start migrate to node 'node3' +info 161 node1/lrm: service vm:102 - end migrate to node 'node3' +info 180 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node3) +info 185 node3/lrm: starting service vm:102 +info 185 node3/lrm: service status vm:102 started +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-cpu-weight/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README new file mode 100644 index 0000000..8ede76c --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/README @@ -0,0 +1,5 @@ +With equal weights, the node imbalance is 27.5%, which is below the +ha-auto-rebalance-threshold of 30%. Test that setting ha-mem-weight to +0 balances the CPU load only, which triggers a rebalancing migration +from the CPU-heavy node3 to node1, which has the lowest CPU load but +the highest memory usage. diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist new file mode 100644 index 0000000..12b01a7 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-mem-weight 0" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect new file mode 100644 index 0000000..1a7e746 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/log.expect @@ -0,0 +1,61 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: using ha-auto-rebalance '1' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-mem-weight 0 +info 120 node1/crm: switching ha-mem-weight from '1' to '0' +info 160 node1/crm: auto rebalance - migrate vm:106 to node1 (expected change for imbalance from 62.6% to 27.8%) +info 160 node1/crm: got crm command: migrate vm:106 node1 +info 160 node1/crm: migrate service 'vm:106' to node 'node1' +info 160 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node1) +info 165 node3/lrm: service vm:106 - start migrate to node 'node1' +info 165 node3/lrm: service vm:106 - end migrate to node 'node1' +info 180 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node1) +info 181 node1/lrm: starting service vm:106 +info 181 node1/lrm: service status vm:106 started +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-zero-mem-weight/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} -- 2.47.3