From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 597F41FF0C0 for ; Wed, 21 Oct 2026 11:01:58 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 434C421796; Wed, 21 Oct 2026 11:00:42 +0200 (CEST) From: Dominik Rusovac To: pve-devel@lists.proxmox.com Subject: [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases Date: Wed, 21 Oct 2026 10:59:30 +0200 Message-ID: <20261021085932.1196302-14-d.rusovac@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261021085932.1196302-1-d.rusovac@proxmox.com> References: <20261021085932.1196302-1-d.rusovac@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792573177984 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.057 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 6JRO24ESLENG2TOU55HXHW5TL6QAU2XB X-Message-ID-Hash: 6JRO24ESLENG2TOU55HXHW5TL6QAU2XB X-MailFrom: d.rusovac@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add tests documenting basic efficacy of auto rebalancing with weights. Overview: - test-crs-dynamic-auto-rebalance-with-increasing-bias: no rebalancing, increment mem-weight until rebalancing is triggered - test-crs-dynamic-auto-rebalance-with-{cpu,mem}-bias: trigger migration that rebalances {cpu,mem} by incrementing {cpu,mem}-weight once Signed-off-by: Dominik Rusovac --- .../README | 5 ++ .../cmdlist | 6 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 62 +++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 4 ++ .../cmdlist | 8 +++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 66 +++++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ .../README | 5 ++ .../cmdlist | 6 ++ .../datacenter.cfg | 6 ++ .../dynamic_service_stats | 8 +++ .../hardware_status | 5 ++ .../log.expect | 62 +++++++++++++++++ .../manager_status | 1 + .../service_config | 8 +++ .../static_service_stats | 8 +++ 27 files changed, 332 insertions(+) create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README new file mode 100644 index 0000000..f67978e --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README @@ -0,0 +1,5 @@ +At equal weights (=1) load imbalance sits at 27.5%, which is below the +ha-auto-rebalance-threshold of 30%. Test that increasing the +ha-auto-rebalance-cpu-weight will increase load imbalance and trigger a +rebalancing migration from cpu-heavy node3 to node2, which sits low on either +resource. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist new file mode 100644 index 0000000..c29b4b6 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist @@ -0,0 +1,6 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-cpu-weight 2" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect new file mode 100644 index 0000000..352aa1c --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect @@ -0,0 +1,62 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1 +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1 +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 2 +info 320 node1/crm: crs settings: 'ha-auto-rebalance-cpu-weight' changed from '1' to '2' +info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 33.3% to 4.4%) +info 360 node1/crm: got crm command: migrate vm:106 node2 +info 360 node1/crm: migrate service 'vm:106' to node 'node2' +info 360 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2) +info 365 node3/lrm: service vm:106 - start migrate to node 'node2' +info 365 node3/lrm: service vm:106 - end migrate to node 'node2' +info 380 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2) +info 383 node2/lrm: starting service vm:106 +info 383 node2/lrm: service status vm:106 started +info 920 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README new file mode 100644 index 0000000..a9974a7 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/README @@ -0,0 +1,4 @@ +At equal weights (=1) load imbalance sits at 18.7%, which is below the +ha-auto-rebalance-threshold of 30%. Test that incrementing +ha-auto-rebalance-mem-weight will eventually trigger a rebalancing migration +from mem-heavy node3. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist new file mode 100644 index 0000000..643ab0b --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdlist @@ -0,0 +1,8 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 3" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 4" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats new file mode 100644 index 0000000..14740f1 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 10.0, "mem": 3221225472 }, + "vm:102": { "cpu": 5.0, "mem": 1073741824 }, + "vm:103": { "cpu": 5.0, "mem": 3221225472 }, + "vm:104": { "cpu": 3.0, "mem": 1073741824 }, + "vm:105": { "cpu": 0.5, "mem": 32212254720 }, + "vm:106": { "cpu": 0.5, "mem": 6442450944 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect new file mode 100644 index 0000000..596ef29 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.expect @@ -0,0 +1,66 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1 +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1 +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2 +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2' +info 420 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 3 +info 420 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '2' to '3' +info 520 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 4 +info 520 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '3' to '4' +info 560 node1/crm: auto rebalance - migrate vm:106 to node2 (expected change for imbalance from 32.9% to 17.3%) +info 560 node1/crm: got crm command: migrate vm:106 node2 +info 560 node1/crm: migrate service 'vm:106' to node 'node2' +info 560 node1/crm: service 'vm:106': state changed from 'started' to 'migrate' (node = node3, target = node2) +info 565 node3/lrm: service vm:106 - start migrate to node 'node2' +info 565 node3/lrm: service vm:106 - end migrate to node 'node2' +info 580 node1/crm: service 'vm:106': state changed from 'migrate' to 'started' (node = node2) +info 583 node2/lrm: starting service vm:106 +info 583 node2/lrm: service status vm:106 started +info 1120 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README new file mode 100644 index 0000000..19cebf9 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README @@ -0,0 +1,5 @@ +At equal weights (=1) load imbalance sits at 27.5%, which is below the +ha-auto-rebalance-threshold of 30%. Test that increasing the +ha-auto-rebalance-mem-weight will increase load imbalance and trigger a +rebalancing migration from mem-heavy node1 to node2, which sits low on either +resource. diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist new file mode 100644 index 0000000..2144ffb --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist @@ -0,0 +1,6 @@ +[ + [ "power node1 on", "power node2 on", "power node3 on" ], + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats new file mode 100644 index 0000000..0d0b3b2 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, + "vm:106": { "cpu": 6.0, "mem": 3221225472 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status new file mode 100644 index 0000000..4395b83 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_status @@ -0,0 +1,5 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 }, + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 68719476736 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect new file mode 100644 index 0000000..5fc901f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect @@ -0,0 +1,62 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node3 on +info 20 node3/crm: status change startup => wait_for_quorum +info 20 node3/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: adding new service 'vm:102' on node 'node1' +info 20 node1/crm: adding new service 'vm:103' on node 'node2' +info 20 node1/crm: adding new service 'vm:104' on node 'node2' +info 20 node1/crm: adding new service 'vm:105' on node 'node3' +info 20 node1/crm: adding new service 'vm:106' on node 'node3' +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:104': state changed from 'request_start' to 'started' (node = node2) +info 20 node1/crm: service 'vm:105': state changed from 'request_start' to 'started' (node = node3) +info 20 node1/crm: service 'vm:106': state changed from 'request_start' to 'started' (node = node3) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 21 node1/lrm: starting service vm:102 +info 21 node1/lrm: service status vm:102 started +info 22 node2/crm: status change wait_for_quorum => slave +info 23 node2/lrm: got lock 'ha_agent_node2_lock' +info 23 node2/lrm: status change wait_for_agent_lock => active +info 23 node2/lrm: starting service vm:103 +info 23 node2/lrm: service status vm:103 started +info 23 node2/lrm: starting service vm:104 +info 23 node2/lrm: service status vm:104 started +info 24 node3/crm: status change wait_for_quorum => slave +info 25 node3/lrm: got lock 'ha_agent_node3_lock' +info 25 node3/lrm: status change wait_for_agent_lock => active +info 25 node3/lrm: starting service vm:105 +info 25 node3/lrm: service status vm:105 started +info 25 node3/lrm: starting service vm:106 +info 25 node3/lrm: service status vm:106 started +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-cpu-weight 1 +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 1 +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-mem-weight 2 +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' changed from '1' to '2' +info 360 node1/crm: auto rebalance - migrate vm:102 to node2 (expected change for imbalance from 33.3% to 3.8%) +info 360 node1/crm: got crm command: migrate vm:102 node2 +info 360 node1/crm: migrate service 'vm:102' to node 'node2' +info 360 node1/crm: service 'vm:102': state changed from 'started' to 'migrate' (node = node1, target = node2) +info 361 node1/lrm: service vm:102 - start migrate to node 'node2' +info 361 node1/lrm: service vm:102 - end migrate to node 'node2' +info 380 node1/crm: service 'vm:102': state changed from 'migrate' to 'started' (node = node2) +info 383 node2/lrm: starting service vm:102 +info 383 node2/lrm: service status vm:102 started +info 920 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config new file mode 100644 index 0000000..60cadca --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_config @@ -0,0 +1,8 @@ +{ + "vm:101": { "node": "node1", "state": "started" }, + "vm:102": { "node": "node1", "state": "started" }, + "vm:103": { "node": "node2", "state": "started" }, + "vm:104": { "node": "node2", "state": "started" }, + "vm:105": { "node": "node3", "state": "started" }, + "vm:106": { "node": "node3", "state": "started" } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats new file mode 100644 index 0000000..9fcd31f --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_service_stats @@ -0,0 +1,8 @@ +{ + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } +} -- 2.47.3