From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 069461FF0C2 for ; Thu, 22 Oct 2026 14:25:07 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1AEB82172F; Thu, 22 Oct 2026 14:24:59 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 22 Oct 2026 14:20:47 +0200 Message-Id: From: "Daniel Kral" To: "Dominik Rusovac" , Subject: Re: [PATCH pve-ha-manager 13/15] test: add auto rebalance with weights test cases X-Mailer: aerc 0.22.0-10-g6373ac9d2179-dirty References: <20261021085932.1196302-1-d.rusovac@proxmox.com> <20261021085932.1196302-14-d.rusovac@proxmox.com> In-Reply-To: <20261021085932.1196302-14-d.rusovac@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792671647429 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.718 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: JQK5MAWE36W47R2HSSMRBCCXI6EMXDXF X-Message-ID-Hash: JQK5MAWE36W47R2HSSMRBCCXI6EMXDXF X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Nice test cases! Would be nice to also have test cases for the case where one of them or even both of them are set to 0, but these are already nice to have. On Wed Oct 21, 2026 at 10:59 AM CEST, Dominik Rusovac wrote: > Add tests documenting basic efficacy of auto rebalancing with weights. > > Overview: > - test-crs-dynamic-auto-rebalance-with-increasing-bias: no rebalancing, > increment mem-weight until rebalancing is triggered > - test-crs-dynamic-auto-rebalance-with-{cpu,mem}-bias: trigger migration > that rebalances {cpu,mem} by incrementing {cpu,mem}-weight once > > Signed-off-by: Dominik Rusovac > --- > .../README | 5 ++ > .../cmdlist | 6 ++ > .../datacenter.cfg | 6 ++ > .../dynamic_service_stats | 8 +++ > .../hardware_status | 5 ++ > .../log.expect | 62 +++++++++++++++++ > .../manager_status | 1 + > .../service_config | 8 +++ > .../static_service_stats | 8 +++ > .../README | 4 ++ > .../cmdlist | 8 +++ > .../datacenter.cfg | 6 ++ > .../dynamic_service_stats | 8 +++ > .../hardware_status | 5 ++ > .../log.expect | 66 +++++++++++++++++++ > .../manager_status | 1 + > .../service_config | 8 +++ > .../static_service_stats | 8 +++ > .../README | 5 ++ > .../cmdlist | 6 ++ > .../datacenter.cfg | 6 ++ > .../dynamic_service_stats | 8 +++ > .../hardware_status | 5 ++ > .../log.expect | 62 +++++++++++++++++ > .../manager_status | 1 + > .../service_config | 8 +++ > .../static_service_stats | 8 +++ > 27 files changed, 332 insertions(+) > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/README > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/cmdlist > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/datacenter.cfg > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/dynamic_service_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/hardware_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/log.expect > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/manager_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/service_config > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-cpu-bia= s/static_service_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/README > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/cmdlist > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/datacenter.cfg > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/dynamic_service_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/hardware_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/log.expect > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/manager_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/service_config > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-increas= ing-bias/static_service_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/README > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/cmdlist > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/datacenter.cfg > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/dynamic_service_stats > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/hardware_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/log.expect > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/manager_status > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/service_config > create mode 100644 src/test/test-crs-dynamic-auto-rebalance-with-mem-bia= s/static_service_stats > > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/READM= E b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README > new file mode 100644 > index 0000000..f67978e > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/README > @@ -0,0 +1,5 @@ > +At equal weights (=3D1) load imbalance sits at 27.5%, which is below the > +ha-auto-rebalance-threshold of 30%. Test that increasing the > +ha-auto-rebalance-cpu-weight will increase load imbalance and trigger a > +rebalancing migration from cpu-heavy node3 to node2, which sits low on e= ither > +resource. > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdli= st b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist > new file mode 100644 > index 0000000..c29b4b6 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/cmdlist > @@ -0,0 +1,6 @@ > +[ > + [ "power node1 on", "power node2 on", "power node3 on" ], > + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], > + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], > + [ "datacenter crs set ha-auto-rebalance-cpu-weight 2" ] Wouldn't it suffice to only issue the last command? Each command list is always executed 5 HA Manager rounds apart, so with the default hold duration of 3 rounds it would already have done something before setting the {cpu,mem}-weight to 1 explicitly. If we also log the first set value as noted in the crs option log patch, then it would also show up in the log here if it's set to something different than the default value. > +] > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datac= enter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacent= er.cfg > new file mode 100644 > index 0000000..01c8114 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/datacenter.c= fg > @@ -0,0 +1,6 @@ > +{ > + "crs": { > + "ha": "dynamic", > + "ha-auto-rebalance": 1 > + } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynam= ic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/d= ynamic_service_stats > new file mode 100644 > index 0000000..0d0b3b2 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/dynamic_serv= ice_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, > + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, > + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, > + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, > + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, > + "vm:106": { "cpu": 6.0, "mem": 3221225472 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardw= are_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardwar= e_status > new file mode 100644 > index 0000000..4395b83 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/hardware_sta= tus > @@ -0,0 +1,5 @@ > +{ > + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.e= xpect b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect > new file mode 100644 > index 0000000..352aa1c > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/log.expect > @@ -0,0 +1,62 @@ > +info 0 hardware: starting simulation > +info 20 cmdlist: execute power node1 on > +info 20 node1/crm: status change startup =3D> wait_for_quorum > +info 20 node1/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node2 on > +info 20 node2/crm: status change startup =3D> wait_for_quorum > +info 20 node2/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node3 on > +info 20 node3/crm: status change startup =3D> wait_for_quorum > +info 20 node3/lrm: status change startup =3D> wait_for_agent_lock > +info 20 node1/crm: got lock 'ha_manager_lock' > +info 20 node1/crm: status change wait_for_quorum =3D> master > +info 20 node1/crm: using scheduler mode 'dynamic' > +info 20 node1/crm: node 'node1': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node2': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node3': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: adding new service 'vm:101' on node 'node1' > +info 20 node1/crm: adding new service 'vm:102' on node 'node1' > +info 20 node1/crm: adding new service 'vm:103' on node 'node2' > +info 20 node1/crm: adding new service 'vm:104' on node 'node2' > +info 20 node1/crm: adding new service 'vm:105' on node 'node3' > +info 20 node1/crm: adding new service 'vm:106' on node 'node3' > +info 20 node1/crm: service 'vm:101': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:102': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:103': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:104': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:105': state changed from 'request_= start' to 'started' (node =3D node3) > +info 20 node1/crm: service 'vm:106': state changed from 'request_= start' to 'started' (node =3D node3) > +info 21 node1/lrm: got lock 'ha_agent_node1_lock' > +info 21 node1/lrm: status change wait_for_agent_lock =3D> active > +info 21 node1/lrm: starting service vm:101 > +info 21 node1/lrm: service status vm:101 started > +info 21 node1/lrm: starting service vm:102 > +info 21 node1/lrm: service status vm:102 started > +info 22 node2/crm: status change wait_for_quorum =3D> slave > +info 23 node2/lrm: got lock 'ha_agent_node2_lock' > +info 23 node2/lrm: status change wait_for_agent_lock =3D> active > +info 23 node2/lrm: starting service vm:103 > +info 23 node2/lrm: service status vm:103 started > +info 23 node2/lrm: starting service vm:104 > +info 23 node2/lrm: service status vm:104 started > +info 24 node3/crm: status change wait_for_quorum =3D> slave > +info 25 node3/lrm: got lock 'ha_agent_node3_lock' > +info 25 node3/lrm: status change wait_for_agent_lock =3D> active > +info 25 node3/lrm: starting service vm:105 > +info 25 node3/lrm: service status vm:105 started > +info 25 node3/lrm: starting service vm:106 > +info 25 node3/lrm: service status vm:106 started > +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-c= pu-weight 1 > +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 1 > +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-c= pu-weight 2 > +info 320 node1/crm: crs settings: 'ha-auto-rebalance-cpu-weight' c= hanged from '1' to '2' > +info 360 node1/crm: auto rebalance - migrate vm:106 to node2 (expe= cted change for imbalance from 33.3% to 4.4%) > +info 360 node1/crm: got crm command: migrate vm:106 node2 > +info 360 node1/crm: migrate service 'vm:106' to node 'node2' > +info 360 node1/crm: service 'vm:106': state changed from 'started'= to 'migrate' (node =3D node3, target =3D node2) > +info 365 node3/lrm: service vm:106 - start migrate to node 'node2' > +info 365 node3/lrm: service vm:106 - end migrate to node 'node2' > +info 380 node1/crm: service 'vm:106': state changed from 'migrate'= to 'started' (node =3D node2) > +info 383 node2/lrm: starting service vm:106 > +info 383 node2/lrm: service status vm:106 started > +info 920 hardware: exit simulation - done > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manag= er_status b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_= status > new file mode 100644 > index 0000000..0967ef4 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/manager_stat= us > @@ -0,0 +1 @@ > +{} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/servi= ce_config b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_= config > new file mode 100644 > index 0000000..60cadca > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/service_conf= ig > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "node": "node1", "state": "started" }, > + "vm:102": { "node": "node1", "state": "started" }, > + "vm:103": { "node": "node2", "state": "started" }, > + "vm:104": { "node": "node2", "state": "started" }, > + "vm:105": { "node": "node3", "state": "started" }, > + "vm:106": { "node": "node3", "state": "started" } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/stati= c_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/st= atic_service_stats > new file mode 100644 > index 0000000..9fcd31f > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-cpu-bias/static_servi= ce_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/README b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/RE= ADME > new file mode 100644 > index 0000000..a9974a7 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/READM= E > @@ -0,0 +1,4 @@ > +At equal weights (=3D1) load imbalance sits at 18.7%, which is below the > +ha-auto-rebalance-threshold of 30%. Test that incrementing > +ha-auto-rebalance-mem-weight will eventually trigger a rebalancing migra= tion > +from mem-heavy node3. > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/cmdlist b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/c= mdlist > new file mode 100644 > index 0000000..643ab0b > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/cmdli= st > @@ -0,0 +1,8 @@ > +[ > + [ "power node1 on", "power node2 on", "power node3 on" ], > + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], > + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], Same here, these seem redundant for the test case alone > + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ], > + [ "datacenter crs set ha-auto-rebalance-mem-weight 3" ], > + [ "datacenter crs set ha-auto-rebalance-mem-weight 4" ] > +] > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-increasing= -bias/datacenter.cfg > new file mode 100644 > index 0000000..01c8114 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/datac= enter.cfg > @@ -0,0 +1,6 @@ > +{ > + "crs": { > + "ha": "dynamic", > + "ha-auto-rebalance": 1 > + } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-inc= reasing-bias/dynamic_service_stats > new file mode 100644 > index 0000000..14740f1 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/dynam= ic_service_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "cpu": 10.0, "mem": 3221225472 }, > + "vm:102": { "cpu": 5.0, "mem": 1073741824 }, > + "vm:103": { "cpu": 5.0, "mem": 3221225472 }, > + "vm:104": { "cpu": 3.0, "mem": 1073741824 }, > + "vm:105": { "cpu": 0.5, "mem": 32212254720 }, > + "vm:106": { "cpu": 0.5, "mem": 6442450944 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/hardware_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasin= g-bias/hardware_status > new file mode 100644 > index 0000000..4395b83 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/hardw= are_status > @@ -0,0 +1,5 @@ > +{ > + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/log.expect b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/log.expect > new file mode 100644 > index 0000000..596ef29 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/log.e= xpect > @@ -0,0 +1,66 @@ > +info 0 hardware: starting simulation > +info 20 cmdlist: execute power node1 on > +info 20 node1/crm: status change startup =3D> wait_for_quorum > +info 20 node1/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node2 on > +info 20 node2/crm: status change startup =3D> wait_for_quorum > +info 20 node2/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node3 on > +info 20 node3/crm: status change startup =3D> wait_for_quorum > +info 20 node3/lrm: status change startup =3D> wait_for_agent_lock > +info 20 node1/crm: got lock 'ha_manager_lock' > +info 20 node1/crm: status change wait_for_quorum =3D> master > +info 20 node1/crm: using scheduler mode 'dynamic' > +info 20 node1/crm: node 'node1': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node2': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node3': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: adding new service 'vm:101' on node 'node1' > +info 20 node1/crm: adding new service 'vm:102' on node 'node1' > +info 20 node1/crm: adding new service 'vm:103' on node 'node2' > +info 20 node1/crm: adding new service 'vm:104' on node 'node2' > +info 20 node1/crm: adding new service 'vm:105' on node 'node3' > +info 20 node1/crm: adding new service 'vm:106' on node 'node3' > +info 20 node1/crm: service 'vm:101': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:102': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:103': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:104': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:105': state changed from 'request_= start' to 'started' (node =3D node3) > +info 20 node1/crm: service 'vm:106': state changed from 'request_= start' to 'started' (node =3D node3) > +info 21 node1/lrm: got lock 'ha_agent_node1_lock' > +info 21 node1/lrm: status change wait_for_agent_lock =3D> active > +info 21 node1/lrm: starting service vm:101 > +info 21 node1/lrm: service status vm:101 started > +info 21 node1/lrm: starting service vm:102 > +info 21 node1/lrm: service status vm:102 started > +info 22 node2/crm: status change wait_for_quorum =3D> slave > +info 23 node2/lrm: got lock 'ha_agent_node2_lock' > +info 23 node2/lrm: status change wait_for_agent_lock =3D> active > +info 23 node2/lrm: starting service vm:103 > +info 23 node2/lrm: service status vm:103 started > +info 23 node2/lrm: starting service vm:104 > +info 23 node2/lrm: service status vm:104 started > +info 24 node3/crm: status change wait_for_quorum =3D> slave > +info 25 node3/lrm: got lock 'ha_agent_node3_lock' > +info 25 node3/lrm: status change wait_for_agent_lock =3D> active > +info 25 node3/lrm: starting service vm:105 > +info 25 node3/lrm: service status vm:105 started > +info 25 node3/lrm: starting service vm:106 > +info 25 node3/lrm: service status vm:106 started > +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-c= pu-weight 1 > +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 1 > +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 2 > +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' c= hanged from '1' to '2' > +info 420 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 3 > +info 420 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' c= hanged from '2' to '3' > +info 520 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 4 > +info 520 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' c= hanged from '3' to '4' > +info 560 node1/crm: auto rebalance - migrate vm:106 to node2 (expe= cted change for imbalance from 32.9% to 17.3%) > +info 560 node1/crm: got crm command: migrate vm:106 node2 > +info 560 node1/crm: migrate service 'vm:106' to node 'node2' > +info 560 node1/crm: service 'vm:106': state changed from 'started'= to 'migrate' (node =3D node3, target =3D node2) > +info 565 node3/lrm: service vm:106 - start migrate to node 'node2' > +info 565 node3/lrm: service vm:106 - end migrate to node 'node2' > +info 580 node1/crm: service 'vm:106': state changed from 'migrate'= to 'started' (node =3D node2) > +info 583 node2/lrm: starting service vm:106 > +info 583 node2/lrm: service status vm:106 started > +info 1120 hardware: exit simulation - done > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/manager_status b/src/test/test-crs-dynamic-auto-rebalance-with-increasing= -bias/manager_status > new file mode 100644 > index 0000000..0967ef4 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/manag= er_status > @@ -0,0 +1 @@ > +{} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/service_config b/src/test/test-crs-dynamic-auto-rebalance-with-increasing= -bias/service_config > new file mode 100644 > index 0000000..60cadca > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/servi= ce_config > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "node": "node1", "state": "started" }, > + "vm:102": { "node": "node1", "state": "started" }, > + "vm:103": { "node": "node2", "state": "started" }, > + "vm:104": { "node": "node2", "state": "started" }, > + "vm:105": { "node": "node3", "state": "started" }, > + "vm:106": { "node": "node3", "state": "started" } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bia= s/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-incr= easing-bias/static_service_stats > new file mode 100644 > index 0000000..9fcd31f > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-increasing-bias/stati= c_service_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/READM= E b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README > new file mode 100644 > index 0000000..19cebf9 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/README > @@ -0,0 +1,5 @@ > +At equal weights (=3D1) load imbalance sits at 27.5%, which is below the > +ha-auto-rebalance-threshold of 30%. Test that increasing the > +ha-auto-rebalance-mem-weight will increase load imbalance and trigger a > +rebalancing migration from mem-heavy node1 to node2, which sits low on e= ither > +resource. > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdli= st b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist > new file mode 100644 > index 0000000..2144ffb > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/cmdlist > @@ -0,0 +1,6 @@ > +[ > + [ "power node1 on", "power node2 on", "power node3 on" ], > + [ "datacenter crs set ha-auto-rebalance-cpu-weight 1" ], > + [ "datacenter crs set ha-auto-rebalance-mem-weight 1" ], same here > + [ "datacenter crs set ha-auto-rebalance-mem-weight 2" ] > +] > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datac= enter.cfg b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacent= er.cfg > new file mode 100644 > index 0000000..01c8114 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/datacenter.c= fg > @@ -0,0 +1,6 @@ > +{ > + "crs": { > + "ha": "dynamic", > + "ha-auto-rebalance": 1 > + } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynam= ic_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/d= ynamic_service_stats > new file mode 100644 > index 0000000..0d0b3b2 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/dynamic_serv= ice_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "cpu": 1.5, "mem": 42949672960 }, > + "vm:102": { "cpu": 0.5, "mem": 21474836480 }, > + "vm:103": { "cpu": 2.0, "mem": 8589934592 }, > + "vm:104": { "cpu": 1.0, "mem": 4294967296 }, > + "vm:105": { "cpu": 9.0, "mem": 5368709120 }, > + "vm:106": { "cpu": 6.0, "mem": 3221225472 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardw= are_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardwar= e_status > new file mode 100644 > index 0000000..4395b83 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/hardware_sta= tus > @@ -0,0 +1,5 @@ > +{ > + "node1": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node2": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 }, > + "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 6= 8719476736 } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.e= xpect b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect > new file mode 100644 > index 0000000..5fc901f > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/log.expect > @@ -0,0 +1,62 @@ > +info 0 hardware: starting simulation > +info 20 cmdlist: execute power node1 on > +info 20 node1/crm: status change startup =3D> wait_for_quorum > +info 20 node1/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node2 on > +info 20 node2/crm: status change startup =3D> wait_for_quorum > +info 20 node2/lrm: status change startup =3D> wait_for_agent_lock > +info 20 cmdlist: execute power node3 on > +info 20 node3/crm: status change startup =3D> wait_for_quorum > +info 20 node3/lrm: status change startup =3D> wait_for_agent_lock > +info 20 node1/crm: got lock 'ha_manager_lock' > +info 20 node1/crm: status change wait_for_quorum =3D> master > +info 20 node1/crm: using scheduler mode 'dynamic' > +info 20 node1/crm: node 'node1': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node2': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: node 'node3': state changed from 'unknown' =3D= > 'online' > +info 20 node1/crm: adding new service 'vm:101' on node 'node1' > +info 20 node1/crm: adding new service 'vm:102' on node 'node1' > +info 20 node1/crm: adding new service 'vm:103' on node 'node2' > +info 20 node1/crm: adding new service 'vm:104' on node 'node2' > +info 20 node1/crm: adding new service 'vm:105' on node 'node3' > +info 20 node1/crm: adding new service 'vm:106' on node 'node3' > +info 20 node1/crm: service 'vm:101': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:102': state changed from 'request_= start' to 'started' (node =3D node1) > +info 20 node1/crm: service 'vm:103': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:104': state changed from 'request_= start' to 'started' (node =3D node2) > +info 20 node1/crm: service 'vm:105': state changed from 'request_= start' to 'started' (node =3D node3) > +info 20 node1/crm: service 'vm:106': state changed from 'request_= start' to 'started' (node =3D node3) > +info 21 node1/lrm: got lock 'ha_agent_node1_lock' > +info 21 node1/lrm: status change wait_for_agent_lock =3D> active > +info 21 node1/lrm: starting service vm:101 > +info 21 node1/lrm: service status vm:101 started > +info 21 node1/lrm: starting service vm:102 > +info 21 node1/lrm: service status vm:102 started > +info 22 node2/crm: status change wait_for_quorum =3D> slave > +info 23 node2/lrm: got lock 'ha_agent_node2_lock' > +info 23 node2/lrm: status change wait_for_agent_lock =3D> active > +info 23 node2/lrm: starting service vm:103 > +info 23 node2/lrm: service status vm:103 started > +info 23 node2/lrm: starting service vm:104 > +info 23 node2/lrm: service status vm:104 started > +info 24 node3/crm: status change wait_for_quorum =3D> slave > +info 25 node3/lrm: got lock 'ha_agent_node3_lock' > +info 25 node3/lrm: status change wait_for_agent_lock =3D> active > +info 25 node3/lrm: starting service vm:105 > +info 25 node3/lrm: service status vm:105 started > +info 25 node3/lrm: starting service vm:106 > +info 25 node3/lrm: service status vm:106 started > +info 120 cmdlist: execute datacenter crs set ha-auto-rebalance-c= pu-weight 1 > +info 220 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 1 > +info 320 cmdlist: execute datacenter crs set ha-auto-rebalance-m= em-weight 2 > +info 320 node1/crm: crs settings: 'ha-auto-rebalance-mem-weight' c= hanged from '1' to '2' > +info 360 node1/crm: auto rebalance - migrate vm:102 to node2 (expe= cted change for imbalance from 33.3% to 3.8%) > +info 360 node1/crm: got crm command: migrate vm:102 node2 > +info 360 node1/crm: migrate service 'vm:102' to node 'node2' > +info 360 node1/crm: service 'vm:102': state changed from 'started'= to 'migrate' (node =3D node1, target =3D node2) > +info 361 node1/lrm: service vm:102 - start migrate to node 'node2' > +info 361 node1/lrm: service vm:102 - end migrate to node 'node2' > +info 380 node1/crm: service 'vm:102': state changed from 'migrate'= to 'started' (node =3D node2) > +info 383 node2/lrm: starting service vm:102 > +info 383 node2/lrm: service status vm:102 started > +info 920 hardware: exit simulation - done > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manag= er_status b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_= status > new file mode 100644 > index 0000000..0967ef4 > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/manager_stat= us > @@ -0,0 +1 @@ > +{} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/servi= ce_config b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_= config > new file mode 100644 > index 0000000..60cadca > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/service_conf= ig > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "node": "node1", "state": "started" }, > + "vm:102": { "node": "node1", "state": "started" }, > + "vm:103": { "node": "node2", "state": "started" }, > + "vm:104": { "node": "node2", "state": "started" }, > + "vm:105": { "node": "node3", "state": "started" }, > + "vm:106": { "node": "node3", "state": "started" } > +} > diff --git a/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/stati= c_service_stats b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/st= atic_service_stats > new file mode 100644 > index 0000000..9fcd31f > --- /dev/null > +++ b/src/test/test-crs-dynamic-auto-rebalance-with-mem-bias/static_servi= ce_stats > @@ -0,0 +1,8 @@ > +{ > + "vm:101": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:102": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:103": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:104": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:105": { "maxcpu": 16.0, "maxmem": 68719476736 }, > + "vm:106": { "maxcpu": 16.0, "maxmem": 68719476736 } > +}