From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 280DA1FF141 for ; Mon, 30 Mar 2026 16:46:00 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1FBF5C4D; Mon, 30 Mar 2026 16:43:02 +0200 (CEST) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH-SERIES cluster/ha-manager/perl-rs/proxmox v3 00/40] dynamic scheduler + load rebalancer Date: Mon, 30 Mar 2026 16:30:09 +0200 Message-ID: <20260330144101.668747-1-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1774881612849 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.065 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: VBHJPRATFFC54ALTRRGJ6H47NMKVBUAP X-Message-ID-Hash: VBHJPRATFFC54ALTRRGJ6H47NMKVBUAP X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Most of the patches for the v3 are already R-b'd by @Dominik (many thanks!). A lot less has changed than from v1 -> v2, I've still added per-patch changelogs to make reviewing the rest more straightforward. Most changes went into - #05 - resource-scheduling: implement generic cluster usage implementation - #09 - resource-scheduling: implement rebalancing migration selection - #13 - pve-rs: resource-scheduling: use generic usage implementation - #19 - datacenter config: add auto rebalancing options - #30 - usage: use add_service to add service usage to nodes and the last few ha-manager patches should still get a review before applying is considered. I'll follow-up with the changes for pve-docs and pve-manager as soon as these patches are in a final state. Otherwise, the patches have been tested with building per-patch build-tests, I'll do some more testing with some Proxmox VE setups tomorrow and give an update here. Thank you very much for the feedback, @Dominik, @Thomas, @Maximiliano, and @Jillian Morgan! fixes v2 -> v3: - correctly set running state for HA resources in usage accounting - fix an error in Usage::add_resource_usage_to_node(), which correctly handles adding a non-existent node (would still move existing resource into moving state) and add a test case for that changes v2 -> v3: - rebased all repositories on master - move unset current_node invariant handling from proxmox-perl-rs to pve-ha-manager - made logic in (proxmox-resource-scheduling) Usage::add_resource() and Usage::add_resource_usage_to_node() more explicit by inling the logic - drop Usage::add_resource_to_nodes() - drop Usage::remove_resource_from_nodes() - drop Resource::moving_to() method - drop Resource::nodenames() and ResourcePlacement::nodenames() methods - s/to_string/to_owned/ where sensible - do not use bail!() but use assert!()s instead - do not compare imbalance score in scheduler tests, but only the order of the migrations returned by score_best_balancing_migration_candidates{,_topsis}() This patch series proposes an implementation for a dynamic scheduler and manual/automatic static/dynamic load rebalancer by implementing the following: - gather dynamic node and service usage information and use it in the dynamic scheduler, and - implement a load rebalancer, which actively moves HA resources to other nodes, to lower the overall cluster node imbalance, while adhering to the HA rules. == Model == The automatic load rebalancing system checks whether the cluster node imbalance exceeds some user-defined threshold for some HA Manager rounds ("hold duration"). If it does exceed on consecutive HA Manager rounds, it will choose the best service migration/relocation to improve the cluster node imbalance and queue it if it significantly improves it by some user-defined improvement ("margin"). The best service motion can be selected by either bruteforce or TOPSIS. This selection method and some other parameters from above can be tweaked at runtime. == Tests == I've added some test cases to ensure more basic decisions are documented. The other tests were in virtualized clusters with adding load dynamically to guests with stress-ng, even though I plan to rely more on real-world load simulators for the next batch of tests. == Benchmarks == I've also done some theoretical benchmarks with the target of being able to handle a 48 nodes cluster and 9.999 HA resources / guests and a worst-case scenario of each HA resource being part of 3 HA rules (pairwise positive and negative resource affinity rules, where each positive resource affinity pair has a common node affinity rule). Generating the migration candidates for the huge cluster with the worst-case HA ruleset takes 243 +- 9 ms. Generating the migration candidates for the huge cluster without the worst-case HA ruleset (to gain the most amount of 459954 migration candidates) takes 356 +- 6 ms. This is expected, because we need to evaluate more HA resources' rules as there are no HA resource bundles. Excluding the generation, the brute force and TOPSIS method for select_best_balancing_migration() were roughly similar both being in the range 350 +- 50 ms for the huge cluster without any HA rules (for the maximum amount of migration candidates) including the serialization between Perl and Rust. == Future ideas == - include the migration costs in score_best_balancing_migrations(), e.g., so that VMs with lots of memory are less likely to be migrated if the link between the nodes is slow, but that would need measuring and storing the migration network link speeds as a mesh - apply some filter like moving average window or exponential smoothing on the usage time series to dampen spikes; triple exponential smoothing (Holts-Winters) is also already implemented in rrdcached and allows for exponential smoothing with better time series analysis but would require changing the rrdcached data structure once more - score_best_balancing_migrations(...) can already provide a size-limited list of the best migrations, which could be exposed to users to allow manual load balancing actions, e.g., from the web interface, to get some insight in the system - The current scheduler can only solve bin covering, but it would be interesting to also allow bin packing if certain criteria are met, e.g., for energy preservation while the overall cluster load is low - Allow individual HA resources to be actively excluded from the automatic rebalancing, e.g., because containers cannot be live migrated. - move the migration candidate generation to the rust-side; the generation on the perl-side was chosen first to reduce code duplication, but it doesn't seem future proof and right to copy state to the online_node_usage object twice (medium priority) == Diffstat == proxmox: Daniel Kral (9): resource-scheduling: inline add_cpu_usage in score_nodes_to_start_service resource-scheduling: move score_nodes_to_start_service to scheduler crate resource-scheduling: rename service to resource where appropriate resource-scheduling: introduce generic scheduler implementation resource-scheduling: implement generic cluster usage implementation resource-scheduling: topsis: handle empty criteria without panics resource-scheduling: compare by nodename in score_nodes_to_start_resource resource-scheduling: factor out topsis alternative mapping resource-scheduling: implement rebalancing migration selection proxmox-resource-scheduling/src/lib.rs | 9 + proxmox-resource-scheduling/src/node.rs | 96 +++++ proxmox-resource-scheduling/src/pve_static.rs | 102 ++--- proxmox-resource-scheduling/src/resource.rs | 117 +++++ proxmox-resource-scheduling/src/scheduler.rs | 407 ++++++++++++++++++ proxmox-resource-scheduling/src/topsis.rs | 6 +- proxmox-resource-scheduling/src/usage.rs | 208 +++++++++ .../tests/scheduler.rs | 379 ++++++++++++++++ proxmox-resource-scheduling/tests/usage.rs | 181 ++++++++ 9 files changed, 1437 insertions(+), 68 deletions(-) create mode 100644 proxmox-resource-scheduling/src/node.rs create mode 100644 proxmox-resource-scheduling/src/resource.rs create mode 100644 proxmox-resource-scheduling/src/scheduler.rs create mode 100644 proxmox-resource-scheduling/src/usage.rs create mode 100644 proxmox-resource-scheduling/tests/scheduler.rs create mode 100644 proxmox-resource-scheduling/tests/usage.rs perl-rs: Daniel Kral (7): pve-rs: resource-scheduling: remove pedantic error handling from remove_node pve-rs: resource-scheduling: remove pedantic error handling from remove_service_usage pve-rs: resource-scheduling: move pve_static into resource_scheduling module pve-rs: resource-scheduling: use generic usage implementation pve-rs: resource-scheduling: static: replace deprecated usage structs pve-rs: resource-scheduling: implement pve_dynamic bindings pve-rs: resource-scheduling: expose auto rebalancing methods pve-rs/Makefile | 1 + pve-rs/src/bindings/mod.rs | 3 +- .../src/bindings/resource_scheduling/mod.rs | 10 + .../resource_scheduling/pve_dynamic.rs | 227 ++++++++++++++++++ .../resource_scheduling/pve_static.rs | 225 +++++++++++++++++ .../bindings/resource_scheduling/resource.rs | 126 ++++++++++ .../src/bindings/resource_scheduling/usage.rs | 81 +++++++ .../bindings/resource_scheduling_static.rs | 215 ----------------- pve-rs/test/resource_scheduling.pl | 1 + 9 files changed, 672 insertions(+), 217 deletions(-) create mode 100644 pve-rs/src/bindings/resource_scheduling/mod.rs create mode 100644 pve-rs/src/bindings/resource_scheduling/pve_dynamic.rs create mode 100644 pve-rs/src/bindings/resource_scheduling/pve_static.rs create mode 100644 pve-rs/src/bindings/resource_scheduling/resource.rs create mode 100644 pve-rs/src/bindings/resource_scheduling/usage.rs delete mode 100644 pve-rs/src/bindings/resource_scheduling_static.rs cluster: Daniel Kral (3): datacenter config: restructure verbose description for the ha crs option datacenter config: add dynamic load scheduler option datacenter config: add auto rebalancing options src/PVE/DataCenterConfig.pm | 53 ++++++++++++++++++++++++++++++++++--- 1 file changed, 49 insertions(+), 4 deletions(-) ha-manager: Daniel Kral (15): env: pve2: implement dynamic node and service stats usage: pass service data to add_service_usage usage: pass service data to get_used_service_nodes add running flag to non-HA cluster service stats usage: use add_service to add service usage to nodes usage: add dynamic usage scheduler test: add dynamic usage scheduler test cases manager: rename execute_migration to queue_resource_motion manager: update_crs_scheduler_mode: factor out crs config implement automatic rebalancing test: add resource bundle generation test cases test: add dynamic automatic rebalancing system test cases test: add static automatic rebalancing system test cases test: add automatic rebalancing system test cases with TOPSIS method test: add automatic rebalancing system test cases with affinity rules Dominik Rusovac (6): sim: hardware: pass correct types for static stats sim: hardware: factor out static stats' default values sim: hardware: fix static stats guard sim: hardware: handle dynamic service stats sim: hardware: add set-dynamic-stats command sim: hardware: add getters for dynamic {node,service} stats debian/pve-ha-manager.install | 1 + src/PVE/HA/Env.pm | 12 + src/PVE/HA/Env/PVE2.pm | 64 +++++ src/PVE/HA/Manager.pm | 220 +++++++++++++++- src/PVE/HA/Rules/ResourceAffinity.pm | 3 +- src/PVE/HA/Sim/Env.pm | 12 + src/PVE/HA/Sim/Hardware.pm | 185 ++++++++++++-- src/PVE/HA/Sim/RTHardware.pm | 4 +- src/PVE/HA/Usage.pm | 64 +++-- src/PVE/HA/Usage/Basic.pm | 9 +- src/PVE/HA/Usage/Dynamic.pm | 155 ++++++++++++ src/PVE/HA/Usage/Makefile | 2 +- src/PVE/HA/Usage/Static.pm | 63 ++++- src/test/Makefile | 1 + .../README | 2 + .../cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 1 + .../hardware_status | 5 + .../log.expect | 11 + .../manager_status | 1 + .../service_config | 1 + .../static_service_stats | 1 + .../README | 7 + .../cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 3 + .../hardware_status | 5 + .../log.expect | 25 ++ .../manager_status | 1 + .../service_config | 3 + .../static_service_stats | 3 + .../README | 4 + .../cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../service_config | 6 + .../static_service_stats | 6 + .../README | 4 + .../cmdlist | 16 ++ .../datacenter.cfg | 8 + .../dynamic_service_stats | 9 + .../hardware_status | 5 + .../log.expect | 80 ++++++ .../manager_status | 1 + .../service_config | 9 + .../static_service_stats | 9 + .../README | 11 + .../cmdlist | 13 + .../datacenter.cfg | 9 + .../dynamic_service_stats | 9 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../service_config | 9 + .../static_service_stats | 9 + .../test-crs-dynamic-auto-rebalance0/README | 2 + .../test-crs-dynamic-auto-rebalance0/cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 1 + .../hardware_status | 5 + .../log.expect | 11 + .../manager_status | 1 + .../service_config | 1 + .../static_service_stats | 1 + .../test-crs-dynamic-auto-rebalance1/README | 7 + .../test-crs-dynamic-auto-rebalance1/cmdlist | 3 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 3 + .../hardware_status | 5 + .../log.expect | 25 ++ .../manager_status | 1 + .../service_config | 3 + .../static_service_stats | 3 + .../test-crs-dynamic-auto-rebalance2/README | 4 + .../test-crs-dynamic-auto-rebalance2/cmdlist | 3 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../service_config | 6 + .../static_service_stats | 6 + .../test-crs-dynamic-auto-rebalance3/README | 4 + .../test-crs-dynamic-auto-rebalance3/cmdlist | 16 ++ .../datacenter.cfg | 7 + .../dynamic_service_stats | 9 + .../hardware_status | 5 + .../log.expect | 80 ++++++ .../manager_status | 1 + .../service_config | 9 + .../static_service_stats | 9 + .../test-crs-dynamic-auto-rebalance4/README | 11 + .../test-crs-dynamic-auto-rebalance4/cmdlist | 13 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 9 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../service_config | 9 + .../static_service_stats | 9 + .../README | 7 + .../cmdlist | 8 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 5 + .../hardware_status | 5 + .../log.expect | 49 ++++ .../manager_status | 1 + .../rules_config | 4 + .../service_config | 5 + .../static_service_stats | 5 + .../README | 12 + .../cmdlist | 8 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 4 + .../hardware_status | 5 + .../log.expect | 53 ++++ .../manager_status | 1 + .../rules_config | 3 + .../service_config | 4 + .../static_service_stats | 4 + .../README | 14 ++ .../cmdlist | 3 + .../datacenter.cfg | 8 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 31 +++ .../rules_config | 3 + .../service_config | 6 + .../static_service_stats | 6 + .../README | 14 ++ .../cmdlist | 3 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 6 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../rules_config | 7 + .../service_config | 6 + .../static_service_stats | 6 + src/test/test-crs-dynamic-rebalance1/README | 3 + src/test/test-crs-dynamic-rebalance1/cmdlist | 4 + .../datacenter.cfg | 7 + .../dynamic_service_stats | 7 + .../hardware_status | 5 + .../test-crs-dynamic-rebalance1/log.expect | 82 ++++++ .../manager_status | 1 + .../service_config | 7 + .../static_service_stats | 7 + src/test/test-crs-dynamic1/README | 4 + src/test/test-crs-dynamic1/cmdlist | 4 + src/test/test-crs-dynamic1/datacenter.cfg | 6 + .../test-crs-dynamic1/dynamic_service_stats | 3 + src/test/test-crs-dynamic1/hardware_status | 5 + src/test/test-crs-dynamic1/log.expect | 51 ++++ src/test/test-crs-dynamic1/manager_status | 1 + src/test/test-crs-dynamic1/service_config | 3 + .../test-crs-dynamic1/static_service_stats | 3 + .../test-crs-static-auto-rebalance1/README | 7 + .../test-crs-static-auto-rebalance1/cmdlist | 3 + .../datacenter.cfg | 7 + .../hardware_status | 5 + .../log.expect | 25 ++ .../manager_status | 1 + .../service_config | 3 + .../static_service_stats | 3 + .../test-crs-static-auto-rebalance2/README | 4 + .../test-crs-static-auto-rebalance2/cmdlist | 3 + .../datacenter.cfg | 7 + .../hardware_status | 5 + .../log.expect | 59 +++++ .../manager_status | 1 + .../service_config | 6 + .../static_service_stats | 6 + .../test-crs-static-auto-rebalance3/README | 3 + .../test-crs-static-auto-rebalance3/cmdlist | 15 ++ .../datacenter.cfg | 7 + .../hardware_status | 5 + .../log.expect | 79 ++++++ .../manager_status | 1 + .../service_config | 9 + .../static_service_stats | 9 + src/test/test_resource_bundles.pl | 234 ++++++++++++++++++ 187 files changed, 2813 insertions(+), 50 deletions(-) create mode 100644 src/PVE/HA/Usage/Dynamic.pm create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis0/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis1/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis2/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis3/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance-topsis4/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance0/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance1/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance2/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance3/static_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance4/static_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/README create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/cmdlist create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/hardware_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/log.expect create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/manager_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/rules_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/service_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance1/static_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/README create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/cmdlist create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/hardware_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/log.expect create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/manager_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/rules_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/service_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance2/static_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/README create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/cmdlist create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/hardware_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/log.expect create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/manager_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/rules_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/service_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance3/static_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/README create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/cmdlist create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/hardware_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/log.expect create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/manager_status create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/rules_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/service_config create mode 100644 src/test/test-crs-dynamic-constrained-auto-rebalance4/static_service_stats create mode 100644 src/test/test-crs-dynamic-rebalance1/README create mode 100644 src/test/test-crs-dynamic-rebalance1/cmdlist create mode 100644 src/test/test-crs-dynamic-rebalance1/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-rebalance1/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-rebalance1/hardware_status create mode 100644 src/test/test-crs-dynamic-rebalance1/log.expect create mode 100644 src/test/test-crs-dynamic-rebalance1/manager_status create mode 100644 src/test/test-crs-dynamic-rebalance1/service_config create mode 100644 src/test/test-crs-dynamic-rebalance1/static_service_stats create mode 100644 src/test/test-crs-dynamic1/README create mode 100644 src/test/test-crs-dynamic1/cmdlist create mode 100644 src/test/test-crs-dynamic1/datacenter.cfg create mode 100644 src/test/test-crs-dynamic1/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic1/hardware_status create mode 100644 src/test/test-crs-dynamic1/log.expect create mode 100644 src/test/test-crs-dynamic1/manager_status create mode 100644 src/test/test-crs-dynamic1/service_config create mode 100644 src/test/test-crs-dynamic1/static_service_stats create mode 100644 src/test/test-crs-static-auto-rebalance1/README create mode 100644 src/test/test-crs-static-auto-rebalance1/cmdlist create mode 100644 src/test/test-crs-static-auto-rebalance1/datacenter.cfg create mode 100644 src/test/test-crs-static-auto-rebalance1/hardware_status create mode 100644 src/test/test-crs-static-auto-rebalance1/log.expect create mode 100644 src/test/test-crs-static-auto-rebalance1/manager_status create mode 100644 src/test/test-crs-static-auto-rebalance1/service_config create mode 100644 src/test/test-crs-static-auto-rebalance1/static_service_stats create mode 100644 src/test/test-crs-static-auto-rebalance2/README create mode 100644 src/test/test-crs-static-auto-rebalance2/cmdlist create mode 100644 src/test/test-crs-static-auto-rebalance2/datacenter.cfg create mode 100644 src/test/test-crs-static-auto-rebalance2/hardware_status create mode 100644 src/test/test-crs-static-auto-rebalance2/log.expect create mode 100644 src/test/test-crs-static-auto-rebalance2/manager_status create mode 100644 src/test/test-crs-static-auto-rebalance2/service_config create mode 100644 src/test/test-crs-static-auto-rebalance2/static_service_stats create mode 100644 src/test/test-crs-static-auto-rebalance3/README create mode 100644 src/test/test-crs-static-auto-rebalance3/cmdlist create mode 100644 src/test/test-crs-static-auto-rebalance3/datacenter.cfg create mode 100644 src/test/test-crs-static-auto-rebalance3/hardware_status create mode 100644 src/test/test-crs-static-auto-rebalance3/log.expect create mode 100644 src/test/test-crs-static-auto-rebalance3/manager_status create mode 100644 src/test/test-crs-static-auto-rebalance3/service_config create mode 100644 src/test/test-crs-static-auto-rebalance3/static_service_stats create mode 100755 src/test/test_resource_bundles.pl Summary over all repositories: 206 files changed, 4971 insertions(+), 339 deletions(-) -- Generated by murpp 0.11.0